How Much VRAM Does an LLM Actually Need?

Why OOM is almost never about the weights

The first time a 13B model hits torch.OutOfMemoryError halfway through loading, the instinct is to rent a bigger card. That instinct is usually wrong, and it is the expensive answer.

Weights load once and then never change. Everything else on the card is allocated per request and scales with how many people you serve at once. A 13B model in 4-bit leaves roughly 19 GB free on a 24 GB card — and a busy endpoint can eat all of it with context alone, while the weights sit there unchanged.

The four consumers of VRAM

The KV cache formula, written out

For each token you keep two tensors (a key and a value) for every layer and every key/value head. So the cost per token is:

2 × layers × kv_heads × head_dim × bytes_per_element

For a 32-layer model with 32 KV heads of dimension 128 in FP16 that is 2 × 32 × 32 × 128 × 2 = 524,288 bytes, or 0.5 MB per token. At 8K context that is 4 GB for a single request. At batch 8 it is 32 GB — and now the card is gone before the weights are even counted.

This is why grouped-query attention changed the economics. A 70B model with 80 layers but only 8 KV heads costs 2 × 80 × 8 × 128 × 2 = 320 KB per token — less per token than the 7B example above, despite having twenty times the parameters. Parameter count tells you the weight size. It tells you nothing useful about the cache.

Recompute it for the exact model you are deploying

import json

cfg = json.load(open("config.json"))

layers    = cfg["num_hidden_layers"]
hidden    = cfg["hidden_size"]
heads     = cfg["num_attention_heads"]
# GQA models expose fewer KV heads than attention heads.
# Falling back to `heads` overestimates KV cache by the
# group factor -- common mistake when sizing Llama-3.
kv_heads  = cfg.get("num_key_value_heads", heads)
head_dim  = cfg.get("head_dim", hidden // heads)

def kv_bytes_per_token(dtype_bytes=2):
    return 2 * layers * kv_heads * head_dim * dtype_bytes

def total_gb(params_b, ctx, batch=1,
             weight_bpp=2, kv_bytes=2, overhead_gb=1.5):
    weights = params_b * 1e9 * weight_bpp / 1024**3
    kv = (kv_bytes_per_token(kv_bytes) * ctx * batch) / 1024**3
    # activations + fragmentation, roughly 15% of weights
    acts = weights * 0.15
    return weights + kv + acts + overhead_gb

print(f"KV/token: {kv_bytes_per_token()/1024:.0f} KB")
for ctx in (4096, 8192, 32768):
    for b in (1, 8):
        print(f"ctx={ctx:6} batch={b}: "
              f"{total_gb(7, ctx, b):5.1f} GB")

Order of operations when the card is too small

• Quantize the weights first — 4-bit cuts the largest fixed cost with almost no engineering effort.
• Quantize the KV cache too. Many people stop after the weights and find the freed memory immediately absorbed by the cache.
• Cut context length before cutting concurrency. Long context is usually a feature request, not a hard requirement.
• Reduce the model. A smaller model improves concurrency and latency at the same time — it is a structural win, not a trick.
• Only then rent a bigger card, and only if precision or context length is a hard constraint rather than a nice-to-have.

Do this before renting anything

• Step 1. Read config.json and note layers, attention heads, and KV heads.
• Step 2. Compute KV bytes per token with the formula above.
• Step 3. Multiply by the context length and the concurrency you actually expect — not the concurrency you hope for.
• Step 4. Add weights at the precision you can afford to run.
• Step 5. Pick the smallest card that leaves 20% headroom, and test at the batch size you sized for.