Tools
VRAM for local LLMs
Pick the model size, its quantization and the context length: the tool separates what the weights, the KV cache, the activations and the runtime overhead each take, then checks the total against common cards. The KV cache is nearly always the surprise: it grows linearly with context and concurrent requests, which is why a model that loads fine at 4K blows up at 64K.
Values below come from each model's published config.json. Editing any of them switches to a custom architecture.
Billions of parameters, experts included. Drives the weights term and nothing else.
num_hidden_layers. The cache is stored once per layer.
num_key_value_heads, not the attention head count. Grouped-query attention cuts it to 8 or fewer on most modern models.
head_dim, usually hidden_size divided by the attention head count.
hidden_size. Only the activations term uses it.
Bytes per parameter. Q4_K_M is the usual sweet spot for local inference: about a quarter of FP16 for a quality loss most people do not notice.
Quantizing the cache is the cheapest way to buy context length, and usually costs less quality than quantizing the weights further.
Positions held in the cache. Double it and you double the KV term, which is why a model that loads fine at 4K runs out of memory at 64K.
Requests in flight at once. Each carries its own cache, so serving several users multiplies that term while the weights stay shared.
Total VRAM
6.20 GiB
weights + cache + overhead
Weights
4.34 GiB
70 %
KV cache
1.00 GiB
16 %
Smallest card that fits
10 GB
RTX 3080 10 GB
- Weights4.34 GiB70.0 %Parameters times bytes per parameter. Fixed for a given checkpoint and quantization.
- KV cache1.00 GiB16.1 %Grows linearly with context length and batch size. This is what makes long contexts expensive to serve.
- Activations64.0 MiB1.0 %Transient buffers during the forward pass, peaking on the prefill chunk.
- Runtime overhead819 MiB12.9 %CUDA context, kernels and allocator slack. Roughly constant, around 0.8 GiB.
- RTX 3060 12 GB52 %
- RTX 3080 10 GB62 %
- RTX 3090 / Ti 24 GB26 %
- RTX 4060 Ti 16 GB39 %
- RTX 4070 / Super 12 GB52 %
- RTX 4070 Ti Super 16 GB39 %
- RTX 4080 / Super 16 GB39 %
- RTX 4090 24 GB26 %
- RTX 5060 Ti 16 GB39 %
- RTX 5070 12 GB52 %
- RTX 5070 Ti 16 GB39 %
- RTX 5080 16 GB39 %
- RTX 5090 32 GB19 %
An estimate, not a guarantee. Real usage moves with the inference engine, the attention implementation, whether the cache is paged, and how much the allocator fragments. Leave a margin of ten percent or so before concluding that a model fits.