How much VRAM do you need to run an LLM locally?
Will it fit your GPU? Normal vs Pingala-8 calculator
In short
- GPU memory needed = the model file + the KV cache for your context + working buffers + what your desktop already uses.
- The file is roughly parameters x bits per weight / 8. Qwen3-4B is 8.05 GB at 16-bit, about 2.5 GB at Q4_K_M and 1.24 GB at 2-bit.
- The KV cache grows with context. For Qwen3-4B it is about 0.6 GB at 4,096 tokens (16-bit cache, calculated).
- On paper, Qwen3-4B fits a 6 GB card at Q4_K_M and a 4 GB card at 2-bit. The tables below show the arithmetic.
If a model does not fit in your graphics card's memory (VRAM), it either fails to load or spills into system RAM, and then it runs far slower. So the first question for any local LLM is simple: will it fit? You can answer it in three steps, before you download anything.
What uses GPU memory
Three things share the card when you run a model with llama.cpp, Ollama or LM Studio:
- The weights. This is the model file itself, loaded in full. It is usually the biggest part, and the one you control by choosing a quantization such as Q8_0, Q4_K_M or a 2-bit format.
- The KV cache. As the model reads your chat, it keeps a key and a value vector for every token in every layer, so it does not have to recompute them. This is the model's working memory of the conversation, and it grows in a straight line with the context length you set.
- Working buffers and the desktop. The runtime needs scratch space for intermediate results (llama.cpp calls it the compute buffer). On a laptop or desktop, the operating system, the browser and the screen already use part of the VRAM.
Step 1: the size of the weights
A model file is close to its number of weights times the bits stored per weight, divided by 8 to get bytes. Qwen3-4B has about 4.0 billion weights. At 16 bits that is 4.0 x 16 / 8 = 8.0 GB, and the real fp16 file is 8.05 GB.
For quantized files, use the real file size rather than the name. A "2-bit" file is never exactly 2 bits per weight: formats store scales next to the weights and keep some tensors, such as the embedding table, at higher precision. Averaged over the whole file, our Pingala-8 Qwen3-4B file is 2.47 bits per weight and llama.cpp's IQ2_XXS is 2.48. Here are measured sizes for one model:
| Qwen3-4B file | Size | Bits per weight | Perplexity | Runs in |
|---|---|---|---|---|
| fp16 (original) | 8.05 GB | 16.01 | 12.46 | Any tool |
| Q8_0 | 4.28 GB | 8.51 | 12.50 | Stock llama.cpp, Ollama, LM Studio |
| Q4_K_M | about 2.5 GB | - | not measured | Stock llama.cpp, Ollama, LM Studio |
| Q2_K | 1.67 GB | 3.32 | 15.01 | Stock llama.cpp, Ollama, LM Studio |
| IQ2_XS | 1.35 GB | 2.69 | 19.02 | Stock llama.cpp, Ollama, LM Studio |
| IQ2_XXS | 1.25 GB | 2.48 | 28.61 | Stock llama.cpp, Ollama, LM Studio |
| Pingala-8 | 1.24 GB | 2.47 | 16.64 | Our patched llama.cpp, NVIDIA CUDA only |
| IQ1_M | 1.13 GB | 2.24 | 57.50 | Stock llama.cpp, Ollama, LM Studio |
Sizes are file bytes divided by 10^9. Perplexity is WikiText-2 at context 4,096, the same test for every file; lower is better (what perplexity means). The Q4_K_M size is Qwen's official file, 2,497,280,256 bytes; we did not measure its perplexity. Source: the benchmark dataset and the paper.
Step 2: add the KV cache
The KV cache depends on the shape of the model, not on how the file is quantized. For each token it stores 2 vectors (a key and a value) x the number of layers x the key-value heads x the head size x the bytes per number. Qwen3-4B's published config has 36 layers and 8 key-value heads of size 128, so at 16-bit each token takes 2 x 36 x 8 x 128 x 2 = 147,456 bytes, about 0.15 MB.
| Context (tokens) | 2,048 | 4,096 | 8,192 | 16,384 | 32,768 |
|---|---|---|---|---|---|
| 16-bit cache (default) | 0.30 GB | 0.60 GB | 1.21 GB | 2.42 GB | 4.83 GB |
| 8-bit cache (q8_0) | 0.16 GB | 0.32 GB | 0.64 GB | 1.28 GB | 2.57 GB |
Qwen3-4B, calculated from the model config, not measured. Llama-3.1-8B (32 layers, 8 key-value heads of size 128) takes 131,072 bytes per token at 16-bit: 0.54 GB at 4,096 tokens.
Two settings shrink the cache. A shorter context (-c in llama.cpp) is the simplest. llama.cpp can also store the cache at 8 bits with -ctk q8_0 -ctv q8_0, which roughly halves it; the 8-bit V cache needs flash attention on (-fa on). This changes the output slightly, and we have not tested it with Pingala-8.
Step 3: leave headroom, then check
Add the compute buffer and whatever the desktop is using. Both depend on the program, its settings and your system, so the reliable way is to look. Run nvidia-smi to see how much VRAM is free before you start, then load the model: llama.cpp prints the size of the model buffer, the KV cache and the compute buffer as it starts. If the total is close to your free memory, cut the context first.
What fits on 4, 6, 8 and 12 GB
Steps 1 and 2 together, for Qwen3-4B with a 4,096-token context and a 16-bit cache (0.60 GB). "Tight" means less than 1.5 GB is left for buffers and the desktop, so expect to shorten the context or close other apps.
| Qwen3-4B file | Weights + cache | 4 GB | 6 GB | 8 GB | 12 GB |
|---|---|---|---|---|---|
| fp16 | 8.65 GB | No | No | No | Yes |
| Q8_0 | 4.88 GB | No | Tight | Yes | Yes |
| Q4_K_M | about 3.1 GB | Tight | Yes | Yes | Yes |
| Q2_K | 2.27 GB | Yes | Yes | Yes | Yes |
| IQ2_XS | 1.95 GB | Yes | Yes | Yes | Yes |
| IQ2_XXS | 1.85 GB | Yes | Yes | Yes | Yes |
| Pingala-8 | 1.84 GB | Yes | Yes | Yes | Yes |
Arithmetic, not a test on these cards. Our speed and quality numbers come from one RTX 5070 Ti Laptop GPU.
Where Pingala-8 fits
Pingala-8 is a 2-bit format we built for the case where the card is the limit. Qwen3-4B goes from 8.05 GB to 1.24 GB and Llama-3.1-8B from 16.08 GB to 2.39 GB, so an 8B model plus a 4,096-token cache (0.54 GB) comes to about 2.9 GB on paper. At the same file size as IQ2_XXS, Pingala-8 Qwen3-4B has a WikiText-2 perplexity of 16.64 against 28.61, and a five-quiz average of 50.8 against 41.9 (the 8-bit model scores 56.3). The bytes you save can go to a longer context instead.
Where it does not
- If Q4_K_M fits, use it. It is a bigger 4-bit file and runs in every common tool. Pingala-8 is for when it does not fit, or when you want the room for context.
- It needs NVIDIA and our runtime. Pingala-8 runs on our patched llama.cpp with CUDA only. Stock llama.cpp, Ollama and LM Studio cannot load it, and there is no Mac (Metal), AMD or Vulkan path.
- It suits one user. It reads long prompts at 0.74x the speed of IQ2_XXS, and serves 2 to 8 chats at once at 0.18x to 0.34x.
- A little more memory buys a lot. With 1.67 GB to spare instead of 1.24 GB, stock llama.cpp's Q2_K scores 15.01 on the same test, lower than our 16.64.
Questions people ask
How much VRAM do I need for a 4B model?
Qwen3-4B needs 8.05 GB for its weights at 16-bit, about 2.5 GB at Q4_K_M and 1.24 GB with a 2-bit format such as Pingala-8 or IQ2_XXS. Add about 0.6 GB of KV cache for a 4,096-token context (16-bit cache, calculated from the model config), plus working buffers and whatever your desktop uses.
How much VRAM do I need for an 8B model?
Llama-3.1-8B is 16.08 GB at 16-bit and 2.39 GB with Pingala-8. Its KV cache takes about 0.54 GB at 4,096 tokens (calculated). A 4-bit file sits in between; check its real file size on the download page.
Can I run an LLM on a 4 GB GPU?
On paper, yes for a 4B model in a 2-bit or 3-bit file: Qwen3-4B at 1.24 to 1.67 GB plus a 0.6 GB cache leaves room on a 4 GB card. Q4_K_M (about 2.5 GB) is tight. Keep the context short and close apps that use the GPU.
What happens if the model does not fit in VRAM?
llama.cpp can keep some layers on the GPU and the rest in system RAM (set fewer GPU layers with -ngl). The model still runs, but it writes much more slowly, because system RAM is slower to read than VRAM.
Try Pingala-8
More guides
- What is a 2-bit LLM, and is 2-bit quantization usable?
- IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
- What is perplexity? LLM perplexity in plain English
- How to run Qwen3-4B on a 4 GB or 6 GB GPU
- How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
- What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
- What is Hadamard rotation in LLM quantization?