What is a 2-bit LLM, and is 2-bit quantization usable?
In short
- A 2-bit LLM stores each weight in about 2 bits instead of 16, so the file is about 6 to 7 times smaller. Qwen3-4B goes from 8.05 GB to about 1.24 GB.
- It always costs quality, and how much depends more on the method than on the bit count. At the same 1.24 GB, WikiText-2 perplexity ranges from 16.64 to 28.61 (the original scores 12.46; lower is better).
- On five quizzes, the 2-bit files we tested at that size keep 74% to 90% of the 8-bit model's average.
- It can be usable for chat on a small GPU if you pick the method with care. Test it on your own task before you rely on it.
A language model is a long list of numbers, its weights. Qwen3-4B has about 4 billion of them, shipped as 16-bit numbers. Quantization stores each weight with fewer bits. At 8 bits the model barely changes; at 4 bits it changes a little; at 2 bits it is a real trade. This guide explains what happens at 2 bits, shows measured numbers for one model, and says when it is worth it.
What "2-bit" means
With 2 bits, a single weight can take only 4 values. Rounding billions of weights to one of 4 levels each would wreck a model, so 2-bit formats do more than that:
- Shared scales. Weights are stored in small blocks, each with its own scale, so the few levels fit the local range of values.
- Codebooks. llama.cpp's IQ2_XXS stores each group of 8 weights as one entry from a fixed table of patterns, which wastes fewer bits than coding each weight alone.
- Trellis codes. IQ2_KT (in ik_llama.cpp) and our Pingala-8 code groups in sequence, so the choice for one group depends on the ones before it. The idea comes from QTIP and, much earlier, from coding for modems.
- Rotations and error feedback. Methods in the QuIP line rotate the weights first, so a few large values do not dominate, and pass each column's rounding error on to the columns not yet coded.
Real files are a little above 2 bits per weight, because scales take space and some tensors stay at higher precision. Over the whole Qwen3-4B file, Pingala-8 averages 2.47 bits per weight and IQ2_XXS 2.48.
How much quality does 2-bit cost?
Here is one model, Qwen3-4B, compressed with several formats and measured with the same tests. Perplexity is how surprised the model is by real text (lower is better; see our perplexity guide). The quiz average covers five standard multiple-choice tests: ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande and MMLU.
| Qwen3-4B file | Size | Bits per weight | Perplexity | vs original | Quiz average | Share of 8-bit quiz |
|---|---|---|---|---|---|---|
| fp16 (original) | 8.05 GB | 16.01 | 12.46 | 1.00x | - | - |
| Q8_0 | 4.28 GB | 8.51 | 12.50 | 1.00x | 56.3 | 100% |
| Q2_K | 1.67 GB | 3.32 | 15.01 | 1.21x | - | - |
| IQ2_XS | 1.35 GB | 2.69 | 19.02 | 1.53x | 47.7 | 85% |
| IQ2_KT | 1.25 GB | 2.48 | 18.15 | 1.46x | 46.5 | 83% |
| IQ2_XXS | 1.25 GB | 2.48 | 28.61 | 2.30x | 41.9 | 74% |
| Pingala-8 | 1.24 GB | 2.47 | 16.64 | 1.34x | 50.8 | 90% |
| IQ1_M | 1.13 GB | 2.24 | 57.50 | 4.62x | - | - |
WikiText-2 test, context 4,096, the same harness for every file. A dash means not measured. The IQ2_KT row is the strongest of six IQ2_KT files we built at this size. The Pingala-8 Qwen3-4B file is the last of many builds we tested on WikiText-2, so treat its number as tuned. Source: the benchmark dataset and the paper.
Three things stand out.
- The method matters as much as the bits. IQ2_XXS, IQ2_KT and Pingala-8 are within 0.3% of each other in file size, yet their perplexity runs from 16.64 to 28.61 and their quiz averages from 41.9 to 50.8.
- Going smaller still costs a lot. IQ1_M, at 2.24 bits per weight, saves about 0.12 GB more than IQ2_XXS and roughly doubles its perplexity, to 57.50.
- A little more size helps a lot. Q2_K, at 3.32 bits per weight and 1.67 GB, reaches 15.01, closer to the original than any of the 1.25 GB files.
So, is it usable?
For a 4B model in our tests, it depends on the method. With a weak one the loss is large: IQ2_XXS keeps 74% of the 8-bit quiz average. With a strong one, Pingala-8 keeps 90%. Our demo shows Pingala-8 Qwen3-4B writing a short email; that is one example, not a measurement. In every case the 2-bit model is clearly weaker than the original (at best 1.34x its perplexity at this size), so 2-bit is a way to fit a model, not a free upgrade.
Model size matters too. The QuIP paper (NeurIPS 2023) found that the gap between 2-bit and 4-bit shrinks as models get bigger. Our Qwen3-14B files sit much closer together than the 4B ones: 9.40 for IQ2_KT, 9.57 for Pingala-8 and 10.54 for IQ2_XXS.
When 2-bit is a poor fit
- When a 4-bit file fits. If Q4_K_M (about 2.5 GB for Qwen3-4B) fits with room for your context, it keeps more of each weight. See how much VRAM you need.
- Tasks we did not measure. Our numbers cover text prediction and five quizzes. We have not measured coding, math, long reasoning, other languages or safety behaviour after compression.
- A smaller model at 4-bit. A smaller model at 4 bits can be the same size as a bigger one at 2 bits. We did not measure that trade, so compare the two on your own task.
- A model retrained for low precision. Ternary Bonsai 4B from prism-ml scores 11.59 at 1.07 GB on the same test, better than the original Qwen3-4B, because it is a retrained model rather than a compressed copy. If you only want a strong small model, that route can beat compression.
Where Pingala-8 fits
Pingala-8 is for compressing a model you already have, with no retraining, when memory is the limit and you have an NVIDIA GPU. Among the formats we tested at the IQ2_XXS size, it gives the lowest perplexity on Qwen3-4B (16.64) and on Llama-3.1-8B at 2.39 GB (9.14, against 10.04 for IQ2_KT and 12.35 for IQ2_XXS; that prediction was written down before the run). It loses on Qwen3-14B: 9.57 against 9.40 for IQ2_KT. It needs our patched llama.cpp with CUDA, so Ollama, LM Studio and Macs cannot run it, and it reads long prompts at 0.74x the speed of IQ2_XXS. For the full picture, see the GGUF quant comparison.
Questions people ask
What is a 2-bit LLM?
A language model whose weights are stored with about 2 bits each instead of the usual 16. Real 2-bit files average a little more, about 2.5 bits per weight over the whole file, because of scales and a few tensors kept at higher precision. Qwen3-4B goes from 8.05 GB to about 1.24 GB.
Is 2-bit quantization worth it?
When the model does not fit your memory any other way, often yes, if you pick the method with care. At the same 1.24 GB, Qwen3-4B files scored between 41.9 and 50.8 on a five-quiz average in our tests, against 56.3 for the 8-bit model. If a 4-bit file fits, use that instead.
Is 2-bit better than 1-bit?
For compressing an existing model, 2-bit loses much less. On Qwen3-4B, llama.cpp's IQ1_M (1.13 GB) has a WikiText-2 perplexity of 57.50, against 16.64 to 28.61 for the 1.24 GB 2-bit files. Models with ternary or 1-bit weights that work well are usually trained that way, which is a different route.
Does a 2-bit model run faster?
Writing a reply for one user is mostly limited by reading weights from memory, so a smaller file can write faster. On our GPU, Pingala-8 wrote at 1.10x the speed of IQ2_XXS and 1.39x the speed of Q4_K_M at batch 1. Reading prompts can be slower: 0.74x IQ2_XXS for Pingala-8.
Try Pingala-8
More guides
- How much VRAM do you need to run an LLM locally?
- IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
- What is perplexity? LLM perplexity in plain English
- How to run Qwen3-4B on a 4 GB or 6 GB GPU
- How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
- What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
- What is Hadamard rotation in LLM quantization?