Genrius AI articles

8 open models, 77-85% smaller: what 2-bit quantization costs and what it keeps

Harshit Goyal, Founder, Genrius AI (ex-Google software engineer, IIIT Delhi alumnus). November 1, 2026. About 6 minutes.

First, what perplexity means

Every number below that is not a size or a speed is a perplexity, so it is worth a minute on what it measures.

A language model reads text one token at a time (a token is a word or a piece of a word) and, before each token, gives a probability to every token that could come next. Perplexity takes the probability the model gave to the token that actually came next, averages its logarithm over a long text, and turns the average back into a plain number. The result has a simple reading: a perplexity of 7 means that, on average, the model was as unsure as if it were choosing among 7 equally likely tokens. A perfect predictor would score 1. Lower is better.

For compression, perplexity is a damage meter. Rounding the weights to 2 bits makes the model a little less sure of the right next token, and perplexity goes up. Two files of the same model and the same size can then be compared directly: the one with the lower perplexity kept more of the original.

Three cautions:

• It is one text. All of our numbers are on WikiText-2, English Wikipedia articles. It says little about code, other languages or long conversations.
• It is not a task score. Lower perplexity usually goes with better answers, but not one for one. On Qwen3-4B, our 41.8% lower perplexity went with a 21% higher average on five quizzes (50.8 against 41.9), which is 90% of the 8-bit score. On Qwen3.5-4B, a 25.6% drop went with 53.8 against 49.1 on zero-shot tasks.
• Only compare like with like. Same model, same text, same context length, same windows. A perplexity from one model card and one from another are often not comparable. Our benchmarking article goes into this.

For more, see what perplexity is, in plain English.

How we measured

• Smaller: the Pingala-8 file in bytes against the model's 16-bit text weights, computed from the base repository's safetensors headers, with vision and multi-token-prediction tensors left out. For Llama-3.1-8B we use the 16.08 GB on its model card.
• Lower perplexity: WikiText-2 at context 4,096, against llama.cpp's IQ2_XXS at about the same file size, built with the same importance matrix (imatrix) we used for our file.
• Speed: token generation for one user (batch 1) against IQ2_XXS of the same model, on the same laptop GPU, an RTX 5070 Ti Laptop with 12 GB. The figures come from each model card.

The eight models

On a phone, swipe the table sideways. Perplexity is WikiText-2, lower is better; "Lower PPL" is how much lower ours is than IQ2_XXS. Speed is token generation at batch 1 against IQ2_XXS of the same model unless the row says otherwise. Each model name links to its Hugging Face page.

What 2-bit keeps

Size: 77-85% smaller

A pure 2-bit file would be 87.5% smaller than 16 bits (2 out of 16). No row reaches that, because some parts of each file keep more bits: the per-block scales, a few sensitive tensors, and above all the token embedding and output head. That last table is large in small models. In Qwen3.5-2B the embedding and output head are one shared matrix, and at the 4-bit Q4_K type it takes 286 MB of a 778 MB file. That is a large part of why the smallest Qwen3.5 models shrink least (77.0% and 79.3%) and the 8B and 27B models shrink most.

Quality: the gap to IQ2_XXS grows as models shrink

Read the "Lower PPL" column from top to bottom. On Qwen3.8-27B our file is 12.6% lower than IQ2_XXS; on the 4B and 8B models about 26-42%; on the models of 2B and below 41-70%; and on Qwen3-0.6B, IQ2_XXS reaches a perplexity of 10,791, which means its output is close to noise, while ours is 41.34. In this table, the smaller the model, the harder 2 bits hits a simple format, and the more a careful coder saves. Large models seem to have more slack, so even the simple format holds up better on them.

The flip side: on the large models the differences between good 2-bit files are small, a few percent, and need careful testing to tell apart.

Speed: about the same as IQ2_XXS

Writing a reply for one user is mostly a matter of reading the weights from memory once per token, so files of the same size tend to run at about the same speed. Ours matches or beats IQ2_XXS on 5 of the 6 models measured against it, from +0% to +9%, with 37.7 against 36.4 tokens/s on Qwen3.8-27B. On Qwen3-4B, our 1.24 GB file is 39% faster than the 4-bit Q4_K_M file, which is twice its size.

What 2-bit costs, and where we lose

The table compares us with IQ2_XXS because it is the common llama.cpp format at this size. It is not the strongest rival. These are the places where we are behind:

• Prompt speed. Reading a prompt runs at 0.74-0.89x the speed of IQ2_XXS. A long document takes longer before the first word of the reply appears.
• Qwen3-0.6B speed. Generation is 30% slower than IQ2_XXS on the smallest model.
• DASH-Q. On Qwen3.8-27B, DASH-Q IQ2_XS scores 6.54 against our 6.89. On Qwen3.5-2B, DASH-Q IQ2_XS scores 13.90 and IQ2_M 13.33, against our 14.06. Both of our version 2 files copy DASH-Q's way of splitting bits between tensors (how we did it), and DASH-Q is still ahead.
• IQ2_KT. ik_llama.cpp's IQ2_KT type beats us on Qwen3-1.7B, 20.89 against 28.23, and narrowly on Qwen3-14B, 9.40 against 9.57.
• Retrained ternary models. Bonsai 4B scores 11.59 at 1.07 GB, better and smaller than our Qwen3-4B file (16.64 at 1.24 GB). Bonsai is a model trained to work with very few bits, not a compression of an existing one. If you just want a strong small model, a retrained one like it is the better choice. Pingala-8 is for compressing models nobody has retrained for you.
• Runtime. Our files need our patched llama.cpp runtime on an NVIDIA GPU with CUDA. Stock llama.cpp, Ollama and LM Studio cannot load them. DASH-Q, unsloth and IQ2_XXS files run in stock llama.cpp.
• Gaps in the speed column. We have not measured Llama-3.1-8B speed or benchmarked the Qwen3.5-2B version 2 file, and the version 1 2B figure is noisy.

When 2-bit makes sense

2-bit is a trade. You give up some quality, measured above, to run a model on hardware that could not hold it otherwise. The clearest case in this table is Qwen3.8-27B: 53.8 GB of weights becomes 8.31 GB and runs fully on one 12 GB laptop GPU. The 12 GB guide lists the three 2-bit files that fit, ours and two rivals. For small models on small cards, the Qwen3-4B and Llama 3.1 8B guides show what fits on 4 and 6 GB.

If your GPU can hold a 4-bit or 8-bit file of the model you want, those keep more of the original; our tables do not argue otherwise. And at 2 bits, check the rival files for your model before picking ours. The 2-bit GGUF leaderboard ranks every file we measured, and DASH-Q is first on both models it covers.

Our guide to 2-bit quantization and the GGUF quant comparison cover the formats in more depth.

More articles

• How we fit a 27B model on a 12 GB laptop GPU
• Benchmarking 2-bit GGUFs fairly: paired windows, and the NaN trap on RTX 50 GPUs
• How we fit Qwen3-4B into 1.24 GB without retraining

Questions, corrections or a model you want compressed next? Open a GitHub issue.