How we fit Qwen3-4B into 1.24 GB without retraining

Harshit Goyal, Founder, Genrius AI (ex-Google, IIIT Delhi). October 24, 2026. About 8 minutes.

What we built

Language models are big files. Qwen3-4B, a capable small model from the Qwen team, is 8.05 GB at the 16-bit precision it ships in. On a laptop, that file has to share the graphics card's memory with everything else the machine is doing.

Pingala-8 is a way to store the same model in 1.24 GB. We do not retrain the model or change what it learned. We take the original weights, the billions of numbers that make up the model, and write each one with about 2 bits instead of 16. A patch to llama.cpp, the open-source program many people use to run models on their own computers, reads the new file and runs it on an NVIDIA GPU.

The hard part of 2-bit storage is keeping the model useful. With 2 bits, each weight can take only a handful of values, and the rounding errors add up across the whole network. llama.cpp already has 2-bit formats. The smallest common one, IQ2_XXS, makes a file of almost exactly the same size as ours. So our goal was easy to state and hard to reach: at that same size, lose less of the model.

Why 2-bit matters

For local AI, memory is the limit. A model runs at full speed only when its weights fit in GPU memory, with room left for the conversation. Smaller files help in three ways.

• More models fit. A 1.24 GB file fits where an 8 GB file never could, and leaves more room for long chats.
• Replies come faster. Writing text one token at a time is mostly a matter of reading weights from memory, so fewer bytes means more tokens per second. On our test laptop, Pingala-8 Qwen3-4B writes about 216 tokens per second, 39% faster than the common 4-bit Q4_K_M file, which is twice its size.
• Downloads and disks shrink. The Qwen3-4B model is a 1.24 GB download.

The catch has always been quality. Push a model to 2 bits with a simple method and it starts to make mistakes the original would not. That gap is what we worked on.

The numbers, and what they cover

All numbers are for Qwen3-4B unless we say otherwise, measured on one GPU, an RTX 5070 Ti Laptop. The full tables, the logs behind every number and the commands to rerun them are in the repository and the paper.

Size

Qwen3-4B goes from 8.05 GB to 1.24 GB, 85% smaller (6.5x). Llama-3.1-8B goes from 16.08 GB to 2.39 GB, also 85% smaller.

Quiz scores

We ran five standard multiple-choice tests of common sense, reading and school knowledge, with no examples given to the model. All three 2-bit files below are about 1.24 GB.

The average is 21% above IQ2_XXS and keeps 90% of the 8-bit score. Per task, the gain over IQ2_XXS is +39% on ARC-Easy, +30% on ARC-Challenge, +21% on HellaSwag, +11% on WinoGrande and +9% on MMLU.

Perplexity

Perplexity measures how well a model predicts real text; lower is better. On WikiText-2 at context 4096, the original 16-bit Qwen3-4B scores 12.46. Pingala-8 scores 16.64, against 18.15 for the strongest IQ2_KT file of the same size we built and 28.61 for IQ2_XXS.

Swipe the chart sideways to see both panels.

On Llama-3.1-8B we wrote our prediction down before running the test. Pingala-8 scores 9.14 against 10.04 for IQ2_KT and 12.35 for IQ2_XXS. With a start token at the beginning of every test window, it is 8.44 against 9.36 for IQ2_KT.

Speed

When writing a reply for one user, Pingala-8 runs at 1.10x the speed of IQ2_XXS, about 216 tokens per second, and 39% faster than Q4_K_M at half its size. Reading a prompt is slower: 0.74x IQ2_XXS. We come back to that below.

Other formats

Two outside formats score better on perplexity, and we want to be clear about both. GSQ Q2_K from ISTA-DASLab scores 14.37, but its file is 1.67 GB, 34% bigger than ours. Ternary Bonsai 4B from prism-ml scores 11.59 at 1.07 GB, which is better and smaller. Bonsai is a retrained model rather than a compression of Qwen3-4B; it even beats the original Qwen3-4B, and making it took a training run, and running it needs their fork.

So if you only want a strong small model in this class, a retrained model like Bonsai is the better choice. Pingala-8 is for compressing the models nobody has retrained for you. Among the formats we tested, it gives the lowest perplexity for the original Qwen3-4B at about 1.24 GB without retraining, and building a file takes a few hours on one GPU.

What we lose

• Qwen3-14B. Before the run, we predicted in writing that we would beat IQ2_KT on Qwen3-14B. We did not: 9.57 against 9.40 on WikiText-2, and a tie on C4. Pingala-8 does not win on every model.
• Reading prompts. Processing a prompt runs at 0.74x IQ2_XXS, and IQ2_KT reads prompts about 1.3x faster than we do. Long documents take longer before the first word appears.
• Many users at once. Decoding 2 to 8 chats in parallel runs at 0.18x to 0.34x IQ2_XXS. Pingala-8 suits one person on one GPU, not a server.
• Where it runs. You need our patched llama.cpp and an NVIDIA GPU with CUDA. Stock llama.cpp, Ollama and LM Studio cannot load the files yet. There is no Metal or Vulkan path, and CPU generation runs at 0.56x to 0.62x IQ2_XXS.
• How much to trust each number. The Qwen3-4B file is the last of many builds we tested on WikiText-2, so treat that result as tuned. The C4 test was held out, and the Llama-3.1-8B and Qwen3-14B runs were pre-registered. Speed comes from one GPU. We have not evaluated the safety behaviour of the compressed models.

How it works

Now the technical part. Most 2-bit formats round each small group of weights to the nearest entry of a fixed table, one group at a time. Pingala-8 uses trellis-coded quantization instead. We split each row of weights into groups of 4 and code the groups in sequence with an 8-state trellis code of the kind Gottfried Ungerboeck designed for modems. The encoder carries a small state from one group to the next, and the state decides which subset (coset) of values the next group may use. Choosing the whole path through the trellis at once lets the same number of bits land closer to the true weights than rounding each group alone.

Two steps come before the coding, both from the QuIP line of work. A random Hadamard rotation mixes the weights, so that a few very large values do not dominate any group. LDLQ rounding then quantizes the columns in order and passes the error of each column on to the columns not yet coded, using statistics from sample text. We also set a scale for each input column by minimizing an incoherence bound.

How much does the trellis memory itself matter? In an ablation at equal bytes, removing it costs 0.076 nats per token. That is about 80% of our margin over a perplexity-tuned IQ2_KT file (a post hoc figure), but only 14% of our gain over IQ2_XXS; most of that larger gain comes from the rest of the pipeline. The work builds directly on QuIP, QuIP# and QTIP, and IQ2_KT by ikawrakow is the closest same-size format we compare against.

The runtime is a CUDA patch to llama.cpp at commit 7ac59a6. The models are GGUF files that only the patched build can read. The shipped file is called B4 in the paper and the logs.

Why the name

Pingala's Chandahsutra, an ancient Sanskrit work on poetic metre, lists the 8 patterns of three long or short syllables, the ganas. Our code has 8 states over short patterns, so we named the format after him. The name is a credit to that tradition; the trellis labels in the code are Ungerboeck's.

How to try it

1. Download pingala8-windows-cuda12.zip (1.6 GB) from the v2.4-oneclick release. On Linux, take pingala8-linux-cuda13.tar.gz.
2. Unzip it and double-click run_qwen3_4b.bat, or run_llama31_8b.bat for Llama-3.1-8B. The model downloads once from Hugging Face. On Linux, run ./run.sh qwen.
3. Chat at http://127.0.0.1:8080. On Windows the page opens by itself. Everything runs on your own machine.

You need an NVIDIA RTX GPU (30, 40 or 50 series) with a recent driver. To build from source or rerun the paper's tests, follow the README. The models are on Hugging Face: Qwen3-4B and Llama-3.1-8B.

Read and cite the paper

The paper has every table, the pre-registration record and the file and line of each number in the logs.

@misc{goyal2026pingala8,
  title     = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size:
               An Equal-Bytes Study in {llama.cpp}},
  author    = {Goyal, Harshit},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23481757},
  url       = {https://doi.org/10.5281/zenodo.23481757}
}

Which model should we compress next? Tell us in the GitHub issues.