Pingala-8 guides

What is trellis-coded quantization? TCQ, QTIP and IQ2_KT explained

Genrius AI guide. October 24, 2026. About 7 minutes.

If you follow 2-bit LLM files, you have met QTIP, EXL3 and IQ2_KT, all called trellis quantization. Here is what that means, where it came from, and what it buys you.

Start with plain rounding

Quantization stores each weight with fewer bits. With 2 bits, a weight can take one of four values. Scalar quantization rounds every weight to the nearest of four levels, with one shared scale per block. It is fast, but every weight is rounded alone.

Vector quantization rounds a group of weights at once, to the nearest entry in a codebook of whole groups, which can follow the shape of real weights better. llama.cpp's IQ2_XXS works this way, with groups of 8 weights and a grid derived from the E8 lattice. The catch is size: a full codebook for groups of 8 at 2 bits per weight would need 2¹⁶ = 65,536 entries, and for groups of 16, 2³², more than 4 billion. So vector formats keep their groups small or their grids highly structured.

What a trellis adds

A trellis is a small state machine drawn over time. At each step the encoder sits in one of a few states, say 8, and the state decides which subset of values the next group may use. The stored bits do two jobs at once: they pick a value from the allowed subset, and they move the machine to its next state. So each group's value depends on the choices before it. The code has memory.

The trick behind it is Gottfried Ungerboeck's set partitioning. Take a codebook with twice as many values as your bits can address, and split it into subsets whose members are far apart. The state picks the subset, so that choice costs no bits. Over a long row, the encoder can reach a far richer set of sequences than independent rounding allows: some of the benefit of a very long group, without a huge codebook.

A toy version at 1 bit per weight: plain rounding has two levels, say -1 and +1. A trellis version uses four levels, -1.5, -0.5, +0.5 and +1.5, split into two subsets, (-1.5, +0.5) and (-0.5, +1.5). In each state only one subset is allowed, so 1 bit still picks the value, and that bit also sets the state for the next weight. The encoder plans the whole row, so it can steer most weights into a state with a close level. Real codes use more states and better subsets, and the gain grows with the number of states: Marcellin and Fischer's classic result is that 256-state TCQ comes within 0.21 dB of the theoretical limit for a memoryless uniform source.

Encoding and decoding

Choosing the bits means finding the best path through the trellis for a whole row. Trying every path is impossible, but the Viterbi algorithm, invented for decoding error-correcting codes, keeps only the best path ending in each state at each step, so the work grows in a straight line with row length times states. For LLMs the best path is the one that changes the layer's output least, so methods weight the error with calibration statistics and can feed each error forward, as QuIP's LDLQ rounding does. This runs once, offline.

The decoder reads the bits, updates the state and looks up (or computes) a value. There is no search. That matters for local chat: writing one token reads every weight once, so decoding must cost only a few operations per weight. A plain trellis decodes in order, so formats reset the state every few dozen symbols, or use a bitshift trellis, where the state is just the last few bits of the stream and any position can be decoded from a short window.

Who uses it for LLMs

• QTIP (Tseng and colleagues, Cornell, NeurIPS 2024) applies TCQ to LLM weights with a bitshift trellis, after a random Hadamard transform that makes the weights look roughly like independent Gaussian samples (see our Hadamard rotation guide). It computes values from the state instead of storing a huge codebook. Its own ablation found that a trellis small enough to store as a table loses accuracy to larger ones. It runs with its own kernels.
• EXL3, the format of turboderp's ExLlamaV3 library, is in its own words a streamlined variant of QTIP, with a fused GPU Viterbi kernel. It is a separate library, not a llama.cpp format.
• IQ2_KT and the other IQ*_KT types live in ik_llama.cpp, a llama.cpp fork by Iwan Kawrakow, who also wrote llama.cpp's i-quants such as IQ2_XXS. The first IQ2_KT took QTIP's "3INST" generator, with a 16-bit seed per group of 8 weights and 2.125 bits per weight; later versions use a different integer generator that, per its pull request, allows faster GPU and CPU kernels. It is the closest prior format to ours.
• Dependent quantization, an 8-state scalar trellis, is part of the MPEG standard for neural network compression (ISO/IEC 15938-17). Newer research such as Proteus, CCQ and CurveTQ looks for cheaper decoders, mostly in its own engines.

Where Pingala-8 fits

Pingala-8 is our 2-bit format: Ungerboeck's 8-state set-partitioned code, used as a vector TCQ on groups of 4 weights, in a GGUF file the size of IQ2_XXS, decoded by a CUDA patch to llama.cpp. Each group is an 8-bit index into one of two 256-entry tables of 4-weight vectors; the second table is a shifted copy (a coset) of the first, and the trellis state picks which one. Decoding a group takes two XORs, a shift and a table read. Before coding, the weights get a Hadamard rotation, and the Viterbi search uses QuIP's LDLQ rounding.

WikiText-2 perplexity at context 4,096, 73 windows (lower is better), and the average of five zero-shot quizzes. The Pingala-8 Qwen3-4B file is the last of many builds we tested on WikiText-2, so treat its number as tuned. Source: the benchmark dataset and the paper.

What we can claim, and what we cannot:

• The coding is not new. It all comes from the work above. We add a fixed-size llama.cpp file and an equal-bytes comparison with IQ2_KT.
• A small trellis gives up accuracy. On a Gaussian test source our code sits about 1.45 dB below QTIP's long trellis, the price of a decoder that keeps up with llama.cpp's generation speed. In our tests a 16-state code was 1.7% better in perplexity on an earlier build; we ship 8 states because only that code has a native build so far.
• The trellis is not most of the gain over IQ2_XXS. At equal bytes, removing the trellis memory costs 0.076 nats per token: about 80% of our margin over a tuned IQ2_KT (a post hoc reading), but only 14% of the gain over IQ2_XXS. The rest comes from the rotation, the LDLQ rounding, the column scales and the table together.
• It does not always win. It beats IQ2_KT on Llama-3.1-8B (8.44 against 9.36, with a BOS at every window) but loses narrowly on Qwen3-14B (9.57 against 9.40). It writes at least as fast as IQ2_XXS on Qwen3-4B, but reads prompts at 0.74x.

To run it you need the patched llama.cpp build from the one-click release and an NVIDIA RTX 30, 40 or 50 series GPU. Stock llama.cpp, Ollama and LM Studio cannot load Pingala-8 files. For all the 2-bit options side by side, see the GGUF quant comparison.

Questions people ask

What is trellis-coded quantization in simple terms?

A way of rounding a long row of numbers in which each choice also decides which values are allowed next. The encoder searches for the best whole path with the Viterbi algorithm; the decoder just follows it. At the same number of bits it usually lands closer to the original numbers than rounding each group alone.

What is the difference between QTIP, EXL3 and IQ2_KT?

QTIP is a research method from Cornell (NeurIPS 2024) that applies trellis-coded quantization to LLM weights with its own kernels. EXL3, in the ExLlamaV3 library, describes itself as a streamlined variant of QTIP. IQ2_KT is a 2-bit trellis type in ik_llama.cpp, a fork of llama.cpp.

Is trellis-coded quantization slow?

Making the file is slow, because the encoder searches many paths, but that is done once, offline. Decoding can be fast: with a small or shift-based trellis each group of weights needs a few integer operations and a table read. Speed then depends on the kernels.

Can I run trellis-quantized models in Ollama or LM Studio?

Not the formats on this page. IQ2_KT files need ik_llama.cpp, EXL3 files need ExLlamaV3, and Pingala-8 files need the patched llama.cpp build from our one-click release. Stock llama.cpp, Ollama and LM Studio cannot load Pingala-8 files.

More guides

• How much VRAM do you need to run an LLM locally?
• What is a 2-bit LLM, and is 2-bit quantization usable?
• IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
• What is perplexity? LLM perplexity in plain English
• How to run Qwen3-4B on a 4 GB or 6 GB GPU
• How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
• What is Hadamard rotation in LLM quantization?