Pingala-8 by Genrius AI
Pingala-8 guides

What is Hadamard rotation in LLM quantization? QuIP, QuaRot and incoherence explained

Genrius AI guide. . About 7 minutes.

In short

  • Low-bit formats round a block of weights with one shared scale. A few large values, the outliers, stretch that scale and waste most of the grid on everyone else.
  • A Hadamard rotation mixes each block using only plus and minus signs. Outliers get spread across the block, the values look more like a bell curve, and the rotation can be undone exactly.
  • QuIP called this incoherence processing; QuIP# made it a fast randomized Hadamard transform; QTIP and EXL3 build on it; QuaRot and SpinQuant rotate the activations too, for 4-bit compute.
  • Pingala-8 rotates every weight matrix with a random-sign Hadamard transform over blocks of 256 columns before its trellis code. We have not measured the rotation's share of our gain on its own.

Recent papers on 2-bit and 4-bit LLMs mention a "random Hadamard transform" or "incoherence processing" early on. The idea is simple, and it explains a good part of why some low-bit formats lose far less than others.

The problem: outliers waste the grid

Quantizers store a block of weights with one shared scale and a few bits per weight. The scale has to reach the largest value in the block. LLM weights mostly sit in a narrow bell curve, but a few are much larger, and some input channels are larger than the rest across the whole matrix. One outlier sets the scale, and the other weights crowd onto the levels nearest zero. At 2 bits, with only four levels, that is expensive.

Older fixes keep outliers at higher precision (SpQR), scale whole channels (SmoothQuant, AWQ) or weight the error by importance (llama.cpp's importance matrix). A rotation instead changes the coordinates so there are no outliers to begin with.

What a Hadamard matrix is

A Hadamard matrix is square, its entries are only +1 and -1, and its rows are at right angles to each other. Divided by the square root of its size, it becomes an orthogonal matrix: it rotates (or reflects) a vector without changing its length, and its transpose undoes it exactly. For power-of-two sizes you build one by doubling, starting from [1]: the matrix next to itself, and below that the matrix next to its negative. The 4 by 4 one is:

+1  +1  +1  +1
+1  -1  +1  -1
+1  +1  -1  -1
+1  -1  -1  +1

Every output mixes every input with equal weight; only the signs change. That is what spreads an outlier across the block.

A toy example

Take four weights with one outlier: 0.5, -0.3, 0.2 and 4.0. Multiply by the matrix above and divide by 2 (the square root of 4): you get 2.2, -1.5, -2.0 and 2.3. The length is unchanged (the squares add up to 16.38 before and after), but the largest value fell from 4.0 to 2.3 and the four values now have similar sizes.

Now round both to a 2-bit grid of four evenly spaced levels, scaled to the largest value. The original keeps the outlier exact but turns the three small weights into 1.33, -1.33 and 1.33, a total squared error of about 3.05. The rotated version, rounded and rotated back, gives 0.77, -0.77, 0.77 and 3.83, a total squared error of about 0.64, nearly five times smaller. The outlier lost a little; everyone else gained a lot. Real weights are messier, but the effect is the same.

Why the model still gives the same answer

A linear layer computes y = W x, weights times input. For any orthogonal matrix Q, y = (W QT)(Q x). So you can store the rotated weights W QT, quantized, and rotate the input by Q at run time; before rounding, the output is identical. Two ways to pay for the input rotation:

  • Rotate on the fly. The fast Walsh-Hadamard transform needs only additions and subtractions, n times log2 n of them: for a block of 256 values, 8 rounds of 256, against 65,536 multiply-adds for a general 256 by 256 rotation.
  • Fold it into the previous layer. Where the maths allows, the rotation is merged into the weights that produce the input, so it costs nothing at run time. This "computational invariance" idea comes from SliceGPT, and QuaRot uses it for most of its rotations.

Most methods also flip signs at random before the Hadamard step, which is called a randomized Hadamard transform. That makes a bad alignment with the fixed pattern very unlikely. The seed is stored, so the rotation is reproducible.

Incoherence: the idea behind QuIP

QuIP (Chee, Cai, Kuleshov and De Sa, Cornell, 2023) named the goal: incoherence. A weight matrix is incoherent when no entry is large compared with the whole, and the calibration statistics (the Hessian) have no dominant direction lined up with one weight. QuIP made both incoherent with random orthogonal matrices, then rounded with LDLQ, an adaptive rounding that feeds each column's error forward to the columns not yet rounded. By its authors' account, it was the first LLM quantization method to give viable results at two bits per weight.

QuIP# (Tseng and colleagues, 2024) replaced QuIP's rotations with the randomized Hadamard transform, which is faster and has better guarantees, and added codebooks built on the E8 lattice. QTIP kept the Hadamard step and replaced the codebook with a trellis code (see our trellis-coded quantization guide). After the rotation the weights look roughly like independent Gaussian samples, which is what lattice and trellis codes are designed for. EXL3, in the ExLlamaV3 library, is a streamlined variant of QTIP.

QuaRot and SpinQuant: rotating activations too

QuaRot (Ashkboos and colleagues, 2024) also rotates the hidden states that flow between layers, removing outliers from the activations, so weights, activations and the KV cache can all be quantized to 4 bits and the matrix multiplies run in low precision. Most rotations are folded into the weights offline; a few run on the fly with fast Hadamard kernels. SpinQuant, from Meta, learns the rotation matrices instead of using random ones. These methods aim mostly at faster matrix multiplies; for one person chatting with a local model, the bottleneck is reading the weights for every token, so the weight format matters most.

Limits worth knowing

  • A rotation saves no bits by itself. It makes the bits you have work better, and the gain depends on the grid, codebook or trellis that follows.
  • Block size is a trade-off. Rotating a whole row spreads outliers furthest but costs more, so many formats rotate in power-of-two blocks.
  • It needs runtime support. The runtime must apply the matching rotation. llama.cpp's IQ2_XXS does not rotate; it uses an importance matrix and a grid derived from the E8 lattice. The first version of ik_llama.cpp's IQ2_KT used no Hadamard transform either, per its pull request.

Where Pingala-8 fits

Pingala-8 is our 2-bit format. Each weight matrix is scaled per input column (with a scale chosen to minimise QuIP's incoherence bound on LDLQ's error, under a simple model of the Hessian), rotated with a random-sign fast Walsh-Hadamard transform over blocks of 256 columns, given one 16-bit scale per 256 weights per row, and rounded with QuIP's LDLQ along an 8-state trellis code. The rotation and the LDLQ rounding are borrowed from QuIP and QuIP#.

Qwen3-4B fileSizePerplexityQuiz average
Q8_0 (reference)4.28 GB12.5056.3
IQ2_XXS1.25 GB28.6141.9
IQ2_KT1.25 GB18.1546.5
Pingala-81.24 GB16.6450.8

WikiText-2 perplexity at context 4,096, 73 windows (lower is better), and the average of five zero-shot quizzes. The Pingala-8 Qwen3-4B file is the last of many builds we tested on WikiText-2, so treat its number as tuned. Source: the benchmark dataset and the paper.

On Qwen3-4B it keeps 90% of the 8-bit model's quiz average, and on Llama-3.1-8B it beats IQ2_KT at the same size (8.44 against 9.36, with a BOS at every window). The caveats:

  • We did not isolate the rotation. At equal bytes the trellis gives 14% of our gain over IQ2_XXS; the rest comes from the rotation, the LDLQ rounding, the column scale and the table together, and we did not separate them.
  • It does not always win. On Qwen3-14B it loses narrowly to IQ2_KT (9.57 against 9.40), and it reads prompts at 0.74x the speed of IQ2_XXS.
  • It needs our runtime. The files need the patched llama.cpp build from the one-click release and an NVIDIA RTX 30, 40 or 50 series GPU. Stock llama.cpp, Ollama and LM Studio cannot load them.

For how 2-bit files compare in practice, see is 2-bit quantization usable? and the GGUF quant comparison.

Questions people ask

Why do LLM quantization methods use a Hadamard transform?

To spread out outliers. The rotation mixes each block of numbers, so a few large values no longer set the scale for the whole block. The rotated values look more like a bell curve, which a low-bit grid or codebook fits better, and the rotation can be undone exactly.

Does rotating the weights change the model's output?

Not by itself. A normalized Hadamard matrix is orthogonal, so rotating the weights and applying the matching rotation to the layer's input gives the same result. Only the rounding that follows changes anything, and the rotation makes that rounding error smaller and more even.

What is the difference between QuIP and QuaRot?

QuIP rotates the weights so that weight-only quantization to 2 bits loses less; QuIP# switched it to a fast randomized Hadamard transform. QuaRot also rotates the hidden states, so weights, activations and the KV cache can all be stored in 4 bits and the matrix multiplies run in low precision.

Is a Hadamard rotation expensive at run time?

It is cheap. The fast Walsh-Hadamard transform uses only additions and subtractions: for a block of 256 values, 8 rounds of 256, against 65,536 multiply-adds for a general 256 by 256 rotation. Some methods fold rotations into neighbouring weights, so those cost nothing at run time.

More guides