Genrius AI articles

Benchmarking 2-bit GGUFs fairly: paired windows, and the NaN trap on RTX 50 GPUs

Harshit Goyal, Founder, Genrius AI (ex-Google software engineer, IIIT Delhi alumnus). November 1, 2026. About 8 minutes.

Why 2-bit files are hard to compare

At 2 bits, good files of the same model often differ by only a few percent in perplexity. On Qwen3.8-27B, the five files we measured span 6.54 to 7.88. A few percent is easy to create or erase by accident: a different context length, a different slice of the test text, a different tokenizer setting, or a GPU kernel that quietly returns garbage.

There is also the question of who is measuring. We make Pingala-8, one of the files in every comparison here. The only answer to that we know is to publish the method, the commands and every window, and to report the comparisons we lose with the same care as the ones we win. Below is the method, two lessons from using it, and one trap we hit on our GPU.

Rule 1: same text, same windows

Every file is scored on the same bytes. We take the WikiText-2 raw test split from Hugging Face, join its rows into one file, and publish its SHA-256 hash in the dataset card. llama-perplexity runs in its strided mode with -c 2049 --ppl-stride 4095, which makes it score 4,096-token windows, 72 of them over the whole text. Batch size is 512 for every file.

For Qwen3.5-2B we score all 72 windows. For Qwen3.8-27B we stop after the first 20, about 82,000 tokens, because of the time a 27B model takes on one laptop GPU. Every file of a model is scored on exactly the same windows, so its numbers line up one for one.

Our IQ2_XXS reference files are built with stock llama-quantize and the same importance matrix (imatrix) we use for Pingala-8, so the comparison is about the format and not about the calibration data. The rival files are the ones their authors published, downloaded and scored unchanged.

One local change to disclose: in our upstream llama.cpp build, the strided path of perplexity.cpp scores each batch as soon as its logits are ready instead of keeping all logits for the window, to save host memory. The tokens scored and the formula are unchanged.

Rule 2: pair the windows

A final perplexity is one number per file. It hides how the file did on each part of the text. llama-perplexity prints the running perplexity P_k after each window k. Since every window has the same number of scored tokens, the mean loss of window k, in nats per token, falls out of two running values:

NLL_1 = ln P_1
NLL_k = k ln P_k - (k-1) ln P_(k-1)

For two files A and B, take the difference d_k = NLL_A,k - NLL_B,k on every window. Report three things: the mean of d, its 95% interval (1.96 times the standard deviation of d, divided by the square root of the number of windows), and the number of windows on which A has the lower loss. A mean difference converts back to a perplexity ratio with exp(): 0.052 nats per token is a ratio of about 1.053, or 5% in perplexity.

Why pair at all? Windows differ a lot in difficulty. A window of plain prose is easy; one full of names, dates and tables is hard. That difficulty is shared by every file scored on the window, so the difference between two files on the same window cancels most of it. What is left is the part that belongs to the files. A comparison of two averages keeps all the difficulty noise in; a paired comparison takes it out.

Here is a short script that does this from two llama-perplexity logs:

import math, re, statistics as st, sys

def window_nll(path):
    # running perplexity [k]P_k as printed by llama-perplexity
    run = {int(k): float(p) for k, p in re.findall(r"\[(\d+)\]([0-9.]+)", open(path).read())}
    nll, prev = [], 0.0
    for k in sorted(run):
        cum = k * math.log(run[k])
        nll.append(cum - prev)
        prev = cum
    return nll

a, b = window_nll(sys.argv[1]), window_nll(sys.argv[2])
n = min(len(a), len(b))
d = [a[i] - b[i] for i in range(n)]
m, h = st.mean(d), 1.96 * st.stdev(d) / math.sqrt(n)
won = sum(x < 0 for x in d)
print(f"dNLL {m:+.3f} +/- {h:.3f} nats/token, A better on {won}/{n}, ppl ratio {math.exp(m):.3f}")

Run it as python3 paired.py a.log b.log. A negative dNLL means file A lost less.

Why averages mislead: one lower average, one real loss

These are the paired results from our recent runs, each one ours minus the other file:

dNLL in nats per token; negative means ours lost less. 27B rows: Qwen3.8-27B, our v2 file, first 20 windows. 2B rows: Qwen3.5-2B, our v2 file, 70 paired windows.

Look at the first two rows. On final perplexity, our Qwen3.8-27B file is 6.89, unsloth UD-IQ2_S is 7.34, and DASH-Q IQ2_XS is 6.54. Read as averages, we beat unsloth by a wider margin (0.064 nats) than DASH-Q beats us (0.052 nats). It is tempting to report a clear win over unsloth and a narrow loss to DASH-Q.

The windows say otherwise. DASH-Q is better than us on every one of the 20 windows, and its interval is tight: plus or minus 0.012. Against unsloth we are better on only 9 of 20 windows, and the interval, plus or minus 0.075, is about six times wider and crosses zero. The difference between our file and unsloth's swings both ways from window to window. On a different 20 windows the average could come out the other way. So we report the DASH-Q result as a loss and the unsloth result as not significant, and that is how the 27B guide states it.

The 2B rows show the other side: with 70 windows, small differences become measurable. DASH-Q IQ2_XS beats our 2B file by only about 1.2% in perplexity, and it is still a clear result, because the interval is plus or minus 0.007.

Pairing also guards us against our own hopes. One build we tried for the 2B, which copied the bit plan of our 27B file, had a final perplexity of 15.80 against 15.62 for version 1. Paired, it was +0.011 plus or minus 0.008, better on only 22 of 70 windows: no gain, and we did not ship it.

The NaN trap on RTX 50 GPUs

The first time we scored the DASH-Q and unsloth files on our laptop, with every layer on the GPU, the perplexity came out as NaN or as absurd values. Our machine is an RTX 5070 Ti Laptop GPU (compute capability sm_120) under WSL2. It happened with llama.cpp upstream at commit 889f2ec and with our own build at 7ac59a6.

All of the affected files contain tensors stored as IQ1_S, IQ2_S or IQ3_S. Mixed 2-bit recipes use these types for some of their tensors. llama.cpp ships an operator test, test-backend-ops, that runs each operator on the GPU and checks it against the CPU. On our machine the IQ1_S, IQ2_S and IQ3_S matrix-multiply cases failed it:

./build/bin/test-backend-ops -o MUL_MAT -b CUDA0 2>&1 | grep -E "iq1_s|iq2_s|iq3_s"

That points at those CUDA kernels on this machine, not at the files. The workaround that gave correct numbers: keep every tensor of those three types on the CPU with llama.cpp's -ot (override-tensor) option, and add --no-op-offload so llama.cpp does not copy those weights back to the GPU for large batches. Every other tensor stays on the GPU.

llama-perplexity -m rival.gguf -f wikitext2_test_hfjoin.raw -c 2049 --ppl-stride 4095 -ngl 99 -b 512 \
  -ot "^(<names of the IQ1_S, IQ2_S and IQ3_S tensors>)$=CPU" --no-op-offload

The NaN troubleshooting page has a short gguf-py script that builds the regex from a file's tensor types, and the dataset card lists the exact commands and the regex we used for each rival file.

How do we know the CPU path is right? We ran a file that the GPU handles correctly, unsloth's UD-IQ2_XXS of Qwen3.5-2B, once on the GPU and once on the CPU, on the same windows. The first 4 windows agreed within 0.2%.

Three limits to keep in mind:

• One machine. We have tested one GPU, one driver setup and two llama.cpp builds. This is what we saw, not a claim about every RTX 50 card. Run test-backend-ops on your own setup first.
• No speed comparison. With part of each rival file on the CPU, its speed on our machine says nothing fair about the file. We report speed only for files that ran fully on the GPU.
• Two runtimes. Rival files ran on upstream llama.cpp; ours ran on our patched runtime. The perplexity formula is the same, and the dataset records the runtime and the tensor placement of every row.

The broader lesson: a NaN is the easy case, because you notice it. Before you trust a perplexity from a new GPU, a new build or a new quant type, check it against a second path once.

The leaderboard

The 2-bit GGUF leaderboard ranks the published 2-bit files of Qwen3.8-27B and Qwen3.5-2B by WikiText-2 perplexity, with the paired difference to the file in first place and the windows won. DASH-Q is first on both models. On Qwen3.8-27B our version 2 file ranks 2 of 5; on Qwen3.5-2B it ranks 3 of 6, ahead of both unsloth files.

The eval dataset behind it has one row per file and window: the running perplexity as logged, the derived window loss, the runtime, the tensor placement and the source log. Download it and you can redo every paired number on this page, or score your own file on the same text and windows and compare.

A checklist for your own comparisons

1. Use one text file for every model file, and publish its hash.
2. Fix the context, stride, window count and batch size, and use the same ones for every file.
3. Build your reference quants with the same imatrix as the file you are testing.
4. Compare per window: mean difference, 95% interval, and windows won. Call a result a win only when the interval leaves zero out.
5. Run test-backend-ops on a new GPU or build, and cross-check one file on two paths.
6. Do not compare speed for a file that ran partly on the CPU.
7. Publish the windows, including the comparisons you lose.

The same method is behind every number in our 8-model article and our 27B article.

More articles

• How we fit a 27B model on a 12 GB laptop GPU
• 8 open models, 77-85% smaller: what 2-bit quantization costs and what it keeps
• How we fit Qwen3-4B into 1.24 GB without retraining

Questions, corrections or a model you want compressed next? Open a GitHub issue.