What is perplexity? LLM perplexity explained in plain English
In short
- Perplexity measures how well a language model predicts real text. Lower is better.
- A perplexity of 12.5 means that, on average, the model is as unsure about each next token as if it were choosing among 12.5 equally likely options.
- There is no universal good score. Compare a compressed model with its own original on the same text and settings: Qwen3-4B scores 12.46, its 8-bit copy 12.50 and 1.24 GB 2-bit copies 16.64 to 28.61.
- It is a fast, sensitive check, but confirm with task scores before you pick a model.
If you compare quantized models, you will meet perplexity everywhere: on model cards, in llama.cpp pull requests and in papers. It is the most common single number for how much a compressed model has lost. Here is what it means, how to read it, and where it can mislead you.
The plain-English version
Give a model a page of real text, one piece at a time, and ask it to guess the next piece before you show it. A model does not make one guess; it spreads its bets, giving each possible next token a probability. If the real next token got a high probability, the model was not surprised. If it got a low one, it was.
Perplexity turns that surprise into one number. Take the probability the model gave each real next token, average the surprise over the whole text (in maths, the average negative log probability), and turn it back into a count. The result reads as "how many options the model was torn between, on average". If a model always gave the right token a probability of 1 in 4, its perplexity would be exactly 4.
A token is a word or a piece of a word, so perplexity is per token, not per word.
Why it matters for quantization
Compressing a model changes its weights, and so its probabilities. Perplexity picks up that change across thousands of predictions at once, which makes it a quick and sensitive way to compare compressed files of the same model. Here is Qwen3-4B on the WikiText-2 test text at context 4,096, with the same settings for every file:
| Qwen3-4B file | Size | Perplexity | Change vs original |
|---|---|---|---|
| fp16 (original) | 8.05 GB | 12.46 | - |
| Q8_0 | 4.28 GB | 12.50 | +0.3% |
| Q2_K | 1.67 GB | 15.01 | +20.5% |
| Pingala-8 | 1.24 GB | 16.64 | +33.6% |
| IQ2_KT | 1.25 GB | 18.15 | +45.7% |
| IQ2_XS | 1.35 GB | 19.02 | +52.7% |
| IQ2_XXS | 1.25 GB | 28.61 | +129.7% |
| IQ1_M | 1.13 GB | 57.50 | +361.6% |
WikiText-2 test, 73 windows, the same harness for every file. The Pingala-8 Qwen3-4B file is the last of many builds we tested on this text, so treat its number as tuned. Source: the benchmark dataset and the paper.
What is a good perplexity score?
No single number is good for every model. A good score is one close to the original model's score on the same test. As a rough reading of the table above (our reading, not a standard):
- Within about 1% (Q8_0): the copy predicts almost exactly like the original.
- About 20% to 50% higher (Q2_K, Pingala-8, IQ2_KT, IQ2_XS): a clear but moderate loss.
- Double or more (IQ2_XXS, IQ1_M): a large loss. On five quizzes, IQ2_XXS keeps 74% of the 8-bit model's average.
Compare like with like
Perplexity numbers are only comparable when everything except the file is the same:
- Same model and tokenizer. Pingala-8 Llama-3.1-8B scores 9.14 and Pingala-8 Qwen3-4B 16.64, but that does not make the Llama file better. The two models split text into different tokens and were trained differently, so their scores sit on different scales.
- Same text. Different texts give different numbers for the same file: Pingala-8 Qwen3-4B scores 16.64 on WikiText-2 (Wikipedia articles) and 24.36 on C4 (web pages).
- Same settings. Context length and how the text is cut into windows matter. The same Pingala-8 Llama-3.1-8B file scores 9.14 in llama.cpp's strided mode and 8.44 when every window starts with a start-of-text token.
When perplexity can mislead
- It does not always match task scores. In our Qwen3-4B tests the two mostly agreed, but not perfectly: IQ2_XS has worse perplexity than IQ2_KT (19.02 against 18.15) and a slightly higher quiz average (47.7 against 46.5), a gap inside the tests' noise.
- Tuning on the test text flatters. If you try many builds and keep the one with the lowest WikiText-2 score, that score is optimistic. Our Qwen3-4B Pingala-8 file is such a case, so we also report C4, which we held out: 24.36 against 27.15 for IQ2_KT and 37.28 for IQ2_XXS, the same order.
- A retrained model is a different model. Ternary Bonsai 4B scores 11.59, below the original Qwen3-4B's 12.46, because it was retrained rather than compressed. Its perplexity tells you how good it is, not how faithful a copy it is.
A closer check: KL divergence
Perplexity only looks at the probability of the right token. KL divergence compares the compressed model's whole spread of probabilities with the original's, token by token, so it measures how closely a copy follows the original. llama.cpp's perplexity tool can report it against a saved run of the original (--kl-divergence-base, then --kl-divergence). Against the 8-bit Qwen3-4B, Pingala-8 scores 0.585, the IQ2_KT files 0.85 to 0.89 and IQ2_XXS 1.40; lower is closer.
How to measure it yourself
llama.cpp ships llama-perplexity. Download the WikiText-2 raw test file, then run every GGUF file you want to compare with the same command, for example:
llama-perplexity -m model.gguf -f wiki.test.raw -c 4096 -ngl 99It prints a running value per window and a final estimate with an error bar. Keep the text, the context and every other flag the same across files. The exact command and text file behind our numbers are in the reproduction guide.
Where Pingala-8 fits
Pingala-8 is a 2-bit format we built to lose less at a fixed size. On Qwen3-4B at 1.24 GB its perplexity is 34% above the original, against 46% for the strongest IQ2_KT file and 130% for IQ2_XXS at the same size, and on the quizzes it keeps 90% of the 8-bit average. Where it does not fit: Q2_K loses less (+21%) if you can spare 1.67 GB, Pingala-8 loses to IQ2_KT on Qwen3-14B (9.57 against 9.40), and it needs our patched llama.cpp on an NVIDIA GPU. The GGUF quant comparison puts these numbers next to speed and quiz scores.
Questions people ask
What does perplexity mean for an LLM?
It measures how well the model predicts real text, one token at a time. Lower is better. A perplexity of N means that, on average, the model was as unsure about each next token as if it were choosing among N equally likely options.
Is lower perplexity always better?
For the same model on the same test with the same settings, yes: lower means it predicts the text better. Across different models, tokenizers, texts or settings, the numbers are not comparable.
What is a good perplexity for a quantized model?
One close to the original model's score on the same test. On WikiText-2, Qwen3-4B's 8-bit file is within 0.3% of the 16-bit original; the 1.24 GB 2-bit files we tested are 34% to 130% higher.
Why does my perplexity differ from published numbers?
The text, the context length, how the text is cut into windows, the tokenizer and the original file all move the number. Use the same command and text for every file you compare.
Try Pingala-8
More guides
- How much VRAM do you need to run an LLM locally?
- What is a 2-bit LLM, and is 2-bit quantization usable?
- IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
- How to run Qwen3-4B on a 4 GB or 6 GB GPU
- How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
- What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
- What is Hadamard rotation in LLM quantization?