IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant should you pick?
In short
- If Q4_K_M fits with room for your context, start there. For Qwen3-4B it is about 2.5 GB.
- Below that, in stock llama.cpp, Ollama or LM Studio, Q2_K (1.67 GB, perplexity 15.01) and IQ2_XS (1.35 GB, 19.02) lose far less than IQ2_XXS (1.25 GB, 28.61) on Qwen3-4B.
- At the IQ2_XXS size on an NVIDIA GPU, trellis formats do much better: IQ2_KT scores 18.15 and our Pingala-8 16.64.
- Skip IQ1_M on a model this small: 57.50.
Open any GGUF model page on Hugging Face and you get a long list of files: Q8_0, Q6_K, Q4_K_M, Q3_K_S, IQ2_XS, IQ2_XXS, IQ1_M and more. They are the same model stored at different precisions. This guide decodes the names, then compares the 2-bit to 4-bit options on one model with one set of tests, so you can pick by your memory and your software.
How to read the names
- The number is the rough bits per weight for most of the model: Q2 is about 2, Q4 about 4, Q8 about 8. Whole files come out a little higher.
- K means a k-quant: weights are stored in small blocks with their own scales, grouped into super-blocks of 256 weights.
- S, M and L are small, medium and large mixes. An M file gives a few sensitive tensors more bits than an S file.
- IQ means an i-quant: groups of weights are stored as entries from a fixed table of patterns (a lattice codebook). They are usually made with an importance matrix, or imatrix: statistics from sample text that tell the quantizer which weights matter most.
- XXS, XS, S and M on i-quants are sizes, smallest first.
- KT is a trellis quant from ik_llama.cpp, a fork by ikawrakow, who also wrote llama.cpp's k-quants and i-quants. It codes groups of weights in sequence, in the spirit of QTIP. Stock llama.cpp cannot load it.
The comparison
Every row is Qwen3-4B, made from the same original and measured with the same WikiText-2 test at context 4,096 (lower perplexity is better). The quiz average covers ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande and MMLU. Speeds are ratios to IQ2_XXS on one RTX 5070 Ti Laptop GPU: "writing" is generating a reply for one user (batch 1), "reading" is processing the prompt. A dash means we did not measure it.
| Format | Size | Perplexity | Quiz average | Writing | Reading | Runs in |
|---|---|---|---|---|---|---|
| fp16 (original) | 8.05 GB | 12.46 | - | - | - | Any tool |
| Q8_0 (reference) | 4.28 GB | 12.50 | 56.3 | - | - | Stock tools |
| Q4_K_Mllama.cpp | about 2.5 GB | - | - | 0.79x | 0.89x | Stock tools |
| Q2_Kllama.cpp | 1.67 GB | 15.01 | - | - | - | Stock tools |
| IQ2_XSllama.cpp | 1.35 GB | 19.02 | 47.7 | - | - | Stock tools |
| IQ2_KTik_llama.cpp | 1.25 GB | 18.15 | 46.5 | 1.00x | 0.95x | ik_llama.cpp |
| IQ2_XXSllama.cpp | 1.25 GB | 28.61 | 41.9 | 1.00x | 1.00x | Stock tools |
| Pingala-8Genrius AI | 1.24 GB | 16.64 | 50.8 | 1.10x | 0.74x | Our patched llama.cpp, NVIDIA CUDA only |
| IQ1_Mllama.cpp | 1.13 GB | 57.50 | - | - | - | Stock tools |
"Stock tools" means stock llama.cpp, Ollama and LM Studio. The IQ2_KT row is the strongest of six IQ2_KT files we built at this size. The Pingala-8 Qwen3-4B file is the last of many builds we tested on WikiText-2, so treat its number as tuned. Its writing speed has a 95% interval of 1.03x to 1.16x, and the paper does not treat that edge over IQ2_XXS as established. The Q4_K_M size is Qwen's official file. Source: the benchmark dataset and the paper.
Each format in a few lines
Q4_K_M is the usual default for local use and runs everywhere. On Qwen3-4B it is about twice the size of the 2-bit files, and at batch 1 on our GPU it wrote at 0.79x the speed of IQ2_XXS, largely because a bigger file takes longer to read from memory. We did not measure its perplexity in this study.
Q2_K has the lowest perplexity of the stock llama.cpp files under 2 GB in this table, 15.01, but at 1.67 GB it is 34% bigger than the 1.25 GB files. If 1.67 GB fits, it is a good pick in any tool.
IQ2_XS, at 1.35 GB, is 9% bigger than IQ2_XXS and much better: 19.02 against 28.61, and 47.7 against 41.9 on the quizzes. If you need stock tools and Q2_K does not fit, try it before IQ2_XXS.
IQ2_XXS is the smallest common 2-bit file. On Qwen3-4B it loses the most of the 2-bit files: 28.61, more than twice the original's 12.46. Its strength is that it runs everywhere.
IQ2_KT is the same size as IQ2_XXS and scores 18.15 and 46.5. It reads prompts at 0.95x IQ2_XXS, about 1.3x faster than Pingala-8, and on Qwen3-14B it beats Pingala-8, 9.40 against 9.57. It needs ik_llama.cpp.
Pingala-8 is our format, the same size again: 16.64 and 50.8, the lowest perplexity and highest quiz average of the 1.25 GB files we tested on Qwen3-4B. On Llama-3.1-8B at 2.39 GB it also leads both same-size formats (9.14 against 10.04 for IQ2_KT and 12.35 for IQ2_XXS). It reads prompts slower (0.74x) and needs our patched llama.cpp on an NVIDIA GPU.
IQ1_M saves about 0.12 GB more and roughly doubles the perplexity of IQ2_XXS, to 57.50. On a model this small, that is a poor trade.
Which one to pick
- Does Q4_K_M fit, with room for your context? Use it. How much VRAM you need shows the arithmetic.
- Need Ollama, LM Studio, a Mac, an AMD card or a CPU? Use the largest stock file that fits: Q2_K, then IQ2_XS, then IQ2_XXS.
- NVIDIA GPU, one user, and you need the IQ2_XXS size? Try Pingala-8 for the lowest perplexity in our tests on Qwen3-4B and Llama-3.1-8B, or IQ2_KT for faster prompt reading and for Qwen3-14B.
- Not tied to one model? A model retrained for low precision can beat any compression: Ternary Bonsai 4B scores 11.59 at 1.07 GB, but needs its authors' llama.cpp fork. A smaller model at 4-bit is also worth testing; we did not measure that trade.
Caveats
These are one model's numbers, and the ranking can change between models: on Qwen3-14B, IQ2_KT beats Pingala-8. Quiz scores for a typical 2-bit file carry standard errors of 0.4 to 3.1 points per task, so small gaps mean little. Perplexity is one test of text prediction (see what perplexity means), so check scores on your own task too. For background on the 2-bit trade itself, read is 2-bit quantization usable.
Questions people ask
Is IQ2_XXS better than Q2_K?
Not on quality. On Qwen3-4B, Q2_K has a WikiText-2 perplexity of 15.01 against 28.61 for IQ2_XXS (lower is better), but its file is 34% bigger: 1.67 GB against 1.25 GB. If 1.67 GB fits, Q2_K is the safer stock choice. IQ2_XXS is for when it does not.
What is the difference between IQ2_XS and IQ2_XXS?
IQ2_XS is the larger of the two. On Qwen3-4B it is 1.35 GB against 1.25 GB, with perplexity 19.02 against 28.61 and a five-quiz average of 47.7 against 41.9.
What is IQ2_KT?
A 2-bit trellis quant in ik_llama.cpp, the llama.cpp fork by ikawrakow. At the IQ2_XXS size on Qwen3-4B, the strongest IQ2_KT file we built scores 18.15 against 28.61 for IQ2_XXS. Stock llama.cpp, Ollama and LM Studio cannot load it.
Is Q4_K_M good enough?
For most local use, Q4_K_M is the usual default and runs in every common tool. Choose a 2-bit file only when Q4_K_M plus the context you need does not fit in your memory.
Try Pingala-8
More guides
- How much VRAM do you need to run an LLM locally?
- What is a 2-bit LLM, and is 2-bit quantization usable?
- What is perplexity? LLM perplexity in plain English
- How to run Qwen3-4B on a 4 GB or 6 GB GPU
- How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
- What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
- What is Hadamard rotation in LLM quantization?