Pingala-8 by Genrius AI
Pingala-8 guides

NaN perplexity with IQ1_S, IQ2_S or IQ3_S GGUFs on an RTX 50 laptop GPU

Genrius AI troubleshooting note. . About 4 minutes.

In short

  • On our machine, an RTX 5070 Ti Laptop GPU (sm_120, WSL2), llama.cpp upstream 889f2ec and our 7ac59a6 build gave NaN or a perplexity near 6.8e6 for GGUFs containing IQ1_S, IQ2_S or IQ3_S tensors.
  • On our machine, test-backend-ops failed for those three CUDA kernels.
  • Workaround that gave correct numbers on our machine: keep those tensors on the CPU with -ot "<regex of their names>=CPU" plus --no-op-offload.
  • We have not tested other cards, drivers or llama.cpp versions. This is what we saw on one machine, not a claim about a general bug.

What we saw on our machine

We measured published 2-bit files of Qwen3.8-27B and Qwen3.5-2B to compare them with our own Pingala-8 Qwen3.8-27B file. On our RTX 5070 Ti Laptop GPU (compute capability sm_120, under WSL2), with every layer on the GPU, these files gave NaN or a WikiText-2 perplexity near 6.8e6 (about 6.8 million) instead of a normal value:

  • unsloth UD-IQ2_S (Qwen3.8-27B)
  • DASH-Q IQ2_XS and IQ2_M (Qwen3.8-27B and Qwen3.5-2B)

All of them contain tensors stored as IQ1_S, IQ2_S or IQ3_S. It happened on two builds on our machine: llama.cpp upstream at 889f2ec, and our own build at 7ac59a6. llama.cpp's own operator test, test-backend-ops, failed for the IQ1_S, IQ2_S and IQ3_S CUDA kernels on this machine, which suggests those kernels rather than the files.

Check your own GPU with test-backend-ops

test-backend-ops runs each operator on your GPU backend and compares it with the CPU. It is built with llama.cpp's tests (on by default in a normal CMake build). For matrix multiplication:

./build/bin/test-backend-ops -o MUL_MAT -b CUDA0 2>&1 | grep -E "iq1_s|iq2_s|iq3_s"

Each line names the case, for example type_a=iq2_s, and ends in OK or FAIL. On our machine the IQ1_S, IQ2_S and IQ3_S cases failed. If all of yours pass, your GPU and build are probably not affected, and the NaN has another cause. Drop -b CUDA0 to test every backend, or add MUL_MAT_ID to -o (comma separated) for mixture-of-experts models.

Workaround: keep those tensors on the CPU

llama.cpp's -ot (--override-tensor) option takes <regex of tensor names>=<buffer type>. Pointing every IQ1_S, IQ2_S and IQ3_S tensor at CPU keeps their matrix multiplications off the GPU kernels that failed, while every other tensor stays on the GPU. Add --no-op-offload so llama.cpp does not copy those CPU-stored weights back to the GPU to run large batches there. This small script builds the regex from the tensor types in the file, using gguf-py (pip install gguf):

# make_ot.py: print an -ot value that keeps every IQ1_S, IQ2_S and IQ3_S tensor on the CPU.
# Usage: python3 make_ot.py model-00001-of-00003.gguf [more parts ...]
import re, sys
from gguf import GGUFReader, GGMLQuantizationType as Q

SLOW = {Q.IQ1_S, Q.IQ2_S, Q.IQ3_S}
names = []
for path in sys.argv[1:]:                  # pass every split part
    for t in GGUFReader(path).tensors:
        if t.tensor_type in SLOW:
            names.append(re.escape(t.name))
if names:
    print("^(" + "|".join(names) + ")$=CPU")
else:
    sys.exit("no IQ1_S, IQ2_S or IQ3_S tensors in these files")

Then pass it to llama.cpp:

OT=$(python3 make_ot.py model-0000*-of-00003.gguf)
./build/bin/llama-perplexity -m model-00001-of-00003.gguf -f wiki.test.raw -ngl 99 -ot "$OT" --no-op-offload

llama-cli and llama-server accept the same two options. Expect slower prompt processing and generation, since part of the model now runs on the CPU.

How we checked that the CPU path is right

As a check that the GPU and CPU paths agree on our machine, we ran the unsloth UD-IQ2_XXS file of Qwen3.5-2B on the same windows once on the GPU and once on the CPU. Window 1 gave 20.66 on the GPU and 20.67 on the CPU, and the first 4 windows agreed within 0.2%. With the workaround above, the IQ1_S, IQ2_S and IQ3_S files then gave normal perplexities on our machine; those are the DASH-Q and unsloth numbers on our Qwen3.8-27B page.

What we do not know

We have tested one machine: one RTX 5070 Ti Laptop GPU under WSL2, with the two llama.cpp builds named above. We do not know whether other RTX 50 cards, native Windows or Linux, other drivers or newer llama.cpp versions behave the same. Run test-backend-ops on your own setup before relying on either the problem or the workaround.

Questions people ask

Why does my 2-bit GGUF give NaN perplexity in llama.cpp?

We can only say what we saw on our machine. On our RTX 5070 Ti Laptop GPU (sm_120, WSL2), llama.cpp upstream 889f2ec and our 7ac59a6 build gave NaN or a perplexity near 6.8e6 for GGUFs containing IQ1_S, IQ2_S or IQ3_S tensors, and test-backend-ops failed for those three CUDA kernels. Run test-backend-ops -o MUL_MAT on your own GPU to see whether yours behaves the same.

How do I keep IQ2_S tensors on the CPU in llama.cpp?

Pass -ot with a regex of those tensor names followed by =CPU, plus --no-op-offload. On our machine that gave correct numbers: a CPU-vs-GPU check on a file the GPU handles agreed within 0.2% over 4 windows.

More guides