Pingala-8 by Genrius AI
Pingala-8 overview

Small-State Coset Trellis Coding at IQ2_XXS Size: An Equal-Bytes Study in llama.cpp

Harshit Goyal, Founder, Genrius AI (former Google software engineer, IIIT Delhi alumnus). Preprint, version 2.4, . DOI 10.5281/zenodo.23481757.

Download the PDF (21 pages, 434 KB) Zenodo record (PDF and supplement)

In short

  • An 8-state trellis code, stored in a standard llama.cpp GGUF file at exactly the size of llama.cpp's IQ2_XXS, with no retraining.
  • Qwen3-4B goes from 8.05 GB (16-bit) to 1.24 GB. WikiText-2 perplexity: 16.64, against 28.61 for IQ2_XXS and 18.15 for the best IQ2_KT file we built at that size.
  • Llama-3.1-8B in 2.39 GB: 8.44 against 9.36 for IQ2_KT. On Qwen3-14B it loses narrowly (9.57 against 9.40).
  • Generation at batch 1 is at least as fast as IQ2_XXS; prompt processing runs at 0.74x.

Abstract

Trellis-coded quantization of LLM weights is established (QTIP, EXL3), and ik_llama.cpp's IQ2_KT already provides a trellis type in a llama.cpp fork. We ask how much a small-state code adds at a fixed file size, that of llama.cpp's IQ2_XXS, and which part of the pipeline supplies the gain. We store Ungerboeck's 8-state set-partitioned code over 4-weight symbols in a GGUF file, decoded by an unmerged CUDA patch to llama.cpp, behind a random Hadamard rotation and QuIP's LDLQ rounding, with input-column scales that minimise, under a diagonal Hessian model, QuIP's incoherence bound on LDLQ's error. On Qwen3-4B (post hoc) the file reaches WikiText-2 perplexity 16.64, against 28.61 for IQ2_XXS and 18.15 for the strongest IQ2_KT file of this size we built; it decodes at batch 1 at least as fast as IQ2_XXS but processes prompts at 0.74x. Pre-registered, it beats IQ2_KT on Llama-3.1-8B (8.44 against 9.36 with a BOS at every window; IQ2_KT made from Q8_0) and loses on Qwen3-14B (9.57 against 9.40; tie on C4). At equal bytes, removing the trellis memory costs 0.076 nats per token: about 80% (post hoc) of the margin over a perplexity-tuned IQ2_KT, but 14% of the gain over IQ2_XXS.

Code, models and logs

Cite

@misc{goyal2026coset,
  title     = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size:
               An Equal-Bytes Study in llama.cpp},
  author    = {Goyal, Harshit},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23481757},
  url       = {https://doi.org/10.5281/zenodo.23481757}
}

Licence: CC BY 4.0. The DOI above always resolves to the latest version.