Pingala-8 guides

Gemma 4 VRAM requirements: 26B-A4B and 31B

Genrius AI guide. October 25, 2026. About 8 minutes.

Will it fit your GPU? Normal vs Pingala-8 calculator

Gemma 4 comes in several sizes; this page covers the two that people try to fit on one consumer GPU, the dense 31B and the 26B-A4B mixture of experts. Both have unusual attention, which our calculator does not cover yet (it shows Gemma 4 as "weights only"), so the KV cache here is derived by hand from the configs, google/gemma-4-31B-it config.json @ 842da37 and google/gemma-4-26B-A4B-it config.json @ 4d7ae49, and from llama.cpp's Gemma 4 code at commit 1a0b56c.

Memory per quantization: Gemma 4 31B

Memory per quantization: Gemma 4 26B-A4B

Memory estimates from public configs, not measurements: we have not run this model. Weights = parameters x bits per weight / 8, with llama.cpp's whole-file bits per weight (FP16 16.0005, Q8_0 8.5008, Q4_K_M 4.8944, IQ2_XXS 2.3824, from the Llama-3.1-8B table in llama.cpp's tools/quantize/README.md). KV cache at 16-bit, context padded to 256 cells as llama.cpp does. GB = 10⁹ bytes. Add about 1.5 GB for the runtime's compute buffers and your desktop. These are the calculator's formulas.

As a check, unsloth's files are 32.64 GB (Q8_0), 18.32 GB (Q4_K_M) and 8.53 GB (UD-IQ2_XXS) for the 31B (unsloth/gemma-4-31B-it-GGUF), and 26.86 GB, 16.95 GB (UD-Q4_K_M) and 9.92 GB (UD-IQ2_XXS) for the 26B-A4B (unsloth/gemma-4-26B-A4B-it-GGUF). Our rows run a little high for Q8_0 and Q4_K_M because the parameter totals include the vision encoder, which ships as a separate mmproj file of about 1.2 GB at BF16. The 26B-A4B 2-bit file is larger than our estimate because unsloth's "UD" files keep some tensors at higher precision.

Which GPU fits at 4k, 8k and 32k context

Gemma 4 31B:

Gemma 4 26B-A4B:

The largest of the four formats in the tables above whose weights plus 16-bit KV cache leave at least 1.5 GB free on the card; "tight" means it fits but with less than 1.5 GB left. Card memory from NVIDIA's spec pages, taken at face value. On paper, not tested on these cards.

The 26B-A4B is the easier fit: about 16 GB at Q4_K_M with a small cache, comfortable on 24 GB, and its 2-bit file fits a 12 GB card even at 32k. The 31B at Q4_K_M fits a 24 GB card at 4k and 8k and is tight at 32k. On 16 GB, neither Q4_K_M file fits, so it is 2-bit or partial offload.

How the architecture affects the KV cache

Both models mix two kinds of layer. In the 31B, 50 of 60 layers use sliding-window attention over the last 1,024 tokens and 10 use full attention (in the 26B-A4B, 25 of 30 and 5). The two kinds have different shapes, and that is where a simple calculator goes wrong:

• Sliding layers (31B): 16 key-value heads of size 256, so 2 x 16 x 256 x 2 = 16,384 bytes per token per layer at 16-bit. llama.cpp caps their cache at GGML_PAD(min(context, 1,024 + 512), 256) = 1,536 cells by default (src/llama-kv-cache-iswa.cpp), so 50 layers cost a fixed 1.26 GB once the context passes 1,536 tokens.
• Full layers (31B): num_global_key_value_heads = 4 heads of global_head_dim = 512, so 8,192 bytes per token per layer, and these grow with the context. llama.cpp's converter writes these as a per-layer head count (conversion/gemma.py), and the model code reads the head size per layer type (src/llama-hparams.cpp, n_embd_head_k).
• K equals V. The config sets attention_k_eq_v. Where a layer has no separate value projection, llama.cpp reuses the key projection as the value (src/models/gemma4.cpp). That saves weights, not cache: the key is then normalized and rotated, the value only normalized, and llama.cpp stores both, so the cache holds the usual two tensors per layer.

For the 31B that gives 1,258,291,200 bytes for the sliding layers plus 81,920 bytes per token for the full layers: 1.93 GB at 8,192 tokens, 12.00 GB at 131,072 and 22.73 GB at the full 262,144-token window. For the 26B-A4B the same rules give 8,192 bytes per sliding layer and 4,096 per full layer per token: 3.00 GB at 131,072 tokens. If you run llama.cpp with --swa-full, every sliding layer keeps the whole context too, and the 31B cache at 32,768 tokens grows from 3.94 GB to 29.53 GB.

The 26B-A4B is a mixture of experts: 128 experts per layer, 8 used per token. As with any MoE, all experts must be loaded, so memory follows the full parameter count; llama.cpp's --cpu-moe and --n-cpu-moe N keep expert weights in system RAM when the card is too small, at a cost in speed.

Pingala-8 and this model

Pingala-8 is a 2-bit format at IQ2_XXS size. A Pingala-8 build of Gemma 4 31B or 26B-A4B is not released yet and its quality is not measured; the size would be about the IQ2_XXS row above, about 9.31 GB or 7.69 GB of weights. Request it here: open the GPU fit calculator and press Request, or open a GitHub issue with the model, your card and the context you need.

What we have measured is on other models. On Qwen3-4B, Pingala-8 is 1.24 GB with a WikiText-2 perplexity of 16.64 against 28.61 for IQ2_XXS at the same size; on Qwen3-14B it loses narrowly to IQ2_KT (9.57 against 9.40). The released files run only in our patched llama.cpp on NVIDIA RTX 30, 40 or 50 series cards; stock llama.cpp, Ollama and LM Studio cannot load them. Details are in the paper and the benchmark dataset.

Questions people ask

How much VRAM does Gemma 4 31B need?

On paper about 19.1 GB of weights at Q4_K_M and 9.3 GB at IQ2_XXS, plus a KV cache of 1.93 GB at 8,192 tokens and about 1.5 GB of working buffers. A 24 GB card fits Q4_K_M up to about 32k tokens, where it gets tight. These are estimates from the public config, not measurements.

How much VRAM does Gemma 4 26B-A4B need?

On paper about 15.8 GB of weights at Q4_K_M and 7.7 GB at IQ2_XXS, plus a KV cache of only 0.48 GB at 8,192 tokens. All 128 experts per layer must be loaded even though 8 run per token.

Why does Gemma 4 31B have a bigger KV cache than its sliding window suggests?

Its 50 sliding layers use 16 key-value heads of size 256, so even capped at 1,536 cells they take 1.26 GB, and its 10 full layers use heads of size 512 that grow with the context. llama.cpp stores both keys and values even where the model reuses the key projection as the value.

Is there a Pingala-8 version of Gemma 4?

Not yet. A Pingala-8 build would be about the IQ2_XXS size, about 9.3 GB for the 31B or 7.7 GB for the 26B-A4B, but it is not released and its quality is not measured. You can request it from the calculator or in our GitHub issues.

More guides

• How much VRAM do you need to run an LLM locally?
• What is a 2-bit LLM, and is 2-bit quantization usable?
• IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
• What is perplexity? LLM perplexity in plain English
• How to run Qwen3-4B on a 4 GB or 6 GB GPU
• How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
• What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
• What is Hadamard rotation in LLM quantization?
• Qwen3.8 27B VRAM requirements: which GPU do you need?
• Can you run gpt-oss-20b on a 12 GB or 16 GB GPU?
• Qwen3.6 35B-A3B on a consumer GPU: VRAM and MoE offload