Pingala-8 guides

Can you run gpt-oss-20b on a 12 GB or 16 GB GPU?

Genrius AI guide. October 25, 2026. About 7 minutes.

Will it fit your GPU? Normal vs Pingala-8 calculator

gpt-oss-20b is OpenAI's smaller open-weight model, and "will it run on my 12 GB or 16 GB card" is the first thing most people ask about it. The answer is unusual, because the file sizes do not follow the usual quantization ladder. This page works it out from the config, openai/gpt-oss-20b config.json @ 6cee5e8, and the published GGUF files.

Memory per quantization

First, what our calculator's formula gives if every weight were stored at each format's bits per weight:

Memory estimates from public configs, not measurements: we have not run this model. Weights = parameters x bits per weight / 8, with llama.cpp's whole-file bits per weight (FP16 16.0005, Q8_0 8.5008, Q4_K_M 4.8944, IQ2_XXS 2.3824, from the Llama-3.1-8B table in llama.cpp's tools/quantize/README.md). KV cache at 16-bit, context padded to 256 cells as llama.cpp does. GB = 10⁹ bytes. Add about 1.5 GB for the runtime's compute buffers and your desktop. These are the calculator's formulas.

Now the files people actually download (sizes from the Hugging Face API, ggml-org/gpt-oss-20b-GGUF and unsloth/gpt-oss-20b-GGUF):

Real file sizes plus the same KV cache, calculated, not measured.

The real files are all between 11.5 and 13.8 GB, whatever their name says. That is because most of the model, the 32 experts in each of the 24 layers, is published in MXFP4 (the config's quantization_config leaves only attention, the router, the embeddings and the output layer unconverted), and the file sizes show that these builds keep the experts in that form. So the formula's FP16 and Q8_0 rows describe a model nobody downloads, and Q4_K_M is no smaller than MXFP4 in practice. The IQ2_XXS row, about 6.23 GB, is what a true 2-bit file of all weights would need; neither repo publishes one.

Which GPU fits at 4k, 8k and 32k context

Using the real MXFP4 and Q4_K_M files, and the on-paper IQ2_XXS size as a last resort:

The largest of the three options whose weights plus 16-bit KV cache leave at least 1.5 GB free on the card; "tight" means it fits but with less than 1.5 GB left. Card memory from NVIDIA's spec pages, taken at face value. On paper, not tested on these cards.

On 16 GB cards (RTX 4060 Ti 16 GB, 4080, 5060 Ti 16 GB, 5070 Ti, 5080) the MXFP4 file fits at every context in the table, and even at the full 131,072 tokens the cache is 3.24 GB, about 15.4 GB in total with no buffers counted, so very long context is tight. On 12 GB cards the MXFP4 file is larger than the card once the cache is added; the Q4_K_M file fits only as "tight" at 4k and 8k, with well under 1.5 GB left for buffers. On 8 GB nothing published fits fully; only a true 2-bit file, which does not exist yet in these repos, would.

Running it on 12 GB: expert offload

gpt-oss-20b is a mixture-of-experts model: 4 of 32 experts run per token, but the router picks different experts each time, so all of them must be loaded somewhere. llama.cpp can keep experts in system RAM with --cpu-moe (all layers) or --n-cpu-moe N (the first N layers) while attention and the cache stay on the GPU. On a 12 GB card, start with a few layers, for example -ngl 99 --n-cpu-moe 4 -c 8192, and lower N until it no longer loads. It writes more slowly than a full-GPU run, but far faster than offloading whole layers of a dense model.

How the architecture keeps the cache small

The config's layer_types alternate: 12 layers use sliding-window attention with a 128-token window and 12 use full attention. Every layer has 8 key-value heads of size 64, so each costs 2 x 8 x 64 x 2 bytes = 2,048 bytes per token at 16-bit. The full layers grow with the context. The sliding layers do not: llama.cpp gives them a separate cache of GGML_PAD(min(context, window + n_ubatch), 256) cells, which is 768 cells with the default n_ubatch of 512 (src/llama-kv-cache-iswa.cpp; the alternating pattern is set by load_swa_pattern(ml, 2) in src/models/openai-moe.cpp).

So the cache is 2,048 x (12 x context + 12 x 768) bytes: 0.22 GB at 8,192 tokens and 0.82 GB at 32,768. Our calculator currently shows gpt-oss as "weights only"; the KV numbers on this page are derived by hand from the config and that llama.cpp source, at commit 1a0b56c.

Pingala-8 and this model

Pingala-8 is a 2-bit format at IQ2_XXS size. A Pingala-8 build of gpt-oss-20b is not released yet and its quality is not measured; the size would be about the IQ2_XXS row above, about 6.23 GB of weights on paper; because the experts ship in MXFP4, a 2-bit build would have to start from a higher-precision copy of the weights. Request it here: open the GPU fit calculator and press Request, or open a GitHub issue with the model, your card and the context you need.

What we have measured is on other models. On Qwen3-4B, Pingala-8 is 1.24 GB with a WikiText-2 perplexity of 16.64 against 28.61 for IQ2_XXS at the same size; on Qwen3-14B it loses narrowly to IQ2_KT (9.57 against 9.40). The released files run only in our patched llama.cpp on NVIDIA RTX 30, 40 or 50 series cards; stock llama.cpp, Ollama and LM Studio cannot load them. Details are in the paper and the benchmark dataset.

Questions people ask

Can gpt-oss-20b run on a 16 GB GPU?

Yes, on paper. The MXFP4 GGUF is 12.11 GB and the KV cache is 0.82 GB at 32,768 tokens, so about 12.9 GB before buffers, leaving room on a 16 GB card.

Can gpt-oss-20b run on a 12 GB GPU?

Not fully on the GPU with the published files: they are 11.5 to 12.1 GB before any cache. Keep some expert layers in system RAM with llama.cpp's --n-cpu-moe, which is slower but works with enough RAM.

Why is the gpt-oss-20b Q4_K_M file almost as big as MXFP4?

Because the expert weights, most of the model, are published in the 4-bit MXFP4 format and these GGUF builds keep them that way. Only the remaining tensors change with the quant name, so the files are all about 11.5 to 13.8 GB.

Is there a Pingala-8 version of gpt-oss-20b?

Not yet. A Pingala-8 build would be about the IQ2_XXS size, about 6.2 GB on paper, but it is not released and its quality is not measured. You can request it from the calculator or in our GitHub issues.

More guides

• How much VRAM do you need to run an LLM locally?
• What is a 2-bit LLM, and is 2-bit quantization usable?
• IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
• What is perplexity? LLM perplexity in plain English
• How to run Qwen3-4B on a 4 GB or 6 GB GPU
• How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
• What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
• What is Hadamard rotation in LLM quantization?
• Qwen3.8 27B VRAM requirements: which GPU do you need?
• Gemma 4 VRAM requirements: 26B-A4B and 31B
• Qwen3.6 35B-A3B on a consumer GPU: VRAM and MoE offload