Qwen3.6 35B-A3B on a consumer GPU: VRAM and MoE offload
Will it fit your GPU? Normal vs Pingala-8 calculator
In short
- Qwen3.6-35B-A3B is a mixture-of-experts model: about 3 billion parameters work on each token, but all 35.95 billion must be in memory. That is about 22.00 GB at Q4_K_M and 10.71 GB at IQ2_XXS on paper.
- Its KV cache is tiny: only 10 of 40 layers keep one, with 2 key-value heads each. 0.74 GB at 32,768 tokens, 5.43 GB at the full 262,144.
- A 24 GB card holds Q4_K_M up to about 32k tokens (tight at 32k). A 16 GB card fits a 2-bit file. On 8 to 12 GB, the usual route is keeping experts in system RAM.
- All numbers are memory estimates from the public config. We have not run this model.
"A3B" sounds small, and people often expect Qwen3.6-35B-A3B to run like a 3B model on a modest card. Each token uses only about 3 billion parameters, which is why models like this write quickly, but it does not fit like a small model. This page works out the memory from the model's config, Qwen/Qwen3.6-35B-A3B config.json @ 995ad96, with the formulas of our GPU fit calculator.
Memory per quantization
| Format | Weights | + 4,096 tokens | + 8,192 tokens | + 32,768 tokens |
|---|---|---|---|---|
| FP16 | 71.91 GB | 72.06 GB | 72.14 GB | 72.65 GB |
| Q8_0 | 38.20 GB | 38.35 GB | 38.43 GB | 38.94 GB |
| Q4_K_M | 22.00 GB | 22.15 GB | 22.23 GB | 22.74 GB |
| IQ2_XXS | 10.71 GB | 10.86 GB | 10.94 GB | 11.45 GB |
Memory estimates from public configs, not measurements: we have not run this model. Weights = parameters x bits per weight / 8, with llama.cpp's whole-file bits per weight (FP16 16.0005, Q8_0 8.5008, Q4_K_M 4.8944, IQ2_XXS 2.3824, from the Llama-3.1-8B table in llama.cpp's tools/quantize/README.md). KV cache at 16-bit, context padded to 256 cells as llama.cpp does. GB = 109 bytes. Add about 1.5 GB for the runtime's compute buffers and your desktop. These are the calculator's formulas.
For comparison, unsloth's published files (unsloth/Qwen3.6-35B-A3B-GGUF) are 36.90 GB for Q8_0, 22.13 GB for UD-Q4_K_M and 10.76 GB for UD-IQ2_XXS. Our estimates count the vision encoder and the multi-token-prediction layer in the parameter total; a text GGUF ships the vision part separately (mmproj, 0.90 GB at BF16).
MoE: all experts sit in memory
The config lists 256 experts per layer, 8 of them routed for each token, plus one shared expert. The router picks different experts for every token, so llama.cpp needs all of them loaded, and the weight column above counts all 35.95 billion parameters. What MoE saves is compute and memory bandwidth per token, not capacity: each token reads only a small slice of the weights, which is why it writes quickly.
That same fact makes MoE models good at partial offload. llama.cpp can keep the expert weights in system RAM and the rest (attention, shared expert, cache) on the GPU, with --cpu-moe for all experts or --n-cpu-moe N for the first N layers. Because each token touches only 8 of 256 experts, this costs less speed than offloading layers of a dense model, though it is still slower than running fully on the GPU. You need enough system RAM to hold the expert weights, roughly the file size minus what stays on the card.
Which GPU fits at 4k, 8k and 32k context
| Card memory | Common NVIDIA cards | 4,096 tokens | 8,192 tokens | 32,768 tokens |
|---|---|---|---|---|
| 8 GB | RTX 3050 8 GB, 3060 Ti, 4060, 4060 Ti 8 GB, 5060; laptop 4060, 4070, 5060 | None fully on the GPU | None fully on the GPU | None fully on the GPU |
| 12 GB | RTX 3060 12 GB, 4070, 5070; laptop 5070 Ti | IQ2_XXS (tight) | IQ2_XXS (tight) | IQ2_XXS (tight) |
| 16 GB | RTX 4060 Ti 16 GB, 4080, 5060 Ti 16 GB, 5070 Ti, 5080; laptop 4090 | IQ2_XXS | IQ2_XXS | IQ2_XXS |
| 24 GB | RTX 3090, 4090; laptop 5090 | Q4_K_M | Q4_K_M | Q4_K_M (tight) |
| 32 GB | RTX 5090 | Q4_K_M | Q4_K_M | Q4_K_M |
The largest of the four formats in the table above whose weights plus 16-bit KV cache leave at least 1.5 GB free on the card; "tight" means it fits but with less than 1.5 GB left. Card memory from NVIDIA's spec pages, taken at face value. On paper, not tested on these cards.
Q4_K_M at about 22 GB is a 24 GB-card format: it fits at 4k and 8k and is tight at 32k, with under 1.5 GB to spare. A 32 GB RTX 5090 holds it with room to spare. Q8_0 at 38 GB does not fit any single consumer card. On 16 GB, a 2-bit file of about 10.7 GB fits at every context in the table. On 12 GB the 2-bit file is tight, and on 8 GB you need expert offload.
Why the KV cache is so small
Like Qwen3.8-27B, this model is a hybrid: of its 40 layers, 30 are linear-attention layers and 10 are full-attention layers (full_attention_interval = 4). Only the full-attention layers keep a per-token cache, and each has just 2 key-value heads of size 256: 2 x 2 x 256 x 2 bytes = 2,048 bytes per token per layer, 20,480 bytes per token in total. The 30 linear-attention layers keep a fixed 32-bit state of 65,863,680 bytes (0.07 GB), counted the way llama.cpp sizes it (src/llama-hparams.cpp): per layer, 3 x (2 x 16 x 128 + 32 x 128) convolution values plus 128 x 32 x 128 recurrent values.
So the context costs very little: 2.75 GB at 131,072 tokens. Qwen3-30B-A3B, the earlier non-hybrid MoE of similar size, needs 98,304 bytes per token (48 layers, 4 key-value heads of 128), almost five times as much. With this model, the weights decide what card you need; the context almost never does.
Pingala-8 and this model
Pingala-8 is a 2-bit format at IQ2_XXS size. A Pingala-8 build of Qwen3.6-35B-A3B is not released yet and its quality is not measured; the size would be about the IQ2_XXS row above, about 10.71 GB of weights. Request it here: open the GPU fit calculator and press Request, or open a GitHub issue with the model, your card and the context you need.
What we have measured is on other models. On Qwen3-4B, Pingala-8 is 1.24 GB with a WikiText-2 perplexity of 16.64 against 28.61 for IQ2_XXS at the same size; on Qwen3-14B it loses narrowly to IQ2_KT (9.57 against 9.40). The released files run only in our patched llama.cpp on NVIDIA RTX 30, 40 or 50 series cards; stock llama.cpp, Ollama and LM Studio cannot load them. Details are in the paper and the benchmark dataset.
Questions people ask
How much VRAM does Qwen3.6-35B-A3B need?
On paper about 22.0 GB of weights at Q4_K_M, 38.2 GB at Q8_0 and 10.7 GB at IQ2_XXS, plus a KV cache of only 0.23 GB at 8,192 tokens and about 1.5 GB of working buffers. These are estimates from the public config, not measurements.
Does a 3B-active MoE model need only 3B worth of VRAM?
No. All 256 experts per layer must be loaded because the router picks different ones for each token, so memory follows the full 35.95 billion parameters. The 3B active parameters make it fast, not small.
Can Qwen3.6-35B-A3B run on a 12 GB or 16 GB GPU?
On 16 GB, a 2-bit file of about 10.7 GB fits fully on paper. On 12 GB the 2-bit file is tight. Otherwise keep the expert weights in system RAM with llama.cpp's --cpu-moe or --n-cpu-moe, which is slower but works with enough RAM.
Is there a Pingala-8 version of Qwen3.6-35B-A3B?
Not yet. A Pingala-8 build would be about the IQ2_XXS size, about 10.7 GB, but it is not released and its quality is not measured. You can request it from the calculator or in our GitHub issues.
Try Pingala-8
More guides
- How much VRAM do you need to run an LLM locally?
- What is a 2-bit LLM, and is 2-bit quantization usable?
- IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
- What is perplexity? LLM perplexity in plain English
- How to run Qwen3-4B on a 4 GB or 6 GB GPU
- How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
- What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
- What is Hadamard rotation in LLM quantization?
- Qwen3.8 27B VRAM requirements: which GPU do you need?
- Gemma 4 VRAM requirements: 26B-A4B and 31B
- Can you run gpt-oss-20b on a 12 GB or 16 GB GPU?