Pingala-8 guides

How to run Qwen3-4B on a 4 GB or 6 GB GPU

Genrius AI guide. October 24, 2026. About 6 minutes.

Qwen3-4B, from the Qwen team, is a popular size for laptops: big enough to be useful, small enough to compress onto a modest graphics card. Many laptops have 4 GB or 6 GB of VRAM, for example RTX 3050 laptops with 4 GB and RTX 4050 laptops with 6 GB. Here is which file to pick, how much context you can keep, and what to change when it does not fit.

The memory budget

You need room for the model file, the KV cache (the model's memory of the conversation) and some working space. For Qwen3-4B the cache takes about 0.15 MB per token at 16-bit, calculated from its published config (36 layers, 8 key-value heads of size 128). Our VRAM guide explains the arithmetic.

File plus a 16-bit KV cache, calculated, not measured on these cards. Add the runtime's compute buffer and what your desktop uses. An 8-bit cache roughly halves the cache part.

On a 6 GB card, Q4_K_M with 8,192 tokens leaves over 2 GB to spare on paper. On a 4 GB card the desktop and buffers take part of the memory, so Q4_K_M at 4,096 tokens leaves under 1 GB, while the 2-bit files leave about 2 GB.

Option 1: Q4_K_M in any tool (6 GB cards)

Qwen publishes official GGUF files, including Qwen3-4B-Q4_K_M.gguf (about 2.5 GB). It works on any GPU, on a CPU and on Macs, in Ollama and LM Studio as well as llama.cpp. With llama.cpp:

llama-server -hf Qwen/Qwen3-4B-GGUF:Q4_K_M -ngl 99 -c 8192

-ngl 99 puts every layer on the GPU and -c sets the context. Then open http://127.0.0.1:8080 in a browser.

Option 2: squeeze it onto 4 GB

• Shorten the context. Use -c 4096 or less. Every 4,096 tokens costs about 0.6 GB at 16-bit.
• Store the cache at 8 bits. -fa on -ctk q8_0 -ctv q8_0 roughly halves the cache, with a small change in output.
• Turn off thinking when you do not need it. Qwen3 writes out its reasoning before the answer by default, and those tokens fill the context. Adding /no_think to a message turns it off for that turn.
• Free the GPU. Close browsers and apps that use GPU acceleration, and check free memory with nvidia-smi.
• Use a smaller file. Q2_K (1.67 GB) and IQ2_XS (1.35 GB) run in stock tools; community GGUF repositories publish them, or you can make them with llama-quantize. IQ2_XXS (1.25 GB) loses much more on this model; see the table below.
• Last resort: split the model. A lower -ngl keeps some layers in system RAM. It runs, but writes much more slowly.

Option 3: Pingala-8 on an NVIDIA RTX card

If your card is an NVIDIA RTX 30, 40 or 50 series, Pingala-8 gives you Qwen3-4B in 1.24 GB, the size of IQ2_XXS, with a much smaller loss than IQ2_XXS in our tests.

1. Download pingala8-windows-cuda12.zip (1.6 GB) from the v2.4-oneclick release. On Linux, take pingala8-linux-cuda13.tar.gz.
2. Unzip it and double-click run_qwen3_4b.bat (on Linux, ./run.sh qwen). The model downloads once from Hugging Face.
3. Chat at http://127.0.0.1:8080. Everything runs on your machine.

The launcher starts llama-server with a 4,096-token context, so by the table above it needs about 1.84 GB plus working buffers. We measured Pingala-8 on one RTX 5070 Ti Laptop GPU, where it writes about 216 tokens per second for one user. We have not measured other cards, and we have not run it on a 4 GB or 6 GB card yet. If you do, please tell us how it went in the GitHub issues.

How the options compare

WikiText-2 perplexity at context 4,096 (lower is better; what it means) and the average of five zero-shot quizzes. A dash means not measured. "Stock tools" means stock llama.cpp, Ollama and LM Studio. The Pingala-8 Qwen3-4B file is the last of many builds we tested on WikiText-2, so treat its number as tuned. Source: the benchmark dataset and the paper. More formats: the GGUF quant comparison.

Where Pingala-8 does not fit

• Not an NVIDIA RTX 30 series or newer. The one-click build does not cover GTX 16 or RTX 20 series cards, AMD or Intel GPUs, Macs or CPU-only machines. Use Q4_K_M or a stock 2-bit file there.
• You use Ollama or LM Studio. They cannot load Pingala-8 files yet.
• Long documents. Pingala-8 reads prompts at 0.74x the speed of IQ2_XXS, so the first word takes longer on long inputs.
• Q4_K_M fits comfortably. On a 6 GB card with a modest context, Q4_K_M is the simpler choice and keeps more of each weight.

Questions people ask

Can Qwen3-4B run on 4 GB of VRAM?

On paper, yes, with a compressed file. A 1.24 GB 2-bit file plus a 4,096-token cache comes to about 1.84 GB. Q4_K_M (about 2.5 GB) comes to about 3.1 GB, which is tight on a 4 GB card once the desktop and working buffers take their share.

Which Qwen3-4B quant should I use on a 6 GB GPU?

Q4_K_M, about 2.5 GB. It runs in every common tool and, on paper, leaves over 2 GB to spare with an 8,192-token context.

How much VRAM does Qwen3-4B need at full precision?

The 16-bit file is 8.05 GB, plus about 0.6 GB of KV cache at 4,096 tokens, so it needs a card with more than 8 GB.

Does Pingala-8 work in Ollama or LM Studio?

Not yet. Pingala-8 files need our patched llama.cpp with CUDA on an NVIDIA GPU. The one-click build for Windows and Linux includes it.

More guides

• How much VRAM do you need to run an LLM locally?
• What is a 2-bit LLM, and is 2-bit quantization usable?
• IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
• What is perplexity? LLM perplexity in plain English
• How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
• What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
• What is Hadamard rotation in LLM quantization?