Pingala-8 guides

How to run Llama 3.1 8B on a 4 GB or 6 GB GPU

Genrius AI guide. October 24, 2026. About 7 minutes.

Llama 3.1 8B, from Meta, is one of the most used open models in its size class. Many laptops have 4 GB or 6 GB of VRAM, for example RTX 3050 laptops with 4 GB and RTX 4050 laptops with 6 GB. At that size the question is not which file is best but which file fits at all. Here is the memory arithmetic, three ways to run it, and what each costs.

The memory budget

Three things share the card: the model file, the KV cache (the model's memory of the conversation) and working buffers for the runtime and your desktop. For Llama 3.1 8B the cache takes 131,072 bytes per token at 16-bit, calculated from its published config: 32 layers and 8 key-value heads of size 128. That is 0.54 GB at 4,096 tokens. Our VRAM guide walks through the arithmetic.

File plus a 16-bit KV cache, calculated, not measured on these cards. Add the runtime's compute buffer and whatever your desktop uses. An 8-bit cache roughly halves the cache part. File sizes are the exact files in the paper: 2,388,210,816 bytes for Pingala-8, 2,389,350,432 for the IQ2_KT file we built and 2,399,212,576 for IQ2_XXS.

On a 4 GB card, any of the 2-bit files fits with 4,096 tokens and leaves over 1 GB for buffers and the desktop; at 8,192 tokens it gets tight. On a 6 GB card they fit with 16,384 tokens on paper. Q4_K_M is the hard case: at about twice the size of a 2-bit file it cannot sit fully on a 4 GB card, and on 6 GB it fits only with a short context and little else using the GPU.

Option 1: Q4_K_M with a short context (6 GB cards)

Q4_K_M is the usual default for local models: it keeps more of each weight than any 2-bit file, and it runs everywhere, in llama.cpp, Ollama and LM Studio, on NVIDIA, AMD and Apple hardware and on a CPU. For chat you want the Instruct version. With llama.cpp:

llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99 -c 4096 -fa on -ctk q8_0 -ctv q8_0

-ngl 99 puts every layer on the GPU, -c 4096 keeps the context short, and -fa on -ctk q8_0 -ctv q8_0 stores the cache at 8 bits, which roughly halves it with a small change in output. Then open http://127.0.0.1:8080. If it runs out of memory, close apps that use the GPU, check free memory with nvidia-smi, and shorten the context before anything else.

Option 2: Q4_K_M on 4 GB, with part of it in system RAM

You can split the model. -ngl sets how many layers live on the GPU; Llama 3.1 8B has 32. Start low, for example -ngl 20, and raise it until the model no longer loads, then step back. The layers left behind run from system RAM, and every new token has to read them through that slower path, so the model writes much more slowly. It is fine for trying the model; for long chats most people find it frustrating.

Option 3: a 2-bit file that fits entirely

A 2-bit file keeps the whole model and a useful context on the GPU. The cost is quality, and how much depends a lot on the format; our 2-bit guide explains the trade-off. These are the three files of this size we measured on Llama 3.1 8B:

Base Llama-3.1-8B, WikiText-2 test, context 4,096, 70 windows, in llama.cpp's strided mode, where windows after the first start without a start-of-text (BOS) token; lower is better (what perplexity means). With a BOS at every window instead, Pingala-8 scores 8.44 against 9.36 for IQ2_KT. IQ2_KT was made from the 8-bit Q8_0 file. We wrote the Pingala-8 prediction down before this run. Only perplexity was measured on this model: no quiz scores and no speed. Source: the paper, Table 1 and Section 4.3.

IQ2_XXS runs in every stock tool and loses the most of the three. IQ2_KT, a trellis-based type from the ik_llama.cpp fork, loses less. Pingala-8, our format, has the lowest perplexity of the three at this size.

Running Pingala-8 Llama 3.1 8B

On an NVIDIA RTX 30, 40 or 50 series card, the one-click build runs it with nothing else to install:

1. Download pingala8-windows-cuda12.zip (1.6 GB) from the v2.4-oneclick release. On Linux, take pingala8-linux-cuda13.tar.gz.
2. Unzip it and double-click run_llama31_8b.bat (on Linux, ./run.sh llama). The model, about 2.4 GB, downloads once from Hugging Face.
3. A page opens at http://127.0.0.1:8080. Everything runs on your machine.

The launcher starts llama-server with a 4,096-token context, so by the table above it needs about 2.9 GB plus working buffers. The files need this patched build: stock llama.cpp, Ollama and LM Studio cannot load them. To build the runtime yourself, follow the quick start in the README.

Know before you try: this is the base Llama-3.1-8B, not the Instruct model. It continues the text you start ("The three main causes of inflation are") rather than answering like an assistant. That is fine for testing, writing continuations and comparing formats; for chat, use an Instruct file in a stock tool, or our Qwen3-4B file from the Qwen3-4B guide.

We have not run Pingala-8 on a 4 GB or 6 GB card yet, and we have not measured its speed on Llama 3.1 8B. If you try it, please tell us how it went in the GitHub issues.

Where Pingala-8 does not fit

• You want a chat model. We publish the base Llama-3.1-8B only.
• Not an NVIDIA RTX 30 series or newer. The build does not cover GTX 16 or RTX 20 series cards, AMD or Intel GPUs, Macs or CPU-only machines.
• You use Ollama or LM Studio. They cannot load Pingala-8 files.
• Long prompts. On Qwen3-4B, where we measured speed, Pingala-8 reads prompts at 0.74x the speed of IQ2_XXS.
• Every model is different. Pingala-8 wins on Llama 3.1 8B and Qwen3-4B, but loses narrowly to IQ2_KT on Qwen3-14B (9.57 against 9.40).

Questions people ask

Can Llama 3.1 8B run on a 4 GB GPU?

Fully on the GPU, only as a 2-bit file: about 2.4 GB, or about 2.9 GB with a 4,096-token cache on paper. A Q4_K_M file is about twice the size of a 2-bit file, so on 4 GB part of it has to stay in system RAM, which makes it write much more slowly.

Which Llama 3.1 8B quant should I use on a 6 GB GPU?

Try Q4_K_M with a short context first; it runs in every common tool, but it is tight on 6 GB. If it does not fit with the context you need, a 2-bit file of about 2.4 GB leaves room for 16,384 tokens on paper.

Is the Pingala-8 Llama 3.1 8B file a chat model?

No. It is a compression of the base Llama-3.1-8B, which continues the text you give it rather than following instructions. For chat, use an Instruct GGUF in a stock tool, or our Qwen3-4B file.

Does Pingala-8 Llama 3.1 8B work in Ollama or LM Studio?

No. Pingala-8 files need the patched llama.cpp build from our one-click release, on an NVIDIA RTX 30, 40 or 50 series GPU. Stock llama.cpp, Ollama and LM Studio cannot load them yet.

More guides

• How much VRAM do you need to run an LLM locally?
• What is a 2-bit LLM, and is 2-bit quantization usable?
• IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
• What is perplexity? LLM perplexity in plain English
• How to run Qwen3-4B on a 4 GB or 6 GB GPU
• What is trellis-coded quantization? TCQ, QTIP and IQ2_KT
• What is Hadamard rotation in LLM quantization?