Pingala-8 guides

Can you run Qwen3.8-27B on a 12 GB GPU?

Genrius AI guide. November 1, 2026. About 6 minutes.

Check your own card and context: Normal vs Pingala-8 calculator

Which files fit in 12 GB

Totals from our GPU fit calculator: the file, plus the 16-bit KV cache, plus 1.5 GB for compute buffers and your desktop. Qwen3.8-27B has a small cache for its size (0.43 GB at 4,096 tokens, 0.69 GB at 8,192, 2.30 GB at 32,768), because only 16 of its 64 layers keep one; the VRAM requirements page shows the arithmetic.

Total memory on a 12 GB card by the calculator: estimates, not measurements, except where marked; the one measured run is our file, fully on the GPU at context 4096 on one RTX 5070 Ti Laptop GPU. "Fits" leaves at least 1.5 GB for buffers; "tight" means the file and cache fit but less than 1.5 GB is left for buffers. File sizes from Hugging Face (ours, jkim96/Qwen3.8-27B-DASHQ-Q2-GGUF, unsloth/Qwen3.8-27B-GGUF); the Q4_K_M and FP16 rows are the formula sizes from the VRAM page. GB = 10⁹ bytes. We have not run any of these files on an RTX 3060, 4070 or any other 12 GB card.

At 32,768 tokens, -fa on -ctk q8_0 -ctv q8_0 stores the full-attention cache at 8 bits, which roughly halves that part of the cache, or you can use a shorter context. Q4_K_M and larger files only run on a 12 GB card with some layers in system RAM (a lower -ngl), which is much slower.

Quality: DASH-Q is ahead

WikiText-2 test at context 4096, the first 20 of 72 windows (a time cap on one laptop GPU), every file on the same windows; lower is better. In our runs the DASH-Q and unsloth files were on stock llama.cpp with their IQ1_S, IQ2_S and IQ3_S tensors on the CPU, because those CUDA kernels gave NaN on our GPU (details). Per-window losses: eval dataset; sortable table: 2-bit GGUF leaderboard.

Speed

Measured for our file only: 37.7 tokens/s token generation at batch 1, every layer on one RTX 5070 Ti Laptop GPU (12 GB, WSL2, -ngl 99). We did not measure the speed of the DASH-Q or unsloth files: in our runs part of each ran on the CPU, so a number from our machine would not be a fair one.

How to run ours

Build llama.cpp at commit 7ac59a6 with the Pingala-8 CUDA patch and the Qwen3.5 loader change, then download the 9-part file and pass part 1:

git clone https://github.com/Genrius-AI/pingala8-llm
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 7ac59a6e3ad851cd41af00f678effab0598ba9a8
git apply ../pingala8-llm/patches/llama_cuda_vs_7ac59a6.patch
git apply ../pingala8-llm/patches/q35_loader_count_delta.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release -DLLAMA_OPENSSL=OFF
cmake --build build -j8 --target llama-cli llama-perplexity llama-bench

hf download Genrius-AI/Qwen3.8-27B-Pingala8-2bit --local-dir qwen3.8-27b-pingala8
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
./build/bin/llama-cli -m qwen3.8-27b-pingala8/Qwen3.8-27B-Pingala8-2bit-00001-of-00009.gguf -ngl 99 -c 4096

These are the steps from the model card (How to run), which also has the measured perplexity command. The architecture flag 120 matches our RTX 5070 Ti Laptop GPU, the only card this file has been run on. This 27B file has not been tested with the one-click build. It is the text model only: no vision tower and no multi-token-prediction block.

When to choose DASH-Q or unsloth instead

• DASH-Q IQ2_XS if quality comes first. Of the five files we measured it has the lowest perplexity, 6.54, better than ours on every one of the 20 windows, for 0.24 GB more than our file.
• DASH-Q or unsloth if you want stock llama.cpp or a tool built on it. Both use standard llama.cpp quant types; ours needs our patched CUDA runtime. We ran them only on stock llama.cpp, not in Ollama or LM Studio.
• unsloth UD-IQ2_S is close to ours on this test: 7.34 against 6.89, a difference that is not significant over 20 windows.
• Pingala-8 v2 if you already use our runtime on an NVIDIA RTX 50 card like ours: it is the smallest of the three files (8.31 GB) and the one we measured fully on the GPU, at 37.7 tokens/s. If your card's IQ1_S, IQ2_S or IQ3_S CUDA kernels give NaN, as ours did, the rival files need the CPU workaround on the NaN page.

Questions people ask

Can I run Qwen3.8-27B on a 12 GB GPU?

Yes, with a 2-bit file. Our Pingala-8 v2 file (8.31 GB) runs with every layer on one 12 GB RTX 5070 Ti Laptop GPU at 37.7 tokens/s, batch 1; the calculator shows 10.24 GB in total at context 4096. The DASH-Q IQ2_XS (8.55 GB) and unsloth UD-IQ2_S (8.37 GB) files also fit at 4k and 8k context on paper.

Does Qwen3.8-27B Q4_K_M fit in 12 GB?

No. Q4_K_M weights are about 17 GB (unsloth's UD-Q4_K_M file is 16.46 GB), more than the card before any cache. You would have to keep layers in system RAM, which is much slower.

Which 2-bit Qwen3.8-27B GGUF has the best quality?

Of the five files we measured, DASH-Q IQ2_XS: WikiText-2 perplexity 6.54 at 8.55 GB. Ours is 6.89 at 8.31 GB, unsloth UD-IQ2_S 7.34 at 8.37 GB (ours lower, but not significant: better on 9/20 windows) and llama.cpp IQ2_XXS 7.88 at 8.43 GB.

Does the Pingala-8 file work in Ollama or LM Studio?

No. It needs the patched Pingala-8 llama.cpp runtime (CUDA); stock llama.cpp, Ollama and LM Studio refuse it with a load error. It has not been tested with the one-click build.

Was it tested on an RTX 3060 or RTX 4070?

No. It has been run on one GPU only, an RTX 5070 Ti Laptop GPU with 12 GB under WSL2. For other 12 GB cards the fit numbers on this page are calculator estimates.

Is it text-only?

Yes. The vision tower and the multi-token-prediction block of Qwen3.8-27B are not included.

More guides

• Qwen3.8 27B VRAM requirements: which GPU do you need?
• Qwen3.5-2B under 1 GB: 2-bit GGUFs measured
• NaN perplexity with IQ1_S, IQ2_S or IQ3_S GGUFs on an RTX 50 laptop GPU
• How much VRAM do you need to run an LLM locally?
• What is a 2-bit LLM, and is 2-bit quantization usable?
• IQ2_XXS vs IQ2_KT vs Q2_K vs Q4_K_M: which GGUF quant?
• What is perplexity? LLM perplexity in plain English
• Can you run gpt-oss-20b on a 12 GB or 16 GB GPU?
• How to run Qwen3-4B on a 4 GB or 6 GB GPU
• How to run Llama 3.1 8B on a 4 GB or 6 GB GPU
• Qwen3.6 35B-A3B on a consumer GPU: VRAM and MoE offload