# How Much VRAM to Run an LLM Locally (Worked Out by Hand) > Work out how much VRAM to run an LLM: add the GGUF file size, the KV cache for your context and 1 to 2 GB of overhead. Real sizes for popular models. [Skip to content](#main) [](/) [Services](/services)[AI OS](/ai-os)[Training](/training)[Education](/education)[Agents](/agents)[Learn](/learn) … [Book a call ](/book) 1. [Home](/) 2. [Guides](/guides) 3. How Much VRAM to Run an LLM Locally, Worked Out by Hand # How Much VRAM to Run an LLM Locally, Worked Out by Hand Work out how much VRAM to run an LLM: add the GGUF file size, the KV cache for your context and 1 to 2 GB of overhead. Real sizes for popular models. Updated 4 November 2026 · 9 min read The short answer to how much VRAM to run an LLM: the model file has to fit in your VRAM, with room left for a KV cache that grows with how much text you feed it, plus 1 to 2 GB of runtime overhead. VRAM is the memory on your graphics card that holds the model while it runs. On an Apple Silicon Mac, it's shared unified memory instead. Parameter count alone won't tell you whether a model fits, because packaging matters as much. [Qwen3.5-9B's GGUF files](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/tree/main) are 17.9 GB at full BF16 precision and 5.68 GB at Q4\_K\_M, so the same model needs a 24 GB card in one form and fits an 8 GB card in the other. Don't want to do the sum yourself? Our [Can I Run This LLM agent](https://koso.ai/agents/can-i-run-this-llm) reads your graphics card from the browser and checks every popular open model against it, including how fast each one runs. The rest of this guide shows the arithmetic by hand, so you can check any model yourself. ## The Formula for How Much VRAM to Run an LLM The sum fits on one line: **VRAM needed = GGUF file size + KV cache + runtime overhead**, with a little headroom on top Start from the file size on the model's Hugging Face page, not from parameters times bits per weight: that file is what gets loaded. The KV cache holds keys and values for your prompt and grows with context length. Overhead covers the CUDA or Metal context, activation buffers and memory fragmentation. [Inventive HQ puts that overhead at roughly 1 to 2 GB](https://inventivehq.com/blog/how-much-vram-to-run-an-llm). The headroom covers your desktop and browser, which use VRAM too when the card drives your display. ## How Quantisation Changes the Model File Size Quantisation sets the file size by storing each weight in fewer bits. Full precision keeps 16 bits per weight (FP16 or BF16), Q8\_0 keeps about 8 bits, and Q4\_K\_M averages about 4 to 5\. The file shrinks with the bit count, and lower quants give up some quality. Q4\_K\_M is where most people start for local use. On Qwen3.5-9B it's about 60% of the size of the Q8\_0 file. ### Real GGUF File Sizes for Popular Open Models File sizes in GB: | Model | Q8\_0 | Q6\_K | Q5\_K\_M | Q4\_K\_M | Q3\_K\_M | | --------------------------------------------------------------------------------------------------------- | ----- | ----- | -------- | -------- | -------- | | [Llama 3.1 8B Instruct (2024)](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF) | 8.54 | 6.60 | 5.73 | 4.92 | 4.02 | | Qwen3.5-9B | 9.53 | 7.46 | 6.58 | 5.68 | 4.67 | | Gemma 4 12B | 12.7 | 9.79 | 8.41 | 7.12 | 5.69 | | [DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF) | 15.70 | 12.12 | 10.51 | 8.99 | 7.34 | | [Mistral Small 24B (January 2025)](https://huggingface.co/bartowski/Mistral-Small-24B-Instruct-2501-GGUF) | 25.05 | 19.35 | 16.76 | 14.33 | 11.47 | | [Qwen3.6-27B](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF) | 28.6 | 22.52 | 19.51 | 16.82 | 13.59 | | [Gemma 4 31B](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) | 32.64 | 25.2 | 21.66 | 18.32 | 14.74 | | [Qwen3 32B](https://huggingface.co/bartowski/Qwen%5FQwen3-32B-GGUF) | 34.82 | 26.88 | 23.21 | 19.76 | 15.97 | Sizes from Hugging Face, November 2026\. Llama and Mistral are older releases, and the DeepSeek row is a Qwen-based distill, not DeepSeek-R1 itself. Quant labels don't always shrink anything. [gpt-oss-20b barely shrinks](https://huggingface.co/unsloth/gpt-oss-20b-GGUF) across quants: Q2\_K is 11.5 GB and F16 is 13.8 GB. OpenAI ships its mixture-of-experts layers already in 4-bit MXFP4, so there's little left to cut. ## How the KV Cache Grows With Context Length The second term in the sum is the KV cache. As the model reads your prompt and writes its reply, it stores a key and a value vector for every token at every attention layer. That saves recomputing them, but the store sits in VRAM too. Our visual guide shows [how an LLM pays attention to earlier tokens](https://koso.ai/learn/how-llms-work), if you want the background. Per token, the cost is 2 (key and value) times attention layers, times KV heads, times head dimension, times bytes per element. At f16 an element is 2 bytes. Multiply the result by your context length. Take Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token. At 32K tokens that's 4.3 GB, and at 128K it's 17.2 GB, over three times the 4.92 GB Q4\_K\_M file from the table above. Those [per-token KV cache figures](https://computingforgeeks.com/how-much-vram-to-run-llm/) hold for models like Llama 3.1 that use full attention on every layer. Newer designs break that. Qwen3.5-9B runs full attention on only 8 of its 32 layers, with 4 KV heads and head dimension 256, so it costs about 32 KiB per token, roughly 1 GiB at 32K ([Qwen3.5-9B's model card](https://huggingface.co/Qwen/Qwen3.5-9B) shows the layout). Gemma 4 31B puts a 1,024-token sliding window on 50 of its 60 layers, so an estimate from its config comes to about 3.3 GiB at 32K. Before you trust the formula for any model, open its config file (here's [Gemma 4 12B's config.json](https://huggingface.co/google/gemma-4-12b-it/raw/main/config.json)) and check _layer\_types_ and _num\_key\_value\_heads_. You can also quantise the cache itself. At q8\_0 each element costs about 1.06 bytes and at q4\_0 about 0.56, against 2 bytes at f16. ## A Worked Example: Qwen3.6-27B on a 24 GB Card 1. **File:** 16.82 GB at Q4\_K\_M, straight from the table above. 2. **KV cache:** [Qwen3.6-27B's config.json](https://huggingface.co/Qwen/Qwen3.6-27B/raw/main/config.json) lists 64 layers, 16 with full attention (the rest are linear attention), 4 KV heads and head dimension 256\. So 2 × 16 × 4 × 256 × 2 bytes = 64 KiB per token, about 2.15 GB at 32,768 tokens. 3. **Overhead:** the 1 to 2 GB we added earlier. Add the three: 16.82 + 2.15 + 1 to 2 = about 20 to 21 GB at 32K context. Against [NVIDIA's spec table](https://www.nvidia.com/en-us/geforce/graphics-cards/compare/?section=compare-specs), a 16 GB RTX 5060 Ti can't hold even the file. A 24 GB RTX 4090 fits it with 3 to 4 GB spare. Context alone flips this. At 128K the cache is 8 GiB, about 8.6 GB, and the total reaches 26 to 27 GB: over 24 GB, so the model spills to system RAM. Quantise the cache to q8\_0 and it drops to about 4.5 GB, for a total of 22.4 to 23.4 GB. That squeezes back into 24 GB, with little room left over. On a [32 GB MacBook Pro](https://www.apple.com/macbook-pro/specs/) the GPU only gets part of the unified memory (the FAQ covers how much), so 20 GB is borderline. A 64 GB M5 Pro is comfortable. ## Why Memory Bandwidth Sets Your Tokens per Second At batch size 1, each new token reads every weight from memory once, so your top speed is [memory bandwidth divided by the bytes read per token](https://solmaz.io/llm-throughput-upper-bounds): roughly the file size for a dense model. Apple lists [307 GB/s for the M5 Pro and 614 GB/s for the M5 Max](https://www.apple.com/newsroom/2026/03/apple-debuts-m5-pro-and-m5-max-to-supercharge-the-most-demanding-pro-workflows/). Divide by the 16.82 GB Qwen3.6-27B file and you get upper limits of about 18 and 36 tokens per second. Real speed lands below the ceiling. In [llama.cpp's Apple Silicon benchmarks](https://github.com/ggml-org/llama.cpp/discussions/4167), an older M4 Pro at 273 GB/s hit 30.7 tokens per second on a 7B Q8\_0 model, against a ceiling near 38, so about 80% of it. Long context lowers the ceiling further, since the KV cache is read every token too. That's also why the 8 GB and 16 GB RTX 5060 Ti run the same model at the same speed: NVIDIA rates both at 448 GB/s. The extra memory buys room for a bigger model. ## What Happens When a Model Doesn't Fit in VRAM Ollama and llama.cpp don't quit: they put as many layers on the GPU as fit and leave the rest in system RAM. Every token still runs through every layer, so the RAM part sets the pace. The bandwidth rule from the last section prices that. Dual-channel DDR5-5600 moves about 90 GB/s (2 channels × 8 bytes × 5,600 MT/s, worked out from the memory spec). An RTX 5060 Ti moves 448 GB/s. Put Qwen3 32B Q4\_K\_M (19.76 GB) on that 16 GB card with about 6 GB in RAM: 31 ms per token on the GPU part, 67 ms on the RAM part. That caps you near 10 tokens/s, against about 23 if the file fit. No measured benchmark for a dense model spilling to RAM turned up in our research, so treat those as ceilings. [One guide reports roughly 1 to 3 tokens per second](https://www.promptquorum.com/local-llms/run-70b-models-24gb-vram) for a 70B Q4 on a 24 GB card with about 11 GB offloaded, with no test rig stated. You can check the split with _ollama ps_: [Ollama's FAQ](https://docs.ollama.com/faq) shows its Processor column reading "48%/52% CPU/GPU" instead of "100% GPU". Before you accept a split, try the cheaper fixes. Keep _num\_ctx_ as low as your work allows, run the KV cache at q8\_0 with flash attention on to roughly halve it, or drop one quant level. ## How Mixture-of-Experts Models Change the Maths Mixture-of-experts (MoE) models spill differently. They split most layers into many small experts and uses only a few for each token. [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) holds 35B parameters but activates only about 3B of them per token. Memory follows the full file: [Qwen3 30B-A3B](https://huggingface.co/bartowski/Qwen%5FQwen3-30B-A3B-GGUF) at Q4\_K\_M is 18.63 GB, close to the dense Qwen3 32B's 19.76 GB above. Speed follows the active part, so the bandwidth rule from earlier gives it a much higher ceiling on the same card. [gpt-oss-120b](https://huggingface.co/ggml-org/gpt-oss-120b-GGUF) follows the same pattern at a much bigger scale, with a 63.39 GB file. That split makes expert offload workable. llama.cpp's _\--cpu-moe_ keeps experts in system RAM and attention on the GPU. Each token touches few experts, so RAM reads stay small. [One tester's 16 GB run](https://www.developersdigest.tech/blog/llama-cpp-moe-offload-n-cpu-moe-vs-moe-cache-2026) put a 78 GB MoE on an RTX 5060 Ti with 62 GB of RAM at 14.95 tokens/s, 19.26 with an expert cache. That cache, _\--moe-cache-mib_, landed in [llama.cpp PR #29887](https://github.com/ggml-org/llama.cpp/pull/29887) on 7 October 2026\. Its author measured 1.57x to 2.2x speed-ups on one system, though a user reported a slowdown on a two-GPU setup, so gains aren't guaranteed. ## VRAM For Local LLMs FAQ ### How much VRAM do I need to run a 70B model? ComputingForGeeks lists Llama 3.3 70B at 42.5 GB in Q4\_K\_M, plus about 10.7 GB of KV cache at 32K context. One 24 GB card can't hold that, and neither can a 32 GB RTX 5090\. Two 24 GB cards (48 GB) hold the file, with room for only a short context. ### Can I run an LLM on an 8 GB GPU? Yes, for 8B to 9B models at Q4\_K\_M. Qwen3.5-9B's 5.68 GB file plus about 1 GB of cache at 32K and 1 GB of overhead comes to about 7.8 GB, which is tight, so a 16K context is safer. Llama 3.1 8B (4.92 GB) fits Ollama's default 4K context but not 32K, where its cache alone is 4.3 GB. ### What can a 24 GB GPU like the RTX 4090 run? Dense models up to about 31B at Q4\_K\_M with a 32K context, such as Qwen3.6-27B, Gemma 4 31B and Mistral Small 24B, plus 30B-class MoE models like Qwen3 30B-A3B. Qwen3 32B needs about 30 GB at 32K, so it spills unless you cap the context. ### Is unified memory the same as VRAM? Not quite. On Apple Silicon the GPU and CPU share one pool, so there's no separate VRAM. macOS lets the GPU use [roughly 65 to 75% of it](https://macos-tahoe.com/blog/mac-unified-memory-local-llm-how-much-ram-m5-ultra-512gb-guide-2026/) by default, so about 21 to 24 GB on a 32 GB Mac. The sysctl iogpu.wired\_limit\_mb raises it until reboot, and [Mac Studio with M5 Ultra](https://www.apple.com/mac-studio/specs/) reaches 512 GB. ### What's the rule of thumb for model size vs VRAM? Q4\_K\_M files run about 0.6 GB per billion parameters: 4.92 GB for an 8B model, 19.76 GB for a 32B. Add 1 to 2 GB of runtime overhead and the KV cache for the context you want, then keep 1 GB spare. ## Final Thoughts Start from the GGUF file size on Hugging Face, then add the cache for the context you'll actually use and 1 to 2 GB of overhead. If the total doesn't fit, shrink the context or quantise the cache before you drop a quant level or let layers spill into RAM. With an MoE model, offload the experts first. To skip the sums, the [Can I Run This LLM](https://koso.ai/agents/can-i-run-this-llm) agent reads your graphics card in the browser and checks every popular open model against it, with an estimate of how fast each one runs. If you plan to build on top of a local model, our chapter on [how AI agents work](https://koso.ai/learn/ai-agents) is a good next read. ## Can I Run This LLM Reads your graphics card from the browser, then measures every popular open model against it: what runs, how fast, and the command to pull it. [Check your PC ](/agents/can-i-run-this-llm) ## Try The Other Agents. Every tool here is a live multi-agent pipeline, free to run, reasoning on display. [All agents ](/agents) [LiveDomain AppraisalSix agents value any domain: real web history, a 1000-point score, real sales, retail and wholesale prices.Appraise a domain ](/agents/domain-appraisal)[LiveCompany ResearchSix agents build a dossier on any company and benchmark it against its rivals.Research a company ](/agents/company-research)[LiveAI Agent ReadinessHow well your website works when ChatGPT and Perplexity come to read it, out of 100.Check your site ](/agents/ai-readiness)[LiveAEO Page AuditChecks whether ChatGPT, Perplexity and Google's AI answers would quote one of your pages, asks ChatGPT, and scores it out of 100.Audit a page ](/agents/aeo-audit)[LiveProduct Market FitA manager agent hires analyst agents to test whether your product fits its market.Analyze your product ](/agents/product-market-fit)[LiveAgent Workflow GeneratorDescribe the agent you want. It asks 2 or 3 sharp questions, then draws the workflow and writes the build spec.Design your agent ](/agents/workflow-generator)[LiveICP GeneratorReads your real pricing page and job board, then builds the ideal customer profile: firmographics, the buying committee, triggers, and who to say no to.Build your ICP ](/agents/icp-generator)[LiveProspect ResearchGive a name and a company. Agents read the public professional web and write your call-prep note: who they are, how to talk to them, and what to open with.Research a prospect ](/agents/prospect-research)[LiveAI Skill BuilderDescribe a capability and it writes you an installable SKILL.md for Claude Code, Cursor and Copilot, after showing you what already exists on skills.sh.Build a skill ](/agents/skill-builder)[LiveAI Slop DetectorRenders your landing page in a real browser, boxes every AI-template tell on a screenshot and scores it out of 100.Scan your page ](/agents/ai-slop-detector)[Coming soonBuild Your Own AgentSoon you'll design an agent like these yourself, right here. Sign in and you're on the list for early access.Get on the list ](/login) A custom AI development agency. Bespoke agents, copilots and products engineered for teams that want a moat, not a subscription. Ideanab Private Ltd is a company registered in England and Wales, no. [16344048](https://find-and-update.company-information.service.gov.uk/company/16344048). Registered office: 124 City Road, London, EC1V 2NX, United Kingdom. ### Company * [Services](/services) * [Custom AI Agency](/custom-ai-agency) * [AI Product Development](/ai-product-development) * [AI OS](/ai-os) * [AI Engineering](/ai-engineering-company) * [AI Consulting](/ai-consulting-company) * [AI Consulting Services](/artificial-intelligence-consulting-services) * [Training](/training) * [Education](/education) * [Free AI Agents](/agents) * [Learn AI](/learn) * [Guides](/guides) * [Fractional CAIO](/caio) * [About Us](/about) * [The Koso Deck](/deck) * [Careers](/careers) * [Docs for agents](/docs) ### Contact * [Contact us](/contact) * [Book a call](/book) * [LinkedIn](https://www.linkedin.com/company/koso) * [Instagram](https://www.instagram.com/okaashish) ### Legal * [Terms of Service](/terms) * [Privacy Policy](/privacy) * [Cookie Policy](/cookies) * [Acceptable Use Policy](/acceptable-use) * [KosoBot](/bot) © 2026 Koso, a brand of Ideanab Private Ltd. All rights reserved.