Local LLM Calculator: Accurate VRAM, RAM & Quantization Sizing Guide

Calculating local LLM memory requirements relies on sizing three distinct elements: model weights, key-value (KV) cache, and runtime execution buffers [8]. A model quantized to 4-bit precision generally requires roughly 0.6 GB of memory per billion parameters [6]. Sizing your system accurately ensures the selected context window and model fit into dedicated VRAM or unified memory without triggering operating system paging or out-of-memory errors [8].

Core LLM Memory Math: Model Weights, KV Cache, and Overhead

Determining the exact memory footprint requires a 3-part memory equation [8]:

$$\text{Total VRAM} = \text{Model Weights} + \text{KV Cache} + \text{Runtime Overhead}$$

Model weights represent the static disk-to-memory footprint of the network layers [8]. The KV cache stores attention states across previous tokens, dynamically expanding as the input and generation sequence grows [1][8]. Runtime overhead covers memory allocators, framework runtime contexts, activations, and compute buffers necessary to execute inference steps [1][8].

Weight Precision and Quantization Multipliers (Q4_K_M vs Q8_0)

Weight footprint scales directly with parameter count and target precision [8]. Direct measurements across parameter counts reported by Ollama demonstrate that a model quantized to Q4_K_M uses roughly 0.6 bytes per parameter, whereas 8-bit precision (Q8_0) uses roughly 1.07 bytes per parameter [8].

Because of this scaling:

A 4-bit (Q4) quantized model requires roughly 0.6 GB of memory per billion parameters [6].
An 8-bit (Q8_0) quantized model requires roughly 1.07 GB of memory per billion parameters [8].
Uncompressed 16-bit precision (FP16 or BF16) requires approximately 2.0 bytes per parameter, or roughly 2.0 GB per billion parameters [1][6].

KV Cache Scaling Across Context Windows

The KV cache footprint scales with context window length, batch size, and KV cache precision [1][4]. Sizing long context windows significantly impacts resident memory:

On a 27B model at a 128K context window, an F16 KV cache consumes 30.1 GiB of resident memory [4].
Downscaling the KV cache precision to Q8_0 (8.5 bits per weight) reduces that 27B 128K context footprint to 26.1 GiB [4].
Downscaling further to Q4_0 (4.5 bits per weight) brings the footprint down to 24.0 GiB [4].

Lowering KV cache precision reduces total allocation when running long sequences or multi-turn agent conversations [1][4].

Accounting for Runtime Buffer Margins and OS Overhead

Runtime overhead accounts for framework-level workspace allocations, attention scratchpads, and CUDA/driver state [1][8]. Multi-Token Prediction (MTP) speculative decoding, for example, adds layer weights, full-context KV allocation, and approximately 2 GiB of runtime buffers [4]. When planning local hardware capacity, calculators rely on distinct buffer safety modes [10]:

Theoretical mode: Ideal minimum allocation with no safety buffer [10].
Conservative mode: Adds a 10% safety buffer over baseline memory math [10].
Production mode: Adds a 25% safety buffer to accommodate multi-user concurrency and dynamic runtime activations [10].

Worked Hardware Examples: From 8GB Laptops to 24GB GPUs

Testing 8GB Unified Memory: phi3:mini vs mistral:7b Real-World Measurements

To test the precision of calculated estimates against real physical memory footprints, consider phi3:mini and mistral:7b evaluated on an 8GB Apple Silicon machine running Ollama [8]:

Metric / Parameter	`phi3:mini` Measurement	`mistral:7b` Measurement
Parameter Count	3.8B parameters [8]	7B parameters [8]
Quantization Level	Q4_0 [8]	Q4_0 [8]
Context Length	4,096 tokens [8]	4,096 tokens [8]
Architecture Detail	32 layers [8]	Standard dense [8]
Calculated Estimate	~3.71 GB [8]	~4.78 GB [8]
Measured Physical Footprint	3,624 MB [8]	Swap activity observed; physical RSS measurement not completed due to OS memory pressure [8]
Calculated vs. Measured Delta	4.7% [8]	N/A — the mistral:7b result demonstrates that calculator estimates alone do not capture OS-level memory pressure; treat the 4.78 GB estimate as a lower bound on an 8GB system [8]

On the 8GB test configuration, phi3:mini fit into physical memory within a 4.7% delta of its 3.71 GB mathematical estimate [8]. Conversely, while mistral:7b mathematically required ~4.78 GB—fitting the 8GB ceiling on paper—the operating system and desktop overhead forced aggressive memory swapping, causing steep performance drops [8].

Single 24GB Consumer GPU Benchmarking: Fitting 15B to 35B Parameter Models

On a single 24GB GPU (such as an RTX 3090 or RTX 4090), local LLMs in the 15B to 35B parameter range fit at Q4 quantization [7][13].

In BenchLM’s documented single-GPU catalog, Alibaba’s Qwen3.6-27B achieves a score of 48/100 using a 24GB RTX 4090 reference setup, representing the highest score among documented single-GPU configurations [7]. Tests across 11 local models spanning 15B to 35B at Q4 quantization on an RTX 3090 with 24GB of VRAM show models such as Gemma 4 31B, Qwen3 32B, and Qwen3.5 27B successfully processing reasoning and benchmark suites within a single 24GB card’s VRAM envelope [13].

Hardware Planning Across Memory Tiers

VRAM Headroom Budgeting:
┌────────────────────────────────────────────────────────┐
│ Discrete GPU (e.g., RTX 4090 24GB): ~90% Usable VRAM   │
│ [===========================================---] ~21.6GB│
└────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────┐
│ Apple Silicon (e.g., Mac 32GB): ~70%–85% Usable RAM    │
│ [====================================----------] ~22.4GB│
└────────────────────────────────────────────────────────┘

Discrete GPU VRAM vs Apple Silicon Unified Memory Budgets

Available memory depends heavily on your system architecture:

Discrete GPUs: ModelFit budgets roughly 90% of dedicated VRAM for model execution [6].
Apple Silicon Unified Memory: Unified architecture shares RAM between the operating system, CPU, and GPU. Calculators budget roughly 70% of unified memory for the model on configurations up to 32GB, scaling to approximately 85% on 128GB and higher configurations [6].

Based on these budgeting rules, consumer hardware tiers unlock specific parameter brackets at Q4_K_M quantization [6]:

Memory Tier	Usable Model Budget (~70%–85%)	Max Dense Model Size (Q4_K_M)	Example Compatible Model
8 GB	~5.6 GB [6]	~9B parameters [6]	Ornith 1.0 9B (~5.6 GB) [6]
16 GB	~11.2 GB [6]	~14B parameters [6]	Qwen3 14B (~11 GB) [6]
24 GB	~16.8 GB [6]	~29.3B parameters [6]	Qwen3.8 27B (~16.5 GB) [6]
32 GB	~22.4 GB [6]	~35B parameters [6]	Ornith 1.0 35B (~21.2 GB) [6]
64 GB	~48.0 GB [6]	~70B parameters [6]	Llama 3.3 70B Instruct (~42 GB) [6]

Handling Mixture of Experts (MoE) Architectures

Mixture of Experts (MoE) architectures execute inference by routing tokens to a subset of parameters per step, but every expert weight must reside in resident VRAM to preserve routing without latency penalties [1][8]. MoE models conserve compute operations during inference rather than memory footprint [1]. When using sizing calculators for MoE designs, total parameter count must be used rather than active parameter count [8].

Calculator Modes: Theoretical, Conservative, and Production Deployment

When budgeting local hardware, calculators like local_inference_calculator provide three planning modes to account for different execution environments [10]:

Buffer Allocation Breakdown:
┌─────────────────────────┬──────────────────────┬──────────────────────────────────┐
│ Theoretical (0% Buffer) │ Conservative (+10%)  │ Production (+25%)                │
├─────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Ideal minimum memory    │ Standard local use   │ Real-world serving concurrency   │
│ Base Weights + KV Cache │ Single-user desktop  │ Dynamic activations & spikes     │
└─────────────────────────┴──────────────────────┴──────────────────────────────────┘

Theoretical Mode (Ideal Minimum): Calculates exact static weights plus baseline KV cache allocation without adding safety buffers [10]. This mode shows whether an LLM can initialize in memory under laboratory conditions [10].
Conservative Mode (10% Buffer): Adds a 10% memory headroom over the mathematical minimum [10]. This serves as the recommended default for desktop interfaces like Ollama or LM Studio, absorbing small context jumps and standard system overhead [8][10].
Production Mode (25% Buffer): Adds a 25% memory cushion to accommodate concurrent request batches, dynamic activation spikes, and multi-tenant server workloads without crashing [1][10].

Local LLM Hardware Sizing FAQ

Does CPU offload help when a model exceeds VRAM, and what is the performance cost?

CPU offload (via llama.cpp’s --n-gpu-layers flag) allows layers not loaded onto the GPU to be computed on the CPU. However, PCIe bandwidth between CPU and GPU is far narrower than on-device memory bandwidth, so each offloaded layer adds latency per token. In practice, offloading even a fraction of layers to CPU RAM can reduce generation throughput substantially compared to a fully GPU-resident run [10][1]. For reference, SitePoint’s measured benchmarks show a fully GPU-resident Llama 4 Scout Q4 on an RTX 4090 achieving ~33 tok/s, while the same model with partial CPU offload drops well below that threshold [14]. Use the theoretical and conservative calculator modes to find a quantization level that keeps all layers on-GPU before resorting to offload.

Why does an 8GB device struggle with Mistral 7B even if it mathematically fits?

Running mistral:7b at a 4,096 context length requires an estimated 4.78 GB of memory [8]. On an 8GB unified memory system, the remaining budget is constrained because the operating system, display server, and active background applications share the same pool [6][8]. When free memory is exhausted, the system initiates aggressive swap memory activity to disk, causing generation speeds to drop sharply [8]. On macOS, unified memory reclaim is managed by the kernel and is generally more graceful than on Windows, where VRAM eviction to pagefile can cause more abrupt performance cliffs — though in both cases the result is a steep drop in tokens per second once physical memory is exhausted.

What quantization format should I choose for the best quality-to-size tradeoff?

The right quantization depends on your available memory and quality tolerance. Q4_K_M (~0.6 GB/B) is the standard starting point for consumer hardware. Q8_0 (~1.07 GB/B) recovers most quality lost at 4-bit and is preferred when memory allows [8]. FP16 (~2.0 GB/B) is the uncompressed baseline used in benchmarks and production deployments [1]. Lower formats such as Q2_K and IQ4_XS trade further quality degradation for smaller footprints and are best treated as last-resort options when a model would otherwise not fit at all.