Local LLM Inference: Memory Bandwidth, KV Cache, Quantization, and Tokens/sec
Learn what actually limits local LLM inference, including memory bandwidth, KV cache, context length, quantization, and real-world tokens per second.
Local LLM Inference: Memory Bandwidth, KV Cache, Quantization, and Tokens/sec
Running local LLMs taught me that having enough RAM to load a model is only the beginning. Interactive inference also depends on memory bandwidth, KV-cache size, context length, quantization, accelerator performance, and runtime efficiency.
For my workloads, roughly 50 output tokens per second is where local inference starts feeling comfortably responsive. That is a personal threshold rather than a universal benchmark.
Why More RAM Does Not Automatically Make an LLM Faster
Memory capacity and memory bandwidth solve different problems:
Capacity = how much data can remain resident
Bandwidth = how quickly that data can be delivered
Large amounts of RAM let you load larger models and keep more model state in memory. They do not automatically increase decoding speed.
For dense transformer models, a useful simplified mental model is:
tokens/sec ≈ effective memory bandwidth / bytes of model data moved per token
This is not a complete performance equation. Compute utilization, kernel efficiency, architecture, batching, quantization, speculative decoding, attention implementation, and cache behavior also matter.
But it explains why a system can have hundreds of gigabytes of free memory and still generate tokens relatively slowly.
Why Batch-Size-One Inference Is Bandwidth Sensitive
Interactive chatbot workloads usually generate one response at a time. At low batch sizes, there is less opportunity to reuse computation while model weights are being moved.
That makes decoding particularly sensitive to memory bandwidth.
It also means headline compute specifications such as TOPS do not tell the whole story. A processor may have enormous theoretical compute capability but still spend time waiting for model data.
When evaluating local LLM hardware, I consider these together:
- memory capacity;
- memory bandwidth;
- accelerator performance;
- model representation;
- KV-cache requirements;
- context length;
- concurrency;
- runtime efficiency.
KV Cache Is the Other Major Memory Consumer
Model weights are not the only significant allocation during inference.
Transformers maintain a KV cache containing key and value representations for previously processed tokens. Without it, information from earlier tokens would need to be repeatedly recomputed.
A simplified memory budget looks like:
Total memory usage ≈
model weights
+ KV cache
+ compute buffers
+ runtime overhead
+ operating system memory
+ other applications
KV-cache requirements depend on factors including:
- transformer layer count;
- number of KV attention heads;
- head dimension;
- number of cached tokens;
- KV-cache precision;
- concurrent sequences.
Architectures using grouped-query attention or multi-query attention can require less KV-cache memory than traditional multi-head attention.
There is therefore no universal amount of KV-cache memory consumed per token.
Long Context Windows Have a Hardware Cost
A model supporting 128K or larger context does not mean that context is free to use locally.
Longer contexts increase cache state and attention work. The effect becomes especially important with:
- long coding sessions;
- large system prompts;
- RAG document sets;
- agent histories;
- parallel conversations;
- concurrent inference requests.
A model may load successfully with plenty of memory remaining, then require substantially more memory once a large context is allocated.
Testing whether a model merely starts is therefore insufficient. Test the context length and concurrency you expect to use in practice.
llama.cpp Exposes These Tradeoffs
llama.cpp exposes separate controls for key and value cache configuration.
Its KV cache can use multiple representations, including F32, F16, BF16, Q8, Q5, and Q4 variants. Cache size also changes with context length, architecture, precision, and the number of sequences.
Reducing KV-cache precision can save meaningful memory, but it introduces another tradeoff that should be measured on the actual workload.
Why Swap Can Destroy Local LLM Performance
Unified-memory systems are attractive because CPU and GPU workloads can share the same memory pool.
However, unified memory is still finite.
When the combination of model weights, KV cache, buffers, applications, and operating-system usage creates enough pressure, macOS can use memory compression and eventually swap.
Swap is backed by SSD storage rather than main memory. That introduces dramatically lower throughput and higher latency than modern system memory.
The system may continue working, but token generation can slow and latency can become inconsistent.
More RAM helps prevent this situation. It does not remove the underlying bandwidth constraint.
Quantization Makes Large Local LLMs Possible
Quantization reduces model-weight precision and therefore memory consumption.
For example, a rough estimate for a 70B-parameter model is:
FP16:
70B × 2 bytes ≈ 140 GB
8-bit:
70B × 1 byte ≈ 70 GB
4-bit:
70B × 0.5 byte ≈ 35 GB
Actual model files differ because quantization formats include scales, metadata, mixed-precision tensors, and other overhead.
The important point is that quantization can turn a model that does not fit locally into one that does.
It can also reduce the amount of model data that must move through memory during inference, potentially improving performance.
Quantization Is Not Free
A quantized model approximates its higher-precision counterpart.
Lower precision can potentially reduce quality, although modern quantization methods can preserve strong performance at moderate precision.
The impact varies with:
- model architecture;
- quantization algorithm;
- number of bits;
- calibration method;
- reasoning workload;
- context length;
- benchmark;
- downstream task.
For summarization, extraction, text generation, or some coding workloads, differences may be difficult to notice. More demanding reasoning, mathematics, edge cases, or tasks near the model's capability limit can be more sensitive.
The practical tradeoff is:
smaller representation
→ lower memory use
→ less bandwidth traffic
→ potentially faster inference
→ potentially lower quality
Smaller Models Can Be More Useful
The largest model that fits in memory is not necessarily the best local model.
For interactive work, latency matters.
A smaller model that solves the task reliably at much higher speed can be more useful than a larger model producing only a few tokens per second.
This is especially relevant for:
- autocomplete;
- short coding tasks;
- structured extraction;
- rewriting;
- classification;
- deterministic tool use;
- constrained-domain RAG.
A useful rule is to choose the smallest model that reliably solves the task at an acceptable speed.
Prefill and Decode Are Different Metrics
"Tokens per second" can describe very different workloads.
Prefill
During prefill, the model processes existing input such as:
- the system prompt;
- conversation history;
- RAG documents;
- the user's message.
Many prompt tokens can be processed in parallel, allowing high prompt-processing throughput.
Decode
Decode is the output-generation phase.
Autoregressive decoding is sequential:
token N → token N+1 → token N+2
Each generated token depends on the previous token, limiting parallelism for a single response.
When I refer to roughly 50 tokens/sec as a practical local threshold, I mean single-response output decoding, not prompt processing or aggregate server throughput.
Benchmarks reporting hundreds or thousands of tokens per second may instead measure batching, prefill, multiple users, or aggregate throughput.
Concurrency Changes the Hardware Requirements
Serving one developer is different from serving multiple users or agents.
Additional sequences increase KV-cache state, change batching behavior, alter memory traffic, and introduce scheduling overhead.
For inference servers, I would separate three measurements:
time to first token
single-request decode tokens/sec
aggregate tokens/sec under concurrency
They describe different aspects of system performance.
RAG Still Consumes Context
Retrieval-augmented generation does not eliminate context costs.
RAG retrieves relevant information and inserts it into the model's prompt. Those tokens still need to be processed and cached.
If a request retrieves 20,000 tokens of documentation, those 20,000 tokens still contribute to prompt processing and context usage.
A strong RAG system should therefore retrieve the smallest amount of information required to answer correctly.
That can improve:
- prompt-processing latency;
- KV-cache usage;
- model focus;
- answer quality;
- infrastructure cost.
Context engineering is partly a hardware optimization.
Agents Amplify Inference Costs
Agents may perform repeated cycles such as:
- reason;
- call a tool;
- observe the result;
- update context;
- reason again;
- call another tool.
A task that produces a 500-token final answer might require thousands of intermediate tokens across many sequential model invocations.
A model generating 10–15 tokens/sec may be tolerable for one response but painfully slow across many agent steps.
Raw tokens/sec also cannot be separated completely from model capability. A faster but weaker model may require more iterations, while a slower but more capable model may solve the task in fewer steps.
Around 50 Output Tokens/sec Is My Practical Threshold
After experimenting with local models, my rough experience is:
- 5–10 tokens/sec: useful for batch processing, document analysis, or privacy-sensitive jobs, but slow for interactive work;
- 20–30 tokens/sec: substantially more comfortable;
- around 50 tokens/sec: responsive enough that generation speed stops dominating many normal workflows.
This is not a universal threshold. Different workloads have different latency requirements.
The real decision depends on the combination of:
model quality
+ generation speed
+ context capability
+ memory requirement
+ privacy
+ operational simplicity
Apple Silicon and DGX Spark Solve Different Problems
High-memory Apple Silicon systems are attractive because unified memory can accommodate models that may not fit into the VRAM of a single conventional GPU.
The M3 Ultra Mac Studio can be configured with up to 512 GB of unified memory, with Apple specifying more than 800 GB/s of unified-memory bandwidth.
That makes systems in this category interesting for large quantized models, local RAG, long contexts, research, evaluation, and workloads where capacity matters more than maximum decode speed.
NVIDIA's DGX Spark takes a different approach. It combines a Grace Blackwell platform with 128 GB of coherent LPDDR5X memory and a CUDA-focused software stack. NVIDIA specifies 273 GB/s of memory bandwidth and positions the platform for models up to 200 billion parameters depending on representation and workload.
Neither example changes the basic rule: memory capacity does not guarantee high single-user decoding speed.
Why Specialized Inference Hardware Can Be Much Faster
Architectures built specifically for inference can have radically different memory systems.
Groq describes rack-scale infrastructure using large amounts of on-chip SRAM and extremely high aggregate SRAM bandwidth, while Cerebras places SRAM directly alongside compute on its wafer-scale processors.
The important distinction is architectural.
Adding hundreds of gigabytes of conventional system RAM is not equivalent to placing model data next to compute with petabytes per second of aggregate on-chip bandwidth.
This is why specialized inference systems can target throughput that normal desktop architectures cannot achieve merely by adding more memory.
How to Choose Hardware for Local LLMs
There is no single specification that determines whether a system is good for local AI.
I evaluate hardware roughly in this order:
- model quality for the target task;
- enough memory for the model and working set;
- memory bandwidth;
- accelerator compute;
- quantization format;
- KV-cache requirements;
- context length;
- concurrency;
- runtime optimization;
- measured performance on the real workload.
A machine with 256 GB of RAM does not automatically run a 40 GB model faster than a system with less memory.
Likewise, a very fast GPU is not useful for a model that cannot remain resident in its available memory.
The system has to be balanced.
Frequently asked questions
Does more RAM increase local LLM tokens per second?
Not necessarily. More RAM increases the amount of model and context data that can remain resident. Decode speed is also strongly affected by memory bandwidth, accelerator performance, model architecture, quantization, and runtime efficiency.
How much RAM does a 70B LLM need?
For model weights alone, a rough theoretical estimate is about 140 GB at FP16, 70 GB at 8-bit, or 35 GB at an idealized 4-bit representation. Real memory requirements are higher because model formats add overhead and inference also needs KV cache, compute buffers, runtime memory, and operating-system resources.
What is KV cache in an LLM?
The KV cache stores key and value representations for previously processed tokens so they do not need to be recomputed during every generation step. Its size grows with context length and depends on the model architecture, cache precision, layer count, attention configuration, and number of concurrent sequences.
Does a longer context window require more memory?
Yes. Longer contexts increase KV-cache requirements and attention work. The impact can become substantial with large RAG prompts, long conversations, coding sessions, agents, or multiple concurrent requests.
Does quantization make local LLM inference faster?
It can. Quantization reduces model size, which lowers memory requirements and can reduce the amount of weight data transferred during inference. The speed and quality impact depends on the model, quantization format, runtime, hardware, and workload.
Is 50 tokens per second good for a local LLM?
For my interactive workloads, around 50 output tokens/sec feels comfortably responsive. This is a practical personal threshold rather than a universal standard. Batch processing, document analysis, and privacy-sensitive workloads can still be useful at much lower generation speeds.
What matters more for local LLMs: VRAM or memory bandwidth?
Both matter for different reasons. Memory or VRAM capacity determines whether the model and its working state fit. Memory bandwidth strongly influences how quickly that data can reach the compute hardware during decoding. A useful system needs enough of both.
Conclusion
Running a local LLM successfully is not just a question of whether the model fits in memory.
A practical local inference system must balance model quality, memory capacity, memory bandwidth, accelerator performance, quantization, KV-cache size, context length, concurrency, and runtime efficiency.
For my workloads, local AI becomes especially compelling when a capable model can sustain roughly 50 output tokens/sec. Beyond that point, model quality and task suitability often matter more than simply maximizing raw generation speed.