Developer & Tech

GPU Memory Calculator

What this does

Estimate GPU memory for running a model from parameter count, precision, batch size and KV cache allowance, with fit checks.

Enter your details

Runs in your browser

Calculator inputs

How to use it

Using the gpu memory calculator

  1. 01

    Enter parameter count

    From the model card, e.g. 7, 13, 70 billion.

  2. 02

    Pick serving precision

    Quantization choice dominates: int4 halves fp16, enabling 70B-class models on single consumer cards.

  3. 03

    Check the fit rows

    Green fits mean the plan clears that GPU’s VRAM with the allowances you entered.

Good to know

Precision is the biggest lever

  • fp32; reference precision, rarely needed for inference
  • fp16/bf16; standard serving precision
  • int8; half of fp16, minor quality cost
  • int4; quarter of fp16, noticeable but often acceptable

The parts weights forget

KV cache scales with batch × sequence length and can rival weights at long contexts; activation buffers spike with batch size; CUDA/runtime overhead claims 1–2 GB regardless. Production capacity plans leave 20–30% headroom above any estimate like this one.

How it's calculated

The math behind this calculator

weights GB ≈ params(B) × bytes/param total ≈ weights + KV-cache + activations

Weight memory equals parameter count times bytes per parameter; 7 billion params at fp16’s two bytes is 14 GB directly. Your entered KV cache and activation allowance tops that up, and fit rows check the total against popular GPUs. All figures are rough planning aids: CUDA contexts, framework buffers and fragmentation add gigabytes no formula captures.

Assumptions & limitations

  • Rough planning figures, not procurement specs.
  • Training needs roughly 4–6× weight memory (optimizer states).
  • KV cache grows with batch size and sequence length beyond the entered flat allowance.

Worked example

A 7B model at fp16 needs about 14 GB for weights; with 1 GB of KV cache headroom it fits comfortably on a 24 GB RTX 4090.

FAQ

Frequently asked questions

Can I fine-tune within these numbers?
No; training adds optimizer states and gradients, typically 4–6× weight memory. LoRA/QLoRA reduce that dramatically but not to inference levels.
Why does batch size barely appear?
Batch multiplies activations and KV cache, not weights. Enter a larger KV cache allowance to reflect high-batch serving.
Is int4 quantization safe for production?
Often yes for chat workloads; quality-sensitive domains (code, math) deserve A/B evaluation first.

Keep exploring

Related calculators