What Is VRAM? GPU Memory for Running LLMs
The first wall you hit when trying to run an LLM is often "out of VRAM." Here's what VRAM actually is — and why your GPU can't load the model.
What is VRAM?
VRAM (Video RAM) is dedicated memory on the GPU. It temporarily holds the data the GPU computes on, physically separate from the CPU's system RAM. It matters for AI because a model's weights and intermediate values must fit in VRAM for the GPU to compute at all.
VRAM vs system RAM
Aspect | System RAM | VRAM |
|---|---|---|
Location | Motherboard (by CPU) | Inside the GPU |
Purpose | CPU compute / OS | GPU compute data |
Speed | Relatively slower | Very fast (high bandwidth) |
AI role | Preprocessing, etc. | Core of model execution |
The key difference is bandwidth. VRAM must feed thousands of GPU cores at once, so it's built far faster — and can't be replaced by system RAM even at equal capacity.
How much VRAM does an LLM need?
Model weights scale with parameters × precision. At FP16, roughly 2GB per billion parameters. So a 7B model needs ~14GB and a 70B model ~140GB.
In practice, KV cache and activations add more, so leave 20–30% headroom. Running a 7B model at FP16 realistically needs 16–18GB of VRAM.
* These are common industry approximations; actual needs vary by implementation and settings.
When VRAM runs short
The model won't load at all (Out of Memory)
Offloading to system RAM causes a sharp slowdown
You must shrink batch size / context, reducing throughput
How to cope
Options include quantization (INT8/INT4 to cut VRAM by half or more), multi-GPU sharding (splitting a large model across GPUs), and GPU partitioning/sharing (running several small models on one GPU). In shared environments, how you divide GPU memory largely determines overall utilization.
To manage limited GPU memory so multiple workloads share it efficiently, explore AIPub's GPU partitioning.