The GPU Utilization Trap: 95% Isn't Real
Your dashboard reads 95% GPU utilization. Yet training is slow, and the quarterly bill is bigger than expected. That reassuring number may guarantee nothing at all. Here's why the GPU utilization metric misleads — and what you should actually watch.
What does GPU utilization measure?
GPU utilization (GPU-Util) measures whether at least one kernel was executing on the GPU during a sampling window. That's the number nvidia-smi and other NVML-based tools show you.
Here's where the trap begins. This metric tells you whether the GPU was working, not how much work it did. Tools slice time into milliseconds and, for each slice, count the GPU as "in use" if even one kernel is resident on any SM (Streaming Multiprocessor). Whether that kernel uses all the GPU's cores or just one, it counts the same.
100% utilization with zero compute is possible
Take an extreme example. Run a kernel that only reads and writes memory while doing no math, and GPU utilization can still read 100% — even though actual floating-point operations (FLOPS) are zero.
There's a real example from NVIDIA's developer forum: run an infinite-loop kernel on a single SM, and utilization reads near 100% regardless of whether the GPU has 100 SMs. Intuitively it should be around 1% (one SM in use), but the metric doesn't account for SM count.
In other words, GPU-Util is a smoke alarm telling you the machine isn't idle — not a fuel gauge measuring how much fuel you're burning.
So what should you watch: SM Efficiency and MFU
To see how much real work a GPU does, you need different metrics.
SM Efficiency — the share of the GPU's SMs that actually had work on them. If GPU-Util asks "did a kernel run," SM Efficiency asks "how many cores participated." NVIDIA DCGM ships it by default.
MFU (Model FLOPs Utilization) — the most honest metric. It's the ratio of the compute a model actually achieved to the hardware's theoretical peak. It measures the gap between the spec sheet and the math actually advancing your model.
MFU tells you what percentage of a GPU's maximum you're really using — the true test of whether an expensive GPU is earning its keep.
Metric comparison
Metric | What it shows | Limitation |
|---|---|---|
GPU-Util | Whether a kernel is running | Ignores compute volume & core participation |
SM Efficiency | How many SMs participated | Doesn't capture compute intensity |
MFU | Real compute vs theoretical peak | Hard to monitor continuously |
What counts as a "good" MFU?
This surprises many people. In the industry, a "good" MFU is around 35–45%, and above 50% is considered excellent. In practice, Google's PaLM reported ~46%, Meta's Llama 3 405B reported 38–43%, and ByteDance's MegaScale hit 55% on a 175B model.
What does that mean? Even world-class teams use only around half of a GPU's theoretical peak. Yet many organizations see nvidia-smi's 90-something percent and report to the CFO that their GPUs are fully utilized — while enormous compute capacity leaks away underneath.
Why this gap matters: it's about money
This isn't a terminology quibble; it's a cost problem. By one analysis, a 128-GPU H100 cluster running at 30% utilization wastes roughly $1.6M/year in idle capacity, because GPUs bill at full rate even while idle.
And if that "30%" is a GPU-Util figure, effective compute by MFU could be even lower. Wrong metric, wrong decisions: "utilization is high, let's buy more GPUs" may really have been "we aren't using the GPUs we already have."
How to measure GPU utilization properly
Approach depends on scale.
On a single host, use nvidia-smi for a baseline — but read the number as an alarm, not a fuel gauge. At cluster scale, the standard is to scrape the DCGM Exporter into Prometheus and visualize in Grafana. Watch not just GPU-Util but SM Efficiency, Tensor Core activity (DCGM's Tensor Pipe Activity), and memory bandwidth to reveal real efficiency.
The key is not trusting a single number. GPU-Util only tells you the GPU isn't idle — not that it's being used well.
FAQ
Q. If nvidia-smi shows 100%, am I using the GPU well?
No. It only means a kernel is running, not that all cores are working at peak. You can hit 100% while mostly moving memory and doing little compute.
Q. Why is MFU hard to monitor continuously?
MFU requires knowing your model structure and actual throughput (tokens/sec), which is cumbersome to track layer-by-layer in real time. Many teams monitor SM Efficiency continuously and check MFU periodically.
Q. Is a 40% MFU a problem?
No. 35–45% is a healthy industry range. Trusting nvidia-smi's "90%, we're fine" is the riskier habit.
Q. How do I see fleet-wide GPU efficiency?
Single-node tools don't scale; the standard is DCGM Exporter + Prometheus + Grafana to collect and visualize across the cluster.
Conclusion: watch effective compute, not the utilization number
Before you relax at "90% utilization," ask how much effective compute that GPU is actually doing. GPU-Util is only a starting point; SM Efficiency and MFU show the real picture.
And tracking these across a cluster by hand isn't realistic. TEN's AIPub monitors your entire AI infrastructure in real time with 50+ proprietary metrics across five layers — from data center to cluster, node, workload, and GPU device — including the very metrics this article covers: SM Active, SM Occupancy, and Tensor Core activity. It also tracks PCIe traffic, NVLink bandwidth, and GPU power and temperature, so you can see where compute actually leaks behind the surface "in use" number — with automatic daily, weekly, and monthly reports.
To diagnose whether your GPUs are truly earning their keep, explore AIPub's monitoring.
References
NVIDIA, DCGM (Data Center GPU Manager) Documentation
NVIDIA Developer, System Management Interface (nvidia-smi)
Chowdhery et al., "PaLM: Scaling Language Modeling with Pathways" (MFU definition)
Trainy, "GPU Utilization is a Misleading Metric" (2025)
PyTorch Profiler Documentation