7 Metrics to Watch on Your GPUs
What to Watch in GPU Monitoring: 7 Metrics Beyond Utilization
A single GPU costs tens of thousands of dollars; a training cluster of dozens becomes a multi-million-dollar asset. Yet many teams judge the health of that asset by one line of nvidia-smi, and within it, a single number: GPU-Util. When the dashboard reads “90% utilization,” everyone moves on. When it reads 30%, they start drafting a procurement request.
The problem is that the 90% guarantees almost nothing. GPU-Util at 90% while throughput sits at half of expectation; adding GPUs and seeing total training time barely drop, these all happen because someone trusted a single number called “utilization.”
This article covers why GPU-Util deceives you, the 7 metrics you must watch alongside it in real operations, and how to read them as layers rather than one flat number.
What GPU-Util is, and what it doesn’t tell you
GPU-Util (GPU Utilization) means “the fraction of the last sampling window during which at least one kernel was running on the GPU.” The key phrase is “at least one.” It measures only whether a kernel ran, not how much of the GPU’s compute capacity that kernel filled.
So this happens: a GPU has thousands of CUDA cores and dozens to hundreds of SMs (Streaming Multiprocessors), and even if just one SM is working while the rest sit idle, GPU-Util can read 100%. Even if the kernel spends most of its time stalled, waiting for data to arrive from memory, utilization still looks high as long as a kernel is “running.”
The most common illusion comes from data-loader bottlenecks. While the GPU waits for the CPU to feed the next batch, leftover kernels from the previous step keep GPU-Util looking high. The operator believes the GPU is maxed out, while real compute efficiency (MFU, Model FLOPs Utilization) sits in the 20-30% range, which is far from rare. For reference, even well-tuned large-scale training reaches an MFU of only 35-45%, and exceeding that takes serious engineering.
The 7 metrics that matter in real operations
Instead of discarding utilization entirely, watch these 7 together to finally see what the GPU is actually doing.
Metric | What it measures | Why it matters |
|---|---|---|
SM Efficiency | How much the streaming multiprocessors were actually engaged | Whether compute units did real work, not just whether a kernel ran. The primary signal of true utilization |
Memory bandwidth usage | How saturated HBM bandwidth is | Inference and many training phases are memory-bound, not compute-bound. High here with low SM means memory-bound |
GPU memory (VRAM) | Allocated VRAM, actually-used VRAM, headroom | The basis for preventing OOM and deciding whether another workload fits on this GPU |
Power draw | Real-time watts per card | Power is operating cost. Throughput-per-watt is the real efficiency measure |
Temperature / clock throttling | Core temperature and whether clocks drop | Thermal throttling silently cuts performance. A frequent cause of unexplained slowdowns |
PCIe / interconnect traffic | Card-to-card and node-to-node data movement | The key to catching communication bottlenecks in distributed training |
Throughput | Tokens / images / samples per second | Ultimately “how much work got done.” The other six are diagnostics that explain this number |
The real value of these metrics emerges when you read them against each other. Some representative combinations:
Low SM + saturated memory bandwidth → memory-bound. Compute units are waiting on data. Adjust batch size, quantization, or KV-cache strategy.
Low SM + low memory bandwidth + high PCIe → communication-bound. The network is the bottleneck in distributed training. Look at the network fabric or communication strategy (e.g., gradient compression).
Low SM + everything low → input bottleneck (data loader / preprocessing) or a scheduling problem. The GPU isn’t being fed work in the first place.
High temperature + dropping clocks → cooling / power issue. A hardware- or data-center-level problem.
No single number can distinguish any of these. “GPU-Util 90%” can look identical across all four situations.
From a single metric to layered observability
Mature GPU operations treat metrics as layers, not a one-line number, because bottlenecks occur at specific layers and a single number can’t tell you which layer is the problem.
From bottom to top, the layers you should observe:
Layer | What you watch | The question it answers |
|---|---|---|
Physical card | Power, temperature, clocks, fans | Is the hardware healthy? Any throttling? |
Compute / memory | SM Efficiency, memory bandwidth, VRAM | Is the GPU actually working? What is it waiting on? |
Workload | Throughput, per-Pod/Job resource use | Is each job efficient? Where is it slow? |
Project / team | GPU-hours per project, allocated vs. used | Who uses resources efficiently? |
Account / org | Org-wide usage patterns, cost allocation | Is investment turning into results? Is buying more the answer? |
This layering matters because many organizations jump straight to the most expensive conclusion: “buy more GPUs.” But when you separate the layers, it often turns out the problem isn’t a GPU shortage, it’s operational inefficiency, using the GPUs you have at only 20-30%. Buying more just replicates that inefficiency at a higher price.
AIPub’s approach: 5-layer observability and 50+ proprietary metrics
TEN’s GPU operations platform, AIPub, is designed around layered monitoring. AIPub collects and visualizes 50+ proprietary metrics across 5 layers, from physical card to node, workload, project, and account. Instead of the single “90% utilization” illusion, operators can track whether compute units are actually working at each layer and which layer the bottleneck forms in, all on one screen.
Add per-card power-limit control, and thermal and energy cost shift from “things you observe” to “levers you operate.” You can cap a card’s power to optimize throughput-per-watt, or shape the whole cluster’s power profile to match cooling headroom. That’s the point where monitoring crosses from after-the-fact reporting into active control.
Before you buy more GPUs, look at how hard the ones you have are actually working at each layer. In most cases, that answer makes you rewrite the procurement request.
Frequently asked questions
Q. If GPU-Util is 100%, aren’t my GPUs being used well?
No. GPU-Util measures only the fraction of time a kernel was running. It can read 100% even if just one SM is active, or if the kernel is stalled waiting on memory. For real compute utilization, look at SM Efficiency and throughput.
Q. How do I measure SM Efficiency?
Through profiling metrics from NVIDIA’s DCGM (Data Center GPU Manager), such as SM Active and SM Occupancy. It isn’t in the default nvidia-smi output; you need DCGM or a monitoring platform that integrates it.
Q. What’s the most important metric for inference workloads?
Memory bandwidth usage and VRAM footprint. LLM inference is mostly memory-bound, so HBM bandwidth saturates before compute units do. Batching and KV-cache management drive performance.
Q. How many metrics should I track?
You don’t need dozens from day one. Start by reading the 7 in this article (SM Efficiency, memory bandwidth, VRAM, power, temperature/clocks, interconnect, throughput) across layers, then add detail around the points where bottlenecks recur.
→ See AIPub’s 5-layer GPU monitoring: