What to Watch in GPU Monitoring
What to Watch in GPU Monitoring
A single GPU can cost tens of thousands of dollars. A cluster of dozens becomes a multi-million-dollar asset. Yet many teams judge the health of that asset by one line of nvidia-smi, and within it, a single number: GPU-Util. When the dashboard reads “90% utilization,” everyone relaxes. When it reads 30%, they panic.
The problem is that the 90% tells you almost nothing.
What GPU-Util doesn’t tell you
GPU-Util from nvidia-smi means “the fraction of the sampling window during which at least one kernel was running on the GPU.” It only sees whether a kernel ran, not how much of the GPU’s compute capacity that kernel actually filled.
So even if just one of the GPU’s thousands of cores is active, and even if that core spends most of its time stalled waiting on a memory load, GPU-Util can read 100%. During a data-loader bottleneck, while the GPU waits for the CPU to feed the next batch, leftover kernels can keep utilization looking high. The operator believes the GPU is maxed out, while real compute efficiency sits at 20-30%, which is far from rare.
This is the utilization trap. So what should you actually watch?
The 5 metrics that matter in operations
Metric | What it measures | Why it matters |
|---|---|---|
SM Efficiency | How much the streaming multiprocessors were actually engaged | Measures whether the compute units did real work, not just whether a kernel ran. The true utilization signal |
Memory bandwidth usage | How saturated HBM bandwidth is | Inference is usually memory-bound, not compute-bound. Low here means compute is waiting on memory |
GPU memory (VRAM) footprint | Allocated vs. actually used VRAM, and headroom | The basis for preventing OOM incidents and deciding whether another workload can be placed |
Power / temperature / clocks | Draw, core temperature, clock throttling | Thermal throttling silently cuts performance; power is cost |
PCIe / interconnect traffic | Card-to-card and node-to-node data movement | The key to catching communication bottlenecks in distributed training |
The point is that these must be read together. Low SM Efficiency with saturated memory bandwidth means a memory bottleneck; both low with high PCIe traffic means a communication bottleneck. No single number can tell you this.
From one metric to layered observability
Mature GPU operations treat metrics as layers, not a one-line number. You watch power, temperature, and clocks at the physical card layer; compute and memory efficiency above that; and workload-, project-, and account-level usage patterns above that again. Only by isolating which layer the bottleneck sits in can you judge whether “buy more GPUs” is really the right answer.
Far more often, the issue isn’t a GPU shortage. It’s that you’re using the GPUs you have at 20%.
AIPub’s approach: 5-layer observability and 50+ metrics
TEN’s GPU operations platform, AIPub, solves this with layered monitoring. AIPub collects and visualizes 50+ proprietary metrics across 5 layers, from physical card to node, workload, project, and account. Instead of the “90% utilization” illusion, operators see whether compute units are actually working at each layer and where the bottleneck forms, all on one screen.
Add per-card power-limit control, and thermal and energy cost become operational levers too. That’s the point where monitoring crosses from after-the-fact reporting into operational control.
Before you buy more GPUs, look at how hard the ones you have are actually working. That answer changes the investment decision.
→ See AIPub’s GPU monitoring: https://ten1010.io/en