Kubernetes GPU Scheduling Explained
There are clearly GPUs free, yet your training job stays "Pending." Anyone who has run GPUs on Kubernetes has hit this. Here's how Kubernetes GPU scheduling works, why this happens, and what to fix.
What is Kubernetes GPU scheduling?
Kubernetes scheduling is the process of deciding which job (Pod) goes on which server (Node). GPU scheduling adds one condition to that: the GPU.
When a training or inference job arrives, Kubernetes finds a node with the GPU that job needs and places it there. In an environment where many users and teams share one GPU cluster, how well this placement works determines overall utilization.
The catch: the default Kubernetes scheduler was designed for general container workloads. Handling an expensive, specialized resource like a GPU exposes a few limits.
Three limits of the default scheduler
1. It only assigns whole GPUs
Ever had a whole GPU locked up by a single small inference job?
Kubernetes assigns GPUs in units of one whole card by default. Even a job that uses 10% of a GPU takes the entire card, leaving the other 90% idle. Multiple small jobs can't share one GPU.
2. It ignores workload characteristics
The default scheduler just finds "an empty node with a GPU" — it doesn't distinguish a large training run from a short inference call. As a result, short jobs get stuck behind long ones, and jobs with different resource needs interfere on the same node.
3. Priority and fairness control is weak
Is one team monopolizing GPUs while others keep waiting in line?
In shared environments, you need a policy for "whose job runs first." The default scheduler alone can't finely control per-team quotas or priorities.
Why jobs wait while resources sit free: fragmentation
When these limits stack up, you get resource fragmentation.
Fragmentation is when GPUs are scattered in leftover pieces but can't be secured contiguously in the amount a job needs. For example, if several nodes each have 0.5 of a GPU free, a job needing 2 GPUs finds nowhere to land and keeps waiting.
Cluster-wide, there are plenty of GPUs — yet it feels like a shortage. The problem isn't GPU count; it's placement.
What efficient GPU placement requires
Solving this takes capabilities beyond the default scheduler.
Problem | Capability needed |
|---|---|
Whole-GPU assignment | GPU partitioning (many jobs share one GPU) |
Workload-blind placement | Workload-aware placement |
No priority | Per-team/project quotas & priority |
Fragmentation | Intelligent scheduling that finds and fills free capacity |
Low visibility | Real-time resource monitoring |
The key is moving beyond "find an empty node with a GPU" to intelligent scheduling that places jobs in the optimal spot by weighing both workload demands and resource state.
Conclusion: the edge is in placement, not count
No matter how many GPUs you secure, inefficient placement keeps utilization low. First distinguish whether jobs wait because of a real GPU shortage — or a scheduling problem.
Can GPUs be split and assigned in the amounts actually needed?
Are jobs placed automatically by workload characteristics?
Do multiple teams share without priority conflicts?
Is idle capacity minimized?
These are solved by operations software, not more hardware.
TEN's AIPub partitions GPUs into blocks so multiple jobs share them, places jobs by weighing resource state and queues, and separates resources per team and project to reduce conflicts — so expensive GPUs don't sit idle due to fragmentation.
To diagnose why your cluster's GPUs stay pending, explore AIPub.