Cloud-Native Inference & GPU Splitting
GRIP: In-place GPU Partitioning for Cloud-Native Inference Serving
A paper by Sangho Yeo of TEN’s RA:X team, “GRIP: In-place Spatial Partitioning of CUDA Graphs for Cloud-Native Inference Serving,” has been accepted to SoCC ‘26 (the ACM Symposium on Cloud Computing, November 2026, Singapore), one of the top venues in cloud computing. This post explains the problem and the solution the paper addresses.
A technique for efficient GPU utilization: spatial partitioning
The real utilization of GPU inference clusters is far lower than most assume. One measurement study¹ reports a median utilization of just 6% for inference GPU clusters, with individual serving containers sitting completely idle for roughly 66% of the measured time². Nine out of ten expensive cards, effectively standing still.
The established answer is spatial partitioning: co-locate multiple tenants on one GPU, and when one tenant is idle, hand its resources to another. That’s the theory.
The problem is that LLM inference relies on CUDA Graph-based optimization. And that is exactly where spatial partitioning and CUDA Graphs collide.
The CUDA Graph dilemma: allocation frozen at capture time
A CUDA Graph bundles many kernel launches into a single graph submitted with one host call. Because it sharply reduces kernel-launch overhead for inference workloads with short per-kernel execution times, major inference frameworks, including vLLM, SGLang, and TensorRT-LLM, depend on it.
But CUDA Graphs carry a stubborn constraint. Once a graph is captured, it freezes the resource allocation (at the level of TPCs, Texture Processing Clusters) as it stood at capture time. Change the allocation afterward, say, “give this tenant more idle TPCs,” and the graph still replays with its original allocation. The idle GPU capacity you worked to reclaim never actually reaches a CUDA Graph-based workload.
Until now, operators faced a lose-lose choice: abandon CUDA Graphs and fall back to the slower kernel path, or keep CUDA Graphs but give up reclaiming idle resources.
GRIP: repartition in place, without recapturing the graph
GRIP (Graph Repartitioning In-Place) solves this head-on. As the name says, it re-partitions the graph in place, with no graph recapture and no application changes.
The key insight is that a narrow window exists in the CUDA Graph lifecycle where allocation is still mutable: after a graph is instantiated but just before it is uploaded to the GPU, the host-side metadata remains editable. GRIP locates this window and, through a transparent substitution layer between the application and the CUDA driver, replaces the running graph instance with one that reflects the new allocation. The application perceives its own graph running as submitted, while in reality the repartitioned graph executes.
GRIP also solves a second problem: non-clusterable TPCs. Some TPCs run ordinary kernels normally yet cannot form a Thread Block Cluster (TBC). When such a TPC lands in an allocation, the TBC kernel fails, and under MPS that failure propagates to co-located tenants and stalls everyone. GRIP uses a deterministic topology-discovery method to identify these TPCs in advance and compose only safe allocations.
Results: up to 2.1x under contention, 2.2x when resources free up
The paper validates GRIP across the Hopper generation (H100, H200), the Blackwell generation (B200, B300), and the RTX PRO 6000, using vLLM-based Qwen3-8B inference and ResNet152 inference.
Baseline | Condition | GRIP’s result |
Temporal partitioning | Under contention | Up to 2.1x higher throughput |
Static spatial partitioning | When idle resources free up | Up to 2.2x higher throughput |
Dedicated GPU (vs. MPS) | Control overhead | At most 5.4% |
Memory overhead | Relative to model execution | Under 1% (at most 236 MiB) |
The point is that GRIP captures both sides. Under contention it keeps spatial partitioning’s strength (concurrent execution); when resources free up it reclaims idle capacity in a way temporal partitioning cannot, all while keeping control cost in the single digits relative to a dedicated GPU.
Why research like this sits underneath the products we ship
What TEN builds are products that operate GPUs efficiently. AIPub partitions and schedules GPUs; RA:X diagnoses and benchmarks infrastructure. For such products to credibly claim they “raise utilization,” there has to be a mechanism that actually works underneath that claim.
Research like GRIP is that foundation. When a way to revive idle GPUs as measurable throughput, inside the real CUDA Graph environments inference servers actually use, is backed by peer-reviewed research, the product’s promise becomes engineering.
Congratulations to Sangho Yeo and the RA:X team. We hope this work, to be presented in Singapore this November, offers one glimpse of the depth at which TEN examines GPU infrastructure problems.
Curious about TEN’s GPU infrastructure technology:
¹ Suyi Li, Lingyun Yang, Haoxuan Yu, Sheng Yao, Tianyuan Wu, Xiaoxiao Jiang, Hanfeng Lu, Kangjin Wang, Chenhao Wang, Shenglin Xu, Lun Wang, Qingyang, Duan, Shenghao Liang, Xiu Lin,WenchaoWu, Yinghao Yu, Guodong Yang, Liping, Zhang, and Wei Wang. 2026. Heterogeneity at Hyperscale: Characterization and Scheduling of Large Production AI Clusters at Alibaba. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26).
² Alibaba Group. 2026. GenAI Serving Top-Down Dataset 2026 (GenTD26). https: //github.com/alibaba/clusterdata. Alibaba Cluster Trace Program.; Xuanzhe Liu, Yihao Zhao, Shufan Liu, Xiang Li, Yibo Zhu, Xin Liu, and Xin Jin. 2024. MuxFlow: Efficient GPU Sharing in Production-Level Clusters with More than 10000 GPUs. Science China Information Sciences 67, 12 (2024), 222101. doi:10.1007/s11432-024-4227-2; Zeyu Zhang and Haiying Shen. 2024. PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference. arXiv:2409.15104 [cs.DC]