Why split LLM inference across multiple GPUs when a single GPU used to handle the entire request? When LLM inference performance falls short of expectations, the problem may not simply be a lack of GPUs. The bottleneck may come from running two fundamentally different types of work on the same GPU at the same time. **Disaggregated Inference** is the architecture that emerged to solve this. Disaggregated Inference runs the Prefill and Decode stages of LLM inference on separate GPUs or GPU [pools.It](http://pools.It) is spreading quickly across major LLM serving technologies and cloud environments, including NVIDIA Dynamo, llm-d, SGLang, LMCache, vLLM-integrated environments, and AWS SageMaker HyperPod. --- ## **LLM Inference Consists of Prefill and Decode** To understand Disaggregated Inference, you first need to understand that LLM inference consists of two stages with very different characteristics.

Category	Prefill	Decode
Primary role	Processes the entire input prompt	Generates output tokens sequentially
Processing method	Processes multiple input tokens in parallel	Generates tokens one by one autoregressively
Primary output	First token and KV cache	Subsequent tokens generated using the KV cache
Main bottleneck	Compute throughput	Memory bandwidth and capacity
Key metric	TTFT	TPOT and inter-token latency

### **Prefill: Processing the Input** Prefill processes the entire prompt entered by the user and creates the first token and **KV cache (Key-Value Cache)**. - ***KV cache:*** *Temporary GPU memory that stores information the LLM computed while processing earlier tokens, so it doesn't repeat the same computation for each new token and improves inference speed.* Because Prefill must process long documents or large contexts all at once, it is generally considered a compute-bound workload that requires high computational throughput. Prefill performance directly affects TTFT (Time to First Token) — the time between submitting a request and receiving the first part of the response. ### **Decode: Continuing the Response** Decode uses the KV cache created during Prefill to generate output tokens one at a time. Every time a token is generated, the GPU repeatedly reads the model weights and KV cache. GPU memory bandwidth and memory access speed are therefore critical. Decode is generally classified as a memory-bandwidth-bound workload. Decode performance affects TPOT (Time per Output Token) — how quickly and smoothly the answer continues. In short, Prefill determines "how fast the answer starts," and Decode determines "how fast the answer continues." --- ## **Why Split Prefill and Decode?** ![Diagram comparing aggregated and disaggregated LLM inference. In the aggregated architecture, one GPU worker handles both Prefill and Decode, which can cause resource interference and require the entire GPU pool to be scaled. In the disaggregated architecture, separate GPU pools handle each stage and transfer the KV cache between them, enabling independent optimization and scaling.](https://dfyvbtcqkbshnnpfclnp.supabase.co/storage/v1/object/public/blog-images/blog-images/1784178165949-p4brjz.png) In a traditional aggregated setup, a single GPU worker handles both Prefill and Decode. - ***GPU worker:*** *A GPU-backed serving unit that runs the model and serves actual inference requests. A group of GPU workers performing the same role is called a GPU pool.* The aggregated design is simple. However, it forces two fundamentally different workloads to compete for the same GPU resources. This resource contention is the main reason Prefill and Decode are separated. Splitting the two stages provides three major benefits. ### **1. Reduce Interference Between Workloads** Suppose a long Prefill request arrives while a Decode worker is already generating tokens. The Prefill computation may consume a large share of the GPU's resources, delaying the Decode process already in progress. To the user, this shows up as token generation suddenly slowing down or stuttering after the answer has started. Separating Prefill and Decode into different GPU pools reduces this interference.The Prefill pool can be optimized around TTFT, while the Decode pool can be optimized around TPOT. This makes it easier to maintain more stable and predictable response latency. ### **2. Scale Only the Stage Experiencing a Bottleneck** Prefill requires high compute throughput. Decode depends more heavily on memory bandwidth and capacity. In an aggregated architecture, however, the same GPU workers must be added regardless of which stage is actually causing the bottleneck. This can result in unnecessary GPU spending. In a disaggregated setup, you scale Prefill workers when input tokens grow, and Decode workers when concurrent users and output tokens grow. Instead of scaling the entire GPU configuration uniformly, you add resources only to the stage where the load actually occurs. ### **3. Apply a Hardware Strategy for Each Stage** Splitting Prefill and Decode also makes it possible to apply different GPU and parallelization strategies to each stage. GPUs with high computational throughput can be assigned to Prefill. GPUs with high memory bandwidth and sufficient memory capacity can be assigned to Decode. Organizations can also consider heterogeneous GPU configurations that combine different GPU models. Instead of forcing two different workloads into a single GPU configuration, each workload can be matched with the resources best suited to its characteristics.This can improve both GPU utilization and cost efficiency. --- ## **How Does Disaggregated Inference Work?** ![Diagram showing the KV cache transfer flow in disaggregated inference. A Prefill worker processes the user prompt in parallel, generates the first token and KV cache, and transfers the cache directly to a Decode worker through RDMA-based high-speed interconnects. The Decode worker repeatedly accesses the model weights and KV cache to generate and stream output tokens sequentially.](https://dfyvbtcqkbshnnpfclnp.supabase.co/storage/v1/object/public/blog-images/blog-images/1784178181783-pfvtxe.png) The basic flow of disaggregated inference is as follows. 1. A Prefill worker Processes the entire input prompt in parallel. 2. It generates the first token and KV cache. This KV cache is containing the context. 3. The KV cache is transferred to a Decode worker. 4. The Decode worker generates the remaining output tokens. In this structure, Prefill and Decode resources can be placed and scaled independently. However, the two stages are not completely independent. Decode cannot begin without the KV cache created during Prefill. So the performance of disaggregated inference depends not only on splitting the GPUs, but on **how fast the KV cache is transferred**. In multi-node environments, high-speed networking technologies such as InfiniBand, RoCE, and AWS EFA may be used alongside RDMA and GPUDirect RDMA. - ***RDMA***(*Remote Direct Memory Access) : Transfer data directly between the memory of different servers with minimal CPU involvement. GPUDirect RDMA transfers data directly between GPU memory to reduce latency.* --- ## **Which Workloads Benefit from Disaggregated Inference?** Disaggregated Inference can be particularly effective in environments with long input contexts, high request concurrency, or strict latency consistency requirements. - Large-scale document analysis and long-context RAG - Coding agents that process large codebases - AI chatbots that require real-time token streaming - LLM inference APIs with high concurrency - Multi-turn and agentic AI systems that invoke LLMs repeatedly Document analysis and long-context RAG have many input tokens, so the Prefill load can grow. Reasoning models and AI agents, on the other hand, generate long outputs and call the LLM repeatedly, so the Decode load can grow. Because the balance of input and output differs by workload, a structure that scales Prefill and Decode resources independently can be an advantage --- ## **Two things to check before adopting Disaggregated Inference** Disaggregated Inference is not automatically more efficient than aggregated inference in every environment. For services with low request volumes or short inputs and outputs, the overhead of separating GPU pools and transferring the KV cache may outweigh the benefits. Also, because Prefill and Decode workers each hold their own copy of the model weights, memory efficiency can drop in small-scale GPU environments. So the decision should be based on your model and traffic characteristics, your TTFT/TPOT targets, and your GPU scale. ### **1. Can you transfer the KV cache fast enough?** The longer the context and the larger the model, the bigger the KV cache you have to transfer. If the network is slow or congested, KV cache transfer time inflates TTFT and the benefit of splitting can disappear. You have to review not just GPU performance but network bandwidth and transfer method too. ### **2. Can you adjust the Prefill-to-Decode resource ratio?** There's no fixed ideal ratio between Prefill and Decode GPUs. The resources you need shift with input/output token counts, concurrent requests, model type, and time of day. If one side is short on workers, requests queue up; if you over-provision, GPUs sit idle. So you need to monitor TTFT, TPOT, request queues, and GPU utilization together, and adjust the ratio accordingly --- ## **Conclusion: the key to disaggregation isn't the GPU — it's operations** Disaggregated Inference processes Prefill and Decode on GPU resources suited to the characteristics of each stage. This reduces interference between workloads and allows resources to be scaled independently according to the actual bottleneck. But splitting the GPUs isn't enough on its own. How many Prefill and Decode pools to run, which path to transfer the KV cache over, which GPU to route each request to, and how to adjust resources as traffic shifts — all of this has to be managed together to translate into real performance and cost efficiency. Ultimately, what matters is the ability to operate the split GPUs and workloads as one stable service. Allocating GPU resources to fit the workload, tracking usage and performance in real time, and running many AI workloads reliably — that operational layer is exactly where AIPub has focused. Don't go it alone. As long contexts and AI agents spread, LLM inference infrastructure keeps getting more complex. Work with TEN's experts to design the GPU configuration and operations approach that fit your organization's models and workloads. 👉[ Learn more about AIPub](https://ten1010.io/en/products/aipub-dev) 👉[ Explore RA:X infrastructure consulting](https://ten1010.io/en/products/rax) 📩[ Talk to an expert](https://ten1010.io/en/inquiry) --- ## **References** 1. [NVIDIA, “What Is Disaggregated Serving?”](https://www.nvidia.com/en-us/glossary/disaggregated-serving/) 2. [AWS, “Disaggregated Prefill and Decode for LLM Inference on SageMaker HyperPod”](https://aws.amazon.com/blogs/machine-learning/disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod/) --- ## **Recommended Reading** - [What Is a Neocloud? GPUaaS Market Growth and Monetization Strategy](https://ten1010.io/en/blog/neocloud-gpuaas-operations-aipub) - [Why GPU Infrastructure Operations Must Evolve in the Agentic AI Era](https://ten1010.io/en/blog/agentic-ai-gpu-infrastructure) - [The Structural Limits of Time-Slicing: How Spatial Partitioning Solves the 30% GPU Utilization Problem](https://ten1010.io/en/blog/the-structural-limits-of-time-slicing-how-spatial-partitioning-solves-the-30-gpu-utilization-problem)

#Decode

What Is Disaggregated Inference? Why LLM Serving Splits Prefill and Decode