What Is LLM Serving? Inference Deployment Explained
What is LLM serving? Learn how requests flow through an inference stack, how engines differ from serving platforms, key metrics like TTFT, and much more.
Nov 18, 2026
Training a model or downloading an open-source LLM can happen fairly quickly. The real work starts after that: turning it into a service people actually use, such as an internal chatbot, document search, or a coding assistant. That stage is serving.
Generating one answer on a laptop and running an API around the clock that answers hundreds of simultaneous users within seconds are entirely different problems. This post covers what LLM serving is, how it is structured, the performance metrics that matter, and what serving demands from GPU infrastructure.
What Is LLM Serving?
LLM serving is the process, and the operating system around it, of hosting a trained large language model on GPU servers and delivering inference results in real time through an API in response to requests from users and applications.
If inference is the act of the model computing an answer, serving is the whole system that delivers that inference reliably, quickly, and to many people.
Aspect | Inference | Serving |
Scope | Model computation for one input | From receiving requests to responses, scaling, and failure handling |
Concerns | Accuracy, compute speed | Latency, throughput, availability, cost |
Example | Answering one question | Running an internal chatbot API for 1,000 employees |
How LLM Serving Works: The Request Flow

Component | Role |
API gateway | Authentication, rate limiting, routing |
Load balancer | Spreads requests across inference servers |
Inference engine | Runs model computation on GPUs (vLLM, SGLang, TensorRT-LLM, and others) |
KV cache | Keeps results for earlier tokens in GPU memory to avoid recomputation |
Autoscaler | Adjusts the number of inference servers with traffic |
Monitoring | Observes latency, throughput, error rate, and GPU status |
An LLM doesn't produce its answer all at once; it generates tokens one at a time. The work splits into a Prefill phase that processes the whole prompt and a Decode phase that generates answer tokens one by one. Because the two phases behave differently, architectures such as Disaggregated Inference have emerged to run them separately.

Inference Engines and Serving Platforms Are Different
This is a common point of confusion. An inference engine such as vLLM is responsible for how efficiently GPUs are used inside a single server (batching, memory management). A serving platform ties many inference servers together and is responsible for deployment, scaling, updates, failure recovery, and monitoring.

Aspect | Inference engine | Serving platform and operations layer |
Examples | vLLM, SGLang, TensorRT-LLM | Kubernetes-based operations platforms |
Responsibility | GPU efficiency within one server | Operating many servers and many models |
Core features | Continuous batching, KV cache management | Autoscaling, zero-downtime deploys, rollback, resource allocation |
Picking a good inference engine alone doesn't complete the service. If the engine is the engine, the serving platform is closer to the vehicle and the road it runs on.
Key LLM Serving Performance Metrics
Metric | Meaning | What users feel |
TTFT (Time To First Token) | Time from request to the first token | "How long until the answer starts?" |
Inter-token latency (TPOT) | Time between generated tokens | "How smoothly does the answer stream?" |
Throughput (tokens/s) | Total tokens generated per second | How many users can be served at once |
p95 and p99 latency | Latency of the slowest 5% and 1% of requests | How often it occasionally feels slow |
Error rate and availability | Share of failed requests, service uptime | Whether the service can be trusted |

Latency and throughput trade off against each other. Batching many requests together raises throughput but can increase latency for individual requests. That's why you need to set targets that fit the nature of your service first.
3 Ways to Serve LLMs: Which to Choose
Approach | Characteristics | Best fit |
Use an external API | Call a model provider's API | Fast start, when sending data outside is allowed |
Self-host on cloud GPUs | Deploy open-source models on cloud GPUs | Model customization, flexible scaling |
Self-host on-premises | Deploy models on in-house GPU servers | Data security, long-term cost control, regulated industries |
Organizations that can't send data outside, such as in finance, manufacturing, and the public sector, often choose to self-host. From that point on, how GPU infrastructure is run directly shapes service quality.
What Serving Demands from GPU Infrastructure
- Always-on occupancy: Training gives GPUs back when it ends; serving holds them 24 hours a day.
- Variable load: Traffic differs sharply between business hours and night. Fix GPUs at peak capacity and they sit idle overnight.
- Many models side by side: Real companies run several services, not one model. Giving each small model a whole GPU drives utilization down fast.
- Zero-downtime updates: When you improve and swap a model, the service must not stop.

So at the serving stage, you need an operations layer that splits GPUs to fit each service, scales with traffic, and swaps versions without downtime. TEN's AIPub provides autoscaling, zero-downtime deploys, rollback, and automatic failure recovery in its Operation workload for LLM and API serving, and its 100-block GPU partitioning lets several inference services share one GPU. We cover how to run it in practice in the next post.
Frequently Asked Questions (FAQ)
Q. Are LLM serving and LLM inference the same thing?
They are used similarly, but the scope differs. Inference is the model computing an answer; serving is the whole system of deployment, scaling, and monitoring that delivers that inference reliably through an API.
Q. Is installing vLLM enough for serving?
You can bring up an API on a single server. But if you need to scale across servers, update without downtime, recover from failures, and manage resources per team, you need an operations layer on top.
Q. How many GPUs does LLM serving need?
It depends on model size, concurrent users, and target response time. The first test is whether the model weights and KV cache fit in GPU memory; the most reliable way to get an exact count is to measure with your real workload.
Q. Does even a small model need a whole GPU?
No. With GPU partitioning, several small models can share one GPU, which can raise utilization substantially.
References
- vLLM Documentation
- NVIDIA, LLM Inference Benchmarking: Fundamental Concepts (TTFT and ITL definitions)
- TEN, AIPub Technical Document (2026.05)
Share article