What Is Kubernetes? Why AI Infrastructure Uses It
What is Kubernetes? Learn how it automates container deployment, scaling, and recovery, how it differs from Docker, and why AI and GPU teams rely on it.
Nov 04, 2026
Contents
What Is Kubernetes?How Docker and Kubernetes DifferKubernetes Architecture at a GlanceKey Kubernetes Terms to KnowWhy Kubernetes Became the Standard for AI InfrastructureWhere Kubernetes Falls Short with GPUsWhy You Need a GPU Operations Layer on Top of KubernetesFrequently Asked Questions (FAQ)When an AI team first gets a GPU server, life is simple. You log in and run your training script. Then the fleet grows to 3 servers, then 10, more teams join, and the questions change.
"Which server should this job run on?" "If one server dies, what happens to our inference service?" "Every researcher uses different library versions, so how do we keep environments consistent?"
The industry's standard answer to these questions is Kubernetes. This post covers what Kubernetes is, how it is structured, the terms you need to know, why it became the standard for AI and GPU infrastructure, and where it falls short.
What Is Kubernetes?
Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and recovery of containerized applications.
Google open-sourced it in 2014 based on its experience running internal systems, and today it is maintained by the CNCF (Cloud Native Computing Foundation). It is often shortened to k8s, because there are 8 letters between the "K" and the "s".
Put simply, Kubernetes ties many servers together so they behave like one large computer. You declare "run 3 of this container," and Kubernetes decides which servers to place them on and restarts them if one fails.
How Docker and Kubernetes Differ
Aspect | Docker | Kubernetes |
Role | Builds and runs containers | Places and manages containers across many servers |
Scope | One server | A cluster (many servers) |
Analogy | The shipping container | Port control deciding which ship each container goes on |
Failure handling | Manual restart | Automatic recovery and rescheduling |
Scaling | Adjust counts by hand | Keeps the declared count, scales automatically with load |
They are not competitors; they do different jobs. Kubernetes runs the images you build with Docker on top of a cluster. Note that since Kubernetes 1.24, dockershim has been removed and Kubernetes uses runtimes such as containerd directly, but images built with Docker still run as-is.
Kubernetes Architecture at a Glance
A Kubernetes cluster has two main parts: the control plane (the brain) and worker nodes (the servers that do the work).
Component | Location | What it does |
API Server | Control plane | Entry point for every request; reads and changes cluster state |
etcd | Control plane | Distributed key-value store that holds cluster state |
Scheduler | Control plane | Decides which node a new Pod runs on |
Controller Manager | Control plane | Continuously reconciles declared state with actual state |
kubelet | Worker node | Runs and watches Pods on the node |
kube-proxy | Worker node | Routes Service traffic (can be replaced by Cilium and others) |
Container runtime | Worker node | containerd and similar; runs the containers |
Key Kubernetes Terms to Know
Term | One-line definition | Example in AI infrastructure |
Pod | The smallest deployable unit; one or more containers | A container running one training script |
Deployment | Keeps a set number of Pods running and manages updates | Always keep 3 inference API servers up |
Service | A stable address for a group of Pods | Internal endpoint for an inference API |
Job | A one-off task that ends when done | Model training, batch inference |
Namespace | A logical partition of the cluster | Separate spaces for Team A and Team B |
ResourceQuota | Resource limits per Namespace | Up to 4 GPUs per team |
Custom Resource | User-defined resources that extend Kubernetes | Foundation for ecosystem tools like GPU Operator and Kubeflow |
Why Kubernetes Became the Standard for AI Infrastructure
- Reproducible environments: Libraries and CUDA versions are baked into the image, so the environment is the same on any server. Fewer "but it worked on my machine" moments.
- Resource scheduling: It looks at free capacity across dozens of servers and places jobs automatically.
- Self-healing and scaling: It restarts a failed inference server and adds replicas when traffic grows.
- An extensible ecosystem: NVIDIA GPU Operator, distributed training tools, and monitoring stacks all run on Kubernetes.
- A common base for on-prem and cloud: You can run in-house GPU servers and cloud the same way, which suits hybrid strategies.
Where Kubernetes Falls Short with GPUs
Kubernetes was originally designed for CPU-centric web services. When it comes to GPUs, there are clear gaps the default features don't cover.
Operational need | Kubernetes default | What GPU operations require |
GPU allocation unit | Whole GPUs as integers ( nvidia.com/gpu: 1) | Fractional allocation sized to the job |
Scheduling | Place in order wherever there is room | Priority queues, co-scheduling for distributed training |
Multi-tenancy | Configure Namespace and RBAC separately | Unified per-team isolation of resources, images, and storage |
Usage visibility | Not provided | GPU usage aggregated per team and project |
Ease of use | YAML and kubectl first | A UI researchers can use right away |
For example, an inference job that needs only 10GB of memory still takes a whole 80GB GPU in default Kubernetes. And a distributed training job that needs 8 GPUs at once can end up holding a partial set while it waits, blocking other jobs' resources in the process.
Why You Need a GPU Operations Layer on Top of Kubernetes
Kubernetes is an excellent foundation for AI infrastructure, but it is not a GPU operations platform by itself. That is why many organizations add an operations layer on top that handles GPU partitioning, scheduling, multi-tenancy, and usage tracking.
TEN's AIPub is a Kubernetes-based GPU workload orchestration platform that provides this layer. Built as Kubernetes Custom Resources, it runs on standard Kubernetes and fills the gaps in the defaults.
- GPU partitioning: Split GPUs into 100 blocks (1% units) or with MIG, and allocate to fit each job
- GPU-aware scheduler: Priority queues and distributed-training-aware scheduling, plus Dynamic Allocation that redistributes idle resources
- Project-based multi-tenancy: Users, nodes, image registries, volumes, and quotas bundled and isolated together
- Web UI without YAML: Create and run Workspaces, Jobs, and Operations from the screen
- In sync with the Kubernetes ecosystem: Roughly 3-month release cycles keep pace with upstream
Understanding Kubernetes is the starting point for running AI infrastructure. And the moment several teams start sharing GPUs, the next question becomes what to run on top of it.
Frequently Asked Questions (FAQ)
Q. Should I learn Docker or Kubernetes first?
Start with Docker. Kubernetes is a tool for running containers, so once you understand images and containers, Pods and Deployments make much more sense.
Q. What does k8s mean?
It is shorthand for Kubernetes, replacing the 8 letters between the first K and the last s with the number 8. The name comes from the Greek word for "helmsman."
Q. How do you use GPUs in Kubernetes?
Install the NVIDIA device plugin or GPU Operator, and the node's GPUs are registered as the
nvidia.com/gpu resource. Pods can then request a number of GPUs. Allocation is whole-GPU by default, though, so partitioning and priority scheduling need additional tools.Q. Do small teams need Kubernetes?
If one team is using 1 or 2 GPU servers, you may not need it yet. The payoff grows when servers multiply, several teams share them, or you need to run inference services reliably.
Q. Is it better to build on Kubernetes yourself or adopt a platform?
Kubernetes itself is open source, but assembling and maintaining GPU partitioning, scheduling, multi-tenancy, and usage tracking takes significant engineering effort. We cover how to decide in "GPU Platform: Build vs Buy TCO."
References
- Kubernetes Documentation, Overview
- Kubernetes Documentation, Kubernetes Components
- Kubernetes Blog, Dockershim Removal FAQ
- NVIDIA, GPU Operator Documentation
- TEN, AIPub Technical Document (2026.05)
Share article