What Is a Token? The Unit of LLM Cost
Look at the ChatGPT API pricing and it’s priced “per token.” NVIDIA calls the AI factory a “factory that mints tokens.” Here’s the token — now the unit for measuring AI’s cost and performance — explained.
What is a token?
A token is the smallest unit by which an LLM processes text. Where a person reads by words, an LLM cuts text into pieces called tokens and processes those.
A common misconception is “token = word.” It’s not. A token can be smaller or larger than a word. Short words become one token; long or complex words split into several. In English, a word averages roughly 1–2 tokens; languages like Korean tend to split more finely due to particle and ending structures.
So a token is the “currency unit of language” that the model understands and generates.
Input tokens and output tokens
Using an LLM, tokens come in two kinds.
Input tokens — the prompt you feed the model: questions, instructions, reference documents.
Output tokens — the answer the model generates.
API pricing usually counts these separately, and output tokens are often pricier than input — generating an answer takes more compute.
Context window: tokens you can hold at once
The context window is the maximum number of tokens a model can process at once. There’s a cap on the combined total of input and output.
A larger context window lets you feed longer documents at once or keep a longer conversation in context. But the longer the context, the more memory (KV cache) and compute processing takes — so supporting long context requires proportionally larger GPU memory.
Why tokens became the unit of AI cost and performance
Tokens became AI’s core unit for two reasons.
A unit of cost — An LLM’s compute scales with the number of tokens processed. So cloud APIs bill per token, and organizations running their own infrastructure gauge efficiency by “cost per token.”
A unit of performance — LLM serving performance is often measured in tokens per second. With the same GPU, generating more tokens per second means serving more users. As covered earlier, this generation speed depends heavily on memory bandwidth more than raw compute.
That’s why NVIDIA calls the AI factory a “token factory” and puts Tokens per Watt forward as a key metric.
FAQ
Q. Are tokens the same as words?
No. A token can be smaller or larger than a word — short words are one token, long ones split into several.
Q. Why are output tokens pricier than input?
Generating an answer one token at a time takes more compute than reading the input.
Q. Is a bigger context window always better?
It handles longer context, which helps — but needs more GPU memory and compute. Judge by your use case.
Q. What determines token throughput?
GPU memory bandwidth, the inference engine, and batching all play in. Inference especially is memory-bound, so bandwidth matters a lot.
Conclusion: it’s about how efficiently you “produce” tokens
Tokens have become the common unit for AI cost and performance. Ultimately, an AI service’s edge comes down to how many tokens you produce, how cheaply, from the same resources.
Token production efficiency depends directly on how well you operate GPUs. To process more tokens on the same GPU, you must allocate resources so GPUs don’t idle, tune power and clocks to the workload, and measure real throughput. TEN’s AIPub partitions GPUs so multiple inference workloads share resources efficiently, controls power per GPU card, and monitors utilization and throughput in real time — raising token production efficiency.
To process more tokens efficiently on the same GPU, explore AIPub.
References
OpenAI, Tokenizer Documentation
NVIDIA, “Tokens per Watt” and AI Factory Technical Blog
Hugging Face, Tokenizers Guide