Inference Optimization in the Blackwell Era: Why Cost-Per-Token Is the New Standard

"If you picked your GPU based on FLOPs/$ alone, your inference costs will make you regret it."

The center of gravity in AI is shifting fast. Training is a one-time cost. Inference is a continuous operating expense — incurred with every user query, every agentic workflow step, and every automated decision.

That's why NVIDIA has been pushing "cost-per-token" as the defining metric since the Blackwell generation. The company has gone so far as to declare that "legacy data centers have evolved into AI token factories." The real output of infrastructure is no longer compute — it's tokens.

This post covers why FLOPs/$ is no longer sufficient, how Blackwell is reshaping inference economics, and what infrastructure operators need to prepare for right now.

Why FLOPs/$ Is No Longer Enough

FLOPs/$ — theoretical compute per dollar — has been the most widely referenced metric for GPU procurement. In the training-centric era, it made sense. The GPU that delivered more compute for less money was the better GPU.

But in inference, there are factors that matter more than raw FLOPs.

Memory bandwidth is the real bottleneck. The most frequent operations in inference are KV-cache access and long context window processing. No matter how powerful a GPU's compute cores are, if memory can't feed data fast enough, the cores sit idle. Most inference workloads are memory-bandwidth-bound, not compute-bound.

Batch efficiency determines throughput. Inference must handle massive numbers of simultaneous user requests. How many requests you can batch together directly determines tokens generated per second — your actual throughput.

Latency defines user experience. In agentic AI workloads especially, Time to First Token (TTFT) determines the responsiveness of the entire workflow. AI coding assistants and agentic applications have surged from 11% to roughly 50% of all AI queries over the past year (OpenRouter State of Inference report), showing just how critical latency requirements have become.

This is why NVIDIA officially declared that "evaluating AI hardware by FLOPs/$ is a fundamentally flawed approach," positioning cost-per-token as the new standard.

How Blackwell Reshapes Inference Economics

Let's start with Blackwell Ultra's (GB300 NVL72) headline numbers.

According to SemiAnalysis InferenceX benchmarks, the GB300 NVL72 delivers up to 50x higher throughput per megawatt and up to 35x lower cost per token compared to the Hopper platform.

The key insight: this isn't about "cheaper per hour."

In fact, Blackwell is more expensive per hour. The GB300 NVL72 costs approximately $2.65/hr — nearly double the $1.41/hr of the Hopper (HGX H200).

But when you compare actual output — token generation — the picture flips entirely. Cost per million tokens: Blackwell at $0.12 versus Hopper at $4.20. Despite higher hourly costs, Blackwell generates 65x more tokens per second, delivering the 35x cost reduction.

This is the reality that FLOPs/$ can't capture. Cost-per-token can.

Blackwell Ultra: Key Specs and Technical Background

Here's the technical foundation behind Blackwell Ultra's performance leap.

Each GPU features 288GB of HBM3e memory with 8TB/s memory bandwidth, allowing large model KV-caches to stay in GPU memory. NVLink provides 130TB/s bandwidth across 72 GPUs, enabling high-speed communication optimized for Expert Parallelism in MoE (Mixture of Experts) models.

NVFP4 (4-bit floating point) quantization support dramatically reduces model size while maximizing throughput. Blackwell Ultra delivers 1.5x more NVFP4 AI compute and 2x more attention-layer acceleration compared to the previous Blackwell generation.

At MLPerf Inference v6.0 (April 2026), Blackwell Ultra recorded the highest throughput and lowest token cost across all categories. For DeepSeek-R1 inference, it achieved 5,842 tokens per second per GPU offline and 2,907 tokens per second in the server scenario — roughly a 5x improvement over Hopper.

Software optimization is equally critical. NVIDIA Dynamo and TensorRT-LLM deliver GPU kernel optimization for low-latency inference, NVLink Symmetric Memory for direct GPU-to-GPU access, and Programmatic Dependent Launch to minimize idle time between operations.

This hardware-software co-design — what NVIDIA calls Extreme Co-design — is the core driver behind Blackwell's performance leap.

The Synergy Between MoE Architecture and Blackwell

MoE (Mixture of Experts) architectures, exemplified by DeepSeek-R1, are emerging as the key to inference efficiency.

MoE models activate only a subset of expert parameters for each input, meaning the actual compute used per inference is a fraction of total model parameters — structurally reducing inference costs. However, to fully realize MoE efficiency, experts must be distributed across multiple GPUs with fast "all-to-all" communication.

The GB300 NVL72's 130TB/s NVLink bandwidth provides the infrastructure to handle this all-to-all traffic at high speed. Expert Parallelism across 72 GPUs becomes practical while keeping latency minimal.

The combination of MoE + NVFP4 + Blackwell Ultra is currently regarded as the optimal configuration for inference cost reduction — and it's this combination that powered the record-breaking MLPerf v6.0 results.

What Inference Optimization Demands from Infrastructure Operations

Inference workloads have fundamentally different operational characteristics from training.

Always on. Training jobs end and release GPUs. Inference serving runs continuously as long as the service is live.

Traffic is volatile. Inference requests spike and dip by time of day and by event.

Latency-sensitive. Especially in agentic AI environments, latency directly impacts user experience and workflow efficiency.

What this demands from infrastructure operations is clear.

• Precise resource allocation. Allocate only as much GPU as inference needs, and make the rest available for other workloads. Training occupies entire GPUs, but inference typically serves multiple models simultaneously with smaller GPU slices. This is exactly where the ability to partition GPUs at the block level becomes especially powerful for inference.
• Real-time monitoring. Track GPU utilization, memory occupancy, PCIe/NVLink traffic, and power consumption to quickly identify inference serving bottlenecks.
• Autoscaling for traffic fluctuations. Scale up during peak hours, scale down during idle periods. Automated scaling is the core of inference cost optimization.

AIPub's block-level spatial partitioning is particularly well-suited to inference environments. By splitting one GPU into 100 blocks, multiple inference models can be served simultaneously on a single GPU without interference. Each model's usage is tracked independently, enabling precise cost-per-token-based cost management.

Making the Blackwell Decision with Data

"Should we deploy Blackwell, or stay with Hopper?"

The answer depends entirely on your workload.

On hourly cost alone, Hopper is cheaper. But on cost-per-token, Blackwell is 35x more efficient. If large-scale inference serving is your primary workload, Blackwell's ROI is overwhelming. If small-scale fine-tuning is the focus, Hopper may still be more cost-effective.

RA:X's benchmark-driven consulting answers the question "Which GPU, and how many?" with data — not guesswork. It runs your actual models and data samples on TEN's reference architecture, comparing GPU options under identical conditions to recommend the optimal configuration for your specific workload.

Conclusion: Inference Economics Are Decided by Infrastructure Operations, Not GPU Specs

Even if Blackwell Ultra cuts cost-per-token by 35x, that efficiency won't be realized without the right infrastructure operations behind it.

You need an operational framework that precisely allocates GPU resources to workloads, tracks usage in real time, automatically redistributes idle resources, and accurately attributes costs by team and project. Only then does Blackwell's potential translate into actual ROI.

Whether you're designing GPU infrastructure from scratch or evaluating a transition from Hopper to Blackwell — don't navigate this alone. Design the optimal infrastructure strategy with TEN's experts, grounded in data.

👉 Learn more about AIPub: https://ten1010.io/en/products/aipub-dev
👉 Explore RA:X consulting: https://ten1010.io/en/products/rax
📩 Talk to an expert: https://ten1010.io/en/inquiry