Data Centers Have Evolved Into 'AI Token Factories'

NVIDIA has called on enterprises to fundamentally rethink how they evaluate artificial intelligence (AI) infrastructure costs. Published through NVIDIA's official blog, the argument centers on a single proposition: cost per token is the only meaningful total cost of ownership (TCO) metric for AI infrastructure.

NVIDIA argues that traditional data centers — once facilities for storing and processing data — have transformed into 'intelligence factories' that manufacture intelligence in the form of tokens. With AI inference now the dominant workload in data centers, the framework for evaluating infrastructure economics must evolve accordingly.

Enterprises Are Still Looking at the Wrong Metrics

Three metrics typically guide AI infrastructure evaluations:

• Compute cost: What enterprises pay for AI infrastructure, whether rented from cloud providers or owned on-premises
• FLOPS per dollar: Raw computing power per dollar spent
• Cost per token: The all-in cost to produce one million tokens

NVIDIA classifies the first two as 'input metrics' and argues they are insufficient. Since the business value AI generates flows from tokens — the output — optimizing only inputs represents a fundamental mismatch.

The 'Inference Iceberg' — What You See vs. What Actually Drives Cost

NVIDIA frames this using the 'inference iceberg' analogy. The GPU hourly cost, the numerator in the cost-per-token equation, sits above the waterline — visible and easy to compare. But what actually determines the unit cost is below the surface: the denominator, or real-world token throughput.

Maximizing the denominator delivers two simultaneous business effects:

1. Lower token cost: Higher throughput per unit time drives down cost per token, expanding profit margins on every AI interaction served.
2. Higher revenue: More tokens per megawatt means more intelligence delivered from the same infrastructure investment.

Surface vs. In-Depth Evaluation Framework

Technical Factors That Determine Real Token Cost

The factors NVIDIA identifies as critical to real-world inference performance include:

MoE (Mixture-of-Experts) model support: The most widely deployed AI model type today generates 'all-to-all' communication patterns. Scale-up interconnects that cannot handle this traffic become bottlenecks.

FP4 precision support: FP4 reduces memory usage and compute cost, but requires an inference stack capable of leveraging it without accuracy degradation.

Speculative decoding and multi-token prediction: Key optimization techniques that improve user interactivity.

Serving layer optimization: Disaggregated serving, KV-aware routing, and KV-cache offloading are central to maximizing real throughput.

Agentic AI workloads: Platforms must also handle the unique requirements of agentic pipelines that go beyond simple query-response patterns.

NVIDIA claims that when all these factors are accounted for, it delivers the lowest cost per token in the industry.

Historical Thread: The Evolution of Inference Economics

The debate over AI infrastructure costs did not emerge overnight. Following the launch of ChatGPT in 2022, enterprise AI adoption surged, and by 2023, the cost of cloud-based AI APIs had become a primary concern. OpenAI, Anthropic, Google, and others raced to simultaneously improve model performance while cutting API prices.

By 2024–2025, LLM inference workloads had become a dominant driver of data center power consumption. The rise of MoE architectures and reasoning-specialized models made simple GPU compute comparisons insufficient for predicting real-world costs.

In 2026, 'token economics' has become the central language of AI infrastructure procurement. NVIDIA's TCO reframing is both a continuation of this trend and a strategic narrative designed to differentiate from competing chips — including AMD, Intel, and custom TPU designs.

[Expert Analysis] Token Economics May Reshape Enterprise AI Strategy

NVIDIA's argument is simultaneously a technical document and a market framing strategy. Establishing 'cost per token' as the industry standard metric directly maps to the competitive advantages of its own software ecosystem — CUDA, TensorRT-LLM, NIM, and beyond.

For enterprise AI infrastructure teams, the framing is compelling. Measuring the cost per million AI responses actually delivered to customers is a more direct proxy for business profitability than a line on a cloud invoice.

However, this metric is difficult to measure in a vendor-neutral way. Cost per token varies significantly depending on the model, workload, and deployment configuration. Enterprises serious about applying this metric to purchasing decisions are likely to demand independent benchmarks based on their own actual workloads.

Looking ahead, the AI infrastructure market is likely to shift from a 'hardware spec competition' to a 'software optimization competition.' Even on identical hardware, inference stack quality can produce multi-fold differences in token throughput — a structural shift that could reshape the competitive landscape not just for NVIDIA, but for the entire AI software ecosystem.