The 5x That Changes Everything: How NVIDIA Cut DeepSeek V4 Token Costs Without a New Chip

In one month, software optimizations alone made DeepSeek V4 five times cheaper to serve on Blackwell hardware. The implications go far beyond one benchmark.

In one month, NVIDIA's software stack made the same Blackwell GPUs generate five times more tokens per second on DeepSeek V4. The operators already owned the hardware. Nothing new was shipped. What changed was the software sitting on top — and the implications stretch far beyond a single benchmark.

The Numbers

SemiAnalysis InferenceX independently benchmarked the timeline:

Twenty-four days. Same rack. Same model. Nothing the operators had to buy.

Where that leaves the economics today:

At nine cents per million tokens — at production interactivity of 105 tokens per second per user — GB200 NVL72 is pricing inference at a level that makes most competing infrastructure look economically non-viable. And it got there in under a month without a transistor changing.

What Actually Changed

Four software optimizations. None revolutionary alone. Together, they compound.

Disaggregated Serving: Splitting the Brain in Two

DeepSeek V4 is a Mixture-of-Experts model — 1.6 trillion parameters total, with roughly 49 billion active on any given token. Running inference on it has two phases that fight each other for the same GPU resources.

Prefill processes the entire input prompt at once. It's compute-bound.

Decode generates output tokens one at a time. It's memory-bandwidth-bound, and the GPU's tensor cores sit half-idle waiting on HBM fetches.

Put both on the same GPU and one phase always bottlenecks the other. Disaggregated serving splits them across different GPUs inside the NVL72 rack, with the NVLink fabric (900 GB/s unidirectional per GPU) transferring the KV-cache between prefill and decode nodes at internal speeds. The gain alone is roughly 40-50%.

Wide Expert Parallelism: 72 GPUs Acting as One

MoE models route each token to a subset of specialized "experts." On standalone GPUs, the overhead of sending tokens between GPUs over PCIe or InfiniBand kills throughput. On NVL72, NVLink connects all 72 GPUs into one coherent fabric. Wide expert parallelism spreads every expert layer across all 72 GPUs, routing tokens at rack-internal bandwidth.

This is why GB200 NVL72 hits 6,644 tok/s/GPU while a standalone B200 — same silicon — hits 976. Seven times the throughput from architecture, not faster transistors.

NVFP4: Four Bits, Half the Bandwidth

Blackwell introduced native NVFP4, a 4-bit floating point format for inference. DeepSeek V4 was trained with FP4 quantization in mind, using Blackwell-native MXFP4. For the active 49 billion parameters running during decode, NVFP4 cuts memory bandwidth requirements roughly in half versus FP8. When decode is memory-bandwidth-bound, that nearly doubles throughput.

Multi-Token Prediction: Stop Going One at a Time

Autoregressive decode is sequential — predict one token, feed it back, predict the next. Each step leaves the GPU partially idle. Multi-token prediction (MTP) predicts several tokens in parallel and verifies them, filling those idle cycles. At 50+ tokens per second per user, those cycles add up to a meaningful gain.

The Stacking Effect

Individually: disaggregated serving adds ~40-50%. Wide EP adds ~200-300%. NVFP4 adds ~80-100%. MTP adds ~30-50%. Together, they compound to the ~5× throughput NVIDIA documented between April 25 and May 19 — and up to 20× from the initial day-zero baseline when all optimizations are layered across the full stack.

The word NVIDIA keeps using is "compound." It's the right word.

The Orchestration Layer: Dynamo

All four optimizations are orchestrated by NVIDIA Dynamo, the inference framework that sits between the model runtime and the hardware. Dynamo handles scheduling disaggregated prefill and decode across the 72-GPU rack, placing experts and routing tokens at the right moment, KV-cache management, and batching strategies that maximize utilization at every interactivity target.

Competitors can write CUDA kernels. What they cannot replicate is the full-stack co-design between Dynamo, the NCCL and NVLink communication libraries, the GPU microarchitecture, and the model architecture. Jensen Huang's "extreme co-design" thesis plays out here — on production hardware operators already own.

Who's Already Running This

Per NVIDIA's June 30 blog post and corroborated by @benitoz, the following inference providers are deployed on the optimized stack:

• Cognition — the company behind Devin, the AI coding agent
• Baseten — one of the largest model-serving platforms
• Deep Infra — inference infrastructure provider that has publicly documented their DeepSeek V4 cost reductions
• Together AI — the open-source model inference platform
• Cursor — the AI-first code editor used by millions of developers

These are not lab experiments. They are production inference fleets serving paying customers.

What the Market Keeps Getting Wrong

The stock market's instinct is to read "cheaper tokens" as deflation. Lower prices per token means less revenue for inference providers. Less revenue means less willingness to buy GPUs. Less GPU demand means NVIDIA is in trouble. Sell the stock.

This is exactly backward.

NVIDIA's revenue doesn't come from selling tokens. It comes from selling GPUs. The operator's ROI math is simple: how many tokens can this GPU serve over its useful life, and what will the market pay for those tokens?

When NVIDIA's software makes the same GPU generate 5× more tokens per second, the operator's ROI on that GPU just went up five times. Yes, token prices will fall — that's the point. But the GPU's output volume rises faster than the token price declines. The operator makes more money on the same capital investment.

Higher GPU ROI → operators order more GPUs → NVIDIA's next quarter is bigger than the last.

This is the Jevons paradox, and it is now NVIDIA's explicit business model. Every 5× cost reduction expands the addressable market: real-time agentic AI, video generation at scale, always-on reasoning loops, AI-first enterprise applications that were uneconomical at the old token prices. The cheaper tokens get, the more use cases become viable. Cheaper tokens don't shrink the market. They flood it.

As @benitoz put it on June 30:

"Software cutting cost on the installed base is a second revenue engine, and cheaper tokens don't shrink the market — they flood it. Jevons isn't a theory anymore. It's Nvidia's business model."

The ASIC Threat, Revisited

The competitive case against NVIDIA has always been: custom silicon will eventually be cheaper than general-purpose GPUs for inference. Google has TPUs. Amazon has Trainium. Microsoft has Maia. Meta has MTIA.

The 5× software optimization story undermines every premise of that thesis.

Custom silicon competes on hardware specs. NVIDIA is now competing on software velocity — the compounding rate of throughput improvement on hardware customers already own. Can a TPU improve its DeepSeek V4 throughput 5× in a month with a software update? The answer is almost certainly no, not because Google lacks talent, but because the ecosystem — CUDA kernels, communication libraries, inference framework, and silicon — must evolve in lockstep. NVIDIA controls the entire stack. ASIC vendors control one layer.

SemiAnalysis's InferenceX benchmarks from December 2025 found that GB200 NVL72 delivered approximately 15× better value than AMD's MI355X on MoE inference workloads. At $0.09 per million tokens on DeepSeek V4, the TCO math for any non-NVIDIA inference deployment on large MoE models is deeply challenging.

The Bigger Picture

The semiconductor sector has been churning sideways — violent rotation between wafer fab equipment names (AMAT, KLAC, LRCX surging) and AI chip names (NVDA, AVGO under pressure) while the SOXX ETF averages it all to a flat line. The 5× token cost story is the microcosm of that debate.

The bears see collapsing token prices and say: margins are compressing before revenues materialize. The capex boom is a bubble. NVIDIA's ROIC narrative will break when hyperscalers get impatient.

The bulls see software compounding at a rate that nobody can match, and say: the moat is widening, not narrowing. And it's widening in software, where margins are effectively infinite, not in silicon, where they're finite.

The sideways price action in NVDA — pinned between $190 and $200 by dealer gamma, compressing from 30× trailing earnings to 15× forward — reflects a market that cannot decide which of these two reads is correct. It cannot price the stock up because the "token deflation equals bad" narrative hasn't broken. It cannot price it down because the structural demand thesis is too immense to bet against. Goldman Sachs now projects the four hyperscalers will spend $5.3 trillion on AI infrastructure from 2025 through 2030, with $725 billion in capex in 2026 alone.

What to Watch

The avalanche of semiconductor earnings beginning in late July will provide the first hard data on which side of this debate is right:

If operators are accelerating Blackwell purchases in response to the software economics NVIDIA just demonstrated, it will show up in WFE guidance first. AMAT and LRCX see the orders before the GPUs ship. But the real test is whether hyperscaler capex commentary during their own July earnings confirms or walks back the $5.3 trillion trajectory.

The Closing Frame

A company that can make its installed base 5× more productive in a month, without asking customers to spend another dollar, is not a hardware company with a software problem. It's a platform company that happens to sell hardware.

The platform is CUDA. The platform is Dynamo. The platform is the communication libraries and the NVLink fabric and the NVFP4 quantization ecosystem and the multi-token prediction research pipeline — all of which compound together every month, on silicon that was already sold.

The market is still pricing NVIDIA like a chipmaker whose cycle peaks when the next process node is delayed. It should be pricing it like a software platform whose value to its customers compounds over time — not depreciates.

Whether it gets that re-rating in the next six months or the next two years depends on one thing: whether the July earnings wave confirms that operators are buying more GPUs because their existing GPUs just got better.

Sources

• NVIDIA Blog — "How NVIDIA's Inference Software Stack Powers the Lowest Token Cost" (June 30, 2026)
• SemiAnalysis InferenceX — DeepSeek V4 Pro 1.6T: B200 vs GB200 NVL72
• SemiAnalysis InferenceX — DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time (June 9, 2026)
• @benitoz — "NVIDIA just cut DeepSeek V4 token cost to roughly one-fifth..." (June 30, 2026)
• @rohanpaul_ai — NVIDIA Blackwell chart showing 5× improvement in one month (June 30, 2026)
• Wccftech — "NVIDIA Slashes DeepSeek v4 Token Costs By Up To 5x Just One Month After Launch" (July 1, 2026)
• Goldman Sachs — "The Assumptions Shaping the Scale of the AI Build-Out" (May 2026), with $5.3T capex revision reported June 3, 2026
• SemiAnalysis InferenceX — NVIDIA Blackwell vs AMD vs Hopper (February 2026)