NVIDIA's full-stack inference software stack is reducing token costs by up to 5x through coordinated optimization across

Organizations scaling from AI pilots to production AI factories have shifted their focus from peak chip specifications to cost per token—how many useful tokens they can deliver per dollar, per watt and within required latency targets. NVIDIA has codesigned its full-stack inference software with GPUs, CPUs, networking and systems, strengthened by a broad open source ecosystem. On the NVIDIA Blackwell platform, this software stack has already reduced token costs by up to 5x on the DeepSeek V4 model in just one month, with leading companies and inference providers already seeing the compounding value.

The nature of AI workloads has fundamentally changed. Traditional web, search and software-as-a-service workloads were relatively predictable, with users loading pages, refreshing feeds or updating business records following similar software paths with scaling achieved by adding more of the same servers. In contrast, agents can reason, plan, call tools, spin up specialist subagents and manage massive context across multi-turn workflows, turning a single request into a distributed computing problem that can span hundreds of subagents, thousands of tasks and multiple large language models running across GPUs, CPUs, DPUs and storage systems. The software stack determines whether that complexity turns into wasted capacity or lower cost per token.

Lower cost per token comes from turning individual optimizations into system-level performance. NVIDIA's inference software stack connects three layers that work as one system, allowing individual optimizations to compound. Disaggregated serving, large expert parallelism over NVIDIA NVLink interconnect technology, NVFP4 precision and multi-token prediction each deliver meaningful gains on their own but combined increase throughput by up to 20x. Capturing these gains in production is complex, requiring coordination across the full inference stack from production operations and model runtimes to kernels, communication libraries and hardware access.

This full-stack foundation is amplified by the open source ecosystem. Many of today's most widely used open source AI frameworks and inference projects are built natively on NVIDIA CUDA, meaning new research and software optimizations run with leading performance on NVIDIA GPUs from day zero. PyTorch, launched in 2016 with native CUDA support, has coevolved with NVIDIA's architecture, giving developers access to innovations such as Tensor Cores, Transformer Engine and NVFP4 directly through a familiar framework. When breakthroughs such as DFlash speculative decode, which delivers up to 15x more throughput on existing hardware, or FastVideo, which generates 1080p videos in less than five seconds, land in PyTorch, they can run instantly on NVIDIA, helping AI factories convert research progress into lower token costs.

This open source momentum ensures that when frontier models like DeepSeek V4 are released, leading inference frameworks like vLLM and SGLang have day-zero deployment recipes for the NVIDIA Blackwell architecture, making the model accessible across millions of Blackwell GPUs. DeepSeek V4 performance on Blackwell improved by up to 5x within about a month across vLLM and SGLang frameworks, cutting token costs to roughly one-fifth of previous levels. This open source flywheel effect creates a cycle where more developers optimize CUDA-native inference paths, more production deployments feed back into the ecosystem and each software improvement increases delivered token output while lowering cost per token over time.