SambaNova's AI inference optimization techniques could significantly cap hyperscaler AI capex by reducing compute requirements for inference workloads.

SambaNova has reframed the AI infrastructure debate away from simply deploying larger GPU clusters toward maximizing efficiency of existing hardware. New benchmark results demonstrate its fifth-generation Reconfigurable Dataflow Units (RDUs) operating alongside Nvidia H200 GPUs to achieve substantially higher inference throughput than GPU-only systems on selected workloads. By separating prompt processing from token generation, the approach lets enterprises keep their existing GPU infrastructure productive while delegating decode operations to specialized accelerators. If production deployments match the demonstrated performance, organizations could reduce inference costs while extending the operational life of installed AI systems.

For years, the AI industry pursued a singular goal: deploy larger GPU clusters. Each new generation increased compute performance, memory bandwidth, power consumption, and cost. This strategy suited training foundation models, where raw floating-point performance determines how quickly massive datasets convert to usable models. Inference, however, presents different economics.

Once a model reaches production, serving millions of users depends less on peak compute capability than on prompt processing, memory management, context handling, and sustained token generation. This shift has prompted AI infrastructure vendors to rethink system architecture rather than simply adding more GPUs.

SambaNova believes heterogeneous inference represents the next phase of AI infrastructure evolution. At the RAISE Summit 2026, the company demonstrated what it describes as a premium inference platform built on heterogeneous architecture. The configuration pairs one Nvidia H200 rack containing four GPUs for the compute-intensive prefill stage with one SambaRack SN50 populated with 16 fifth-generation RDUs dedicated to decode operations. This extends the heterogeneous blueprint first shown at COMPUTEX, where Nvidia B200 GPUs handled prefill while SambaNova SN40 accelerators performed decode. The latest configuration replaces the SN40 with the new SN50 and targets emerging agentic AI workloads.

Independent benchmarking by Artificial Analysis using the MiniMax M2.7 model reported decode performance reaching 850 tokens per second on short-context workloads while sustaining more than 450 tokens per second on long-context inference. Earlier benchmark disclosures had reported approximately 763 tokens per second. The newer results reflect continued optimization of the SN50 platform while establishing a new performance baseline for MiniMax inference.

Inference naturally divides into two distinct operations. The prefill phase processes prompts, performs matrix computations, and generates the key-value cache—work at which GPUs excel due to its highly parallel nature. Decode follows a fundamentally different pattern: each output token depends on previous tokens, making memory bandwidth and data movement far more important than peak arithmetic throughput. SambaNova specifically designed its RDU architecture for this portion of the inference pipeline. Rather than replacing GPUs, the system assigns each processor the workload it executes most efficiently.

For cloud providers and enterprise operators, this distinction fundamentally changes infrastructure economics. Existing H200 GPU clusters continue handling compute-intensive prefill work while SN50 systems provide dedicated decode capacity. Organizations can increase inference throughput without replacing entire GPU installations.

The benchmark reveals more than higher token-generation rates; it highlights SambaNova's underlying dataflow architecture. Traditional GPU architectures emphasize massive parallel execution supported by increasingly large memory systems. SambaNova instead focuses on reducing data movement by overlapping communication with computation and keeping active data closer to execution resources. The company argues that sustained inference depends more on architectural efficiency than on advertised peak FLOPS.

The SN50 represents SambaNova's fifth-generation Reconfigurable Dataflow Unit and replaces the SN40L introduced during 2024. According to the company, the new processor increases FP16 performance by approximately 2.5× while raising FP8 throughput by roughly 5×, reaching approximately 1.6 PFLOPS FP16 and 3.2 PFLOPS FP8. While these figures remain below the theoretical peak performance of Nvidia's latest Blackwell-class GPUs, SambaNova argues that peak specifications rarely translate directly into production inference because memory movement, synchronization overhead, and utilization often determine real-world performance.

Memory architecture illustrates this philosophy. Each SN50 incorporates 432 MB of on-chip SRAM, 64 GB of HBM2E delivering approximately 1.8 TB/s of bandwidth, and between 256 GB and 2 TB of DDR5 memory. SambaNova intentionally retained HBM2E instead of adopting newer memory technologies, citing lower cost and improved supply availability. The larger DDR5 pool allows models and key-value caches to move between memory tiers in milliseconds, supporting rapid model switching while maintaining high utilization.

SambaNova joins a growing list of companies embracing disaggregated inference. Nvidia introduced the concept through its NVL72 systems by allocating different GPU resources to prefill and decode workloads, later expanding that strategy following its acquisition of Groq engineering talent. AMD, AWS, Cerebras, and several emerging accelerator companies have introduced similar architectures. The industry increasingly recognizes that no single processor architecture delivers optimal efficiency across every stage of AI inference. For hyperscalers operating hundreds of thousands of accelerators, modest utilization improvements translate directly into lower operating costs and higher service capacity.

Perhaps SambaNova's most compelling argument centers on installed infrastructure rather than new hardware. Most enterprises already own significant investments in Nvidia accelerators, many of which continue providing substantial compute capability despite newer GPU generations. Rather than encouraging replacement, SambaNova proposes extending their useful life. Existing H100 and H200 clusters continue executing prefill while SN50 racks perform decode. Enterprises gain higher inference throughput without purchasing complete replacement GPU clusters. Unlike many next-generation AI systems requiring liquid cooling, SambaNova's SN50 racks remain air-cooled and fit within existing data center infrastructure. CIOs evaluating AI expansion can therefore increase inference capacity without undertaking expensive facility upgrades.

SambaNova recently strengthened both its financial position and industry partnerships. The company completed the first close of a $350 million financing round as part of a broader $1 billion Series F, producing an estimated valuation of approximately $11 billion. General Atlantic led the financing, with participation from Intel Capital, Vista Equity Partners, Cambium Capital, and several additional institutional investors. Intel's participation addresses speculation that the company intended to acquire SambaNova; instead, Intel chose a multi-year engineering partnership focused on heterogeneous AI infrastructure. The collaboration combines Intel Xeon processors with SambaNova RDUs through joint hardware-software optimization aimed at enterprise AI deployments. Intel identified Vector Core Compute as one of the first deployment partners, while Together AI will become an early large-scale commercial customer. For Intel, the partnership expands its AI infrastructure strategy beyond GPUs following limited success with previous data center GPU and Gaudi initiatives.

SambaNova views today's 16-RDU system as only the beginning. Future configurations will expand to 128 and, eventually, 256 RDUs connected through a switched interconnect delivering approximately 2.2 TB/s of bidirectional chip-to-chip bandwidth per accelerator. Those larger systems target AI agents, coding assistants, retrieval-augmented generation, enterprise search, and other production inference workloads where latency, throughput, utilization, and cost per generated token determine profitability.

The architecture also addresses an emerging challenge: enterprise AI increasingly depends on customized models. Departments, customers, and applications often require separate models rather than sharing one foundation model, lowering accelerator utilization because each model occupies valuable memory resources. SambaNova's large DDR5 memory pool allows models and key-value caches to move rapidly between memory tiers, allowing infrastructure providers to maintain higher utilization while accommodating model specialization.