SK Hynix and SanDisk publish HBF (High Bandwidth Flash) specification: 512 GiB flash cache co-packaged with GPUs to address AI inference memory wall; endorsed by Google and Tenstorrent.
The text you've provided is already in English (though it reads like a machine translation). I'll refine it into polished, professional English while preserving all technical terms, numbers, and organization names:
---
HBM has demonstrated that by stacking and co-packaging storage media adjacent to GPUs, the data movement bottleneck in AI computation can be mitigated.
Now, SK Hynix and SanDisk have formally applied a similar approach to NAND, placing high-capacity flash storage next to various xPUs, including GPUs, via HBF to address the memory wall problem in AI inference.
Recently, the two companies released the "High Bandwidth Flash (HBF) High-Level Base Die Specification, Version 0.7.0" through OCP. Google and Tenstorrent are acknowledged in the specification as providers of feedback and recommendations.
Positioned between HBM and SSD, HBF is high-capacity flash storage situated adjacent to xPUs, jointly managed by Base Die and host software.
The application scope extends from static model weights to KV Cache, multiple models, mixture-of-experts, and agent workloads.
The physical constraints of NAND in write operations, durability, and management persist. The complexity arising from these constraints is transferred to the software stack of the Base Die and accelerators.
The memory wall problem has persisted from the CPU era into today's data-centric computing systems. As GPU performance improvements continue to outpace memory response times, data movement increasingly becomes a performance bottleneck.
Simultaneously, the growth in large language model scale and context length has created demands for higher memory bandwidth and significantly increased memory capacity.
Data involved in AI processing can be divided into two categories: one includes model inputs and intermediate results generated by successive processing layers—dynamic data commonly referred to as activations; the other includes weights (parameters) that define the model.
During inference, model weights generally remain unchanged. However, their total volume may far exceed what a single processing chip or its local cache can accommodate.
For sufficiently large models, some weights may be stored on SSDs within the rack or in network-attached storage further away from the processor.
This creates a multi-tiered data path: non-volatile storage provides long-term capacity, weights needed for computation are moved into DRAM (possibly HBM), and then cached in processor-side SRAM.
When a weight is first accessed, data may need to traverse several storage levels to reach the processor. If this weight is later evicted from cache and subsequently accessed again, the same path must be traversed once more.
The concept behind HBF mirrors the thinking that led to HBM: moving storage media closer to compute units through stacking and co-packaging.
However, the media employed and their roles differ. HBM provides low-latency, high read/write bandwidth working memory using DRAM; HBF leverages higher-capacity NAND to handle large-scale, read-heavy data originally stored on SSDs or network storage—such as model weights—while maintaining high aggregate read bandwidth.
In some system designs, model weights pre-loaded in HBF can bypass the traditional DRAM staging step and transfer directly to processor-side cache. While such a design complicates cache and memory management, it saves transfer overhead from external storage to DRAM.
As Sharad Chole, Chief Scientist and co-founder of Expedera, describes it, HBF aims to bridge the gap between "high-bandwidth access and high storage capacity." This characterization is more precise than simply describing HBF as a faster SSD or flash-based HBM.
The OCP specification formally defines HBF as a non-coherent, memory-centric flash device situated adjacent to GPUs, TPUs, or other xPUs, designed to enhance HBM by adding terabyte-scale additional memory capacity near the processor.
HBF neither directly replaces HBM nor simply extends its capacity. Instead, it introduces another memory tier between high-bandwidth memory and traditional storage, keeping a larger proportion of model data physically closer to compute units.
Base Die is more than a passive interconnect layer. While maintaining communication with the host xPU, Base Die manages the UCIe protocol and controls data movement between the host interface and NAND die.
Additionally, it handles host and NAND commands, ECC encoding and decoding, error reporting, transfer scheduling, read/write/erase status, NAND initialization, and TSV redundancy mapping.
This division of labor distinguishes HBF from "simply placing flash die next to the processor": NAND stacking provides capacity, while Base Die provides the control, interface, and reliability mechanisms necessary for the stack to function as a chiplet memory device.
The xPU connects to HBF via UCIe 3.0, a standard for intra-package die-to-die interconnect. Above the UCIe (Universal Chiplet Interconnect Express) interface, AXI serves as the communication protocol. UCIe handles the physical and link layers, while AXI carries read/write operations between the host and HBF.
The specification provides detailed descriptions of the UCIe protocol layer, Flit format, and AXI channels. However, the key point for understanding HBF's system role is that it requires a dedicated chiplet interface with corresponding link layer implementations on both the xPU and HBF sides—unlike PCIe SSDs, which can connect to existing accelerators without special modifications.
A single HBF stack supports up to 16 host channels. Each channel uses a dedicated UCIe link and provides access to its own NAND resources. Requests issued through one channel cannot access data connected to another.
Each channel has its own contiguous local address space. Base Die maps this local address space to physical NAND banks, dies, and blocks. At the system level, host software maps the host's global address to the local address space of available HBF channels.
The result is not a single, automatically unified flash pool, but rather a channelized architecture whose performance depends partly on how the host distributes data and requests among multiple independent resources.
This distinction is important because the specification requires that HBF and HBM be separately managed when used in the same system. HBF does not automatically become part of HBM's coherent address space. Software must decide which data belongs to which memory tier and how data should move, partition, and be accessed.
The reference configuration provided in the specification uses 16 NAND dies, 16 banks per channel, and 4 KiB NAND pages, with a total capacity of 512 GiB (approximately 550 GB), and allows for higher-capacity HBF stacks.
The maximum bandwidth target of approximately 3 TB/s results not from individual NAND achieving performance close to HBM, but from the convergence of 16 host channels combined with parallelism across multiple dies, banks, and arrays. At maximum configuration, each channel uses a 64-bit interface with data lane rates up to 32 GT/s.
To truly leverage these parallel resources, data must be distributed across sufficient channels, dies, and banks, and the host must maintain sufficient request concurrency with Base Die performing corresponding scheduling.
Therefore, approximately 3 TB/s is more accurately described as the specification target for maximum configuration, rather than measured performance from actual chips or inference workloads.
Even if aggregate read bandwidth reaches HBM-class levels, HBF is not equivalent to HBM. They differ significantly in latency, write capability, access granularity, durability, and memory semantics.
HBF's core advantage is pairing high-capacity NAND with highly parallel reads, rather than achieving DRAM-equivalent performance across all workloads.
Model weights range from several GiB to tens or even hundreds of GiB, must be fetched during token generation, and typically remain unchanged during inference. This makes them better suited for read-optimized, high-capacity NAND tiers compared to data that undergoes continuous rewrites.
As memory analyst Jim Handy summarizes: "Training continuously changes model weights, while inference typically keeps them unchanged."
However, the OCP specification assigns broader application scope to HBF. Its applications include single LLM serving, storing and switching multiple LLMs, mixture-of-experts models, multimodal models, agent workloads, AI parameter loading, and KV Cache read/write.
For a single model, the specification describes distributing model parameters across all host channels for parallel reading. For multiple models, it provides two possible layouts.
Keeping multiple models in HBF reduces the need to reload entire models from external SSDs when the active model switches. While this doesn't eliminate model-switching latency, it eliminates the separate SSD-to-accelerator loading step in traditional deployments.
For mixture-of-experts, multimodal, and agent workloads, data placement on HBF can follow two strategies: either scatter data across all channels to pursue maximum bandwidth, or place data on designated channels prioritizing capacity allocation and workload isolation.
During the prefill stage, model layers compute and write KV Cache; after entering the decode stage, the attention module reads previously generated KV Cache and adds new cache content when generating subsequent tokens.
Unlike model weights, which remain largely unchanged during inference, KV Cache continuously grows and undergoes constant read/write operations.
The specification expects the host to understand LLM and AI workload structure and arrange KV Cache data to optimize read/write performance. Thus, HBF is defined not only as a parameter-loading device, but also as a possible storage target for data generated during inference runtime.
This expands HBF's potential value, especially when longer contexts and agent workloads increase memory capacity demands. However, it also exposes the architecture directly to NAND's weakest characteristic—frequent writes.
NAND flash's physical structure determines its read/write access pattern. Flash chips are internally divided into blocks, each subdivided into pages.
The specification uses 4 KiB-sized NAND pages and supports 4 KiB-aligned burst writes. Write requests smaller than 4 KiB are cached in Base Die before actual writing to the NAND core die occurs only after a complete 4 KiB page is accumulated.
Within NAND blocks, HBF requires sequential writes and does not support direct random overwrites of already-programmed pages. To rewrite any data in a block, the entire block must first be erased, then written anew from the beginning in the prescribed page order.
These rules align relatively well with large, predetermined model weight layouts—weights are write-once, read-many data that can be written sequentially in complete pages.
However, for dynamically generated data, this is challenging, as their size, lifecycle, and update patterns can change during inference.
For data like KV Cache that require frequent fine-grained writes, the specification does not guarantee that all KV Cache workloads will perform well on HBF; actual performance depends on how the host combines small writes, lays out KV Cache pages, and avoids patterns leading to inefficient NAND block utilization.
Because model weights and KV Cache have different read/write patterns, the specification notes that mixing them in the same region may reduce durability and capacity utilization. Therefore, it recommends partitioning HBF at channel granularity and provides two example strategies.
Non-uniform partitioning allocates only enough channels to store active model weights and allocates remaining capacity to KV Cache.
This indicates that HBF's data layout cannot be entirely automated by hardware. The host must understand workload characteristics, estimate capacity requirements for weights and KV Cache, and decide how much bandwidth and durability budget to allocate to each.
Since each HBF channel has an independent address space, host software must map global addresses to different channels, decide how data is interleaved, and separately manage HBF and HBM.
Maintenance of the NAND medium is not entirely encapsulated within Base Die. Wear leveling can be performed by Base Die or controlled by the host through region remapping.
The host can adjust the mapping between logical and physical addresses to distribute programming and erase operations more evenly, but remapping commands do not automatically migrate existing data. The host must first stop relevant accesses before rewriting data to new locations.
Data retention and read disturb also require regular management. Base Die detects and reports NAND status, while the host may need to refresh data, retry reads, isolate failed capacity, or wait for device recovery.
However, HBF does not transfer all functions handled by traditional SSD controllers to host software. Base Die remains responsible for NAND commands, ECC, request scheduling, and error detection, but data layout, partial wear management, and anomaly recovery require host participation.
HBF's division of responsibilities differs from both HBM and traditional SSDs. The former, based on DRAM, does not face NAND's management complexity; the latter, while also using NAND, typically handles address translation, garbage collection, and wear management in the SSD controller, with upper-level software largely unaware of these processes.
HBF occupies a different position: Base Die retains device-side control capabilities but does not fully encapsulate all NAND media management. To fully leverage its capacity and parallel bandwidth, the xPU host and its software must still understand channels, data layout, and partial media status.
This is HBF's fundamental tradeoff: it moves high-capacity NAND closer to compute while requiring tighter coordination among hardware, firmware, runtime software, and workload data layout.