Last-Level Caches Matter Even More in the HBM Era

By André Bonnardot

August 17, 2026

A decade ago, a high-end server processor might have accessed 50 to 100 GB/s of memory bandwidth. At the time, that seemed impressive. Today, AI accelerators routinely consume 3 to 8 TB/s of memory bandwidth, roughly 30 to 100 times that of their predecessors.

This dramatic increase in demand has driven the rapid adoption of high-bandwidth memory (HBM) in artificial intelligence (AI), high-performance computing (HPC), and other data-intensive applications. By placing multiple stacks of DRAM dies in the same package as the main system-on-chip (SoC) die, HBM provides memory bandwidth levels that would be difficult—or impossible—to achieve with traditional DDR memory alone.

Many people assume that once HBM enters the picture, memory bottlenecks become a thing of the past. Unfortunately, the reality is more complicated. HBM offers a much bigger pipe, but more bandwidth does not automatically translate into a faster or more efficient system. As a result, last-level caches (LLCs) are more important today than ever before

Modern SoCs are data-movement machines

Today’s SoCs are astonishingly complex devices containing hundreds of intellectual property (IP) blocks, including CPUs, GPUs, NPUs, DSPs, hardware accelerators, DMA engines, security functions, and more. These blocks communicate via one or more networks-on-chip (NoCs), with some advanced SoCs employing dozens of NoCs, each optimized for different traffic types and performance requirements.

To read the full article on Embedded, click here.

Explore Arteris IP:

• FlexNoC Interconnect IP
• CodaCache Last-Level Cache IP
• FlexGen Smart Network-on-Chip (NoC) IP
• FlexWay Interconnect IP