Redefining XPU Memory for AI Data Centers Through Custom HBM4 – Part 1

By Archana Cheruliyil, Alphawave Semi

November 25, 2024

Part 1: An overview of HBM

This is the first of a three-part series from Alphawave Semi on HBM4 and gives an overview on the HBM standard. Part 2 will provide insights on HBM implementation challenges, and part 3 will introduce the concept of a custom HBM implementation.

Relentless Growth in Data Consumption

Recent advances in deep learning have had a transformative effect on artificial intelligence (AI) and the ever-increasing volume of data and the introduction of transformer-based models, such as GPT, has revolutionized natural language understanding and generation leading to improvements in virtual assistants, chatbots and other natural-language processing (NLP) applications. Training transformer-based models requires enormous datasets and computational resources, so while these models open new opportunities for AI-driven insights across industries, they also create challenges around data management, storage, and processing.

The Memory Wall and How to Address It

As AI models grow in both size and complexity, they generate and process increasingly massive datasets, leading to performance bottlenecks in memory systems. These memory-intensive operations strain the memory hierarchy, especially in high-throughput scenarios like training large neural networks. We see CPU processing power continue to increase, tracking with Moore’s Law, however, memory access speed has not kept up at the same rate. Specialized AI hardware, while capable of extreme parallelism, is constrained by memory latency and bandwidth. This bottleneck, often referred to as the memory wall, can significantly affect overall system performance. To address these challenges and narrow the memory-performance gap, advancements are being explored in areas like 3D stacked memory technologies commonly known as High Bandwidth Memory (HBM).

Click here to read more ...