A novel 3D buffer memory for AI and machine learning

A 3D charge-coupled device with IGZO channel: a promising CXL type-3 buffer memory

By Maarten Rosmeulen, Program Director of the Storage Memory program at imec

How AI and machine learning change the traditional compute architecture

For many decades, dynamic random-access memory (DRAM) has been the main memory in traditional Von Neumann compute architectures. Its role is temporarily storing data and program code and feeding these to the processor’s cache memories through double data rate (DDR) data buses. DRAM is byte-addressable, which means that it can address one or a few bytes at a time. One of the most critical metrics is the short latency, i.e., the ability to address the first byte within a ~50ns timeframe. This requirement is most needed for quickly retrieving program code, which typically contains branched instructions distributed randomly within the DRAM memory chip.

DRAM density could increase through technology scaling to address the growing demand for DRAM and keep pace with the performance improvement of the processor’s logic part. Unfortunately, since about 2015, DRAM cost scaling – expressed as cost per bit – has increasingly struggled to keep up with Moore’s Law.

Parallel to this evolution, data-intensive applications such as AI and machine learning are changing the Von Neumann compute architecture. Not only more but also more specialized processor cores (GPUs, TPUs, ...) now operate in parallel to perform the application-specific tasks. As these applications are extremely data-hungry, ever larger streams of data (not so much program code) go from the memory to the processors, increasing the demand for DRAM memory. New interconnect standards are being introduced to complement the parallel DDR buses to support large data transfers. One of these is the compute express link (CXL), an open, high-bandwidth processor-memory interconnect standard that allows for the more efficient use of DRAM memory. CXL supports a variety of use cases, giving rise to different types of standards, referred to as types 1, 2, and 3. The latter, also called type-3 buffer memory, can be envisioned as an off-chip pool of memories that feeds the various processor cores with large blocks of data through a high-bandwidth CXL switch.

To read the full article, click here