Ask the Experts: The State of AI

On this episode of Ask the Experts, we had the opportunity to chat with Steven Woo, Rambus Fellow & Distinguished Inventor, about the latest developments in AI and the implications for hardware and computing architecture. Specifically, he discussed the exciting new inference technique called chain-of-thought and the innovations needed to support its computational demands.

Key Topics of this episode of Ask The Experts

• What is Chain-of-Thought?
• How does Chain-of-Thought enable memory innovations?
• How can we manage the power demands of AI systems?
• Power Management ICs (PMICs) and AI
• The Future of AI

What is Chain-of-Thought?

Chain-of-thought is an improvement in the way large language models operate, where complex questions and prompts are broken down into simpler steps to achieve the answer to a more complex problem. This mimics human reasoning, where smaller steps are taken to reason through a problem. Large language models in AI are now starting to mimic this process, allowing for higher quality answers.

How does Chain-of-Thought enable memory innovations?

Chain-of-thought requires more compute power as each step in the reasoning process takes its own amount of compute capability. This approach puts more bandwidth and capacity pressure on the computing architecture. Innovations such as HBM4 with 3D stacking, multi-level signaling (PAM3) in GDDR7, and multiplexing in MRDIMM are being developed and deployed to provide more memory bandwidth and capacity.

How can we manage the power demands of AI systems?

As AI systems scale up, the requirements for greater power also rise. Cooling systems are transitioning to liquid cooling due to its higher capacity to transfer and move heat compared to air. Additionally, power delivery systems are shifting from 12-volt to 48-volt infrastructure to manage the increased current demands. Future discussions are exploring even higher voltages, such as 400-volt infrastructure.

Power Management ICs (PMICs) and AI

Changes in voltage levels require power management ICs (PMICs) to convert the voltage close to the components that consume it. This ensures high-quality power with minimal losses. PMICs are expected to support larger numbers of voltage conversions and show up in more places within the chassis.

The Future of AI

AI is a game-changing technology, bringing new use cases and applications. Large language models are improving in accuracy, and robotics are being trained in virtual worlds to navigate and perform tasks. These assistive technologies make humans more productive and open up new possibilities for further improvements in technology.

Key Quote

“Chain-of-thought is really an improvement in the way some of these large language models operate, where very large and complex questions and prompts are broken down into simpler steps first to achieve the answer to a more complex problem. This really mimics human reasoning where in order to arrive at the answer to a complex problem, we take smaller steps, and we reason through it.”