LLM Inference on RISC-V Embedded CPUs
By Yueh-Feng Lee, Andes Technology
The advancement of large language models (LLMs) has significantly enhanced natural language processing capabilities, enabling complex text understanding and generation tasks. This presentation focuses on optimizing the open-source LLaMA CPP project for the RISC-V P extension. By running the TinyLLaMA 1.1B model on the Andes Voyager development board using a quad-core CPU supporting the RISC-V P extension, performance results show that the model can achieve near real-time response. This work highlights the potential of RISC-V as an efficient platform for deploying advanced AI models in resource-constrained environments, contributing to the growing field of edge computing and embedded AI applications.
Related Semiconductor IP
- RISC-V Debug & Trace IP
- RISC-V IOPMP IP
- Gen#2 of 64-bit RISC-V core with out-of-order pipeline based complex
- 64-bit RISC-V core with in-order single issue pipeline. Tiny Linux-capable processor for IoT applications.
- Multi-core capable RISC-V processor with vector extensions
Related Videos
- MIPS on the RISC-V Shift: ‘Physical AI Is Agentic AI at the Edge’
- Evaluating Asynchronous Machine Learning and Scientific Computing Applications on RISC-V, Arm and X86
- RISC-V at NVIDIA: One Architecture, Dozens of Applications, Billions of Processors
- LLMs On The Edge
Latest Videos
- The Future of AI Scale-Up: Latest Advancements in UALink Technology
- Evaluating Asynchronous Machine Learning and Scientific Computing Applications on RISC-V, Arm and X86
- When AI Meets Hardware Security
- Sameer Wasson: AI Is Flipping Chip Design Upside Down
- Interview with GlobalFoundries VP at MIPS ‘Physical AI is Agentic AI at the Edge’ Taipei Event