GenAI v1-Q launched with 4 bits Quantization support to accelerate larger LLMs at the Edge

The new version brings a 276% speed increase for the top LLMs in low-cost systems, while maintaining their intelligence.

Spain, September 24, 2024 -- Raiderchip has presented a new HW accelerator product adding 4-bits and 5-bits Quantization support (Q4_K and Q5_K) to the extraordinary efficiency of the base GenAI v1. The new Generative AI LLM hardware accelerator runs inside FPGA devices, and is ideal for boosting their capabilities with low-cost DDR and LPDDR memories, incrementing inference speed by 276%.

^{Click to enlarge}

GenAI v1-Q running the Llama 2-7B LLM model with 4 bits Quantization on a low-cost Versal FPGA with LPDDR4 memory

The new acceleration engine increases not only inference speed but also lowers memory requirements by up to 75%, allowing the largest and most intelligent LLM models to fit into smaller systems, lowering the overall cost, while keeping real-time speed, and also reducing energy consumption. All of this with minimal impact on model accuracy and intelligence perception.

The GenAI v1-Q, which like its predecessor is already available for a wide range of FPGAs, aims to expand the range of available features. In the words of its CTO, Victor Lopez, ‘We seek to offer maximum flexibility to our customers, with highly configurable hardware that allows them to balance criteria such as accuracy, inference speed, model size, unit cost of hardware, or energy consumption goals according to their needs, finding the perfect balance that best fits their objectives.’

The current demonstrator accelerates the 4 bits quantized Meta’s Llama 2-7B using barely 4 GB of memory, whereas the vanilla version requires 16 GB of DDR.

Companies interested in trying the GenAI v1-Q may reach out to Raiderchip for access to our demo or a consultation on how our IP cores can accelerate their AI workloads.

More information at https://raiderchip.ai/technology/hardware-ai-accelerators