Partitioning Strategies to Optimize AI Inference for Multi-Core Platforms

By _Ceva's Experts blog

January 29, 2024

Not so long ago, AI inference at the edge was a novelty easily supported by a single NPU IP accelerator embedded in the edge device. Expectations have accelerated rapidly since then. Now we want embedded AI inference to handle multiple cameras, complex scene segmentation, voice recognition with intelligent noise suppression, fusion between multiple sensors, and now very large and complex generative AI models. Such applications can deliver acceptable throughput for edge products only when run on multi-core AI processors. NPU IP accelerators are already available to meet this need, extending to 8 or more parallel cores and able to handle multiple inference tasks in parallel. But how should you partition expected AI inference workloads for your product to take maximum advantage of all that horsepower? That’s the subject of this article.

Six paths to exploit parallelism for AI inference

As in any parallelism problem, we start with a defined set of resources for our AI inference objective: some number of available accelerators with local L1 cache, shared L2 cache and a DDR interface, each with defined buffer sizes. The task is then to map the network graphs required by the application to that structure, optimizing between total throughput and resource utilization.

One obvious strategy is in processing large input images which must be split into multiple tiles – partitioning by input map where each engine is allocated a tile. Here, multiple engines search the input map in parallel looking for the same feature. Conversely you can partition by output map – the same tile is fed into multiple engines in parallel, and you use the same model but different weights to detect different features in the input image at the same time.

Click here to read more ...