MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators
By Dong Liu 1, Yanxuan Yu 2
1 University of California, Los Angeles
2 Columbia University
Abstract
Autoregressive transformer decoding is constrained by irregular key-value (KV) cache movement on tiled accelerators. Prior compression and DRAM-placement systems still concentrate traffic on centralized memory paths that bottleneck long-context serving. We present MeshKV, a KV cache fabric that moves blocks as packetized flows over a lightweight NoC. It co-designs (i) TaKV affine striping to spread homes and cut hotspot load, (ii) Mare multicast with verified duplicate suppression, and (iii) Pad, which overlaps prefetch, tile multiply, and streaming softmax behind credit-aligned FIFOs. Together they convert bisection back-pressure into useful KV transfer. On our 8x8 FPGA implementation with LLaMA-2-7B and Mistral-7B at 8K-32K, MeshKV reduces interconnect traffic by up to 58%, improves KV bandwidth utilization by 2.1x, and delivers up to 1.9x multi-stream throughput.
Index Terms — Transformer decoding, KV cache, network-on-chip, accelerator architecture, long-context LLM, multicast routing, spatial locality, FPGA
To read the full article, click here
Related Semiconductor IP
- FlexGen Multi-Die Smart Network-on-Chip (NoC) IP
- NoC Interconnect IP Generator
- NoC Silicon IP for RISC-V based chips supporting the TileLink protocol
- NoC Verification IP
- FlexGen Smart Network-on-Chip (NoC) IP
Related Articles
- SoC design: When a network-on-chip meets cache coherency
- Tackling Network-on-Chip (NoC) Scaling Challenges with a System-technology Co-optimization Approach
- HyNoC: A Hybrid Circuit-Switch/Wormhole Network-on-Chip for Distributed VLIW Computing on FPGA
- Processor forum examines embedded cache, architectures
Latest Articles
- FlexSpIM: An Event-Based Digital Compute-In-Memory Accelerator with Flexible Operand Resolution and Layer-Wise Hybrid Stationarity
- MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators
- Analog Pin Directionality as an Exfiltration Attack Surface in Mixed-Signal ICs
- SIMT-Aware Lockstep Verification and Functional-Coverage Closure Methodology for an Open-Source RISC-V GPGPU: A UVM 1.2 Environment
- Efficient Hardware Information-Flow Tracking for Pre-Silicon Security Testing