MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators
By Dong Liu 1, Yanxuan Yu 2
1 University of California, Los Angeles
2 Columbia University

Abstract
Autoregressive transformer decoding is constrained by irregular key-value (KV) cache movement on tiled accelerators. Prior compression and DRAM-placement systems still concentrate traffic on centralized memory paths that bottleneck long-context serving. We present MeshKV, a KV cache fabric that moves blocks as packetized flows over a lightweight NoC. It co-designs (i) TaKV affine striping to spread homes and cut hotspot load, (ii) Mare multicast with verified duplicate suppression, and (iii) Pad, which overlaps prefetch, tile multiply, and streaming softmax behind credit-aligned FIFOs. Together they convert bisection back-pressure into useful KV transfer. On our 8x8 FPGA implementation with LLaMA-2-7B and Mistral-7B at 8K-32K, MeshKV reduces interconnect traffic by up to 58%, improves KV bandwidth utilization by 2.1x, and delivers up to 1.9x multi-stream throughput.
Index Terms — Transformer decoding, KV cache, network-on-chip, accelerator architecture, long-context LLM, multicast routing, spatial locality, FPGA
To read the full article, click here
Related Semiconductor IP
- FlexGen Multi-Die Smart Network-on-Chip (NoC) IP
- NoC Interconnect IP Generator
- AXI NoC Non-Coherent Mesh
- FlexGen Smart Network-on-Chip (NoC) IP
- NoC System IP
Related Articles
- SoC design: When a network-on-chip meets cache coherency
- HyNoC: A Hybrid Circuit-Switch/Wormhole Network-on-Chip for Distributed VLIW Computing on FPGA
- Processor forum examines embedded cache, architectures
- Designing a high-performance, low-cost switch fabric chip-set using COT
Latest Articles
- PVTSizing 2.0: A Fast and Reliable Analog Circuit Sizing Operator for Agentic EDA Workflows
- An Agentic Python Framework for LLM-Driven UVM Testbench Synthesis with Closed-Loop Compiler Feedback
- Reconfiguring System-Level Protection Through Bounded Delegation in RISC-V MCUs
- SkipLane: Write-Once, Conflict-Free Accumulation for Zero-Skipping CNN Engines on FPGAs
- Priority-Aware Deficit Round Robin for AXI4 Bandwidth Fairness in ASAP-7nm Process