Listening at the Speed of Memory: Live Speech Recognition on Chimera

By Jacques van Rhyn and Shayan Yassami

November 10, 2026

How fast can a small chip transcribe live speech? For Moonshine v2 on a Chimera QC-Nano core, memory sets the limit, and our implementation reaches 99% of it.

Live captions are only useful if they keep up with the person talking. On smart glasses, a hearing aid or in a car's cabin, that means transcribing speech as it arrives, on a small processor with a modest memory system.

Moonshine v2 is a speech recognition model built for exactly that. We brought it up on one Chimera QC-Nano core, the smallest core in our GPNPU family, and measured it in our instruction-set simulator (ISS).

Moonshine works in two steps. First it listens: the encoder turns every 80 ms of audio into a summary of what was heard. Then it writes: the decoder produces the transcript one piece of a word at a time. On QC-Nano, listening to 80 ms of audio takes 3.36 ms and writing each piece takes 7.11 ms. Both steps spend almost all their time reading the model from memory, and both run at 99% of the fastest that memory allows. Starting from techniques proven in our Kernel Arena, we optimized the code in about two and a half hours—stopping only when we hit the memory bandwidth roofline.

A Speech Model That Streams

Moonshine v2 comes from Moonshine AI, formerly Useful Sensors, and its weights are released under the MIT license. Most speech recognizers use a full-attention encoder: every audio frame attends to every other frame. Whisper, which we brought up on Chimera earlier this year, works on 30-second windows. Models like Whisper are accurate, but their encoder has to wait for the whole utterance before the decoder can write its first word, so the wait grows with the length of the utterance.

Moonshine v2 bounds attention with a sliding window. In each of its six encoder layers, a frame looks at the 15 frames before it and at most 3 after it. A new 80 ms chunk of audio therefore produces new encoder output without revisiting the past, and the decoder can start while the speaker is still talking. The authors report accuracy on par with models six times its size. We used the Tiny streaming checkpoint, with 44 million parameters.

That is what makes live captions possible. Our demo updates the transcript every 400 ms while audio arrives: "Yeah" became "Okay.", then "Where is it?", "Where is my brother?" and finally "Where is my brother now?" Once two consecutive updates agree on a word and it is at least 3 seconds old, the word is locked in, so later updates rewrite only the end of the sentence. The demo streams short utterances, up to five seconds of audio and 30 tokens of transcript. Captioning a whole conversation will also need the encoder output that the decoder reads to be segmented and reset as it grows, which the demo doesn't do yet.

Streaming must not change the answer. Between chunks, our implementation carries a small state: the tails of two convolutions and one window of history per layer, with 240 ms of lookahead in total. Chunk by chunk, it reproduces the encoder run on the whole utterance at once, to within 0.000006. Every call on the ISS is checked against a CPU reference on the same inputs, and the final transcript matches the floating-point PyTorch model token for token. Our PyTorch implementation also matches Hugging Face Transformers' own Moonshine v2 word for word on 29 of 31 public utterances; the other two differ by one punctuation mark where the model's top two choices nearly tie. These are reproducible checks on one speaker's public recordings, not a full accuracy benchmark.

The encoder runs once per 80 ms chunk and hands its state to the next chunk. The decoder runs once per token. Both stream their weights through a ring buffer in on-chip memory.

What the Roofline Means

Every call to a neural network has two costs: the arithmetic it does and the bytes it reads. It can finish no faster than the slower of the two. That lower bound is the roofline.

For Moonshine on a small core, the bytes win by a wide margin. The encoder reads 21.4 MB of 16-bit weights per 80 ms chunk to do about 80 million multiply-accumulates (MACs). The decoder reads 45.1 MB per token, its weights plus the attention memory, for about 22 million. That is roughly 4 and 0.5 MACs per byte. This QC-Nano configuration can do about 40 MACs in the time it takes to bring in one byte at 6.4 GB/s.

So both programs are bound by memory bandwidth, by a factor of ten or more, and the roofline is simply the bytes divided by the bandwidth: 3.34 ms for the encoder and 7.05 ms for the decoder.

Measured time per call against the time needed to read that call's bytes at 6.4 GB/s.

In practice, the encoder runs about 24 times faster than real time. With live updates on, the whole demo utterance takes 29 encoder calls and 30 decoder calls and keeps the core busy for about 15% of the audio's duration.

We also checked the result from the other direction. Halving the bandwidth to 3.2 GB/s makes both programs 1.99× slower, and the bytes per call implied by the new times match the bytes each call has to read. Arithmetic, control or synchronization limits would not scale that cleanly.

How We Got to 99%

Being memory-bound is common for edge models; running at the bound is not. Every moment the memory bus sits idle is lost for good, and a transformer gives it plenty of reasons to wait. Layer norms, attention, residual adds and state updates all sit between the weight-heavy projections. Run them one after another with the weight reads, and the bus waits while they compute.

We didn't start from scratch. Kernel Arena doubles as a database of optimizations: every leading kernel is a technique already proven on Chimera. We began with four Arena leader kernels and the ops from our first Moonshine bring-up, and the key technique, streaming weights through a ring buffer in on-chip memory, came straight from the Arena. The first full run on the ISS came 39 minutes after we switched to Moonshine v2. The decoder reached 99% of its roofline at 2 hours 1 minute and the encoder 99.4% at 2 hours 28 minutes.

We built both programs so the bus never waits:

• One kernel per program. The encoder and the decoder each compile into a single fused kernel, written with the Chimera Compute Library (CCL). We composed them in ChiPy, the Chimera SDK's Python interface for composing neural-network modules, which also runs the same operations on the CPU as a reference.
• Streamed weights. Every weight tile streams from external memory through a ring buffer in on-chip memory, in the order the computation uses it. Activations stay on chip, so normalization, attention and state updates run while the next weights are already arriving.
• No small reads stuck in the queue. Memory transfers complete in the order they are issued, so a small read issued behind a megabyte of weights waits for all of it. We load the carried state and inputs before the weight stream starts, which saves 415,000 cycles, more than 10%, per chunk.

How It Compares

Synaptics publishes figures for the same Moonshine checkpoint on its Astra SL2610, an edge AI processor with a Torq NPU. Like our QC-Nano configuration, the SL2610's Torq NPU is a 1 TOPS engine with 512 MACs, so the two are in the same compute class. The fastest memory the SL2610 supports, 16-bit LPDDR4x-3200, peaks at 6.4 GB/s according to its datasheet: the same bandwidth as our QC-Nano configuration. Synaptics doesn't say which memory its benchmark used, so 6.4 GB/s is the most bandwidth that benchmark could have had.

Sources: SL2610 times from Synaptics' Hugging Face model card (Torq v2.1.0, mean values, measured on hardware); the encoder time is derived from the reported real-time factor of 0.38. Quadric times are measured in our simulator. SL2610 NPU rating (up to 1 TOPS, 512 MACs) from the overview and section 2.3 of the Astra SL2610 datasheet; memory bandwidth from section 2.6: up to 6.4 GB/s, the peak of its fastest supported memory (16-bit LPDDR4x-3200). The memory used for Synaptics' benchmark is not stated.

Synaptics' model card lists an encoder real-time factor of 0.38, which works out to about 30 ms per 80 ms chunk, and a decoder speed of 36.6 tokens per second, about 27 ms per token. Neither implementation quantizes the model to integers. On those figures, QC-Nano runs the encoder about 9 times faster and the decoder about 3.8 times faster. The two sets of numbers aren't measured the same way: Synaptics' come from its hardware and ours from our simulator, and the two may not count the same work in a call. Our bandwidth-halving check shows that our implementation is limited by memory; the model card doesn't say what limits the SL2610's.

Measurement note: Moonshine Streaming Tiny on one Chimera QC-Nano core, 8 MACs per processing element, 4 MB of on-chip memory, 1.0 GHz and 6.4 GB/s of external memory bandwidth, measured in the instruction-set simulator (ISS). Weights are FP16. Times are steady-state device cycles per call; host audio chunking, token selection and text decoding are outside them. The demo streams utterances of up to five seconds of audio and 30 decoded tokens. The roofline is the bytes each call reads from external memory divided by 6.4 GB/s.

Listening at the Limit

At the roofline, the remaining lever is the number of bytes. Fewer bytes per weight or more memory bandwidth both translate directly into speed, because nothing else is in the way.

Speech joins the perception, driving and game-playing models we have brought to Chimera this fall. Each one has a different shape, and each one runs on the same programmable processor. Moonshine v2 shows that it can run them efficiently, too: at the limit of the memory system, on our smallest core.

The ring buffer behind that result came from the Arena, and it is going back into the Chimera SDK. That is the flywheel: the Arena finds a technique, the SDK gets it, and every customer model benefits.

The Moonshine v2 demo ships in Chimera SDK 26.11, including the implementation, notebook and ISS workflow. Contact Quadric to discuss your next speech workload.

Explore Quadric IP:

• GPNPU Processor IP - 32 to 864TOPs
• GPNPU Processor IP - 16 to 108 TOPs
• GPNPU Processor IP - 4 to 28 TOPs