Audio AI at the Edge: When the Tiniest Devices Learn to Listen

A field guide to the applications, constraints, and design choices behind on-device audio intelligence – and a sneak peek at a whitepaper we’ve been writing.

Over the last three posts, we went deep, really deep, into a single problem: making speech clearer with Environmental Noise Cancellation, from the basics, through the classic signal-processing toolbox, and into the deep-learning era. That series was a microscope. This post is the opposite, a wide-angle lens on the whole world of Audio AI now living on small, power-starved devices, and a preview of the whitepaper we’ve been writing, A Glimpse into Audio AI on the Edge. Consider it the trailer.

Noise cancellation, it turns out, is just one act in a much bigger show. Your earbuds wake to your voice, your doorbell flags a smashing window, your hearing aid pulls a friend out of a crowded room, a factory sensor catches a failing bearing, all without a round trip to the cloud. Let’s map that landscape: what Audio AI does at the edge, what makes the edge so unforgiving, and what you have to weigh before you ship.

And the timing is no accident. Edge AI is already a ~$25–40 billion market in 2025, on track to pass $100 billion within a decade. The sub-1-watt hardware tier, Ceva’s home turf, is a fast-growing slice of it. The accelerant is generative AI moving on-device: small language models, voice agents, and conversational assistants that used to live in the cloud are migrating onto the device in your hand and the bud in your ear, and audio is the most natural interface for all of it.

At Ceva, this is the design space we work in every day: helping teams bring audio, sensing, and AI workloads onto constrained devices while balancing user experience, power efficiency, and product reliability.

First, what do we mean by “the edge”?

“Edge” has become an elastic word. I’m not talking about a laptop with a beefy GPU or a server in a closet; I mean the genuinely constrained end: microcontrollers (MCUs) with kilobytes of RAM, audio DSPs, and dedicated neural processing units (NPUs) sipping a fraction of a watt.

That’s a wide continuum on its own:

Figure 1: “Edge” spans a whole continuum. The further left you sit, the more every kilobyte and microjoule is a decision, and that constraint is the whole story.

Why move audio AI on-device at all?

If the cloud is cheap and powerful, why bother squeezing models onto a chip the size of a fingernail? Four reasons, and most real products care about all of them:

Latency. A wake word that lags by half a second feels broken, and conversational audio has a tight delay budget: ITU-T G.114 caps the comfortable one-way limit near 150 ms end-to-end, leaving an on-device enhancement block only tens of milliseconds of its own. The cloud can’t reliably promise that.

Privacy. Audio is intimate: it can carry your identity, your mood, even hints about your health. Keep it on the device and the raw signal never leaves your pocket. For voice biometrics and medical use, that’s not a nice-to-have; it’s the requirement.

Energy and cost. Streaming audio to a server, 24/7, burns radio power and racks up bills. On-device inference avoids both.

Reliability. A safety device that goes deaf when the Wi-Fi drops isn’t a safety device. The edge keeps working offline.

Put those four together and you get the things that actually move product: longer battery life, a leaner bill of materials, privacy you can print on the box, and a faster path to market.

Earbuds are where all four constraints collide: latency you can hear, audio too personal to stream, a battery measured in hours, and no guarantee of a connection.

The expanding menu: what Audio AI actually does at the edge

The ENC series covered one task; the whitepaper covers a whole table-full. They sort into five families, and you’ve already met them all: the always-on wake word in your earbuds, the noise cancellation in your hearables, the speech recognition that turns talk into text, the speaker ID and glass-break detection guarding a smart home, the intent engine and offline voice in your car, and an emerging frontier of emotion-aware interfaces and on-device music. Figure 2 lays them out.

Figure 2: The application landscape, in five families. What ties them together is one stubborn thread: every task was born in the cloud, on big models, and now has to earn its place on hardware a thousand times smaller.

Why the edge is hard: living within kilobytes and milliwatts

Here’s where it gets interesting. A model that scores beautifully on a benchmark laptop can be undeployable on the device you’re actually targeting. The constraints stack up:

• Memory and model size. A hearing-aid or MCU budget can be a few hundred kB total, for the model and the working buffers. That rules out most off-the-shelf architectures.
• A stark example: a Keyword Transformer needs ~5.4M parameters; a broadcasted-residual BC-ResNet hits comparable ~98% accuracy on Speech Commands with ~320k parameters and ~89M MACs. Same task, an order-of-magnitude different bill.
• Power and thermal. “Always-on” means the energy-per-inference is multiplied by all the time. Wearables also can’t get warm. Both push you toward the smallest model that’s good enough, never the most accurate one.
• Real-time, streaming latency. Edge audio is a stream, not a file. Models have to process it frame by frame within a hard latency budget, with no peeking at the future.

Robustness in the wild: on a factory floor, the acoustic signature of a failing bearing has to cut through noise no lab dataset fully captures.

And yet it works, often spectacularly. To make the point concrete: the standard MLPerf Tiny keyword-spotting model fits in about 50 kB, and INT8 quantization shrinks a model roughly 4× (~75%) for typically under a point of accuracy loss; speaker verification has been quantized to under 1 MB while keeping equal-error-rate under 2% on the standard VoxCeleb1 test; non-autoregressive on-device TTS such as FastSpeech 2 reaches a mean opinion score near natural speech (≈3.8 on LJSpeech); and (the eyebrow-raiser) at ISSCC 2024 CEA-Leti showed keyword spotting under one microwatt (a ~0.99 µW front-end, 91% on a 10-word task). The edge is small, but it is not weak.

Deploying audio AI on the edge: what you actually have to decide

You’ve picked a task and a target device. What are the real decisions? The short version of the playbook:

1. Choose the model family for the budget, not the leaderboard. Depthwise-separable CNNs (DS-CNN), temporal convolutions (TC-ResNet), and broadcasted-residual networks (BC-ResNet) trade a sliver of accuracy for an order-of-magnitude in efficiency. Start from the constraint and work backward.
2. Compress deliberately. Quantization (ideally quantization-aware training, not just post-hoc), pruning, and knowledge distillation are the most impactful levers we have: across every task we surveyed, QAT and distillation preserve quality while collapsing footprint.
3. Co-design the model and the hardware. The best results come from optimizing them together: neural architecture search constrained by latency and energy, mixed-precision quantization guided by sensitivity, operator fusion in the compiler. Hardware-aware co-design has reported energy-efficiency gains upward of 40% (over 50% in some studies) with little accuracy cost.
4. Evaluate beyond accuracy. The one teams skip, and the one the whitepaper argues hardest. PESQ, WER, and EER say nothing about whether the model fits or how long the battery lasts. You also have to report latency (ms/frame), energy per inference (µJ), memory footprint, real-time factor, and ideally the full energy–accuracy Pareto curve.
5. Mind the benchmarking gap. Initiatives like MLPerf Tiny, SUPERB, and HEAR are great starts, but most benchmarks still emphasize accuracy alone, and audio-centric multi-task edge evaluation barely exists yet. Until that’s standardized, comparing two “state-of-the-art” edge models is harder than it should be.

Figure 3 (illustrative): On the edge you choose a point along the curve, not the top of it. The most accurate model often lives past the power budget; the deployable one sits on the knee. Schematic, not measured data.

That last point is close to my heart; it’s exactly the kind of problem we’re tackling in the Audio AI working group at the Edge AI Foundation (formerly the tinyML Foundation), where standardizing how we report edge performance is a recurring theme.

The part we can’t outsource: privacy, bias, and trust

Running on-device is a privacy advantage, not a free pass. Voiceprints still need protecting: secure enclaves, on-device encryption, and federated learning all help, though each costs compute that fights your efficiency budget. Datasets skew heavily toward English, Mandarin, and clean studio audio, so models can generalize poorly to other languages, accents, and noisy conditions, and compression can quietly amplify that bias. And as voice cloning gets cheaper, provenance and watermarking move from academic curiosities to product requirements. Responsible Audio AI isn’t a separate chapter; it’s a design constraint riding along with all the others.

---

Explore Ceva IP:

• AI SDK for Ceva-NeuPro NPUs
• Vision AI DSP
• NPU IP for Embedded ML
• Scalable Edge NPU IP for Generative AI

---

Where Ceva comes in

This is the world we build for. The trade-offs above (accuracy versus footprint, latency versus power, capability versus battery) are exactly what our edge audio and AI portfolio is designed to make tractable. Ceva-NeuPro-Nano, which took the Artificial Intelligence award at embedded world 2026, targets the embedded-ML and TinyML end of the spectrum; Ceva-NeuPro-M scales up to heavier and generative workloads at up to 350 TOPS/W; Ceva-ClearVox delivers the noise-cancellation payoff we spent three posts on; and Ceva-NeuPro-Studio plus our audio DSPs handle the unglamorous-but-essential job of getting a trained model to run efficiently on real silicon. It’s resonating in the market: ten customers licensed NeuPro IP in 2025 across consumer, industrial, automotive, infrastructure, and PC. The point isn’t any single block; it’s having the right tool for wherever you land on that MCU-to-NPU continuum.

From waveform to silicon: the whole journey, from captured audio to a model running efficiently on-device, is what the portfolio is built around.

Coming soon: the whitepaper (and a quieter, smarter edge)

If this whetted your appetite, the whitepaper goes much deeper: a task-by-task tour across wake words, enhancement, ASR, biometrics, language understanding, and synthesis, with the datasets, architectures, metrics, hardware tiers, and open challenges behind each.

We started this journey trying to make a single voice clearer in a noisy room. We’re ending it somewhere much bigger: an entire generation of devices that listen, understand, and respond – locally, privately, and on a power budget you can measure in microwatts. The cloud taught machines to hear. The edge is teaching them to listen everywhere.

The whitepaper, A Glimpse into Audio AI on the Edge, lands soon. Pre-register to get it the day it drops. Or,  if your team is already looking at Audio AI for a battery-powered, always-on, or privacy-sensitive design, talk to Ceva’s experts about the best path from model to silicon.

Catch up on the ENC series:

• Part 1: Enhancing Audio Quality with Environmental Noise Cancellation in Sound Processing
• Part 2: Environmental Noise Cancellation (ENC): Noise Types and Classic Methods for Speech Enhancement
• Part 3: AI Audio for Voice Enhancement: Deep into the Deep

Sources & further reading

• Edge AI market sizing: 2025 market estimates and long-range forecasts from Grand View Research, Precedence Research, and Fortune Business Insights.
• MCU model size & power budget: C. Banbury et al., MLPerf Tiny Benchmark (2021): the reference keyword-spotting model is a 52.5 kB DS-CNN, and target devices run at 10–250 MHz under 50 mW. arXiv:2106.07597
• Keyword spotting on microcontrollers / INT8 4× compression: Y. Zhang et al., Hello Edge: Keyword Spotting on Microcontrollers (2017). arXiv:1711.07128
• Keyword Transformer (~5.4M parameters): A. Berg et al., Keyword Transformer: A Self-Attention Model for Keyword Spotting, Interspeech 2021. arXiv:2104.00769
• BC-ResNet (~321k parameters, 89.1M MACs): B. Kim et al., Broadcasted Residual Learning for Efficient Keyword Spotting, Interspeech 2021. arXiv:2106.04140. See also the survey: I. López-Espejo et al., Deep Spoken Keyword Spotting: An Overview (2021), arXiv:2111.10592.
• Sub-1 MB speaker verification: B. Liu et al., Extremely Low Bit Quantization for Mobile Speaker Verification Systems Under 1MB Memory, Interspeech 2023 (binarized ResNet34 “b-vector”: 0.97 MB, ~27× compression, 1.72% EER on the VoxCeleb1 original test set). ISCA Archive
• On-device TTS quality: Y. Ren et al., FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2021), MOS 3.83 on LJSpeech. Microsoft Research. On-device example: Apple, Advancing Speech Accessibility with Personal Voice (2023), Apple ML Research.
• Sub-microwatt keyword spotting: CEA-Leti at ISSCC 2024, “0.4 V 988 nW Time-Domain Audio Feature Extraction for Keyword Spotting Using Injection-Locked Oscillators” (91% on 10 words). Coverage
• Conversational latency budget: ITU-T Recommendation G.114, One-way transmission time (≤150 ms one-way for transparent interactivity). ITU-T G.114
• Hardware-aware co-design gains: H. Bouzidi et al., HADAS: Hardware-Aware Dynamic Neural Architecture Search for Edge Performance Scaling (2022), up to 57% energy-efficiency gains. arXiv:2212.03354
• Edge NPU efficiency: Ceva-NeuPro-M (up to 350 TOPS/W) and Ceva-NeuPro-Nano (10–200 GOPS/core).
• Benchmarks: SUPERB (S. Yang et al., Interspeech 2021); HEAR (J. Turian et al., NeurIPS 2021); MLPerf Tiny (above).
• Edge AI Foundation (formerly the tinyML Foundation): org