PaperlayerView exact revision on arXiv

Source-linked AI summary

O\'ido: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural Accelerator

Daniel Varela Sánchez, Guillermo Martín-Coello Juárez

arXiv:2610.13447v1cs.SDcs.CLeess.AS

TL;DR

Low-cost microcontrollers generally cannot run phone- or server-class recognizers, leaving products to use cloud inference or fixed command lists; model size, memory bandwidth, and runtime support also constrain local speech recognition. Oído runs Conformer speech recognition with per-frame dynamic int8 quantization and on-chip decoding, alongside a chunked-attention streaming mode with cached int8 state. Int8 post-training quantization with per-frame activation scales costs 0.02 (clean) and 0.12 (other) absolute WER points; all int8 configurations make fewer errors than laptop baselines.

  • Problem

    Low-cost microcontrollers generally cannot run phone- or server-class recognizers, leaving products to use cloud inference or fixed command lists; model size, memory bandwidth, and runtime support also constrain local speech recognition.

    Source: §1 · p. 1

  • Method

    Oído runs Conformer speech recognition with per-frame dynamic int8 quantization and on-chip decoding, alongside a chunked-attention streaming mode with cached int8 state.

    Source: §1 · p. 1, §5.1 · p. 3, §5.4 · p. 3

  • Results

    Int8 post-training quantization with per-frame activation scales costs 0.02 (clean) and 0.12 (other) absolute WER points; all int8 configurations make fewer errors than laptop baselines.

    Source: §7.1 · p. 5

  • Takeaways & Limitations

    At about twice real time on one core, Oído suits short commands where several seconds of delay are acceptable, not live transcription or conversation.

    Source: §9 · p. 9

    On one core the recognizer runs at 1.97× real time, and the two-core mode gives wrong transcripts on silicon.

    Source: §8.2 · p. 8

Abstract

from arXiv · show

Speech recognition on low-cost microcontrollers is, in practice, limited to keyword spotting or a fixed list of commands. We present O\'ido, an inference engine that runs open-vocabulary English speech recognition entirely on an ESP32-S3 (240 MHz dual-core Xtensa LX7, 512 KB of SRAM, 8 MB of PSRAM, 16 MB of flash, no neural accelerator). O\'ido executes a 13M-parameter Conformer-CTC model with int8 weights, per-frame dynamic int8 activations and int8 attention on the chip's SIMD instructions, with a schedule designed around flash and PSRAM bandwidth. On LibriSpeech test-clean/test-other the on-device arithmetic reaches 3.70%/8.23% word error rate (WER), within 0.12 points of full precision, and 3.33%/7.15% with an on-chip 1.3M-parameter language model; a transducer head gives 2.99%/6.67% and an int4 variant fits in 8.3 MB. This is 2.3-2.8x fewer errors than the vendor's recognizer on the same chip, and fewer than Whisper tiny.en and Moonshine tiny in full precision on a laptop, including in real noise and reverberation. Measured on a physical board, the firmware runs at 1.97x the duration of the audio on one core, with transcripts identical to an instruction-exact emulator: accurate, but not yet real time (a 4 s command is transcribed about 8.7 s after it ends). The two-core schedule gives wrong results on silicon and is disabled. A streaming variant shows partial text while the speaker talks and delivers the final text 4.1-11.1 s after clips of 3.6-10.6 s (7.2-22.4 s in utterance mode) at 4.86%/11.03% WER; a Spanish model makes 34-60% fewer errors than multilingual Whisper tiny. Code and models are released.

1 Introduction

Low-cost microcontrollers typically offer only fixed command recognition, while running larger speech models is constrained by memory, bandwidth, and runtime support. Oído shows that an accurate Conformer model can run on a commodity microcontroller, though not yet in real time.

1 Introduction

  • Int8 inference loses 0.02 and 0.12 absolute WER points on LibriSpeech test-clean and test-other versus full precision, without retraining.The engine uses int8 inference for Conformer speech recognition on the ESP32-S3.Source: §1 · p. 1
  • The bandwidth-aware schedule reduces modelled flash and PSRAM weight traffic by 2.4× through utterance-level row blocking, K-split accumulation, and a batched front end.Kernel optimizations also remove 9% of executed instructions; weights are streamed once per 64-frame block while current activations and int32 accumulation tiles stay in SRAM.Source: §1 · p. 1, Figure 1 · p. 2
  • The on-chip decoding contributions include a recurrent language model and an RNN-T head that evaluates its joint network across several frames while remaining equivalent to greedy search.The evaluation covers seven test sets, fourteen noise and reverberation conditions, voice-command latency and memory, and comparisons with laptop-class models and the vendor recognizer.Source: §1 · p. 1
  • The streaming mode uses chunked-attention fine-tuning and per-layer int8 state caches to show partial text during speech and retain only the last partial chunk after speech stops.The engine measures the streaming trade-off at the chip’s current speed.Source: §1 · p. 1
  • The engine, firmware, tools, and models are open source.The release includes models.Source: §1 · p. 1

2 Related Work

Prior work spans compact speech models, quantization, and microcontroller deployments. Oído builds on these lines while targeting speech recognition without a neural accelerator.

2 Related Work

  • A 10.3 M-parameter Conformer reaches 2.7%/6.3% LibriSpeech with RNN-T, while Whisper tiny [35] and Moonshine tiny [23] target laptop-class devices.The paper uses NVIDIA’s Conformer-CTC Small [26], reported at 3.7%/8.1% greedy in full precision, and its transducer counterpart.Source: §2 · p. 2
  • Ding et al. [13] report native 4-bit quantization-aware training without LibriSpeech accuracy loss, while Rybakov et al. [36] report 2-bit results at 3.1%/7.0% and 10.5 MB.Oído instead uses post-training int8 with per-frame activation scales and reserves quantization-aware training for int4.Source: §2 · p. 2
  • Prior microcontroller speech-recognition demonstrations include accelerator-backed systems and Espressif’s fixed-command MultiNet7; the authors report no lower published microcontroller LibriSpeech WER.The cited systems include Cortex-M55 platforms with Ethos accelerators.Source: §2 · p. 2

3 Platform

The platform is an ESP32-S3 module with SIMD instructions and limited on-chip SRAM, backed by larger but slower external memories. The placement strategy keeps working data on-chip while storing and streaming model weights from external memory.

3 Platform

  • The ESP32-S3 has two 240 MHz Xtensa LX7 cores, a single-precision FPU, and a 128-bit SIMD extension used by the engine.The listed SIMD instructions support int8 vector multiply-accumulate operations.Source: §3 · p. 2
  • The module provides 512 KB of internal SRAM, 8 MB of octal PSRAM, and 16 MB of quad-SPI flash, with flash streaming at roughly 30–40 MB/s.Flash and PSRAM share a 64 KB data cache; PSRAM is several times faster than flash.Source: §3 · p. 2
  • A 13 M-parameter int8 model cannot fit in SRAM, only partly fits in PSRAM alongside working buffers, and fits in flash, so frequently read matrices are copied to PSRAM while per-block activations stay in SRAM.Table 1 is identified as the Conformer-CTC Small model architecture.Source: §3 · p. 2, Table 1 · p. 2

4 Model

Oído re-implements NVIDIA’s Conformer-CTC Small and checks its outputs against the released reference. The model runs in utterance mode with full attention, trading a wait until speech ends for avoiding chunked-streaming accuracy costs.

4 Model

  • Oído runs NVIDIA’s Conformer-CTC Small, re-implemented from released weights and checked against the reference implementation.Features agree within 2 × 10^-5, and frame-level argmax labels match exactly on the checked test utterances.Source: §4 · p. 2
  • Utterance mode delimits each utterance with voice activity detection, then processes all frames with full attention, matching the model’s training.This avoids the accuracy cost of chunked streaming.Source: §4 · pp. 2–3
  • In utterance mode, text appears only after the speaker stops, making latency the trade-off for avoiding chunked-streaming accuracy costs.The paper’s module includes the chip, PSRAM, flash, and antenna.Source: §4 · pp. 2–3

5 Engine

The engine combines integer inference with a bandwidth-aware schedule and on-chip decoding. A cached chunked mode supports streaming, and board behavior is checked against an instruction-exact emulator.

5.1 Quantization

  • Per-frame dynamic int8 activation scales preserve precision across speech and silence; attention uses int8 dot products, and the second convolution fuses patch quantization into its gather.LayerNorm, SiLU, GLU gating, and depthwise convolution run in fp32.Source: §5.1 · p. 3

5.2 Kernels

  • SIMD matrix products use an x-stationary kernel for K ≤ 256, row streaming for K = 704, and K-split for K > 1024; two-core execution gives wrong silicon transcripts.All reported measurements therefore use one core.Source: §5.2 · p. 3

5.3 Bandwidth-aware schedule

  • 18 MB/s to 7.5 MB/s: blocking, K-split accumulation, and batched front-end convolution reduce modeled weight streaming, at a cost of 2.9% more executed instructions.Layers process 64-frame blocks, reusing each weight row across the block.Source: §5.3 · p. 3
  • The PSRAM working set peaks at 1.9 MB for a 15-second utterance and 2.4 MB at the 20-second maximum, leaving remaining PSRAM for frequently read weights.The positional-encoding table is shared across layers, and buffers with non-overlapping lifetimes alias.Source: §5.3 · p. 3

5.4 Decoding

  • CTC decoding supports greedy argmax or GRU prefix-beam fusion; transducer search batches up to eight frames without changing greedy outputs, and int4 quantization-aware training targets flash-space savings.The transducer discards and recomputes rows after a token emission; its outputs matched sequential greedy search on LibriSpeech.Source: §5.4 · pp. 3–4, Figure 2 · p. 4

5.5 Streaming

  • Chunked streaming training varies chunk size and left context, retains full context with probability 0.3, limits convolution to the chunk, and uses fixed feature normalization plus noise and room-impulse augmentation.The clean-read-speech-only fine-tune lost much of the original model’s robustness.Source: §5.5 · p. 4
  • At C=32, streaming runs at 1.96× real time versus 2.04× for utterance mode on one core, while fixed-size caches retain prior state and leave only the last partial chunk at stop.The caches use about 2 MB in PSRAM at C=32 and L=128.Source: §5.5 · p. 4

5.6 System

  • On 27 clips, board and emulator transcripts matched, but host log-mel rounding matched the board on only 22; all accuracy numbers use the host build.The firmware captures audio into an 8-second PSRAM ring buffer and segments utterances using voice activity detection.Source: §5.6 · p. 4

6 Evaluation Methodology

The evaluation combines full LibriSpeech tests with held-out speech sets, noise and reverberation conditions, laptop baselines, and measurements on an ESP32-S3 board. The protocols distinguish host-compiled chip arithmetic, laptop subsets, and vendor-published figures.

Datasets

  • Evaluation covers LibriSpeech test-clean/test-other (2,620/2,939 utterances) plus held-out Common Voice (1,500), AMI individual headset microphones (1,000), People’s Speech (1,000), and Earnings-22 (4,179 segments).People’s Speech uses the corpus’s own noisy references.Source: §6 · pp. 4–5

Baselines

  • Laptop baselines are full precision and use up to 500 evenly spaced utterances per set; shared segmentation and text normalization apply, though some baselines may have seen held-out utterances during training.The listed laptop models include Whisper tiny.en, Moonshine tiny, and Vosk small.Source: §6 · p. 5

Robustness

  • Robustness tests mix 300 test-clean utterances with real noise at 5 or 10 dB SNR, four-talker babble, simulated reverberation, and a far-field noise-plus-reverberation condition.Simulated rooms use RT60 values of 0.4 and 0.8 seconds; the far-field condition uses 0.6-second reverberation and kitchen noise at 10 dB.Source: §6 · p. 5

Speed

  • Board speed tests use a 240 MHz ESP32-S3 with flash clips processed one after another; real-time factor includes language-model beam search but excludes audio capture.The emulator advances virtual time by one nanosecond per executed instruction and serves as a functional reference; streaming tests feed clips at speech rate and time final-text appearance.Source: §6 · p. 5

Baselines

  • Table 2 reports host-compiled int8 arithmetic for chip rows, full-precision laptop results on 500-utterance subsets, and vendor-published WER; the vendor evaluation protocol is unspecified.The table also reports model-file sizes in ESP-SR.Source: Table 2 · p. 5, §6 · p. 5

7 Results

Oído achieves strong accuracy across English domains and noise conditions, while streaming and Spanish variants extend its scope; on the ESP32-S3, however, inference remains slower than real time. The results quantify accuracy, robustness, latency, and memory on the board.

7.1 Accuracy and quantization

  • Per-frame activation scales let int8 post-training quantization cost 0.02 (clean) and 0.12 (other) absolute WER points; the GRU language model reduces LibriSpeech errors 10–13% relative, and the transducer head brings on-chip results to 2.99%/6.67%.Quantization-aware fine-tuning yields int4 4.61%/9.98% greedy; on-chip int8 models beat laptop baselines on Common Voice, VoxPopuli and AMI but trail Whisper tiny.en on Earnings-22 (20.8–22.4% vs 20.3%). The speech-transcript LM helps every set, whereas the book-only LM improves LibriSpeech, Common Voice and Earnings-22, leaves VoxPopuli unchanged, and slightly hurts AMI and People’s Speech.Source: §7.1 · p. 5

7.2 Robustness

  • 31–45% fewer errors on average across 14 noise and reverberation conditions: on-chip int8 models outperform Whisper tiny.en and Moonshine tiny.Car noise at 5 dB SNR costs less than 0.3 points, kitchen and living-room noise roughly 1–2 points, and cafeteria noise 4.7–6.0 points; four-talker babble, strong reverberation, and far-field use are hardest, where beamforming would help more than a different recognizer.Source: §7.2 · pp. 5–6, Table 4 · p. 6

7.3 Streaming

  • 4.86%/11.03% WER at C=32 with the language model, versus 3.33%/7.15% in utterance mode; chunking reduces accuracy but streaming remains more accurate than the vendor recognizer and Whisper tiny.en.Adding noise, music, babble, and simulated reverberation improves mean WER across 14 conditions from 8.9% to 6.2% in full-context mode and from 12.0% to 9.0% at C=32. Streaming final text arrives 4.1–11.1 s after 3.6–10.6 s clips, versus 7.2–22.4 s in utterance mode.Source: §7.3 · p. 6, Table 5 · p. 6

7.4 Spanish

  • 34–60% fewer errors than multilingual Whisper tiny on four Spanish sets with the language model; streaming at C=32 costs about 2 points.The comparison favors Oído because it was trained on corpus training splits while Whisper was zero-shot, and no Spanish noise benchmark is reported.Source: §7.4 · p. 7, Table 6 · p. 7

7.5 Speed and memory on the board

  • 1.97× real time for the one-core int8 CTC model with language model; measured firmware transcripts match the emulator on all 27 clips.The int4, transducer, and Spanish models run at 1.76×, 1.89×, and 2.06× real time. Int8 occupies 14.0 MB of flash, int4 8.3 MB; the maximum-length PSRAM working set is 2.4 MB and at least 125 KB of internal SRAM remains free, including 72 KB with streaming caches. Bandwidth-aware tiling cuts modeled weight traffic 2.4× but was not measured on silicon.Source: Table 7 · p. 7, §7.5 · p. 7

7.6 Latency

  • About 7 s from the end of a 3.7 s command to text in utterance mode, versus about 4 s in streaming mode; neither is conversational.Both timings include the 0.8 s end-of-speech wait; utterance-mode compute scales with clip length at the measured real-time factor.Source: §7.6 · p. 7

8 Discussion and Limitations

Oído runs a 14 MB model from flash with per-frame dynamic quantization and whole-utterance, block-wise execution, but one-core instruction issue remains the main cost. The two-core mode is not reliable on silicon, and the current one-core system is slower than real time; streaming trades accuracy for shorter waits.

8 Discussion and Limitations

  • The board’s remaining one-core cost is instruction issue; matrix kernels spend most instructions on loads, accumulator reads and loop overhead, while scheduling cuts modeled flash and PSRAM weight traffic 2.4×.Table 9 reports that an optimization cuts flash traffic from 18 to 7.5 MB/s while adding instructions.Source: §8 · pp. 7–8, Table 9 · p. 8

8.1 The two-core mode

  • At 240 MHz, two-core execution corrupts activations and sometimes crashes, so the measured system uses one core; the cause remains unknown.Parallelizing K=704 products corrupts, and even the incorrect two-core build runs at 0.78 times the one-core time rather than the instruction-count prediction of 0.55 because of memory contention.Source: §8.1 · p. 8

8.2 Limitations

  • On one core, inference runs at 1.97× real time, so a 4 s command appears about 8.7 s after it ends in utterance mode.The quoted utterance-mode delay includes a 0.8 s end-of-speech wait.Source: §8.2 · p. 8
  • Streaming loses about 1.5 test-clean WER points versus utterance mode, while final text still arrives roughly (RTF−1) T after an utterance of length T.At the chip’s present speed, streaming buys a factor of two in latency, not more.Source: §8.2 · p. 8
  • The board study used one board and short clips; accuracy came from the host build, while board transcripts matched the emulator and the host differed on some hard clips.The RTF figures cover 27 CTC clips and 10–12 clips for other models, all at most 10.6 s; the emulator estimates the 20 s utterance’s 2.4 MB PSRAM peak.Source: §8.2 · p. 9
  • The evaluation covers English and Spanish; other languages require a new vocabulary and fine-tuning, and transducer weights are not redistributed.The transducer weights are distributed under NVIDIA’s NGC terms.Source: §8.2 · p. 9
  • Laptop baselines use 500-utterance subsets, MultiNet7 results are vendor-reported, and same-model deployments on TensorFlow Lite Micro or ESP-DL were not compared..Source: §8.2 · p. 9

9 Conclusion

Oído runs open-vocabulary English speech recognition on a $5 microcontroller without a neural accelerator. Its LibriSpeech accuracy is close to full precision, but on one core it remains slower than real time.

9 Conclusion

  • Oído runs open-vocabulary English recognition on a $5 microcontroller without an accelerator, reaching 3.70%/8.23% LibriSpeech WER and 2.99%/6.67% with a transducer head.The 3.70%/8.23% result is within 0.12 points of full precision; the one-core board runs about twice real time, limiting it to short commands where several seconds of delay are acceptable.Source: §9 · p. 9
Loading 2610.13447v1…