Source-linked AI summary
Orukeet: Multilingual ASR with Frozen Gabor Kernels
Nathan Roll, Irene Yi, Büşra Marşan, Vianney Grenez, Gabriel Stein, Momcilo Mrkaic, Pavle Padjin, Vladimir Zeljkovic, Calbert Graham
TL;DR
Multilingual ASR must handle variation across languages, accents, and domains while preserving an efficient recognizer design. Orukeet addresses this by fitting and freezing Gabor replacements for half of an adapted Parakeet encoder’s temporal filters, then training the remaining parameters. Across FLEURS, pooled WER falls from 11.01% to 9.85%, and Orukeet improves most evaluated partitions.
Problem
Multilingual ASR evaluation must cover variation across 25 languages, accents, and domains.
Method
Orukeet fits Gabor functions to learned temporal kernels, fixes the 12,288 selected replacements, and trains the remaining recognizer parameters.
Results
Pooled FLEURS WER falls from 11.01% for Parakeet to 9.85% for Orukeet, with improvements on 23 of 25 languages.
Takeaways & Limitations
The fitted kernels are materialized as ordinary convolution weights, preserving Parakeet’s tensor shapes and inference operators.
Abstract
from arXiv · showhide
Orukeet replaces half of an adapted Parakeet encoder's temporal filters with 12,288 fitted Gabor kernels, freezes these replacements, and trains the remaining parameters on multilingual and multi-accent data. Final adaptation and checkpoint selection use LibriSpeech test-other. Across 20,146 FLEURS recordings in 25 languages, pooled word error rate (WER) falls from Parakeet's 11.01% to Orukeet's 9.85%, a 10.6% relative reduction. Orukeet has lower WER on 23 of the 25 languages. Orukeet outperforms Parakeet on 61 out of 74 tested splits, including LibriSpeech test-clean (1.46% vs. 1.53% WER), test-other (2.86% vs. 3.14%), and FLEURS English (3.82% vs. 4.28%). All comparisons decode the same audio with matched NeMo settings. The fitted kernels are stored as ordinary convolution weights, retaining Parakeet's architecture and inference operators.
1 A fixed structure inside a learned recognizer
Orukeet inserts a fixed structure into an adapted Parakeet recognizer by replacing half of its temporal filters with fitted Gabor functions while training the remaining network.
- 1 A fixed structure inside a learned recognizer: Orukeet replaces the closest half of Parakeet’s 24-block temporal depthwise kernels with fitted Gabor functions and freezes those replacements during training.Each block contains 1,024 nine-tap temporal depthwise kernels; the remaining recognizer parameters are trained.
- 1 A fixed structure inside a learned recognizer: The method selects learned filters whose Gabor fits keep the initial change small, then compares the final checkpoint with stock Parakeet across speech conditions.The comparison covers read speech, accents, and domains.
2 Fitting, freezing and adaptation
Orukeet globally ranks fitted Gabor replacements, fixes the 12,288 best fits, and adapts the remaining network through recovery, distillation, and final LibriSpeech test-other selection.
- 2 Fitting, freezing and adaptation: Each nine-tap kernel is fitted at t ∈ {−4, . . . , 4} by ranking normalized squared error, using linear least squares for amplitude and phase.The search evaluates initial parameter combinations and refines the strongest candidates in float64.
- 2 Fitting, freezing and adaptation: 12,288 lowest-error Gabor fits are selected globally and fixed, with 175–748 replacements per layer and 6.32% median relative RMS error.The selected fits’ pooled squared error equals 0.4244% of their original weight energy.
- 2 Fitting, freezing and adaptation: Recovery trains unfixed parameters with transducer loss, teacher matching, and distillation before AdamW adaptation and a final 168-update pass.The final pass uses LibriSpeech test-other for checkpoint selection.
- 2 Fitting, freezing and adaptation: The exported checkpoint verifies that all 12,288 fitted kernels remain byte-identical to their fitted functions while the remaining parameter tensors change.Tokenizer assets, signal-processing buffers, and batch-normalization statistics also remain unchanged.
3 Recognition across languages, accents and domains
Matched decoding shows Orukeet improving Parakeet across multilingual read speech and accent/domain evaluations, with pooled WER reductions and wins across most partitions.
- 3 Recognition across languages, accents and domains: Both models decode identical audio with matched NeMo greedy-batch TDT settings, FP32 weights, BF16 CUDA autocast, and identical batching.Empty transcripts remain included in the scores.
- 3 Recognition across languages, accents and domains: Pooled WER counts substitutions, deletions, and insertions using normalized reference-word counts, while language macros weight each language equally.Independent full-partition rescoring reproduces every reported value.
- 3 Recognition across languages, accents and domains: WER improves on 25 of 27 read-speech partitions, including 23 of 25 FLEURS languages, with pooled FLEURS WER falling from 11.01% to 9.85%.The corresponding language macro falls from 11.07% to 9.96%.
- 3 Recognition across languages, accents and domains: Pooled accent/domain WER falls from 16.72% to 15.25%, with Orukeet improving 36 of 47 partitions and all 20 English partitions.For 5,120 English recordings, WER falls from 9.51% to 8.84%.
4 Checkpoint and reproduction
Orukeet is distributed as a reproducible NeMo checkpoint that preserves Parakeet’s tensor shapes and inference operators, with accompanying weights, fits, audits, and evaluation records.
- 4 Checkpoint and reproduction: Inference uses ordinary depthwise convolution with the same tensor shapes and operator counts as Parakeet.The exported model stores materialized convolution weights while retaining fitting parameters in the repository.
- 4 Checkpoint and reproduction: The repository and release checkpoint include source weights, training recipes, fitted functions, export audits, hashes, and reproducible evaluation records.Metric records retain unrounded scores, per-record edit counts, manifest identities, and checkpoint hashes.
- 4 Checkpoint and reproduction: Table 1 covers complete LibriSpeech and FLEURS test partitions, with pooled FLEURS scores summing errors and reference words across all 25 languages.Its macro row weights languages equally.