PaperlayerView exact revision on arXiv

Source-linked AI summary

Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation

Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, Xiaobin Hu, Shuicheng Yan, Chunyan Miao

arXiv:2605.19833v1cs.SDcs.AIcs.CLcs.MMeess.AS

TL;DR

Robust ASR still struggles with severe, compositional real-world acoustic distortions. Mega-ASR combines scalable compound-data construction with progressive acoustic-to-semantic optimization, achieving state-of-the-art robustness and over 30% relative WER reductions on challenging scenarios.

  • Problem

    Existing robust ASR datasets mainly cover isolated acoustic conditions, leaving recognition under mixed real-world effects insufficiently addressed.

  • Method

    Mega-ASR combines Voices-in-the-Wild-2M with progressive acoustic-to-semantic fine-tuning and dual-granularity WER-gated policy optimization.

  • Results

    Mega-ASR achieves the best overall robustness across CHiME-4, VOiCES, and NOIZEUS, with average WER 6.70 versus Qwen3-ASR’s 7.93.

  • Takeaways & Limitations

    Relative WER reductions exceeding 30% on challenging real-world acoustic conditions support Mega-ASR as a scalable paradigm for robust in-the-wild ASR.

Abstract

from arXiv · show

Despite rapid advances in automatic speech recognition (ASR) and large audio-language models, robust recognition in real-world environments remains limited by an "acoustic robustness bottleneck": models often lose acoustic grounding and produce omissions or hallucinations under severe, compositional distortions. We propose Mega-ASR, a unified ASR-in-the-wild framework that combines scalable compound-data construction with progressive acoustic-to-semantic optimization. We introduce Voices-in-the-Wild-2M, covering 7 classic acoustic phenomena and 54 physically plausible compound scenarios, and train Mega-ASR with Acoustic-to-Semantic Progressive Supervised Fine-Tuning and Dual-Granularity WER-Gated Policy Optimization. Extensive experiments demonstrate that Mega-ASR achieves significant advantages over prior state-of-the-art systems on adverse-condition ASR benchmarks (45.69% vs. 54.01% on VOiCES R4-B-F, and 21.49% vs. 29.34% on NOIZEUS Sta-0). On complex compositional acoustic scenarios, Mega-ASR further delivers over 30% relative WER reduction against strong open- and closed-source baselines, establishing a scalable paradigm for robust ASR in-the-wild.

1 Introduction

Although ASR models approach 1% WER on standard benchmarks, robust recognition in complex real-world acoustic conditions remains insufficient. MEGA-ASR addresses this gap by targeting compound environments, single-model performance across conditions, and semantic recovery under severe distortion.

  • Motivation: 1% WER is approached by state-of-the-art ASR models on widely used benchmarks, but these results do not resolve harder real-world acoustic settings.The introduction motivates a shift toward ASR-in-the-wild^2 because environmental conditions can be singly or compoundly complex.
  • Proposed framework: MEGA-ASR is proposed as a framework for strengthening ASR under complex acoustic environments encountered in the wild.Its stated goal is to improve recognition under much harder settings than standard benchmarks.
  • Contributions: MEGA-ASR aims to achieve state-of-the-art accuracy across individual conditions, superior performance on compound real-world recordings, and semantic recovery under severe conditions.These three capabilities are presented as the framework’s core contributions.

2 Related Work

Recent ASR foundation models perform strongly on standard benchmarks, but this performance does not ensure robustness in real-world deployment. Deployment speech can contain simultaneous acoustic degradations, motivating robust ASR research beyond clean or mildly noisy conditions.

  • ASR Foundation Models and Robust Speech Recognition: Recent ASR foundation models include encoder-decoder systems, large-scale self-supervised models, and audio-language models.
  • ASR Foundation Models and Robust Speech Recognition: Strong performance under clean or mildly noisy conditions does not imply robustness in deployment.
  • ASR Foundation Models and Robust Speech Recognition: Deployment speech is often corrupted by simultaneous degradations, including noise, far-field propagation, reverberation, obstruction, device distortion, and transmission dropout.

3 VOICES-IN-THE-WILD-2M

VOICES-IN-THE-WILD-2M is a scalable, spectrogram-level simulated dataset designed to model compound real-world acoustic conditions beyond isolated effects. The authors also release Voices-in-the-wild-Bench, a 5,000-clip English/Mandarin evaluation set spanning seven atomic phenomena.

  • Dataset motivation and construction: VOICES-IN-THE-WILD-2M uses spectrogram-level code-based simulation to address real-world environments where multiple acoustic effects can push WER beyond 30%.Existing robust-ASR datasets mainly cover isolated conditions with mild WER typically between 4%–10%.
  • Dataset motivation and construction: The pipeline synthesizes acoustic conditions from clean speech clips because curating online materials was costly and fundamentally unscalable.Atomic phenomena are simulated directly on spectrograms, then composed into realistic scenarios.
  • Dataset motivation and construction: 54 hybrid scenarios expand seven meta-scenarios to cover diverse real-world acoustic degradations at scale.The dataset is constructed by composing atomic acoustic effects into compound scenarios.
  • Evaluation benchmark: Voices-in-the-wild-Bench contains 5,000 English/Mandarin clips covering the same seven atomic phenomena, including 3,500 synthetic clips and 1,500 real-world recordings.The real-world recordings were collected from internet sources and 16 human participants.

4 Mega-ASR

Mega-ASR combines Acoustic-to-Semantic Progressive Supervised Fine-Tuning with Dual-Granularity WER-Gated Policy Optimization for robust recognition under complex acoustic conditions. It also uses environment-aware routing to preserve complementary clean-speech, hotword, and streaming capabilities.

  • Mega-ASR-Base builds on Qwen3-ASR through Acoustic-to-Semantic Progressive Supervised Fine-Tuning to improve perceptual robustness and semantic recovery.
  • Dual-Granularity WER-Gated Policy Optimization supplies token- and sentence-level rewards while dynamically modulating granularity to mitigate WER reward failure.
  • Training errors are predominantly word-level confusions at WER<=30% but shift to sentence-level hallucinations and omissions beyond that threshold, motivating granularity-aware optimization.The standard WER reward conflates these regimes and saturates under heavy degradation.
  • The dynamic reward combines token-level refinement for local information recovery with sentence-level reconstruction for semantic preservation, using WER-gated mirrored fusion.The token-level component addresses acoustic confusion, while the sentence-level component targets hallucinations, omissions, and broader structural failures.
  • Environment-aware inference routes each utterance to Mega-ASR or another model to retain noise robustness alongside clean-speech, hotword, and streaming ASR capabilities.A lightweight binary classifier with LoRA predicts whether an input requires Mega-ASR’s noise-robust model.

5 Experiments

Mega-ASR is evaluated on clean, adverse, and compound-condition ASR benchmarks against representative conventional, audio-language, and omni-modal systems. Results show competitive clean and multilingual recognition, improved semantic-level outcomes, and ablation evidence supporting progressive adaptation and reward design.

  • Datasets and Evaluation: Evaluation spans standard ASR, adverse-condition ASR, and compound acoustic scenarios covering realistic multi-factor environments.Standard benchmarks include LibriSpeech, CommonVoice22, FLEURS, AISHELL-1, WenetSpeech, and VoxPopuli; adverse benchmarks include CHiME-4, VOiCES, and NOIZEUS.
  • Competitive general ASR with adaptive routing: With adaptive routing, LibriSpeech WER improves from 1.78/3.57 to 1.63/3.37, while Fleurs zh/en reaches 3.86/3.17.The model remains highly competitive on clean and multilingual benchmarks against Qwen3-ASR, Seed-ASR, and Kimi-Audio.
  • Semantic-level gains: Missed-content drops from 14.2 to 5.9, showing gains beyond WER through reduced hallucination and dropped utterances.The semantic-level improvements are reported against Qwen3-ASR on the average Voices and Noizeus evaluation.
  • A2S-SFT and DG-WGPO ablation: Removing the first two progressive stages yields 8.31/8.79 WER, remaining 0.72/0.67 behind Mega-ASR-Base and supporting staged acoustic-to-semantic adaptation.Vanilla DAPO with Rwer alone also outperforms vanilla GRPO by 0.11/0.13 WER on Voices/Noizeus.
  • Reward design: 3.2× lower time-cost accompanies comparable WER for the rule-based reward versus the LLM-judge variant, whose training takes 62.23s per step.Differences remain within roughly 0.1 on Voices and Noizeus and 0.11 on Voi-R.
  • Hyperparameter sensitivity: At αdyn=0.8, V.N.R. rises from 7.4 to 9.1 and Nz from 7.6 to 8.1, while αdyn=0.4 raises V.F. by 0.4 and V.F.R. by 0.3.The results indicate that over-weighting the gating term harms WER-driven recognition, whereas lowering it hurts far-field subsets.

6 Case study

The case study compares MEGA-ASR with GEMINI-3-PRO and QWEN3-ASR on challenging audio involving far-field reconstruction, hallucination, and entity recovery. MEGA-ASR aligns with the reference transcript in the far-field case, while competing systems produce empty or incorrect outputs.

  • Far-field reconstruction: 100.0% WER: QWEN3-ASR returns an empty transcription for the far-field case at Peak -5.2 dB.The case study compares QWEN3-ASR with GEMINI-3-PRO and MEGA-ASR on the same challenging audio.
  • Case-study design: The comparison covers far-field reconstruction, content hallucination, and entity recovery against GEMINI-3-PRO and QWEN3-ASR.Figure 6 frames these scenarios as tests of semantic reconstruction, hallucination resistance, and fine-grained detail recovery.
  • Far-field reconstruction: MEGA-ASR achieves WER 0.0% on far-field semantic reconstruction, faithfully aligning with the reference transcript.The audio is described as extremely unintelligible and evaluated under strong environmental robustness.

7 Conclusion · A Qualitative Case Studies

MEGA-ASR addresses the acoustic robustness bottleneck through scalable compound-data construction and progressive acoustic-to-semantic optimization. Qualitative case studies show that it converts catastrophic baseline failures under severe degradation into correct or near-correct transcriptions while preserving utterance semantics.

  • 7 Conclusion: MEGA-ASR targets severe, compositional acoustic distortions that limit current ASR and large audio-language models.The framework is designed to overcome the acoustic robustness bottleneck in real-world speech recognition.
  • 7 Conclusion: 7 classic acoustic phenomena and 54 physically plausible compound scenarios define the VOICES-IN-THE-WILD-2M dataset.The dataset provides the large-scale compound-data foundation of the unified framework.
  • 7 Conclusion: Acoustic-to-Semantic Progressive Supervised Fine-Tuning and Dual-Granularity WER-Gated Policy Optimization provide MEGA-ASR’s robust perceptual training approach.These two methods are identified as central components of the framework.
  • A Qualitative Case Studies: The qualitative examples cover off-audio hallucination, empty-output collapse, dropout-induced semantic drift, noisy semantic drift, and entity-level recovery.They illustrate how MEGA-ASR changes baseline error modes under severe acoustic degradation and are intended to complement, not replace, quantitative evaluation.
  • A Qualitative Case Studies: Baseline failures often involve cross-lingual hallucination, empty outputs, severe semantic drift, and missing key entities rather than local substitutions.Figure 9 highlights that these failures are qualitatively different from ordinary word-level errors.
  • A Qualitative Case Studies: MEGA-ASR often converts catastrophic baseline failures into correct or near-correct transcriptions while preserving the utterance’s semantic backbone.Minor lexical differences may remain even when the main meaning is recovered.

B Additional Robust Benchmark Results … C.4 Construction of Compound Acoustic Scenarios

The paper reports complementary robustness comparisons across CHiME-4, NOIZEUS, and VOiCES, then details a hierarchical, physically constrained procedure for constructing compound acoustic scenarios. The construction progresses from primitive transformations to atomic effects and 54 anchor–modifier scenario categories with order-preserving chain merging.

  • B Additional Robust Benchmark Results: The comparison evaluates Qwen3-ASR-1.7B, Merged-v2, and quality-routed 3-LoRA across CHiME-4, NOIZEUS, and VOiCES under varied adverse acoustic conditions.Table 10 reports benchmark-average WER, while Tables 11–13 provide subset-level breakdowns.
  • B Additional Robust Benchmark Results: Both enhanced variants improve three-benchmark average robustness over Qwen3-ASR-1.7B; Merged-v2 leads VOiCES and NOIZEUS, while quality-routed 3-LoRA leads CHiME-4.The results associate always-on adaptation with far-field and controlled noisy conditions, and quality-aware routing with balancing real and simulated noisy speech.
  • C.1 Hierarchical Simulation Pipeline: VOICES-IN-THE-WILD-2M decomposes in-the-wild speech degradation hierarchically into primitive acoustic effects, atomic acoustic effects, and compound acoustic scenarios.This hierarchy avoids directly enumerating complex real-world environments.
  • C.2 Primitive Acoustic Effects: Eight controllable primitive transformations model signal-level degradation, including noise, echo, reverberation, distortion, resampling, filtering, loudness changes, and frame-level stutter.Each primitive exposes interpretable parameters tied to global severity and remains modular for ordered composition.
  • C.3 Construction of Seven Atomic Acoustic Effects: Seven atomic effects represent common in-the-wild conditions by combining ordered primitive chains with one dominant degradation mechanism and realistic secondary artifacts.Examples include far-field reverberation, low-pass attenuation, and loudness reduction, plus barrier, echo&reverb, recording coloration, distortion, and transmission dropout conditions.
  • C.4 Construction of Compound Acoustic Scenarios: The compound-scenario design separates three scene-defining anchors—far-field, Echo&reverb, and obstructed—from four portable modifiers—recording coloration, electronic distortion, noise, and transmission dropout.Anchors determine acoustic geometry, while modifiers attach flexible degradations across environments without combining mutually distinctive propagation conditions.
  • C.4 Construction of Compound Acoustic Scenarios: 54 acoustic scenario categories comprise seven single-effect, 18 two-effect, 13 three-effect, and 16 higher-order scenarios built from anchor–modifier compositions.Final chains preserve within-scene primitive order, remove cross-scene duplicate operators except additive noise, and retain intentional repeated filters within an atomic effect.

C.5 Severity Sampling and Difficulty Calibration

Mega-ASR calibrates sample difficulty with a shared global severity variable derived from a latent uniform draw, enabling coherent control across heterogeneous acoustic effects. Among four candidate mappings, linear sampling provides the best balance of coverage, learnability, and difficulty diversity in pilot robustness experiments.

  • Severity variable: Each sample draws x ∼U(0, 1) and maps it to a shared severity value m ∈[0, 1] across all primitive effects.This keeps degradation globally coherent rather than varying arbitrarily between effects.
  • Candidate mappings: The four mappings produce distinct difficulty profiles: linear is even, sqrt-forward favors hard samples, sqrt-backward favors easy samples, and gaussian-mid concentrates on medium difficulty.All mappings transform the same uniform variable x into the common severity range [0, 1].
  • Parameter calibration: A shared severity interface instantiates heterogeneous parameters, with larger m corresponding to stronger or otherwise more difficult effects depending on parameter semantics.Examples include stronger reverberation, larger distortion drive, lower target loudness, and higher stutter probability.
  • Empirical selection: The linear mapping provides the most balanced coverage of easy, medium, and hard samples and achieves the best overall robustness in pilot supervised-fine-tuning experiments.Sqrt-forward may reduce early learnability, sqrt-backward under-exposes challenging conditions, and gaussian-mid under-represents both extremes.

D Router Implementation and Training Details · D.1 Motivation

Mega-ASR uses an environment-aware router to select between the original Qwen3-ASR backbone and the MEGA-ASR LoRA branch, preserving clean-domain capabilities while targeting degraded speech. The router performs model selection only and leaves transcript generation and ASR decoding unchanged.

  • D.1 Motivation: The router addresses the limitation that always using robust Mega-ASR weights is not optimal for every input.Mega-ASR is optimized for acoustically degraded speech, whereas the original backbone retains strong clean-domain behavior.
  • D.1 Motivation: The original Qwen3-ASR backbone preserves complementary clean-speech recognition capabilities.These capabilities include clean-speech recognition, hotword recognition, and streaming-style inference.
  • D.1 Motivation: Mega-ASR introduces a lightweight environment-aware router for input-dependent model selection.The router predicts whether an input utterance should be processed by the original backbone or the robust branch.
  • D.1 Motivation: The router outputs a binary decision for each input audio clip.It determines whether the clip is clean or degraded before model selection.
  • D.1 Motivation: Clean inputs are routed to the base Qwen3-ASR model, while degraded inputs are routed to the MEGA-ASR LoRA branch.This routing scheme selects the model according to the predicted acoustic condition.
  • D.1 Motivation: The router does not generate transcripts or modify the ASR decoding process.Its role is restricted to choosing which ASR model processes the input audio clip.
  • D.1 Motivation: This design makes Mega-ASR a plug-and-play robustness module rather than a full replacement for the original ASR system.The original Qwen3-ASR model remains available for clean inputs and complementary capabilities.

D.2 Router Model Architecture

The router is a lightweight audio-quality classifier that predicts whether an input is clean or degraded using log-Mel acoustic features and a single-layer Transformer to minimize routing overhead.

  • D.2 Router Model Architecture: The router predicts a binary clean-versus-degraded label from log-Mel acoustic features using a single-layer Transformer architecture.This design minimizes routing overhead.
  • D.2 Router Model Architecture: The model extracts 80-dimensional log-Mel features, downsamples them with a convolutional frontend, adds sinusoidal positional encoding, and applies one Transformer encoder layer.Attention pooling then forms an utterance-level embedding for a linear binary classification head.

D.3 Router Training Data … D.6 Inference Overhead

Mega-ASR uses a lightweight binary router to distinguish clean from degraded speech and switch a single Qwen3-ASR model between base and LoRA branches. The design achieves over 99.5% routing accuracy while adding no measurable inference overhead in practice.

  • D.3 Router Training Data: Clean speech is labeled 0 and routed to the original Qwen3-ASR backbone, while degraded speech is labeled 1 and routed to the MEGA-ASR LoRA branch.Clean data comes from LibriSpeech, AISHELL-1, CommonVoice22, and WenetSpeech; degraded data comes from VOICES-IN-THE-WILD-2M.
  • D.3 Router Training Data: The final router dataset contains 552,651 clean samples and 674,107 degraded samples, split into 1,104,084 training, 61,337 validation, and 61,337 test samples.The split covers the complete final router dataset.
  • D.3 Router Training Data: Over 99.5% binary classification accuracy on the held-out development set shows that a lightweight router can reliably detect acoustic differences between clean and degraded inputs.This supports using the router as the front-end decision mechanism for branch selection.
  • D.4 Training Objective and Optimization: The router predicts pθ(y | x) for binary labels y ∈{0, 1}, with y = 1 denoting degraded speech, and is optimized using standard cross-entropy loss.This formalizes environment detection as binary classification.
  • D.4 Training Objective and Optimization: Audio is resampled to 16 kHz, converted to mono, truncated to 30 seconds, represented with log-Mel spectrograms, and trained with augmentation and AdamW-based optimization.Training also uses padding, random gain perturbation, weak additive noise, warmup cosine scheduling, gradient clipping, label smoothing, and mixed precision.
  • D.5 Integration with Qwen3-ASR and LoRA Delta Switching: A single Qwen3-ASR-1.7B instance preloads robust-adapter LoRA deltas, activating them for degraded inputs and retaining base weights for clean inputs.With γ = 0.5, inputs route to the MEGA-ASR LoRA branch when pdirty(x) ≥γ and to the Qwen3-ASR base branch otherwise.
  • D.6 Inference Overhead: 371 seconds versus 374 seconds for direct Qwen3-ASR inference yields a −0.8% relative difference on CHiME-4, indicating no measurable practical overhead.Precomputed delta-tensor switching avoids full-model reloading, and batch grouping further reduces unnecessary switching.

E Training and Implementation Details … F Additional Related works

Mega-ASR uses staged, parameter-efficient A2S-SFT followed by multimodal DG-WGPO to progressively improve acoustic grounding and semantic transcription recovery. The related-work discussion situates this approach among traditional robust ASR, large audio language models, and established speech benchmarks.

  • E.1 A2S-SFT Hyperparameters: A2S-SFT uses three LoRA-based phases initialized from Qwen3-ASR-1.7B: acoustic adaptation, semantic adaptation, and joint acoustic-semantic adaptation.The effective batch size is 128 unless otherwise specified.
  • E.1 A2-SFT Hyperparameters: Phase I expands training from WER < 30% to WER < 50% to WER < 70%, providing an acoustic warm start before harder samples.Only the acoustic encoder and speech-to-LLM aligner are updated during this curriculum stage.
  • E.1 A2S-SFT Hyperparameters: Phases II and III use the full targeted split: Phase II updates only LLM-side LoRA parameters, while Phase III jointly updates the encoder, aligner, and LLM.Direct full-module training was less stable, whereas encoder-aligner-only training gave limited gains on heavily corrupted samples requiring semantic recovery.
  • E.2 DG-WGPO Hyperparameters: DG-WGPO starts from the A2S-SFT LoRA-merged checkpoint and adapts Qwen3-ASR into a multimodal DAPO-style policy model.The adaptation preserves audio preprocessing and prompt construction while supporting group-based policy optimization.
  • E.2 DG-WGPO Hyperparameters: K = 12 candidate transcriptions are sampled per prompt, using stochastic decoding to maintain intra-group diversity while monitoring hallucinations, overlong outputs, and format violations.Temperature selection balances candidate potential, base-ability preservation, and valid-output rate.
  • E.2 DG-WGPO Hyperparameters: The reward uses τ = 0.5, soft-error discount αs = 0.4, and dynamic-reward weight αdyn = 0.6, emphasizing token refinement below the WER gate and structural recovery above it.This design targets medium- and high-WER examples where standard WER rewards become less discriminative.
  • F Additional Related works: Traditional robust ASR relies on enhancement, normalization, speaker adaptation, multi-condition training, and language-model rescoring, but is mainly evaluated by word error rate.These methods target robustness to noise, reverberation, channel mismatch, speaker variation, and domain shift rather than semantic reasoning over speech.
  • F Additional Related works: Large Audio Language Models enable audio-conditioned instruction following, question answering, and reasoning, while datasets such as LibriSpeech, TED-LIUM, Switchboard, and Common Voice support varied ASR evaluation.LALMs can use linguistic knowledge and contextual reasoning to recover corrupted or ambiguous speech.
Loading 2605.19833v1…