PaperlayerView exact revision on arXiv

Source-linked AI summary

SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise

Yuejie Li, Ke Yang, Yueying Hua, Berlin Chen, Jianhao Nie, Yueping He, Caixin Kang

arXiv:2602.12783v3cs.IRcs.AI

TL;DR

Spoken-query retrieval lacks unified robustness evaluation because existing benchmarks simplify queries, separate ASR from retrieval, or assume clean text. SQuTR builds a diverse spoken-query benchmark with controlled acoustic conditions and evaluates cascaded and end-to-end systems. Retrieval performance consistently decreases with increasing noise, with system-dependent degradation and persistent difficulty under extreme noise.

  • Problem

    Existing spoken-query evaluations use simple queries or uncontrolled environments, while ASR and IR benchmarks separately measure transcription or clean-text retrieval rather than unified downstream robustness.

  • Method

    SQuTR derives spoken queries from six English and Chinese retrieval benchmarks and evaluates retrieval pipelines under four controlled acoustic conditions using speech from 200 diverse speakers and real-world noise.

  • Results

    Retrieval effectiveness consistently declines as noise increases across ASR front-ends and lexical, dense, and end-to-end systems, with degradation patterns differing by model scale and design.

  • Takeaways & Limitations

    SQuTR provides a controllable, reproducible testbed showing that noise-robust spoken retrieval remains an open challenge with substantial room for improvement.

  • Takeaways & Limitations

    SQuTR’s prior benchmark comparison identifies limitations in SVQ’s mostly single-hop, fact-oriented queries, short general-domain passages, and absence of graded SNR control.

Abstract

from arXiv · show

Spoken query retrieval is an important interaction mode in modern information retrieval. However, existing evaluation datasets are often limited to simple queries under constrained noise conditions, making them inadequate for assessing the robustness of spoken query retrieval systems under complex acoustic perturbations. To address this limitation, we present SQuTR, a robustness benchmark for spoken query retrieval that includes a large-scale dataset and a unified evaluation protocol. SQuTR aggregates 37,317 unique queries from six commonly used English and Chinese text retrieval datasets, spanning multiple domains and diverse query types. We synthesize speech using voice profiles from 200 real speakers and mix 17 categories of real-world environmental noise under controlled SNR levels, enabling reproducible robustness evaluation from quiet to highly noisy conditions. Under the unified protocol, we conduct large-scale evaluations on representative cascaded and end-to-end retrieval systems. Experimental results show that retrieval performance decreases as noise increases, with substantially different drops across systems. Even large-scale retrieval models struggle under extreme noise, indicating that robustness remains a critical bottleneck. Overall, SQuTR provides a reproducible testbed for benchmarking and diagnostic analysis, and facilitates future research on robustness in spoken query to text retrieval.

1 Introduction

Spoken queries are increasingly important, but noise, environmental interference, and speaker variability degrade ASR and downstream retrieval. SQuTR addresses fragmented evaluation with a diverse benchmark and controlled protocol for measuring robustness across acoustic conditions.

  • Motivation: Background noise, environmental interference, and speaker variability degrade ASR, with transcription errors often causing substantial retrieval drops.These effects create a direct robustness challenge for spoken-query retrieval in real-world settings.
  • Evaluation gap: Existing evaluations separate ASR robustness, measured with WER, from text-only IR benchmarks, preventing unified comparison of spoken-query retrieval systems.Speech benchmarks focus on transcription, while IR benchmarks generally assume clean textual queries.
  • Prior benchmark limitations: SVQ uses mainly single-hop, fact-oriented queries and environment-based recordings, without explicitly controlled graded SNR levels.Its short, general-domain corpora also limit contextual and task complexity.
  • SQuTR benchmark: SQuTR derives spoken queries from six English and Chinese retrieval benchmarks, preserving broader query complexity and task diversity.The source datasets span finance, multi-hop and open-domain QA, medical retrieval, general-domain retrieval, and passage retrieval.
  • Controlled conditions: SQuTR synthesizes speech from 200 diverse speakers and injects real-world noise across four controlled SNR conditions for reproducible robustness analysis.The conditions are Clean, Low Noise, Medium Noise, and High Noise.
  • Findings: Retrieval effectiveness consistently declines as acoustic noise increases across ASR models of varying sizes and architectures.The experiments cover multiple ASR front-ends and both lexical and dense retrievers, highlighting robustness as a practical challenge.

2 Related Work

Prior work advances spoken retrieval through cascaded and end-to-end paradigms, while ASR and IR benchmarks largely evaluate separate components. SQuTR targets this gap by combining retrieval-oriented datasets with controlled acoustic conditions.

  • Spoken-query retrieval: Cascaded systems transcribe speech with ASR before applying lexical or neural text retrievers, whereas end-to-end systems map speech directly to retrieval representations.Cascaded systems benefit from modularity and reusable IR infrastructure, but remain sensitive to transcription errors.
  • ASR robustness: ASR robustness benchmarks typically use WER or CER and generally stop at transcription, leaving downstream retrieval effects unmeasured.This limits end-to-end robustness assessment for spoken-query retrieval systems.
  • SQuTR dataset: Table 1 summarizes SQuTR’s source datasets using query count, corpus size, and average query and document lengths.The caption defines Query as #queries, Corp. as corpus size, and Qlen/Dlen as average query/document length.
  • SQuTR acoustic conditions: Table 2 summarizes four SQuTR acoustic conditions that differ only in noise intensity: Clean, Low Noise, Medium Noise, and High Noise.The conditions are generated from clean speech under controlled acoustic perturbations.
  • Information retrieval benchmarks: Standard IR datasets and evaluation suites are text-only, assuming clean textual queries and omitting uncertainty from spoken input and acoustic variability.This creates a mismatch between mature text retrieval evaluation and spoken-query conditions.

3 Dataset Construction

SQuTR constructs a large spoken-query benchmark from six multilingual IR datasets, then adds controlled acoustic noise and a unified retrieval evaluation protocol. The dataset combines quality-controlled speech, diverse noise conditions, and user-facing evaluation from audio to ranked documents.

  • Source Queries: SQuTR reuses queries from six widely used English and Chinese IR benchmarks, preserving their document collections and relevance annotations.The source queries span factoid QA, multi-hop QA, general IR, short keyword-style queries, and longer natural-language questions.
  • Speech Synthesis: Speech is synthesized with CosyVoice-3 using 200 speakers, while three candidate renditions per query are filtered using reference-ASR WER/CER.Speaker profiles vary in gender, age, and accents, and queries are grouped by English and Chinese.
  • Acoustic Conditions: Four conditions—Clean, Low Noise, Medium Noise, and High Noise—are created by injecting environmental recordings at predefined SNR levels.Noise sources include public transportation, offices, households, public venues, and babble noise.
  • Quality Control: Automated checks and bilingual human verification assess synthesis artifacts, volume, truncation, naturalness, noise assignment, and semantic consistency.Samples failing manual inspection are discarded and regenerated; included audio receives at least one human-verification round.
  • Dataset Statistics: 37,317 unique queries are paired with four acoustic conditions, producing 149,268 evaluation instances.Each instance is defined as a query–acoustic-condition pair.
  • Evaluation Protocol: The unified protocol evaluates cascaded and end-to-end systems directly from spoken input to ranked text documents using nDCG@10, Recall@k, and MRR@k.Metrics are computed on the final ranked list to reflect user-facing robustness under acoustic conditions.

4 Experiments

The experiments evaluate cascaded and end-to-end spoken-query retrieval systems across languages, acoustic conditions, ASR front-ends, and retrieval architectures. Results show that noise degrades retrieval, with robustness varying by model design, scale, and retrieval backend.

  • Evaluated Systems: SQuTR evaluates systems across four acoustic conditions, multiple ASR front-ends, and lexical, dense, and end-to-end retrieval approaches.The evaluated cascaded systems pair ASR with text retrieval, while Omni-Embed-Nemotron-3B directly maps speech to retrieval representations.
  • Main Results: Retrieval performance degrades across systems as acoustic noise increases from Clean to High Noise.The benchmark is designed to quantify robustness under graded acoustic conditions and discriminate architectural capabilities.
  • Main Results: Qwen3-Embedding-8B sustains stronger performance at 0dB, whereas traditional dense retrievers such as BGE-Large show a more pronounced monotonic drop.These differing degradation patterns provide discriminative power for robustness evaluation.
  • Main Results: Qwen3-Embedding-8B (Chinese) drops from 0.8033 on Text to 0.7760 on Clean Speech and 0.7302 at High Noise.Spoken-query retrieval remains below the text upper bound even in clean conditions, and the gap widens with noise.
  • Robustness and Stability Analysis: BM25 has lower variance than BERT-based dense retrievers, with σ= 0.031 versus BGE-Base at σ= 0.100.Dense retrievers can show strong clean-condition performance while remaining more sensitive to acoustic perturbations.
  • Robustness and Stability Analysis: Scaling Qwen3 from 0.6B to 8B reduces variance from σ: 0.094 →0.070 while improving mean effectiveness.Omni-Embed-Nemotron-3B further narrows the gap with σ= 0.054.
  • Ablation Study: Retrieval backend choice is more critical than ASR size: Whisper-Tiny with Qwen3-Embedding-8B significantly outperforms Whisper-Large-v3 with BM25.Lexical and dense retrieval maintain similar degradation trends, remaining stable through Medium Noise before sharply dropping at High Noise.
  • Ablation Study: BM25 fluctuations under Low Noise (20dB) are attributed to stochastic error variance, while dense retrieval shows a smoother monotonic decline.Modern ASR models saturate at 20dB, making random hallucination and deletion patterns relevant to exact keyword matching.

5 Conclusion

SQuTR defines a reproducible benchmark for spoken query retrieval under graded acoustic noise and formalizes its signal degradation process. The benchmark models reverberant speech plus additive noise, controls mixing through SNR, and normalizes the resulting signal.

  • SQuTR evaluates spoken query to text retrieval under graded acoustic noise using a controllable and reproducible benchmark.
  • The distorted signal combines reverberant speech with additive noise through a linear superposition model.
  • Room acoustics enter through convolution with a Room Impulse Response, producing the target reverberant speech signal.
  • The additive noise sequence is randomly sampled from the DEMAND and NOISEX-92 datasets.
  • The mixing coefficient α is solved from global RMS amplitudes for a target SNR_dB.
  • The mixed signal is globally normalized with headroom factor β = 0.9 to prevent digital clipping.

B Automated Quality Control Protocol

The automated quality-control protocol filters synthesized speech using objective ASR consistency and subjective LLM-based audio-quality evaluation. It therefore checks both semantic accuracy and perceptual fidelity before acceptance.

  • The filtering function maps each generated audio sample to a binary acceptance decision.
  • The pipeline integrates objective consistency checking with subjective proxy evaluation.
  • A large-scale ASR model compares synthesized speech with ground-truth text after normalization using Word Error Rate.
  • Two independent Judge-LLMs score audio quality on a 1–5 scale using Chain-of-Thought reasoning.
  • Figure 4 presents pseudocode for quality control combining ASR-based semantic accuracy with LLM-consensus perceptual fidelity.

C Detailed Experimental Setup

The experiments strictly control the computational environment, inference hyperparameters, and model-specific configurations to support reproducibility.

  • The study controls the computational environment, inference hyperparameters, and model-specific configurations for reproducible experiments.

C.1 Computational Environment

The experimental setup uses a high-performance computing cluster and documents hardware and software specifications. It also standardizes ASR input processing and evaluates models spanning languages, scales, and configurations.

  • C.1 Computational Environment: All experiments run on a high-performance computing cluster, with hardware and software specifications listed in Table 8.
  • C.1 Computational Environment: The setup evaluates ASR robustness across different parameter scales and languages using a multi-model strategy.
  • C.1 Computational Environment: The paper header identifies SQuTR as a robustness benchmark for spoken query to text retrieval under acoustic noise.
  • C.1 Computational Environment: The dataset is downsampled from 24kHz to 16kHz during inference for ASR systems optimized for 16kHz input.
  • C.1 Computational Environment: The ASR evaluation covers English Whisper models, Chinese Paraformer-Large, and multilingual baseline systems.

C.3 Retrieval Model Configuration

The evaluation covers 15 distinct cascaded retrieval architectures and applies model-specific task instructions through official prompt templates. Table 9 documents the prefixes used for all embedding models.

  • 15 distinct cascaded retrieval architectures were evaluated.
  • Model-specific task instructions, or prefixes, activate the asymmetric retrieval capabilities of modern embedding models.
  • Table 9 lists the model-specific query instructions used for all embedding models.

C.4 Evaluation Metrics Standard

Evaluation uses normalized nDCG@10 under the NIST trec_eval standard, with relevance handling adapted to graded or binary datasets. The pipeline also addresses multi-GPU ranking and audio-quality assessment through structured LLM judging.

  • Normalized Discounted Cumulative Gain at rank 10, nDCG@10, is reported according to the NIST trec_eval standard.
  • Graded relevance is preserved, while binary-dataset relevance is mapped to {0, 1}.
  • Multi-GPU inference gathers embeddings across devices to ensure accurate global ranking statistics.
  • The quality-control prompt defines explicit failure modes and handling for text-to-speech edge cases such as truncation.
  • The LLM-as-a-Judge pipeline uses role-playing, Chain-of-Thought reasoning, negative constraints, and a strictly defined JSON schema.

E Extended Experimental Results

Extended results provide retrieval diagnostics across six sub-datasets under different acoustic conditions. Additional tables compare ASR robustness for BM25 and Qwen3-Embedding-8B retrievers across the same datasets.

  • Retrieval diagnostics are reported across all six sub-datasets in Tables 10 through 21.
  • Tables 10–15 report main experimental results for FiQA, HotpotQA, NQ, DuRetrieval, MedicalRetrieval, and T2Retrieval under different acoustic conditions.
  • The main-result tables use nDCG@10, MRR@10, and Recall@10, with best results highlighted in bold.
  • Tables 16–21 compare ASR robustness for BM25 and Qwen3-Embedding-8B on the six sub-datasets under different acoustic conditions.
  • The ASR-robustness comparison tables report nDCG@10, MRR@10, and Recall@10, with best results highlighted in bold.
Loading 2602.12783v3…