PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Retrieval, Worse Robustness:How Multi-hop RAG Amplifies Upstream ASR Errors

Zhenghua Bao

arXiv:2608.22872v1cs.CLcs.IReess.AS

TL;DR

ASR errors enter spoken RAG before retrieval, but their propagation through multi-hop architectures remains underexplored. The paper evaluates four RAG configurations across synthesized accents and three benchmarks, finding that richer structures amplify rather than absorb errors, especially when query entities are corrupted. The study is limited by synthetic speech, English-only evaluation, and restricted voice coverage.

  • Problem

    The impact of upstream ASR errors on retrieval-augmented generation, especially multi-hop systems with graph linking or iterative reformulation, has received little attention.

  • Method

    The authors evaluate four RAG methods on three multi-hop QA benchmarks using TTS-synthesized queries in four English accents and compare them with clean-text oracle inputs.

  • Results

    Structurally complex retrieval methods amplify rather than absorb ASR errors, widening the oracle–NG F1 gap by 36–67% relative to naive dense retrieval across all three benchmarks.

  • Takeaways & Limitations

    Query-entity corruption is the dominant failure mechanism, while lightweight surface-form mitigations close at most a small fraction of the gap.

  • Takeaways & Limitations

    The evaluation uses synthesized speech, one voice per accent, English only, and limited validation beyond Whisper-large-v3 and one benchmark-condition check.

Abstract

from arXiv · show

Speech-based applications pass spoken queries through automatic speech recognition (ASR) before any retrieval module, so ASR errors enter the pipeline as a fixed upstream constraint. We empirically test whether two extensions to standard retrieval-augmented generation (RAG), entity-graph linking and iterative reformulation, absorb or amplify these errors. Using four English accents synthesized through neural TTS, we evaluate four RAG configurations on three multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA and MuSiQue) against a clean-text oracle. Although the structurally richer configurations generally retain higher absolute F1 under ASR input, both extensions amplify the error: the F1 gap from clean text to the highest-WER accent is 36-67% larger under their combination than under naive dense retrieval, on all three benchmarks. The dominant failure mode is corruption of one or more query entities, accounting for 87-96% of degradation cases on 2WikiMultiHopQA across all four methods. Two lightweight surface-form mitigations leave most of the gap intact, indicating that downstream retrieval structure amplifies remaining entity errors. We release code and data at https://github.com/ZhenghuaBao/spoken-multihop-rag .

1 Introduction

This study examines how accented-speech ASR errors propagate through multi-hop RAG and whether richer retrieval structures absorb or amplify them. Across controlled accents and benchmarks, structurally complex methods amplify errors despite higher clean-text performance, with query-entity corruption dominant and lightweight mitigations recovering little.

  • Main finding: 36.5% larger drop: on HotpotQA, the oracle–Nigerian F1 gap rises from 0.104 for Naive RAG to 0.142 for IRCoT+HippoRAG2.The richer method still achieves higher clean-text F1, but loses more performance under the highest-WER accent.
  • Motivation: ASR errors receive limited attention in RAG, despite multi-hop QA adding cross-document retrieval and reasoning dependencies.The study distinguishes general query corruption from named-entity corruption and tests whether graph linking and iterative reformulation change robustness.
  • Study design: The evaluation spans three multi-hop QA benchmarks, four English accents, and four RAG methods, totaling 12,000 spoken queries.Questions are synthesized with TTS and evaluated against clean-text oracle inputs.
  • Failure mechanism: Query-entity corruption accounts for 87–96% of degradation cases on 2WikiMultiHopQA across all four methods.The corresponding ranges are 67–82% on HotpotQA and 54–78% on MuSiQue.
  • Mitigation: N-best decoding and phonetic entity correction leave most of the performance gap intact, indicating amplification by downstream retrieval structure.These mitigations are used as diagnostic probes rather than as a runtime branch.

2 Related Work

Prior work establishes ASR disparities and downstream error propagation, but has focused mainly on single-hop or single-pass settings. This paper extends the question to multi-hop retrieval, where graph linking and iterative reasoning introduce additional propagation points.

  • ASR error propagation: Prior studies report higher ASR WER for several dialectal, regional, and second-language varieties and document downstream effects in NLP tasks.The cited work spans gender and dialect bias, commercial ASR systems, and global English accents.
  • Multi-hop RAG: Multi-hop QA requires chaining evidence across multiple documents, creating challenges distinct from single-hop open-domain QA.Naive dense retrieval can struggle when a bridging entity is absent from the query.
  • Graph retrieval: HippoRAG builds an entity-linked knowledge graph and retrieves through personalized PageRank seeded from query entities.HippoRAG2 additionally retrieves propositional facts and combines fact-level information.
  • Spoken QA: Prior spoken QA research primarily studies single-hop reading comprehension or conversational QA with mitigation inside a single retrieve-and-read pass.This study instead examines propagation during multi-hop reasoning.

3 Pipeline

The pipeline synthesizes fixed text questions in four accented English voices, transcribes them with Whisper, optionally applies separately evaluated surface-form mitigations, and runs one of four RAG methods. A clean-text oracle bypasses speech processing for comparison.

  • Speech processing: Each question is synthesized in US, Indian, Filipino, and Nigerian accents, then transcribed with Whisper-large-v3.The four voices share comparable speaking style and audio quality, reducing but not eliminating speaker and channel confounds.
  • RAG methods: Four RAG methods include Naive RAG, HippoRAG2, IRCoT+Naive, and IRCoT+HippoRAG2.The iterative configurations generate follow-up search queries, while the combined configuration adds graph-based linking.
  • Mitigation conditions: The mitigation stage contains separate experimental conditions—no mitigation, N-best decoding, or phonetic entity correction—not a runtime branch.Both mitigations operate on the ASR transcription before retrieval and require no retraining.
  • Oracle: The oracle condition feeds the original text directly to retrieval, providing a clean-text upper bound.It bypasses the speech pipeline entirely.
  • Mitigation methods: N-best decoding samples alternative ASR hypotheses, while phonetic correction replaces extracted entities using corpus-based phonetic matching.The two probes target surface-level forms of entity error.

4 Experimental Setup

The experiments use three multi-hop QA benchmarks, controlled accented speech, and a real-speech validation set. Performance is measured with normalized token-level F1 and EM, while ASR quality and degradation cases are defined explicitly against clean-text oracle inputs.

  • Benchmarks: The study samples 1,000 validation questions from each of HotpotQA, 2WikiMultiHopQA, and MuSiQue.The associated supporting and distractor passages form each benchmark’s retrieval corpus.
  • Real-speech validation: The authors transcribe 500 real Nigerian-accented utterances to validate whether synthesized speech produces comparable ASR error patterns.This set is used only for ASR error-pattern validation, not multi-hop QA evaluation.
  • Answer metrics: F1 and Exact Match are computed after SQuAD-style normalization and averaged within each dataset-accent cell.The generated answer is evaluated as the raw output string without an acceptance or confidence check.
  • ASR metrics: WER is computed by Levenshtein alignment between each transcription and the original question after lowercasing and whitespace tokenization.Per-accent gaps use ΔF1 = F1_oracle − F1_accent.
  • Degradation analysis: A degradation case is correct oracle performance followed by incorrect performance from the ASR transcription, with correctness defined as token-level F1 ≥0.5.Cross-condition F1 and EM significance uses paired bootstrap with 10,000 resamples.

5 Results and Analysis

Across three benchmarks, ASR degradation increases with WER, while structural retrieval extensions amplify rather than absorb the clean-text-to-ASR gap. Entity corruption dominates failures, and lightweight surface-form mitigations recover only a small fraction of the loss.

  • ASR Error Degrades Multi-hop QA and Scales with WER: ASR degradation scales with WER across accents and individual questions.Questions above 20% WER degrade in 25.2% of Naive RAG cases and 37.8% of IRCoT+HippoRAG2 cases.
  • Structural Complexity Amplifies Rather Than Absorbs ASR Errors: 36.5%, 42.3%, and 67.4% larger oracle–NG F1 gaps occur for IRCoT+HippoRAG2 than Naive RAG on HotpotQA, 2WikiMultiHopQA, and MuSiQue, respectively.The combined configuration has the highest clean-text F1 on all three benchmarks but the largest oracle–NG gap.
  • Structural Complexity Amplifies Rather Than Absorbs ASR Errors: Iterative reformulation contributes to amplification through cumulative retrieval rounds, while the structural gap persists even after partial closure.At step 5, HippoRAG2’s gap is 0.131 versus 0.115 for Naive RAG, remaining 14% larger.
  • Entity Corruption is the Dominant Failure Mechanism: 87–96% of degradation cases on 2WikiMultiHopQA involve query-entity corruption across all four methods.Entity corruption remains the largest category on HotpotQA and MuSiQue as well, accounting for 67–82% and 54–78% of cases.
  • Entity Corruption is the Dominant Failure Mechanism: A single corrupted entity can erase retrieval signal during later reformulation and linking steps.In the case study, mild corruption breaks both IRCoT variants while base retrievers remain intact; severe corruption causes complete failure for Naive RAG and both iterative variants.
  • Lightweight Mitigations Reveal a Structural Gap: Lightweight surface-form mitigations close only a small fraction of the oracle–NG gap.N-best decoding recovers −2.2% to +2.5%, while phonetic correction recovers +4.4% to +11.1%.

6 Conclusion

Across three benchmarks and four English accents, upstream ASR errors degrade multi-hop QA in proportion to WER. Structurally complex retrieval amplifies these errors, with entity corruption dominant and lightweight surface mitigations recovering only a small fraction of the gap.

  • 6 Conclusion: ASR errors degrade multi-hop QA performance, and degradation tracks upstream WER across four architectures, three benchmarks, and four English accents.
  • 6 Conclusion: 36–67%: combining entity-graph linking and iterative reformulation widens the oracle–NG F1 gap relative to naive dense retrieval on all three benchmarks.Richer configurations generally remain more accurate under ASR input but lose a larger fraction of their clean-text performance.
  • 6 Conclusion: 87–96%: entity corruption accounts for degradation cases on 2WikiMultiHopQA and remains the largest category on HotpotQA and MuSiQue across all four methods.
  • 6 Conclusion: Lightweight surface-form mitigations close at most a small fraction of the gap, suggesting the failure is not explained by random ASR noise or surface-level entity confusion alone.

Limitations

The evaluation is limited by synthetic speech, narrow model and language coverage, proxy-based error attribution, and limited real-speech validation. These constraints leave generalization to diverse speakers, languages, models, and real spoken multi-hop QA unresolved.

  • Limitations: The evaluation uses synthesized speech rather than recorded human speech, with a single voice per accent.
  • Limitations: Real Nigerian utterances show comparable entity-corruption rates, but the validation covers only one dataset and synthetic voice condition rather than end-to-end real spoken multi-hop QA.
  • Limitations: The evaluation is restricted to English, while additional voices, speaker attributes, code-switched languages, and tonal languages may exhibit different ASR error patterns.
  • Limitations: Named-entity corruption is identified with a rule-based proxy, and human validation would strengthen confidence in the attribution.
  • Limitations: A single gpt-4o-mini generator is used, so larger or open-source LLMs may handle corrupted input differently, particularly inside IRCoT.

Ethical Considerations

The paper frames its accent comparisons as controlled ASR error conditions rather than demographic estimates, and treats fairness implications cautiously. Its diagnostic mechanism and mitigations are presented as research findings without anticipated dual-use concerns beyond general RAG and ASR risks.

  • Ethical Considerations: Accent comparisons are controlled error conditions, not quantitative claims about specific demographic populations.
  • Ethical Considerations: A single TTS voice per accent does not represent within-accent speaker variation, limiting demographic interpretation of the findings.
  • Ethical Considerations: The amplification finding suggests a fairness implication for voice-driven deployments, but the paper does not claim a fairness result.
  • Ethical Considerations: Concrete fairness claims require multi-hop QA evaluation on real spoken queries because comparable per-entity corruption rates do not establish matching end-to-end RAG behavior.
  • Ethical Considerations: The identified mechanism is a diagnostic finding rather than an attack design, and the benchmarked mitigations are public, lightweight, and computationally modest.

C Exact Match Results

Exact Match follows the same structural amplification pattern as F1, with smaller absolute gaps but tight agreement across conditions. The largest reported EM gap occurs for IRCoT+HippoRAG2 on 2WikiMultiHopQA.

  • C Exact Match Results: Exact Match requires exact string match after normalization and is stricter than F1.
  • C Exact Match Results: Pearson r > 0.95: EM gaps track F1 gaps tightly across all cells.
  • C Exact Match Results: The method-level amplification ordering observed for F1 is preserved on every dataset under EM.
  • C Exact Match Results: 0.189: the largest EM gap occurs on 2WikiMultiHopQA under IRCoT+HippoRAG2, mirroring the structural amplification pattern reported for F1.

D Error-Type Distributions Across Benchmarks and Methods

Entity corruption is the most frequent degradation category across all four RAG methods and all three benchmarks. Severe garbling usually co-occurs with entity corruption, while MuSiQue shows one limited exception.

  • Cross-benchmark analysis: Entity corruption remains the largest single failure category across HotpotQA and MuSiQue accents under Naive RAG.
  • Cross-benchmark analysis: Severe garbling reaches 44% on HotpotQA NG and co-occurs with entity corruption rather than replacing it.
  • Cross-benchmark analysis: “Other content change” reaches 28% on MuSiQue US, where it meaningfully competes with entity corruption in the dataset’s only such cell.The passage attributes this pattern to MuSiQue’s small degradation sample (n = 46) and low overall WER (5.2%).
  • Cross-method analysis: Entity corruption accounts for 67–82% of HotpotQA, 87–96% of 2WikiMultiHopQA, and 54–78% of MuSiQue degradation cases across methods.

E Real-World Speech Validation

Validation on real Nigerian-accented speech indicates that the synthesized condition understates ASR difficulty, while entity corruption rates remain comparable between real and synthesized speech.

  • Dataset: The validation corpus contains short Nigerian-accented English utterances from multiple speakers and is used only to validate ASR error patterns.
  • Transcription and WER: Real Nigerian speech reaches 28.9% mean WER, compared with 17.1% on 2WikiMultiHopQA, 14.5% on HotpotQA, and 7.9% on MuSiQue under synthesized speech.
  • Transcription and WER: A noticeable fraction of real-speech utterances exceeds 50% WER because of recording-channel and speaker variation unavailable in TTS.
  • Entity corruption rate: Among 282 real utterances containing entities, 51.8% have at least one mistranscribed entity and 40.7% of 607 entity instances are altered.
  • Entity corruption rate: The synthesized Nigerian 2WikiMultiHopQA condition alters 43.8% of entity instances, 1.08× the real-speech rate.

F ASR Sensitivity Analysis

Replacing Whisper-large-v3 with SeamlessM4T-v2-large increases WER while preserving the cross-method ordering of oracle–ASR gaps. Absolute F1 changes remain small, but the largest gap widens.

  • Results: SeamlessM4T raises mean WER on 2WikiMultiHopQA NG audio from 17.1% to 28.0%, a 64% increase.
  • Results: Absolute F1 changes by at most 0.019 under SeamlessM4T, for IRCoT+HippoRAG2.
  • Results: The oracle–ASR gap ordering remains Naive RAG < HippoRAG2 < IRCoT+Naive < IRCoT+HippoRAG2 under both ASR systems.
  • Results: The largest gap grows from 0.195 under Whisper to 0.214 under SeamlessM4T.
Loading 2608.22872v1…