PaperlayerView exact revision on arXiv

Source-linked AI summary

To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

Debajyoti Mazumder, Mamta, Abhirama Subramanyam Penamakuri

arXiv:2609.30227v1cs.LGcs.AIcs.CLcs.SD

TL;DR

Speech-based fact verification asks whether LALMs can verify spoken claims against retrieved textual evidence, despite possible differences from text-based performance. VERISPEAK is a controlled benchmark of 3,879 spoken claims, and its experiments find a text-speech gap and stronger claim-evidence comparison with retrieval plus explicit reasoning.

  • Problem

    It is unclear whether LALMs can perform evidence-grounded fact verification when claims are spoken, since verification requires identifying the claim and comparing it with retrieved textual evidence.

    Source: §1 · p. 2

  • Method

    VERISPEAK provides 3,879 balanced spoken claims across temporal, geographical, and relational facts for controlled comparisons of text, speech, retrieval, and reasoning settings.

    Source: §1 · p. 2

  • Results

    LALMs show a consistent text-speech gap, retrieval alone only partially improves speech verification, and a thinking-tuned LALM reaches 86.1% accuracy under transcript-based RAG with CoT.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Speech fact-checking requires not only speech recognition and evidence retrieval but also preserving the spoken claim and reasoning across claim and evidence.

    Source: §7 · p. 9

    The benchmark covers celebrity-biography temporal, geographical, and relational claims, uses single-speaker synthesized speech, and retrieves from a fixed evidence collection.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Online misinformation increasingly appears in spoken formats such as news clips, podcasts, interviews, political speeches, and social media videos, creating a need for fact-checking systems that can verify claims directly from speech. We introduce VeriSpeak, a probe benchmark for studying speech-based fact verification in Large Audio Language Models (LALMs). VeriSpeak contains 3,879 spoken claims spanning temporal, geographical, and relational facts, with balanced true and false labels. The benchmark is designed to examine whether factual verification ability transfers from text to speech, and whether retrieval-augmented LALMs can use textual evidence to correctly support or refute spoken claims. Our experiments reveal a consistent text-speech modality gap: LALMs that verify written claims reliably often fail on the same claims when spoken. Moreover, retrieval alone provides limited gains because models frequently conflate retrieved evidence with the spoken claim. In contrast, retrieval combined with explicit reasoning improves claim-evidence comparison, with a thinking-tuned LALM reaching 86.1% accuracy. VeriSpeak highlights that effective speech misinformation detection requires not only speech understanding, but also grounded reasoning over retrieved evidence. The dataset is publicly available via Hugging Face at https://huggingface.co/datasets/abhiram4572/VeriSpeak.

1 Introduction

VERISPEAK investigates whether factual verification transfers from text to speech and whether LALMs can compare spoken claims with retrieved textual evidence. It finds a text-speech gap, limited gains from retrieval alone, and improved claim-evidence comparison with explicit reasoning.

1 Introduction

  • VERISPEAK contains 3,879 spoken claims across temporal, geographical, and relational facts, with balanced correct and incorrect labels.The benchmark is designed for controlled comparisons across text-only, text-input LALM, speech-input LALM, retrieval, and reasoning settings.Source: §1 · p. 2
  • The study asks whether text-based verification transfers to speech, where the modality gap arises, how retrieval approaches compare, and when reasoning helps evidence use.The motivation is that speech fact verification requires identifying the spoken claim and comparing it with textual evidence.Source: §1 · p. 2
  • LALMs show a large, consistent text-speech gap: strong verification on written claims does not reliably transfer to the same claims when spoken.The paper also reports that transcript-based RAG is generally more reliable than audio-query RAG, but retrieval alone only partially improves speech verification.Source: §1 · p. 2
  • 86.1% accuracy with retrieval and reasoning was reached by a thinking-tuned LALM, while standard LALMs often conflate retrieved evidence with the spoken claim.The reported result highlights the importance of preserving the claim-evidence distinction during verification.Source: §1 · p. 2
  • VERISPEAK’s results indicate that speech verification requires not only speech recognition and evidence retrieval but also robust claim-evidence separation.Source: §1 · p. 2

2 Related Work

Prior fact-verification and multimodal work establishes retrieve-and-verify methods and cross-modal challenges, while reliable evidence-grounded verification of spoken claims remains a distinct question. VERISPEAK studies whether speech-capable models can keep spoken claims separate from retrieved text while judging veracity.

Fact Verification

  • Fact verification commonly retrieves evidence before predicting whether a claim is supported or refuted, and performance depends on retrieving useful evidence and using it correctly.Multimodal fact verification has mainly examined visual-text misinformation; speech verification instead pairs an acoustic claim with typically textual evidence.Source: §2 · p. 2

Large Audio Language Models

  • LALMs can follow instructions over speech and audio, but those capabilities do not by themselves establish reliable factual verification.Verification requires recovering the spoken claim, keeping it distinct from external textual evidence, and reasoning over both sources.Source: §2 · p. 3
  • VERISPEAK examines whether speech-capable models can perform evidence-grounded verification when the claim is spoken and the evidence is textual.The model must preserve the spoken claim as the verification target while reasoning over retrieved text.Source: §2 · p. 3

3 VeriSpeak: A Probe-Benchmark for Speech Fact Verification

VERISPEAK is constructed as a controlled probe benchmark with balanced labels and three fact categories. Its pipeline creates atomic claims and plausible incorrect counterparts, then synthesizes speech under consistent speaker conditions.

3 VeriSpeak: A Probe-Benchmark for Speech Fact Verification

  • VERISPEAK uses temporal, geographical, and relational categories to probe qualitatively different knowledge types and diagnose whether the text-speech gap varies by category.Temporal facts concern years and dates; geographical facts concern locations and nationalities; relational facts include employment, kinship, and organizational affiliations.Source: §3 · p. 3
  • Atomic, pronoun-free claims are generated from biographies so probe items express a single unambiguous fact.Items shorter than four words or containing unresolved pronouns are discarded.Source: §3 · p. 3
  • Incorrect counterparts are created with category-specific perturbations, including year shifts, same-type geographical substitutions, relation-word swaps, and entity swaps.Celebrities without a valid negative are discarded to preserve balanced labels.Source: §3 · pp. 3–4
  • All claims are synthesized with a single-speaker Coqui TTS setup to control speaker variation, and the resulting 3,879-item probe has balanced correct and incorrect claims.The set includes 1,451 year, 2,226 location, and 202 relation items; each item has audio, an automatic Wav2Vec 2.0 transcript, and a ground-truth label.Source: §3 · p. 4
  • The benchmark uses automatic transcripts despite transcription errors to reflect realistic deployment conditions and test multimodal reasoning under that added difficulty.The 659 subject entities span 60 countries; the dataset reflects the underlying knowledge base’s composition.Source: §3 · p. 4

4 Experimental Setup

The experiments compare speech-capable LALMs with their text-only backbones under controlled input, retrieval, and reasoning conditions. They include transcript-based and audio-query retrieval variants and report category-level and average accuracy comparisons.

4 Experimental Setup

  • Models are drawn from three families, with each LALM paired to its associated text-only LLM backbone and evaluated on both text and speech inputs.Audio-Flamingo-next-think is also evaluated to examine explicit reasoning in speech-based verification.Source: §4 · p. 4
  • The controlled conditions vary input modality, model interface, retrieved evidence, and explicit reasoning.They include text-only LLM and LALM conditions, speech-only LALM with and without chain-of-thought prompting, and transcript-based retrieval with and without chain-of-thought prompting.Source: §4 · pp. 4–5
  • Transcript-based retrieval uses the ASR transcript to retrieve textual evidence while the spoken claim remains the LALM’s verification target.The audio is transcribed with wav2vec 2.0; multi-e5 is the default retriever, with e5-large-v2 and msmarco-MiniLM-L12-v3 also tested.Source: §4 · p. 5, Figure 3 · p. 5
  • The audio-query retrieval variant uses the audio claim itself as the query and a CLAP audio-text retriever to match it with textual evidence.A separate text-only LLM condition receives the written claim with multi-e5-retrieved evidence as an upper bound.Source: §4 · p. 5
  • Accuracy is reported for each factual category and averaged across Year, Location, and Relation, with comparisons against paired text-only and speech-only baselines.Positive ∆ indicates degradation relative to the text-only baseline; positive Lift indicates improvement over the speech-only baseline.Source: §4 · p. 5

5 Research Questions

VERISPEAK reveals a consistent gap between written- and speech-based factual verification, while retrieval alone offers modest gains. Explicit reasoning improves use of retrieved evidence, especially for a thinking-tuned LALM.

RQ2

  • Speech-input degradation occurs across all four models, while text-mode degradation varies: ∆speech ranges from 6.5 to 27.2 points, and some LALMs match or outperform their text-only backbones on text.The decomposition distinguishes the LALM-versus-backbone text difference from the additional drop when that LALM receives speech.Source: §5 · p. 6

RQ1

  • 75.3% text-only accuracy for Qwen-7B contrasts with 50.4% for Qwen-Audio-Chat and 49.2% for Qwen2-Audio-7B in speech-only verification.The reported gaps also span Audio-Flamingo-3 and Phi-4-multimodal.Source: §5 · p. 5, Table 2 · p. 6

RQ3

  • 2.3 points: transcript-based RAG’s average Lift across standard LALMs; it improves all four models but leaves substantial gaps to text-only performance.Audio-query RAG with CLAP has an average Lift of −0.3 points, and multi-e5 is the most reliable transcript retriever overall, with an exception for Qwen-Audio-Chat.Source: §5 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
  • 65% of sampled extracted claims matched retrieved evidence rather than the spoken claim, while 31% matched the original claim and 4% were other or ambiguous.This diagnostic shows how retrieval can shift the model’s verification target away from the spoken claim.Source: §5 · p. 7

RQ6

  • 86.1% accuracy: Audio-Flamingo-next-think reaches this with transcript-based RAG and CoT, a 19.7-point Lift over speech-only; it remains 6.3 points below the text-only RAG upper bound.With the same setup, standard Audio-Flamingo-3 reaches 65.5%.Source: Table 5 · p. 8, §5 · p. 8

RQ4–RQ5

  • CoT alone lowers speech-only accuracy for standard LALMs, but adding it to transcript-based RAG improves accuracy for every standard model, averaging 4.9 points.The contrast supports the paper’s account that reasoning helps when retrieved evidence is available for claim–evidence comparison.Source: §5 · pp. 7–8

6 Additional Analysis

The additional analyses find variation in transcription quality and measurable verification degradation from model-generated transcripts. Word error rate alone does not capture the disproportionate importance of errors in fact-critical information.

WER Analysis

  • 0.13–0.23: overall WER varies across LALMs; approximately 74% of Audio-Flamingo-3’s WER edit operations involve names or other proper nouns.Non-name tokens have an error rate of approximately 3%, and names and locations can be label-critical.Source: §6 · p. 9

Transcription-based verification contrast

  • 5.6–11.6 points: ∆recognition estimates the verification degradation when a paired text LLM receives an LALM-generated transcription instead of the original claim.The contrast holds the paired text LLM constant and changes the text supplied to it.Source: §6 · p. 9, Table 7 · p. 9

Additional analysis

  • WER alone does not fully determine verification performance, because errors in entities, years, locations, or relations can disproportionately affect the final prediction.This qualifies the use of transcription error rate as a measure of downstream fact-verification impact.Source: §6 · p. 9

7 Conclusion and Future Work

VERISPEAK is a controlled benchmark for speech fact verification, and its results expose a persistent text–speech gap that retrieval alone only partly addresses. The conclusion emphasizes explicit reasoning and identifies speech–text parity as an open goal.

Conclusion

  • VERISPEAK contains 3,879 spoken claims spanning temporal, geographical, and relational facts for controlled evidence-grounded verification.The benchmark is designed to study fact verification in speech.Source: §7 · p. 9
  • Retrieval alone only partially addresses the modality gap, while transcript-based RAG with CoT brings a thinking-tuned LALM to 86.1% accuracy.The authors attribute the remaining challenge to preserving the spoken claim and comparing it with retrieved evidence.Source: §7 · p. 9

Future Work

  • Speech–text verification parity remains an important research direction because the 86.1% result still trails the fully text-based RAG upper bound.For open-world or time-sensitive claims, the authors say retrieval remains necessary, with retrieved text treated as evidence rather than the verification target.Source: §7 · p. 9
  • Future work aims to extend VERISPEAK to multilingual claims, diverse accents and dialects, natural speech, broader fact domains, and richer evidence sources.Source: §7 · p. 9

Limitations

VERISPEAK’s controlled probe design limits the claims it covers and the speech and evidence conditions it represents.

Limitations

  • The benchmark covers temporal, geographical, and relational claims from celebrity biographies, leaving scientific, medical, numerical, causal, and multi-hop claims for future work.Source: Limitations · p. 9
  • Claims use single-speaker TTS to isolate input modality, so the benchmark does not capture spontaneous speech, background noise, diverse accents, dialects, or multilingual speech.Source: Limitations · p. 10
  • Retrieval uses a fixed evidence collection for controlled comparisons rather than noisy, conflicting, or time-sensitive open-world sources.Source: Limitations · p. 10

Ethical Considerations

VERISPEAK uses public-figure information and synthetic false claims for controlled evaluation, and is not intended as a factual source about individuals.

Ethical Considerations

A Additional Analysis

The stratified results show broad but uneven gains from transcript-based retrieval and reasoning, with variation across fact type, gender, and country. The remaining modality gap depends on the strength of the text-only baseline as well as speech-side accuracy.

Fact type

  • Temporal facts gain +11.44 points on average, location +9.51, and relation +6.68; AF-Next-Think reaches 88.56% on Year, 86.66% on Location, and 83.17% on Relation.Relation facts remain hardest, in part because verification requires preserving role direction and subject–relation–object structure.Source: A Additional Analysis · p. 14, Table 8 · p. 15

Gender

  • Male-subject claims gain +10.57 points versus +9.25 for female-subject claims, but their remaining gap is larger: +7.74 versus +6.25.Across the four LALMs, final accuracy averages approximately 67.1% for male-subject claims and 65.4% for female-subject claims.Source: A Additional Analysis · p. 14, Table 8 · p. 15, Table 9 · p. 15

Country

  • US-subject claims have the largest average Lift (+9.92) and smallest remaining gap (+5.11), while England and India retain gaps of +11.41 and +12.78 despite high AF-Next-Think accuracies.Germany has the lowest average final accuracy and smallest average Lift (+8.07); its smaller +7.16 gap does not indicate better speech verification.Source: A Additional Analysis · p. 14, Table 8 · p. 15, Table 9 · p. 15

Fact type

  • Year facts have an average remaining gap of −2.96, while Location and Relation retain gaps of +13.32 and +11.76, respectively.AF-Next-Think surpasses its text-only baseline on Year, while the gap persists for the other two categories.Source: A Additional Analysis · p. 14, Table 9 · p. 15

Takeaway

  • Across slices, transcript-based RAG with CoT improves speech verification, but gains are uneven; fact type varies most, while gender effects are modest and model differences remain dominant.The thinking-tuned model generalizes more reliably than standard LALMs across factual, gender, and country slices.Source: A Additional Analysis · p. 15

B Additional Failure Modes

Output diagnostics show that some weak speech-verification results reflect formatting failures, but vanilla chain-of-thought prompting remains unreliable even when outputs are parseable. Qwen2-Audio-7B also collapses to one verdict in speech-only verification.

B Additional Failure Modes

  • Qwen2-Audio-7B predicts No for every speech-only C2 sample, yielding 49.2% accuracy; other near-50% pairs produce both verdict classes.Thus, constant-answer behavior alone does not explain the other near-random results.Source: B Additional Failure Modes · p. 15
  • 29.8% of Phi-4-multimodal C3 outputs are parseable, rising to 82.4% in C5; its below-random C3 accuracy partly reflects format non-compliance.Most model–setting pairs have parseability rates above 90%.Source: B Additional Failure Modes · p. 16
  • Qwen models remain approximately 99% parseable in C3, yet vanilla CoT does not improve on their speech-only C2 results; Audio-Flamingo-3 shows a similar pattern above 90%.The weak CoT results generally cannot be attributed to formatting failures, although Phi-4-multimodal has an additional model-specific format-following problem.Source: B Additional Failure Modes · p. 16

C Detailed Results for AF-next-think

This appendix reports category-wise results for Audio-Flamingo-next-think, describes the extraction prompts and deterministic negative-fact generation, and provides qualitative examples. It also specifies implementation details and the authors’ AI-use statement.

C Detailed Results for AF-next-think

D Atomic Fact Extraction Prompts

  • The extraction prompts request atomic, pronoun-free facts from the supplied biography, with separate prompts for temporal, location, and relation facts.Outputs are filtered to remove short facts and unresolved pronouns, while locally coreferential possessives are retained when the antecedent is named in the sentence.Source: D Atomic Fact Extraction Prompts · pp. 16–18
  • Incorrect counterparts are generated by deterministic perturbations: years shift within ±10, locations swap with same-category candidates, and relations use keyword or entity swaps.Sentences without a producible negative are dropped to preserve balanced correct and incorrect counts per celebrity.Source: D Atomic Fact Extraction Prompts · pp. 18–19

E Additional Qualitative Samples (Visual)

F Implementation Details

  • Experiments used PyTorch and Hugging Face Transformers, ran on three NVIDIA A6000 GPUs with 48 GB each, and report results averaged over three runs.Source: F Implementation Details · p. 19

G AI Use Statement

  • AI assistance was used only to polish manuscript writing and improve Figure 1’s visual presentation; the authors state that the ideas, experiments, analyses, and conclusions are their own.Source: G AI Use Statement · p. 19
Loading 2609.30227v1…