Source-linked AI summary
SEAR: Spoofing Evidence-Grounded Audio Reasoning Benchmark for Audio Language Models
Rong Wan, Suliu Qin, Jiaxi Li, Wei Xie, Wenwu Wang, Xiaolong Han, Lu Yin, Xilu Wang
TL;DR
Existing ADD benchmarks evaluate verdicts or rationale plausibility without verifying the acoustic evidence behind them. This paper introduces SEAR and BAEA, which evaluates ALMs on evidence-grounded ADD and equips a frozen ALM with controlled acoustic tools. Six-ALM experiments reveal a gap between plausible rationales and verifiable evidence reasoning, while BAEA-FIXED improves performance and misleading evidence degrades detection and grounding.
Problem
Existing ADD benchmarks assess verdicts or rationale plausibility but do not evaluate whether ALMs identify and quantify fine-grained acoustic anomalies and use them consistently.
Source: §1 · p. 1Method
Results
Six-ALM experiments reveal a gap between plausible rationales and verifiable evidence reasoning; BAEA-FIXED improves performance, while misleading evidence degrades detection and grounding.
Source: §1 · pp. 1–2, §4.4 · p. 4Takeaways & Limitations
The findings highlight the importance of reliable acoustic evidence acquisition and use for deepfake detection and rationale grounding.
Source: §1 · p. 1, §4.4 · p. 4
Abstract
from arXiv · showhide
Audio language models (ALMs) are increasingly used for audio deepfake detection (ADD), yet existing benchmarks assess their verdicts or rationale plausibility without verifying the underlying acoustic evidence. To address this issue, we first introduce spoofing evidence-grounded audio reasoning (SEAR), a four-task AQA benchmark to evaluate ALM-based ADD through acoustic evidence identification and quantification, deepfake detection, and forensic rationale generation. We further propose a bona-fide-based acoustic evidence agent (BAEA), which equips a frozen ALM with controlled acoustic tools under \textsc{fixed} or \textsc{adaptive} evidence-acquisition policies. Experiments with six ALMs reveal a clear gap between plausible rationales and verifiable acoustic evidence reasoning, while BAEA-\textsc{Fixed} improves final verdicts and forensic rationales on both evaluation partitions. Controlled interventions further show that misleading evidence degrades both detection and grounding performance.
1. INTRODUCTION
Existing ADD evaluations do not establish whether ALM verdicts and rationales use verifiable acoustic anomalies. SEAR evaluates that evidence-grounding problem, and BAEA adds controlled acoustic analysis to a frozen ALM; experiments show plausible rationales can exceed evidence reasoning, while BAEA-FIXED improves performance.
1. INTRODUCTION
- Prior ADD benchmarks assess detection outcomes or rationale plausibility, but do not test whether ALMs identify and quantify fine-grained anomalies and use them consistently.The paper asks whether ALMs can identify and quantify signal-level anomalies and use that evidence to support forensic rationales and deepfake verdicts.Source: §1 · p. 1
- SEAR is a four-task AQA benchmark that evaluates acoustic evidence identification and quantification, deepfake detection, and forensic rationale generation.It is constructed through analysis of 35 acoustic features, controlled AQA generation, and quality control.Source: §1 · p. 1, Figure 1 · p. 2
- BAEA equips a frozen ALM with controlled signal analysis and a bona-fide reference distribution, using FIXED or ADAPTIVE evidence acquisition.FIXED ranks deviations across all 35 features; ADAPTIVE lets the ALM select features before tool execution.Source: §1 · p. 1
- Experiments with six ALMs reveal a gap between plausible rationales and reasoning with verifiable acoustic evidence; misleading evidence degrades detection and grounding.BAEA-FIXED improves evidence identification and quantification, rationale grounding, and deepfake detection over its backbone.Source: §1 · pp. 1–2
2. OUR BENCHMARK
SEAR evaluates ADD through four tasks spanning verdicts, acoustic evidence, and forensic rationales. It constructs reference evidence and questions from acoustic features and benchmark audio, then applies automated and human quality checks.
2.1. Tasks Design
- SEAR separates evaluation into binary deepfake verdicts (T1), cue identification (T2), acoustic measurement (T3), and evidence-grounded forensic rationale generation (T4).T2 and T3 are evaluated against constructed reference acoustic evidence; T4 rationales must explain verdicts using verifiable signal evidence.Source: §2.1 · p. 2
2.2. Audio Sources and Acoustic Feature Extraction
- Audio comes from ASVspoof 2019 LA and 2021 LA, while 35 utterance-level features capture spectral, cepstral, energy, temporal, and source-related characteristics.The feature set includes 20 time-averaged MFCCs and 15 complementary measurements.Source: §2.2 · p. 2
2.3. Reference Evidence and AQA Generation
- Reference anomalies are screened using effect size and folded ROC AUC, then flagged feature-value pairs are ranked by balanced accuracy to construct task answers.Templates and distractors are controlled across T1–T3; T4 turns triggered findings into an evidence-grounded rationale, without exposing partition labels or attack identities to evaluated models.Source: §2.3 · pp. 2–3
2.4. Quality Control
- Generated AQAs undergo automated checks for answer uniqueness, numerical traceability, and cross-record consistency, plus review by six humans of a stratified subset of 5,400.Review covers correctness, ambiguity, evidence consistency, and linguistic clarity.Source: §2.4 · p. 3
3. METHOD
BAEA augments a frozen ALM with controlled acoustic measurements referenced to bona-fide training audio. It offers fixed or adaptive evidence acquisition and constrains downstream decisions and rationales to measured evidence.
3. METHOD
- BAEA computes 35 pre-registered features and estimates each feature’s median and median absolute deviation exclusively from bona-fide 19LA training audio.The robust deviation indicates direction and acoustic rarity; a large deviation is treated as evidence, not direct proof of spoofing.Source: §3 · p. 3
- FIXED measures all 35 features and returns ranked, category-balanced deviations, whereas ADAPTIVE lets the ALM select features under controller-enforced validity and evidence-budget constraints.The ALM can invoke only whitelisted tools and cannot modify their outputs.Source: §3 · p. 3
- For T2 and T3, BAEA selects the minimum-error option against controlled measurements; for T1 it uses audio and evidence, and for T4 it retains only supported ALM interpretation.T1 spoof scores are averaged across original and A/B-swapped option orders; T4 excludes unmeasured features, unsupported values, and attack metadata.Source: §3 · p. 3
4. EXPERIMENTS
Across SEAR’s four tasks, six ALMs struggled with fine-grained evidence despite strong rationale scores; BAEA-FIXED improved on its backbone, while misleading evidence degraded detection and grounding.
4.1. Experimental Setup
- The evaluation covers six ALMs and five metrics—F1, EER, ACC, B-F1, and Ref-G—and samples 2,000 audio items per partition, totaling 68,004 AQAs.In a fixed-template ablation on 200 19LA Evaluation recordings, T1 EER varied by up to 15.0 points, so reported T1 EER values are position-ensembled.Source: §4.1 · p. 3
4.2. ALMs’ Performance on SEAR
- T1 best F1 spans 45.07–52.47%; T2 stays near 25% random-choice accuracy and T3 at or below 25.79%, while T4 B-F1 reaches 83.26–85.17%.The disparity indicates that plausible rationale generation does not guarantee reliable identification or quantification of the supporting evidence.Source: §4.2 · pp. 3–4, Table 1 · p. 4
- Oracle-assisted T2/T3 context raised T4 Ref-G by 0.635–1.313, changed B-F1 by at most 0.79 points, and did not consistently improve T1.Oracle retains model answers and is not a pure gold-only upper bound.Source: §4.2 · p. 4, Table 2 · p. 4
4.3. Tool-Augmented Acousitc Evidence Reasoning
- On 19LA, FIXED cut T1 EER from 36.34% to 24.53%, and on 21LA from 36.43% to 33.54%; ADAPTIVE underperformed FIXED on T1 and T4 in both partitions.FIXED also improved T2/T3 and T4 Ref-G over its backbone; Tool-only reproduced T2/T3 decisions but had dataset-dependent T1 performance.Source: Table 3 · p. 4, §4.3 · p. 4
4.4. Effect of Evidence Quality
- Swapped evidence raised T1 EER by 30.28 and 18.08 points and reduced Ref-G by 0.605 and 0.755 on the two evaluation partitions.Matched evidence improved T4 Ref-G but did not consistently improve T1.Source: §4.4 · p. 4, Table 4 · p. 4
5. CONCLUSION
The conclusion presents SEAR as a four-task ADD benchmark and reports a gap between plausible rationales and fine-grained acoustic evidence, alongside BAEA’s bona-fide-based grounding approach.