Source-linked AI summary
From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection
Mengzhe Geng, Yujia Lu, Patrick Littell, Manuela Kunz, Xie Chen
TL;DR
Speech-deepfake detectors often reduce each utterance to a score that cannot explain why borderline cases need different follow-up. This paper keeps passive, keyed-probe, retrieval, and profile evidence in an auditable record for late calibration, improving matched-set operating results while retaining reviewable provenance rather than claiming a stronger standalone detector.
Problem
Most detectors end with one utterance-level score, leaving the reasons behind borderline decisions unavailable for trust, deferral, or review.
Method
The paper stores passive, keyed-probe, retrieval, and speaker-profile cues with explicit disagreement coordinates, then applies late calibration.
Results
Late calibration over the full decision record reaches 8.43% EER on the 4,080-example matched subset, while the fixed retrieval-augmented rule lowers EER from 15.84% to 11.91%.
Takeaways & Limitations
The decision record preserves evidence behind surfaced utterances while still producing one operating score for thresholding and review.
Takeaways & Limitations
The best passive WavLM run reaches 6.71% EER, and the matched subset does not support a source-isolated or speaker-isolated zero-day claim.
Abstract
from arXiv · showhide
Speech deepfakes can mimic a speaker's voice convincingly enough to deceive listeners and automated systems. This has driven strong progress in speech deepfake detection, but most detectors still end with one score per utterance. That score is useful for ranking systems, yet it says little about why a borderline item should be trusted, deferred, or reviewed. Two utterances can fall in the same score band for different reasons, for example because passive and retrieval evidence disagree or because the keyed probe is unavailable. We ask whether the final decision can remain scalar without discarding that provenance. We answer this question with an auditable decision record that carries four aligned cues into a late calibration step: a passive detector score, a conditional keyed-probe score on a marked derivative, retrieval support, and a speaker-profile margin, together with explicit disagreement coordinates. On the 4,080-example ASVspoof 5 Track 1 matched subset, the fixed retrieval-augmented rule improves on retrieval-only evidence, reducing equal error rate (EER) from 15.84% to 11.91%, and late calibration over the full record reaches 8.43% EER. At a 33.75% review budget, the exposed cue union covers 82.85% of the calibrated model's errors. The best passive WavLM run still reaches 6.71% EER, so we do not present the decision record as a stronger standalone detector. Its contribution is to preserve the evidence behind each surfaced utterance while still producing one operating score for thresholding and review.
1 INTRODUCTION
The paper argues that a single detector score can hide why borderline speech-deepfake cases need different follow-up. It preserves four aligned evidence streams through late calibration, keeping a scalar operating score while retaining reviewable provenance.
- Motivation: Borderline utterances with similar detector scores can require different follow-up because their supporting cues differ.The distinction may reflect passive–retrieval conflict or an unavailable probe.
- Contribution: The decision record retains passive, probe, retrieval, and speaker-profile cues before final calibration.Calibration preserves the underlying evidence while producing one scalar decision score.
- Research question: The study evaluates whether late calibration improves operating decisions while preserving a usable basis for review on 4,080 matched examples.The matched subset contains examples for which all four cues are available.
- Scope: The paper treats the keyed probe as an auxiliary measurement and does not claim watermark robustness or attribution.The probe remains one evidence stream rather than replacing passive detection.
2 RELATED WORK
Related work establishes strong scalar passive baselines and motivates preserving semantic distinctions among passive, proactive, retrieval, and profile evidence. The paper positions its decision record as a calibrated, interpretable alternative to collapsing these streams into one score.
- Passive detection: Prior ASVspoof work provides comparable passive-detector benchmarks across generator families, channels, and postprocessing conditions.Frozen self-supervised systems, including WavLM, provide strong scalar references.
- Interpretability: The method responds to related work showing that ranking metrics alone are insufficient when score reliability and provenance also matter.Its distinguishing emphasis is calibrated and interpretable score fields.
- Probe semantics: A proactive watermark probe is meaningful only relative to its key, embedding rule, and threat model.The paper therefore treats the probe as an auxiliary measurement rather than a replacement for passive detection.
- Retrieval and profiles: Retrieval and profile matching address missing generator, channel, or speaker relationships in supervised training data.These methods supply support-set and speaker-context evidence distinct from passive waveform scoring.
3 METHOD
Each utterance becomes a fixed-format record containing passive, probe, retrieval, profile, and neighbor-closeness measurements. A late calibrator combines visible score features with explicit disagreement coordinates so thresholding and review retain the original evidence.
- Decision record: Each utterance record contains passive probability, keyed-probe score, retrieval vote, profile margin, and raw nearest-neighbor closeness.The retrieval vote uses k = 10 support neighbors.
- Evidence semantics: The streams are kept side by side because passive, probe, retrieval, and profile measurements have different semantics.The probe depends on its key, embedding rule, and threat model, while retrieval and profile provide support-set and speaker-context evidence.
- Late calibration: Late calibration operates on a visible score block and explicit disagreement coordinates rather than an early hidden fusion.The original fields remain available for thresholding, selective review, and inspection.
- Feature construction: The disagreement vector records |sp − sw| and |fpw − sr| alongside fixed passive–watermark, retrieval-augmented, and profile-augmented averages.These coordinates test whether visible conflicts add information beyond linear score fusion.
- Calibration protocol: The learned decision-record family uses cross-fit logistic calibration with absolute- or squared-gap parameterizations, while worked review analyses use absolute gaps.Only out-of-fold fitting learns the orientation of raw neighbor closeness.
- Evaluation aim: The method asks whether auxiliary streams reduce held-out-family errors while making borderline utterances easier to review.WavLM-large supplies the scalar-detector reference.
4 EXPERIMENTAL PROTOCOL
The experiments use a restrictive matched ASVspoof 5 Track 1 subset with family-held-out retrieval support, optional probe fields, and out-of-fold calibration. The protocol includes overlap audits, uncertainty analysis, and explicit limits on source-isolation claims.
- Data and splits: The primary matched subset contains 4,080 utterances from 705 speakers, with 3,215 bona fide and 865 spoof examples.It is formed by intersecting passive–probe and retrieval/profile branches while holding out synthesis families A09–A16.
- Protocol: The protocol uses family-held-out retrieval support, optional watermark probing, paired matched examples, and out-of-fold calibration.These are the four fixed protocol choices summarized for the main comparisons.
- Scope limitations: The matched subset is intentionally restrictive and does not establish equivalence to the full retrieval distribution or a source-isolated zero-day claim.The checks bound join effects on available retrieval scores but leave broader source and channel overlap as limitations.
- Retrieval setup: Retrieval and profile scoring use k = 10 nearest neighbors after support-set standardization, with held-out-family support removed before scoring.The audit found no held-out-family top-10 spoof neighbor or exact query, speaker, or audio-path reuse, but source independence remains coarse.
- Comparison design: The main matched comparison fixes one HuBERT retrieval branch rather than selecting the best retrieval-sweep point.This keeps the paired comparison distinct from sensitivity analyses.
- Metrics and uncertainty: The evaluation reports EER, minDCF, and ECE, with pooled EER defined using one threshold over evaluated examples.Paired class-stratified bootstrap intervals use 5,000 resamples.
- Comparison coverage: The matched comparison covers utterances for which passive, probe, retrieval, and profile measurements are simultaneously available.Lower values are better, and Table 3 isolates disagreement-coordinate effects beyond fixed combinations.
- Uncertainty limitation: Source-level resampling is not attempted because the matched subset exposes only one coarse source tag.Speaker-resampled intervals are reported separately.
5 RESULTS
On the 4,080-example matched subset, retrieval-led fusion improves operating performance, while explicit disagreement features support further calibrated gains and more interpretable review. The record remains useful for selective inspection even when passive WavLM detection is stronger standalone.
- Operating performance: 8.43% EER is reached by the learned cross-fit record, improving on 11.91% for the fixed retrieval-augmented rule while remaining above the best passive WavLM run at 6.71%.The fixed rule lowers pooled EER from 15.84% to 11.91%; the learned record restores calibration to 0.0709 ECE.
- Operating performance: 48.67% probe-only EER shows that the keyed probe contributes mainly as a disagreement cue rather than as strong standalone class evidence.Removing the probe raises pooled EER only slightly from 11.91% to 12.46%.
- Ablations and interpretation: 9.86% family-macro EER with profile evidence comes alongside 12.96% pooled EER and 0.3270 ECE, demonstrating a trade-off rather than uniform operating improvement.The paper therefore reads the matched comparison as retrieval-led improvement with preserved disagreement for later calibration and review.
- Ablations and interpretation: 8.67% EER for the no-watermark scalar expansion and 8.78% for fold shuffling indicate that the learned gain is not cleanly attributable to example-level watermark linkage.The authors retain the absolute-gap model because its disagreement coordinates are easier to interpret, despite the squared-gap model's 8.33% point estimate.
- Ablations and interpretation: 2.02% mean Fold EER reduction, from 9.23% to 7.21%, is concentrated across families rather than uniform, with A12 showing the largest improvement.Per-family changes range from -0.12% on A14 to +6.57% on A12; A11–A12 account for 1.46% of the mean reduction.
- Selective review: 4.87% residual error after deferring 10% is achieved by the learned record, versus 8.09% for the fixed retrieval-augmented rule and 12.45% for retrieval kNN.The fixed rule is slightly more precise at the same budget, but the learned record pulls more total mistakes toward inspection.
- Selective review: 82.85% of calibrated-rule errors are covered by the four-cue union at a 33.75% review budget, compared with 32.85% for passive-margin and 40.99% for retrieval-margin queues.The learned-margin queue reaches 88.66%, while the four-cue union retains explicit inspection reasons with 20.70% precision versus 22.15%.
- Selective review: 62.1% of covered errors trigger at least two cues, while single-cue cases are mostly near-threshold decisions.Worked examples include one retrieval-rescued spoof and one retrieval-driven failure.
6 CONCLUSION
The paper argues that keeping multiple evidence cues visible through late calibration preserves reviewable reasons while still producing an operating score. Its gains are concentrated in passive failure modes, and the keyed probe mainly exposes disagreement rather than acting as an independent detector.
- The decision record keeps passive, probe, retrieval, and profile cues visible until late calibration.
- The matched evaluation improves the operating score over fixed scalar fusion and supports review at a fixed inspection load.
- Most improvement comes from a small set of passive failure modes, especially A12 and A13, rather than uniformly across families.
- The keyed probe contributes mainly by exposing disagreement with passive and retrieval streams, not by independently detecting classes.
A ADDITIONAL EVIDENCE
The appendix collects supporting protocol, comparison, boundary-check, and stress-analysis material for the main text.
- The appendix collects protocol details supporting the main text.
- The appendix includes passive-reference comparisons supporting the main text.
- The appendix includes boundary checks and stress analyses supporting the main text.
A.1 MATCHED SUBSET AND OPERATING CONVENTION
The additional evidence documents the examples, score orientation, cost conventions, and coverage of the paper’s score subsets.
- The main-text examples, score orientation, and cost conventions are documented in Tables 4–6.
- Table 4 reports bona fide and spoof counts, broad corpus source-tag counts, and duration when referenced audio files are available.
- Table 5 defines the passive-score orientation gap and reports inverted EER using the deterministic score complement.
- Table 6 tests cost-orientation sensitivity by treating either bona fide speech or spoof speech as the target class.
A.2 PASSIVE-DETECTOR CONTEXT
The passive-detector context compares frozen SSL systems across matched, full-development, outside-corpus, and self-voice-conversion settings. WavLM provides the main matched passive reference, while training-size and duration controls contextualize the runs.
- Full-development SSL duration controls use 20k-per-class runs with 40,000 scored examples and compare crops with the same backbone’s 4 s median.
- Training-size controls aggregate completed 4 s runs by frozen SSL backbone and per-class training cap.
- Outside-corpus and self-voice-conversion results provide passive-detector context beyond the main matched comparison.
- The matched WavLM passive context uses the same 4,080 examples as the decision-record comparison.
- The lowest-EER WavLM-large run among 22 matched passive SSL runs has a 9.60% EER sweep median.
A.3 PROBE AND RETRIEVAL BOUNDARY CHECKS
The appendix bounds how the probe, retrieval, and profile streams are interpreted and checks retrieval-neighbor overlap under the available metadata. The matched join changes retrieval-score distributions, while finer-grained source and channel independence remains unresolved.
- Probe boundary: The probe channel is interpreted through raw statistics, thresholds, and operating counts, with separate conditions for probe availability and failure behavior.The reported controls distinguish keyed-probe evidence from authentication of an original utterance.
- Subset sensitivity: Matched-subset sensitivity analysis compares retained examples with matched-size bootstrap draws and omitted spoof examples to quantify retrieval-score shifts.Larger spoof-example retrieval scores indicate harder retrieval cases.
- Retrieval and profile boundary: Retrieval and profile neighbors are audited after excluding exact family, utterance, speaker-ID, and audio-path reuse under the available metadata.The audit uses held-out synthesis-family exclusion for spoof examples and reports coarse corpus-level source tags.
- Retrieval and profile boundary: Broader source and channel overlap remains a limitation because the available metadata does not establish finer-grained independence.Coarse source tags do not separate prompt, channel, or recording source.
A.4 FAMILY, CALIBRATION, AND STRESS EVIDENCE
The appendix shows that pooled gains concentrate in a small family subset, while calibration and cue-union analyses characterize where the decision record helps inspection. Active selection and attack-stress results remain separate from the central matched-subset claim.
- Family evidence: The pooled gain is concentrated in a small subset of families, and excluding A12 and A13 largely removes the pooled advantage.Most of the net reduction comes from bona fide corrections in those families.
- Review evidence: The four-cue union is intended for triage rather than optimizing a single ranking rule, and it is compared with scalar review queues at the same load.Cue analysis uses the same EER-threshold convention as the retrospective selective-risk analysis.
- Review evidence: Figure 6 presents calibrated decision-record examples with passive, probe, retrieval, and profile scores plus status, watermark, calibration, neighbor, disagreement, and review fields.The examples use the calibrated operating threshold, with the fixed score shown as a reference.
- Calibration evidence: The equal-width calibration view places most miscalibration near the upper score range, with equal-mass bins providing a sensitivity check.The calibration-bin analysis also reports fold-isotonic ECE and sparse-bin counts.
- Scope boundaries: Active selection changes ordering across label budgets: disagreement is best at the two smallest budgets, while random selection is best at the two largest.The study therefore treats active selection as a separate experimental-design question.
- Scope boundaries: Attack stress separates perturbations near the clean operating range from model-specific failures, with noise, aggressive quantization, and 2.5 kHz highpass producing the largest failures.Attack sensitivity is kept outside the main matched-subset claim.