PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Temporal Localisation of Audio Deepfakes in Multi-Speaker Conversations

Soumyadeep Roy

arXiv:2609.10051v1cs.SDcs.LG

TL;DR

Voice-cloning fraud can replace only short spans of genuine multi-speaker conversations, but utterance-level detectors cannot localize those injections. This paper formalizes temporal deepfake localization and introduces a training-free hysteresis pipeline around frozen detectors, achieving strong temporal localization across backbones while approaching supervised performance.

  • Problem

    Utterance-level detectors emit one binary label per clip and cannot identify surgically injected synthetic speech within mixed-content conversations.

  • Method

    A training-free five-stage pipeline wraps a frozen binary detector with temporal metrics, window-score processing, and a calibration-selected two-threshold hysteresis decoder.

  • Results

    Across three frozen detectors, the pipeline reaches t-IoU 0.90, comes within approximately 0.04 t-IoU of a trained localiser, and achieves below-6% false alarms on genuinely real speech.

  • Takeaways & Limitations

    The paper establishes a first zero-shot TDLMC baseline and a reusable benchmark, with residual false alarms attributed to backbone behavior rather than the decoder.

  • Takeaways & Limitations

    The benchmark uses constructed rather than live dialogue, and the method remains a baseline with approximately 3.5-second boundary displacement and an inflated global false-alarm rate.

Abstract

from arXiv · show

Voice-cloning fraud increasingly relies on surgical injection: a genuine conversation in which only one or two sentences are replaced by synthetic speech. Utterance-level deepfake detectors emit a single real/fake label per clip and cannot report where the synthetic speech lies. We formalise this as Temporal Deepfake Localisation in Multi-Speaker Conversations (TDLMC), show that equal error rate and min-DCF are ill-posed once a file contains both classes, and propose temporal metrics for this regime. Our contribution is a training-free five-stage pipeline that wraps a frozen binary detector and adds segment-level output with no retraining, using a two-threshold hysteresis finitestate-machine decoder to turn noisy window scores into coherent intervals. On 180 constructed multi-speaker conversations from ASVspoof 5, the system attains temporal intersection-over-union 0.90, temporal detection rate 0.95, and MS-DCF 0.26 with a strong backbone, and its false-alarm rate on genuine speech is below 6%, falling under 2% on genuine real multi-speaker dialogue (AMI). Under an identical pipeline, a trained localiser improves temporal IoU by only about 0.04, bounding the cost of forgoing supervision. Evaluated across three frozen detectors under one decoder whose constants are selected on a held-out calibration split, and with a controlled analysis attributing the residual false-alarm rate to a backbone domain gap rather than to the decoder, this provides the first zero-shot baseline and a reusable benchmark for TDLMC.

I. INTRODUCTION

Voice-cloning fraud can replace only decisive sentences inside otherwise genuine conversations, while utterance-level detectors output one label for an entire clip. The paper formalises this localisation problem and introduces a training-free baseline using frozen detectors.

  • I. INTRODUCTION: Surgical voice-cloning attacks replace decisive sentences while leaving the surrounding conversation genuine.This makes whole-clip binary classification insufficient for identifying the injected span.
  • I. INTRODUCTION: Prior localisation methods train on temporally labelled data under single-speaker or supervised assumptions.The paper distinguishes its setting from partial-spoof detection, W-TDL, and LENS-DF.
  • I. INTRODUCTION: The paper formalises TDLMC and shows utterance-level EER and min-DCF are ill-posed for mixed-content files.Its contributions also include temporal metrics, a training-free pipeline, and a controlled failure-mode analysis.

A. Conversation construction

The benchmark constructs controlled 60-second multi-speaker conversations from normalized ASVspoof 5 utterances, varied codecs, and four real/fake patterns. It provides timestamped ground truth while representing a controlled probe rather than natural live dialogue.

  • Conversation construction: 180-second conversations combine four normalized speaker utterances with crossfades and one of seven codecs.A JSON manifest records ground-truth timestamps in samples.
  • Conversation construction: Four patterns—RRRR, FFFF, RFFR, and RFRF—probe false alarms, detection ceilings, fraud injection, and rapid switching.
  • Conversation construction: The corpus-agnostic protocol concatenates utterances rather than modeling natural live-dialogue turn-taking.Results therefore characterize localization under controlled conditions, not field fraud.

B. Metrics for mixed-content files

Mixed-content conversations invalidate file-level utterance metrics because one file contains both genuine and fake speech. The paper instead evaluates frame- and segment-level temporal localization using complementary overlap, detection, false-alarm, boundary, and cost measures.

  • Metrics for mixed-content files: EER and min-DCF are ill-posed for files containing both genuine and fake speech because no single operating point captures their temporal miss/false-alarm trade-off.They remain defined at frame level, which the evaluation uses.
  • Metrics for mixed-content files: Temporal IoU measures overlap between predicted and true fake unions, while TDR and TFAR measure fake-second recall and real seconds falsely flagged.
  • Metrics for mixed-content files: Segment boundary displacement averages matched prediction-to-ground-truth endpoint errors while reporting unmatched false positives and misses separately.This prevents easy segments from masking boundary or detection failures.
  • Metrics for mixed-content files: Segment precision, recall, and F1 are additionally reported at t-IoU thresholds 0.3, 0.5, and 0.7 because union overlap can hide merged intervals.
  • Metrics for mixed-content files: MS-DCF pools second-level statistics globally with miss cost 1.0 and false-alarm cost 0.5, while degenerate all-real or all-fake patterns are marked N/A where appropriate.

III. ZERO-SHOT LOCALISATION PIPELINE

The zero-shot pipeline converts waveform windows scored by a frozen detector into coherent fake intervals without temporal labels. Overlap-add scoring, smoothing, hysteresis decoding, and calibration-selected post-processing improve test localization while preserving a training-free design.

  • III. ZERO-SHOT LOCALISATION PIPELINE: The five-stage pipeline uses a frozen backbone and no temporal labels to produce fake intervals.
  • III. ZERO-SHOT LOCALISATION PIPELINE: Window scores from 2-second windows advanced by 1-second hops are overlap-added onto a 10-fps timeline, whose independent temporal resolution remains bounded by the hop.
  • III. ZERO-SHOT LOCALISATION PIPELINE: Median and Gaussian smoothing feed a two-threshold FSM that enters FAKE at 0.55 and returns to REAL below 0.35.The dead band absorbs oscillation near boundaries.
  • III. ZERO-SHOT LOCALISATION PIPELINE: Post-processing removes intervals shorter than 2.0 seconds and merges gaps shorter than 1.5 seconds.
  • III. ZERO-SHOT LOCALISATION PIPELINE: 0.904 test t-IoU and 0.258 MS-DCF are achieved by the calibrated configuration, improving from 0.893 and 0.290 with the hand-set baseline.The decoder constants were selected on the calibration split, leaving the test split untouched.

IV. RESULTS AND DISCUSSION

The zero-shot pipeline performs strongly on constructed conversations, transfers across real dialogue with low genuine-speech false alarms, and approaches supervised localisation while residual errors primarily reflect backbone and boundary limitations.

  • Constructed benchmark: 0.053 mean TFAR and 0.040 pooled TFAR are obtained on conversations containing genuine speech, while FFFF’s global false-alarm figure is a labelling artifact.On the speaker-disjoint subset, t-IoU is 0.87, TDR is 0.93, and TFAR is 0.22; this measures within-partition stability rather than cross-corpus generalisation.
  • Multi-backbone: 0.90 t-IoU is achieved by DF Arena 1B, compared with 0.64 for wav2vec2-XLSR and 0.49 for AASIST under the identical decoder.The decoder transfers across backbones, but AASIST degenerates because its fixed input length forces 2 s windows to be tiled.
  • Comparison to a trained localiser: 0.909 t-IoU from a trained localiser exceeds DF Arena 1B’s 0.871 on a rebuilt instance, while the zero-shot system retains lower MS-DCF, 0.220 versus 0.280.The zero-shot pipeline comes within approximately 0.04 t-IoU of the trained localiser without temporal supervision.
  • Real conversational validation: 0.017 and 0.004 TFAR are measured on genuine AMI dialogue with DF Arena 1B and wav2vec2-XLSR, respectively, both below 2% of genuine conversational seconds.The decoder and calibration are locked from the constructed benchmark, and AMI is not refit.
  • False-alarm attribution: 0.27 TFAR remains with oracle window scores versus 0.28 for real scores, implicating window/boundary structure and adjacent genuine segments rather than the hysteresis decoder.The backbone’s clean-genuine window EER is 0.04, but perfect oracle scores do not remove the false alarms.
  • Metric decomposition and limitations: 3.53 s global matched SBD and segment-F1 falling from 0.50 at t-IoU 0.3 to 0.31 at 0.7 expose boundary-precision limits hidden by union t-IoU.The constructed benchmark is not live dialogue, and short injected segments from out-of-distribution generators remain bounded by the backbone domain gap.

V. CONCLUSION

The paper presents a zero-shot framework for locating synthetic speech in multi-speaker conversations and reports a reusable benchmark with comparisons across backbones, supervision levels, and decoder stages.

  • The framework formalises temporal deepfake localisation and provides a zero-shot baseline with released benchmark resources.It targets mixed-content conversations where utterance-level metrics are ill-posed and introduces temporal evaluation metrics.
  • The pipeline wraps a frozen detector and uses hysteresis decoding to produce coherent fake intervals without retraining.Figure 2 illustrates reconstructed confidence over time with separate entry and exit thresholds.
  • The evaluation is organised around a primary global-results table, a multi-backbone comparison, a trained-localiser comparison, and real-dialogue validation.The supplied tables cover the calibration-selected primary decoder, identical-decoder backbone comparisons, supervised comparison, and AMI validation.
  • A decoder ablation isolates the contributions of smoothing, hysteresis, and refinement under a fixed hand-set configuration.The ablation caption identifies median smoothing as the largest gain and reports a coherence trade-off for hysteresis that refinement recovers.
Loading 2609.10051v1…