PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolDF: Tool-Integrated Reasoning for Mixed-Authenticity Audio Deepfake Detection

Taewoo Kim, Young Han Lee, Nam In Park, Chanwoo Kim

arXiv:2609.03620v1eess.AScs.AIcs.SD

TL;DR

Mixed-authenticity audio combines genuine and manipulated cues across temporal regions or overlapping sources, challenging clip-level single-domain detection and requiring evidence localization. ToolDF uses a supervised tool-integrated ALLM orchestrator that adaptively selects separation and domain experts, achieving the best overall composite-type detection performance with interpretable localized evidence. The paper also releases a benchmark for mixed-authenticity audio deepfake detection.

  • Problem

    Mixed-authenticity audio deepfake detection is underaddressed because existing clip-level, single-domain methods may miss localized manipulations and mixed source evidence.

  • Method

    ToolDF trains an ALLM with supervised tool-use trajectories to analyze audio structure, selectively separate overlapping sources, route components to domain-specific experts, and aggregate their evidence.

  • Results

    ToolDF achieves the best overall performance on composite-type detection while localizing interpretable evidence to temporal regions and acoustic sources.

  • Takeaways & Limitations

    ToolDF supports adaptive and interpretable audio deepfake detection for complex acoustic scenes through component-level evidence aggregation.

  • Takeaways & Limitations

    ToolDF depends on external separation modules and domain-specific detectors, so their errors and coverage constrain final performance.

Abstract

from arXiv · show

Audio deepfake detection is commonly formulated as clip-level binary classification of single-domain audio. However, real-world manipulated audio can exhibit mixed authenticity, where genuine and manipulated cues coexist across temporal transitions, overlapping sources, or both. This setting requires not only detecting manipulated audio but also localizing the components that provide evidence for the decision. We propose ToolDF, a tool-integrated reasoning framework for mixed-authenticity audio deepfake detection. ToolDF employs an audio large language model as an orchestrator trained with supervised tool-use trajectories. It adaptively analyzes the audio scene, selectively performs source separation, routes components to domain-specific experts, and aggregates their evidence into an interpretable verdict. We further introduce a mixed-authenticity ADD benchmark covering temporal transitions, acoustic overlaps, and hybrid mixtures. Experimental results show that ToolDF achieves the best overall performance on composite-type detection, achieving macro-F1 gains of 3.72 and 14.39 points over the strongest monolithic baseline and a fixed pipeline, respectively, while providing interpretable evidence localized to temporal regions and acoustic sources. Our source code and dataset are publicly available online.

1 Introduction

The paper introduces mixed-authenticity audio deepfake detection, where genuine and manipulated cues coexist across time or sources, and proposes ToolDF to analyze, route, and aggregate evidence adaptively.

  • Motivation: Mixed-authenticity detection addresses audio inputs containing genuine and manipulated cues across temporal transitions, overlapping sources, or both.The task requires identifying not only whether audio is manipulated but also which temporal segments or acoustic sources support the decision.
  • Motivation: Existing clip-level classifiers, domain-specific detectors, fixed pipelines, and holistic ALLM classifiers are limited by localized manipulations, out-of-domain sources, artifacts, or opaque evidence.The paper motivates adaptive routing because separation can be unnecessary or artifact-inducing, while direct ALLM prediction does not exploit specialized detectors.
  • ToolDF: ToolDF uses an ALLM orchestrator to analyze audio structure, selectively separate overlapping sources, route components to domain-specific detectors, and aggregate responses into an interpretable verdict.Its expert coverage includes speech, singing, music, and environmental sound detectors.
  • ToolDF: Supervised fine-tuning on structured tool-use trajectories trains structural analysis, tool selection, response interpretation, and evidence aggregation rather than only final-label prediction.The trajectories provide intermediate supervision for the model’s reasoning and tool-use process.
  • Evaluation: ToolDF achieves the best overall performance on composite-type detection while providing interpretable traces that identify contributing segments, sources, and detector outputs.The benchmark covers single-type and composite-type manipulation scenarios, including temporal segments and overlapping sources.

2 Related Work

Prior ADD work largely assumes a dominant acoustic domain, while source separation and ALLMs address parts of composite-audio analysis but leave fixed processing and black-box limitations. ToolDF combines expert tools with supervised orchestration to reason over mixed-authenticity inputs.

  • Audio Deepfake Detection: Traditional ADD methods expanded from speech to singing, music, and environmental sound but generally assume each clip belongs to one dominant acoustic domain.This single-domain assumption limits direct application to composite audio inputs.
  • Source Separation: Source separation can isolate source-specific manipulation cues, but applying it to every input may introduce artifacts in signals that do not require separation.Recent work therefore motivates adaptive rather than uniformly fixed separation.
  • ALLMs and Tool-Integrated Reasoning: Direct ALLM binary classification can operate as a black box and fail to exploit domain-specific detectors.Tool-augmented models instead invoke external modules and integrate intermediate responses.
  • ALLMs and Tool-Integrated Reasoning: ToolDF trains an ALLM with supervised tool-use trajectories to orchestrate analysis of mixed-authenticity inputs, route components to expert detectors, and aggregate their outputs.This operationalizes tool-integrated reasoning for audio deepfake detection.

3 Methodology

ToolDF formulates mixed-authenticity detection as structured reasoning over heterogeneous audio components, using an orchestrator to analyze, route, execute tools, and aggregate evidence into a clip-level verdict.

  • 3.1 Task Formulation: Mixed-authenticity audio is represented as heterogeneous components with content types, support regions, and component-level authenticity labels.The standard single-domain task is the special case K = 1, while mixed-authenticity detection also identifies evidence supporting the clip-level decision.
  • 3.2 ToolDF Framework: ToolDF uses an audio large language model as an orchestrator for audio understanding, tool-use planning, localized execution, and evidence aggregation.The structured trajectory links component identification, expert routing, localized tool calls, and the final verdict.
  • 3.2 ToolDF Framework: The orchestrator identifies temporal segments and acoustic sources, then conditionally invokes source separation when overlapping sources are detected.The provided rollout illustrates analysis of speech, singing, background sound, and environmental noise before planning tool calls.
  • 3.2 ToolDF Framework: Localized tool execution routes temporal segments or separated sources to selected experts, whose component-level predictions provide evidence for the final decision.Tool observations are inserted into the trajectory as binary authenticity predictions with normalized confidence scores.
  • 3.2 ToolDF Framework: Evidence aggregation returns fake when any component prediction is fake and otherwise returns real.This early-fail rule connects component-level evidence to the clip-level authenticity verdict.
  • 3.3 Supervised Trajectory Learning: ToolDF is trained with supervised fine-tuning on structured trajectories spanning audio understanding, planning, tool calls, evidence summaries, and final decisions.Ground-truth component annotations populate the audio-understanding blocks, plans, localized calls, tool observations, evidence summaries, and verdicts; optimization covers orchestrator-produced tokens.

4 Mixed-Authenticity ADD Benchmark

The benchmark extends audio deepfake detection beyond isolated domains by combining public corpora across speech, singing, music, and environmental sound, with single-type and composite-type evaluations.

  • The benchmark combines public audio deepfake datasets from multiple acoustic domains to evaluate heterogeneous audio scenes.Existing benchmarks typically focus on a single content type and do not sufficiently test components with different authenticity cues.
  • Its source corpora include ASVspoof2019 for speech, CtrSVDD for singing, FakeMusicCaps and MusicCaps for music, and EnvSDD for environmental sounds.
  • The benchmark constructs both isolated single-type and composite-type evaluation settings from authentic and synthetic domain-specific subsets.

5 Experiments

The experiments evaluate ToolDF and baselines across single-type and composite-type settings using macro-F1, with specialized experts and a fixed separation-based pipeline as comparison points.

  • The benchmark evaluates both conventional single-type and proposed composite-type settings using source-corpus splits, with a MusicCaps exception to prevent data leakage.
  • ToolDF uses source separation for overlapping vocal-background mixtures and routes separated or target segments to domain-specific XLSR-AASIST detectors.
  • The evaluation compares end-to-end monolithic models with component-level frameworks, including a fixed pipeline that always performs source separation.
  • The Fixed Pipeline routes separated foreground and background streams to specialized vocal and non-vocal detectors, using an early-fail fake decision rule.
  • Macro-F1 averages the F1 scores of real and fake classes equally, preventing the majority class from dominating evaluation.

6 Results

ToolDF improves composite-type detection over monolithic and fixed-pipeline baselines while maintaining structured reasoning and strong localization across temporal and source-mixing scenarios.

  • 6.2 Evaluation on Mixed-Authenticity ADD: Single-domain strength does not guarantee composite generalization: XLSR-AASIST’s single-type S-Avg. is 86.57, but the best composite C-Avg. among single-type-trained baselines is 57.83.
  • 6.2 Evaluation on Mixed-Authenticity ADD: 81.89 C-Avg. is ToolDF’s best composite-input result, exceeding XLSR-AASIST by 3.72 points and reaching 91.21 on temporal transitions.ToolDF also achieves 76.81 on hybrid mixtures, while ALLM4ADD’s composite C-Avg. drops to 65.04.
  • 6.2 Evaluation on Mixed-Authenticity ADD: ToolDF reaches 81.89 C-Avg. versus 67.50 for the Fixed Pipeline, whose mandatory separation yields 78.55 on acoustic overlaps.ToolDF adaptively decides when to separate, whereas separation may be unnecessary or introduce artifacts on other composite types.
  • 6.2 Evaluation on Mixed-Authenticity ADD: ToolDF provides explicit tool-execution trajectories that expose the segments, sources, and detector outputs contributing to each decision.Its performance is only 0.96 points below the Oracle variant.
  • 6.3 Ablation and Localization Analysis: The full ToolDF model performs best across ablations while maintaining parsing rates close to 100%.Removing planning causes the largest strict-F1 drop, while removing description reduces C2 parsing rate to 23.69.
  • 6.3 Ablation and Localization Analysis: ToolDF achieves event-level macro-F1 scores of 94.24 on temporal transitions and 87.62 on hybrid mixtures.Across single-type domains, segment- and event-level macro-F1 scores range from 93.64 to 99.97.

7 Conclusion

ToolDF combines audio-scene analysis, adaptive expert routing, and component-level evidence aggregation for mixed-authenticity audio deepfake detection. It achieves strong composite-type performance while localizing the temporal and acoustic evidence behind its decisions.

  • ToolDF uses an audio large language model to adaptively route heterogeneous audio components to specialized expert detectors.It aggregates component-level evidence into an interpretable clip-level verdict.
  • ToolDF achieves the best overall performance on composite-type detection while providing fine-grained localization of supporting temporal regions and acoustic sources.

Limitations

ToolDF’s performance depends on the reliability and coverage of external tools, while the benchmark may not fully represent real-world manipulated audio. The evaluation also excludes conventional partial-spoof benchmarks.

  • Errors in source separation or domain-specific detector outputs can propagate to ToolDF’s final decision.Overall performance is bounded by the reliability and coverage of its underlying expert models.
  • The benchmark’s composition from existing public datasets may not capture the diversity and distributional complexity of real-world manipulated media.Its temporal-transition and acoustic-overlap rules may also be simpler than those in real-world or adversarially edited audio.
  • The evaluation excludes conventional partial-spoof benchmarks focused on localized manipulations within a single speech utterance.Extending ToolDF with a partial-spoof detector remains future work.

A Mixed-Authenticity Benchmark Construction

The benchmark converts single-source clips into structured component annotations and uses them to synthesize composite mixed-authenticity examples. These annotations supervise both final authenticity decisions and intermediate tool-selection reasoning.

  • The construction pipeline first annotates single-source clips and then synthesizes composite examples for the C1, C2, and C3 settings.
  • Each component is annotated with a content type, temporal support region, and real-or-fake authenticity label.Content types include speech, singing, music, and environmental sound.
  • Single-source captions describe observable acoustic properties while excluding authenticity-related information to avoid label leakage.The captions are generated with Qwen3-Omni-30B-A3B-Captioner.
  • Component captions and metadata form the audio_understanding block of supervised ToolDF trajectories.The supervision covers acoustic structure as well as the final clip-level authenticity label.
  • Vocal support regions are estimated with WebRTC-VAD, whereas non-vocal components use the full clip duration.This distinguishes localized vocal activity from continuous background components.

A.2 Composite Mixture Construction

Composite examples cover temporal transitions, acoustic overlaps, and hybrid mixtures across vocal and non-vocal domains. Their construction preserves component support regions and authenticity labels for structured evaluation.

  • The benchmark combines components from different acoustic domains while preserving their support regions and authenticity labels.
  • Temporal Transition (C1): C1 creates temporal transitions by concatenating speech and singing in both orders.It evaluates real-to-real, real-to-fake, fake-to-real, and fake-to-fake authenticity combinations.
  • Acoustic Overlap (C2): C2 mixes a foreground speech or singing component with a music or environmental-sound background.The vocal support region remains localized while the background spans the full mixture duration.
  • The captioning prompt restricts outputs to observable acoustic traits and excludes authenticity-related information.This design is intended to avoid label leakage during component description.
  • Hybrid Mixture (C3): C3 combines a speech-to-singing transition with an overlaid music or environmental-sound background.This setting requires reasoning over both temporal segmentation and source-level separation.

A.3 Dataset Statistics

The benchmark contains 973,649 examples, including 379,900 composite examples across three mixed-authenticity settings. The accompanying materials define these configurations and document the ToolDF orchestrator’s structured tool-use protocol.

  • Dataset Statistics: 973,649 total examples include 379,900 composite examples from the C1, C2, and C3 settings.The composite pool is fake-heavy because a clip is labeled fake when at least one constituent component is fake.
  • ToolDF Prompt Configuration: The exact system prompt is provided for reproducibility during supervised fine-tuning and inference of the ALLM orchestrator.It enforces structured generation and prevents fabricated tool results or deviations from the prescribed reasoning protocol.
  • Composite Configurations: C1 covers temporal speech–singing transitions, C2 covers vocal–non-vocal acoustic overlaps, and C3 covers hybrid mixtures combining both.
  • ToolDF Prompt Configuration: The ToolDF system prompt requires the orchestrator to use provided tools to judge authenticity rather than deciding independently.
Loading 2609.03620v1…