Source-linked AI summary
ToolDF: Tool-Integrated Reasoning for Mixed-Authenticity Audio Deepfake Detection
Taewoo Kim, Young Han Lee, Nam In Park, Chanwoo Kim
TL;DR
Mixed-authenticity audio combines genuine and manipulated cues across temporal transitions or overlapping sources, creating a need to detect manipulation and localize its evidence. ToolDF uses an audio-language-model orchestrator to analyze audio structure, route components to expert detectors, and aggregate evidence; it achieves the best overall performance on composite-type detection while localizing supporting regions and sources.
Problem
Mixed-authenticity audio combines genuine and manipulated cues across temporal transitions or overlapping sources, requiring detection of manipulation and localization of the evidence.
Source: §1 · p. 1Method
ToolDF uses an audio large language model as an orchestrator to analyze audio structure, selectively invoke expert detectors and source separation, and aggregate evidence, trained with supervised tool-use trajectories.
Source: §1 · p. 2Results
Takeaways & Limitations
The findings highlight TIR's potential as a framework for interpretable and adaptive audio deepfake detection in complex acoustic scenes.
Source: §7 · p. 9Because the benchmark composes existing public datasets, it may not fully reflect real-world diversity, editing artifacts, source interactions, or distributional complexity.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Audio deepfake detection is commonly formulated as clip-level binary classification of single-domain audio. However, real-world manipulated audio can exhibit mixed authenticity, where genuine and manipulated cues coexist across temporal transitions, overlapping sources, or both. This setting requires not only detecting manipulated audio but also localizing the components that provide evidence for the decision. We propose ToolDF, a tool-integrated reasoning framework for mixed-authenticity audio deepfake detection. ToolDF employs an audio large language model as an orchestrator trained with supervised tool-use trajectories. It adaptively analyzes the audio scene, selectively performs source separation, routes components to domain-specific experts, and aggregates their evidence into an interpretable verdict. We further introduce a mixed-authenticity ADD benchmark covering temporal transitions, acoustic overlaps, and hybrid mixtures. Experimental results show that ToolDF achieves the best overall performance on composite-type detection, achieving macro-F1 gains of 3.72 and 14.39 points over the strongest monolithic baseline and a fixed pipeline, respectively, while providing interpretable evidence localized to temporal regions and acoustic sources. Our source code and dataset are publicly available online.
1 Introduction
ToolDF addresses mixed-authenticity audio, where genuine and manipulated cues coexist across temporal transitions or overlapping sources, with an ALLM orchestrator that routes expert tools and aggregates evidence. The paper introduces a benchmark and reports that ToolDF performs best overall on composite-type detection while providing interpretable evidence.
1 Introduction
- Mixed-authenticity detection concerns audio with coexisting genuine and manipulated cues across temporal transitions, overlapping sources, or both, and requires localizing evidence-bearing segments or sources.The paper frames this as a task beyond deciding whether an entire clip is real or fake.Source: §1 · pp. 1–2
- Existing detectors can fail on out-of-domain sources or localized manipulations, fixed separation pipelines may introduce artifacts, and holistic ALLM classification lacks specialized evidence (Li et al., 2024; Zang et al., 2024b; Zhang et al., 2026; Gu et al., 2025; Guo et al., 2026).These limitations motivate an adaptive framework that analyzes input structure and selectively invokes expert tools.Source: §1 · p. 1
- ToolDF uses an ALLM orchestrator to analyze audio structure, selectively route components to domain-specific experts, separate overlapping sources, and aggregate tool responses into an interpretable decision.Its experts cover speech, singing, music, and environmental sound.Source: §1 · p. 2
- Supervised fine-tuning on structured tool-use trajectories teaches ToolDF structural analysis, tool planning, interpretation of expert responses, and evidence aggregation beyond the final authenticity label.The trajectories provide intermediate supervision for structural analysis, tool selection, and evidence aggregation.Source: §1 · p. 2
- ToolDF achieves the best overall performance on composite-type detection, especially when reasoning over temporal segments and overlapping sources, and exposes interpretable tool-execution traces.The benchmark covers single-type and composite-type manipulation scenarios and is publicly released.Source: §1 · p. 2
2 Related Work
Prior ADD work largely assumes one acoustic domain and one clip-level authenticity label, while newer composite-audio and ALLM approaches leave gaps in adaptive analysis, domain coverage, or evidence transparency. ToolDF addresses these gaps by adaptively separating sources and using a supervised tool-use orchestrator to route components to expert detectors and aggregate evidence.
2.1 Audio Deepfake Detection
- ADD methods expanded from speech benchmarks and architectures to singing, music, and environmental sounds, but still assume one dominant domain and one authenticity label per clip.Examples include ASVspoof (Yamagishi et al., 2021), AASIST (Jung et al., 2022), and cross-domain efforts (Zang et al., 2024a,b; Afchar et al., 2025; Xie et al., 2025, 2026a).Source: §2.1 · p. 2
2.2 Multi-Source and Composite Audio ADD
- Fixed separation can add artifacts to single-source audio, while adaptive separation methods remain speech-limited; ToolDF separates only when structurally needed and detects across acoustic domains.Prior approaches include fixed preprocessing (Zang et al., 2024b) and adaptive separation for overlaps (Zhang et al., 2026).Source: §2.2 · p. 2
2.3 ALLM and Tool-Integrated Reasoning
- Direct ALLM classification is a black box with limited evidence insight and misses domain-specific detectors; ToolDF trains a supervised tool-use orchestrator to route components and aggregate evidence.Tool-augmented language models invoke external expert modules and integrate intermediate responses (Schick et al., 2023; Qin et al., 2024; Qian et al., 2025).Source: §2.3 · pp. 2–3
3 Methodology
ToolDF treats mixed-authenticity audio as a structured reasoning task: an audio language model orchestrates localized expert analysis and combines component-level evidence into a clip-level verdict. The methodology also trains the orchestrator to follow this protocol using supervised trajectories.
3.1 Task Formulation
- Each clip may contain heterogeneous components with content types, support regions, and authenticity labels; detection must predict the clip label and localize supporting evidence.Conventional single-domain detection is the special case of one component spanning the entire clip.Source: §3.1 · p. 3
3.2 ToolDF Framework
- ToolDF uses an audio language model as an orchestrator that analyzes the scene, plans tool use, invokes domain-specific experts, and aggregates their evidence into a clip-level verdict.The verdict is fake if any component prediction is fake; otherwise, it is real.Source: §3.2 · pp. 3–4
- The orchestrator identifies component types and support regions, prioritizes source separation for overlapping vocal and background components, then routes components to experts for localized predictions.A tool call specifies the tool and input region, such as a temporal segment or separated source.Source: §3.2 · p. 4
3.3 Supervised Trajectory Learning
- Supervised fine-tuning trains the orchestrator to generate structured trajectories using ground-truth component annotations, while optimizing only tokens produced by the orchestrator.Ground-truth observations condition later trajectory generation and determine the evidence summary and final verdict under the early-fail rule.Source: §3.3 · pp. 4–5
4 Mixed-Authenticity ADD Benchmark
The benchmark evaluates ADD in heterogeneous audio scenes where components may carry different authenticity cues, extending beyond the single-content-type focus of existing benchmarks.
4 Mixed-Authenticity ADD Benchmark
- The benchmark targets mixed-authenticity audio scenes that single-content-type benchmarks do not sufficiently evaluate.It combines public deepfake datasets spanning multiple acoustic domains.Source: §4 · p. 5
- It uses authentic and synthetic subsets from ASVspoof2019 (Nautsch et al., 2021), CtrSVDD (Zang et al., 2024a), FakeMusicCaps (Comanducci et al., 2025), MusicCaps (Agostinelli et al., 2023), and EnvSDD (Yin et al., 2025).The corpora cover speech, singing, music, and environmental sounds.Source: §4 · p. 5
- Evaluation includes isolated speech, singing, music, and environmental clips plus composite splits for temporal transitions (C1), acoustic overlaps (C2), and hybrid mixtures (C3).C1 combines speech and singing transitions; C2 overlaps foreground vocals with background music or environmental sound; C3 combines transitions and overlaps.Source: §4 · p. 5
5 Experiments
Experiments evaluate ToolDF in single-type and composite-type settings against end-to-end systems and a fixed component pipeline, using clip-level macro-F1.
5.1 Dataset
- Single-type evaluation follows source-corpus splits, while accessible MusicCaps and FakeMusicCaps samples use a 6:2:2 random split and unavailable originals’ fake pairs are held out for testing.Table 1 summarizes benchmark statistics; composite-type configurations are provided in Appendix A.Source: §5.1 · p. 6
5.2 Implementation Details
- ToolDF fine-tunes Qwen2.5-Omni-3B on supervised trajectories and uses Demucs with domain-specific XLSR-AASIST experts to route segments or separated sources.Experts cover speech, singing, music, and environmental sound; full-duration and separation-aware variants address transitions and separation artifacts, with thresholds set by development-set equal error rates.Source: §5.2 · p. 6
5.3 Baselines
- Comparisons include four end-to-end models and a fixed Demucs-plus-expert pipeline that labels a clip fake if either detector predicts synthetic origin.The monolithic baselines are XLSR-AASIST (Tak et al., 2022), XLSR-Conformer, WPT-XLSR-AASIST (Xie et al., 2026a), and ALLM4ADD (Gu et al., 2025).Source: §5.3 · p. 6
5.4 Evaluation Metric
- Clip-level macro-F1 equally averages real- and fake-class F1 scores, and Tables 2 and 3 report macro-F1 across single-type and composite-type settings.S-Avg. and C-Avg. denote averages over single-type and composite-type settings, respectively.Source: §5.4 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7
6 Results
ToolDF achieves the strongest composite-input performance while adaptively routing audio components for interpretable evidence, and it also maintains strong fine-grained localization. Results show that single-type performance alone does not ensure mixed-authenticity generalization, while ablations highlight the importance of structured reasoning stages.
6.1 The Single-Type Generalization Gap
- Single-type-trained baselines’ best composite C-Avg. is 57.83, despite XLSR-AASIST’s highest single-type S-Avg. of 86.57.ALLM4ADD performs best among evaluated baselines on overlapping scenarios C2 and C3, but its composite performance remains substantially below its single-type performance; the generalization gap motivates ToolDF’s structured, tool-integrated reasoning.Source: §6.1 · p. 7
6.2 Evaluation on Mixed-Authenticity ADD
- ToolDF achieves the highest composite C-Avg. of 81.89, exceeding XLSR-AASIST by 3.72 points and the Fixed Pipeline’s 67.50.It scores 91.21 on temporal transitions (C1) and 76.81 on hybrid mixtures (C3); the Fixed Pipeline scores 78.55 on acoustic overlaps (C2), but performs lower on C1 and C3. ToolDF adapts whether to separate sources and closely approaches its Oracle variant, with a 0.96-point gap.Source: §6.2 · pp. 7–8
6.3 Ablation and Localization Analysis
- The full ToolDF model performs best across ablation configurations while maintaining parsing rates close to 100%; removing planning causes the largest strict-F1 drop across splits.Removing audio understanding also lowers performance, while excluding description evidence reduces the C2 parsing rate to 23.69%.Source: §6.3 · p. 8, Table 4 · p. 8
- ToolDF’s composite event-level macro-F1 reaches 94.24 on temporal transitions (C1) and 87.62 on hybrid mixtures (C3), localizing manipulation evidence beyond clip-level classification.Across single-type domains, segment- and event-level macro-F1 scores range from 93.64 to 99.97.Source: §6.3 · p. 8, Table 5 · p. 8
7 Conclusion
ToolDF uses an audio large language model to analyze audio structure, adaptively route heterogeneous components to specialist detectors, and aggregate component-level evidence into an interpretable clip-level verdict. On the mixed-authenticity benchmark, it achieves the best overall performance on composite-type detection while localizing temporal regions and acoustic sources that support its decisions.
7 Conclusion
- An audio large language model orchestrates ToolDF’s analysis of audio structure and adaptive routing of heterogeneous components to specialized expert detectors.Source: §7 · p. 8
- ToolDF achieves the best overall performance on composite-type detection while localizing the temporal regions and acoustic sources that support its decisions.Source: §7 · p. 9
- ToolDF aggregates component-level evidence into an interpretable clip-level verdict, highlighting TIR’s potential for interpretable and adaptive detection in complex acoustic scenes.Source: §7 · p. 9
Limitations
ToolDF’s performance depends on the reliability and coverage of its external tools and expert models. Its benchmark and evaluation also leave gaps in real-world complexity and conventional partial-spoof coverage.
Limitations
- Errors from source separation or detector misclassification can propagate to ToolDF’s final decision, which is bounded by the reliability and coverage of its expert models.ToolDF depends on external source-separation modules and domain-specific deepfake detectors.Source: Limitations · p. 9
- Composing existing public datasets enables controlled evaluation with component-level annotations but may not capture real-world media diversity, artifacts, source interactions, complexity, or adversarial editing.The benchmark’s temporal-transition and acoustic-overlap composition rules may be simpler than real-world cases.Source: Limitations · p. 9
- The evaluation excludes conventional partial-spoof benchmarks such as PartialSpoof (Zhang et al., 2022), and adding a partial-spoof detector as a speech expert remains future work.Source: Limitations · p. 9
A Mixed-Authenticity Benchmark Construction · A.1 Audio Caption and Component Annotation
The benchmark construction pipeline annotates single-source clips with component-level acoustic and authenticity information, then uses these annotations and captions to support composite-example synthesis and supervised ToolDF trajectories.
A Mixed-Authenticity Benchmark Construction
- Single-source clips receive component metadata that supports synthesis of composite examples in the C1, C2, and C3 settings.The appendix describes converting clips into structured component annotations before using them to construct composite mixtures.Source: A Mixed-Authenticity Benchmark… · p. 11
A.1 Audio Caption and Component Annotation
- Each component is labeled by content type, active support region, and authenticity as real or fake.Content types include speech, singing, music, and environmental sound; support regions mark when each component is active.Source: §A.1 · p. 11
- Qwen3-Omni-30B-A3B-Captioner2 generates concise captions describing observable acoustic properties of each component.Descriptions vary by component type and cover properties such as vocal delivery, singing style, instrumentation, and environmental sound sources.Source: §A.1 · p. 11
- Caption prompts prohibit authenticity-related descriptions, preventing label leakage while retaining acoustic information for trajectory construction.The prompt bars references to real, fake, synthetic, AI-generated, or model-produced audio.Source: §A.1 · p. 11
- Captions and component metadata form the trajectory’s <audio_understanding> block, supervising intermediate acoustic structure alongside the final clip-level authenticity label.This structure supports tool selection and evidence aggregation.Source: §A.1 · p. 11
- WebRTC-VAD estimates vocal support regions to exclude prolonged silence, while music and environmental components use the full clip duration.The full duration is used for non-vocal signals because they typically serve as continuous background components in the mixtures.Source: §A.1 · p. 11
A.2 Composite Mixture Construction
Composite examples combine annotated components from different acoustic domains while preserving support regions and authenticity labels; a clip is fake if any constituent component is fake, and real otherwise.
Temporal Transition (C1)
- C1 concatenates speech and singing in both orders, sampling R→R, R→F, F→R, and F→F to test localized manipulation across temporal changes.The first three combinations use equal sampling budgets; F→F uses remaining fake source pools.Source: §A.2 · p. 11
Acoustic Overlap (C2)
- C2 overlaps a speech or singing foreground with music or environmental sound, creating four vocal-background subconfigurations.The vocal keeps its localized support region while the background is active throughout; subset sizes follow source-pool availability, yielding larger singing and environmental-sound subsets.Source: §A.2 · pp. 11–12
Hybrid Mixture (C3)
- C3 overlays a speech-to-singing temporal transition with music or environmental sound, requiring reasoning over temporal segmentation and source separation.The background is active throughout, while speech and singing retain their updated temporal support regions.Source: §A.2 · p. 12
A.3 Dataset Statistics · B System Prompt for ToolDF
The benchmark contains 973,649 examples, including 379,900 composite examples, and its appendix documents both the composite configurations and the ToolDF orchestrator’s system prompt. The prompt supports reproducible tool-integrated reasoning during training and inference.
A.3 Dataset Statistics
- 973,649 examples make up the complete benchmark, including 379,900 composite examples across C1, C2, and C3.Table 6 reports detailed composite-type statistics across train, development, and evaluation splits.Source: §A.3 · p. 12, Table 6 · p. 13
- Composite examples are fake-heavy because a clip is labeled fake when at least one constituent component is fake.This label rule explains the fake-heavy composition of the composite pool.Source: §A.3 · p. 12
- C3’s evaluation split has a higher proportion of real samples after basename-level resplitting of FakeMusicCaps samples to prevent background-track leakage.Source: §A.3 · p. 12
B System Prompt for ToolDF
- The exact ALLM orchestrator system prompt is provided for both supervised fine-tuning and inference to support reproducibility of the TIR trajectory.Figure 4 presents the prompt used in both settings.Source: B System Prompt for ToolDF · p. 12, Figure 4 · p. 13
- The prompt enforces structured generation and prevents fabricated tool results or deviations from the prescribed reasoning protocol.These constraints apply during supervised fine-tuning and inference.Source: B System Prompt for ToolDF · p. 12, Figure 4 · p. 13
A.3 Dataset Statistics
- C1 covers temporal speech–singing transitions, C2 vocal/non-vocal acoustic overlaps, and C3 hybrid mixtures combining both.Table 6 describes the composite-type configurations; the supplied passage does not provide its cell values.Source: Table 6 · p. 13