PaperlayerView exact revision on arXiv

Source-linked AI summary

TUE-Detector: A Tool-Using Expert MLLM-Based Detector for AI-Generated Videos

Yichen Wu, Haoxuan Qu, Yongxing Dai, Yan Bai, Yihang Lou, Yuqi Lin, Hossein Rahmani, Jun Liu

arXiv:2608.30704v1cs.CV

TL;DR

AI-generated video detection must identify subtle-yet-measurable artifacts as generated videos become increasingly similar to real footage. TUE-Detector trains a general MLLM into a task-tailored tool-using expert that discovers and reasons over evidence, and extensive experiments demonstrate its efficacy.

  • Problem

    Reliable AI-generated video detection remains challenging because generated videos can resemble real videos while differing through subtle-yet-measurable unnatural artifacts.

  • Method

    TUE-Detector progressively trains a general MLLM into a task-tailored tool-using expert and equips it with suitable tools to collect and reason over evidence of unnaturalness.

  • Results

    TUE-Detector achieves superior performance on evaluated benchmarks, and extensive experiments demonstrate its efficacy.

  • Takeaways & Limitations

    Tool-mediated evidence discovery provides the framework’s supported approach for reliable AI-generated video detection through adaptive tool use and evidence-based reasoning.

  • Takeaways & Limitations

    The first-stage model may overfit demonstrated trajectory patterns, while the initial heuristic toolset may remain insufficiently adapted to model behavior and diverse artifacts.

Abstract

from arXiv · show

AI-generated video detection, which aims to distinguish AI-generated videos from real ones, has recently received increasing research attention. To perform this task reliably, a key challenge lies in accurately identifying subtle-yet-measurable unnatural artifacts. In this work, we address this challenge from a novel perspective of tool-mediated evidence discovery and propose Tool-Using Expert MLLM-based AI-generated Video Detector (TUE-Detector), a novel framework for AI-generated video detection. TUE-Detector trains a general MLLM into a task-tailored tool-using expert detector that learns to invoke suitable tools, collect concrete evidence of unnaturalness, and reason over the evidence for reliable detection. Meanwhile, TUE-Detector further introduces novel designs to equip the expert detector with high-quality and suitable tools. Extensive experiments demonstrate the effectiveness of our framework.

1. Introduction

AI-generated videos increasingly resemble real footage, making detection depend on finding subtle measurable artifacts. TUE-Detector addresses this through tool-mediated evidence discovery, training an MLLM to use suitable tools and reason over collected evidence.

  • Motivation: AI-generated videos can closely resemble real videos, with differences appearing as subtle shape, texture, motion, position, color, or boundary artifacts.Figure 1 highlights subtle shape changes and texture alterations as examples.
  • Motivation: Reliable detection requires identifying subtle-yet-measurable unnatural artifacts that provide concrete evidence for distinguishing real from generated videos.These cues may be difficult to notice directly but can be measured in visual or temporal properties.
  • Approach: TUE-Detector frames detection as tool-mediated evidence discovery, in which an MLLM examines diverse cues, verifies evidence of unnaturalness, and reasons toward a final judgment.The framework is designed to transform a general MLLM into a task-tailored tool-using expert detector.
  • Challenges: Because artifacts vary across regions, frames, and temporal scales, effective detection requires adaptive decisions about which tools to use, where, when, and how to integrate their outputs.A fixed tool-use pattern is insufficient for the diverse cues found across videos.
  • Approach: TUE-Detector uses progressive two-stage training: it first builds tool-use ability with a fixed heuristic toolset, then improves both the detector and toolset.The process addresses the difficulty of jointly developing suitable tools and tool-use ability from a general MLLM.
  • Results: TUE-Detector achieves superior performance on the evaluated benchmarks.The contribution statement reports this outcome without specifying a numerical value.

2. Related Work

Prior AI-generated video detectors include CNN-, RNN-, and MLLM-based approaches, while tool use has been explored across other tasks. TUE-Detector differs by training a general MLLM into a task-tailored tool-using expert for video detection.

  • AI-generated video detection: Earlier AI-generated video detection studies explored CNN-based and RNN-based neural architectures.The related-work passage also notes growing use of MLLMs as their visual understanding developed.
  • Tool usage: External-tool assistance has been studied for open-domain question answering, mathematical reasoning, fact verification, and image forensic analysis.These examples situate tool usage across several problem settings beyond AI-generated video detection.
  • Distinction: TUE-Detector is presented as the first work, to the authors’ knowledge, to train a general MLLM into a task-tailored tool-using expert for AI-generated video detection.Its stated purpose is to identify subtle-yet-measurable unnatural artifacts in videos.

3. Method

TUE-Detector uses two-stage training to turn a general MLLM into a tool-using video detector. It learns tool-mediated evidence discovery, improves trajectory supervision through influence estimation, and jointly evolves the model and toolset.

  • Training overview: TUE-Detector trains a general MLLM into a task-tailored tool-using detector through a two-stage process.Stage 2 starts from the preliminary detector and heuristic toolset obtained in Stage 1.
  • Training Stage 1: Stage 1 constructs a heuristic toolset and trains the MLLM to select tools, collect evidence, interpret outputs, and reason toward a verdict.Training uses retained teacher-generated trajectories while masking tool-result tokens so learning focuses on active reasoning and tool-call generation.
  • Training Stage 1: Influence scores estimate whether imitating each retained trajectory would improve or hurt validation performance, serving as a proxy for trajectory utility.The scores are converted into adaptive trajectory weights for the training loss.
  • Training Stage 1: Ridge-regression influence estimation avoids explicit Hessian inversion and enables batch-wise scoring of retained trajectories.This addresses the large-MLLM Hessian cost and avoids separate inverse-Hessian-vector computations for individual trajectories.
  • Training Stage 2: Stage 2 jointly improves the detector and toolset because the model may overfit demonstrated patterns while the initial heuristic tools remain static and mismatched.The framework alternates model optimization with user-experience-guided tool evolution to improve tool-use capability and tool suitability.

4. Experiments

Experiments evaluate TUE-Detector on ViF-Bench and GenVideo, reporting benchmark-specific metrics and comparisons with existing detection methods. Ablations show that both proposed strategies and both training stages contribute to performance.

  • Main Results: Evaluation uses ViF-Bench and GenVideo with benchmark-specific protocols and metrics.ViF-Bench reports ACC, R, and F1; GenVideo uses many-to-many zero-shot evaluation with R and F1.
  • Main Results: TUE-Detector achieves the best average performance across ViF-Bench generators and near-saturated performance on GenVideo.GenVideo is retained mainly for cross-generator comparability, while ViF-Bench serves as the primary discriminative benchmark.
  • Main Results: 98.82% accuracy is achieved on real videos in ViF-Bench.The reported figure comes from Table 1, whose paired splits contain real and fake videos.
  • Ablation Studies: Removing either influence-based teacher-knowledge guidance or user-experience-guided tool evolution causes a significant performance drop.The ablation compares each variant with the full framework.
  • Ablation Studies: Removing either Stage 1 or Stage 2 significantly lowers performance relative to the complete two-stage process.Direct prompting with the updated toolset and omitting Stage 1 task-specific training also produce lower performance.

5. Conclusion

The conclusion presents TUE-Detector as a tool-mediated evidence-discovery framework for AI-generated video detection. It trains a general MLLM into a task-tailored tool-using expert, equips it with suitable tools, and reports efficacy through extensive experiments.

  • TUE-Detector frames AI-generated video detection as tool-mediated evidence discovery.
  • The framework trains a general MLLM into a task-tailored tool-using expert detector.
  • Extensive experiments demonstrate the efficacy of TUE-Detector.

Supplementary Material

The supplementary material contains additional ablations, visualizations, case studies, experimental details, mathematical derivations, toolset details, prompts, and dataset and model licenses.

  • Section A presents additional ablation studies.
  • Section B presents additional visualizations and case studies, while Section C provides experimental details and a derivation of influence reweighting.
  • Sections D–F provide toolset details, referenced prompts, and dataset and model licenses.

A. Additional Ablation Studies

Additional ablations show that TUE-Detector’s performance depends on complementary tool evaluation, adaptive toolset evolution, trained tool use, evidence-grounded reasoning, and several robust design choices. The framework remains effective across teacher models, prompts, damping settings, and practical inference conditions.

  • Tool-use experience: Both tool-use breadth and evidential effectiveness are necessary, as removing either evaluation aspect lowers performance.The two aspects measure how broadly a tool is used and how effective its returned evidence is.
  • Training the model to use tools: Training the model to use tools substantially outperforms directly prompting off-the-shelf models, showing that the toolset alone is insufficient.Training teaches when to invoke tools, integrate returned evidence, and make detection decisions.
  • Training design: Exposing ground-truth labels to the teacher and removing the token-level tool-output mask are evaluated as alternative designs whose performance is lower than the proposed configuration.The token-level mask focuses Stage 1 learning on tool-use decisions and evidence-based reasoning.
  • Robustness and efficiency: Performance remains consistent across teacher models and paraphrased prompts, while κ = 10 or 20 is optimal and κ = 10 is used in experiments.Across κ values from 1 to 50, the framework consistently outperforms the previous state-of-the-art method; inference takes approximately 0.5 seconds per clip on 8 NVIDIA H200 GPUs.
  • Trajectory weighting: Influence-based trajectory reweighting outperforms LLM-judge weighting, and ridge-reformulated influence computation enables a practically feasible training procedure.The external judge rates reasoning trajectories on a 1-to-5 scale, while the ridge reformulation addresses practical computation.
  • Toolset update strategy: Removing components from the toolset update strategy reduces performance, indicating that brainstorming and crossover provide complementary training signals.The evolved toolset uses selection, mutation, and exploration, including brainstorming and crossover.
  • Tool-mediated reasoning: Masking tool-returned evidence substantially reduces performance to near-baseline accuracy, confirming that classifications use observed evidence rather than tool calls alone.The masked variant retains reasoning and invocations but replaces returned evidence with a fixed placeholder.

B. Additional Qualitative Results

This section introduces additional visualisations and case studies.

  • Additional qualitative results: The section presents additional visualisations and case studies.No further qualitative finding is stated in the supplied passage.

B.1. Qualitative Comparisons

Qualitative comparisons illustrate that TUE-Detector identifies varied temporal, spatial, and interaction inconsistencies in fake videos and uses them to predict FAKE.

  • Qualitative comparisons: The case studies use chronological frames, ground-truth labels, conclusions, and numerical diagnostics returned during tool interaction.Full reasoning traces are omitted to focus the comparison on observable evidence.
  • Qualitative comparisons: TUE-Detector identifies temporal instability in foreground people and predicts FAKE.This example concerns a campaign-news clip.
  • Qualitative comparisons: TUE-Detector identifies non-rigid flashlight deformation and predicts FAKE.The reported evidence focuses on object deformation.
  • Qualitative comparisons: TUE-Detector identifies temporal instability in hoodie-logo text and predicts FAKE.The example highlights instability in both the logo and its text.
  • Qualitative comparisons: TUE-Detector identifies temporal inconsistency in liquid and stirring-stick interaction and predicts FAKE.The evidence concerns the consistency of an interaction over time.
  • Qualitative comparisons: TUE-Detector identifies temporal instability around a rear puppy and adjacent background and predicts FAKE.The example highlights instability spanning the subject and nearby background.

C.1. Additional Details of Stage 1

Stage 1 uses teacher-guided tool-using trajectories and implementation settings to train the detector to inspect video evidence before reaching Real/Fake judgments.

  • Training settings: The training configuration applies LoRA with rank r=32, α=64, dropout 0, and a learning rate decayed from 5×10−5 to 2×10−5.LoRA is applied only to language-side projections.
  • Reference construction: Stage 1 uses teacher-generated trajectories constructed from 16 uniformly sampled frames, heuristic tools, and a reference trajectory schema.The resulting SFT corpus contains 4,034 rows with final verdicts matching the binary ground truth by construction.
  • Tool-mediated reasoning: Teacher trajectories guide the model to inspect subtle visual artifacts and reason toward a final detection verdict using tool calls and evidence.Examples include temporal instability in people, objects, logos, liquids, and backgrounds, alongside contrasting stable interpretations.
  • Reference reliability: Teacher reliability logs report 66/4,100 internal attempts failing validation and 63/4,034 samples requiring a format retry.The two percentages are 1.610% and 1.562%, respectively, and the layers are not additive.
  • Qualitative examples: The qualitative examples contrast tool-supported identification of temporal or physical instability with baseline descriptions of stable, plausible content.The examples include distorted chess players and a skydiver who unnaturally vanishes, both judged Fake.

C.2. Additional Implementation Details of Stage 2

Stage 2 jointly improves the detector’s tool-use policy and toolset through reward shaping and periodic tool evolution, while validating candidate tools before retention.

  • Optimization: Stage 2 uses full-parameter GRPO with four rollouts per prompt, batch size 4, learning rate 5 × 10−7, and KL regularisation coefficient 0.10.These settings define the reinforcement-learning update against a reference policy.
  • Reward design: The reward combines final-verdict correctness, output-format validity, tool-name drift penalties, and anchor-based rewards for correcting or regressing from Stage 1 predictions.Incorrect final verdicts receive no direct result bonus, while malformed structure triggers a hard format penalty.
  • Anchor shaping: The Stage 1 prediction cache remains frozen throughout Stage 2, so current rollouts are rewarded for improving on or penalized for regressing from the starting checkpoint.The anchor is keyed by video ID and is not recomputed during reinforcement learning.
  • Tool evolution: Tool evolution runs every ∆evo = 20 training steps and classifies tools by usage breadth and evidential effectiveness before retaining, deleting, mutating, brainstorming, or crossing them over.Tools in QHH are retained and tools in QLL are deleted; other quadrants provide mutation sources.
  • Candidate validation: Candidate tools are trial-run on calibration examples and added only after successful import, execution, signature, and XML-schema checks.Failures feed concrete error messages and summaries back into subsequent evolution prompts.
  • Exploration operators: Brainstorm proposes probes for uncovered gaps, whereas crossover combines complementary evidence channels into a joint firing condition.Examples target region stability, unstable text, scene complexity, background support, and sparse foreground motion.

C.3. Additional Implementation Details of Testing

During testing, the trained policy invokes the evolved tools to reason toward a verdict, typically using three to four tools per video.

  • Testing process: The trained policy typically invokes 3–4 tools per video during evaluation.first_frame_jump_analyzer is the most frequently invoked forensic tool.
  • Testing process: Evaluation uses the evolved toolset and follows the same tool-mediated reasoning process used during training to reach the final verdict.The complete evolved registry is listed in Table 28, with additional usage analyses in Tables 5 and 24.

C.4. Additional Details of ViF-Bench

ViF-Bench evaluates detectors on paired real and fake videos spanning 19 modern generators, using shared real-video counterparts and standardized frame sampling.

  • Benchmark composition: ViF-Bench contains about 5,000 fake and 5,000 real videos, with fake videos drawn from 19 state-of-the-art generators.The real videos come from Panda-70M and Kinetics-400, while generators cover Text to Video and Image to Video settings.
  • Benchmark composition: The official paired test selection uses 2,949 fake videos and a shared pool of 163 real videos reused across all 19 generators.Per-generator fake counts range from 111 to 163.
  • Frame sampling: Testing samples 16 evenly spaced frames per video, and the benchmark’s approximately 115,000-pixel frames remain below every listed inference pixel cap.Thus, the caps are non-binding and the compared setups receive the same frame input on ViF-Bench.
  • Training data: Training uses ViF-CoT-4K, a 4,034-row dataset of grounded Chain-of-Thought labels covering seven generators and multiple Text to Video and Image to Video variants.Six covered generators also appear in the ViF-Bench test split, while one CogVideoX Image to Video variant is training-only.
  • Evaluation protocol: For each generator, matched Real/Fake pairs are evaluated with Accuracy, Recall, and F1 using Fake as the positive class, then averaged equally across generators.The paired construction matches real videos by source clip ID.

C.5. Additional Details of GenVideo

This section describes the GenVideo evaluation setup and supporting influence-based trajectory-weighting derivation. The method uses masked SFT objectives, empirical-Fisher approximations, and ridge regression to produce weights for iterative training.

  • GenVideo evaluation: The GenVideo many-to-many test set contains about 18,588 videos, including 10,000 real and 8,588 fake videos.Real videos come from MSR-VTT, Youku-mPLUG, and Kinetics-400, while fakes come from ten generators.
  • GenVideo evaluation: The ten GenVideo training generators differ from the ten test generators, making the many-to-many evaluation fully zero-shot.The training split includes Pika, ZeroScope, DynamiCrafter, VideoCrafter, SVD, OpenSora, Latte, SEINE, I2VGen-XL, and Stable Diffusion.
  • Influence-based weighting: The estimator applies influence analysis to the same masked token-level objective optimized by SFT, then approximates the Hessian with a damped empirical Fisher.Masked per-sample gradients define both the empirical-Fisher approximation and the resulting ridge problem.
  • Influence-based weighting: The push-through reformulation converts the damped inverse calculation into a ridge problem over sample coefficients, avoiding a large parameter-space inverse.The implementation solves this problem with a normalized stochastic gradient method over trainable LoRA coordinates and layer blocks.
  • Influence-based weighting: The influence-weight normalization preserves the average loss scale, removes samples with nonpositive utility, and introduces no additional weighting hyperparameter.The normalization keeps the total weight equal to J, while samples with u_j ≤ 0 receive zero weight.

D. Toolset Details

This section specifies TUE-Detector’s heuristic forensic tools, structured tool outputs, and prompting procedures. The tools test measurable visual or motion anomalies under explicit conditions, while the prompts require tool-assisted reasoning before a real/fake verdict.

  • Tool outputs: Tool outputs are packed into structured XML containing analyzed signals, computed numeric values, threshold status, and a manual override explanation.The manual explanation guides the registry’s verdict override and is not surfaced to the student in raw form.
  • Forensic tools: The first_frame_jump_analyzer compares the first-frame bright and sharp differences against robust within-clip MAD-z baselines.It flags the first frame when either channel exceeds the threshold, indicating that the frame is out of distribution relative to the rest of the clip.
  • Forensic tools: The background motion analyzer checks tracked background drift only when the camera is confidently static and enough background points support a reliable percentile estimate.It fires when both the 90th-percentile displacement and per-frame step exceed their thresholds.
  • Tool registry: The initial heuristic toolset contains 13 tools: 12 code tools plus toolbox_guide.The registry is documented in Table 27, and the updated toolset is compared with it in Table 28.
  • Prompting procedure: The teacher and student prompts require reasoning over sampled frames, 1–3 code-tool calls, and toolbox_guide as the final tool call before deciding Real or Fake.The prompts instruct models to select tools that can further inspect video artifacts and prohibit fabricated tool results.
Loading 2608.30704v1…