PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination

Shuo Liang, Yixing Ma, Pengfei Zhou, Xingyan Chen, Zihan Mei, Manting Li, Feihan Chen, Zhiwen Wang, Bin Xu, Haotian Zhang, Jiajun Song, Shiya Su, Run Liu, Zhenghang Ni, Yifa Yu, Jintao Hong, Bolong Feng, Yifei Liu, Zirui Zhang, Jingxuan Zhang, Songlin Zhao, Yifan Bai, Kang Tan, Yizhe Liu, Junhao Du, Yongtao Ge, Zhaopan Xv, Xinyuan Zhang, Mengru Ma, Chunhua Shen, Wei Wang, Yang You, Zheng Zhu, Kaipeng Zhang, Wangbo Zhao

arXiv:2608.14391v1cs.CVcs.AI

TL;DR

Existing benchmarks offer limited evidence about detecting AI-generated videos depicting real-world crises and about detector reliability under generation and dissemination conditions. This paper introduces RA-Bench to evaluate these gaps and finds that current detectors do not generalize reliably across realistic generated videos.

  • Problem

    Existing studies provide limited systematic evidence on detector performance and generator behavior for socially consequential real-world crisis videos.

    Source: §1 · p. 3

  • Method

    The paper constructs RA-Bench from real-video anchors and generated clips, then evaluates detector generalization, generation properties, human judgments, and social dissemination.

    Source: §1 · pp. 3–4, §3.1 · p. 6

  • Results

    Across detector families, performance fails to transfer consistently across RA-Bench sources; traditional-detector AUC falls to source-level means of 43.9–57.3%.

    Source: §4.1.4 · pp. 15–16, §4.2.4 · p. 20

  • Takeaways & Limitations

    Current methods cannot reliably detect realistic AI-generated videos, especially when crisis content evolves and circulates in real-world settings.

    Source: §1 · p. 4

    RA-Bench is a snapshot covering nine I2V sources and visual-only detection, while its I2V and social-dissemination stages are evaluated separately under controlled conditions.

    Source: §5 · p. 25

Abstract

from arXiv · show

Recent video generators can fabricate realistic depictions of wars, disasters, public emergencies, and other real-world crises, creating substantial risks of misinformation. Existing benchmarks, however, provide limited evidence on detector and generator behavior in such settings, including how detectability varies with generation conditions, how people perceive generated videos, and whether detectors remain reliable during social dissemination. To address this gap, we introduce RA-Bench, a benchmark for AI-generated video detection that uses Real videos as Anchors. RA-Bench contains 17,886 videos, comprising 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from four open-source and five closed-source generators. Based on RA-Bench, we organize our evaluation along three dimensions. We first assess detector generalization across seven traditional detectors, ten zero-shot multimodal models under three review settings, and two MLLMs specifically fine-tuned on AI-generated video detection. Across these methods, none of the three detector families generalizes consistently across RA-Bench instances. We then examine how detectability varies with generation quality, conditioning information, and sampling seeds. These analyses show that generation properties affect detector families differently, while source-level detection patterns remain stable across seeds. Finally, we study human authenticity judgments and detector reliability during social dissemination. We find that videos that mislead people are also difficult for current detectors, and that social dissemination makes detection harder. Together, these findings show that current methods struggle to detect realistic AI-generated videos, highlighting the need for detectors robust to evolving video generators.

1. Introduction

Realistic AI-generated videos of real-world crises can deceive viewers and provoke panic, while existing studies provide limited evidence about detector reliability in these settings. RA-Bench addresses this gap with a large benchmark and systematic evaluation showing that current detectors struggle across sources, generation conditions, human-deceptive videos, and social dissemination.

Motivation

  • Existing benchmarks primarily study general video content, leaving detector performance and generator behavior insufficiently analyzed for real-world crises and other socially consequential events.This gap makes it unclear whether current detectors can reliably protect society from increasingly realistic video-generation models and their misuse.Source: §1 · p. 3

Benchmark

  • RA-Bench contains 17,886 videos, including 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from four open-source and five closed-source generators.The benchmark conditions image-to-video generation on each anchor’s first frame to simulate plausible misinformation scenarios involving real images and generated subsequent events.Source: §1 · p. 3

Evaluation scope

  • RA-Bench evaluates detector generalization, generation-property effects, human authenticity judgments, and robustness during social dissemination using traditional detectors, zero-shot multimodal models, and fine-tuned MLLMs.The evaluation examines generation quality, conditioning information, sampling seeds, human-deceptive cases, and dissemination through RA-Bench-HumanProof and RA-Bench-LastMile.Source: §1 · pp. 3–4

Main findings

  • None of the three detector families generalizes consistently across RA-Bench sources, with traditional-detector source-level means of 43.9–57.3% AUC and rankings that change across generators.Zero-shot multimodal models remain sensitive to prompts and generation sources, while fine-tuned MLLMs also show weak performance in reported settings.Source: §1 · p. 4
  • Generation properties affect detector families differently, while source-level detection patterns remain relatively stable across sampling seeds.A 50-point Condition Fidelity increase corresponds to a 14.4-point decrease in Gemini Diagnostic FakeR, and the seven-detector mean AUC changes from 33.4% to 50.9% and 44.6% across T2V, first-frame I2V, and first+last-frame I2V.Source: §1 · p. 4
  • Videos that mislead people are also difficult for detectors, while social dissemination further weakens detection, reducing mean FakeR from 46.0% to 1.4% in the Full condition.On the 633-video HumanProof subset, Gemini Binary and Diagnostic achieve 54.7% and 54.5% BAcc, while traditional detectors average 47.5% AUC.Source: §1 · p. 4

2. Related Work

Prior benchmarks and detector taxonomies have expanded alongside increasingly realistic video generators, but remain limited for socially consequential real-world crises. RA-Bench addresses this gap by evaluating representative detector families in these settings.

Benchmarks for AI-Generated Video Detection

  • AI-generated-video benchmarks initially paired real videos with generated counterparts and evaluated cross-generator transfer, then expanded scale, generator coverage, evaluation protocols, and MLLM-based explanation.Examples include GVF (Ma et al., 2025), GenVideo (Chen et al., 2024), GenVidBench (Ni et al., 2026), and AIGVDBench (Ma et al., 2026).Source: §2 · p. 4
  • Despite this progress, existing benchmarks provide limited coverage of misuse scenarios involving real-world crises and other socially consequential events.RA-Bench is introduced to systematically assess detector robustness and limitations in these settings.Source: §2 · p. 5

Realistic video generation models

Detectors for AI-generated videos

  • AI-generated-video detectors comprise traditional discriminative approaches and MLLM-based approaches, with the latter offering explanations beyond binary classification.Traditional methods include video, generated-image, and generated-video detectors that respectively model video labels, image artifacts, or temporal evidence across frames.Source: §2 · p. 5

3. RA-Bench

RA-Bench is a crisis-focused benchmark pairing 1,830 real-video anchors with 16,056 generated clips from image-to-video models. Its construction combines social-risk source collection, scene-level preprocessing, two-round human review, standardized postprocessing, and paired generation from shared semantics.

3. RA-Bench

  • RA-Bench contains 1,830 real-video clips and 16,056 generated clips, built by pairing reviewed crisis-event anchors with generated counterparts.The benchmark construction pipeline is summarized in Figure 2, while its composition is summarized in Figure 3.Source: §3 · p. 6

3.1. Source Data Collection

  • The authors collect 675 public videos and organize them into 10 social-risk categories and 44 subcategories without imposing uniform category quotas.The categories span crises and socially consequential events including disasters, war, politics, public safety, health, technology, and large public events.Source: §3.1 · p. 6

3.2. Automated Preprocessing

  • Scene-cut detection produces 5,774 reviewable clips, with neighboring near-duplicates flagged using ResNet-18 frame- and clip-level cosine similarities.PySceneDetect and FFmpeg perform segmentation, while duplicate warnings are provided to manual reviewers.Source: §3.2 · p. 7

3.3. Manual Review

  • Two-round review by seven reviewers and four adjudicators retains 2,426 clips and rejects 3,348 based on quality, semantic fit, duration, duplication, and source-risk criteria.Each clip is independently assessed by two reviewers in Round 1, while disagreements and uncertain cases proceed to Round 2.Source: §3.3 · p. 7

3.4. Retained Clip Postprocessing

  • Postprocessing yields 1,830 clips from 338 unique source videos by bounding duration to 3–15 seconds, unifying H.264 encoding, pruning redundancy, and recording licensing metadata.The released real set totals 18,448 seconds, about 5.1 hours, with a mean duration of 10.08 seconds.Source: §3.4 · p. 7

3.5. Paired Video Generation

  • Each anchor is captioned and paired with image-to-video counterparts conditioned on the caption and first frame, using four open-source and five closed-source generators.Generated clips use durations proportional to their real counterparts within 2–8 seconds, share H.264 encoding, and total 16,056 clips; closed-source safety filters reduce some paired subsets.Source: §3.5 · pp. 7–8, Figure 3 · p. 8

4. Experiments and Analysis

This section evaluates detection across RA-Bench generation sources, prompts, protocols, generation conditions, human judgments, and social dissemination. It examines how detector generalization and reliability vary across these settings, including the effects of generation properties, sampling seeds, human deception, and dissemination.

4.1.1. Traditional Detectors Do Not Transfer Reliably to RA-Bench

  • Traditional detector AUC falls from 67.6–98.6% in public references to 43.9–57.3% source-level means on RA-Bench, with different detectors leading across generators.At 5% FPR, the seven-detector mean identifies only 2.8–7.5% of generated videos, and temporal reallocation improves mean AUC by just 0.99 points.Source: §4.1.1 · pp. 9–12

4.1.2. Zero-Shot Multimodal Models Remain Unreliable Across Prompts and Sources

  • Gemini-3.1-Pro-Preview is the strongest zero-shot model at 63.4/62.9 Binary BAcc/macro-F1, but no model combines strong performance with stable behavior across prompts and sources.Prompt changes can reverse class preference, structured outputs do not consistently improve detection, and scaling helps selected source–prompt pairs without producing reliable generalization.Source: §4.1.2 · pp. 12–14

4.1.3. Fine-Tuned MLLMs Exhibit Protocol Dependence and Class Bias

  • Replacing timestamps with frame indices reduces Skyra’s exact-5-second FakeR gap from 48.6 to 1.5 points and lowers mean BAcc to 54.4–54.9%, while BusterX++ remains strongly biased toward Real.Skyra’s residual per-source BAcc spans 31.1–61.3%, whereas BusterX++ FakeR remains 4.1–9.1% with 93.0–93.8% real recall.Source: §4.1.3 · pp. 14–15

4.1.4. Summary of Detector Generalization

  • Across all three detector families, public benchmark performance, larger model scale, and task-specific fine-tuning do not ensure generalization across RA-Bench sources.Traditional detectors, zero-shot multimodal models, and fine-tuned MLLMs each exhibit source-, prompt-, or protocol-dependent weaknesses.Source: §4.1.4 · pp. 15–16

4.2.1. Generation Quality Does Not Uniformly Determine Detection Difficulty

  • Higher Condition Fidelity and temporal consistency are associated with weaker fake evidence, whereas dynamic clips strengthen Gemini’s fake evidence but leave traditional-detector scores nearly unchanged.Gemini Diagnostic fake recall decreases by 9.6, 14.4, and 12.4 points for interquartile increases in Quality, Condition Fidelity, and Combined Quality; dynamic clips add 5.5–8.4 points of Gemini fake evidence.Source: §4.2.1 · p. 17

4.2.2. Real-Image Conditioning Affects Detector Families Differently

  • From T2V to first-frame and first+last-frame I2V, mean fine-tuned-MLLM FakeR falls from 70.5% to 42.7% and 28.3%, while traditional-detector AUC changes in both directions.First-frame conditioning raises traditional-detector mean AUC from 33.4% to 50.9%, but further conditioning lowers it to 44.6%; detector-specific changes range from −25.6 to +6.0 points.Source: §4.2.2 · pp. 17–18

4.2.3. Detection Results Vary Little Across Sampling Seeds

  • Across sampling seeds, source-level detector patterns remain stable: the largest traditional-detector mean AUC range is 1.05 points, with pairwise Spearman correlations of 0.978–0.989.Seed 0 differs from three-seed averages by at most 0.20 AUC points, and only three cells cross 50% AUC while remaining near random ranking.Source: §4.2.3 · p. 20
  • Across three seeds, source-level seven-detector AUC and fine-tuned-MLLM FakeR vary only slightly, so the primary benchmark patterns are not seed-specific.The largest FakeR/BAcc ranges are 2.35/1.17 points for Skyra-SFT, 2.46/1.23 for Skyra-RL, and 0.71/0.36 for BusterX++; BusterX++ remains below 10% FakeR for every source–seed combination.Source: §4.2.3 · p. 20

4.2.4. Summary of Generation Properties

  • Generation quality and conditioning affect detector families differently, whereas source-level detection patterns remain stable across sampling seeds.Fine-tuned-MLLM mean FakeR falls from 70.5% for T2V to 42.7% for first-frame I2V and 28.3% for first+last-frame I2V, while traditional-detector AUCs move in both directions.Source: §4.2.4 · p. 20

4.3.1. Human Recognition Varies Sharply Across Generation Sources

  • Human recognition varies sharply by generation source, and real crisis videos are also sometimes mistaken for generated content.Reviewers labeled 60.3% of generated-video judgments Generated, versus 71.9% of real-video judgments Real; 22.8% of real-video judgments were labeled Generated and 5.3% Uncertain.Source: §4.3.1 · p. 21

4.3.2. RA-Bench-HumanProof Remains Difficult Across Detector Families

  • Human-deceptive videos remain difficult across detector families, with traditional detectors near random ranking and fine-tuned models showing source- and prior-dependent failures.HumanProof contains 633 generated videos labeled Real by all five reviewers; traditional detectors average 47.5% AUC and 4.4% T@5%, while timestamp-free Skyra reaches 53.7% BAcc and BusterX++ only 3.9% FakeR.Source: §4.3.2 · pp. 21–22, Table 8 · p. 22

4.3.3. Social Dissemination Simulation Further Weakens Detection

  • Social dissemination weakens detection, with fine-tuned MLLMs shifting toward Real and traditional detectors changing rankings inconsistently.Figure 11 reports mean MLLM BAcc falling from 59.5% on Original to 47.1% under Full, while FakeR falls from 46.0% to 1.4%; traditional-detector AUC falls from 51.4% to 47.3%, and Original–Full ranking correlation is 0.07.Source: Figure 11 · p. 23, §4.3.3 · pp. 23–24

4.3.4. Summary of Human Perception and Social Dissemination

  • Generated crisis-event videos can be difficult for both viewers and detectors, while social dissemination further reduces their detectability.Reviewers identified 68.6% of open-source but 52.9% of closed-source videos as generated; on HumanProof, Gemini reached 54.7%/54.5% BAcc, traditional detectors averaged 47.5% AUC, and Full dissemination reduced fine-tuned-MLLM FakeR from 46.0% to 1.4%.Source: §4.3.4 · p. 24

5. Discussion

RA-Bench shows that detector reliability depends on the generation source, evaluation protocol, generation properties, and social dissemination, while human deception exposes failures that detector scores alone miss. The benchmark is a controlled snapshot whose scope should expand to evolving generators, end-to-end forgery workflows, and realistic online contexts.

Detector reliability is conditional

  • Detector reliability cannot be summarized by one benchmark score: no evaluated detector family transfers consistently across sources and protocols.Traditional detectors change rank across generation sources; multimodal models vary with prompts and sources; fine-tuned MLLMs depend on temporal-label format or favor Real.Source: §5 · p. 24

Generation properties affect detector families differently

  • Generation conditions affect detector families differently: fine-tuned MLLM mean FakeR falls from 70.5% to 42.7% and 28.3% across T2V, first-frame I2V, and first+last-frame I2V.Stronger Condition Fidelity and temporal consistency are associated with weaker fake evidence, while traditional-detector AUCs move in both directions; small seed variation suggests these patterns are stable across samples.Source: §5 · p. 24

Human deception identifies socially consequential failures

  • 633 generated videos were labeled Real by all five reviewers, forming RA-Bench-HumanProof, where Gemini performs near chance across all prompt formats.Reviewers also labeled 22.8% of real-video judgments as Generated, while traditional detectors showed only a small additional decline because source-matched performance was already near random.Source: §5 · p. 25

Social dissemination shifts detector predictions toward Real

  • Under Full social dissemination, traditional-detector mean AUC falls from 51.4% to 47.3%, while fine-tuned MLLM mean FakeR drops from 46.0% to 1.4%.Downsampling, frame-rate reduction, and a news-style badge each contribute to predictions shifting toward Real.Source: §5 · p. 25

Limitations

  • RA-Bench is a controlled snapshot covering nine I2V sources and visual-only detection, so future versions should add evolving generators, detectors, end-to-end forgery workflows, and realistic social context.The current benchmark separates generation and simulated dissemination, while human judgments omit surrounding online context.Source: §5 · p. 25

6. Future Directions

Future work should improve detector robustness and interpretability, incorporate active watermarking and audio-visual evidence, and use detection to advance video-generation realism. These directions respond to the benchmark’s observed difficulty with realistic generated videos and unreliable detection evidence.

Robust detection

  • Develop robust detectors using temporal consistency, lighting variations, and object motion to handle realistic videos across diverse scenarios and post-processing operations.Existing methods show a substantial performance gap on realistic videos from state-of-the-art generators (van den Oord and Roman, 2024; Team Seedance et al., 2026).Source: §6 · p. 25

Detector interpretability

  • Establish benchmarks for the correctness and faithfulness of MLLM-generated detection evidence, and explore structured reasoning agents and post-training methods.MLLMs provide evidence beyond traditional logits, but the reliability of that reasoning remains largely unverified; relevant directions include agents (Yao et al., 2025; Xie et al., 2024) and post-training (Agarwal et al., 2024).Source: §6 · pp. 25–26

Active watermarking

  • Use active watermarking and co-design it with passive detectors to improve reliability and evaluate robustness against post-processing during social dissemination.Watermarks can embed identifiable signals during generation, while the benchmark can assess their resilience under dissemination operations (Fernandez et al., 2024; Hu et al., 2025; Su et al., 2025).Source: §6 · p. 26

Audio-visual detection

  • Extend detection benchmarks beyond visual cues to jointly evaluate audio-visual evidence as generators produce natively aligned audio-video content.Seedance 2.0 (Team Seedance et al., 2026) is identified as an example motivating this multimodal direction.Source: §6 · p. 26

Generation realism

  • Use detector performance as a realism signal and reward model for post-training video generators, because higher-quality generated videos are harder for humans and detectors to identify.Best-performing detectors can support generation-model post-training (Liu et al., 2025).Source: §6 · p. 26

7. Conclusion · Appendix

RA-Bench evaluates AI-generated video detection across realistic crisis settings, showing that current detectors do not generalize reliably across generators, generation conditions, or social dissemination. Human-misleading videos are also difficult for detectors, underscoring persistent detection limitations.

7. Conclusion

  • RA-Bench contains 17,886 videos, including 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from nine generators.The benchmark uses Real videos as Anchors and includes four open-source and five closed-source generators.Source: §7 · p. 26
  • Mean FakeR fell from 46.0% to 1.4% after social dissemination among evaluated fine-tuned MLLMs, showing substantially reduced detection reliability.RA-Bench-LastMile simulates social dissemination and measures its effect on AI-generated video detection.Source: §7 · p. 26
  • None of the three detector families generalizes consistently across generation sources in RA-Bench.Traditional detectors fall sharply from public-reference results, zero-shot multimodal models remain sensitive to prompts and sources, and fine-tuned MLLMs depend on timestamp cues or show class bias.Source: §7 · p. 26
  • Generation properties affect detector families differently, while source-level detection patterns remain stable across three sampling seeds.Condition Fidelity, temporal consistency, dynamic content, and real-image conditioning show different relationships with traditional-detector and MLLM outputs.Source: §7 · p. 26
  • Human review found 633 generated videos judged Real by all five assigned reviewers, forming RA-Bench-HumanProof.These human-misleading videos are difficult for detectors: Gemini performs near chance under all three prompt formats.Source: §7 · p. 26
  • Together, the results show that current methods cannot reliably detect realistic AI-generated videos in real-world crisis settings.The evaluation covers detector generalization, generation properties, human perception, and social dissemination.Source: §7 · p. 26

A. RA-Bench Construction and Data Documentation … B. Detector Evaluation Protocols and Extended Results

RA-Bench constructs a socially risky real-video benchmark through structured taxonomy, provenance-aware collection, automated preprocessing, manual review, postprocessing, and standardized generation from real anchors. The resulting dataset preserves broad category coverage while documenting review outcomes, rights status, generation conditions, and qualitative real-versus-generated comparisons.

A.1. L2 Taxonomy and Source Distribution

  • RA-Bench organizes 675 source videos into 10 broad social-risk domains and 44 event-specific categories before segmentation, retaining the collected distribution without quota balancing.Table 9 reports category definitions and source counts at the source-video level.Source: §A.1 · p. 32, Table 9 · p. 32

A.2. Source Inventory and Rights Basis

  • Government and public-institution sources contribute 348 videos, while open-license repositories contribute 191; together they provide 539 of 675 sources, or 79.9%.Rights bases are recorded at the source level, but redistribution decisions are made separately for each source during rights review.Source: §A.2 · p. 33, Table 10 · p. 33

A.3. Automated Preprocessing Details

  • Scene segmentation yields 5,774 reviewed clips using PySceneDetect, minimum 15-frame scenes, short-scene merging, and FFmpeg re-encoding; a near-duplicate prefilter produces 1,269 warnings across 350 clusters without automatic removal.Warnings are reviewed manually rather than treated as exclusion decisions.Source: §A.3 · pp. 33–34

A.4. Manual Review Details

  • Manual review resolves all 5,774 clips through two independent Round 1 assessments and adjudication of disagreements or uncertain cases, retaining 2,426 clips and rejecting 3,348.Seven volunteers conduct Round 1 review, while four additional volunteers adjudicate routed clips using visual, semantic, duration, duplication, and source-risk criteria.Source: §A.4 · p. 34

A.5. Postprocessing Details

  • Postprocessing produces 1,830 real clips from 338 source videos after duration bounding and 595 homogeneity removals, totaling 18,448 seconds with a 10.08-second mean duration.Released media includes 1,319 publicly cleared or conditional anchors; 511 additional anchors are represented only by metadata and source URLs.Source: §A.5 · p. 34

A.6. Generation Details

  • Generation uses a two-stage visual-caption and generator-agnostic prompting pipeline, sharing each prompt and anchor first frame across sources while applying source-specific duration and temporal-grid mappings.Prompts are visual-only, identity-free, grounded in observable content, and paired with the real anchor’s first frame for I2V generation.Source: §A.6 · pp. 35, 37, 38

A.7. Qualitative Real-versus-Generated Examples

  • Figures 12 and 13 qualitatively show generated clips spanning all ten social-risk domains and matched cross-generator outputs that preserve initial layouts while varying subsequent motion and visual drift.Figure 12 pairs real anchors with generated counterparts; Figure 13 fixes the anchor, prompt, and first-frame reference while changing open-source configurations.Source: Figure 12 · p. 40, §A.7 · p. 40, Figure 13 · p. 41

B.1. Detector Evaluation Protocol and Model Inputs

The evaluation pairs generated videos with matched real anchors, preserves released detector protocols, and compares methods using common metrics without auxiliary metadata. It also treats temporal evidence as part of each end-to-end system while separately controlling temporal allocation in an eight-frame ablation.

Evaluation protocol

  • Each generation source is evaluated against matched real anchors, with benchmark averages equally weighting nine RA-Bench sources and excluding the Wan2.2 control.Open-source generators contribute 1,830 pairs each; closed-source generators use only anchors with returned outputs.Source: §B.1 · p. 41
  • Released checkpoints or API versions, preprocessing, temporal sampling, score extraction, and score direction are preserved across methods unless explicitly ablated.Continuous outputs use paired AUC and T@5%; discrete outputs use BAcc, macro-F1, and FakeR, with RealR additionally reported for class-preference analyses.Source: §B.1 · p. 41

Model inputs

  • Zero-shot multimodal models receive only sixteen temporally ordered frames resized to short side 256, producing binary, rated, or diagnostic verdict outputs without auxiliary metadata.They are not given filenames, generator identities, seeds, dataset labels, or other auxiliary information.Source: §B.1 · pp. 41–42, Table 14 · p. 42
  • Detectors observe different temporal evidence: frame-level models sample isolated frames, DeCoF emphasizes opening frames, D3 and ReStraV use local windows, and multimodal models summarize multiple frames.These differences follow released inference protocols, so the main comparison treats each detector as an end-to-end system.Source: §B.1 · p. 42

Temporal-sampling control

  • The temporal-sampling control changes only allocation of a fixed eight-frame input, contrasting uniform sampling with four event-centered consecutive frames plus four uniformly spaced frames.The experiment covers eight fixed-length RA-Bench sources and the fixed-duration Wan2.2 control, reports macro-average AUC across nine settings, and uses a source-block bootstrap for the five-detector mean confidence interval.Source: §B.1 · pp. 42–43

B.2. Traditional-Detector Reference Transfer and Operating Points

Traditional-detector rankings transfer poorly from public references to RA-Bench, and practical detection remains weak across low-false-positive and high-recall operating points. Detector performance and operating trade-offs vary substantially by generation source, with no method consistently reliable across sources.

Reference transfer

  • Public detector rankings transfer poorly: the public-to-RA-Bench Spearman correlation is 0.26, with UnivFD falling from second to sixth and NPR rising from sixth to third.The analysis uses six detectors sharing the AIGVDBench LTX-I2V reference; ReStraV remains in absolute comparisons but is excluded from rank correlation.Source: Table 15 · p. 43, §B.2 · p. 43

Operating points

  • Source-specific operating points remain unstable: ReStraV’s F@95% ranges from 35.8% on Kling to 96.0% on Seedance2.0.Among 63 detector–source pairs, 54 have T@5% below 10% and 52 require at least 80% FPR for 95% TPR, so no detector maintains both goals across sources.Source: Figure 14 · p. 44, §B.2 · pp. 44–45
  • Across nine sources, seven-detector mean T@1% is 0.4–1.7%, T@5% is 2.8–7.5%, and F@95% is 83.1–91.8%.These stricter endpoints use complete ROC curves rather than default thresholds; random-ranking values are 1%, 5%, and 95%, respectively.Source: §B.2 · pp. 43–44, Table 16 · p. 44
  • Similar AUC can mask different trade-offs: ForgeLens has higher T@5% than DeCoF, while DeCoF achieves lower F@95% across both open- and closed-source groups.ForgeLens versus DeCoF: open-source T@5% is 18.3% versus 3.9% and F@95% is 84.7% versus 73.2%; closed-source values are 9.7% versus 2.0% and 85.3% versus 77.6%.Source: §B.2 · p. 45

B.3. Zero-Shot Multimodal Model Protocol and Extended Analysis

The unified zero-shot evaluation fixes visual evidence and excludes metadata while comparing Binary, Rating, and Diagnostic prompts across source-equal generator groups. Results remain near chance and require class-conditional, continuous-ranking, and structured-output analyses because aggregate accuracy obscures prompt-dependent biases and diagnostic collapse.

Unified evaluation protocol

  • All models receive 16 ordered 256-pixel frames without metadata, while Binary, Diagnostic, and Rating vary only the requested output format and retain the same real-versus-generated task.Aggregates weight generation sources equally: Open averages four generators, Closed five, and All nine; the fixed-duration Wan2.2 control is excluded.Source: §B.3 · p. 45

Aggregate performance

  • Aggregate zero-shot performance remains near chance for many model–prompt pairs, with no prompt consistently improving as model scale increases across source groups.Qwen3.5-122B-A10B improves from 53.0/45.7 BAcc/macro-F1 under Binary to 54.8/50.8 under Diagnostic, while Qwen3.5-27B declines from 53.2/46.7 to 51.3/37.2; Qwen3.7-Plus and Gemini-3.1-Pro-Preview Rating AUC also decreases between open and closed sources.Source: §B.3 · pp. 46–47, Table 18 · p. 47

Prompt-dependent operating points

  • Prompt-dependent class recall reveals qualitatively different operating points that can share near-50% BAcc, so FakeR and RealR are necessary to expose class preference.Qwen3.5-0.8B shifts from Real-favoring Binary predictions to nearly all-fake Diagnostic and Rating predictions, whereas Qwen3.5-4B and 9B shift oppositely; GPT-5.5 predicts below 4.5% FakeR and above 99% RealR across prompts.Source: §B.3 · pp. 47–48, Figure 15 · p. 47

Continuous ranking and categorical decisions

  • Rating scales are not comparable across models: Qwen3.5-2B scores generated and real videos 81.7 and 77.5, while Qwen3.5-9B scores them 1.6 and 1.1 despite both near-chance AUC.GPT-5.5 reaches 62.1% paired AUC but only 36.9% verdict macro-F1, whereas Gemini-3.1-Pro-Preview is the only model above 60% on both measures.Source: §B.3 · p. 48, Figure 16 · p. 48

Five-aspect diagnostic

  • Diagnostic outputs often collapse to one global score: Qwen3.5-0.8B, 4B, 9B, and 27B exceed 94% exact-five collapse while aspect AUC remains near chance.Gemini-3.1-Pro-Preview reaches 62.6–63.8% aspect AUC but repeats one value across all dimensions in 55.2% of outputs; GPT-5.5 rarely collapses, yet Temporal and Motion AUC are 44.9% and 48.6%.Source: §B.3 · p. 48, Figure 17 · p. 49

Extended analysis

  • Reliable zero-shot assessment therefore reports class-conditional recall, Rating AUC, explicit verdict metrics, Diagnostic aspect AUC, and collapse tests rather than treating one aggregate score as sufficient.These measures distinguish prompt-induced class preference, continuous ranking, and whether structured responses contain dimension-specific information.Source: Table 17 · p. 45, §B.3 · p. 49

B.4. Fine-Tuned MLLM Protocol Sensitivity and Class-Conditional Behavior … C.2. Generation Settings and Detectability

The analyses show that fine-tuned MLLM detection is highly sensitive to temporal-label artifacts and class-conditional biases, while generation quality and conditioning affect detector families differently. Adding matched end frames consistently lowers Skyra fake recall across seeds, demonstrating that detectability depends on generation protocol as well as source and quality.

B.4. Fine-Tuned MLLM Protocol Sensitivity and Class-Conditional Behavior

  • The temporal-label effect reflects label-correlated metadata: exact 5.00-second timestamps occur in 53.2% of generated versus 4.0% of real ViF-CoT-4K samples and 99.9% versus 58.8% in ViF-Bench.On RA-Bench, exact-5-second clips receive higher fake recall for every eligible source under official timestamps, while frame indices remove the gap; single-stratum Wan2.2 and Hailuo show opposite shifts consistent with a temporal prior.Source: §B.4 · pp. 49–50
  • Frame-index Skyra performance remains source-dependent, with BAcc ranging from 31.1% to 61.0% for SFT and 29.7% to 61.3% for RL, despite aligned source patterns.Source-wise correlations remain high under timestamps and frame indices (ρ=0.93 and 0.95), while BusterX++ predicts Real for about 93.6% of inputs, yielding FakeR of only 4.1–9.1% despite RealR of 93.0–93.8%.Source: §B.4 · pp. 50–51
  • Replacing timestamps with frame indices reduces Skyra-SFT and Skyra-RL’s exact-5-second fake-recall gaps from 48.6 to 1.5 and 43.4 to 1.4 points, respectively.The same intervention changes class preference: source-equal FakeR rises 4.0 and 5.5 points, while full-set RealR falls 32.0 and 34.5 points, leaving BAcc near 55%.Source: §B.4 · p. 49, Figure 18 · p. 51

C.1. Generation Quality and Detectability

  • VBench++ quality aggregates vary narrowly across sources, yet equal Combined Quality can accompany sharply different detection difficulty: LTX and Seedance2.0 both score 83.7 while Gemini Diagnostic fake recall is 74.5% versus 32.3%.The benchmark evaluates eight quality and conditioning dimensions on all 16,056 generated clips, using custom RA-Bench inputs rather than official VBench++ leaderboard data.Source: §C.1 · pp. 51–53
  • Condition Fidelity has the largest aggregate association with Gemini outputs, whereas VBench-I2V Quality Score has the largest association with the traditional-detector mean.Subject and background consistency measures are negative across detector outputs, Aesthetic and Imaging Quality show no clear Gemini Diagnostic association, and Dynamic Degree changes oppositely for Gemini but is nearly null for traditional detectors.Source: §C.1 · p. 53, Table 22 · p. 53

C.2. Generation Settings and Detectability

  • Adding a matched last frame lowers FakeR by 17.4 and 19.3 points for Skyra-SFT and by 17.5 and 15.6 points for Skyra-RL, while BusterX++ falls only 2.0 points.Because the real-video control is fixed within each configuration, these BAcc differences are driven by FakeR; BusterX++ starts at only 6.0% FakeR under first-frame I2V.Source: §C.2 · p. 54, Table 23 · p. 54
  • Across seeds 0, 42, and 123, adding the matched last frame consistently decreases FakeR by 15.1–17.4 points for Skyra-SFT, 16.2–17.9 for Skyra-RL, and 2.0–2.5 for BusterX++.The unchanged direction and small cross-seed ranges indicate that the seed-0 generation-setting effects are not specific to one sampled realization.Source: Figure 19 · p. 54, Table 24 · p. 55, §C.2 · p. 55

C.3. Stability Across Generation Seeds · D. Human Evaluation and Social Dissemination

Across controlled seed changes, traditional-detector rankings and source-level conclusions remain highly stable, while fine-tuned MLLM performance varies only modestly. The supplied passages contain no evidence for the merged section’s human-evaluation or social-dissemination subsection.

C.3. Stability Across Generation Seeds

  • The controlled comparison regenerates clips for the same 1,830 real-video anchors with seeds 42 and 123, holding prompts, conditioning images, and other settings fixed.Closed-source generators are excluded because their APIs lack controllable seeds; fixed-duration Wan2.2 is also omitted.Source: §C.3 · p. 55
  • Traditional detector AUC means vary by at most 1.05 points across seeds, with seed-0 source means within 0.20 AUC points of three-seed averages.The comparison covers 28 detector–source pairs across seeds 0, 42, and 123.Source: §C.3 · p. 55, Table 25 · p. 56
  • Across the 28 detector–source values, seed-pair Spearman correlations are 0.981, 0.978, and 0.989, despite ReStraV on LTX showing the largest 5.74-point AUC range.Only NPR on Wan2.2 dynamic, D3 on Wan2.2-Lightning, and UnivFD on LTX cross 50% AUC, remaining near-random under all seeds.Source: §C.3 · p. 55, Table 25 · p. 56
  • Fine-tuned MLLM seed sensitivity is limited: FakeR ranges are 2.35, 2.46, and 0.71 points for Skyra-SFT, Skyra-RL, and BusterX++, respectively.Because RealR is constant within each model, the corresponding BAcc ranges are half as large.Source: §C.3 · p. 55, Table 26 · p. 57
  • The corresponding BAcc ranges are 1.17, 1.23, and 0.36 points for Skyra-SFT, Skyra-RL, and BusterX++, respectively.Table 26 reports source-wise seed sensitivity using the stated official-timestamp prompt or released evaluation pipeline.Source: §C.3 · pp. 55–56, Table 26 · p. 57
  • Skyra preserves pronounced source differences across seeds, whereas BusterX++ remains below 10% FakeR for every source–seed combination.These results indicate that the source-level conclusions are not driven by seed-0 generations.Source: §C.3 · p. 56

D.1. Human Evaluation Protocol and Source-Level Recognition

Human evaluation on RA-Bench shows moderate recognition of generated videos, with substantial source-level differences and stable open-versus-closed performance patterns despite large reviewer variation.

Human Evaluation Protocol and Source-Level Recognition

  • The Stage 1 analysis covers 17,850 videos and 53,550 judgments, comprising 1,812 real videos and 16,038 generated videos reviewed by three different reviewers each.The remaining 36 videos were reserved for internal assignment-level quality control and excluded from reported recognition statistics and RA-Bench-HumanProof construction.Source: §D.1 · p. 57
  • Reviewers independently label each video as Real, Uncertain, or Generated after standardized training, with randomized interleaving of sources and no displayed source or label information.An answer is correct when real videos are labeled Real or generated videos are labeled Generated; Uncertain responses are reported separately.Source: §D.1 · p. 57
  • 60.3% of generated-video judgments and 71.9% of real-video judgments are correct, while source-equal human FakeR is 68.6% for open-source and 52.9% for closed-source generators.Seedance2.0 and Kling are hardest to recognize as generated, at 40.7% and 45.1% FakeR, whereas OmniWeaving reaches 75.8%.Source: Table 27 · p. 58, §D.1 · p. 58
  • Although individual FakeR ranges from 18.8% to 87.5%, every reviewer has higher source-equal FakeR on open-source than closed-source generators, with a median gap of 15.7 points.Leaving out any reviewer preserves the complete nine-source ranking, and the largest source-level response-rate change is 2.9 points.Source: §D.1 · p. 58

D.2. RA-Bench-HumanProof Construction and Detector Evaluation

RA-Bench-HumanProof retains 633 generated videos that five reviewers independently judged Real, paired with their real anchors for source-matched detector evaluation. Human-deceptive videos expose overlapping failures: Gemini loses fake-side evidence, traditional detectors remain weak, and timestamp-dependent MLLMs are unreliable.

RA-Bench-HumanProof construction

  • The benchmark retains 633 generated videos after five Real judgments, comprising 119 open-source and 514 closed-source videos.Stage 1 selects 1,080 candidates, and Stage 2 retains 58.6%; closed-source retention is 61.6% versus 48.4% for open-source candidates.Source: §D.2 · pp. 58–59

Paired evaluation and source matching

  • Detector evaluation uses 633 generated–real pairs linked to 511 unique real videos and reports BAcc, FakeR, RealR, paired AUC, and T@5%.Source-matched references preserve RA-Bench-HumanProof’s source proportions while controlling source mixture, not conditional selection.Source: §D.2 · pp. 59, 61

Detector-family results

  • Gemini’s Binary and Diagnostic BAcc each decrease by 6.5 points, while FakeR decreases by 15.6 and 17.3 points, respectively, after human selection.The decline is concentrated on generated videos: FakeR falls to 34.3% and 30.0%, while RealR remains 75.2% and 79.0%.Source: §D.2 · p. 60
  • Skyra-SFT and Skyra-RL retain 72.0% and 74.5% BAcc with official timestamps, but both fall to 53.7% with frame indices.BusterX++ reaches 49.4% BAcc by labeling only 3.9% of generated videos as fake while retaining 94.9% of real videos.Source: §D.2 · pp. 60–61
  • Human-deceptive videos expose overlapping detector failures: Gemini loses fake-side evidence, traditional detectors remain weak, and timestamp-dependent MLLMs are unreliable.Videos repeatedly judged Real by reviewers also receive weak or unreliable fake evidence, although human and detector failures are not equivalent.Source: §D.2 · pp. 60–61, Table 29 · p. 60

D.3. RA-Bench-LastMile Protocol and Complete Results

RA-Bench-LastMile evaluates detector robustness on 150 matched real-event anchors and 1,350 generated videos per condition across six social-dissemination conditions. Dissemination substantially disrupts both traditional and fine-tuned MLLM detectors, with detector-specific failure patterns and very low Full-condition reliability.

Evaluation subset

  • The protocol retains 150 real-event anchors spanning 41 of 44 L2 subcategories, pairing each condition with 1,350 generated videos for 1,500 videos and 9,000 instances overall.Anchors are selected deterministically using subcategory-aware sampling and fixed SHA-256 ordering; the Wan2.2 fixed-duration variant is auxiliary rather than a separate source.Source: §D.3 · p. 61

Social dissemination simulation

  • Six dissemination conditions apply a common VP9-to-H.264 transcode, with spatial, temporal, presentation, and Full variants evaluated as additions to that baseline.Full applies all four operations sequentially, and the same transformations are applied identically to each generated video and its matched real anchor.Source: §D.3 · p. 61, Table 30 · p. 61

Evaluation and uncertainty

  • Results use equal-weight averages across nine generation sources with anchor-clustered bootstrap confidence intervals, and excluding pre-existing upper-right badges changes AUC by at most 2.8 points and FakeR by 4.3 points.The robustness analysis uses 2,000 stratified bootstrap replicates and complete predictions across all sources and conditions.Source: §D.3 · p. 62, Table 32 · p. 63

Complete detector results

  • Under Full dissemination, the seven-detector mean AUC falls 4.2 points and T@5% reaches only 2.8%, while detector responses vary sharply and ordering becomes unstable.ForgeLens loses 26.0 AUC points, whereas DeCoF and D3 gain 2.4 and 4.8 points; the Original-to-Full ordering correlation is 0.07.Source: §D.3 · p. 62, Table 31 · p. 63
  • Under Full dissemination, every fine-tuned MLLM configuration reaches at most 2.4% FakeR, with spatial, temporal, and presentation transformations producing distinct failures.Spatial downsampling reduces mean FakeR from 29.9% to 7.9%; 8-fps conversion disproportionately harms official-timestamp Skyra configurations, while the news badge shifts mean FakeR from 29.9% to 14.0% and RealR from 79.1% to 88.5%.Source: §D.3 · p. 62, Table 31 · p. 63

Related papers

Loading 2608.14391v1…