PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination

Shuo Liang, Yixing Ma, Pengfei Zhou, Xingyan Chen, Zihan Mei, Manting Li, Feihan Chen, Zhiwen Wang, Bin Xu, Haotian Zhang, Jiajun Song, Shiya Su, Run Liu, Zhenghang Ni, Yifa Yu, Jintao Hong, Bolong Feng, Yifei Liu, Zirui Zhang, Jingxuan Zhang, Songlin Zhao, Yifan Bai, Kang Tan, Yizhe Liu, Junhao Du, Yongtao Ge, Zhaopan Xv, Xinyuan Zhang, Mengru Ma, Chunhua Shen, Wei Wang, Yang You, Zheng Zhu, Kaipeng Zhang, Wangbo Zhao

arXiv:2608.14391v1cs.CVcs.AI

TL;DR

Existing studies offer limited evidence about detecting AI-generated videos of real-world crises. This paper introduces RA-Bench to evaluate detectors, generation conditions, human judgments, and dissemination, finding that current detectors do not generalize reliably.

  • Problem

    Existing studies provide limited systematic evidence about detector performance and generator behavior for videos depicting real-world crises and socially consequential events.

  • Method

    RA-Bench uses 1,830 real-video anchors and 16,056 clips from nine generators to evaluate detectors, generation properties, human judgments, and social dissemination.

  • Results

    Detector performance fails to transfer consistently across RA-Bench sources, while generation properties affect detector families differently and social dissemination makes detection harder.

  • Takeaways & Limitations

    Current methods cannot reliably detect realistic AI-generated videos, especially crisis depictions, underscoring the need for systems effective as generators evolve and videos circulate.

  • Takeaways & Limitations

    RA-Bench is a time-sensitive snapshot covering nine I2V sources and visual-only detection, with I2V generation and dissemination evaluated as separate controlled stages.

Abstract

from arXiv · show

Recent video generators can fabricate realistic depictions of wars, disasters, public emergencies, and other real-world crises, creating substantial risks of misinformation. Existing benchmarks, however, provide limited evidence on detector and generator behavior in such settings, including how detectability varies with generation conditions, how people perceive generated videos, and whether detectors remain reliable during social dissemination. To address this gap, we introduce RA-Bench, a benchmark for AI-generated video detection that uses Real videos as Anchors. RA-Bench contains 17,886 videos, comprising 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from four open-source and five closed-source generators. Based on RA-Bench, we organize our evaluation along three dimensions. We first assess detector generalization across seven traditional detectors, ten zero-shot multimodal models under three review settings, and two MLLMs specifically fine-tuned on AI-generated video detection. Across these methods, none of the three detector families generalizes consistently across RA-Bench instances. We then examine how detectability varies with generation quality, conditioning information, and sampling seeds. These analyses show that generation properties affect detector families differently, while source-level detection patterns remain stable across seeds. Finally, we study human authenticity judgments and detector reliability during social dissemination. We find that videos that mislead people are also difficult for current detectors, and that social dissemination makes detection harder. Together, these findings show that current methods struggle to detect realistic AI-generated videos, highlighting the need for detectors robust to evolving video generators.

1. Introduction

RA-Bench addresses the limited evaluation of AI-generated videos depicting real-world crises by using real videos as anchors and testing detectors, generation properties, human judgments, and social dissemination. The evaluation finds that current detection methods do not reliably generalize to realistic generated crisis videos, especially after social dissemination.

  • Benchmark: RA-Bench contains 1,830 real-video anchors from 675 source videos across 10 social-risk categories and 44 subcategories.Image-to-video generators are conditioned on each anchor’s first frame to simulate real-image-based misinformation.
  • Evaluation: The study evaluates seven traditional detectors, ten zero-shot multimodal models under three review settings, and two fine-tuned MLLMs across detector generalization, generation properties, and dissemination.Generation analyses vary quality, conditioning information, and sampling seeds.
  • Detector generalization: 67.6–98.6% public-reference AUCs fall to 43.9–57.3% source-level means for traditional detectors on RA-Bench, with rankings changing across generators.None of the three detector families generalizes consistently across RA-Bench sources.
  • Social dissemination: 46.0% to 1.4% is the reduction in mean FakeR under the Full social-dissemination condition across five fine-tuned configurations.The findings indicate that social dissemination makes detection harder and reinforce the need for systems robust to evolving generators and real-world circulation.

2. Related Work

Prior AI-generated video benchmarks have expanded in scale, generator coverage, and evaluation protocols, but remain limited in real-world crisis and socially consequential misuse scenarios. Existing detectors are broadly organized into traditional discriminative and MLLM-based approaches, motivating RA-Bench’s systematic evaluation of detector robustness in these settings.

  • Benchmarks for AI-Generated Video Detection: Existing benchmarks progressed from pairing real videos with generated counterparts to broader coverage of generators, world-simulation settings, and source backtracking.Early efforts include GVF and GenVideo, which evaluate cross-generator transfer; later work expands benchmark scale, generator coverage, and evaluation protocols.
  • Benchmarks for AI-Generated Video Detection: Existing benchmarks remain limited in covering misuse scenarios involving real-world crises and other socially consequential events.RA-Bench is introduced to address this gap and systematically assess detector robustness and limitations in these settings.
  • Detectors for AI-Generated Videos: AI-generated video detectors are broadly divided into traditional discriminative approaches and MLLM-based approaches.Traditional approaches include video classification, generated-image detection, and generated-video detection models.
  • RA-Bench Construction: RA-Bench constructs real-video anchors from public crisis-related videos through scene-level segmentation, near-duplicate screening, two-round manual review, and release-oriented postprocessing.Each retained anchor is paired with image-to-video generated counterparts using first-frame conditioning and a shared prompting pipeline.

3. RA-Bench

RA-Bench is built from real-world crisis videos organized into social-risk categories, rigorously reviewed and standardized, then paired with image-to-video counterparts generated from the same scene semantics. The resulting benchmark contains 1,830 real-video clips and 16,056 generated clips from four open-source and five closed-source generators.

  • Source collection: 675 public videos are organized into 10 social-risk categories and 44 subcategories to represent crises and other socially consequential events.The categories include weather and natural disasters, war and armed conflict, politics and governance, public safety, accidents and infrastructure failures, economic and social panic, public health, technology, space and exploration, and additional domains.
  • Clip preparation: 5,774 scene-level clips are produced from 675 source videos, then filtered for near-duplicates before human review.Scene cuts are detected with PySceneDetect and split using FFmpeg; a pretrained ResNet-18 compares cosine similarities between neighboring clips from the same source video.
  • Human review: Two-round manual review retains 2,426 clips using criteria covering visual quality, semantic fit, duration suitability, duplicate content, and source-related risks.Seven volunteers conduct the first round, with two independent reviewers per clip; disagreement and uncertainty are adjudicated by another four volunteers in Round 2.
  • Anchor standardization: Real clips are standardized to 3–15 seconds and H.264 encoding, with homogeneity pruning and licensing review applied before forming the final anchor set.The shared codec reduces the risk that detectors exploit codec configurations as label cues.
  • Generated-video construction: Each real clip is paired with I2V counterparts conditioned on its caption and first frame, yielding 16,056 generated clips alongside 1,830 real anchors.The generated set uses four open-source and five closed-source generators; open-source models cover all anchors, while safety filters reduce some closed-source paired subsets.

4. Experiments and Analysis

Across traditional detectors, zero-shot multimodal models, and fine-tuned MLLMs, detection performance fails to generalize consistently across RA-Bench sources. Prompt and temporal-label sensitivity, generation properties, and social-risk conditions further limit reliability.

  • Zero-shot multimodal models: Zero-shot multimodal models remain limited, and scaling does not remove sensitivity to prompt format or generation source.For Qwen3.5, Rating AUC rises from 51.0 to 65.8 on LTX but only from 50.5 to 52.7 on Hailuo when scaling from 0.8B to 122B-A10B.
  • Fine-tuned MLLMs: Timestamp replacement lowers Skyra-SFT and Skyra-RL mean BAcc from 68.5% and 69.5% to 54.4% and 54.9%, respectively.Using the same 16 visual frames, the exact-5-second FakeR gaps shrink from 48.6 to 1.5 points for Skyra-SFT and from 43.4 to 1.4 points for Skyra-RL.
  • Fine-tuned MLLMs: Under frame indices, Skyra per-source BAcc still spans 31.1–61.0% for Skyra-SFT and 29.7–61.3% for Skyra-RL, showing residual source-specific failures.BusterX++ instead keeps FakeR between 4.1% and 9.1% with macro-F1 between 36.1% and 40.8%.
  • Traditional detectors: Traditional-detector mean AUC falls from 67.6–98.6% in public references to 43.9–57.3% across RA-Bench sources, with different detectors leading by generator.Public-reference rankings transfer poorly: Spearman correlation is 0.26, while DeCoF and ForgeLens each lead four sources and ReStraV leads Kling.
  • Generation properties: An interquartile increase in VBench-I2V Quality Score, Condition Fidelity, or Combined Quality decreases Gemini Diagnostic fake recall by 9.6, 14.4, or 12.4 percentage points, respectively.The traditional-detector mean also decreases by 9.5–11.5 percentile points across the three scores.

5. Discussion

The discussion shows that detector reliability is conditional on generation source, review settings, and dissemination, while human deception exposes failures that detector scores alone miss. RA-Bench is also a changing snapshot requiring versioned updates as generators and detectors evolve.

  • Detector reliability is conditional: Detector reliability varies across generation sources and review conditions, so performance on one generator does not establish transfer to another.Traditional detectors fall from public-reference performance and change rank across sources; zero-shot multimodal models vary with prompt format and source, while fine-tuned MLLMs depend on temporal-label format.
  • Human deception identifies socially consequential failures: 22.8% of real-video judgments are labeled Generated, while 633 generated videos are labeled Real by all five assigned reviewers.These 633 videos form RA-Bench-HumanProof, where Gemini performs near chance under all three prompt formats; traditional detectors show only a small additional decline.
  • Social dissemination shifts detector predictions toward Real: Under Full dissemination, traditional-detector mean AUC falls from 51.4% to 47.3%, while fine-tuned MLLM mean FakeR falls from 46.0% to 1.4%.Social dissemination shifts predictions toward Real and substantially changes traditional-detector rankings.
  • Limitations: RA-Bench is a snapshot of a rapidly changing field, covering nine I2V generation sources and visual-only detection rather than the complete misuse landscape.The benchmark should be maintained through versioned updates as new generators and detectors may quickly change its difficulty.

6. Future Directions

Future work should develop detectors that remain robust on increasingly realistic AI-generated videos and improve transparency through structured reasoning. Active watermarking in video generation models is another proposed direction for more reliable detection.

  • Robust detection: Future detectors should leverage multiple cues to address the performance gap on realistic videos produced by state-of-the-art generators.The passage identifies increasingly realistic AI-generated videos as a key challenge for existing detection methods.
  • Structured reasoning: MLLM-based agents and post-training approaches could decompose AI-generated video detection into structured reasoning steps.This decomposition is proposed to further enhance transparency.
  • Active watermarking: Active watermarking techniques in video generation models are proposed to improve the reliability of AI-generated video detection.The passage presents active watermarking as a future direction for reliable detection.

7. Conclusion · Appendix

RA-Bench provides a large benchmark for detecting AI-generated videos depicting real-world crises. Its evaluation finds inconsistent detector generalization across generation sources and family-specific sensitivity to generation properties.

  • 7. Conclusion: 17,886 videos comprise 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips.The generated clips come from four open-source and five closed-source generators.
  • 7. Conclusion: RA-Bench uses Real videos as Anchors to evaluate AI-generated video detection.
  • 7. Conclusion: None of the three detector families generalizes consistently across generation sources.
  • 7. Conclusion: Traditional-detector performance falls sharply from public-reference results.
  • 7. Conclusion: Zero-shot multimodal models remain sensitive to prompts and sources.
  • 7. Conclusion: Fine-tuned MLLMs depend on timestamp cues or exhibit strong class bias.
  • 7. Conclusion: Generation properties affect detector families differently.The passage identifies Condition Fidelity, temporal consistency, dynamic content, and real-image conditioning as properties with different relations to detector behavior.

A. RA-Bench Construction and Data Documentation … B. Detector Evaluation Protocols and Extended Results

RA-Bench documents a ten-domain, two-level taxonomy and a staged pipeline that converts 675 source videos into 1,830 real anchors and matched generated clips. Its evaluation materials standardize review, rights, prompting, generation, and qualitative comparisons across sources and configurations.

  • A.1. L2 Taxonomy and Source Distribution: 675 source videos are organized into 10 broad social-risk domains and 44 event-scope categories before segmentation and manual review, without quota rebalancing.Counts refer to source videos rather than review clips or released anchors.
  • A.2. Source Inventory and Rights Basis: 539 of 675 source videos (79.9%) come from government or public-institution sources and open-license repositories, with rights recorded at source level.Final redistribution decisions are made separately for each source during rights review.
  • A.4. Manual Review Details: 2,426 of 5,774 reviewed clips are retained after two-reviewer assessment and adjudication, while 3,348 are rejected.Seven volunteers produce 11,548 clip–reviewer decisions; disagreements or uncertain judgments receive one of four adjudicators.
  • A.5. Postprocessing Details: 1,830 real clips from 338 source videos remain after duration bounding and homogeneity pruning, totaling 18,448 s with a mean duration of 10.08 s.Duration bounding removes one clip, while homogeneity pruning removes 595 clips from the post-review pool.
  • A.6. Generation Details: The pipeline uses visual-only Stage A captions and shared generator-agnostic Stage B prompts, with the same prompt sent to every generator.Prompts are grounded in visible content, identity-free, single-shot, and constrained to 100–180 words.
  • A.6. Generation Details: For real-anchor duration treal ∈[3, 15] s, the shared generation target is tgen = clamp(treal · 8/15, 2, 8) s before source-specific temporal-grid conversion.Wan2.2 and OmniWeaving use 4n+1 at 16 fps, LTX uses 8n+1 at 24 fps, and Hailuo discretizes targets to 6- or 10-second outputs.
  • A.7. Qualitative Real-versus-Generated Examples: Qualitative comparisons cover all ten social-risk domains and matched cross-generator configurations, showing preserved conditioning-frame layouts alongside differing subsequent motion and visual drift.Each filmstrip contains six uniformly sampled frames, with the real first frame marked as the generator’s conditioning input.

B.1. Detector Evaluation Protocol and Model Inputs

The evaluation pairs generated videos with matched real anchors, preserves released detector protocols, and reports common metrics without post-hoc score inversion. Methods receive differing temporal evidence, while zero-shot multimodal models use sampled frames without auxiliary metadata and the temporal-sampling control isolates frame allocation.

  • Evaluation protocol: Each open-source generator contributes 1,830 generated–real anchor pairs, while closed-source evaluation uses returned videos; benchmark results equally weight nine generation sources.The fixed-duration Wan2.2 control is excluded from benchmark averages and marked with an asterisk.
  • Evaluation protocol: Released checkpoints or API versions, preprocessing, temporal sampling, and score extraction are preserved; continuous outputs use AUC and T@5%, while discrete outputs use BAcc, macro-F1, and FakeR.Continuous score directions remain fixed, with no post-hoc inversion when AUC is below 50%.
  • Model inputs: Zero-shot multimodal models receive sixteen temporally ordered frames at short side 256 and no filename, generator identity, seed, dataset label, or other auxiliary metadata.Their outputs are binary verdicts, 0–100 ratings with verdicts, or five-aspect diagnostics with verdicts.
  • Model inputs: Detectors use different temporal evidence: frame-level models sample isolated frames, DeCoF emphasizes opening frames, D3 and ReStraV use local windows, and multimodal models receive broader summaries.The main comparison therefore treats each detector as an end-to-end system under its released inference protocol.
  • Temporal-sampling control: The temporal-sampling control changes only allocation of a fixed eight-frame input across eight fixed-length RA-Bench sources and the fixed-duration Wan2.2 control, excluding dynamic-duration Wan2.2.It reports macro-average AUC over nine settings with a source-block-bootstrap confidence interval for the five-detector mean.

B.2. Traditional-Detector Reference Transfer and Operating Points

Public detector rankings transfer poorly to RA-Bench, while stricter operating-point results show weak detection across generation sources and trade-offs that vary by detector and source. Similar AUC can conceal substantially different low-FPR and high-recall behavior.

  • Reference transfer: 0.26 Spearman correlation separates public and RA-Bench mean detector rankings, with UnivFD moving from second to sixth and NPR from sixth to third.Per-source correlations range from −0.37 to 0.54, with a median of 0.31.
  • Operating points: 0.4%–1.7% is the seven-detector mean TPR range at 1% FPR across nine RA-Bench sources, rising to 2.8%–7.5% at 5% FPR.Reaching 95% TPR requires mean FPR of 83.1–91.8%; 54 of 63 detector–source pairs remain below 10% T@5%.
  • Operating points: 83.1–91.8% mean FPR is required to reach 95% TPR, and 52 of 63 detector–source pairs require at least 80% FPR.These stricter endpoints translate the low AUCs into weak practical operating regions.
  • Operating-point trade-offs: ForgeLens and DeCoF have nearly tied mean AUC on open-source generators (64.2% versus 63.4%) and closed-source generators (58.7% versus 58.9%), but different operating trade-offs.ForgeLens has higher T@5% (18.3%/9.7%) with F@95% of 84.7%/85.3%, whereas DeCoF has lower T@5% (3.9%/2.0%) and F@95% of 73.2%/77.6%.
  • Source variability: 35.8%–96.0% is ReStraV’s F@95% range across sources, from Kling to Seedance2.0, showing source-specific instability.No evaluated detector maintains both low false-positive rates and high recall across generation sources.

B.3. Zero-Shot Multimodal Model Protocol and Extended Analysis

Zero-shot multimodal models were evaluated under a standardized metadata-free protocol using 16 ordered frames and three prompt formats. Results show that aggregate BAcc obscures prompt-dependent class preferences, divergent continuous and categorical behavior, and weak aspect-specific diagnostic reasoning.

  • Evaluation protocol: All models received the same 16 temporally ordered 256-pixel frames without filenames, sources, seeds, datasets, watermarks, or other metadata, under Binary, Diagnostic, and Rating prompts.Aggregate results weight generation sources equally rather than individual clips; Open, Closed, and All cover four, five, and nine generators, respectively.
  • Aggregate performance: Aggregate performance remains near chance for many model–prompt pairs, with no prompt consistently benefiting larger models and source-group performance varying across models.Qwen3.5-122B-A10B improves from 53.0/45.7 BAcc/macro-F1 under Binary to 54.8/50.8 under Diagnostic, while Qwen3.7-Plus and Gemini-3.1-Pro-Preview show lower Rating AUC on closed-source generators.
  • Prompt-dependent operating points: Prompt formats can induce opposite class preferences while leaving BAcc near 50%, so FakeR and RealR are necessary to expose operating-point differences.Qwen3.5-0.8B shifts from a Real-favoring Binary rule to predicting nearly every video as fake under Diagnostic and Rating; Qwen3.5-4B and 9B shift oppositely.
  • Continuous versus categorical outputs: 62.1% paired AUC can coexist with 36.9% verdict macro-F1, showing that continuous ranking and categorical decisions must be evaluated separately.GPT-5.5 assigns low scores to both classes while predicting almost every video as real; Gemini-3.1-Pro-Preview is the only model with both paired AUC and verdict macro-F1 above 60%.
  • Diagnostic aspect analysis: 94%+ exact-five collapse occurs for Qwen3.5-0.8B, 4B, 9B, and 27B, whose Diagnostic aspect AUC remains near chance because they repeat one global score.Gemini-3.1-Pro-Preview reaches 62.6–63.8% aspect AUC but assigns the same value across all five dimensions in 55.2% of outputs; GPT-5.5’s Temporal and Motion AUCs are 44.9% and 48.6%.

B.4. Fine-Tuned MLLM Protocol Sensitivity and Class-Conditional Behavior … C.2. Generation Settings and Detectability

Temporal-label formatting materially changes Skyra’s class preferences and exposes label-correlated priors, while generation quality and conditioning affect detector outputs in distinct, non-causal ways. Adding a matched last frame lowers fine-tuned-MLLM fake recall consistently across seeds, showing that generation-setting effects are robust rather than seed-specific.

  • B.4. Fine-Tuned MLLM Protocol Sensitivity and Class-Conditional Behavior: 32.0 points: Skyra-SFT loses this much full-set RealR with frame indices, while Skyra-RL loses 34.5 points; both BAcc values fall to about 55%.Skyra-SFT gains 4.0 points in source-equal FakeR and Skyra-RL gains 5.5 points, indicating changed class preference rather than uniformly reduced recall.
  • B.4. Fine-Tuned MLLM Protocol Sensitivity and Class-Conditional Behavior: 53.2% of generated ViF-CoT-4K samples end at exactly 5.00 seconds, compared with 4.0% of real samples, exposing a label-correlated temporal prior.In ViF-Bench, the corresponding proportions are 99.9% generated and 58.8% real, so the final temporal tag can act as a label prior.
  • B.4. Fine-Tuned MLLM Protocol Sensitivity and Class-Conditional Behavior: 48.6 to 1.5 points: replacing official timestamps with frame indices nearly eliminates the source-equal exact-5-second fake-recall gap for Skyra-SFT.For Skyra-RL, the corresponding mean gap falls from 43.4 to 1.4 points; both prompts use identical sampled frames.
  • C.1. Generation Quality and Detectability: 83.7: LTX and Seedance2.0 share this Combined Quality score, yet their Gemini Diagnostic fake recalls are 74.5% and 32.3%, respectively.Across sources, the VBench-I2V Quality Score spans 72.9–78.0, Condition Fidelity spans 91.1–96.4, and Combined Quality spans 83.7–86.7.
  • C.1. Generation Quality and Detectability: Condition Fidelity has the largest aggregate association with all three Gemini outputs, whereas the VBench-I2V Quality Score has the largest association with the traditional-detector mean.Subject and background consistency measures are negative across every detector output, while Aesthetic and Imaging Quality show no clear association with Gemini Diagnostic.
  • C.2. Generation Settings and Detectability: 1,830 anchor-derived prompts are evaluated under T2V, first-frame I2V, and first+last-frame I2V using matched prompts and seeds.T2V has no real-image condition; the two I2V settings use the matched first frame or matched first and last frames, with seed 0 as the main comparison and seeds 0, 42, and 123 for cross-seed analysis.
  • C.2. Generation Settings and Detectability: 17.4 and 19.3 points: adding the matched last frame lowers Skyra-SFT FakeR by these amounts across the two prompt variants.Skyra-RL decreases by 17.5 and 15.6 points, while BusterX++ decreases by only 2.0 points because its first-frame-I2V FakeR is already 6.0%.
  • C.2. Generation Settings and Detectability: 15.1 to 17.4 points: across seeds, the first+last-frame versus first-frame FakeR decrease for Skyra-SFT remains within this range.The corresponding ranges are 16.2 to 17.9 points for Skyra-RL and 2.0 to 2.5 points for BusterX++; the direction is unchanged across seeds.

C.3. Stability Across Generation Seeds · D. Human Evaluation and Social Dissemination

Across controlled seed evaluations, detector performance varies little at the source level and preserves stable cross-source patterns. Fine-tuned MLLM source-level conclusions likewise remain consistent beyond seed 0.

  • C.3. Stability Across Generation Seeds: 1,830 real-video anchors per generator are reused for seeds 0, 42, and 123, with prompts, conditioning images, and all other generation settings held constant.Closed-source providers are omitted because their APIs do not expose a controllable seed.
  • C.3. Stability Across Generation Seeds: 1.05 AUC points is the maximum variation across seeds for traditional detectors’ source-level seven-detector means.Seed-0 source means differ from their three-seed averages by at most 0.20 AUC points.
  • C.3. Stability Across Generation Seeds: 5.74 AUC points is the largest detector-specific range, observed for ReStraV on LTX.The remaining 28 detector–source comparisons are reported in Table 25.
  • C.3. Stability Across Generation Seeds: 0.981, 0.978, and 0.989 are the pairwise Spearman correlations across 28 AUC values for seed pairs 0/42, 0/123, and 42/123.Only NPR on Wan2.2 dynamic, D3 on Wan2.2-Lightning, and UnivFD on LTX cross 50% AUC, remaining close to random ranking under all seeds.
  • C.3. Stability Across Generation Seeds: 2.35, 2.46, and 0.71 points are the largest FakeR ranges for Skyra-SFT, Skyra-RL, and BusterX++, respectively.Because RealR is constant for each model, the corresponding BAcc ranges are half the FakeR ranges: 1.17, 1.23, and 0.36 points.
  • C.3. Stability Across Generation Seeds: BusterX++ remains below 10% FakeR for every source–seed combination, while Skyra retains pronounced source differences under all three seeds.These results indicate that source-level conclusions are not driven by the seed-0 generations used in the primary benchmark.

D.1. Human Evaluation Protocol and Source-Level Recognition

The human evaluation used a standardized three-review protocol over 17,850 videos, measuring separate recognition of real and generated content. Recognition was moderate overall, varied substantially by generator source, and remained consistently higher for open-source than closed-source generators across reviewers.

  • Human Evaluation Protocol: 17,850 videos received 53,550 judgments in the standard three-review stream, covering 1,812 real videos and 16,038 generated videos.Each video was assigned to three different reviewers; the remaining 36 videos were reserved for internal assignment.
  • Human Evaluation Protocol: Reviewers completed standardized training and labeled each video Real, Uncertain, or Generated, with an optional 16-frame contact sheet.The interface presented one video at a time and did not provide source or label information.
  • Source-Level Recognition: 60.3% of generated-video judgments and 71.9% of real-video judgments were correct, while Uncertain responses were reported separately.A judgment counted as correct when real videos were labeled Real or generated videos were labeled Generated.
  • Source-Level Recognition: 68.6% open-source FakeR exceeded 52.9% closed-source FakeR, while Seedance2.0 and Kling were hardest to recognize at 40.7% and 45.1%.OmniWeaving reached 75.8% FakeR, showing that the pooled 60.3% masked substantial source-level differences.
  • Source-Level Recognition: Reviewer FakeR ranged from 18.8% to 87.5%, yet every reviewer achieved higher source-equal FakeR for open-source than closed-source generators.The reviewer-level difference had a median of 15.7 points and ranged from 0.6 to 38.2 points.

D.2. RA-Bench-HumanProof Construction and Detector Evaluation

RA-Bench-HumanProof selects generated videos that repeatedly appear real to five reviewers and evaluates detectors using source-matched real anchors. Human-selected videos expose overlapping but distinct detector failures, while conditional construction and repeated anchors limit interpretation.

  • Construction: 1,080 of 16,038 generated videos passed Stage 1, and only candidates labeled Real by both additional reviewers entered RA-Bench-HumanProof.Any Uncertain or Generated response in Stage 2 excluded a candidate.
  • Evaluation: 633 generated–real pairs represent 511 unique real anchors, with evaluation reporting BAcc, FakeR, RealR, paired AUC, and T@5% as appropriate.The source-matched reference controls source proportions but not conditional selection from five human judgments.
  • Detector-family results: 49.5% to 47.5%: traditional detectors’ mean AUC decreased by 2.0 points, while mean T@5% changed from 4.6% to 4.4%.Individual AUCs ranged from 33.1% to 59.0%, and the source-matched baseline was already near random ranking.
  • Detector-family results: Gemini’s Binary and Diagnostic FakeR decreased by 15.6 and 17.3 points, reaching 34.3% and 30.0%, while corresponding RealR remained 75.2% and 79.0%.Its Binary and Diagnostic BAcc each fell by 6.5 points, indicating weaker fake-side evidence on videos misleading reviewers.
  • Detector-family results: 72.0% and 74.5% BAcc with official timestamps fell to 53.7% when timestamps were replaced with frame indices; BusterX++ labeled only 3.9% of generated videos fake while retaining 94.9% of real videos.These results indicate timestamp reliance or weak visual decisions rather than reliable detection.

D.3. RA-Bench-LastMile Protocol and Complete Results

RA-Bench-LastMile evaluates matched real and generated videos under controlled social-dissemination conditions, using equal-source aggregation and anchor-clustered bootstrap uncertainty. Social transformations substantially alter detector behavior, with traditional detectors showing unstable rankings and fine-tuned MLLMs exhibiting condition-specific failures.

  • RA-Bench-LastMile Protocol: The subset retains approximately 10% of common anchors per subcategory when more than 10 are available, one anchor for 1–10, and none for zero.A fixed SHA-256 ordering of normalized clip identifiers selects retained anchors within each subcategory.
  • RA-Bench-LastMile Protocol: Each dissemination condition contains 150 real videos and 1,350 matched generated videos, with T2–T4 added to the common T1 transcode.T1 applies VP9 and H.264 encoding; spatial, temporal, and presentation operations are evaluated separately or jointly under Full.
  • Complete Results: Traditional-detector and fine-tuned-MLLM metrics are averaged equally across nine generation sources, with 95% uncertainty estimated from 2,000 anchor-clustered bootstrap replicates.Bootstrap resampling is stratified by L1 category and uses random seed 20260718.
  • Complete Results: The seven-detector mean AUC decreases by 4.2 points under Full, while ForgeLens loses 26.0 points and DeCoF and D3 gain 2.4 and 4.8 points.Detector-ordering Spearman correlation with Original ranges from 0.07 to 0.46 across dissemination conditions, indicating unstable rankings.
  • Complete Results: Spatial downsampling reduces the five-configuration mean FakeR from 29.9% to 7.9%, the largest isolated FakeR loss among fine-tuned MLLM conditions.Conversion to 8 fps lowers BAcc by 11.7 and 12.9 points for official-timestamp Skyra configurations, versus 3.4 and 2.4 points for frame-index controls.
Loading 2608.14391v1…