BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos

• Diffusion-based video generation has advanced across closed- and open-source systems, but generated videos still exhibit artifacts that undermine realism and user trust.The introduction cites Gen3 [7] and SORA alongside open-source models including CogVideoX [32], HunyuanVideo [14], and Wan 2.1 [27].Source: §1 · pp. 1–2
• Forensic classification systems such as AIGVDet [26], DeCoF [1], and DIVID [16] output binary labels, limiting insight into where artifacts occur.Source: §1 · p. 2
• BrokenVideos contains 3,254 AI-generated videos from diverse generation methods, with pixel-level artifact masks produced using SAM2.Source: §1 · p. 2
• Fine-tuned segmentation models achieve state-of-the-art performance in artifact localization, demonstrating BrokenVideos’ usefulness for training and evaluation.Source: §1 · p. 2

2.1 Video Generation

• Video generation progressed from early text-prompt animation and CNN-based UNets to autoregressive and DiT-based systems, while imperfections remain.Examples include Make-A-Video [23], ModelScope [28], AnimateDiff [11], VideoCrafter [4], VideoPoet [13], CogVideoX [32], HunyuanVideo [14], and Wan 2.1 [27]; closed-source examples include SORA, Kling, and Gen3.Source: §2.1 · p. 2

2.2 Video Object Segmentation

• Artifact localization can be framed as video object segmentation, but artifacts are irregular and amorphous rather than semantically clear objects.Both tasks require high spatial precision and temporal consistency.Source: §2.2 · p. 3

2.3 Video Artifact Detection

• Existing artifact detectors target specific defects or threshold quality scores, while public datasets do not address artifacts unique to AI-generated content.The related work identifies limited spatial precision in quality-assessment approaches and calls for fine-grained datasets tailored to video generation.Source: §2.3 · p. 3

3 BrokenVideos Dataset

• BrokenVideos is curated for artifact localization through prompt engineering, large-scale generation, and region-level, pixel-accurate annotation.The dataset is designed to support training and evaluation of artifact detection and localization models.Source: §3 · p. 3

3.1 Motivation

• The dataset addresses gaps in region-level annotation, operational artifact definitions, and diversity across scenes, artifact types, and generation models.Its stated design criteria include reproducible definitions, pixel-level masks, broad diversity, and sufficient scale and annotation quality.Source: §3.1 · p. 3

3.2 Dataset Construction

• A hybrid construction strategy combines in-house generated videos with an existing benchmark dataset to balance diversity and annotation quality while limiting computational costs.Broken videos from both sources were systematically identified and annotated with fine-grained masks.Source: §3.2 · p. 3

3.2.1 Artifact Categorization and Annotation

• The annotation taxonomy comprises five artifact categories, and the protocol handles co-occurring types with multi-label, multi-instance annotations.Annotators localize regions at frame level and assign them to categories; overlap across space and time increases annotation workload and complexity.Source: §3.2.1 · pp. 4–5
• SAM2-GUI supports interactive frame-level mask creation, multiple regions per frame, and temporal propagation followed by user inspection and refinement.Masks are saved as frame-level PNGs and can be exported as visualization videos.Source: §3.2.1 · p. 5

3.3 Dataset Statistics

• 3,254 videos and approximately 336,000 manual artifact-region annotations form the dataset’s scale foundation for training and evaluating localization methods.Videos range from 416*624 to 1760*1152 resolution.Source: §3.3 · p. 5
• 80% of videos contain 1–2 artifact regions, 17% contain 3–6, and 3% contain 7–10, covering typical and multi-defect localization scenarios.The annotations support evaluation across both typical and complex cases.Source: §3.3 · p. 5
• 53% of videos feature Human Characters, followed by Fauna and Flora (12%), Robotic Entities (8%), Urban Landscapes (8%), Virtual Environments (7%), Natural Landscapes (7%), and Vehicles (5%).Figure 3 presents the topic distribution.Source: Figure 3 · p. 5, §3.3 · p. 5

4.1 Training and Test Datasets

• Training uses 3,091 broken videos with manually annotated masks and 748 clean videos labeled normal; validation includes 150 broken and 150 normal videos.The authors use clean videos to discourage overfitting to artifacts and encourage recognition of normal videos.Source: §4.1 · p. 5

4.2 Model Fine-Tuning

• SAMWISE, GLUS, and VideoLISA were fine-tuned on BrokenVideos using 8 A800 GPUs, with model-specific schedules and training durations.The passage reports 8 epochs for SAMWISE, 3,000 steps for GLUS, and 2,000 steps for VideoLISA.Source: §4.2 · p. 5

4.3 Performance Evaluation

• GLUS reaches J&F 0.6387 after fine-tuning versus 0.1202 before, with gains across J, F, and J&F on artifact, normal, and mixed videos.The table evaluates localization models on BrokenVideos.Source: Table 2 · p. 6, §4.3 · p. 6
• VideoLISA's J&F score on normal videos rises from 0.2282 to 0.4967 after fine-tuning.The authors report considerable gains on normal videos as well as artifact-heavy scenarios.Source: §4.3 · p. 6

5 Conclusion

• BrokenVideos contains 3,254 AI-generated videos from multiple models, each paired with human-verified pixel-level masks identifying visual artifacts.The authors report that extensive experiments show improved localization after training state-of-the-art video MLLMs on the dataset.Source: §5 · p. 6