Source-linked AI summary
VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes
Yan Ma, Jiadi Su, Zhulin Hu, Ethan Chern, Linhao Zhang, Tiantian Mi, Pengfei Liu
TL;DR
Video pretraining data pipelines are largely closed and difficult to inspect or reuse, while testing recipe hypotheses often requires substantial infrastructure. VIDAFORGE provides an executable, traceable five-stage workflow and evaluates coverage–quality variants in early Wan 2.1 and V-JEPA 2.1 pretraining. Across both objectives, broader-coverage Mixed achieves the highest downstream benchmark scores, while loss-based evaluation favors different recipes.
Problem
Video foundation-model data pipelines rarely expose recipe decisions or controlled evidence about how filtering, deduplication, and captioning affect pretraining.
Method
VIDAFORGE executes five-stage recipes from raw videos to training datasets, preserves intermediate outputs and sample-level decisions, and enables traceable alternative datasets for controlled comparisons.
Results
Across Wan 2.1 and V-JEPA 2.1, broader-coverage Mixed achieves the highest downstream benchmark scores under matched early-pretraining budgets, while loss-based evaluations favor different recipes.
Takeaways & Limitations
VIDAFORGE connects video data-recipe choices with pretraining dynamics and downstream performance, supporting research on individual recipe decisions.
Abstract
from arXiv · showhide
Video foundation models increasingly rely on large-scale pretraining data, yet the end-to-end data pipelines behind them remain largely closed and difficult to inspect or reuse. Researchers seeking to understand how video data recipes affect model pretraining often need to build substantial infrastructure before testing even a focused hypothesis. We present VIDAFORGE, an open research infrastructure that represents a video data recipe as an executable five-stage workflow from raw videos to training datasets. A decision in this workflow can be varied to construct alternative datasets while preserving how every sample was produced. To demon strate this research workflow, we compare data recipes with different coverage and quality in early from-scratch pretraining of Wan 2.1 and V-JEPA 2.1. Across both learning objectives, the broader-coverage recipe achieves the highest downstream benchmark scores, while loss-based evaluation favors different recipes. This study demonstrates how VidaForge connects data-recipe choices to downstream model performance. We further release VIDAFORGE-3M, containing 3.14 million scene level clips totaling 6,475 hours, with fine-grained annotations and curation signals for video data-recipe research.
1 Introduction
Video pretraining data pipelines are largely closed, leaving researchers with limited visibility into recipe decisions and their effects. VIDAFORGE addresses this gap with open, traceable infrastructure and a study of coverage–quality choices.
- Leading video foundation models rarely disclose concrete choices for filtering, deduplication, captioning, or controlled data-recipe ablations.
- Researchers often must build and validate a traceable raw-video-to-training-data pipeline before testing even a focused hypothesis.
- VIDAFORGE executes video data recipes from raw videos to training datasets while preserving stage outputs, parameters, and sample-level decisions.
- The project studies coverage–quality trade-offs across generative and representation pretraining and releases VIDAFORGE-3M with 3.14 million clips and 6,475 hours.
2 VidaForge
VidaForge represents a recipe as five executable transformations from raw videos to target-specific training datasets. It supports controlled recipe variants by preserving intermediate outputs and processing records while scaling processing across heterogeneous video workloads.
- Workflow representation: VidaForge models a recipe R as five stage-level transformations from a fixed raw-video snapshot D0 to a target-specific dataset D5.
- Workflow representation: Changing a processing decision creates an alternative dataset while preserving each sample’s complete production trace, a property called attribution-ready.
- Processing stages: The workflow standardizes inputs, creates scene-level clips, selects clips using quality and duplication signals, annotates content, and packages model-ready datasets.
- Recipe variation: Saved outputs allow researchers to modify segmentation, selection, or annotation and rerun only dependent later stages.
- Scalable execution: Processing steps use CPU, GPU, and concurrent inference workers according to their computational requirements.
3 Coverage or Quality? Studying Video Data Recipes with VidaForge
VidaForge makes coverage–quality recipe comparisons possible through matched early-pretraining runs for Wan 2.1 and V-JEPA 2.1. Broader coverage improves downstream benchmarks, while loss-based evaluations favor different recipes.
- Recipe design: The study compares Mixed, Selected, and Rejected recipes that trade coverage against filtering quality under matched training conditions.Mixed uses broader coverage, Selected higher filtering scores, and Rejected lower filtering scores.
- Recipe design: Selected and Rejected occupy different semantic regions, while Mixed spans their combined coverage.The comparison uses Cosmos-Embed embeddings and t-SNE visualization of the candidate pool.
- Wan 2.1: Rejected achieves the lowest Wan training loss, whereas Selected achieves the lowest final validation loss.Thus, training and validation losses favor different recipes.
- Wan 2.1: Mixed ranks first on Wan VBench Quality, Semantic, and Total, improving Total over Selected by 0.53–1.15 points across matched seeds.The same ordering remains after excluding Dynamic Degree.
- V-JEPA 2.1: Rejected has the lowest V-JEPA loss, but Mixed achieves the highest final SSv2 accuracy under all three seeds.This mirrors the divergence between loss-based and task-level evaluation.
- Overall finding: Across both learning objectives, broader-coverage Mixed achieves the highest downstream benchmark scores despite Selected’s higher filtering scores and fewer duplicates.Pretraining losses do not reliably identify the better training dataset.
4 Conclusion
VidaForge concludes that executable, traceable workflows connect video data-recipe decisions to model-level effects. The paper also situates this capability among existing workflows and releases a large annotated dataset.
- Conclusion: VidaForge provides an executable and traceable path from raw videos to video pretraining experiments.The workflow supports studying individual recipe decisions and their model-level effects.
- Future work: Future work can extend the selection study to segmentation and annotation and track their effects throughout training.The conclusion identifies these as additional recipe decisions for investigation.
- Related work: VidaForge complements existing data-processing and recipe-training workflows by linking recipe execution, sample histories, and model evaluation.The related-work comparison distinguishes workflows by their implemented connections and experimental coverage.
- Infrastructure: Its five-stage pipeline preserves versioned outputs, identities, parameters, and processing records across ingestion, segmentation, and later stages.These records allow a training sample’s processing path to be reconstructed.
C.2 Multi-Level Caption Specification
The annotation prompts define structured inputs and outputs for multi-level captions, camera descriptions, and semantic tags. Versioned schemas and validation support consistent downstream packaging.
- Caption specification: Caption receives ordered frames, timestamps, clip duration, Camera output, and optionally extracted audio, then returns four separately stored strings.Separate fields let packaging recipes choose how much text to provide to a target model.
- Caption specification: The caption v1 schema accepts exactly four strings and records mode, model, schema version, and prompt version.These metadata fields accompany the saved annotation.
- Semantic tags: Tag independently classifies sampled frames using timestamps and duration without Camera or Caption text.Subjects and actions are multi-label, while the remaining tag fields use one dominant label.
- Validation: Validation prevents unknown from co-occurring with another label and none from co-occurring with another action.The validated JSON object is stored with separate tag fields for analysis and sampling.
- Camera annotation: Camera prompts classify motion across dimensions including steadiness, translation, rotation, intrinsic change, and object-centric movement.The guidelines distinguish intentional camera motion from subject motion and unintended shake.
- Camera annotation: Camera labels use unknown when evidence is insufficient and reserve no-* labels for motion explicitly absent.The prompt also requires exactly one JSON object matching the schema.
E.1 Dataset Construction and Overview
VIDAFORGE-3M is an open release produced through the VidaForge pipeline, providing millions of scene-level clips and associated annotations and curation records.
- 3,141,246 scene-level clips totaling 6,475.1 hours are released after processing 800,000 source videos through VidaForge.The dataset is constructed from the video portion of LLaVA-OneVision-2-Data.
- Each released clip includes automatically generated Camera, Caption, and Tag annotations, quality measurements, and duplicate relations.Annotation-complete clips pass all three annotation operations and their schema checks.
- Table 5 summarizes the released dataset statistics over all 3,141,246 clips.
E.2 Large-Scale Data Processing Throughput
VidaForge records execution and dataset-wide distributions across media, quality, duplication, annotations, and high-level clip attributes. These records expose the composition and reusable signals of VIDAFORGE-3M.
- Processing throughput: The VidaForge-3M run records every measured operation from media conversion through annotation, including workload, output, elapsed time, throughput, and hardware.Table 6 reports these execution statistics for each pipeline step.
- Media composition: Clip durations range from 1 to 10 seconds, with mean 7.42 seconds and median 9.70 seconds; 454 × 256 is the most common resolution at 73.63%.The next most common resolutions are 1280 × 720 at 11.64% and 340 × 256 at 8.68%.
- Curation signals: VIDAFORGE-3M retains quality measurements and separate PDQ and Cosmos-Embed duplicate-group assignments for every released clip.These signals support alternative thresholding and deduplication recipes without rerunning duplicate matching.
- Annotations: Every released clip includes Camera, Caption, and Tag annotations, with four caption levels offering progressively richer descriptions.The median caption lengths are 105, 362, 615, and 924 Unicode characters, respectively.
- Annotations: Tag annotations combine single-label fields with multi-label Subjects and Actions, while steadiness and scene dynamics describe camera stability and visible temporal activity.Figure 12 notes that percentages in multi-label panels need not sum to 100%.
F Recipe Construction and Distribution Analysis
The experiment creates comparable training pools from a shared ingestion and segmentation pipeline, then separates clips through selection into Selected and Rejected groups.
- Pool construction: 200,000 standardized source videos produce 716,000 scene-level clips, of which 352,000 are admitted and 364,000 rejected.Optical quality is the primary rejection reason for 45.9% of rejected clips.
- Pool construction: Primary rejection reasons are optical quality 45.9%, motion 20.7%, semantic duplication 19.2%, aesthetics 12.4%, perceptual duplication 1.8%, and visible text 0.07%.
- Training pools: Three 10,000-clip validation sets leave 580,000 training clips, split into 290,000 Selected and 290,000 Rejected clips.Mixed processes the full set once, while Selected and Rejected process their respective halves twice.
F.2 Selection Scores, Duplicates, and Coverage
The study contrasts Mixed, Selected, and Rejected recipes using quality, duplication, and semantic coverage. Selection changes both the measured curation properties and the regions occupied in embedding space.
- Coverage: Under approximately 580k processed clips, Mixed uses approximately 580k different clips, while Selected and Rejected each use approximately 290k different clips for two passes.
- Selection scores: Mean motion is 0.252 for Rejected versus 0.408 for Selected, mean optical score is 0.777 versus 0.994, and mean aesthetic score is 0.191 versus 0.230.These measurements confirm that the component pools differ along the dimensions used by selection.
- Duplicates: PDQ duplicate rates are 2.77%/0.00%/4.63% and semantic duplicate rates are 17.64%/0.53%/23.44% for Mixed/Selected/Rejected.
- Coverage: Selected- and Rejected-dominant regions in the Cosmos-Embed map show that selection changes semantic composition across the candidate pool.The visualization uses reduced embeddings and smoothed local membership dominance.
- Distribution analysis: Figure 13 compares normalized distributions of four selection scores across Mixed, Selected, and Rejected, while Figure 14 reports PDQ and semantic duplicate clips by within-dataset group size.The duplicate-group bars are percentages of the corresponding final training dataset.
G Training and Evaluation Details
Wan 2.1 and V-JEPA 2.1 compare the same three recipe definitions under matched within-family training conditions. Each recipe uses three training seeds, with validation sets isolated by parent video.
- Wan 2.1 and V-JEPA 2.1 use the same Mixed, Selected, and Rejected recipe definitions.
- Both model families use 3 training seeds per recipe, with architecture, optimization, data loading, updates, and evaluation inputs fixed within each family.
- Three 10k-clip validation sets are sampled from the Selected, Mixed, and Rejected pools, with parent videos excluded from all training recipes.
- Each validation set is evaluated at every checkpoint after distributed drop-last processing.
G.2 Wan 2.1 Protocol and Diagnostics
Wan 2.1 training diagnostics measure flow-matching loss across three seeds, while final VBench evaluation uses controlled generation settings. Training and validation losses favor different recipes, whereas Mixed leads VBench aggregates.
- VBench evaluation: Mixed has the highest mean VBench scores with and without Dynamic Degree, indicating its advantage extends beyond the motion metric.VBench uses the final checkpoint at update 5,950 with matched generation settings across runs.
- VBench evaluation: Final VBench aggregates report mean ± std across 3 Wan training seeds, with higher scores better and Total† excluding Dynamic Degree.
- VBench evaluation: Final VBench Total ranks Mixed first under all 3 seeds, while the ordering of Selected and Rejected varies.
G.3 V-JEPA 2.1 Protocol and Diagnostics
V-JEPA 2.1 evaluates masked-prediction training and representation utility across fixed-seed protocols. Loss is non-monotonic because target features evolve, while SSv2 accuracy continues improving and Mixed ultimately ranks highest.
- Protocol: V-JEPA 2.1 trains the 1B encoder from scratch for 2,280 updates with 3 training seeds per recipe.
- Evaluation metric: The reported aggregate uses Dynamic Degree, which measures motion presence.
- Protocol: Reported variability uses the sample standard deviation, with datasets and evaluation protocols fixed across training seeds.
- Loss diagnostics: V-JEPA training and validation curves reach minima near the schedule midpoint and then rise, with Rejected retaining the lowest loss.Loss compares predictions against target features that change through exponential moving averaging.
- Representation utility: SSv2 accuracy continues improving while prediction loss rises, separating the non-monotonic pretraining objective from action-recognition utility.
- Loss diagnostics: Validation-loss median cross-seed standard deviation is 0.0016 for Mixed, versus 0.0064 for Selected and 0.0089 for Rejected.Selected and Rejected therefore show approximately 4× and 5.5× the variability of Mixed.
- Representation utility: Representation utility is measured with a frozen attentive probe on the complete Something-Something V2 benchmark.
- Representation utility: Mixed ranks above Selected and Rejected under every pretraining seed, with mean SSv2 gains of 0.543 and 1.157 percentage points.
G.4 Random-290k Comparison
The Random-290k comparison matches distinct-clip counts and approximately one-pass budgets to separate selection effects from data reuse. Recipe rankings vary by learning objective and training stage, while later full-schedule results favor Mixed for both downstream tasks.
- Matched comparison: Random-290k uses the first approximately 290k clips of Mixed once, matching the distinct-clip scale and one-pass exposure of Selected and Rejected.
- Matched comparison: Intermediate checkpoints compare Wan at step 3149 using 1 seed and V-JEPA at update 1140 using all 3 seeds.
- Intermediate results: For Wan, Random-290k has the highest intermediate VBench Quality and Total, exceeding Selected by 0.847 Total points, while Selected leads Semantic.
- Intermediate results: For V-JEPA, Selected leads intermediate SSv2 accuracy, but Mixed overtakes it later and ranks first at the final checkpoint under all 3 seeds.
- Matched comparison: Mixed denotes Random-290k at the intermediate checkpoints, whereas final comparisons use the complete recipe datasets.
- Interpretation: Recipe rankings depend on the learning objective and training stage, while final comparisons include different amounts of data reuse.
- Recipe construction: The study’s examples cover optical, motion, aesthetic, and visible-text filtering plus PDQ and Cosmos-Embed deduplication.