FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editing

Motivation

• Current FFP methods often require costly per-video fine-tuning or auxiliary inputs, limiting computational efficiency and generalization.The paper attributes this reliance to training data that is too short, low-resolution, and insufficiently diverse to teach robust temporal priors.Source: §1 · p. 2

Contributions

• FFP-300K contains 290,441 original/edited video pairs at 720p and 81 frames, built through a two-track synthesis pipeline.The dataset targets limited data length, resolution, and task diversity.Source: §1 · p. 2
• FreeProp introduces AST-RoPE to remap spatio-temporal geometry, anchoring first-frame appearance while matching source-video motion.Its self-distillation strategy uses identity propagation as a regularizer for stable relations between the edited first frame and later frames, preventing semantic drift.Source: §1 · p. 2
• Evaluation on EditVerseBench reports superior visual fidelity and temporal coherence against academic and commercial models.The comparisons include EditVerse and Aleph.Source: §1 · pp. 2–3

Video Editing Models

• Instruction-based editing interprets natural-language prompts, while FFP separates user-driven first-frame modification from automated temporal propagation.Instruction-based methods include inversion-based approaches such as VideoSwap [7] and VideoDirector [30], and inversion-free approaches such as InsV2V [5], LucyEdit, and EditVerse.Source: §2 · p. 3
• Some FFP methods require per-video fine-tuning [19] or auxiliary guidance such as depth maps, optical flow, or predicted masks [15] [16].The paper contrasts these dependencies with propagation conditioned solely on the source video and edited first frame.Source: §2 · p. 3

Video Editing Datasets

• Existing video-editing datasets include synthetic instruction-based data and object-level editing data, while IVEBench [4] mainly targets evaluation.The paper identifies low resolution, short clips, and unclear distinctions between local and global edits as limits for FFP development.Source: §2 · p. 3

3. Scalable FFP Data Construction Pipeline

• The two-track modular pipeline produces semantically aligned 720p pairs through separate local-editing and global-stylization branches.The branches are designed for distinct editing categories and together support instruction-based and FFP video editing.Source: §3 · p. 3

3.1. Local Editing

• Local editing uses vision-language reasoning, segmentation masks, and VACE synthesis for object swapping and removal.Swap tasks favor eroded masks without bounding boxes, while removal favors mask-plus-box conditioning for complete erasure and background reconstruction.Source: §3.1 · pp. 3–4

3.2. Global Stylization

• Global stylization generates a source video from an artistic image, then guides stylization with a style caption, reference image, and depth maps.Depth maps provide structural guidance to preserve geometric structure.Source: §3.2 · p. 4

3.3. Quality Control and Curation

• Removal curation screens nearly 40,000 candidates, manually verifies 14,389 samples, fine-tunes VACE, and regenerates the removal subset.Final semantic verification checks correspondence between edit instructions and visual transformations.Source: §3.3 · pp. 4–5
• 290,441 high-quality source/edited pairs remain after filtering and deduplication, standardized to 720p and 81 frames.The dataset includes 143,913 stylization, 40,000 removal, and 106,528 swap/modification tasks.Source: §3.3 · p. 5

4. Methodology

• The framework is designed to handle temporal consistency without explicit run-time conditions, balancing first-frame appearance propagation with source-motion fidelity.Its two core innovations are adaptive positional encoding and a self-distillation objective.Source: §4 · p. 5

4.1. Preliminary

• For FFP, the model combines the edited first-frame latent, source-video latent, noisy latent, and a first-frame mask as input to the DiT backbone.The model is fine-tuned on FFP-300K with a flow-matching objective.Source: §4.1 · p. 5

4.2. Adaptive Spatio-Temporal RoPE

• AST-RoPE predicts source-conditioned spatial and temporal scaling factors, adapting token positions to the video’s content and separating appearance anchoring from motion modeling.The factors are predicted from the source latent by a lightweight transformer and a two-head MLP.Source: §4.2 · p. 5
• Spatial scaling reduces the perceived positional distance from the first frame, while temporal scaling adjusts inter-frame distance to reflect source-video motion.For rapid motion, a smaller temporal scale reduces perceived distances between frames.Source: §4.2 · p. 6

4.3. Self-Distillation with Identity Propagation

• An identity propagation task reconstructs the target video from itself, providing a teacher representation that the standard FFP task is trained to match.The authors describe this teacher as having perfect knowledge of the source video’s specific motion.Source: §4.3 · p. 6
• Motion-relation and first-frame-consistency losses align the student’s temporal relationships and edit trajectory with the identity task, aiming to prevent drift or fading.The motion loss compares inter-frame relationships; the MMD loss aligns the evolution of token-wise relationships to the first frame.Source: §4.3 · p. 6

5.2. Quantitative Comparison

• Both variants achieve the best performance across automated EditVerseBench metrics; Ours-81f leads temporal consistency and video-level alignment, while Ours-33f leads perceptual quality and semantic correctness.Ours-81f scores 0.991 CLIP, 0.991 DINO, and 25.925 video-level text alignment; Ours-33f scores 20.419 Pick Score and 7.631 VLM Score.Source: Table 1 · p. 7, §5.2 · p. 7

5.1. Experiment Setup

• On 125 stable-structure EditVerseBench videos, evaluation uses six metrics and extends VLM assessment to 10 sampled frames; two model variants use 81-frame and 33-frame training videos.The setup replaces GPT-4o with Qwen2.5-VL-72B-Instruct for evaluation reproducibility.Source: §5.1 · pp. 6–7

5.3. Qualitative Comparison

• Qualitative comparisons show longer outputs and better general quality from our method, while competitors exhibit unsuitable first-frame edits, content loss, flickering, or mosaic artifacts.The comparison covers four representative editing tasks; gray placeholders indicate methods unable to generate videos that long.Source: §5.3 · p. 7, Figure 4 · p. 8

5.4. User Study

• In a 15-participant study, our method receives the highest mean scores for editing accuracy, motion accuracy, and video quality.Each participant assessed eight randomly selected EditVerseBench videos.Source: §5.4 · p. 7, Table 2 · p. 8

5.5. Ablation Study

• The dataset-trained baseline already performs strongly, while adding AST-RoPE and self-distillation further improves visual quality and text alignment.The ablation compares the unmodified baseline, AST-RoPE alone, and the full combination.Source: §5.5 · p. 8, Table 3 · p. 8

• The paper identifies a foundational data gap as the limitation behind complicated runtime guidance and limited generalization in FFP.Source: §6 · p. 8
• Its response combines FFP-300K with a model using AST-RoPE and self-distillation to strengthen first-frame reference and preserve source motion.Source: §6 · p. 8
• The paper reports that this dual approach achieves state-of-the-art fidelity and temporal coherence.Source: §6 · p. 8

7.1. Dataset Construction

• Qwen2.5-VL-72B-Instruct produces prompts for both local editing and global stylization in the two-track construction pipeline.Source: §7.1 · p. 11

7.1.1. Local Editing

• Local-editing instructions distinguish swaps, which replace an object, from modifications, which change attributes while retaining its identity.The prompts also identify editable objects and describe stable scene context for edits.Source: §7.1.1 · p. 11

7.1.2. Global Stylization

• Global stylization prompts describe the source scene and reference-image style while constraining added motion and unnatural shifts in color or lighting.Source: §7.1.2 · pp. 11–12

7.2. Dataset Analysis

• The local-editing object distribution is long-tailed, spanning prominent people and handheld items as well as furniture, animals, nature, vehicles, buildings, and everyday objects.Source: §7.2 · p. 12
• Indoor Activities (12,710 videos, 29.2%), Performance & Entertainment (8,225, 19.0%), and Urban Scenes (6,904, 15.9%) dominate scene content, while remaining categories typically each account for below 8%.Source: §7.2 · p. 12

7.3. Visualization of FFP-300K

• Examples from both tracks maintain spatial coherence and temporal consistency; local editing covers object-level remove and swap edits, while global stylization transfers reference-image appearance across the source video.Global stylization pairs each source video with multiple reference images.Source: §7.3 · p. 13

8.1. Attention Head Classification Heuristic

• Each attention-map grid represents attention from source latent frame i to target latent frame j, and its density counts scores exceeding ε = 10^-6.The threshold filters negligible positive scores produced by softmax.Source: §8.1 · p. 13
• A head is temporal when its maximum non-diagonal density exceeds its minimum diagonal density; otherwise, it is spatial.This tests whether the strongest cross-frame signal exceeds the weakest within-frame signal.Source: §8.1 · p. 13
• Final head labels use majority voting over classifications from 10 diverse validation videos rather than relying on one sample.The authors use this aggregation because head behavior can depend on video content.Source: §8.1 · p. 13

9.1. Experiments on UNICBench

• UNICBench contains 128 FFP-suitable videos covering add, delete, change, and stylization tasks.The test set was filtered using the same principle as EditVerseBench, and evaluation uses the same metrics.Source: §9.1 · p. 13
• Our method achieves the best performance on all UNICBench metrics, with qualitative comparisons indicating more accurate editing and better visual quality.Table 4 compares three types of video-editing methods; Figure 12 shows the qualitative comparison.Source: Table 4 · p. 14, §9.1 · p. 14

9.2. More Results on EditVerseBench

• Supplementary EditVerseBench visualizations show add, remove, change, and stylization results against source videos, with separate landscape and portrait examples.Figures 9–11 provide the task and orientation-specific visualizations.Source: §9.2 · p. 14, Figure 9 · p. 16, Figure 10 · p. 17

9.3. More Results on UNICBench

• Supplementary UNICBench visuals compare source videos, our FFP results, and UNIC; other cases juxtapose UNIC instruction-based outputs with our FFP results.Figure 13 presents the FFP-based comparison.Source: §9.3 · p. 14, Figure 13 · p. 19