PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoCoF: Unified Video Editing with Temporal Reasoner

Xiangpeng Yang, Ji Xie, Yiyuan Yang, Yue Ma, Yan Huang, Min Xu, Qiang Wu

arXiv:2512.07469v2cs.CV

TL;DR

Existing video editing approaches trade mask-dependent precision for unified methods that lack explicit spatial cues. VideoCoF predicts an edit-region latent before generating the target video and aligns temporal positions. On VideoCoF-Bench, it reports improvements over ICVE using 50k video pairs; multi-shot extrapolation shows slight degradation at 512 frames.

  • Problem

    Expert editing models rely on external masks and per-task overhead, while unified temporal in-context models lack explicit spatial cues and can have weak instruction-to-region accuracy.

    Source: §1 · p. 2

  • Method

    VideoCoF predicts reasoning tokens representing an edit region before generating target video tokens, and uses RoPE alignment that accounts for the reasoning latent.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    With 50k video pairs, VideoCoF improves instruction-following score by +15.14% and success ratio by +18.6% over ICVE, which uses ∼1M pretraining videos plus 150k for finetuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results show that VideoCoF achieves state-of-the-art quantitative and qualitative performance on VideoCoF-Bench with 50k video pairs.

    Source: §1 · p. 2

    In multi-shot 16× extrapolation, performance remains stable for the first 500 frames but shows slight degradation at 512 frames.

    Source: §7 · p. 12

Abstract

from arXiv · show

Existing video editing methods face a critical trade-off: expert models offer precision but rely on task-specific priors like masks, hindering unification; conversely, unified temporal in-context learning models are mask-free but lack explicit spatial cues, leading to weak instruction-to-region mapping and imprecise localization. To resolve this conflict, we propose VideoCoF, a novel Chain-of-Frames approach inspired by Chain-of-Thought reasoning. VideoCoF enforces a ``see, reason, then edit" procedure by compelling the video diffusion model to first predict reasoning tokens (edit-region latents) before generating the target video tokens. This explicit reasoning step removes the need for user-provided masks while achieving precise instruction-to-region alignment and fine-grained video editing. Furthermore, we introduce a RoPE alignment strategy that leverages these reasoning tokens to ensure motion alignment and enable length extrapolation beyond the training duration. We demonstrate that with a minimal data cost of only 50k video pairs, VideoCoF achieves state-of-the-art performance on VideoCoF-Bench, validating the efficiency and effectiveness of our approach. Our code, weight, data are available at https://github.com/knightyxp/VideoCoF.

1. Introduction

VideoCoF addresses the precision–unification trade-off by predicting an edit region before generating the edited video, then aligns temporal positions to support motion alignment and length extrapolation. It reports strong VideoCoF-Bench results using 50k video pairs and describes its framework, reasoning, training, and data curation.

1. Introduction

  • Expert models use external masks for precise localized edits but require additional inputs and per-task overhead, while unified temporal in-context models are mask-free but often less accurate without explicit spatial cues.The paper frames this as a trade-off between precision and unification.Source: §1 · p. 2
  • With 50k video pairs, VideoCoF improves instruction-following score by +15.14% and success ratio by +18.6% over ICVE, which uses ∼1M pretraining videos plus 150k for finetuning.The authors report state-of-the-art quantitative and qualitative performance on VideoCoF-Bench.Source: §1 · p. 2

2. Related Work

  • Related work includes training-based editing, mask-dependent or temporal in-context approaches, and visual chain-of-thought; the paper investigates chain-of-frames reasoning for unified editing.The authors note that prior in-context approaches can require masks or lack accuracy and length extrapolation, while visual reasoning for unified video editing remained unexplored.Source: §2 · pp. 2–3

3.1. VideoCoF Framework

  • VideoCoF uses a VideoDiT to process source, reasoning, and target frames in one temporally concatenated sequence, with self-attention and language control through cross-attention.Its framework uses temporal RoPE for source-to-target alignment and reasoning-token RoPE for explicit spatial guidance.Source: §3.1 · p. 3

3.2. Chain of Frames

  • Chain of Frames predicts a grounding region between the original and edited video, making editing a “seeing, reasoning, then editing” process.The method uses a gray-highlighted region as the reasoning-frame target because video diffusion models can be insensitive to black or white grounding masks.Source: §3.2 · pp. 3–4

3.3. RoPE Design for Length Extrapolation

  • Naive sequential temporal indices hinder length extrapolation, while a naive reset creates index collisions; assigning source and target indices [1, F] and reasoning index 0 isolates the reasoning token.The stated design prevents artifact leakage while maintaining length generalization.Source: §3.3 · p. 4

3.4. Training and Inference Paradigm

  • Training noises and supervises the reasoning and target frames while keeping source latents clean; inference initializes the reasoning-editing block from noise and holds source latents fixed.The model predicts a velocity field and minimizes mean squared error between predicted and true velocities.Source: §3.4 · pp. 4–5

3.5. Video Data Curation

  • The data pipeline targets complex instance-level video edits by identifying multi-object scenes, segmenting instances, generating edit triplets, and filtering for quality.It distills a 50k-video subset from Se˜norita 2M using aesthetic-quality and editing-fidelity scores.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate VideoCoF on a benchmark designed for instruction-guided, instance-level editing and compare its performance with baselines and design ablations. Results report stronger benchmark performance with 50k video pairs, alongside tests of reasoning frames, RoPE alignment, and reasoning-frame format.

4.1. Implementation Details.

  • Training uses 50k curated 33-frame video samples, with a four-frame reasoning clip; inference takes about 10 seconds for 33 frames on one H100 GPU.The model is trained for 8k iterations on 16 H100 GPUs.Source: §4.1 · pp. 5–6

4.3. Comparison on VideoCoF-Bench

  • VideoCoF achieves the best instruction-following and success ratio across categories using 50k pairs, versus ICVE’s 1M pretraining videos plus 150k fine-tuning examples.The comparison covers five baselines on VideoCoF-Bench.Source: Table 1 · p. 6, §4.3 · p. 7
  • In Figure 6 examples, VideoCoF edits specified instances more faithfully across removal, addition, swap, and local-style tasks.Examples include removing the right instance, placing an added girl inside a washing machine, and editing the largest cup among similar objects.Source: Figure 6 · p. 6, §4.3 · p. 7

4.2. VideoCoF-Bench and Experimental Setting

  • VideoCoF-Bench contains 200 videos across four editing tasks, with 50 samples per task and half of each task’s samples involving instance-level cases.The tasks are object removal, object addition, object swap, and local style transfer.Source: §4.2 · p. 6

4.4. Ablation Study

  • The ablation reports 10.65% higher instruction-following, 5.46% higher success ratio, and 4.16% higher CLIP-T for VideoCoF than the first-column baseline.The comparison is between the first ([0, 2F −1]) and third (VideoCoF) columns of Table 2.Source: Table 2 · p. 7, §4.4 · p. 7
  • In Table 2, the third column shows a 3.4% relative increase in preservation score over the first column; improved DINO also confirms better preservation of the original video’s spatio-temporal structure during editing.Source: §4.4 · pp. 7–8
  • Fixing the reasoning latent at index 0 while source and edited frames range from 1 to F avoids the motion misalignment and first-frame interference of the naive 0-index design.The paper reports artifacts when the reasoning latent and first editing frame both use index 0.Source: §4.4 · p. 8
  • A progressive gray reasoning mask performs best among the tested formats, while the black mask fails deletion and the red mask deletes content on the wrong side.The tested alternatives include a black mask, a 50% transparent red highlight, a gray mask, and progressive gray transparency.Source: Table 3 · p. 8, §4.4 · p. 8

5. Conclusion

The conclusion presents VideoCoF as a unified video-editing model that reasons over edit regions before generating edits. It reports state-of-the-art performance with 50k video pairs and RoPE-enabled extrapolation up to 16× the training length.

5. Conclusion

  • VideoCoF uses a Chain-of-Frames process in which the model predicts the editing region before generating the edit.The conclusion frames this as a “see, reason, then edit” procedure for unified video editing.Source: §5 · p. 8
  • RoPE alignment that accounts for the reasoning latent enables inference at up to 16× the training length.Source: §5 · p. 8
  • The paper reports state-of-the-art performance using only 50k video pairs.The conclusion presents this as evidence of the efficiency and effectiveness of its temporal reasoning design.Source: §5 · p. 8

Supplementary Material

The supplementary material lists additional design discussion, comparisons, ablations, implementation details, metrics, and future directions.

Supplementary Material

6. Discussion on RoPE Design

The discussion distinguishes VideoCoF’s video-to-video length-extrapolation problem from source-target alignment and text-to-video extrapolation. It attributes the distinction to temporal-index placement and reports that applying UltraViCo to the baseline still fails.

6. Discussion on RoPE Design

  • VideoCoF addresses length extrapolation in video-to-video editing, whereas UNIC focuses on source-target alignment.Source: §6 · p. 12
  • VideoCoF pre-positions reasoning frames so their indices do not overlap the target video; UNIC post-positions them, creating unavoidable index collisions as videos grow.Source: §6 · p. 12
  • The paper distinguishes its source-target alignment and temporal-index-collision limits from the motion or content repetition addressed by RIFLEx and UltraViCo, and reports that UltraViCo still fails on its baseline.Source: §6 · p. 12, Figure 10 · p. 12

7. Editing Length Upper Bound

Extrapolation capacity differs by setting: single-shot testing supports 16×, while multi-shot remains stable through 500 frames and shows slight degradation at 512 frames.

  • Single-shot testing supports 16× extrapolation; multi-shot remains stable through 500 frames but shows slight degradation at 512 frames.Source: §7 · p. 12

8. Full Comparison

VideoCoF leads on instruction following and success ratio across four editing tasks, while its preservation and quality scores are slightly below ICVE's.

  • VideoCoF achieves the highest instruction-following and success-ratio scores across Object Removal, Object Addition, Object Swap, and Local Style Transfer.Source: §8 · p. 12
  • VideoCoF's Preservation and Quality scores are slightly lower than ICVE's, which uses larger-scale pretraining and a 150K SFT dataset versus VideoCoF's 50k.Source: §8 · p. 12
  • VideoCoF achieves the highest CLIP-T scores across all four tasks.Source: §8 · p. 12

9. TGVE+ and V2VBench

VideoCoF achieves the best performance on both TGVE+ and V2VBench; TGVE+ is used in place of TGVE because TGVE has only 304 samples.

  • Because TGVE has only 304 samples, the evaluation uses the TGVE+ subset from EVE, which contains 1,417 samples across seven instruction-editing tasks.Source: §9 · p. 12
  • VideoCoF achieves the best performance on both TGVE+ and V2VBench.Source: Table 4 · p. 12, Table 5 · p. 12, §9 · p. 12

10. More Ablation Studies

The ablations examine reasoning-frame length and the temporal triptych prompt, which structures video content as original, grounded, then edited scenes.

10.1. Ablation on Reasoning Frames

  • Four reasoning frames achieve the best performance; frames 1–4 map to one latent frame, while five frames introduce a second latent and temporal complexity.Source: Table 6 · p. 13, §10.1 · p. 13

10.2. Ablation on Temporal Triptych Prompt

  • The temporal triptych prompt significantly enhances instruction following and overall success rates, and is presented as a practically zero-cost alternative to ICVE's costly instruction-mode pretraining.Source: Table 7 · p. 13, §10.2 · p. 13
  • The temporal triptych prompt describes video evolution as the original scene, then the grounded instruction, and finally the scene after the edit instruction.Source: §10.2 · p. 13, Figure 12 · p. 13

11. Implementation Details

This section describes a 50k-pair training dataset spanning four editing tasks and a benchmark curated without overlap with the training domain.

11.1. Training Dataset

  • The training dataset contains 50k samples balanced across object addition, removal, swapping, and local stylization.It combines filtered open-source and high-quality synthetic data.Source: §11.1 · p. 13, Table 8 · p. 14
  • Addition and removal contribute 25k samples, while swapping and local style contribute another 25k.The removal subset includes 5,000 multi-instance samples; swap and style data use GPT-4o prompts and Grounding DINO masks.Source: §11.1 · pp. 13–14

11.2. VideoCoF-Bench

  • VideoCoF-Bench combines 50 manually curated Pexels samples per task with EditVerse and UNIC-Bench samples, removing UNIC identity images for text-driven evaluation.The benchmark has no overlap with the training domain and includes varied scenes and complex instructions.Source: §11.2 · p. 14

12. Metrics

The section reports semantic and perceptual evaluation measures and outlines directions for scaling VideoCoF and extending it beyond local video editing. VideoCoF is reported to lead on instruction following and success ratio, while broader tasks and larger datasets remain future directions.

12. Metrics

  • GPT-4o judges three sampled frames per video pair for instruction following, visual quality, preservation, and binary success.Success requires target identification, accurate modification, and preservation of other instances.Source: §12 · p. 14
  • VideoCoF achieves superior performance across the reported metrics, including instruction following and success ratio on VideoCoF-Bench.The comparison covers four video-editing tasks and includes baselines trained with substantially more data.Source: §12 · p. 14, Table 9 · p. 16
  • CLIP-T measures instruction alignment, while CLIP-F and DINO measure adjacent-frame consistency at semantic and structural levels.The section presents these as quantitative measures of visual alignment and temporal consistency.Source: §12 · pp. 14–15

13. Future Directions

  • Future work includes scaling training to 200k or millions of samples, joint image-video editing, and extending the reasoning framework to global editing and ID-driven tasks.The paper currently focuses on local editing; proposed extensions include full-frame masks for global editing and identity images as reasoning frames.Source: §13 · p. 15, §12 · p. 15

Related papers

Loading 2512.07469v2…