Region-Constraint In-Context Generation for Instructional Video Editing

Zhongwei Zhang; Fuchen Long; Wei Li; Zhaofan Qiu; Wu Liu; Ting Yao; Tao Mei

Source-linked AI summary

Region-Constraint In-Context Generation for Instructional Video Editing

Zhongwei Zhang, Fuchen Long, Wei Li, Zhaofan Qiu, Wu Liu, Ting Yao, Tao Mei

arXiv:2512.17650v1Dec 19, 2025cs.CVcs.MM

TL;DR

Instruction-based video editing lacks a practical way to localize edits from text alone and to prevent interference between edited and non-edited regions. ReCo addresses this with joint source-target in-context denoising, region-aware latent and attention regularization, and ReCo-Data; it reports consistent gains across four editing tasks.

Problem
Instruction-based video editing remains insufficiently explored when driven solely by textual instructions, particularly for localizing editing regions and avoiding interference between editing and non-editing areas.
Method
ReCo width-wise concatenates source and target videos for joint denoising and applies latent-space and attention-space regional regularization during diffusion training.
Results
ReCo consistently outperforms existing baselines on total score S across four video editing tasks, including scores of 8.23 on Add and 8.74 on Replace.
Takeaways & Limitations
ReCo provides a region-constrained in-context generation paradigm that improves editing accuracy and quality while preserving non-edited content.
Takeaways & Limitations
The latent-space regional constraint is presented as a hypothesis that editing-region discrepancies should be high while non-editing-region discrepancies should be low.

Abstract

from arXiv · show

The In-context generation paradigm recently has demonstrated strong power in instructional image editing with both data efficiency and synthesis quality. Nevertheless, shaping such in-context learning for instruction-based video editing is not trivial. Without specifying editing regions, the results can suffer from the problem of inaccurate editing regions and the token interference between editing and non-editing areas during denoising. To address these, we present ReCo, a new instructional video editing paradigm that novelly delves into constraint modeling between editing and non-editing regions during in-context generation. Technically, ReCo width-wise concatenates source and target video for joint denoising. To calibrate video diffusion learning, ReCo capitalizes on two regularization terms, i.e., latent and attention regularization, conducting on one-step backward denoised latents and attention maps, respectively. The former increases the latent discrepancy of the editing region between source and target videos while reducing that of non-editing areas, emphasizing the modification on editing area and alleviating outside unexpected content generation. The latter suppresses the attention of tokens in the editing region to the tokens in counterpart of the source video, thereby mitigating their interference during novel object generation in target video. Furthermore, we propose a large-scale, high-quality video editing dataset, i.e., ReCo-Data, comprising 500K instruction-video pairs to benefit model training. Extensive experiments conducted on four major instruction-based video editing tasks demonstrate the superiority of our proposal.

1. Introduction

ReCo frames instruction-based video editing as region-constrained in-context generation to address localization and source-content interference. It jointly denoises source-target videos and applies latent and attention constraints, supported by a 500K-pair dataset and evaluations across four tasks.

1. Introduction: ReCo introduces a region-constraint in-context generation framework for instruction-based video editing.The framework targets accurate editing-region localization and reduced interference between editing and non-editing areas.
1. Introduction: ReCo jointly denoises width-concatenated source and target videos while using the source video as an explicit condition.The source and target videos are concatenated along the left-right dimension and processed together during editing generation.
1. Introduction: Latent regularization increases source-target discrepancy in editing regions and decreases it in non-editing areas.This constraint emphasizes intended modifications while reducing unexpected generation outside the edited region.
1. Introduction: Attention regularization reduces attention from target editing regions to corresponding source-region tokens, mitigating interference during novel-object generation.The regularization is applied to attention maps of DiT blocks.
1. Introduction: ReCo-Data contains 500K instruction-video pairs, and experiments evaluate ReCo on four major instruction-based video editing tasks.The dataset is designed to facilitate training and community research, while experiments assess editing accuracy and quality.

2. Related Work

Prior work spans image-editing data pipelines, training-free and training-based video editing, and methods for temporal coherence. ReCo builds on in-context image editing while adding region-wise constraints to address video-editing localization and token interference.

Instruction-based Image Editing: Instruction-based image editing established data-construction recipes that later work refined to train more capable editors.The passage identifies InstructPix2Pix as a representative approach and describes subsequent extensions of its data pipeline.
Instruction-based Video Editing: Early instruction-based video editors adapted text-to-image diffusion models frame by frame, but lacking temporal modeling caused temporal inconsistency.Later methods introduced token-merging or similarity constraints to improve temporal coherence.
Instruction-based Video Editing: Training-based video editing has been limited by scarcity of large-scale, high-quality paired training data.Prior responses included one-shot tuning, propagation-based editing, and direct training on source-target videos with instructions.
Instruction-based Video Editing: ReCo applies in-context generation to video editing while specifically modeling region-wise constraints for localization and reduced token interference.Its stated novelty extends the in-context recipe beyond image editing to relationships between editing and non-editing regions.

3. Our Approach

ReCo reformulates instructional video editing as joint source-target in-context denoising and constrains editing and non-editing regions in latent and attention spaces. It also introduces ReCo-Data, a 500K-pair dataset designed to support high-quality instructional video editing.

In-context video generation: ReCo width-wise concatenates source and target video latents for joint video denoising in an in-context generation framework.The framework jointly learns source-video reconstruction and target-video editing, with a video condition branch for calibrated denoising.
Regional Constraint in Latent Space: The latent regional constraint increases source-target discrepancy in editing regions while reducing it in non-editing regions.It uses one-step backward denoised latents, partitions them into source and target parts, and applies a binary editing mask.
Regional Constraint in Attention Space: The attention constraint reduces target-editing queries’ reliance on corresponding source-editing keys and on the broader source video.It instead encourages attention toward contextually relevant target regions to mitigate token interference and preserve coherent generated content.
Overall objective: ReCo combines the basic in-context flow-matching loss with latent-space and attention-space regional constraints.The overall objective weights the two constraints using trade-off parameters λ1 and λ2.

4. Experiments

ReCo is evaluated on four instruction-based video editing tasks using benchmark scores, visual comparisons, and regional-constraint ablations. It consistently outperforms baselines, while dataset-quality analysis and ablations examine training data usability and the roles of latent and attention constraints.

Experimental Settings: ReCo-Data contains 500K high-quality instruction-video pairs spanning object addition, removal, replacement, and global stylization.Its reported high-quality-sample ratio is 91.6%, compared with 17.9%–29.2% for three existing datasets.
Experimental Settings: The benchmark evaluates edit accuracy, video naturalness, and video quality through sub-dimension scores and aggregated metrics SEA, SV N, SV Q, and S.It contains 480 video-instruction pairs, with 120 pairs for each of four editing tasks.
Comparisons with State-of-the-Art Methods: ReCo consistently outperforms existing baselines across four video editing tasks on the overall score S.On local editing, ReCo scores 8.23 on Add and 8.74 on Replace, exceeding Ditto by 0.67 and Lucy-Edit by 2.02, respectively.
Comparisons with State-of-the-Art Methods: ReCo produces more accurate edits, higher video quality, and better background consistency than competing methods in visual comparisons.Ditto can re-render whole videos or alter non-editing regions, while Lucy-Edit and InsViE show instruction-following or artifact problems.
Ablation Study on Regional Constraint: Removing latent regional constraints causes a dramatic SEA drop, indicating substantially worse editing accuracy and incorrect changes outside the intended region.In one replacement example, ReCoLC− removes a nearby dog while replacing the boy.
Ablation Study on Regional Constraint: Removing attention regularization lowers SV N and produces unnatural object scale, whereas ReCo generates objects with more natural size and coherence.The ablation example shows ReCoAC− generating an oversized parrot relative to its environment.

5. Conclusions

ReCo integrates region-constraint modeling into in-context instructional video editing through joint source-target denoising and latent and attention regularization. ReCo-Data provides 500K instruction-video pairs, and experiments cover four editing tasks.

ReCo jointly denoises width-concatenated source-target videos while regularizing one-step denoised latents and attention maps to model editing versus non-editing regions.Latent regularization increases differences in editing regions and decreases them elsewhere; attention regularization suppresses source-region token interference during novel object generation.
ReCo-Data contains 500K instruction-video pairs spanning a wide range of editing tasks.
Extensive experiments evaluate ReCo across four instruction-based video editing tasks.

1. Construction Pipeline of ReCo-Data

ReCo-Data addresses the scarcity of large-scale, high-quality paired video-editing data through a six-stage construction pipeline. The process combines filtering, segmentation, instruction and condition generation, synthesis, augmentation, and re-captioning to produce approximately 800K video pairs before final filtering.

Data Construction Pipeline: The pipeline filters raw videos, segments objects, generates instructions, constructs conditions, synthesizes videos, and filters and re-captions the results.These six stages are shown in Figure 7.
Object Segmentation: Object masks are produced by extracting entity nouns, detecting bounding boxes, applying NMS, and generating mask sequences with SAM 2.
Instruction Generation: Local editing prompts use captions and highlighted key frames, whereas stylization prompts generate diverse styles and target video descriptions.
Condition Pair Construction: Task-specific condition strategies use ObjectClear for removal, source-target swapping for addition, VACE for replacement, and first-frame style transfer plus depth maps for stylization.
Synthesis and Filtering: The process creates approximately 800K video pairs, then uses VLLM filtering and re-captioning to construct the final ReCo-Data dataset.Data augmentation includes reversible replacement and cross-task reuse of shared clean backgrounds; synthesis and captioning required approximately 76,800 GPU hours and $13,600, respectively.

2. VLLM-based Evaluation Benchmark

The VLLM-based benchmark evaluates video editing beyond traditional generation metrics, using 480 test pairs across four tasks and nine sub-dimensions. It aggregates ratings into edit accuracy, video naturalness, video quality, and an overall score.

Testing Data: The benchmark evaluates 480 video-instruction pairs, evenly split across object addition, removal, replacement, and video stylization.Stylization evaluation uses 10 source videos with 12 styles each.
Evaluation Metrics: VLLM rates editing results across nine sub-dimensions grouped into edit accuracy, video naturalness, and video quality.The dimensions cover instruction alignment, realism and coherence, and visual quality over time.
Evaluation Metrics: Edit accuracy measures semantic accuracy, scope precision, and preservation of non-edited content.
Evaluation Metrics: Video naturalness measures appearance, scale, and motion naturalness, while video quality measures visual fidelity, temporal stability, and edit stability.
Score Aggregation: Each sub-dimension receives a 0–10 VLLM rating; category scores use geometric means and the overall score is their arithmetic mean.

3. Implementation of Baselines and ReCo

ReCo is benchmarked against recent video-editing methods selected according to their ability to support the relevant tasks. The model is built on Wan and trained with AdamW using a two-stage learning-rate schedule and latent and attention constraints.

Baseline Settings: For object addition, replacement, and stylization, ReCo is compared with InsViE, Lucy-Edit, and Ditto under task-specific baseline settings.InsViE inputs are adapted to its 49-frame, 480 × 720 resolution requirement.
Implementation Details: ReCo uses the Wan architecture and AdamW, with learning rates of 1 × 10^-4 initially and 2 × 10^-5 during fine-tuning.
Implementation Details: The latent and attention constraint weights typically fall within [−1, 1], while the flow-matching MSE is approximately 0.03.

4. Generalization Ability of ReCo

ReCo demonstrates generalization beyond conventional edits by handling abstract and creative instructional video-editing tasks. Its evaluation considers edit accuracy, video naturalness, and video quality.

4. Generalization Ability of ReCo: ReCo generalizes to abstract and creative editing tasks, including halos, confetti, idea lightbulbs, and smoke effects.The authors attribute this ability to leveraging priors from a pretrained video diffusion model.
4. Generalization Ability of ReCo: The reported examples include synthesizing a halo, generating cascading confetti, placing an idea lightbulb, and creating smoke from a computer.
4. Generalization Ability of ReCo: Video-editing assessment evaluates edit accuracy, video naturalness, and video quality across four instructional editing tasks.