PaperlayerView exact revision on arXiv

Source-linked AI summary

One Editor, Many Edits: A Unified Training-Free Framework for Diverse Video Editing

Adheesh Sunil Juvekar, Onkar Kishor Susladkar, Kiet A. Nguyen, Muntasir Wahed, Nabeel Bashir, Xiaona Zhou, Tianjiao Yu, Vedant Shah, Ismini Lourentzou

arXiv:2609.04190v1cs.CVcs.AI

TL;DR

Video editing needs a unified way to preserve temporal consistency while supporting diverse instruction-guided and subject-guided edits. EditVid addresses this with a training-free local–global framework built on frozen image MM-DiTs, achieving the strongest training-free performance across the reported evaluations while remaining competitive with training-based editors.

  • Problem

    Existing video editing approaches must handle temporal coherence without sacrificing the semantic control and edit diversity of image editors.

  • Method

    EditVid combines sparse causal memory, correspondence-based post-attention token injection, and adaptive latent blending in frozen image MM-DiTs.

  • Results

    EditVid achieves the strongest performance among training-free methods across FiVE, IVEBench, VLM evaluation, and human preference studies, while remaining competitive with training-based editors.

  • Takeaways & Limitations

    Modern image-editing priors can serve as a strong foundation for temporally consistent instruction-guided and subject-guided video editing without video-specific training.

  • Takeaways & Limitations

    The framework primarily targets appearance-focused spatial editing and may become less stable for explicit motion changes, severe occlusion, or very large appearance changes.

Abstract

from arXiv · show

Video editing spans diverse editing paradigms, yet achieving high-quality instruction-guided and subject-guided editing within a single unified framework remains challenging. We introduce EditVid, a training-free framework combining sparse causal memory for local coherence, correspondence-based post-attention token injection for long-range identity preservation, and soft latent blending for edit locality. The same framework supports instruction-guided and reference-guided edits, including style transfer, attribute modification, object insertion, part-level editing, and subject replacement. On FiVE, EditVid achieves 78.16 FiVE-Acc, compared with 58.95 for the strongest evaluated training-free baseline, while obtaining competitive results on IVEBench. A user study further shows a 51.8\% overall preference for EditVid over 7 competing methods.

1. Introduction

EditVid addresses the challenge of temporally consistent, diverse video editing by combining local and global temporal mechanisms in a training-free framework. It supports instruction-guided and subject-guided editing while preserving identity and edit-relevant content.

  • Motivation and framework: EditVid extends frozen image MM-DiT editors to training-free, temporally consistent instruction-guided and subject-guided video editing.The framework is intended to reuse strong image-editing priors across diverse editing settings.
  • Motivation and framework: Adjacent-frame key–value memory provides short-range coherence, while correspondence-based token transfer preserves long-range appearance and identity.The design separates local temporal stabilization from long-range visual preservation.
  • Motivation and framework: Adaptive latent blending retains instruction-irrelevant source content while allowing requested edits to dominate elsewhere.The preservation weights are derived from discrepancies between source and edited trajectories.
  • Scope and evaluation: Evaluations span FiVE, IVEBench, VLM-based assessment, human studies, temporal-scope tests, and robustness analyses.The authors report the highest FiVE-Acc among training-free methods and competitive IVEBench performance.
  • Scope and evaluation: EditVid supports diverse editing settings, including instruction-guided and subject-guided video editing using MM-DiT-based image editors as priors.The contribution statement emphasizes broad applicability rather than a single editing task.

2. Related Work

Training-free video editing methods expose a trade-off between image-prior editability and video-prior temporal coherence. EditVid targets this gap with sparse temporal memory, correspondence-based token injection, and soft latent blending.

  • Training-free video editing: Training-free methods commonly adapt text-to-image diffusion models by inverting source videos and editing their noisy latent trajectories under temporal constraints.Existing approaches intervene at different stages, including attention-level reuse of neighboring-frame features.
  • Research gap: Image-prior methods offer editability but require explicit temporal coupling, whereas video-prior methods inherit temporal coherence while remaining constrained by their generators.This trade-off leaves strong image-editing priors relatively underexplored for video editing.

3 PRELIMINARIES

EditVid applies training-free correspondence-based token injection to subject-guided video editing, preserving subject appearance without learned subject modules or optimization.

  • EditVid approach: The framework combines sparse causal memory, correspondence-based token injection, and soft latent blending for training-free video editing.Together, these components address temporal coherence, subject preservation, and locality.
  • Subject-guided editing: Existing subject-guided editing methods often require learned subject modules, test-time tuning, feature transfer, or external image-to-video propagation.These requirements distinguish prior training-based and training-free approaches from EditVid.
  • EditVid approach: EditVid preserves subject appearance by injecting correspondence-matched visual tokens from an anchor frame into subsequent frames.The method leverages the position-disentangled behavior of MM-DiT visual tokens without optimization.

3. Preliminaries

The preliminaries describe conditional flow matching and MM-DiT processing for video editing, where visual latent tokens are edited while conditioning tokens provide semantic context. Figure 2 summarizes how sparse temporal cues are injected into a frozen MM-DiT.

  • Conditional Flow Matching: Conditional flow matching defines a conditional probability path between data and Gaussian noise and learns a compatible conditional vector field.The latent state is constructed from paired data and noise samples along the path.
  • EditVid overview: EditVid injects sparse spatio-temporal cues into a frozen MM-DiT through previous-frame memory, anchor-based token injection, and soft latent blending.The diagram emphasizes preserving subject alignment and unedited regions while retaining prompt- or reference-driven edits.
  • Conditional Flow Matching: In practice, a flow-matching latent state z is obtained by interpolating sampled data and Gaussian-noise endpoints through ψ_t.The corresponding target velocity is u = ∂tψ_t.
  • MM-DiT: MM-DiT editing operates on visual tokens derived from noised VAE latent grids, while text or reference-image inputs are encoded as conditioning tokens.For frame i, x(i)(t) denotes visual tokens and c(i) denotes conditioning tokens.
  • MM-DiT: MM-DiT blocks use separate early visual and conditioning streams before later self-attention over their concatenation, with RoPE encoding positional geometry.The output head predicts velocity updates only for visual tokens.

4. Methodology

EditVid edits video in the visual latent stream while keeping text or reference-image conditioning fixed. It combines sparse causal context and previous-frame attention states to introduce bounded temporal coherence.

  • Temporal interactions operate only on visual latent tokens, while conditioning tokens remain unchanged as semantic context.
  • The separation between visual and conditioning streams preserves semantic guidance while temporal information propagates through visual-token attention states.
  • EditVid constructs a compact causal context from the immediately preceding frame’s visual latent tokens.This previous-frame anchor propagates information across frames to promote temporal smoothness.
  • The temporal module augments the current frame’s attention context with previous-frame visual tokens, using their key–value states as temporal context.RoPE keeps consecutive-frame tokens geometrically compatible in attention.

4 METHODOLOGY

EditVid separates local causal propagation from global correspondence-based transfer to maintain temporal consistency across nearby and distant frames. Soft latent blending further preserves unedited background structure while allowing changed regions to follow the edit.

  • Previous-frame attention provides local temporal coherence, but longer-range reuse can become sensitive to RoPE’s relative geometry.The method therefore distinguishes local continuity from long-range spatially displaced interactions.
  • Correspondence-based global token injection transfers anchor-frame information to matched spatial locations across frames.The procedure uses inversion-based correspondence construction, global token injection, and correspondence dropout.
  • Unmatched locations retain their original representations, while aligned injection improves consistency under large motion and occlusion.
  • Causal attention and global injection together stabilize object identity and appearance through sparse, spatially aligned feature propagation.
  • Soft latent blending derives a spatial mask from continuous latent changes and blends edited tokens with the source trajectory at each denoising step.The mask softly anchors regions close to the source while allowing substantially changed regions to follow the target edit.
  • Soft blending preserves background structure in unedited areas without rigidly masking moving subjects.

5. Experiments

EditVid is evaluated across benchmark, curated-set, ablation, qualitative, and human-study settings, showing strong edit correctness, video fidelity, temporal consistency, and robustness among training-free methods. The experiments also examine temporal-memory scope, correspondence filtering, and preservation of instruction-irrelevant content.

  • Benchmark comparisons: 78.16 FiVE-Acc improves on the strongest prior training-free result of 58.95, with especially pronounced gains on multi-choice, open-ended, and intersection edits.These gains indicate more reliable fine-grained object-level edit execution rather than only stronger source similarity.
  • Benchmark comparisons: EditVid achieves the strongest training-free IVEBench results in Total score, Instruction Compliance, and Video Fidelity, while ranking second overall in Total score.On the curated evaluation, it leads five of six criteria and has the second-highest Background Consistency.
  • Human evaluation: EditVid receives the highest preference share across all four user-study criteria in comparisons involving 22 participants, 20 video examples, and seven training-free baselines.The criteria cover edit accuracy, perceptual quality, temporal coherence, and content preservation.
  • Ablation studies: Removing soft latent blending lowers FiVE-Acc from 81.67 to 80.00, while removing spatio-temporal attention produces the same accuracy drop and the lowest MF-S.The ablations support soft blending for preserving instruction-irrelevant content and local temporal context for motion consistency.
  • Ablation studies: Using only the immediately preceding frame reaches 77.50 FiVE-Acc and 91.63 MF-S, whereas longer temporal contexts increase memory or sharply degrade both measures.This motivates bounded local attention for adjacent frames and separate handling of longer-range interactions.
  • Ablation studies: Confidence and cycle-consistency filtering yield the strongest robustness, including the best performance under viewpoint change, occlusion, and non-rigid motion.The results indicate that reliable long-range identity propagation depends on selectively transferring correspondences rather than indiscriminate token copying.
  • Qualitative results: In an elephant color-edit example, EditVid applies the requested blue appearance across frames while preserving structure, motion, and the surrounding scene.The comparison contrasts its balanced instruction adherence and source consistency with baselines that edit conservatively, incompletely, or with greater temporal variation.

6. Conclusion

EditVid is a unified, training-free framework for temporally consistent instruction-guided and subject-guided video editing. It combines local temporal memory, global correspondence-based injection, and adaptive latent blending within frozen image-editing MM-DiTs.

  • EditVid combines sparse causal memory, correspondence-based global token injection, and adaptive latent blending for unified video editing.These components target short-range coherence, long-range identity preservation, and preservation of instruction-irrelevant content.
  • The inference procedure partitions videos into contiguous chunks and uses fixed text or reference conditioning tokens while processing each chunk jointly.Chunk sizes are 36, 24, or 16 frames depending on input resolution.
  • Sparse causal memory uses previous-frame and cached cross-chunk key–value context, while global injection operates in selected vital layers using precomputed correspondences.Temporal memory is updated after each denoising step, and edited visual tokens are decoded and reassembled after the final step.

A.2. Implementation Details

The implementation builds EditVid on FLUX.2-Klein-9B with fixed inference settings and detailed correspondence, temporal-memory, and soft-blending procedures. Evaluation combines FiVE recognition metrics with complementary preservation, alignment, quality, and temporal-consistency measures.

  • Implementation: EditVid uses FLUX.2-Klein-9B with bfloat16, guidance scale 1.0, and four denoising steps.Temporal attention modifications are applied at all denoising steps.
  • Inference: The inference algorithm partitions videos into contiguous chunks, computes fixed conditioning tokens, selects the first frame as anchor, and precomputes correspondence maps.Correspondences are computed at diffusion time 0.25 from the penultimate double-stream block.
  • Inference: Each chunk begins from noisy visual latents and is denoised across steps with temporal memory and correspondence-based global token injection.The procedure uses source conditioning tokens and applies injection through the precomputed anchor-to-frame maps.
  • Soft Latent Blending: Soft latent blending computes per-token differences between edited and inversion-derived source trajectories to preserve unedited regions without rigidly masking moving subjects.The difference map is normalized using low and high quantiles before producing blending weights.
  • Evaluation: FiVE-Acc evaluates object-level editing through FiVE-YN, FiVE-MC, FiVE-U, FiVE-∩, and overall FiVE-Acc, while complementary metrics assess preservation, alignment, quality, and temporal stability.These metrics are used alongside FiVE-Acc rather than as the sole measure of edit correctness.
  • Evaluation: Additional evaluations include CLIP-T, CLIP-Dir, Warp-Err, BRISQUE, and DINO-Ref for text alignment, edit direction, temporal consistency, perceptual quality, and reference-subject fidelity.Reported columns may use FiVE’s scaled conventions, requiring division by the stated factors to recover raw values.

B ADDITIONAL EXPERIMENTS

Additional experiments assess EditVid across curated general and subject-guided editing sets, runtime-quality trade-offs, and a second image-editing backbone. Results show strong training-free performance across alignment, temporal consistency, perceptual quality, and reference preservation.

  • Additional Metrics: EditVid leads training-free methods on six subject-guided metrics and achieves the best general-editing CLIP-Dir, Warp-Err, BRISQUE, and PickScore-Dir.It ranks second on general-editing CLIP-T and PickScore.
  • Additional Metrics: EditVid outperforms the best evaluated training-based methods on 10 of 13 reported metrics and matches the best training-based subject-guided Warp-Err.The reported advantages span edit alignment, temporal consistency, perceptual quality, human-aligned preference, and reference-subject preservation.
  • Performance Profiling: 78.16 FiVE-Acc is achieved at 3.15 seconds per frame, compared with baseline FiVE-Acc values ranging from 26.77 to 58.95.The runtime-quality-memory comparison uses seconds per frame on the x-axis, FiVE-Acc on the y-axis, and bubble size for peak GPU memory.
  • Performance Profiling: Runtime and memory measurements are obtained on a single NVIDIA H200 GPU, and EditVid provides a favorable trade-off among editing accuracy, runtime, and memory consumption.The comparison includes AnyV2V and FlowDirector as relevant baselines.
  • Cross-Backbone Evaluation: The same training-free temporal framework transfers to FLUX.1-Kontext, achieving mask-free VLM scores of 8.125 PF, 8.000 EQ, and 9.042 BC.The FLUX.2-Klein-9B variant remains stronger across all three criteria.

C. Additional Qualitative Results

Qualitative experiments span global style transfer, material and appearance changes, localized edits, and reference-guided replacement. EditVid is presented as preserving motion, pose, identity, and scene structure while applying diverse edits more consistently than competing methods.

  • Qualitative Coverage: The qualitative suite covers global style transfer, object-level material and appearance edits, localized part-level modifications, and subject-guided identity replacement.The examples test strong appearance changes, non-rigid motion, localized attributes, and reference-identity transfer.
  • Comparison: Existing methods trade off preservation against edit strength, whereas EditVid consistently applies requested edits while retaining source motion, pose, and scene structure.This contrast is especially evident for strong transformations and subject-guided replacement.
  • Scope: EditVid primarily targets appearance-focused spatial editing rather than explicit changes to speed, trajectory, or motion direction.Performance may become less stable under severe occlusion or very large appearance changes that weaken correspondences.
  • Global Style Transfer: EditVid applies Ghibli stylization while preserving the duck’s pose, motion, and surrounding scene.Most baselines remain near the source appearance or introduce structural changes; AnyV2V produces stronger stylization but relies on an external image editor.
  • Object-Level Editing: EditVid changes a butterfly into glass and a breakdancer into a holographic breakdancer while preserving shape, motion, and scene context.The examples test object-level material and appearance changes under motion and pose variation.
  • Localized Editing: EditVid localizes wolf-eye and hair-color edits while preserving identity, hairstyle, clothing, pose, and surrounding content.Competing methods either under-apply the requested edit or alter broader aspects of the subject.
  • Subject-Guided Replacement: EditVid transfers lion and polar-bear identity from reference images while preserving source pose, trajectory, motion, and scene interaction.Competing methods more often retain source identity, weaken correspondence, deform geometry, or introduce temporal inconsistency.
Loading 2609.04190v1…