PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing

Xiangpeng Yang, Linchao Zhu, Hehe Fan, Yi Yang

arXiv:2502.17258v1cs.CV

TL;DR

Multi-grained video editing across class, instance, and part levels remains challenging because text-to-region control is semantically misaligned and diffusion features couple across regions. VideoGrain addresses this with zero-shot space-time cross- and self-attention modulation, and reports state-of-the-art results on benchmarks and real-world videos without parameter tuning.

  • Problem

    Multi-grained video editing spans class-level, instance-level, and part-level changes, but existing methods mix distinct instances and suffer semantic misalignment and feature coupling.

  • Method

    VideoGrain modulates space-time cross- and self-attention to focus local prompts on corresponding regions and separate features between regions.

  • Results

    VideoGrain enables class-level, instance-level, and part-level editing and achieves state-of-the-art qualitative and quantitative results without parameter tuning.

  • Takeaways & Limitations

    The approach provides a zero-shot framework for fine-grained video editing across multiple editing granularities.

  • Takeaways & Limitations

    Generation quality remains limited by the base model, and the T2I-based method struggles with large shape deformations and significant appearance changes.

Abstract

from arXiv · show

Recent advancements in diffusion models have significantly improved video generation and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include semantic misalignment of text-to-region control and feature coupling within the diffusion model. To address these difficulties, we present VideoGrain, a zero-shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region while minimizing interactions with irrelevant areas in cross-attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive experiments demonstrate our method achieves state-of-the-art performance in real-world scenarios. Our code, data, and demos are available at https://knightyxp.github.io/VideoGrain_project_page/

1 INTRODUCTION

VideoGrain introduces multi-grained video editing across class, instance, and part levels, addressing semantic misalignment and feature coupling through unified space-time attention modulation.

  • Task Definition: Multi-grained video editing covers class-level, instance-level, and part-level modifications to objects, distinct instances, and object parts.Class-level editing modifies objects within one class; instance-level editing assigns different edits to different instances; part-level editing adds objects or changes attributes.
  • Motivation: Existing methods often mix features across instances because diffusion models treat distinct instances as same-class segments during editing.This instance agnosticism causes artifacts and limits multi-grained edits.
  • Method: VideoGrain proposes ST-Layout Attn, which jointly modulates space-time cross- and self-attention for text-to-region control and feature separation.Cross-attention controls textual features over regions, while self-attention connects frame tokens across time.
  • Method: Cross-attention amplifies each local prompt within its spatially disentangled region while suppressing irrelevant areas.This targets semantic misalignment caused by applying global prompts uniformly across frame tokens.
  • Method: Self-attention increases intra-region focus and reduces inter-region interactions to limit feature coupling and texture mixing.The design makes each query attend to its target region.
  • Contributions: The method enables class-level, instance-level, and part-level editing and reports state-of-the-art qualitative and quantitative results without parameter tuning.The introduction describes this as a zero-tuning result on existing benchmarks and real-world videos.

2 RELATED WORK

Prior video-editing methods use attention, masks, correspondence, optical flow, or video-generation priors, but multi-grained editing remains difficult under layout, motion, and tuning constraints.

  • Image Diffusion Editing: Image-editing approaches manipulate attention maps or use masks for single-attribute foreground editing while preserving backgrounds.Examples include Pix2Pix-Zero, Prompt2Prompt, DiffEdit, and Latent Blend.
  • Multi-grained Editing: Multi-grained image-generation methods optimize subject attention or use layout masks and dense captions, but single-frame controls do not preserve video integrity and temporal consistency.Attention and Excite, DPL, layout-based modulation, and attention refocusing address related image-editing problems.
  • Image Diffusion Video Editing: Image-diffusion video editors improve consistency with inversion, attention fusion, optical flow, depth, or edge controls, yet struggle with local details, non-grid motion, or multi-grained edits.The cited methods also include cases with high memory use, layout-preservation problems, and detail loss.
  • Video Diffusion Editing: Video-diffusion editing methods provide temporal priors or motion transfer, but current models struggle with fixed views and non-grid human motion, while editing methods require parameter tuning.These constraints motivate approaches that address multi-grained editing without tuning.

3 METHOD

VideoGrain addresses multi-grained video editing by modulating space-time cross- and self-attention to align local prompts with target regions and separate regional features. Its zero-shot pipeline combines semantic layout and instance segmentation with unified attention modulation for class-, instance-, and part-level edits.

  • 3.1 MOTIVATION: K-Means self-attention features capture semantic layout but fail to distinguish separate instances, while cross-attention weights overlap or leak across target regions.These failures produce feature homogeneity, semantic misalignment, and unsuccessful instance-level edits.
  • 3.2 PROBLEM FORMULATION: Multi-grained editing targets class-level, instance-level, and part-level changes across video regions.Examples include editing same-class objects jointly, editing individual instances differently, and modifying specific parts.
  • 3.3 OVERALL FRAMEWORK: VideoGrain uses DDIM Inversion, self-attention clustering, SAM-Track instance segmentation, and ST-Layout Attn during denoising.The framework integrates the attention mechanism into frozen Stable Diffusion and supports paired instance- or part-level prompts with locations.
  • 3.4 SPATIAL-TEMPORAL LAYOUT-GUIDED ATTENTION: The unified modulation increases positive attention and decreases negative attention, with intensity varying over time and increasing for smaller regions.The regularization term controls modulation across denoising steps and compensates for layout-size differences.
  • 3.4 SPATIAL-TEMPORAL LAYOUT-GUIDED ATTENTION: Cross-attention amplifies each local prompt within its corresponding spatial region and suppresses attention to irrelevant areas for text-to-region control.The method represents prompt-location pairs as positive relationships and prompt-outside-region pairs as negative relationships, redistributing attention toward target areas.
  • 3.4 SPATIAL-TEMPORAL LAYOUT-GUIDED ATTENTION: Self-attention strengthens intra-region awareness and restricts inter-region interactions across frames to prevent feature coupling and preserve instance separation.Cross-frame diffusion feature similarities provide positive and negative relations, so a query such as a nose patch attends only to its target instance.

4 EXPERIMENTS

Experiments evaluate VideoGrain across multi-grained editing scenarios, qualitative comparisons, efficiency, attention behavior, and component ablations. Results show strong instance- and part-level control, efficient execution, and complementary benefits from cross- and self-attention modulation.

  • Qualitative Results: VideoGrain handles class-level, instance-level, and part-level edits, including separate vehicle transformations and edits in complex occluded scenes.Examples include transforming wolves into pigs, SUVs into firetrucks, vans into school buses, and edits involving Spider-Man and Wonder Woman.
  • Qualitative Comparison: VideoGrain separates same-class instances and supports part-level edits, while baselines mix features or fail to edit intended regions.Qualitative comparisons cover animal instances, human instances, sunglasses, boxing gloves, and backgrounds.
  • Efficiency Comparison: VideoGrain achieves the fastest editing time with the lowest memory usage in the 16-frame efficiency comparison.The comparison measures editing time plus GPU and CPU memory on a single A6000 GPU.
  • Attention Weight Distribution: With ST-Layout Attn, the target prompt’s attention is correctly distributed instead of spilling across unrelated regions.The attention visualization uses the prompt “An Iron Man is playing tennis on a snow court.”
  • Ablation Study: Cross-attention modulation raises CLIP-T by 7.4% and Q-edit by 63.9%, while adding self-attention modulation decreases Warp-Err by 43.9% and raises Q-edit by 80.6%.Cross modulation improves text-to-region control; self modulation reduces feature mixing and structure distortions.

5 CONCLUSION

VideoGrain addresses multi-grained video editing by modulating space-time cross- and self-attention. The paper reports improvements across class-level, instance-level, and part-level editing.

  • VideoGrain modulates cross-attention for text-to-region control and self-attention for feature separation between regions.Cross-attention focuses local prompts on corresponding regions, while self-attention increases intra-region awareness and reduces inter-region interactions.
  • Extensive experiments show VideoGrain surpasses previous video editing methods across class-level, instance-level, and part-level editing.

6 ETHICS STATEMENT

The ethics statement identifies potential misuse of multi-grained video editing, including deceptive identity-altering videos. It suggests invisible watermarking as a possible mitigation.

  • Multi-grained video editing could be misused to create deceptive videos by altering identities.
  • Invisible watermarking is suggested as a strategy to reduce malicious use of generated videos.

A EVALUATE SAM-TRACK MASKS’ IMPACT

The SAM-Track study tests joint and sequential multi-area editing and compares VideoGrain with methods using instance-level grounding. The results indicate that layout masks alone do not guarantee successful edits, whereas VideoGrain enables zero-shot multi-grained editing.

  • Experimental Design: The study compares joint editing of multiple areas with sequential editing of three areas using separate masks.
  • Results: Joint editing modifies only the left man because inaccurate cross-attention weights leave other areas unchanged.
  • Results: Sequential editing initially succeeds but later accumulates denoising errors, producing blurred details.
  • Grounding Comparison: Ground-A-Video struggles with multi-grained editing even when supplied with instance-level grounding comparable to SAM-Track masks.
  • Conclusion: SAM-Track provides layout guidance but does not guarantee successful edits, while VideoGrain performs zero-shot multi-grained editing.

B VIDEOGRAIN CAN WORK WITHOUT SAM-TRACK MASKS

VideoGrain can perform multi-area, part-level, and temporally consistent edits without strictly relying on SAM-Track masks. It also supports foreground-only subject edits while preserving backgrounds.

  • B VIDEOGRAIN CAN WORK WITHOUT SAM-TRACK MASKS: VideoGrain achieves high-quality multi-area editing using coarse layouts from clustered DDIM inversion self-attention features, without SAM-Track masks.The method is not strictly dependent on SAM-Track segmentation.
  • B VIDEOGRAIN CAN WORK WITHOUT SAM-TRACK MASKS: The method separately edits the left or right subject, or both jointly, while keeping the background unchanged.
  • B VIDEOGRAIN CAN WORK WITHOUT SAM-TRACK MASKS: Part-level editing changes human clothing attributes and structure, and modifies animal head or body colors while preserving unrelated details.Examples include recoloring a gray shirt, changing a half-sleeve shirt into a black suit, and preserving a cat’s belt color.
  • B VIDEOGRAIN CAN WORK WITHOUT SAM-TRACK MASKS: Full-frame ST-Layout Attention preserves texture details and prevents flickering, producing temporally consistent, layout-unified editing.Per-frame attention causes feature coupling, while sparse-causal attention produces flickering and misses details.

F CONTROLNET ABLATION

The ControlNet ablation shows that VideoGrain can edit multiple regions without explicit structural conditioning, although complex-motion cases may develop structural inconsistencies. Latent blending preserves unedited background regions during denoising.

  • F CONTROLNET ABLATION: VideoGrain performs simultaneous multi-region editing without ControlNet, but complex-motion results may show structural inconsistencies.Depth or pose conditioning supplies structural guidance in certain complex-motion cases.
  • F CONTROLNET ABLATION: Latent Blend uses masks to focus denoising on edit regions while keeping background regions identical to the source video.The method blends latent states during denoising using an aggregated mask.
  • F CONTROLNET ABLATION: For each frame, attribute masks are merged with a logical OR operation to form a global foreground mask.The resulting masks are aggregated across frames before latent blending.
  • F CONTROLNET ABLATION: The method also reports instance editing for animals and shape editing for cars.

J EXPERIMENTAL DETAILS

The experiments use specified replacement and blending settings for FateZero, four keyframes for TokenFlow on 16-frame videos, and default settings for several comparison methods.

  • J EXPERIMENTAL DETAILS: An ablation examines how increasing the number of frames affects ST-Layout Attention’s attention-weight distribution.
  • J EXPERIMENTAL DETAILS: TokenFlow uses SD editing with four keyframes by default for 16-frame videos.
  • J EXPERIMENTAL DETAILS: ControlVideo, Ground-A-Video, and DMT use their default hyperparameter settings, while ControlNet is reimplemented on the compared T2I codebases.The reimplementation is intended to ensure fairness across T2I-based methods.

K LIMITATIONS.

VideoGrain’s training-free, T2I-based design inherits limitations from the base model and struggles with large shape deformations and significant appearance changes.

  • K LIMITATIONS.: Generation quality remains limited by the base model because VideoGrain is training-free.Artifacts may occur when the Stable Diffusion generation prior is not ideal.
  • K LIMITATIONS.: The T2I-based method struggles with large shape deformations and significant appearance changes.The authors identify incorporating motion priors from T2V models as a possible future direction.
Loading 2502.17258v1…