PaperlayerView exact revision on arXiv

Source-linked AI summary

UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing

Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, Jiang Bian

arXiv:2402.13185v4cs.CV

TL;DR

Video editing methods have made progress on appearance, while motion editing remains underexplored. UniEdit uses a pre-trained text-to-video generator and auxiliary branches for tuning-free motion and appearance editing; experiments report performance beyond state-of-the-art methods across these tasks.

  • Problem

    Video appearance editing has advanced, but motion editing remains underexplored and existing methods lack motion priors and control over inter-frame dependencies.

    Source: §1 · p. 2

  • Method

    UniEdit uses a pre-trained text-to-video generator in an inversion-then-generation framework, with auxiliary branches providing motion and source features for the main editing path.

    Source: §1 · p. 2

  • Results

    UniEdit supports video motion editing and various appearance-editing scenarios, and experiments report performance superior to existing state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    UniEdit provides a unified tuning-free framework for video motion and appearance editing, with motion and appearance edits possible by calling it twice.

    Source: §6 · p. 8

    Simultaneous motion and appearance editing remains unexplored, and automatic determination of the multiple hyper-parameters warrants investigation.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.g., from eating to waving), which distinguishes video editing from image editing, is underexplored. In this work, we present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversion-then-generation framework. To realize motion editing while preserving source video content, based on the insights that temporal and spatial self-attention layers encode inter-frame and intra-frame dependency respectively, we introduce auxiliary motion-reference and reconstruction branches to produce text-guided motion and source features respectively. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers. Extensive experiments demonstrate that UniEdit covers video motion editing and various appearance editing scenarios, and surpasses the state-of-the-art methods. Our code will be publicly available.

1. Introduction

Video editing methods have advanced in appearance editing, but motion editing remains limited by missing motion priors and control over inter-frame dependencies. UniEdit uses a tuning-free, inversion-then-generation framework with auxiliary branches to edit motion and appearance while retaining source content and structure.

1. Introduction

  • Video appearance editing can maintain frame-wise consistency, but existing methods fall short on motion editing because they lack motion priors and control over inter-frame dependencies.The contrast includes replacing guitar-playing movement with waving.Source: §1 · p. 2
  • Prior motion-editing methods fine-tune a generator on the source video, but balancing generation with content preservation can restrict motion diversity and introduce unwanted content changes.Source: §1 · p. 2
  • UniEdit uses a pre-trained text-to-video generator in an inversion-then-generation framework, with a motion-reference branch injecting text-guided motion features through temporal self-attention.A reconstruction branch injects spatial self-attention features to preserve non-edited content; its spatial attention maps also replace those in the main path to retain structure during appearance editing.Source: §1 · p. 2
  • Experiments report that UniEdit supports video motion editing, a wide range of appearance editing, and zero-shot text-image-to-video generation, with performance superior to existing state-of-the-art methods.Source: §1 · p. 2

2. Related Works

The related work reviews learned video-generation priors and editing methods for appearance and motion. It positions UniEdit as using a pre-trained video-generation prior for tuning-free text-guided editing.

2.1. Video Generation

  • Video-generation research has used GANs, language models, and diffusion models, with learned real-world video distributions supporting text-to-video and image-to-video generation.Other work adds structure guidance or customizes generation; UniEdit leverages a pre-trained model’s learned prior for tuning-free editing.Source: §2.1 · p. 2

2.2. Video Editing

  • Video editing seeks instruction-aligned outputs while maintaining other source-video characteristics, and is categorized as appearance or motion editing.Source: §2.2 · p. 2
  • Appearance-editing methods address temporal consistency through inter-frame propagation or edit-then-propagate strategies, but source-structure constraints make them unsuitable for direct motion editing.Examples include Pix2Video, Video-P2P [36], and approaches using nearest-neighbor fields, optical flow, or temporal deformation fields [41].Source: §2.2 · pp. 2–3
  • Prior motion-editing approaches use text guidance, user-provided motion, or specific motion representations; UniEdit is described as the first tuning-free approach to text-guided motion and appearance editing.Dreamix [40] fine-tunes a model per source video, while MoCA [64] generates motion from text and struggles to preserve non-edited motion such as background dynamics.Source: §2.2 · p. 3

3. Preliminaries: Video Diffusion Models

The preliminaries recap text-guided video diffusion models and the attention modules used in their denoising UNets. The paper uses the stated roles of spatial and temporal attention in its later design.

3. Preliminaries: Video Diffusion Models

  • The paper’s preliminaries recap the architecture of common text-guided video diffusion models as the foundation for UniEdit.Source: §3 · p. 3

3.1. Overall Architecture

  • Text-to-video diffusion models typically extend pre-trained text-to-image models with temporal layers, and often generate at low spatial and temporal resolutions before cascaded upsampling.Other adaptations include changes to efficiency, training strategy, or control signals; inference denoises Gaussian noise over T steps and decodes latents when the model is trained in latent space.Source: §3.1 · p. 3

3.2. Attention Mechanisms

  • Denoising UNet blocks include convolution, spatial self-attention, spatial cross-attention, and temporal self-attention modules; these respectively support text-semantic fusion, intra-frame dependency modeling, and inter-frame dependency modeling with temporal consistency.The attention operation uses Q, K, and V derived from its inputs, with d denoting hidden-state dimension.Source: §3.2 · p. 3

4. UniEdit

UniEdit uses an inversion-then-generation pipeline with auxiliary branches that supply source-content and motion features to a main editing path. The framework adapts this design for appearance editing, mask-guided coordination, and image-and-text-conditioned video generation.

4. UniEdit

  • UniEdit denoises an inverted latent in a target-prompt-conditioned main path, supported by reconstruction and motion-reference branches and extended with mask-guided coordination and text-image-to-video generation.The reconstruction branch supplies source features, while the motion-reference branch supplies motion features.Source: §4 · p. 3

4.1. Tuning-Free Video Motion Editing

  • For motion editing, reconstruction-branch spatial-attention features preserve source content, while target-prompt-conditioned motion-reference features enter temporal self-attention to guide motion.The method also applies spatial structure control in early denoising steps to help fuse motion with source content.Source: §4.1 · pp. 3–4

4.2. Tuning-Free Video Appearance Editing

  • Appearance editing removes the motion-reference branch and uses reconstruction-branch query and key features in spatial self-attention to retain frame composition while allowing edited textures and styles.The paper distinguishes this structure control from content preservation, which inherits source textures and style.Source: §4.2 · p. 5

4.3. Mask-Guided Coordination

  • Mask-guided coordination uses foreground/background masks in self-attention and branch-feature fusion to help distinguish foreground from background and improve content consistency between edited and source videos.Masks may come from thresholded cross-attention maps or an off-the-shelf segmentation model.Source: §4.3 · p. 5

4.4. T2V Models are Zero-Shot TI2V Generators

  • For image-to-video generation, UniEdit edits a pseudo-video formed with simulated camera movement or an animation-generated video, using a text prompt to guide the result.The animation-generated input may contain random motion that does not match the prompt.Source: §4.4 · pp. 5–6

5. Experiments

UniEdit is evaluated for both motion and appearance editing, with comparisons and ablations examining content preservation, prompt alignment, and temporal consistency. The experiments also analyze how its attention modules and auxiliary branches contribute to editing behavior.

5.1. Implementation Details

  • Experiments instantiate UniEdit with LaVie and process videos at 320×512 resolution, taking 1–2 minutes per video on an NVIDIA A100 GPU.The framework is not limited to a specific video diffusion model.Source: §5.1 · p. 6

5.2. Comparison with State-of-the-Art Methods

  • Qualitative examples cover motion changes, object replacement, style transfer, and background modification, with target-prompt alignment and temporal consistency.The examples are shown in Figures 1 and 4, with additional cases in Appendix B.Source: §5.2 · p. 6
  • UniEdit better preserves source content in appearance editing and outperforms motion-editing baselines by a large margin.Most motion-editing baselines fail to align with the target prompt or preserve source content.Source: §5.2 · pp. 6–7, Figure 5 · p. 7
  • UniEdit outperforms baseline methods in quantitative comparisons of temporal consistency, textual alignment, and user preference.The user study asked 10 participants to rate edited videos on a five-grade scale.Source: §5.2 · p. 7, Table 1 · p. 7

5.3. Ablation Study and Analysis

  • Temporal cross-frame attention maps overlap with optical flow more than spatial query maps, motivating motion injection through temporal self-attention.Content preservation and structure control are conducted through spatial self-attention.Source: §5.3 · p. 7, Figure 6 · p. 8
  • The reconstruction branch alone preserves identity and background but deviates from the prompt, while motion injection alone causes dramatic background changes.UniEdit combines both branches and achieves content preservation alongside text adherence.Source: §5.3 · p. 8
  • Without spatial structure control, the source structure is lost; with the proposed scheme, edited results adhere to that structure.Figure 8 compares the ablation in its upper rows with results using the scheme in its lower rows.Source: §5.3 · p. 8
  • Injecting features at fewer denoising steps produces a more stylized output than injection at more steps, supporting flexible video editing.The example compares t2=15 with t2=50.Source: §5.3 · p. 8

6. Conclusion and Limitations

UniEdit uses auxiliary motion-reference and reconstruction branches to support video motion and appearance editing. The authors identify simultaneous editing and automatic hyperparameter selection as unresolved limitations.

6. Conclusion and Limitations

  • UniEdit is designed for video motion and appearance editing, using auxiliary motion-reference and reconstruction branches with features injected into the main editing path.The conclusion states that the framework performs motion editing and various appearance editing tasks.Source: §6 · p. 8
  • Motion and appearance editing can be achieved by calling UniEdit twice, but simultaneous editing remains unexplored, and automatic hyperparameter determination remains an open question.The authors also suggest adding watermarks to prevent misuse of the research-purpose approach.Source: §6 · p. 8

A. Detailed Experimental Settings

The experiments use LaVie as the base text-to-video model and evaluate editing with both human ratings and comparisons against baselines. UniEdit receives the best frame-consistency rating and significantly outperforms the baselines on textual alignment.

A. Detailed Experimental Settings

  • The method uses LaVie’s pretrained text-to-video model, with edited clips optionally passed to temporal interpolation and video super-resolution models.The experiments use LaVie’s base model and open-source pretrained weights.Source: A. Detailed Experimental Settings · p. 13
  • Input videos are resized to 320 × 512, normalized, and set to 16 frames; editing takes 1–2 minutes per source video on an NVIDIA A100.The preprocessing follows LaVie’s training configuration.Source: A. Detailed Experimental Settings · p. 13
  • Real videos use 50 DDIM inversion steps and 50 deterministic sampling steps; the user study rates frame consistency and textual alignment on a 1–5 scale.The study collected ratings from 10 experienced volunteers across 20 randomly selected editing samples.Source: A. Detailed Experimental Settings · p. 13
  • UniEdit has the best frame-consistency rating and significantly outperforms all baselines on textual alignment.The authors report these human-rating results in Table 1.Source: A. Detailed Experimental Settings · p. 13
  • Baselines are implemented from official repositories, with MasaCtrl adapted to video and Stable Diffusion-based methods evaluated at 512 × 512.The resolution is chosen to match Stable Diffusion’s default configuration.Source: A. Detailed Experimental Settings · p. 13

B.1. More Ablation Study

The ablation compares UniEdit with and without mask-guided coordination; the authors report improved editing performance with the proposed coordination.

B.1. More Ablation Study

  • Mask-guided coordination improves editing performance compared with the ablated version without it.Figure 9’s second line omits mask-guided coordination, while its last line shows UniEdit with the proposed coordination.Source: §B.1 · p. 13

B.2. More Comparison with State-of-the-Art Methods

The appendix points readers to additional comparisons with state-of-the-art methods in Figures 10 and 11.

B.2. More Comparison with State-of-the-Art Methods

  • Figures 10 and 11 provide additional comparisons with state-of-the-art methods.The passage identifies the figures as further comparisons but does not report their outcomes.Source: §B.2 · p. 13

B.3. More Results of UniEdit

The appendix collects further UniEdit results, including an ablation, more appearance- and motion-editing examples, and text-image-to-video generation.

B.3. More Results of UniEdit

Loading 2402.13185v4…