Shape-aware Text-driven Layered Video Editing

1. Introduction

• Independent frame edits can flicker, while propagating a single edited keyframe can distort regions that were unseen in that keyframe.These are two distinct challenges to temporal consistency in video editing.Source: §1 · p. 1
• Fixed UV sampling in atlas-based methods supports appearance manipulation but prevents shape editing because the mapping retains each frame’s original shape information.Text2LIVE is described as an example of this limitation.Source: §1 · p. 2
• The method derives semantic correspondence between an input keyframe and its edited version, then maps the resulting shape and appearance changes into atlas space and across frames.It uses a pre-trained image-editing tool for the edited keyframe.Source: §1 · p. 2
• Atlas texture and deformation are further optimized with a pre-trained diffusion model to refine shape-aware edits and address unseen regions.The optimization uses a gradient-update procedure described in DreamFusion [38].Source: §1 · p. 2
• The paper’s stated contributions are enabling shape-aware video editing, formulating frame-dependent shape deformation, and using diffusion guidance for atlas completion.The method is reported to handle cases where a moving object undergoes 3D transformation.Source: §1 · p. 2

2. Related Work

• Video editing faces temporal-consistency and computational-complexity challenges, and methods include flow-based propagation, GAN inversion, and unified layer representations.GAN-based approaches are described as limited to datasets with restricted diversity, such as portraits or animal faces.Source: §2 · p. 3
• NLA learns a canonical appearance layer and per-frame UV mappings from video reconstruction, enabling edits to unified layers to be broadcast consistently across frames.The method uses test-time optimization on the input video.Source: §2 · p. 3
• Text2LIVE and Loeschcke et al. [31] use pretrained CLIP to guide test-time atlas editing, but fixed UV mappings restrict them to appearance edits.The fixed mappings preserve original shape information in the frames.Source: §2 · p. 3
• The paper builds on NLA but extends text-driven video editing to enable shape editing.This contrasts with the shape restriction attributed to existing atlas-based methods.Source: §2 · p. 3

3. Method

• NLA atlases and per-frame UV maps support shape-and-appearance editing while preserving the input video's motion.The method assumes a single moving object, represented by foreground and background atlases.Source: §3 · p. 3

3.1. Keyframe editing

• A selected keyframe is edited from the text prompt, then semantic correspondence aligns the edited and source frames to estimate shape deformation.Stable Diffusion is given as an example image-editing tool.Source: §3.1 · p. 4

3.2. Deformation formulation

• The method transforms keyframe displacement vectors into atlas space and propagates them through per-frame UV maps to deform each frame's UV map.A deformation matrix accounts for coordinate changes caused by warping, and thin-plate splines approximate the warp smoothly.Source: §3.2 · pp. 4–5

3.3. Atlas optimization

• Inaccurate correspondence can distort frames, while a single edited keyframe leaves unseen atlas pixels incomplete; diffusion-guided optimization refines correspondence and fills those pixels.The diffusion model provides pixel-level guidance across selected frames, while atlas-space updates maintain temporal consistency.Source: §3.3 · pp. 5–6
• Optimization uses an L1 keyframe reconstruction loss, total-variation atlas loss, and a mask loss comparing deformed target and source object masks.The implementation optimizes on 3–5 selected frames for 600–1000 iterations.Source: §3.3 · p. 6

4.1. Experimental Setup

• The DAVIS [37] videos contain moving objects across 50–70 frames, and each edit prompt describes a target object with a different shape.Baselines use Stable Diffusion [44], with multi-frame interpolation, single-frame propagation, and Text2LIVE [2] as comparison methods; sample results and code are noted as supplementary or forthcoming.Source: §4.1 · p. 6, §4 · p. 6

4.2. Visual Comparison

• Across “blackswan→duck,” “boat→yacht,” and “dog→cat,” our method produces consistent appearance and shape edits while preserving motion, unlike the compared baselines.Multi-frame and single-frame baselines show inconsistencies or propagation distortions, while Text2LIVE changes appearance but retains source shape.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3. Ablation Study

• Fixed UV mapping does not restore target shape; semantic correspondence maps edits into the atlas, deformation restores shape, and atlas optimization refines missing pixels and distortions.The reported artifacts include the car’s roof and back wheel.Source: Figure 6 · p. 7, §4.3 · p. 8

4.4. Application

• Interpolating atlas deformation maps transitions object shapes over time without additional frame interpolation, and atlas textures can also be interpolated.The foreground atlas is an unwrapped texture, so the method edits a video frame and maps it back to the atlas; the background atlas can be edited directly as a panorama.Source: §4.4 · p. 8, Figure 7 · p. 8

4.5. Limitations

• Complex motion can make NLA’s many-to-one atlas mapping imperfect, producing artifacts such as distortion around crossing hind legs.Poor semantic correspondence initialization can hinder atlas optimization, while manual user correction can improve results.Source: §4.5 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

• The method addresses the appearance-only limitation of existing methods with shape-aware text-driven video editing.It uses a deformation formulation in a layered video representation and pretrained text-to-image diffusion guidance to refine unseen regions.Source: §5 · pp. 8–9
• The paper presents a deformation formulation that transforms the mapping field for target shape edits and refines unseen regions using diffusion guidance.The authors state that the method facilitates shape and texture editing applications.Source: §5 · p. 9
• The authors caution that creative video-editing tools should be protected against misuse or malicious use.They state that the code will be released under a license similar to Stable Diffusion’s, focused on ethical and legal use.Source: §5 · p. 9