Video Editing via Factorized Diffusion Distillation

• The approach separates video editing into precisely editing each frame and maintaining temporal consistency across edited frames.Source: §1 · p. 1
• Factorized Diffusion Distillation aligns image-editing and video-generation teachers into a video-editing student without supervised video-editing data.The student receives Score Distillation Sampling and adversarial-loss supervision from frozen adapters.Source: §1 · p. 1
• EVE sets state-of-the-art results on TGVE and on added object removal, addition, and texture-editing tasks.Source: §1 · p. 3
• EVE extends TGVE with temporally aware ViCLIP metrics and three additional editing operations: adding, removing, and changing object textures.Source: §1 · p. 3
• The alignment approach also supports personalized image-editing models by combining image-editing and LoRA adapters.Source: §1 · p. 3

• SDEdit preserves general structure but loses subject identity and textures, making it unsuitable for precise editing.Source: §2 · p. 3
• Cross-attention and extracted features can preserve structure or motion, but may omit information that should persist or preserve information that should change.Source: §2 · p. 3
• Inversion-based methods improve faithfulness at the cost of latency, while synthetic-data approaches adapt image-editing strategies to video.Source: §2 · p. 3
• This work distills separate editing and temporal capabilities from image-editing and video-generation teachers using unsupervised, factorized multi-teacher distillation.Unlike ADD, it learns a new capability rather than reducing diffusion steps and generates its own supervision.Source: §2 · pp. 3–4

3.1 Architecture

• The architecture stacks image-editing and video-generation adapters on the same frozen text-to-image backbone.Source: §3.1 · p. 4, §3 · p. 4
• The video-generation adapter uses Emu Video’s trained temporal layers on top of the frozen Emu text-to-image model.Source: Video Generation Adapter · p. 5

Image Editing Adapter

• The image-editing adapter is a ControlNet trained on the Emu Edit dataset, conditioned on an input image and textual edit instruction.Source: Image Editing Adapter · p. 5
• Both adapters process a noisy input video conditioned on the input video, editing instruction, and desired output caption to enable video editing.Source: Combining The Adapters · p. 5

Combining The Adapters

• The image adapter supplies frame-level precision while the video adapter supplies temporal consistency, but each alone lacks the other capability.Source: Combining The Adapters · p. 5
• The combined model is designed to edit videos that are both temporally consistent and faithful to the input, though it still produces significant artifacts.Source: Combining The Adapters · p. 5

Final Architecture

• The final architecture freezes both adapters and trains low-rank adaptation weights over the shared text-to-image backbone.Source: Final Architecture · p. 5

3.2 Factorized Diffusion Distillation

• FDD jointly distills an image editing adapter and a video generation adapter into a video-editing student using only input data, without supervised video-editing examples.Each training datum contains an output video caption, an editing instruction, and an input video.Source: §3.2 · p. 5
• FDD applies SDS losses from both teachers after independently noising student-generated videos and asking each teacher to predict the noise.The SDS loss compares sampled noise with each teacher’s prediction, with frozen teachers and weighting c(t).Source: §3.2 · p. 6
• The image editing teacher provides faithful, precise frame editing feedback, while the video generation teacher provides temporal-consistency feedback.Source: §3.2 · p. 6
• Because distillation produced blurry results, the method adds separate adversarial objectives for frame editing and video generation.The edit discriminator compares input-frame/instruction/output-frame triples, while the video discriminator compares videos and captions.Source: §3.2 · p. 6
• The student combines edit and video generator losses, each weighted by α and β, while the discriminators combine their corresponding objectives.In practice, α = β = 0.5 and λ = 2.5.Source: §3.2 · p. 6

3.3 Implementation Details

• Training uses k = 3 diffusion steps because this is the maximum number that fits into memory.Source: K-Bin Diffusion Sampling · p. 7
• The T diffusion steps are divided into k evenly sized bins, with one random timestep selected from each corresponding bin during generation.This avoids the discrepancy caused by using the same training timesteps while selecting a larger k at inference.Source: K-Bin Diffusion Sampling · p. 7

• Frozen DINO features feed trainable discriminator heads, following a base architecture similar to.Source: Discriminator Architecture · p. 7
• The image discriminator combines image, text, and noisy-image projections through an additional attention layer.Source: Discriminator Architecture · p. 7
• The video discriminator applies a single temporal attention layer over projected DINO features independently at each pixel.Source: Discriminator Architecture · p. 7

Metrics.

• The evaluation combines CLIPFrame and PickScore with temporally aware ViCLIPdir and ViCLIPout metrics, plus human judgments of text alignment, structure, quality, and overall preference.CLIPFrame and PickScore are inherited from TGVE, while ViCLIP metrics address temporal information.Source: Metrics · pp. 7–8

Unsupervised Dataset

• The FDD dataset uses 1,600 videos from Emu Video’s high-quality dataset, with Llama-2 generating seven editing instructions per video, one for each Emu Edit task.The tasks are Add, Remove, Background, Texture, Local, Style, and Global.Source: Unsupervised Dataset · p. 8

Editing Tasks Unseen During Alignment

• Although alignment trains on only a subset of image-editing tasks, EVE improves on unseen tasks such as segmentation, pose extraction, sketch conversion, and depth-map derivation.The result suggests that the student acquires broader teacher knowledge than the subset exposed during alignment.Source: Editing Tasks Unseen During Alignment · p. 8, Figure 4 · p. 11

4.1 Video Editing Benchmark

• TGVE+ extends the benchmark with object removal, object addition, and texture alteration tasks, broadening evaluation beyond TGVE’s original task categories.The expanded benchmark is publicly released to support ongoing video-editing research.Source: §4.1 · p. 8

4.2 Baseline Comparisons

• EVE achieves state-of-the-art results on TGVE and TGVE+, with human raters preferring it over all baselines and objective metrics favoring it except CLIPFrame.Human raters preferred EVE over STDF and Fairy in 72.4% and 71.7% of comparisons, respectively.Source: §4.2 · p. 9, Table 1 · p. 9

4.3 Ablations

• Ablations show that initializing from pretrained adapters is important, FDD is necessary when combining separately trained adapters, and both SDS and adversarial losses are needed for successful alignment.SDS has the larger impact, while adversarial loss alone provides partial alignment; sampling steps from k bins further improves FDD.Source: §4.3 · pp. 10–11

4.4 Additional Combinations of Adapters

• FDD also aligns image-editing adapters with subject- and style-driven LoRA adapters for personalized and stylized editing.The experiments combine editing with dog, toy-robot, line-art, and sticker adapters.Source: §4.4 · p. 11, Figure 5 · p. 12
• After alignment, edits become more consistent with reference subject identity and style, whereas unaligned plug-and-play combinations can fail to preserve them or produce significant artifacts.The comparison includes vanilla Emu Edit, unaligned adapter attachment, and aligned adapters.Source: §4.4 · pp. 11–12

• Performance is upper bounded by the capabilities of the teacher models, including the image editing adapter teacher.Source: §5 · p. 12
• Training from scratch performs poorly, so the method requires teachers trained as adapters over a frozen text-to-image backbone.Source: §5 · p. 12

• Video editing is factorized into precise editing of individual frames and temporal consistency among edited frames.Source: §6 · p. 12
• The two-stage method trains separate image editing and video generation adapters on a frozen text-to-image backbone, then aligns them with Factorized Diffusion Distillation.Source: §6 · pp. 12–13
• Emu Video Edit establishes a new state-of-the-art in video editing without supervised video editing data.Source: §6 · pp. 12–13
• The approach also demonstrates potential for learning other tasks by aligning additional adapters.Source: §6 · p. 13