PaperlayerView exact revision on arXiv

Source-linked AI summary

MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance

Ernie Chu, Tzuhsuan Huang, Shuo-Yen Lin, Jun-Cheng Chen

arXiv:2308.10079v3cs.CV

TL;DR

Video translation with temporal consistency remains challenging, while existing video diffusion approaches can require substantial training resources. MeDM uses optical-flow-derived pixel correspondence coding to mediate pre-trained image diffusion models, and reports superior results across multiple benchmarks without additional video data. Its scope is constrained by optical flows' potential misalignment for objects with significant contour changes and by biases inherited from the underlying diffusion models.

  • Problem

    Video translation with temporal consistency remains challenging, and video diffusion models can demand substantial training videos and computational resources.

    Source: §1 · p. 1

  • Method

    MeDM uses optical-flow-based pixel correspondence coding and a global pixel repository to mediate image diffusion scores during denoising.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    MeDM outperforms existing methods on multiple benchmarks in video quality and temporal consistency.

    Source: §1 · p. 1, §6 · p. 7

  • Takeaways & Limitations

    MeDM achieves video-to-video translation with pre-trained image diffusion models without additional video data for fine-tuning or test-time optimization.

    Source: §1 · p. 1, §6 · p. 7

    Optical flows may misalign with objects undergoing significant contour changes, and MeDM inherits biases from its underlying diffusion models.

    Source: §6 · p. 7

Abstract

from arXiv · show

This study introduces an efficient and effective method, MeDM, that utilizes pre-trained image Diffusion Models for video-to-video translation with consistent temporal flow. The proposed framework can render videos from scene position information, such as a normal G-buffer, or perform text-guided editing on videos captured in real-world scenarios. We employ explicit optical flows to construct a practical coding that enforces physical constraints on generated frames and mediates independent frame-wise scores. By leveraging this coding, maintaining temporal consistency in the generated videos can be framed as an optimization problem with a closed-form solution. To ensure compatibility with Stable Diffusion, we also suggest a workaround for modifying observation-space scores in latent Diffusion Models. Notably, MeDM does not require fine-tuning or test-time optimization of the Diffusion Models. Through extensive qualitative, quantitative, and subjective experiments on various benchmarks, the study demonstrates the effectiveness and superiority of the proposed approach. Our project page can be found at https://medm2023.github.io

1 Introduction

Image diffusion models produce high-quality text-guided images, but video translation with temporal consistency remains challenging. MeDM mediates frame-wise diffusion scores using optical-flow correspondences and reports improved video quality and temporal consistency without additional video fine-tuning or test-time optimization.

1 Introduction

  • Video generation and translation with temporal consistency remains challenging despite advances in text-to-image and image-to-image diffusion models.Dedicated text-to-video models demand substantial training videos and computational resources, while zero-shot image-model adaptations can flicker or require intricate processes.Source: §1 · p. 1
  • MeDM outperforms state-of-the-art alternatives in video quality and temporal consistency across benchmarks and tasks.The paper also reports promising subjective evaluations for text-guided video editing and video anonymization.Source: §1 · pp. 1–2
  • MeDM uses explicit optical flows to encode pixel correspondences across frames, mediating independent image scores through a global pixel repository.Pixels along a trajectory stay consistent with their corresponding repository pixels, and harmonization is framed as an optimization problem with a closed-form solution.Source: §1 · p. 1
  • The optical-flow coding integrates into image-based denoising cycles, and MeDM requires neither additional video data for fine-tuning nor test-time optimization.The method iteratively aligns corresponding noisy pixels across frames at each denoising step.Source: §1 · p. 1
  • MeDM proposes a workaround for modifying observation-space scores in latent diffusion models, enabling more general latent-model applications.Source: §1 · p. 1
  • The paper reports effective real-world applications with promising subjective evaluations, including text-guided video editing and video anonymization.Source: §1 · p. 2

2 Related Work

Related work covers zero-shot adaptation of image diffusion models for video translation and harmonization of multiple diffusion samples. The paper positions MeDM as handling more general views across video time and space.

2 Related Work

  • The section reviews text-guided video-to-video generation and the mediation of multiple diffusion generations.Source: §2 · p. 2

2.1 Diffusion-Based Video-to-Video Translation

  • Zero-shot image-model adaptations include methods for self-attention, key-frame propagation, cross-frame interaction, and motion-enhanced conditioning, but artifacts and flickering remain visible.The section frames these techniques as approaches to video transformation and contrasts them with MeDM's harmonious editing.Source: §2.1 · p. 2, §2 · p. 2

2.2 Mediating Multiple Diffusion Samples

  • MultiDiffusion treats overlapping image views as a least-squares harmonization problem and averages their intersection during denoising.Its tests mainly use rectangular crops and quantized masks from low-dimensional latent vectors; MeDM handles views spanning temporal and spatial dimensions, including nonconsecutive video pixels.Source: §2.2 · p. 2

3 Optical Flow-Based Temporal Correspondence

The section uses optical-flow correspondences to encode video pixels in a shared repository, then harmonizes independently generated frames by solving a temporal-consistency objective. Its coding assumes that pixels viewing the same underlying points have identical colors, disregarding lighting-related shade changes.

3.1 Flow Coding

  • The coding assumes that pixels viewing the same underlying analog points have identical colors, while intentionally disregarding shade changes from surrounding and environmental lighting.The authors state that addressing lighting effects is planned for future research.Source: §3.1 · p. 2
  • Flow Coding uses adjacent-frame optical flows and occlusions to append previously unseen pixels to a global repository and propagate their codes along motion trajectories.Each frame is encoded as indices into the repository, allowing pixels on the same trajectory to share a repository entry.Source: §3.1 · p. 3

3.2 Global Harmonization

  • Because the diffusion model generates each frame independently, the method defines a temporal-consistency loss between noisy samples and values fetched from the coded pixel repository.The mixing function G harmonizes associated pixels in Xt before they are assigned to the repository entries.Source: §3.2 · p. 3
  • A closed-form post-hoc solver minimizes the distance between noisy frames and repository values by unweighted averaging of pixels that share a Flow Coding index.The repository contains fewer unique pixels than the frame tensors, so associated pixels are mixed before their values are assigned to it.Source: §3.2 · pp. 3–4

4 Mediating Image Diffusion Samples with Temporal Correspondence Guidance

This section applies temporal correspondence guidance during image-diffusion denoising, where frames are otherwise generated independently. It also describes a latent-space workaround for LDM compatibility and uses SDEdit and ControlNet to address blurry results from direct score harmonization.

4.1 Diffusion Models

  • A diffusion model predicts either a reconstruction of the data or a noise residual from a noisy sample to construct a less noisy data distribution.The noisy sample at step t is defined using a data point and a predefined noise schedule.Source: §4.1 · p. 4

4.2 Temporal Correspondence Guidance

  • The method modifies noisy samples at each diffusion-model evaluation to apply temporal correspondence guidance, with weight w controlling the guidance strength.The authors report that an extrapolation formulation caused serious early saturation in their experiments.Source: §4.2 · p. 4
  • Equation 9 is incompatible with latent diffusion models because their denoising occurs at a smaller spatial dimension and their autoencoders cannot directly cast noisy samples or noise predictions into the needed space.The proposed workaround estimates the original samples from predicted noise and uses pretrained autoencoders to cast them between latent and observed spaces.Source: §4.2 · p. 4

4.3 Enhancing Video Sharpness

  • Directly harmonizing independent score estimates from standard diffusion models produced suboptimal blurry video, so the method incorporates SDEdit and ControlNet.The text identifies ControlNet as effective at fusing input structural information with pretrained diffusion-model semantic priors.Source: §4.3 · p. 4

5 Experiments

Experiments evaluate rendering from synthetic 3D assets, real-video editing and anonymization, and the roles of ControlNet and SDEdit. The reported results include lower flow error, favorable human ratings, and strong outcomes in editing and anonymization.

5 Experiments

  • Experiments use 20-step DDIM (w=0.8), Stable Diffusion, UniMatch flow, and 71 online raters; some SDXL modality adapters were unavailable during development.The setup uses Informative Drawings for lineart conditions unless otherwise specified.Source: §5 · pp. 4–5

5.1 Video Rendering

  • On Sintel, MeDM's videos are high-quality and fluent, enhance complex-texture realism in assistive rendering, and receive the highest quality and realism ratings.The visual comparison covers full and assistive rendering; participants also rated sharpness and fluency.Source: §5.1 · p. 5, Figure 5 · p. 6, Table 2 · p. 7
  • MeDM has the lowest EPE among methods for each video-rendering task and is closest to the animation's EPE.EPE compares estimated flows from generated videos with ground-truth flows to assess temporal consistency.Source: Table 1 · p. 5, §5.1 · p. 5

5.2 Video-to-Video Translation

  • On DAVIS 2016, MeDM—especially its InstructPix2Pix variant—produces the best video quality and image-text alignment in subjective scores.The comparison tests editing with absurd text descriptions and includes lineart and InstructPix2Pix variants.Source: §5.2 · p. 5, Figure 6 · p. 6, Table 3 · p. 7
  • On CelebV-HQ, a larger share of participants could not recognize celebrities in MeDM-anonymized videos than in DeepPrivacy videos; ratings also favored realism and retained detail.The anonymization procedure adds noise at strength 0.5T before denoising.Source: §5.2 · pp. 5, 7

5.3 Without ControlNet or SDEdit

  • Removing either ControlNet or SDEdit increases EPE, and removing both prevents faithful adherence to motion structure indicated by optical flows.The ablation evaluates the two components used throughout the experiments.Source: §5.3 · p. 7

6 Conclusion and Future Work

MeDM uses optical-flow guidance to preserve temporal correspondence and reports superior results across multiple benchmarks without additional video data. Its future directions include improved correspondence methods and structural translation, while the method retains limits tied to optical flow and underlying diffusion-model biases.

6 Conclusion and Future Work

  • MeDM employs optical flows to guide temporal correspondence, preserving consistency in efficient, high-quality video-to-video translation.Source: §6 · p. 7
  • MeDM avoids additional video data and achieves superior results on multiple benchmarks compared with existing methods.The user-feedback table evaluates anonymization through recognizability, realism, and faithfulness; participants were limited to those familiar with the celebrities.Source: §6 · p. 7, Table 4 · p. 7
  • The method uses discretized adjacent-frame flows; point tracking could improve correspondence but is currently costlier and less robust, while large contour changes may challenge flow alignment.MeDM also inherits biases from its underlying diffusion models, though community-driven LoRA and fine-tuned checkpoints may help mitigate them.Source: §6 · p. 7

A System

  • The implementation heavily uses Hugging Face Diffusers, with ControlNet lineart and InstructPix2Pix weights hosted on Hugging Face.Source: A System · p. 10

B Flow Coding

This section supplements the Flow Coding toy example with a complete algorithm.

B Flow Coding

  • Algorithm 1 provides the complete Flow Coding procedure to complement the toy example in Figure 3.Source: B Flow Coding · p. 10

C Reducing Discretization Errors

The method addresses errors caused by discretizing per-pixel optical flows, including accumulated errors during code propagation. Distant-frame flows can counteract accumulation, but their estimation errors may limit the benefit.

C Reducing Discretization Errors

  • Rounding per-pixel flow destinations to integer coordinates introduces errors in code propagation, motivating a mitigation technique.Source: C Reducing Discretization Errors · p. 10
  • Distant-frame optical flows can counteract accumulated discretization errors by prioritizing their codes during frame encoding, but their greater estimation errors may prevent performance gains.The distance d is the number of frames preceding the current frame; the index is clamped to be non-negative.Source: C Reducing Discretization Errors · p. 10

D Local Harmonization

Local harmonization relaxes global consistency by smoothing pixel values over time, allowing color transitions that accommodate varying shading. It uses Flow Coding and an inverse pixel repository, but is not physically correct and adds substantial computation.

D Local Harmonization

  • Local harmonization smooths pixels observing the same object point in temporal order, relaxing global consistency to accommodate shading changes.The sequence is convolved with a predefined Gaussian kernel.Source: D Local Harmonization · p. 10
  • Flow Coding propagates pixel indices along rounded optical-flow destinations, assigns new codes at occlusions, and outputs encoded frames plus the unique-pixel count.Source: D Local Harmonization · p. 10
  • The inverse pixel repository groups coordinates by code; local harmonization uses these indexed pixels, but is not physically correct and costs O(T H W).It provides visually appealing, fluent color transitions over time.Source: D Local Harmonization · p. 10
  • Local harmonization takes almost five times as long as Stable Diffusion combined with ControlNet.The increased execution time is attributed to building the reverse repository and harmonizing at each diffusion step.Source: D Local Harmonization · p. 11

D.1 Experiments

Local harmonization tests show a fluidity–global-consistency trade-off as the Gaussian kernel’s smoothing level changes; the user study uses a one-time anonymization-identity survey.

D.1 Experiments

  • The experiment assesses local harmonization with discrete 1D Gaussian kernels of varying width ℓ.The kernel’s sharpness is regulated by σ; the experiment varies σ to control harmonization strength.Source: §D.1 · p. 11
  • Videos become more fluid as r diminishes, at the cost of a more rigorous global-consistency constraint.The experiments compare generated-video EPE against r, the ratio of the kernel’s maximum to minimum weight.Source: §D.1 · p. 11

E User Study

  • Each survey participant encounters the anonymization survey only once and guesses the identity concealed in a video.Participants receive no prior celebrity context, so source videos used to assess methods cannot be reused.Source: E User Study · p. 11

F Additional Samples

The additional materials pair MPI Sintel video samples with visuals on execution time, local harmonization, and the anonymization survey.

F Additional Samples

  • Figure 7 plots execution-time distribution for generating videos across MPI Sintel, shading the components proposed in this work.Source: Figure 7 · p. 12
  • Figure 9 depicts the anonymization survey sequence: participants see the anonymized video before the original, and responses count only when they know the celebrity.Source: Figure 9 · p. 12
  • Figure 8 presents local harmonization results; panel 8a marks global-harmonization EPE with a red line.Source: Figure 8 · p. 12
  • Figure 10 shows Sintel lineart-ControlNet videos generated from pure noise; Figure 11 uses noise level 0.5T.These are additional visual samples of generated videos.Source: F Additional Samples · p. 11, Figure 10 · p. 13, Figure 11 · p. 14

Related papers

Loading 2308.10079v3…