PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Image Dynamics

Zhengqi Li, Richard Tucker, Noah Snavely, Aleksander Holynski

arXiv:2309.07906v3cs.CV

TL;DR

Learning realistic scene motion is difficult, although still images can suggest plausible distributions of natural oscillations. This paper predicts spectral volumes from single images with a frequency-coordinated diffusion model and renders their motion trajectories into videos. The approach outperforms prior single-image animation baselines on reported image and video synthesis quality and supports looping and interactive dynamics.

  • Problem

    Learning realistic scene motion is difficult because the physical properties underlying observed motion are hard to measure at scale.

    Source: §1 · p. 1

  • Method

    A frequency-coordinated diffusion model predicts spectral volumes from single images, which are converted to motion textures for image-based rendering.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    The approach significantly outperforms prior single-image animation baselines in image and video synthesis quality, with lower FVD and DT-FVD distances.

    Source: §7.1 · p. 8

  • Takeaways & Limitations

    Motion priors capture lower-dimensional structure for long-range pixel variation and support coherent long-term generation, finer animation control, looping, and interactive dynamics.

    Source: §1 · p. 2

    Predicting only lower spectral frequencies can miss nonoscillating motion and high-frequency vibrations, while trajectory quality may degrade for thin moving objects or large displacements.

    Source: §8 · p. 8

Abstract

from arXiv · show

We present an approach to modeling an image-space prior on scene motion. Our prior is learned from a collection of motion trajectories extracted from real video sequences depicting natural, oscillatory dynamics such as trees, flowers, candles, and clothes swaying in the wind. We model this dense, long-term motion prior in the Fourier domain:given a single image, our trained model uses a frequency-coordinated diffusion sampling process to predict a spectral volume, which can be converted into a motion texture that spans an entire video. Along with an image-based rendering module, these trajectories can be used for a number of downstream applications, such as turning still images into seamlessly looping videos, or allowing users to realistically interact with objects in real pictures by interpreting the spectral volumes as image-space modal bases, which approximate object dynamics.

1. Introduction

Natural scenes contain subtle oscillations, but learning realistic scene motion is difficult because underlying physical properties are hard to measure at scale. The paper learns a generative prior over image-space motion from real videos, predicting spectral volumes that support animation and interactive dynamics.

1. Introduction

  • Realistic scene motion is difficult to learn because the physical properties driving it are hard to measure and capture at scale.Observed 2D motion can nevertheless provide a basis for simulating plausible dynamics.Source: §1 · p. 1
  • Still images can suggest distributions of plausible motion, since natural dynamics such as candle flicker and tree sway are often predictable.The paper frames computationally modeling these image-conditioned motion distributions as a research problem.Source: §1 · pp. 1–2
  • The method learns an image-space motion prior from automatically extracted spectral volumes in real videos and samples them from a single image using diffusion.Spectral volumes represent dense, long-range pixel trajectories and suit oscillatory scenes such as trees and flowers.Source: §1 · p. 2
  • A predicted spectral volume converts into a motion texture of long-range per-pixel trajectories that can animate the input image.The predicted trajectories are rendered into future frames using an image-based rendering module.Source: §1 · p. 2
  • Compared with raw-RGB priors, motion priors capture lower-dimensional structure explaining long-range pixel variation and support more coherent long-term generation and finer animation control.Demonstrated applications include seamless looping, motion editing, and interactive dynamics through image-space modal bases.Source: §1 · p. 2

2. Related Work

Prior work generates videos from text or animates still images using external motion sources, priors, or annotations, while motion-prediction work often targets optical flow or closed-domain tasks. This paper instead learns a spectral-volume prior in advance for application to single images.

Generative synthesis

  • Text-conditioned video generation can inherit artifacts such as incoherent motion, unrealistic temporal texture changes, and violations of physical constraints like mass preservation.These methods extend generated image tensors along a time dimension.Source: §2 · p. 2

Animating images

  • Still-image animation methods use direct video generation or image-based rendering driven by videos, priors, or user annotations, but often need extra guidance or use limited motion representations.Image-based rendering can yield greater temporal coherence and realism than directly generating video content.Source: §2 · p. 2

Motion models and motion priors

  • Fourier-shaped noise and modal analysis have modeled natural oscillatory motion, and Chuang et al. adapted spectral methods to animate annotated 2D pictures.Inspired by Davis, this work extracts spectral-volume representations from training videos and predicts motion from single images with diffusion.Source: §2 · p. 2
  • Prior motion models predict deterministic estimates or distributions, but many focus on optical flow rather than full trajectories or on recognition rather than synthesis.Generative motion prediction has also been explored in closed domains such as humans and animals [2] [19] [28] [72] [91] [107].Source: §2 · pp. 2–3

Videos as textures

  • Dynamic textures model videos as space-time samples of stochastic processes, while video textures use frames and transition probabilities to produce looping or varying videos.Unlike much of this work, the paper learns priors in advance for application to single images.Source: §2 · p. 3

3. Overview

Given a still image, the system predicts a spectral volume, converts it into per-pixel motion trajectories, and renders the image into a video. The motion texture also supports looping and interactive-dynamics applications.

3. Overview

  • The system takes a still image and uses a latent diffusion model to predict a spectral volume for oscillatory motion.The pipeline has motion-prediction and image-based-rendering modules.Source: §3 · p. 3
  • An inverse discrete Fourier transform converts the predicted spectral volume into a motion texture that specifies each input pixel's position at future time steps.Source: §3 · p. 3
  • Neural image-based rendering animates the input RGB image from the motion texture, supporting seamless looping and interactive-dynamics applications.Source: §3 · p. 3

4. Predicting motion

The method represents long-term pixel trajectories in the frequency domain, where natural oscillatory motion can be described by a small set of harmonic components. It predicts spectral volumes from an input image with a diffusion model, using frequency coordination to produce coherent predictions.

4.1. Motion representation

  • A motion texture is a sequence of time-varying 2D displacement maps that specifies each input pixel’s position at future times.The spectral volume is the temporal Fourier transform of per-pixel trajectories, providing a frequency-domain representation of that motion.Source: §4.1 · p. 3
  • Natural oscillatory motion is represented as a superposition of harmonic oscillators, motivating a spectral volume that encodes per-pixel trajectories in the temporal Fourier domain.The representation avoids predicting a separate displacement field for every output frame.Source: §4.1 · p. 3
  • The method models K = 16 low-frequency Fourier coefficients, which the authors found sufficient to realistically reproduce natural motion in varied videos and scenes.The average motion power spectrum from 1,000 five-second clips decreases exponentially with frequency.Source: Figure 2 · p. 3, §4.1 · p. 4

4.2. Predicting motion with a diffusion model

  • Frequency-adaptive normalization scales coefficients using per-frequency training-set statistics and a square-root transform, yielding a more even coefficient distribution.This addresses the concentration of higher-frequency coefficients near zero under image-dimension scaling.Source: §4.2 · pp. 4–5
  • The model predicts one frequency slice at a time, then fine-tunes interleaved attention layers across frequency bands to coordinate the slices.This avoids the over-smoothed outputs of a single large-channel U-Net and the uncorrelated predictions of independently generated slices.Source: §4.2 · p. 5
  • Average VAE reconstruction error improves from 0.024 to 0.018 with frequency-coordinated denoising rather than a single 2D U-Net.The authors also report that this design improves video generation quality.Source: §4.2 · p. 5

5. Image-based rendering

The renderer converts a predicted spectral volume into a time-specific motion field, then uses multi-scale feature warping and synthesis to render a future frame. Motion magnitude supplies the splatting weights, and the rendering networks are trained against real target frames.

Image-based rendering

  • A spectral volume is inverse-transformed into a per-pixel motion texture, whose field forward-warps encoded input features to render a future frame.The synthesis decoder receives the warped features and produces the rendered image.Source: §5 · p. 5
  • Feature-pyramid softmax splatting warps each feature scale using the resized motion field, addressing holes and overlapping source pixels.The module extracts multi-scale input features before warping them.Source: §5 · p. 5, Figure 4 · p. 5
  • Per-pixel splatting weights use predicted flow magnitude as a depth proxy, assuming larger motions indicate foreground and small or zero motions indicate background.Motion-derived weights are used because learnable weights were ineffective for single-view disocclusion ambiguities.Source: §5 · p. 5
  • The feature extractor and synthesis networks are jointly trained on sampled real-video frame pairs, using estimated flow to warp features and VGG perceptual loss against target frames.Source: §5 · p. 5

6. Applications

The applications use predicted motion to animate still images, adjust motion speed or amplitude, create seamless loops, and simulate interactive object dynamics. Looping guidance constrains motion during denoising, while modal analysis turns spectral components into a force-responsive motion basis.

Image-to-video

  • A predicted spectral volume animates a still image; interpolating its motion texture produces slow motion, while scaling coefficient amplitudes magnifies or minifies motion.Source: §6 · pp. 5–6

Seamless looping

  • At each denoising step, motion self-guidance encourages each pixel’s start- and end-frame position and velocity to match, enabling seamless loops from a model trained on regular non-looping clips.The supplemental comparison reports less distortion and fewer artifacts than an appearance-based looping baseline.Source: §6 · p. 6

Interactive dynamics from a single image

  • Evaluating spectral volumes at resonant frequencies approximates image-space modal bases, which drive simulated responses to user-defined forces.The modal coordinates are simulated with explicit Euler integration of decoupled mass-spring-damper equations; unlike prior methods, this interaction starts from a single picture rather than video.Source: §6 · p. 6

7. Experiments

The experiments compare the approach with single-image animation and video-generation baselines using quantitative metrics, X-t visualizations, ablations, and a user study. Results favor the full approach on video quality and temporal coherence, while revealing limitations for certain motion and content conditions.

7. Experiments

  • The evaluation uses a dataset of 3,015 natural-scene videos, with over 150K image-motion pairs, and compares trained baselines using image- and video-quality metrics.The test split contains 10% of the videos; video evaluation includes FVD and DTFVD at 16- and 32-frame windows, plus sliding-window measures.Source: §7 · pp. 6–7

7.1. Quantitative results

  • The approach significantly outperforms prior single-image animation baselines on image and video synthesis quality, with lower FVD and DT-FVD distances and no sliding-window quality degradation over time.Figure 6 reports sliding-window FID at 30 frames and DTFVD at 16 frames.Source: Figure 6 · p. 7, §7.1 · p. 8

7.2. Qualitative results

  • In X-t comparisons, generated dynamics more closely resemble real reference motion than competing methods; some baselines show unrealistic appearance or motion, while others produce over-smooth or non-oscillatory motion.Figure 5 places the ground-truth slices before baseline outputs and the approach’s output.Source: Figure 5 · p. 7, §7.2 · p. 8

7.3. Ablation study

  • All tested simpler or alternative configurations perform worse than the full model, and gains from adding frequency bands become marginal above 16.The ablations vary frequency count, normalization, frequency coordination, spectral-volume prediction, and rendering.Source: Table 2 · p. 7, §7.3 · p. 8

7.4. Comparing to large video models

  • Users preferred the approach over AnimateDiff, ModelScope, and Gen-2 in 80.9% of comparisons on 30 randomly selected test videos.The compared large video models predict video volumes directly; the reported figure examples show input adherence problems or drift and distortion over time.Source: §7.4 · p. 8, Figure 7 · p. 8

8. Discussion and conclusion

The paper presents spectral volumes as a learned image-space prior for natural oscillatory motion and uses them to animate still images. It reports photo-realistic animations that outperform prior baselines and support looping and interactive applications, while noting limitations tied to frequency coverage and trajectory quality.

Discussion and conclusion

  • Predicting only lower spectral frequencies can miss nonoscillating motion or high-frequency vibration, while thin objects, large displacements, and extensive new content can degrade rendering.The paper suggests learned motion bases as a possible way to address the frequency limitation.Source: §8 · p. 8
Loading 2309.07906v3…