Source-linked AI summary
Understanding Object Dynamics for Interactive Image-to-Video Synthesis
Andreas Blattmann, Timo Milbich, Michael Dorkenwald, Björn Ommer
TL;DR
The paper addresses the challenge of controlling articulated object motion through local interactions rather than predicting motion holistically. It learns a video-based generative model for pixel-level pokes, and experiments assess fine-grained control and the plausibility of its generated dynamics.
Problem
Method
A hierarchical recurrent generative model learns object dynamics from video sequences and predicts the response to a pixel-level interaction without ground-truth interaction labels or object-shape assumptions.
Source: §1 · p. 2Results
Experiments on four datasets of humans and plants assess fine-grained interaction and the visual and temporal plausibility of generated dynamics against state-of-the-art video prediction methods.
Source: §1 · p. 2Takeaways & Limitations
The model produces natural responses to localized interactions and can represent dynamics across object parts and unseen object instances.
Source: §4.2 · p. 7, §5 · p. 8
Abstract
from arXiv · showhide
What would be the effect of locally poking a static scene? We present an approach that learns naturally-looking global articulations caused by a local manipulation at a pixel level. Training requires only videos of moving objects but no information of the underlying manipulation of the physical scene. Our generative model learns to infer natural object dynamics as a response to user interaction and learns about the interrelations between different object body regions. Given a static image of an object and a local poking of a pixel, the approach then predicts how the object would deform over time. In contrast to existing work on video prediction, we do not synthesize arbitrary realistic videos but enable local interactive control of the deformation. Our model is not restricted to particular object categories and can transfer dynamics onto novel unseen object instances. Extensive experiments on diverse objects demonstrate the effectiveness of our approach compared to common video prediction frameworks. Project page is available at https://bit.ly/3cxfA2L .
1. Introduction
The paper aims to predict natural object motion from a local pixel-level interaction, learning from video without ground-truth interactions or object-shape assumptions. It evaluates this control and the plausibility of generated dynamics on diverse articulated objects.
1. Introduction
- The motivating analogy is learning how the scale of a poke affects different parts and generalizing those dynamics to novel objects.Children learn that subtle touches may affect individual leaves, while stronger interactions may affect larger constituents.Source: §1 · p. 1
- Prior work on object dynamics focuses on extrapolating observed motion or coarse controls, leaving direct local interaction and prediction of its effects on other parts as a novel challenge.Such prediction requires modeling natural interrelations between object parts.Source: §1 · pp. 1–2
- A generative model predicts natural video responses to pixel-level interactions, learning fine-grained object dynamics from video sequences without ground-truth interactions or object assumptions.The model uses a hierarchical recurrent architecture informed by physics intuitions.Source: §1 · p. 2
- Experiments on four video datasets of humans and plants assess fine-grained interaction and compare generated dynamics with state-of-the-art video prediction methods.The comparison evaluates visual and temporal quality.Source: §1 · p. 2
2. Related Work
Prior video prediction methods generally model motion holistically, while existing controllable approaches impose representation, annotation, or interaction constraints. The paper positions its latent dynamics model as enabling complex articulated motion from a single-pixel interaction.
2. Related Work
- Video prediction methods use past frames to model motion holistically, including latent-space approaches that still represent dynamics holistically.Warping methods can produce high-quality short-term predictions but struggle with longer sequences.Source: §2 · p. 2
- A spring-mass approach for rigid objects is limited to small deviations around a rest state and cannot model complex dynamics.The passage attributes this boundary to the method's restrictive assumptions.Source: §2 · p. 2
- The proposed latent dynamics model targets complex articulated motion controlled by an interaction at a single pixel, avoiding constraints that limit earlier approaches.Earlier methods may rely on keypoints, action labels, motion sequences, holistic trajectories, or multiple flow vectors.Source: §2 · p. 2
3. Interactive Image-to-Video Synthesis
The method turns a pixel-level poke into a trajectory of object states, then decodes those states into frames. A hierarchical recurrent model captures dynamics at multiple spatial scales, while training infers poke supervision from video and separates object-space pretraining from dynamics learning.
3. Interactive Image-to-Video Synthesis
- A local poke shifts one image location, and the model predicts its physically plausible effects across the depicted object over time.It encodes the initial image as an object state and the poke’s target location as a latent interaction; a dynamics model predicts states that a decoder renders as frames.Source: §3 · pp. 2–3
3.1. A Hierarchical Model for Object Dynamics
- A hierarchy of recurrent predictors models higher-order dynamics, with each level operating on a different spatial feature scale.The hierarchy extends first-order recurrent updates to capture complex, subtle motion, and a UNet decoder receives latent information at multiple scales to retain image detail.Source: §3.1 · pp. 3–4
- The hierarchical model jointly adds temporal and spatial detail to capture interaction-driven dynamics and display them in image space.Its predictors operate across feature levels, and the decoder renders the predicted multiscale states as frames.Source: §3.1 · p. 4
3.2. Learning Dynamics from Poking
- Training pokes are simulated by sampling a pixel displacement from optical flow between a video’s initial and final frames.The observed video sequence supplies the natural response used to supervise the interaction-conditioned generator.Source: §3.2 · pp. 4–5
- The method first pretrains the encoder and decoder to reconstruct frames, then trains the dynamics predictor in that fixed object space while fine-tuning the decoder.This staged procedure addresses the difficulty of learning dynamics when the object-state space changes during end-to-end training.Source: §3.2 · p. 5
4. Experiments
The experiments assess interactive image-to-video synthesis through implementation and dataset details, qualitative analysis of generated dynamics and learned object structure, quantitative comparisons, ablations, and additional experiments.
4.1. Implementation Details and Datasets
- The model was evaluated for object-dynamics understanding on four datasets spanning highly articulated humans and plants; Poking-Plants varies in shape and texture, while Tai-Chi-HD has varied backgrounds and camera movements.Main-paper experiments generally generate 10-frame sequences at 128 × 128, unless otherwise specified; Tai-Chi-HD videos are 256 × 256.Source: §4.1 · p. 6
4.2. Qualitative Analysis
- On plants and humans, pokes produce plausible motion in both the targeted region and other object parts, including for unseen people and varied poke directions and locations.Motion correlations reflect object structure: trunk pokes correlate nearby plant pixels, while leaf pokes correlate mainly neighboring pixels.Source: §4.2 · p. 7
4.3. Quantitative Evaluation
- Across datasets, the model achieves lower FVD than video-prediction methods and performs significantly better than Hao et al. for controlled synthesis, especially on FVD.The controlled-synthesis comparison also reports better LPIPS scores across datasets.Source: Table 1 · p. 6, Table 2 · p. 7, §4.3 · p. 8
4.4. Ablation Studies
- The hierarchical predictor has the largest ablation impact; removing trajectory loss or object-encoder pretraining also worsens FVD scores.The authors attribute the hierarchy’s benefit to modeling higher-order dynamics across spatial feature scales.Source: §4.4 · p. 8, Table 3 · p. 8
4.5. Additional Experiments
- Additional experiments examine generalization to unseen plant types and alternative interpretations of the poke.These experiments are reported in Appendix B.Source: §4.5 · p. 8
5. Conclusion
The model captures articulated object-part dynamics and generates natural responses to localized interactions, with training from unlabeled video and no assumptions on object shape. Supplementary visualizations illustrate its behavior across varied plants and additional settings.
5. Conclusion
- The model learns articulated dynamics by capturing relationships among object parts, enabling videos that respond naturally to localized interactions.It can be learned from unlabeled video without limiting assumptions on object shape.Source: §5 · p. 8
Preliminaries
- The supplementary material includes qualitative comparisons and additional experiments on impulse-based pokes, plant generalization, and RNN hierarchy depth.It also documents implementation and evaluation details.Source: Preliminaries · p. 9
A. Additional Visualizations
- The additional visualizations compare synthesized sequences with ground truth and, for related-method comparisons, show competing methods across datasets.The supplementary videos cover simulated and human-initiated pokes.Source: A. Additional Visualizations · p. 9
A.1. PokingPlants
- Across varied plant shapes and appearances, larger pokes produce larger, farther-reaching motion, while subtle pokes move nearby parts more finely.The examples also indicate long-term relations between object regions and separation of foreground plants from background clutter.Source: §A.1 · p. 9
A.2. iPER
Visualizations show plausible responses to localized interactions across human and outdoor scenes, including complex motion and background separation. Comparisons also show fine-motion and visual-quality differences from competing methods.
A.2. iPER
- On iPER, sparse localized simulated pokes yield realistic global human motion, including combinations of arm and leg movement, with foreground-background separation in an out-of-distribution scene.The examples include unseen actors in indoor and outdoor scenes.Source: §A.2 · p. 10
A.3. Tai-Chi-HD
- On Tai-Chi-HD, the model produces plausible global motions and interaction responses despite textured backgrounds, camera movement, and outdoor scenes.The visualizations also show object-background separation under camera movement and clutter.Source: §A.3 · p. 10
A.4. Human3.6M
- On Human3.6M, pokes control walking direction and interactions with torso, head, and arms generate complex human motion, although appearance changes after a few predicted frames.Source: §A.4 · p. 10
A.5. Qualitative Comparison
- Against SAVP [45], the model synthesizes subtle plant oscillations that the competitor struggles with; against Hao et al., it improves motion consistency and visual quality.The Hao et al. approach holistically moves the body on iPER, rather than moving distinct body parts individually.Source: §A.5 · p. 10
B. Additional Experiments
The additional experiment interprets a poke as an initial localized impulse rather than a pixel shift. It describes training and evaluates videos from this alternative model, including generalization to unseen plants.
B. Additional Experiments
- The alternative experiment treats the poke as an initial localized impulse on the object instead of a shift from poke location to target.Source: B. Additional Experiments · p. 11
- The supplementary experiment trains this impulse interpretation on PokingPlants and presents samples of the resulting synthesized videos.Source: B. Additional Experiments · p. 11
- The trained impulse model is tested for generalization to plant types not seen during training, obtained through web search.Source: B. Additional Experiments · p. 11
B.1. Training Setting
The alternative training setting maps a poke to an initial impulse and matches poke magnitude to motion magnitude observed in training videos. This trains responses in which larger-motion sequences correspond to larger impulses.
B.1. Training Setting
- This training setting normalizes estimated flow magnitudes to [0, 1], retaining direction and relative magnitude while removing the exact target-pixel shift.Source: §B.1 · p. 11
- The latent interaction is treated as an initial force, and training pairs lower-motion videos with smaller pokes and higher-motion videos with larger pokes.Source: §B.1 · p. 11
- Average motion per training sequence is estimated from flow magnitudes between consecutive frames, enabling poke magnitudes to be sampled according to sequence motion.The resulting model synthesizes object responses to an initial impulse.Source: §B.1 · p. 11
B.2. Results
The additional experiments show that poke magnitude tracks the strength of the generated motion and that the model predicts plausible dynamics for unseen plant types. The derivation relates common recurrent architectures to the two-stage Runge–Kutta method.
B.2. Results
- Small pokes produce subtle movements that return to rest, whereas larger pokes produce larger motions that do not return within the 25 predicted frames.The model is trained to reconstruct 10-frame sequences and predicts 25 frames at inference.Source: §B.2 · p. 11
B.3. Generalization to Unseen Types of Plants
- On four unseen web-found plant types, the model predicts plausible dynamics and captures relations between plant parts under different poke magnitudes.For one example, it also separates foreground from a substantially textured background.Source: §B.3 · pp. 11–12
C. Derivation of Equation (3)
- The derivation introduces a correspondence between common RNN architectures, including LSTMs and GRUs, and the two-stage Runge–Kutta method.The GRU update is compared with the Runge–Kutta equations to clarify the correspondence; the passage directs readers to Niu et al. for the exact proof.Source: §C.1 · p. 12, C. Derivation of Equation (3) · p. 12
C.2. Derivation of RNN-Hierarchy
The hierarchy recasts an N-th order evolution function as a system of first-order ODEs, then approximates those equations with recurrent predictors linked across levels.
C.2. Derivation of RNN-Hierarchy
- The table caption describes increasing hierarchy depth as improving prediction accuracy and dynamics consistency.It identifies LPIPS and PSNR as prediction-accuracy measures and FVD as a dynamics-consistency measure.Source: Table 4 · p. 12
- An N-th order evolution function is represented as a hierarchy of variables whose elements correspond to successive time derivatives of the original variable.This maps the higher-order ODE onto a system of first-order ODEs.Source: §C.2 · p. 12
- Each first-order ODE is approximated in discrete time with RK2 and solved by a GRU cell, using the next-state approximation of the preceding variable as input.The resulting hierarchy of predictors is Equation (3) in the main paper.Source: §C.2 · p. 12
D. Ablation Study: Modelled Order of ODE-Derivative
The ablation finds improved reported metrics as the RNN hierarchy models higher derivative orders; the section also specifies the compared settings and network architecture.
D. Ablation Study: Modelled Order of ODE-Derivative
- The ablation compares hierarchy depths from N = 1 to N = 3 on human videos from iPER and plant videos from PP.Each added RNN operates at a higher spatial size and receives a skip connection between the encoder and generator at that level.Source: D. Ablation Study: Modelled Order of… · p. 12
- In Tab. 4, the proposed model’s deeper RNN hierarchy obtains lower FVD than baselines; among baselines, N = 1 performs worse than N = 2, while increasing N lowers LPIPS and PSNR.Source: D. Ablation Study: Modelled Order of… · pp. 12–13
E.1. Network Details
- For 64 × 64 and 128 × 128 videos, the image-to-sequence model uses three and four Conv-GRU cells, respectively, with encoder-matched hidden dimensions.The encoders use strided convolutions and a final ResNet block; the decoder uses ResNet blocks with transposed-convolution upsampling.Source: §E.1 · p. 13
E.2. Training Details
Training samples poke locations using optical-flow magnitude and adds artificial background pokes to teach foreground-background separation; the section also details discriminator and evaluation settings.
E.2. Training Details
- Training samples most poke locations from pixels whose optical-flow magnitude exceeds the flow-map mean, while one tenth are sampled from the background.Artificial background pokes are trained to reconstruct a still sequence by repeating the source image.Source: §E.2 · p. 13
- The discriminators use hinge loss, with gradient penalty for the temporal discriminator and a feature-matching loss weighted by 2.The spatial discriminator is trained on 16 sampled images from predicted and ground-truth sequences.Source: §E.2 · p. 13
- Video-prediction baselines are compared at 64 × 64 resolution using two context frames to predict 10 frames, while the controlled-synthesis comparison uses 128 × 128 videos.For the SAVP comparison on PokingPlants, predictions use 64 × 64 resolution for a fair comparison.Source: §E.2 · p. 13
F. Evaluation Details
Evaluation compares generated and real video distributions in pretrained I3D feature space for FVD, and averages framewise accuracy scores across each sequence. Supplementary examples visualize pokes and target locations across four datasets.
F. Evaluation Details
- FVD compares I3D-feature distributions from 1,000 generated sequences and 1,000 random ground-truth videos of the same length.The I3D model is pretrained on Kinetics.Source: F. Evaluation Details · p. 14
- Accuracy metrics compare each generated frame with its corresponding ground-truth frame, then average the T frame scores per sequence.Source: F. Evaluation Details · p. 14
- Additional examples span PokingPlants, iPER, Tai-Chi-HD, and Human3.6M; rows show frames, columns show sequences, and red marks show the poke and target.Source: Figure 8 · p. 16, Figure 10 · p. 17, Figure 14 · p. 19