PaperlayerView exact revision on arXiv

Source-linked AI summary

iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

Andreas Blattmann, Timo Milbich, Michael Dorkenwald, Björn Ommer

arXiv:2107.02790v2cs.CV

TL;DR

Predicting how a static scene responds to a local poke requires modeling both the controlled motion and its plausible effects on the rest of an object. iPOKE learns an invertible mapping between kinematics and video from local control, and reports stronger video quality and sample diversity than competing approaches. Its control is sparse, and the method is intended to generalize across object classes and instances.

  • Problem

    Given a static image and a local poke, a system should synthesize multiple plausible future dynamics without requiring dense motion guidance or poke-force annotations.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    iPOKE learns an invertible mapping between a video and residual object kinematics conditioned on the initial frame and local poke, enabling kinematics sampling and video synthesis.

    Source: §3.1 · pp. 3–4, §3 · p. 3

  • Results

    iPOKE outperforms all competing approaches by large margins in video quality and sample diversity, while methods with comparable FVD fail to generate diverse samples and vice versa.

    Source: §4.3 · p. 8

  • Takeaways & Limitations

    The model is presented for local control of object motion across arbitrary categories, with kinematics transferable to unseen instances.

    Source: §5 · p. 8

    The conditioning is limited to a single local poke: a shift at one image location between the start and end frames.

    Source: §3 · p. 3

Abstract

from arXiv · show

How would a static scene react to a local poke? What are the effects on other parts of an object if you could locally push it? There will be distinctive movement, despite evident variations caused by the stochastic nature of our world. These outcomes are governed by the characteristic kinematics of objects that dictate their overall motion caused by a local interaction. Conversely, the movement of an object provides crucial information about its underlying distinctive kinematics and the interdependencies between its parts. This two-way relation motivates learning a bijective mapping between object kinematics and plausible future image sequences. Therefore, we propose iPOKE -- invertible Prediction of Object Kinematics -- that, conditioned on an initial frame and a local poke, allows to sample object kinematics and establishes a one-to-one correspondence to the corresponding plausible videos, thereby providing a controlled stochastic video synthesis. In contrast to previous works, we do not generate arbitrary realistic videos, but provide efficient control of movements, while still capturing the stochastic nature of our environment and the diversity of plausible outcomes it entails. Moreover, our approach can transfer kinematics onto novel object instances and is not confined to particular object classes. Our project page is available at https://bit.ly/3dJN4Lf.

1. Introduction

iPOKE targets stochastic video synthesis that responds to sparse local control, predicting how a poke affects an object’s other parts. It learns object kinematics from video and aims to produce diverse, plausible motion across object classes and instances.

1. Introduction

  • The task is to infer multiple plausible future videos from a static image and a single-pixel displacement, without poke-force annotations in training.The learned kinematics should generalize to similar unseen objects rather than remain instance-specific.Source: §1 · p. 1
  • iPOKE uses a local mouse drag to control part of an object while synthesizing plausible holistic motion and diverse kinematic outcomes.Its invertible mapping links object kinematics with video sequences; the framework is intended for arbitrary object classes and unseen instances.Source: Figure 1 · p. 1, §1 · p. 2
  • The approach occupies a sparsely controlled middle ground between uncontrolled prediction and methods requiring dense per-pixel guidance or a target frame.Source: §1 · p. 2
  • Experiments cover four datasets with complex, articulated objects, including humans and plants, and compare against stochastic and controlled video-prediction methods.The introduction reports that these comparisons demonstrate plausible, diverse articulations inferred from local control.Source: §1 · p. 2

2. Related Work

Prior video synthesis work includes deterministic or stochastic prediction, motion transfer, and interpolation, while controllable methods impose varying restrictions on motion representation or guidance. iPOKE is positioned against these limitations with learned, stochastic kinematics for local control.

2. Related Work

  • Video synthesis work commonly addresses future prediction, with methods producing either a deterministic sequence or a distribution over likely futures.Other approaches transfer predefined holistic motion or interpolate between starting and ending frames; the literature also includes several model architectures.Source: §2 · p. 2
  • Earlier controllable methods use fixed linear models limited to oscillation, or parametric representations that restrict object categories and control granularity.The paper contrasts these restrictions with learned natural kinematics and fine-grained local manipulation.Source: §2 · p. 2
  • Optical-flow-controlled approaches either lack temporally coherent motion and require per-frame guidance, or omit ambiguity in uncontrolled object parts.The paper describes as warping-based and [6] as using a hierarchical dynamics model.Source: §2 · p. 2
  • Invertible neural networks are bijective functions used to transform probability distributions; iPOKE uses normalizing flows to learn residual object kinematics.Source: §2 · p. 2

3. Approach

iPOKE synthesizes plausible videos from an initial frame and sparse local motion controls by modeling the remaining object kinematics as residual information. Its conditional invertible mapping supports stochastic sampling, while a compact video representation and simulated training pokes address computational cost and data requirements.

3. Approach

  • A local poke specifies motion at one image location, leaving the model to represent how the remaining object parts move under sparse control.The poke is a 4D signal comprising a location and displacement; training can use up to five simultaneous pokes.Source: §3 · p. 3

3.1. Invertible Controlled Video Synthesis

  • A conditional bijection maps residual kinematics to videos and inverts videos to recover their residual kinematics.The residual captures video information not present in the initial frame and control.Source: §3.1 · p. 3
  • Minimizing KL divergence to an independent residual prior encourages residuals independent of the frame and control, while enabling sampling for stochastic synthesis.The authors use a standard normal prior and state that this objective avoids the cVAE trade-off between synthesis quality and capturing variations.Source: §3.1 · p. 4

3.2. Architecture for Tractably Learning τθ

  • A pre-trained sequence autoencoder supplies a compact video encoding for the invertible model, reducing the dimensionality that would otherwise make cINN training infeasible.The autoencoder uses a 3D-ResNet encoder, a GRU, and an image decoder; it is trained to reconstruct videos with static and temporal discriminators.Source: §3.2 · p. 4
  • A fully convolutional cINN incorporates the poke’s spatial location, while passing portions of each block’s output forward reduces memory demands and helps avoid vanishing gradients.The control is represented as a two-channel spatial map, and the model uses masked convolutions.Source: §3.2 · p. 5

3.3. Automatic Simulation of User Control

  • To avoid collecting control-video pairs for every target category, the authors simulate training pokes from optical flow in available videos.They sample foreground motion locations and train with a variable number of one to five pokes; inference uses mouse drags rather than optical flow estimates.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate iPOKE on diverse human and plant datasets, examining controlled synthesis, kinematics transfer, learned part interactions, and quantitative performance. Results indicate that it combines video quality and sample diversity, supports targeted control and adjustable future uncertainty, and transfers motion between object instances.

4.1. Datasets

  • Across PP, iPER, Tai-Chi-HD, and Human3.6m, the evaluation covers plants and varied human motion, including real-world background and camera variation in Tai-Chi-HD.The four datasets contain different object and motion settings, with training and test splits described for each.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2. Qualitative Evaluation

  • Transferred motion follows the source sequence rather than its appearance, while random residual samples produce different motion except at the controlled body part.The examples support a residual representation of kinematics distinct from object appearance and user control.Source: §4.2 · p. 6
  • Pixel-wise motion correlations show that human body parts connected to a poked location move together, while poking individual plant leaves usually has little effect on the rest of the plant.The analysis estimates correlations from optical-flow shifts across 1,000 randomly sampled controls at a fixed location.Source: §4.2 · p. 6, Figure 7 · p. 6

4.3. Quantitative Evaluation

  • iPOKE outperforms stochastic video prediction methods in video quality and diversity across all considered datasets, and beats sparse-control baselines in FVD and LPIPS while matching their keypoint-based variant in LPIPS.The authors report significantly better temporal and visual consistency than sparse-control baselines, with comparable LPIPS to Hao et al. using ground-truth keypoint trajectories.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • Increasing the number of local pokes reduces prediction errors and between-sample variance, giving users a way to adjust future ambiguity.The study measures prediction error and standard deviation across 50 samples on PP and iPER.Source: §4.3 · p. 8, Figure 9 · p. 8
  • The cINN-based model performs better than its cVAE counterpart on PP and Human3.6m, consistent with the authors’ reported variability-versus-quality trade-off in cVAE.The ablation uses a variational counterpart with the same video-autoencoding framework.Source: Table 3 · p. 8

5. Conclusion

iPOKE learns an invertible mapping between generated videos and object kinematics, enabling local manipulation and diverse kinematic samples across object categories. Supplementary material provides additional visualizations, human-input demonstrations, comparisons, and implementation and evaluation details.

5. Conclusion

  • iPOKE uses an invertible mapping between generated video sequences and a learned kinematics representation, with probabilistic learning to sample diverse realizations under ambiguous local manipulation.The conclusion describes control through simple mouse drags and states that the approach applies to arbitrary object categories.Source: §5 · p. 8

Preliminaries

  • Supplementary visualizations cover controlled stochastic synthesis across datasets, kinematics transfer, control sensitivity, human-user inputs, GUI demonstrations, and qualitative baseline comparisons.The supplementary materials also include the objective derivation, model and baseline implementation details, and expanded metric descriptions.Source: Preliminaries · pp. 9–10, A. Additional Visualizations · p. 10

A.1. Controllable Stochastic Video Synthesis

Across plants and human-motion datasets, iPOKE generates plausible responses to local controls while varying motion in other object parts. It also transfers source-sequence kinematics to still images of people with different appearances.

A.1. Controllable Stochastic Video Synthesis

  • For five varied plants, iPOKE reaches the controlled part’s intended position while generating distinct plausible motion in the remaining parts.The examples cover plants with substantially varying shapes and appearances.Source: §A.1 · p. 10
  • On iPER test-set people unseen during training, iPOKE generalizes appearance and moves poked body parts toward their targets while retaining motion variance.Source: §A.1 · p. 10
  • On Human3.6m, iPOKE generates realistic-looking walking and sitting motion across small and large controls, but large motion can alter appearance.The authors note that the training set contains only five distinct appearances, making appearance generalization difficult.Source: §A.1 · p. 10
  • On Tai-Chi, iPOKE generates realistic-looking sequences despite diverse backgrounds, although the dataset’s human motion is less large and diverse than iPER and Human3.6m.Source: §A.1 · p. 10

A.2. Kinematics Transfer

  • Kinematics transfer preserves subtle source-motion details on a target person of arbitrary appearance, while random residual samples produce different motion except at the controlled location.The authors present this as empirical evidence that residual kinematics are independent of the initial frame and control.Source: §A.2 · p. 11

A.3. Control Sensitivity

iPOKE’s responses follow both simulated and human-defined controls, including fine-grained plant pokes and background inputs. Its GUI supports successive interactions without heavy artifacts or implausible kinematics.

A.3. Control Sensitivity

  • With the same initial image and poke location, varying the shift vector makes the poked body part approach its corresponding target, indicating control sensitivity.Source: §A.3 · p. 11

B. Control Inputs of Human Users

B.1. PokingPlants

B.2. iPER

  • For human-defined iPER controls, poked body parts reach their targets; a background poke instead yields a still sequence with the person unaffected.The training procedure explicitly generates background controls to teach the model to ignore them.Source: B. Control Inputs of Human Users · p. 11, §B.2 · p. 12

B.3. Graphical User Interface

  • In uncut GUI demonstrations, iPOKE processes multiple human control inputs successively without heavy artifacts or implausible kinematics.Source: §B.3 · p. 12

C. Qualitative Comparison with Competing Methods

The qualitative comparisons contrast iPOKE’s controlled, diverse motion with competing methods’ limitations in coherent motion or sample diversity. The comparisons cover controllable and stochastic video synthesis on iPER and PokingPlants.

C. Qualitative Comparison with Competing Methods

C.1. Controllable Video Synthesis

  • On iPER and PokingPlants, iPOKE produces fluent, realistic object kinematics, while Hao et al.’s frame warping misses complex structured motion on iPER and yields blurry outputs on PokingPlants.The authors attribute the lower FVD scores of Hao et al. to its difficulty capturing complex structured motion on iPER.Source: §C.1 · p. 12, C. Qualitative Comparison with… · p. 12

C.2. Stochastic Video Synthesis

  • Compared with IVRNN, iPOKE’s samples follow user interactions while varying motion in other object parts; IVRNN samples show little visible variation and blur fine-grained plant motion.IVRNN was selected as the best competing method on these datasets when video quality and diversity were considered together.Source: §C.2 · p. 12, C. Qualitative Comparison with… · p. 12

D. Derivation and Implications of Learning Objective

The derivation connects minimizing a KL divergence to independence between residual kinematics and conditioning. The residual is intended to retain information about the video not present in the initial frame and control.

D.1. Independence of r and (x0, c)

  • The intended role of r is to contain information about X that is absent from the initial frame x0 and control c.Source: §D.1 · p. 12
  • The derivation uses KL-divergence properties and the unknown true marginal distribution p(r).Source: §D.1 · p. 13
  • The KL divergence between p(r|x0, c) and an independent prior q(r) upper-bounds mutual information between r and (x0, c), so its minimizer forces independence.Source: §D.1 · p. 13

D.2. Derivation of Objective Function

The derivation uses a KL divergence to obtain an objective for learning a probabilistic generative model whose latent residual is independent of the conditioning. It also contrasts this objective with the trade-off in cVAE-based models.

D.2. Derivation of Objective Function

  • The derivation begins with the KL divergence, changes the integration variable from r to X, and substitutes earlier equations before selecting a standard normal q(r).The final objective takes an expectation over (x0, c).Source: §D.2 · p. 13
  • Minimizing KL[p(r|x0, c)∥q(r)] yields a probabilistic generative model that captures the data distribution and induces independence between r and (x0, c).The derivation chooses q(r) = N(0, I) and takes an expectation over (x0, c).Source: §D.2 · pp. 13–14
  • Unlike cVAE-based models, the objective induces both data-distribution modeling and latent-conditioning independence without balancing two loss terms.The cVAE trade-off is between reconstruction quality and independence of the latent variable from the conditioning.Source: §D.2 · pp. 13–14

E. Implementation Details of iPOKE

The implementation combines a pretrained video autoencoder with a conditional invertible model, and adds training that teaches the model to ignore background pokes.

E. Implementation Details of iPOKE

E.1. Video Autoencoding Framework

  • The video autoencoder encodes videos with a 3D-ResNet, predicts subsequent latent states with a GRU, and decodes them into frames; it is pretrained before the invertible model.Its reconstruction loss combines pixel-space L1 and perceptual losses, alongside spatial and temporal discriminator objectives.Source: §E.1 · p. 14
  • The autoencoder uses an 8 × 8 latent grid, with 32 or 64 channels depending on the dataset, and is trained on 10-frame sequences at 64 × 64 or 128 × 128 resolution.The channel count is 32 for iPER and Tai-Chi, and 64 for PP and Human3.6m.Source: §E.1 · p. 15

E.2. Conditional Invertible Neural Network

  • The conditional invertible model uses pretrained encoders for the source frame and control, then a convolutional normalizing flow built from flow blocks and modified Glow blocks.Masked convolutions provide tractable Jacobians and efficient sampling; the flow uses 15 blocks.Source: §E.2 · p. 15
  • The conditional invertible model and its conditioning encoders are trained with Adam; the flow uses 15 blocks, and its learning rate is annealed from zero to 10^-3 before decreasing.The learning-rate schedule warms up for 500 steps; the implementation modifies Glow blocks to use shuffling layers instead of invertible 1 × 1 convolutions.Source: §E.2 · p. 15

E.3. Further Training Details of iPOKE

  • Background-poke training samples artificial controls on background pixels and trains the model to reconstruct a repeated source frame, encouraging it to ignore controls unrelated to the object.Foreground control locations are sampled from locations whose optical-flow magnitude exceeds the flow-map mean.Source: §E.3 · p. 16

F. Implementation Details of Baselines

The baseline implementations cover stochastic video prediction, controllable synthesis, and a variational counterpart used in the ablation study.

Video Prediction Models

  • Stochastic video prediction baselines [40] [10] [24] use two context frames to predict 10-frame, 64 × 64 sequences; SAVP produced no visual diversity with the tested settings.Models were trained from scratch except SRVP on Human3.6m, for which pretrained weights were available.Source: F. Implementation Details of Baselines · p. 16

Controlled Video Synthesis Models

  • Controllable synthesis baselines [6] generate 128 × 128 videos; the models were trained on the evaluated datasets, while [6] used available pretrained models.For, the procedure samples five flow vectors, tracks them into trajectories, and warps the source frame; iPER ground-truth-keypoint trajectories use keypoint shifts.Source: F. Implementation Details of Baselines · p. 16

Variational Counterpart of iPOKE

  • The variational counterpart uses the autoencoder components but regularizes its video encoding toward a standard normal prior and initializes the GRU with video, source-frame, and control features.Its best reported KL weight is λKL = 0.1, with KL annealing for five epochs to avoid posterior collapse.Source: F. Implementation Details of Baselines · pp. 16–17

G. Evaluation Details

Evaluation measures motion consistency, synthesis quality, and diversity using generated video samples and corresponding ground-truth data.

G. Evaluation Details

  • Motion consistency uses FVD by comparing I3D feature distributions from 1,000 generated sequences and 1,000 ground-truth videos.The I3D model was pretrained on Kinetics, and the last conditioning frame is concatenated with each generated sequence.Source: G. Evaluation Details · p. 17
  • Synthesis-quality metrics compare each generated frame with its corresponding ground-truth frame, then average the T frame scores for each sequence.The reported metrics use 1,000 predicted sequences and corresponding ground-truth videos.Source: G. Evaluation Details · p. 17
  • Motion diversity is computed between five realizations per video using LPIPS and pixel-space MSE.The authors report no change in numbers when using more realizations.Source: G. Evaluation Details · p. 17
Loading 2107.02790v2…