PaperlayerView exact revision on arXiv

Source-linked AI summary

What If : Understanding Motion Through Sparse Interactions

Stefan Andreas Baumann, Nick Stracke, Timy Phan, Björn Ommer

arXiv:2510.12777v1cs.CV

TL;DR

Real-world scene motion can have multiple plausible outcomes, while dense deterministic predictions commit to a single trajectory. FPT predicts local motion distributions conditioned on sparse pokes, and its evaluations report versatility across established tasks alongside directly interpretable motion distributions.

  • Problem

    Scene dynamics can have multiple plausible outcomes, which dense deterministic predictions represent by committing to a single trajectory.

    Source: §1 · pp. 1–2

  • Method

    FPT directly predicts interpretable distributions of local motion conditioned on sparse interactions called pokes.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Evaluations demonstrate versatility across domains and competitive performance on established tasks including dense motion generation on faces and articulated objects.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    Directly interpretable motion distributions support usable predictions in interactive environments, and moving-part segmentation emerges from the method’s design.

    Source: §5 · p. 9

Abstract

from arXiv · show

Understanding the dynamics of a physical scene involves reasoning about the diverse ways it can potentially change, especially as a result of local interactions. We present the Flow Poke Transformer (FPT), a novel framework for directly predicting the distribution of local motion, conditioned on sparse interactions termed "pokes". Unlike traditional methods that typically only enable dense sampling of a single realization of scene dynamics, FPT provides an interpretable directly accessible representation of multi-modal scene motion, its dependency on physical interactions and the inherent uncertainties of scene dynamics. We also evaluate our model on several downstream tasks to enable comparisons with prior methods and highlight the flexibility of our approach. On dense face motion generation, our generic pre-trained model surpasses specialized baselines. FPT can be fine-tuned in strongly out-of-distribution tasks such as synthetic datasets to enable significant improvements over in-domain methods in articulated object motion estimation. Additionally, predicting explicit motion distributions directly enables our method to achieve competitive performance on tasks like moving part segmentation from pokes which further demonstrates the versatility of our FPT. Code and models are publicly available at https://compvis.github.io/flow-poke-transformer.

1. Introduction

Scene motion is uncertain and often multimodal, while dense deterministic predictions commit to a single possible future. FPT instead predicts distributions of local motion conditioned on sparse pokes, supporting direct interpretation of possible outcomes and their uncertainty.

1. Introduction

  • Dense deterministic predictions commit to one trajectory, although real-world motion can have multiple plausible outcomes.The paper describes single-trajectory prediction as potentially biased toward one future and less useful than considering multiple outcomes in planning, robotics, and autonomous systems.Source: §1 · pp. 1–2
  • FPT predicts distributions of local motion conditioned on sparse pokes rather than committing to one dense video outcome.Repeated or targeted pokes let observers explore how interactions change the possible motion outcomes.Source: §1 · p. 2
  • Directly accessible motion distributions expose multimodality and uncertainty for interpreting how physical interactions affect a scene.The paper frames such predictions as useful for identifying modes and quantifying uncertainty directly.Source: §1 · p. 2
  • The method supports interactive simulation, moving-part segmentation, and dense motion prediction from a single frame.Unlike optical-flow estimation and tracking, it predicts possible future motion without a future frame.Source: §1 · p. 2
  • Sparse motion modeling is presented as efficient, with predictions in 25ms and throughput above 160k parallel predictions per second on a modern GPU.The paper also reports learning general motion understanding from unstructured web videos.Source: §1 · p. 2

2. Related Work

Prior work predicts motion through rendered videos, dense flow, or specialized physical representations, which can make motion understanding less accessible or limit the motion space. FPT is positioned as directly modeling local motion distributions and their uncertainty.

Motion-based Editing

  • Image- and video-editing methods use pokes or drags to guide image warping or video generation, prioritizing adherence to guidance over physical realism.The cited approaches include motion-supervised warping, predicted dense optical flow, and models extended with ControlNets or LoRAs.Source: §2 · pp. 2–3

Motion Generation

  • Motion-generation methods range from dense flow or RGB video synthesis to discrete flow bins and single continuous-flow predictions.Motion-I2V permits sparse movement conditioning, while DragNUWA predicts dense RGB video and requires another model to estimate motion.Source: §2 · p. 3

Physical Scene Understanding

  • Poke-based and articulated-part methods predict RGB outcomes, making motion representations harder to access and not providing an underlying uncertainty distribution.The paper contrasts these limitations with its direct modeling of motion distributions.Source: §2 · p. 3

Physical Representations

  • Fourier-based physical motion methods model oscillatory dynamics but are described as less flexible for the broader, often non-oscillatory motion space.The cited extensions apply these representations to 3D scenes.Source: §2 · p. 3

Generative Models in Computer Vision

  • Generative models can represent conditional distributions, but sampling-focused GAN and diffusion methods offer limited direct access to distribution structure.The paper builds on GIVT’s directly parameterized Gaussian mixture distributions and extends them to non-diagonal covariances for motion.Source: §2 · p. 3

3. Method

FPT models the conditional distribution of pointwise motion given an image and sparse motion pokes. Its transformer predicts accessible multimodal distributions, which the paper uses for dense motion prediction and moving-part segmentation.

3.1. Problem Setting

  • Given an image and sparse pokes, FPT predicts the movement distribution of arbitrary query points, where each flow describes displacement from the current time to a future time.The poke set specifies movements at selected image locations and conditions the predicted pointwise motion.Source: §3.1 · p. 3

3.2. Flow Poke Transformer

  • The transformer processes poke and query tokens, cross-attends to image features, and uses arbitrary-precision relative positions for sparse, off-grid motion.Queries attend to pokes and image tokens, not to other queries, allowing multiple query distributions to be evaluated in parallel.Source: §3.2 · p. 4, Figure 2 · p. 4
  • A projection head directly predicts a Gaussian mixture with full covariance matrices, making multimodal motion distributions and uncertainty accessible.The model predicts a lower-triangular factor with a positive diagonal to construct positive semi-definite covariance matrices.Source: §3.2 · p. 4
  • Training uses nested random poke sets and query-causal attention to train predictions for multiple conditioning sets efficiently.The described attention pattern reduces computation while retaining the same number of trained predictions.Source: §3.2 · pp. 4–5

3.3. Downstream Applications

  • Dense motion can be produced by parallel pointwise means or autoregressive sampling, with parallel means potentially averaging modes.Autoregressive sampling iteratively adds sampled query flows to the poke set to produce coherent dense-flow samples.Source: §3.3 · p. 5
  • Moving-part segmentation measures how conditioning on a poke changes each query point’s motion distribution using KL divergence.The divergence is zero when the two points’ movements are independent and otherwise quantifies their motion interdependence.Source: §3.3 · p. 5

4. Experiments

FPT predicts multimodal local motion distributions conditioned on sparse pokes, and its experiments assess those predictions, compare motion estimation across tasks, and test moving-part segmentation. Results show useful mode confidence and uncertainty estimates, competitive face-motion performance, and gains on articulated-object tasks after fine-tuning.

4.1. Dataset and Implementation Details

  • The experiments use WebVid clips for general pretraining, evaluate motion at 64² resolution by default, and report query-distribution inference under 25 ms on one H200.The main model uses a 220M-parameter transformer and is trained for 800k steps; parallel prediction throughput is about 160k predictions per second per image.Source: §4.1 · pp. 5–6

4.2. Evaluation of FPT’s Key Abilities

  • Conditioned on sparse pokes, FPT predicts realistic, multimodal motion distributions that reflect scene interactions and fine-grained object motion.The model can also autoregressively sample diverse, realistic global scene motion without poke conditioning.Source: §4.2 · p. 6, Figure 3 · p. 6, Figure 6 · p. 7
  • FPT predicts diverse motion modes whose confidences become more concentrated as poke count increases, and modes with higher confidence are more accurate by PCK.The mode closest to ground truth has above-average confidence; prediction error also correlates with uncertainty, with Pearson ρ=0.66 for sampling, ρ=0.64 for the mean, and ρ=0.62 for the most confident mode.Source: §4.2 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7

4.3. Comparisons on Motion Prediction

  • On face motion, FPT has lower EPE than Motion-I2V and InstantDrag, but slightly worse PCK than InstantDrag at low poke counts; its PCK catches up with more pokes.Using predicted motion for image warping, FPT consistently outperforms both methods by LPIPS distance to ground-truth images.Source: §4.3 · p. 7, Table 1 · p. 7
  • Fine-tuning on Drag-A-Move lets FPT outperform DragAPart and PuppetMaster in articulated-object motion estimation, while zero-shot FPT beats Motion-I2V but trails the in-domain methods.The evaluation is out of distribution for FPT; qualitative results also report better handling of complex multi-poke situations than extracted flow from DragAPart.Source: §4.3 · p. 8

4.4. Segmenting Moving Parts

  • FPT’s divergence-based moving-part score outperforms prior methods, including the benchmark’s DragAPart baseline; its mIoU remains 0.54 and 0.52 when the threshold is halved or doubled.Without the divergence-based score, the method achieves an mIoU of 0.415; qualitative predictions are spatially continuous and make fewer critical segmentation mistakes.Source: §4.4 · p. 8, §4.3 · p. 8

5. Conclusion

The conclusion presents FPT as a general-purpose model that represents local motion distributions conditioned on pokes, capturing multimodality and uncertainty. It reports competitive results across domains while emphasizing that standard evaluations do not fully measure its primary strength: directly interpretable motion distributions for interactive environments.

5. Conclusion

  • FPT models multimodal, stochastic scene dynamics as interpretable local-motion distributions conditioned on targeted pokes.The authors contrast direct distribution prediction with approaches that model only particular possible outcomes.Source: §5 · p. 8
  • FPT performs competitively on dense face-motion generation and articulated-object tasks despite being designed and trained for sparse, general-purpose motion understanding.The conclusion also identifies moving-part segmentation as a capability that emerges from the method’s design.Source: §5 · pp. 8–9
  • The authors caution that established task evaluations do not fully reflect FPT’s primary strength: interpretable, usable motion-distribution predictions for interactive environments.They frame FPT as a versatile, general-purpose motion model and a foundation for future approaches to motion understanding.Source: §5 · p. 9

A. Implementation Details

The implementation samples sparse flow pokes, embeds them with image features, and uses query tokens to predict local flow distributions. The model uses query-causal attention and dataset-specific training and camera-motion settings.

A. Implementation Details

  • The WebVid model trains for 800k steps at 5.0×10−5 with AdamW and 5k-step linear warmup; the second trains for 1M steps with 100k warmup and cosine decay.Source: A. Implementation Details · p. 12
  • Flow pokes are embedded with local image features, mapped into tokens, and combined with query tokens marking where flow distributions are predicted.Poke flows are tanh-mapped to [−1, 1] and sinusoidally embedded; query tokens combine a learned token with positional encoding.Source: A. Implementation Details · p. 12
  • The 12-block, width-768 transformer uses query-causal self-attention, image-feature cross-attention, 2D Axial RoPE, and camera-static conditioning.An output MLP projects transformer features before GMM parameter prediction because direct prediction sometimes produced unreasonable distributions and poor performance.Source: A. Implementation Details · p. 12
  • Camera-static status is determined heuristically from scene motion, using 40% of the frame moving at most 3px for the first training and a 1%-of-frame-side threshold for the second.The first threshold is evaluated at 448² training resolution.Source: A. Implementation Details · p. 12
  • Queries attend to pokes but not to one another, allowing parallel inference for multiple queries belonging to the same poke set.Poke tokens precede query tokens; with one query per set, each query attends to one more poke than its predecessor.Source: A. Implementation Details · p. 12

B. Ablations

The ablations examine pretrained vision-encoder training and motion-distribution parameterization. They show trade-offs between instance-specific motion, low-poke performance, and preserving multimodal predictions.

B. Ablations

  • The extended version of Tab. 1 reports quantitative hyperparameter ablations used to motivate choices in the main paper; all comparison models follow the original training recipe and stop at 250k steps due to compute constraints.Source: B. Ablations · p. 13
  • Jointly training the pretrained vision encoder improves performance over freezing it at low poke counts, where instance segmentation capabilities remain relevant.The frozen-encoder example transfers the man’s hand movement to the woman’s hand; the jointly trained model separates their movements.Source: B. Ablations · p. 13
  • Pretrained vision-encoder initialization boosts performance in this setting, though the authors hypothesize that initialized and uninitialized versions may converge with very long training.Freezing a DINOv2-initialized feature extractor reduces instance-segmentation capability relative to jointly training it.Source: B. Ablations · p. 13
  • Too many GMM components reduce performance with few pokes, while a single component performs better there but prevents multimodal prediction.Diagonal covariance matrices slightly reduce average performance and prevent angled distributions that represent directional uncertainty.Source: B. Ablations · p. 13

C. Extension to 3D Motion

Although the main evaluation reasons about motion in the image plane, the architecture is extended to predict 3D motion. The demonstrated version produces plausible out-of-plane movement, including without pokes.

C. Extension to 3D Motion

  • The paper’s evaluation is 2D, but the architecture is described as extendable to higher dimensions.The qualitative 3D example comes from continued training of a 2D FPT checkpoint.Source: C. Extension to 3D Motion · p. 13
  • A 3D FPT variant fine-tuned on 3D track data produces plausible out-of-plane motion, while its in-plane motion prediction resembles that of a 2D FPT model, in qualitative samples.Source: C. Extension to 3D Motion · p. 13, Figure C · p. 14
  • The 3D model predicts movement toward and away from the camera without pokes, while keeping the background nearly static and segmenting animal motion across all three dimensions.Combining its Z-direction flow with input-image depth can indicate which image parts will be occluded in the future.Source: C. Extension to 3D Motion · p. 14

D. Additional Qualitative Examples

Additional examples visualize motion distributions for different pokes and query locations, alongside qualitative comparisons on articulated-object motion. They illustrate varying motion possibilities across parts and interactions.

D. Additional Qualitative Examples

  • On Drag-A-Move, FPT qualitatively captures complex conditioning from multiple pokes better than DragAPart and PuppetMaster in this setup, while Motion-I2V often misses local conditioning.The figure compares FPT with Motion-I2V, DragAPart, and PuppetMaster.Source: Figure D · p. 14
  • For a downward giraffe-head poke, nearby queries assign greater likelihood to downward motion, while farther queries retain more no-motion probability because the neck may stay still.Arrow opacity in the distribution overlay indicates the relative likelihood of each mode.Source: D. Additional Qualitative Examples · p. 15
  • For the block tower, the likelihood of collapse depends on how quickly the brick is removed, and the tower may instead remain stationary.Source: D. Additional Qualitative Examples · p. 15
  • The cup may remain still or move with the hand holding it, depending on which hand moves; remaining still when that hand moves is very improbable.Source: D. Additional Qualitative Examples · p. 15
  • Predicted movement magnitude varies with query height on a tree, and the model also captures rotational movement effects.Source: D. Additional Qualitative Examples · p. 15
  • The additional samples visualize motion inferred from a single image, optionally conditioned on pokes, using input images and flow-prediction panels.The appendix also includes pointwise predictions and dense-motion samples.Source: D. Additional Qualitative Examples · pp. 14, 16

Related papers

Loading 2510.12777v1…