PaperlayerView exact revision on arXiv

Source-linked AI summary

Envisioning the Future, One Step at a Time

Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella, Mahdi M. Kalayeh, Björn Ommer

arXiv:2604.09527v1cs.CVcs.AIcs.LG

TL;DR

Dense simulators make open-set, long-horizon future exploration costly by modeling appearance alongside motion. The paper instead performs step-wise autoregressive diffusion over sparse point trajectories and introduces OWM for evaluation, reporting comparable or better accuracy with dramatically lower sampling cost.

  • Problem

    Dense video and latent-space approaches spend substantial capacity modeling dense scene appearance, making large-scale exploration of open-set future hypotheses costly.

  • Method

    The paper models open-set future dynamics as step-wise autoregressive diffusion over sparse point trajectories from a single image, optionally conditioned on initial motion.

  • Results

    The model matches or surpasses dense approaches in accuracy while enabling exploration of far more futures within the same compute budget.

  • Takeaways & Limitations

    Dynamics-centric sparse modeling makes rapid exploration of diverse, physically coherent long-range futures practical under compute- and time-constrained settings.

  • Takeaways & Limitations

    The formulation assumes a static camera and relies on pseudo ground-truth trajectories from off-the-shelf trackers, inheriting their biases and failure modes.

Abstract

from arXiv · show

Accurately anticipating how complex, diverse scenes will evolve requires models that represent uncertainty, simulate along extended interaction chains, and efficiently explore many plausible futures. Yet most existing approaches rely on dense video or latent-space prediction, expending substantial capacity on dense appearance rather than on the underlying sparse trajectories of points in the scene. This makes large-scale exploration of future hypotheses costly and limits performance when long-horizon, multi-modal motion is essential. We address this by formulating the prediction of open-set future scene dynamics as step-wise inference over sparse point trajectories. Our autoregressive diffusion model advances these trajectories through short, locally predictable transitions, explicitly modeling the growth of uncertainty over time. This dynamics-centric representation enables fast rollout of thousands of diverse futures from a single image, optionally guided by initial constraints on motion, while maintaining physical plausibility and long-range coherence. We further introduce OWM, a benchmark for open-set motion prediction based on diverse in-the-wild videos, to evaluate accuracy and variability of predicted trajectory distributions under real-world uncertainty. Our method matches or surpasses dense simulators in predictive accuracy while achieving orders-of-magnitude higher sampling speed, making open-set future prediction both scalable and practical. Project page: http://compvis.github.io/myriad.

1. Introduction

The paper frames open-set future prediction as a sparse, step-wise dynamics problem rather than dense scene rendering. Its autoregressive diffusion model and OWM benchmark support fast exploration of diverse, physically plausible futures from a single image.

  • Motivation: Open-ended future prediction is difficult because hidden contacts and stochastic interactions can produce many possible scene dynamics.The paper motivates abstraction over sparse, important changes as a way to reason through this complexity.
  • Motivation: Dense video and latent-space simulators expend capacity on entire-scene appearance, making large-scale branching over futures prohibitively costly.These models predict dense scene representations instead of focusing directly on sparse motion trajectories.
  • Approach: An autoregressive diffusion model enables fast rollouts of diverse futures and can optionally condition motion generation on initial motion cues.The formulation supports exploring how user-provided actions may affect uncertain futures.
  • Evaluation: OWM evaluates whether open-world models produce physically consistent and diverse trajectory distributions under real-world uncertainty.The benchmark is based on diverse in-the-wild videos and complements evaluation across structured and open-set domains.
  • Approach: The method formulates visual motion prediction as open-set, step-wise distributions over sparse point trajectories from a single image.Its autoregressive structure advances short transitions while modeling uncertainty growth over time.
  • Results: The approach matches or surpasses dense models in accuracy while enabling exploration of far more futures within the same compute budget.The paper characterizes the sampling advantage as orders of magnitude over dense video models.

2. Related Work

The related work is organized by visual tax, granularity, domain, and prediction paradigm. The paper positions its method as sparse, step-wise, user-granular, and open-set, contrasting with dense, one-shot, or domain-limited alternatives.

  • Comparison axes: The paper compares motion-prediction literature across visual tax, granularity, domain, and paradigm.Paradigm distinguishes single-shot prediction from step-by-step modeling of how systems evolve.
  • Dense prediction: Video and world models generate dense future sequences but incur a visual tax by modeling appearance alongside scene motion.This makes open-ended prediction and especially branching computationally expensive.
  • Dense prediction: Direct motion and feature-space models avoid some pixel generation but can increase computational demands, model excessive motion, or accumulate errors.The passage contrasts their efficiency and error behavior with the paper’s sparse trajectory focus.
  • Domain: Physics-engine approaches can be accurate when dynamics and parameters are fully in-domain, but they fail to generalize to truly open-set everyday scenes.Their limitation follows from dependence on estimated physical properties and predefined engines.
  • Paradigm: One-shot methods face high capacity requirements for multi-contact and long-horizon scenarios or incur a visual tax through pixel-level reasoning.The paper motivates step-wise prediction as an alternative to accounting for multiple interactions in one prediction.
  • Granularity and domain: Prior sparse-object methods usually target narrow domains and abstract inputs, whereas this work handles unconstrained scenes at user-specified inference granularity.The paper emphasizes parsing and multi-step reasoning directly from appearance at sparse decision points.
  • Positioning: The paper claims advantages across all four axes by modeling sparse user-defined points, generalizing to open-set scenes, and reasoning step by step.This contrasts with pixel-level, closed-environment, and single-shot approaches.

3. Methodology

The method predicts open-set future dynamics as autoregressive, step-wise inference over sparse point trajectories, using motion tokens, shared positional encoding, efficient reasoning blocks, and a flow-matching head. It models incremental motion distributions while supporting trajectory interactions, optional motion hints, uncertainty growth, and efficient sampling.

  • Trajectory formulation: Given a reference image and sparse visible query points, the model predicts a distribution over their full future trajectories.The joint trajectory distribution is factorized over time and trajectories.
  • Trajectory formulation: Incremental motions are accumulated from the initial points, and an optional initial motion hint can guide predicted trajectories.This formulation represents trajectory evolution through stepwise transitions.
  • Motion representation: Motion tokens combine trajectory motion, appearance at each trajectory origin, local context at the current position, and randomized trajectory identity.Random unit-sphere identifiers scale to arbitrary numbers of trajectories and avoid reliance on fixed indices.
  • Token interactions: Shared positional encoding represents each motion token’s current and original spatial positions together with time, while allowing attention to image and motion tokens.The encoding preserves both origin-based appearance context and current-position local context.
  • Efficient backbone: Fused parallel reasoning blocks combine self-attention, cross-attention, and feedforward computation to reduce kernel launches and improve rollout throughput.The design targets efficient exploration of many motion hypotheses.
  • Posterior modeling: A flow-matching head models stepwise motion distributions, with cached conditioning and scale-cascade inputs for heavy-tailed motion values.Compared with GMM parameterization, the authors report faster convergence and more accurate predictions.

4. Benchmarking Efficient Open-World Motion Prediction

The benchmark evaluates open-world motion prediction as distributions of feasible trajectories rather than point estimates. OWM uses diverse static-camera videos, supplemented by physics datasets, and measures accuracy and search efficiency under standardized hypothesis budgets.

  • Benchmark motivation: OWM addresses evaluation under open-world uncertainty by assessing distributions of feasible trajectories consistent with the single observed future.The benchmark limits distribution complexity to make evaluation feasible from one ground-truth observation.
  • OWM composition: OWM contains 95 diverse in-the-wild videos with static cameras, reference frames, query points, and verified ground-truth motion lasting 2.5s to 6.5s.The dataset is used solely for evaluation and is intended for public release.
  • Evaluation task: The task predicts multiple future trajectory samples from one RGB image, a short two-frame warmup hint, and provided query points.Video-generation baselines receive full additional frames and are tracked afterward.
  • Hypothesis generation: Best-of-N samples five trajectory sets, while the primary best-within-timelimit protocol measures how many hypotheses methods generate within five minutes on an Nvidia H200.Both protocols evaluate the hypothesis closest to the ground-truth observation.
  • Metrics: Prediction error uses the mean squared pointwise distance over query points and time for the closest generated trajectory, without penalizing alternative plausible futures.The metric is described as akin to a one-sided Wasserstein distance over motion space.

5. Experiments

Experiments show that sparse, step-wise motion prediction achieves accurate open-world and physical forecasting while enabling efficient hypothesis exploration and billiard-shot planning. The model’s posterior uncertainty tracks error, and architectural choices improve sampling speed and prediction quality.

  • 5.2. Motion Prediction: On OWM, the model matches observed motion more accurately than state-of-the-art video generators while using fewer parameters and substantially less computation.Under Best-within-5 min, its efficiency advantage widens the accuracy gap by avoiding RGB world simulation.
  • 5.2. Motion Prediction: Qualitative rollouts respect scene constraints, object kinematics, and coordinated motion across multiple points on the same object.
  • 5.3. Action Selection by Envisioning Futures: Sparse trajectory models improve throughput by at least an order of magnitude over image models, enabling higher accuracies in billiard planning.Direct final-state prediction is faster but insufficiently accurate, while full-trajectory diffusion significantly underperforms.
  • 5.3. Action Selection by Envisioning Futures: Under a fixed compute budget, the model surpasses dense world models for billiard-shot accuracy by sampling many candidate futures at low latency.Candidate actions vary the cue-ball impulse, and the selected plan maximizes expected goal reward.
  • 5.4. Calibration: The model’s posterior uncertainty predicts final trajectory error well above pixel-level error, indicating useful calibration of uncertainty.The relationship is approximately linear in log-log space beyond the low-error concentration.
  • 5.5. Ablations: Fused attention yields ∼2× faster sampling for 32-timestep rollouts and ∼3.7× faster sampling at batch size 1.Removing the scale cascade severely degrades prediction quality, while the flow-matching head outperforms a GMM head in accuracy and convergence.

6. Conclusion

The paper concludes that autoregressive diffusion over sparse trajectories enables efficient, probabilistic exploration of diverse futures and supports motion-conditioned forecasting and planning. OWM evaluations show accurate long-range predictions with reduced sampling cost across realistic and physical settings.

  • The autoregressive model explores multimodal trajectory distributions step by step, supporting efficient training, inference, and conditioning on user-provided initial motion.
  • Across diverse domains, the method achieves accurate long-range predictions while dramatically reducing sampling cost compared with future-frame generation.The resulting efficiency facilitates exploring possible actions and outcomes, including billiard-shot selection.
  • The main formulation assumes a static camera and relies on pseudo-ground-truth trajectories from off-the-shelf trackers.Dynamic viewpoints and tracker biases or failures remain scope boundaries for the current method.

A. Additional Implementation Details

The implementation combines transformer blocks with a flow-matching posterior head and efficiency-oriented conditioning and sampling. A logarithmic scale cascade stabilizes motion-distribution modeling, while autoregressive sampling advances one motion step at a time.

  • The model uses pre-normalized transformer blocks with RMSNorm, rotary positional embeddings, scaled cosine attention, and GELU feedforward activations.
  • The flow-matching posterior head models a distribution over stepwise motion and uses an efficient architecture for sampling.Its three-layer head has width 1024 and uses a standard flow-matching objective.
  • Conditioning on flow-matching time and motion parameters is factorized so values can be precomputed or reused across sampling loops.This reduces repeated computational overhead during the many forward passes required for sampling.
  • The posterior head applies a logarithmically spaced input scale cascade to stabilize motion modeling across different value ranges.The cascade forms 512 exponentially separated coefficients and transforms noisy motion inputs component-wise.
  • Autoregressive sampling draws one posterior motion sample, updates the latest trajectory position, and then samples the next step.The flow-matching ODE is solved with an Euler solver using 50 sampling steps unless otherwise specified.

A.4. Training Data

Training uses diverse in-the-wild videos, stabilized tracks, billiard simulations, and curated physical-motion data. OWM evaluates trajectory prediction under realistic uncertainty using verified static-camera scenes and ensemble-based hypothesis budgets.

  • The training pipeline uses three data sources: open-set videos, reprojected 3D data, and billiard physics simulations.Open-set videos provide unconstrained motion, while billiard trajectories come from a physics engine.
  • Open-set videos are sourced from the internet without overlap with evaluation data and converted into tracked motion examples.Frames are center-cropped and resized to 512 × 512, while off-the-shelf tracking supplies pseudo-ground-truth annotations.
  • OWM Dataset: OWM contains 95 diverse static-camera videos covering varied motion, with manually selected start frames and points and manually verified tracking quality.The benchmark also includes 97 Physics-IQ samples and 134 Physion samples, excluding the Drape subset.
  • Metrics and Hypothesis Generation: OWM evaluates trajectory prediction with MSE and permits ensembles of hypotheses under Best-of-5 and Best-in-5min settings.The five-sample setting is fixed, while the time-limited ensemble size depends on each method’s throughput.
  • Billiard Planning: Billiard planning samples candidate actions, predicts their rollouts, scores them by goal reward, and executes the action with the highest expected reward.Planning accuracy is measured by whether the target ball reaches the predefined goal region.
  • Baselines: Baselines include image-to-video, framewise autoregressive video, full-trajectory diffusion, single-step prediction, and trajectory-prediction methods.

B.1. Number of Function Evaluations

The experiments examine inference scaling and multi-step prediction, showing that additional denoising evaluations improve billiard accuracy until gains diminish, while shorter autoregressive steps improve long-horizon modeling. The approach also remains competitive in zero-shot trajectory forecasting and across OWM subsets.

  • Number of Function Evaluations: More denoising function evaluations reduce billiard endpoint error, but improvements diminish beyond 10 evaluations.The main evaluations use 50 evaluations to balance quality and speed.
  • Trajectory ID Embedding: Randomized trajectory embeddings outperform learnable and absent embeddings while enabling zero-shot extrapolation to different trajectory counts.The model was trained with 16 trajectories and generalizes to larger and smaller numbers with minimal degradation.
  • Multi-Step Reasoning: Short-step autoregressive prediction improves billiard endpoint error compared with predicting the full 0.5-second horizon in one step.The standard model predicts ∆t = 0.01 s per step, while the single-step variant performs significantly worse than both multi-step variants.
  • Zero-Shot Trajectory Forecasting: In zero-shot ETH-UCY forecasting, the method achieves competitive results with canonical task-specific baselines despite operating directly on images.The baselines are trained on in-domain data, whereas the method is applied zero-shot.
  • OWM Subsets: The method remains competitive with substantially larger video baselines across OWM subsets, including intricate multi-agent interactions.Under a time constraint, its fast inference achieves the best results across all subsets by exploring more potential futures.

C. Additional Qualitative Samples

Additional samples show that the model produces coherent, physically plausible, and context-informed motion with full, partial, or no motion hints. Qualitative comparisons also indicate strong performance on open-world, physical-interaction, and billiard scenarios.

  • Physical Reasoning: On Physics-IQ, the method captures realistic physical interactions where video-generation models often produce simplified or absent motion.On Physion, the method captures rigid-body dynamics while video models hallucinate overly complex motion.
  • Hinted Samples: Initial motion hints produce complex, appearance-informed trajectories with consistent motion across entities and realistic long-term behavior.The samples illustrate trajectory unrolling from hinted motion.
  • Un-Hinted Samples: Appearance-only conditioning produces physically correct and coherent motion, including cars remaining still when pedestrians block an intersection.Unhinted trajectories are generally simpler but remain realistic from the input image alone.
  • Partially-Hinted Samples: A single motion hint can yield coherent motion across linked objects and preserve multimodal outcomes when multiple behaviors are plausible.The Jenga example demonstrates multimodality under the same initial motion hint.
  • Billiard Samples: In billiard simulations, predictions closely match observed ground-truth motion up to minor stochastic details.The visualization compares predicted trajectories from an initial impulse with overlaid ground-truth simulations.

D. Language Model Usage

The paper reports using large language models for text refinement, draft feedback, and initial auxiliary utility-function implementations. It states that the paper’s scientific content, results, and novel ideas were not generated by these models.

  • Language Model Usage: Large language models were used to refine text, improve grammar, and suggest alternative phrasing.The listed models were OpenAI GPT-5.2 and Claude Opus 4.6.
  • Language Model Usage: The models also provided feedback on early drafts and proposed initial implementations for auxiliary utility functions.Those implementations were subsequently verified and reworked by the authors.
  • Language Model Usage: The authors state that no scientific content, experimental results, or novel ideas were generated by the language models.
Loading 2604.09527v1…