PaperlayerView exact revision on arXiv

Source-linked AI summary

Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos

Jailing Lin, Jikuan Zhang, Jianhua Sun

arXiv:2608.31025v1cs.CV

TL;DR

Inferring intrinsic object dynamics from monocular observations is difficult because visual evidence is indirect and existing approaches trade efficiency against physical accuracy. Analytic Dynamics learns a physics-grounded representation from privileged simulated states and distills it into a feed-forward video model, which reports accurate, generalizable inference at 3.83 milliseconds. The approach is evaluated with a benchmark pairing trajectories, videos, and material annotations, while its real-world scope remains limited by idealized constitutive models.

  • Problem

    Existing methods either require costly per-scene optimization or lack intermediate physical abstractions, while visual observations alone can encourage appearance and geometry shortcuts.

  • Method

    Analytic Dynamics learns a structured representation from simulated positions, displacements, and deformation gradients, then aligns monocular-video representations with the frozen teacher for material classification and parameter regression.

  • Results

    Analytic Dynamics outperforms existing baselines while performing inference in 3.83 milliseconds and generalizing to unseen objects.

  • Takeaways & Limitations

    Privileged physical states provide a physics-grounded learning target that guides visual models toward dynamics-relevant patterns beyond object-specific visual correlations.

  • Takeaways & Limitations

    The simulator uses a limited set of idealized constitutive models and omits effects such as damping, viscoelasticity, heterogeneity, and complex contact behavior.

Abstract

from arXiv · show

Inferring object dynamics from visual observations is essential for intelligent agents to reason about and interact with the physical world, yet remains challenging due to the fundamental gap between visual evidence and intrinsic dynamics. Existing methods either rely on costly per-scene optimization, limiting efficiency and scalability, or directly map visual evidence to intrinsic dynamics without intermediate physical abstractions, making them prone to appearance and geometry shortcuts. To bridge this gap, we propose Analytic Dynamics, a feed-forward dynamics inference framework that introduces an intermediate physics-grounded dynamics representation between visual observations and intrinsic dynamics. Specifically, we leverage privileged physical states, including position, displacement, and deformation gradient fields, which are available in simulation, to learn a structured dynamics representation that is difficult to discover from visual observations alone. By aligning visual representations with this space, we equip visual models with a physics-grounded inductive bias, guiding them to capture dynamics-relevant patterns for material model classification and parameter regression. To facilitate this research, we develop a dynamics data generation pipeline and benchmark containing paired physical state trajectories, rendered videos, and ground-truth material models and parameters. Extensive experiments demonstrate that Analytic Dynamics achieves efficient, accurate, and generalizable dynamics inference from monocular videos.

Introduction

Analytic Dynamics addresses the efficiency–accuracy gap in intrinsic dynamics inference by distilling privileged physical-state representations into a feed-forward monocular-video model. The framework targets material models and parameters while reducing reliance on appearance and geometry cues.

  • Per-scene inverse-physics methods are accurate but require extensive simulation–optimization iterations for each scene, limiting computational efficiency and scalability.
  • Visual-only approaches can produce noisy, coarse-grained predictions or exploit appearance and geometry shortcuts because intrinsic dynamics are not directly observable.
  • Analytic Dynamics introduces a physics-grounded representation between monocular observations and intrinsic dynamics through two-stage privileged dynamics distillation.
  • The framework uses simulated position, displacement, and deformation-gradient fields to guide visual learning of constitutive material models and parameters.
  • 9,000 dynamics instances across 60 objects pair monocular videos with physical-state trajectories and ground-truth material annotations.
  • 3.83 milliseconds is the reported inference time, while experiments also demonstrate strong out-of-distribution generalization.

Related Work

Prior work spans per-scene inverse-physics optimization and feed-forward property estimation using visual or language-model priors. Analytic Dynamics instead uses paired simulated trajectories and videos to transfer physics-grounded representations into efficient visual inference.

  • Per-scene inverse-physics methods optimize physical parameters through differentiable rendering and simulation under prescribed or learned constitutive models.
  • Feed-forward methods transfer material semantics and physical priors from pretrained vision-language models to 3D scene representations for zero-shot property estimation.
  • The related feed-forward line is described as transferring priors rather than learning generalizable physical regularities.
  • Analytic Dynamics generates paired particle trajectories and monocular mask videos, using simulator-only physical states to train a trajectory teacher and distill it into a video student.

Methodology

The method constructs synchronized physical-state trajectories and monocular mask videos, learns a dynamics-relevant teacher representation from privileged states, and distills it into a video model. The deployed student predicts material families and parameters from monocular videos alone.

  • Problem Formulation and Overview: The learning objective maps a monocular mask video observed from viewpoint r to material family y_i and material parameters θ_i.
  • Problem Formulation and Overview: Because monocular video indirectly observes physical response, training-only physical-state trajectories provide a direct signal for guiding the video model.
  • Data Generation: The data pipeline pairs simulated object configurations with tracked particle states and synchronized deforming-surface videos sharing material annotations.
  • Physical-State Trajectory: Each physical-state trajectory contains tracked positions, normalized frame displacements, and deformation gradients, with particle identity preserved through simulation.
  • Privileged Dynamics Representation Learning: Stage I uses a P4Transformer and projector to encode spatiotemporal particle interactions into teacher representation zT_i.
  • Privileged Dynamics Representation Learning: The teacher is trained with material-family classification and normalized parameter regression, then frozen as the representation target for paired videos.
  • Visual Representation Distillation: Stage II encodes mask videos with a UniFormer, aligns visual and teacher representations using cosine and ℓ1 distances, and retains task supervision.
  • Visual Representation Distillation: At inference, the teacher is discarded and the video model directly predicts intrinsic dynamics from a monocular mask video.

Experiments

Experiments evaluate Analytic Dynamics on unseen objects using paired dynamics data, material inference metrics, representation analyses, and visual reconstruction. The method accurately infers dynamics and reconstructs observed behavior efficiently, while per-scene optimization remains competitive on some metrics.

  • Experimental Setup: The benchmark contains 9,000 MPM-based dynamics instances spanning jelly, plasticine, and sand, with synchronized particle trajectories and monocular mask videos.Instances vary object shape, material parameters, external excitation, and viewpoint.
  • Evaluation: Evaluation measures material classification accuracy, material-specific parameter errors, aggregate RegL1, and PSNR after re-simulating objects with inferred parameters.E and σy use log-MAE; ν and ϕ use MAE.
  • Unseen-Object Comparison: On unseen objects, the full model achieves the best overall performance, while VLMs and Video-LLMs retain classification ability but show substantially larger parameter errors.Per-scene optimization remains competitive on several metrics despite its high computational cost.
  • Ablation: Alignment outperforms direct label supervision, and adding task supervision further improves prediction of material classes and parameters.Alignment captures dynamics-relevant information, whereas task supervision makes that information predictive of the target outputs.
  • Visual Reconstruction: Analytic Dynamics more accurately reproduces JellyBall elastic deformation and SandChocolate flow and accumulation than VLM and Video-LLM baselines.On ClayCake, per-scene optimization achieves higher reconstruction accuracy, while the proposed method produces visually comparable deformation without test-time optimization.
  • Overall Result: Analytic Dynamics infers material models and parameters from monocular videos in 3.83 ms and generalizes to unseen objects.The result combines quantitative evaluation with visual dynamics reconstruction.

Analysis of Learned Dynamics Representations

The learned representation is organized around intrinsic dynamics rather than object shape and viewpoint. Privileged-state ablations show that deformation gradients and displacement together provide the most transferable guidance for the video model, while synthetic training transfers to real observations with a simulator-to-reality gap.

  • Representation Organization: Without alignment, Stage II representations mix stiffnesses and remain entangled with shape and viewpoint; alignment organizes similar Young’s moduli smoothly across unseen objects.The aligned representation supports cross-shape neighborhoods ordered by stiffness.
  • Representation Organization: Interpolating two unseen jelly representations increases predicted Young’s modulus monotonically from 20718 to 697376 while simulated deformation changes from large to small.This supports a continuous, ordered representation of material variation.
  • Privileged-State Ablation: Position alone preserves geometry and yields the worst Stage I performance, whereas adding deformation gradients reduces Stage I RegL1 error to 0.046.Deformation gradients provide strong cues for intrinsic dynamics.
  • Privileged-State Ablation: The most accurate teacher is not the best student: x+F performs best in Stage I, but x+v+F performs best in Stage II.Displacement supplies visible inter-frame motion cues, while deformation gradients capture fine-grained material responses.
  • Real-World Generalization: On two real scenes, the synthetic-trained model predicts parameters and reconstructs dynamics without fine-tuning, providing initial evidence of transfer to real observations.PSNR is used as an indirect measure because ground-truth real-world material parameters are unavailable.
  • Real-World Generalization: The remaining real-world gap is attributed mainly to simulator mismatch, including omitted damping, viscoelasticity, heterogeneity, and complex contact behavior.Richer constitutive models and more accurate physical modeling are identified as directions for improving generalization.

Conclusion

Analytic Dynamics uses privileged particle states to learn a physics-grounded representation and distill it into a video model for material inference. The framework achieves accurate, generalizable inference with 3.83 ms inference time, but is currently limited to idealized constitutive models.

  • The framework learns a physics-grounded dynamics representation from privileged particle states and distills it into a monocular-video model.
  • The benchmark pairs monocular mask videos with tracked physical-state trajectories and material annotations for systematic training and evaluation.
  • Extensive experiments show accurate material inference, strong generalization to unseen objects, and transfer to real-world observations.
  • 3.83 ms inference time supports efficient feed-forward material inference.
  • The framework is trained on a limited set of idealized constitutive models, motivating extension to richer material behaviors and interactions.

Overview

The appendix documents dataset-generation implementation, extended experiments and analyses, the MPM algorithm, and prompts used for vision-language baselines.

  • The appendix adds implementation details for dataset generation, extended experimental results and analyses, MPM details, and vision-language baseline prompts.

Additional Dataset and Benchmark Details

The benchmark is generated with MPM simulations that record physical states and render paired monocular mask videos across varied geometries, materials, parameters, and views.

  • MPM converts normalized meshes into volumetric particles and binds particles to mesh vertices so simulated motion can be transferred to rendered surfaces.
  • Simulations use a 32-resolution grid, 5 × 10^-4 time step, gravity g = 9.8m/s^2, shared contact settings, and 1,200 steps.
  • Each sequence contains 25 monocular 512 × 512 mask-video frames, with four training views and one randomly selected evaluation view.
  • The benchmark covers 60 geometries and three material families: jelly, plasticine, and sand.
  • Training uses 30 geometries and 100 parameter configurations per material family, producing 9,000 training instances.
  • The paired-data algorithm advances MPM states, records physical trajectories at output frames, updates the mesh, and renders multi-view videos.
  • The unseen-object split contains 30 geometries absent from training, with 10 parameter configurations per geometry and material family, producing 900 instances.

Additional Experimental Results

Additional experiments show strong parameter interpolation on seen geometries, improved cross-shape and temporal generalization from alignment, and 3.83 ms feed-forward inference.

  • Evaluation on Seen Objects: On seen geometries with new parameters, material classification is nearly perfect and the full model achieves the best overall regression performance.
  • Evaluation on Seen Objects: Alignment gains are smaller on seen geometries, where the unaligned model slightly leads for plastic yield stress and sand friction angle.
  • Privileged-State Ablation on Seen Objects: All distilled students outperform the Stage-II baseline; on seen geometries, x + v reaches 0.024 Stage-II RegL1 versus 0.027 for x + F and the full state.
  • Privileged-State Ablation on Seen Objects: The full privileged state primarily benefits cross-shape generalization, where combining motion and deformation responses reduces shape dependence.
  • Temporal OOD Generalization on Unseen Objects: At 15 observed frames, alignment reduces overall RegL1 from 0.224 to 0.138, while shorter observations generally degrade performance.
  • Representation and Efficiency: UMAP representations become more compact and material-discriminative after alignment, while inference takes 3.83 ms in a single forward pass.

Constitutive Material Families and Parameters

The benchmark models jelly, plasticine, and sand with distinct elastic and plastic constitutive components, each governed by material-specific parameters. Jelly is purely elastic, plasticine supports irreversible yielding, and sand uses pressure-dependent frictional plasticity with friction angle as its varied parameter.

  • Jelly: Jelly combines CorotatedElasticity with IdentityPlasticity, representing a purely elastic material without irreversible deformation.
  • Jelly: Jelly behavior is controlled by Young’s modulus E and Poisson’s ratio ν, governing stiffness and volumetric deformation, respectively.
  • Plasticine: Plasticine combines SigmaElasticity with VonMisesPlasticity, introducing irreversible deformation once the yield limit is exceeded.
  • Plasticine: Plasticine is governed by Young’s modulus E, Poisson’s ratio ν, and yield stress σy, with σy determining the onset of permanent deformation.
  • Sand: Sand combines SigmaElasticity with DruckerPragerPlasticity, modeling pressure-dependent yielding and frictional flow.
  • Sand: The benchmark varies only sand’s friction angle ϕf because it governs shear resistance and observed frictional dynamics, while E and ν remain fixed.

Material Point Method Formulation

The simulations use an explicit Material Point Method that represents materials with Lagrangian particles while computing interactions on an Eulerian grid. Each step transfers particle information to the grid, updates grid dynamics, and transfers updated velocities back to particles before constitutive stress evaluation.

  • MPM overview: Material Point Method represents material using Lagrangian particles and evaluates physical interactions on a background Eulerian grid.
  • MPM update: The explicit MPM solver uses particle-to-grid transfer, grid update, and grid-to-particle transfer at every simulation step.
  • Particle state: Each particle stores position, mass, reference volume, velocity, deformation gradient, and affine velocity matrix.
  • Grid update: Particle mass and momentum are accumulated on grid nodes, where internal forces and velocities are computed with boundary and collision responses.
  • Grid-to-particle transfer: Updated grid velocities are interpolated back to particles using quadratic B-spline interpolation and an updated affine velocity matrix.
  • Constitutive update: The trial deformation gradient is projected by φP and mapped through φE to compute stress, while purely elastic materials use identity projection.

Prompts for Vision-Language Baselines

The vision-language baselines infer one global material model and its continuous parameters from either multi-view images or monocular video. Their prompts prioritize observed motion and deformation for model selection, then use material-specific cues to estimate physically plausible parameters.

  • Baseline inputs: GPT-5.5 receives multi-view RGB images, whereas Gemini 3.5 Flash receives monocular RGB video for material inference.
  • Output constraints: Both prompts require selecting exactly one model from elastic, plasticine, and sand and outputting only its required scalar parameters.
  • Inference procedure: The inference procedure first selects a global material model, then estimates the continuous parameter values required by that model.
  • Evidence prioritization: Dynamic cues are primary evidence, while visual appearance is secondary when motion is ambiguous.
  • Material classification cues: Reversible deformation, bouncing, or oscillation indicates elasticity; retained dents, flattening, or spreading indicate plasticine; granular flow and piling indicate sand.
  • Parameter cues: For elastic and plasticine materials, Young’s modulus reflects deformation resistance, while plasticine yield stress controls the onset and resistance of permanent deformation.
  • Parameter cues: For sand, friction angle is larger for steep stable piles and stronger flow resistance, and smaller for broad spreading or shallow piles.
Loading 2608.31025v1…