PaperlayerView exact revision on arXiv

Source-linked AI summary

Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos

Jailing Lin, Jikuan Zhang, Jianhua Sun

arXiv:2608.31025v1cs.CV

TL;DR

Inferring intrinsic object dynamics from monocular observations is difficult because visual evidence is indirect and existing approaches trade efficiency against physical accuracy. Analytic Dynamics learns a physics-grounded representation from privileged simulated states and distills it into a feed-forward video model, which reports accurate, generalizable inference at 3.83 milliseconds. The approach is evaluated with a benchmark pairing trajectories, videos, and material annotations, while its real-world scope remains limited by idealized constitutive models.

  • Problem

    Existing methods either require costly per-scene optimization or lack intermediate physical abstractions, while visual observations alone can encourage appearance and geometry shortcuts.

    Source: Introduction · p. 1

  • Method

    Analytic Dynamics learns a structured representation from simulated positions, displacements, and deformation gradients, then aligns monocular-video representations with the frozen teacher for material classification and parameter regression.

    Source: Introduction · pp. 1–2, Privileged Dynamics Representation… · p. 4

  • Results

    Analytic Dynamics outperforms existing baselines while performing inference in 3.83 milliseconds and generalizing to unseen objects.

    Source: Introduction · p. 2, Comparison on Unseen Objects · p. 6

  • Takeaways & Limitations

    Privileged physical states provide a physics-grounded learning target that guides visual models toward dynamics-relevant patterns beyond object-specific visual correlations.

    Source: Conclusion · p. 7, Privileged-State Ablation on Seen… · p. 11

    The simulator uses a limited set of idealized constitutive models and omits effects such as damping, viscoelasticity, heterogeneity, and complex contact behavior.

    Source: Real-World Generalization · p. 7

Abstract

from arXiv · show

Inferring object dynamics from visual observations is essential for intelligent agents to reason about and interact with the physical world, yet remains challenging due to the fundamental gap between visual evidence and intrinsic dynamics. Existing methods either rely on costly per-scene optimization, limiting efficiency and scalability, or directly map visual evidence to intrinsic dynamics without intermediate physical abstractions, making them prone to appearance and geometry shortcuts. To bridge this gap, we propose Analytic Dynamics, a feed-forward dynamics inference framework that introduces an intermediate physics-grounded dynamics representation between visual observations and intrinsic dynamics. Specifically, we leverage privileged physical states, including position, displacement, and deformation gradient fields, which are available in simulation, to learn a structured dynamics representation that is difficult to discover from visual observations alone. By aligning visual representations with this space, we equip visual models with a physics-grounded inductive bias, guiding them to capture dynamics-relevant patterns for material model classification and parameter regression. To facilitate this research, we develop a dynamics data generation pipeline and benchmark containing paired physical state trajectories, rendered videos, and ground-truth material models and parameters. Extensive experiments demonstrate that Analytic Dynamics achieves efficient, accurate, and generalizable dynamics inference from monocular videos.

Introduction

Analytic Dynamics addresses the efficiency–accuracy gap in intrinsic dynamics inference by distilling privileged physical-state representations into a feed-forward monocular-video model. The framework targets material models and parameters while reducing reliance on appearance and geometry cues.

  • Per-scene inverse-physics methods are accurate but require extensive simulation–optimization iterations for each scene, limiting computational efficiency and scalability.Source: Introduction · p. 1
  • Visual-only approaches can produce noisy, coarse-grained predictions or exploit appearance and geometry shortcuts because intrinsic dynamics are not directly observable.Source: Introduction · p. 1
  • Analytic Dynamics introduces a physics-grounded representation between monocular observations and intrinsic dynamics through two-stage privileged dynamics distillation.Source: Introduction · pp. 1–2
  • The framework uses simulated position, displacement, and deformation-gradient fields to guide visual learning of constitutive material models and parameters.Source: Introduction · p. 1
  • 9,000 dynamics instances across 60 objects pair monocular videos with physical-state trajectories and ground-truth material annotations.Source: Introduction · p. 2
  • 3.83 milliseconds is the reported inference time, while experiments also demonstrate strong out-of-distribution generalization.Source: Introduction · p. 2

Related Work

Prior work spans per-scene inverse-physics optimization and feed-forward property estimation using visual or language-model priors. Analytic Dynamics instead uses paired simulated trajectories and videos to transfer physics-grounded representations into efficient visual inference.

  • Per-scene inverse-physics methods optimize physical parameters through differentiable rendering and simulation under prescribed or learned constitutive models.Source: Per-Scene Dynamics Optimization · p. 2
  • Feed-forward methods transfer material semantics and physical priors from pretrained vision-language models to 3D scene representations for zero-shot property estimation.Source: Feed-Forward Dynamics Inference · p. 2
  • The related feed-forward line is described as transferring priors rather than learning generalizable physical regularities.Source: Feed-Forward Dynamics Inference · pp. 2–3
  • Analytic Dynamics generates paired particle trajectories and monocular mask videos, using simulator-only physical states to train a trajectory teacher and distill it into a video student.Source: Figure 2 · p. 3

Methodology

The method constructs synchronized physical-state trajectories and monocular mask videos, learns a dynamics-relevant teacher representation from privileged states, and distills it into a video model. The deployed student predicts material families and parameters from monocular videos alone.

Problem Formulation and Overview

Data Generation

Physical-State Trajectory

Privileged Dynamics Representation Learning

Visual Representation Distillation

Experiments

Experiments evaluate Analytic Dynamics on unseen objects using paired dynamics data, material inference metrics, representation analyses, and visual reconstruction. The method accurately infers dynamics and reconstructs observed behavior efficiently, while per-scene optimization remains competitive on some metrics.

Experimental Setup

  • The benchmark contains 9,000 MPM-based dynamics instances spanning jelly, plasticine, and sand, with synchronized particle trajectories and monocular mask videos.Instances vary object shape, material parameters, external excitation, and viewpoint.Source: Datasets and Metrics · p. 5

Evaluation

  • Evaluation measures material classification accuracy, material-specific parameter errors, aggregate RegL1, and PSNR after re-simulating objects with inferred parameters.E and σy use log-MAE; ν and ϕ use MAE.Source: Datasets and Metrics · p. 5, Table 1 · p. 5

Unseen-Object Comparison

  • On unseen objects, the full model achieves the best overall performance, while VLMs and Video-LLMs retain classification ability but show substantially larger parameter errors.Per-scene optimization remains competitive on several metrics despite its high computational cost.Source: Comparison on Unseen Objects · p. 5, Baselines · p. 5

Ablation

  • Alignment outperforms direct label supervision, and adding task supervision further improves prediction of material classes and parameters.Alignment captures dynamics-relevant information, whereas task supervision makes that information predictive of the target outputs.Source: Comparison on Unseen Objects · p. 6

Visual Reconstruction

  • Analytic Dynamics more accurately reproduces JellyBall elastic deformation and SandChocolate flow and accumulation than VLM and Video-LLM baselines.On ClayCake, per-scene optimization achieves higher reconstruction accuracy, while the proposed method produces visually comparable deformation without test-time optimization.Source: Figure 3 · p. 5, Comparison on Unseen Objects · p. 6

Overall Result

  • Analytic Dynamics infers material models and parameters from monocular videos in 3.83 ms and generalizes to unseen objects.The result combines quantitative evaluation with visual dynamics reconstruction.Source: Comparison on Unseen Objects · p. 6

Analysis of Learned Dynamics Representations

The learned representation is organized around intrinsic dynamics rather than object shape and viewpoint. Privileged-state ablations show that deformation gradients and displacement together provide the most transferable guidance for the video model, while synthetic training transfers to real observations with a simulator-to-reality gap.

Representation Organization

  • Without alignment, Stage II representations mix stiffnesses and remain entangled with shape and viewpoint; alignment organizes similar Young’s moduli smoothly across unseen objects.The aligned representation supports cross-shape neighborhoods ordered by stiffness.Source: Analysis of Learned Dynamics… · p. 6
  • Interpolating two unseen jelly representations increases predicted Young’s modulus monotonically from 20718 to 697376 while simulated deformation changes from large to small.This supports a continuous, ordered representation of material variation.Source: Analysis of Learned Dynamics… · p. 6

Privileged-State Ablation

  • Position alone preserves geometry and yields the worst Stage I performance, whereas adding deformation gradients reduces Stage I RegL1 error to 0.046.Deformation gradients provide strong cues for intrinsic dynamics.Source: Effect of Privileged Physical States · pp. 6–7
  • The most accurate teacher is not the best student: x+F performs best in Stage I, but x+v+F performs best in Stage II.Displacement supplies visible inter-frame motion cues, while deformation gradients capture fine-grained material responses.Source: Effect of Privileged Physical States · p. 7, Figure 5 · p. 7

Real-World Generalization

  • On two real scenes, the synthetic-trained model predicts parameters and reconstructs dynamics without fine-tuning, providing initial evidence of transfer to real observations.PSNR is used as an indirect measure because ground-truth real-world material parameters are unavailable.Source: Real-World Generalization · p. 7, Figure 6 · p. 7
  • The remaining real-world gap is attributed mainly to simulator mismatch, including omitted damping, viscoelasticity, heterogeneity, and complex contact behavior.Richer constitutive models and more accurate physical modeling are identified as directions for improving generalization.Source: Real-World Generalization · p. 7, Figure 6 · p. 7

Conclusion

Analytic Dynamics uses privileged particle states to learn a physics-grounded representation and distill it into a video model for material inference. The framework achieves accurate, generalizable inference with 3.83 ms inference time, but is currently limited to idealized constitutive models.

  • The framework learns a physics-grounded dynamics representation from privileged particle states and distills it into a monocular-video model.Source: Conclusion · p. 7
  • The benchmark pairs monocular mask videos with tracked physical-state trajectories and material annotations for systematic training and evaluation.Source: Conclusion · p. 7
  • Extensive experiments show accurate material inference, strong generalization to unseen objects, and transfer to real-world observations.Source: Conclusion · p. 7
  • 3.83 ms inference time supports efficient feed-forward material inference.Source: Conclusion · p. 7
  • The framework is trained on a limited set of idealized constitutive models, motivating extension to richer material behaviors and interactions.Source: Conclusion · p. 7

Overview

The appendix documents dataset-generation implementation, extended experiments and analyses, the MPM algorithm, and prompts used for vision-language baselines.

  • The appendix adds implementation details for dataset generation, extended experimental results and analyses, MPM details, and vision-language baseline prompts.Source: Overview · p. 10

Additional Dataset and Benchmark Details

The benchmark is generated with MPM simulations that record physical states and render paired monocular mask videos across varied geometries, materials, parameters, and views.

Additional Experimental Results

Additional experiments show strong parameter interpolation on seen geometries, improved cross-shape and temporal generalization from alignment, and 3.83 ms feed-forward inference.

Evaluation on Seen Objects

  • On seen geometries with new parameters, material classification is nearly perfect and the full model achieves the best overall regression performance.Source: Evaluation on Seen Objects · p. 10
  • Alignment gains are smaller on seen geometries, where the unaligned model slightly leads for plastic yield stress and sand friction angle.Source: Evaluation on Seen Objects · p. 10

Privileged-State Ablation on Seen Objects

Temporal OOD Generalization on Unseen Objects

Representation and Efficiency

Constitutive Material Families and Parameters

The benchmark models jelly, plasticine, and sand with distinct elastic and plastic constitutive components, each governed by material-specific parameters. Jelly is purely elastic, plasticine supports irreversible yielding, and sand uses pressure-dependent frictional plasticity with friction angle as its varied parameter.

Jelly

  • Jelly combines CorotatedElasticity with IdentityPlasticity, representing a purely elastic material without irreversible deformation.Source: Jelly Material · p. 12
  • Jelly behavior is controlled by Young’s modulus E and Poisson’s ratio ν, governing stiffness and volumetric deformation, respectively.Source: Jelly Material · p. 13

Plasticine

  • Plasticine combines SigmaElasticity with VonMisesPlasticity, introducing irreversible deformation once the yield limit is exceeded.Source: Plasticine Material · p. 13
  • Plasticine is governed by Young’s modulus E, Poisson’s ratio ν, and yield stress σy, with σy determining the onset of permanent deformation.Source: Plasticine Material · p. 13

Sand

  • Sand combines SigmaElasticity with DruckerPragerPlasticity, modeling pressure-dependent yielding and frictional flow.Source: Sand Material · p. 13
  • The benchmark varies only sand’s friction angle ϕf because it governs shear resistance and observed frictional dynamics, while E and ν remain fixed.Source: Sand Material · p. 13, Table 4 · p. 13

Material Point Method Formulation

The simulations use an explicit Material Point Method that represents materials with Lagrangian particles while computing interactions on an Eulerian grid. Each step transfers particle information to the grid, updates grid dynamics, and transfers updated velocities back to particles before constitutive stress evaluation.

MPM overview

MPM update

Particle state

Grid update

Grid-to-particle transfer

Constitutive update

Prompts for Vision-Language Baselines

The vision-language baselines infer one global material model and its continuous parameters from either multi-view images or monocular video. Their prompts prioritize observed motion and deformation for model selection, then use material-specific cues to estimate physically plausible parameters.

Baseline inputs

Output constraints

Inference procedure

Evidence prioritization

Material classification cues

Parameter cues

Related papers

Loading 2608.31025v1…