PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysiFormer: Learning to Simulate Mechanics in World Space

Yiming Chen, Yushi Lan, Andrea Vedaldi

arXiv:2606.27364v1cs.CV

TL;DR

Video-based physical world models entangle geometry and motion with viewpoint, lighting, and occlusion. PhysiFormer predicts complete future mesh trajectories in world coordinates with diffusion, outperforming autoregressive baselines across physical-consistency measures and generalising across materials and geometries.

  • Problem

    Video representations ambiguously entangle physical geometry and motion with viewpoint, lighting, and occlusion, motivating direct 3D-mesh dynamics modelling.

  • Method

    PhysiFormer uses a diffusion transformer to jointly generate world-space trajectories for rigid and elastic mesh objects from initial positions, velocities, and material properties.

  • Results

    PhysiFormer improves trajectory accuracy, rigidity preservation, and momentum-based physical consistency over autoregressive baselines while generalising to unseen geometries, larger object counts, and mixed materials.

  • Takeaways & Limitations

    Coordinate-space diffusion supports unified multi-material, multi-object mechanics modelling and diverse plausible futures from incomplete physical-property conditioning.

  • Takeaways & Limitations

    The model leaves some constant physical properties implicit, so its Markovian dynamics are only approximately satisfied when relevant state information is not provided.

Abstract

from arXiv · show

We present PhysiFormer, a diffusion transformer for physically-plausible 3D object motion. Unlike video world models that operate in view-dependent pixel space, PhysiFormer represents objects as 3D meshes expressed in world coordinates. Given the initial vertex positions and velocities, as well as object material type, rigid or elastic, the model samples future vertex trajectories. While related neural physics approaches build on ad-hoc latent spaces or explicitly enforce rigidity and causality, PhysiFormer shows that excellent results can be obtained without any such inductive biases, by casting vertex trajectory prediction as a single denoising diffusion process directly in world coordinates. The probabilistic formulation captures uncertainty in the learned dynamics, enabling diverse plausible futures from initial conditions, making this framework potentially useful for applications with unobserved uncertainty. The model features attention factorised over time, space, and objects for efficiency, enabling permutation-invariant multi-object reasoning without needing explicit object encoding. Trained on over 100k simulated trajectories, PhysiFormer generates rigid and elastic mechanics, and generalises to mixed-material settings, unseen real-world geometries, and larger object counts. It substantially outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency. Our results position coordinate-space diffusion as a promising step toward view-invariant, geometry-aware world modelling for robotics, graphics, and physical design. Visualisations, code, and models are available at https://yimingc9.github.io/physiformer.

1 Introduction

PhysiFormer reformulates mechanical prediction as diffusion-based generation of physically plausible 3D mesh trajectories in world space, supporting rigid, elastic, and mixed-material objects. Its one-shot trajectory generation, factorised attention, and probabilistic formulation enable efficient multi-object reasoning, uncertainty modelling, and generalisation beyond the simulated training scenes.

  • Core formulation: The model directly represents mechanical systems as 3D meshes, avoiding the indirect and ambiguous view-dependent representation of video.This formulation is motivated by the ubiquity of 3D meshes in applications and parallels physics simulators’ use of 3D models.
  • Core formulation: PhysiFormer predicts full-sequence future vertex trajectories from initial mesh positions, velocities, and material conditions in a single forward pass.It supports rigid, deformable, and mixed-material multi-object dynamics, with mesh topology imposed at inference time.
  • Model design: A diffusion transformer generates trajectories as samples from a distribution of physically plausible futures, capturing mechanics including inertia, gravity, and collisions.The probabilistic formulation can represent uncertainty in properties such as mass and friction that are not provided as conditioning variables, enabling diverse plausible futures.
  • Model design: Factorised self-attention across time, space, and objects improves efficiency while enabling permutation-invariant reasoning over multiple objects without explicit object encoding.The model uses small modifications to a general-purpose Diffusion Transformer for mechanics modelling.
  • Generalisation: Training on simulated rigid or elastic collision scenes generalises to novel object combinations, geometries, configurations, mixed materials, and larger object counts.The training scenes contain either all rigid or all elastic objects undergoing collisions and complex motion.
  • Comparison with autoregressive models: Compared with autoregressive alternatives, PhysiFormer generates the entire trajectory at once, avoiding training–testing mismatch and irreducible error accumulation that can deform rigid objects over time.The introduction reports that PhysiFormer obtains far better performance than autoregressive approaches with the same input formulation.

2 Related Work

Prior work spans traditional physics-based simulation, learned particle and grid dynamics, per-scene optimization, autoregressive visual prediction, and diffusion-based world simulation. These approaches provide physical realism or scalable learning, but face computational, supervision, simulator-assumption, or physical-fidelity limitations.

  • Traditional Physical Simulation: Traditional simulators model motion, materials, contacts, and time integration across discrete and continuum representations, but remain computationally demanding.Their applications include cloth, thin shells, and fluids.
  • Per-scene Optimized Physical Dynamics: Per-scene optimized dynamics combine 3D representations with learned dynamics or inverse-physics optimization, but often require dense multi-view capture or tracking supervision.These methods typically overfit individual scenes to recover deformable object motion from videos and rely on strong simulator assumptions.
  • Learning-based Physics Simulation: Learning-based simulators use convolutional networks for regular grids and graph neural networks to represent particle states, interactions, and message-passing updates.GNNs provide a framework for modeling more general dynamics than regular-grid methods.
  • Autoregressive Prediction in Visual Feature Spaces: Deterministic autoregressive prediction can operate in visual foundation-model feature spaces and decode latent futures into outputs such as depth and segmentation.DINO-Foresight predicts latent futures from four context frames.
  • Diffusion Models for World Simulation: Video diffusion models generate convincing visual simulations, but often violate Newtonian dynamics and fail to learn physical laws from videos alone.These limitations reduce their suitability for applications such as robotics where physical fidelity is critical.

3 Method

PhysiFormer models the conditional distribution of 3D vertex trajectories from initial positions, velocities, and material properties using a diffusion transformer. Its structured attention factorizes space, time, and objects for efficient, order-insensitive multi-object reasoning, while one-shot generation outperforms autoregressive rollout in practice.

  • Trajectory generation: One-shot trajectory generation is substantially more effective than autoregressive rollout and avoids exposure bias from teacher-forced training.The method models the full conditional trajectory distribution rather than iteratively predicting p(Xt+1, Vt+1 | Xt, Vt, y).
  • Structured Attention: The attention design implicitly distinguishes objects without explicit object identifiers and remains insensitive to object ordering in the token sequence.Global and object-level spatial attention provide multi-object structure through factorization rather than object-identity embeddings.
  • Conditioning: The model conditions each vertex on separate initial-position and initial-velocity embeddings, plus rigid or elastic material embeddings broadcast from objects to vertices.Rigid and elastic materials are represented by inputs 0 and 1, respectively.
  • Trajectory modeling: PhysiFormer learns p(X | X0, V0, y), denoising noised vertex trajectories conditioned on initial positions, velocities, and scene context.The model predicts clean trajectories with a Diffusion Transformer backbone and flow-matching objective.
  • Structured Attention: Alternating temporal, full-spatial, and object-level attention reduces cost from O(T^2N^2) to O(TN^2 + NT^2).This factorization lets each vertex attend across time while combining local and global spatial information.

4 Experiments

Experiments evaluate PhysiFormer on synthetic rigid and deformable-object dynamics, using randomized multi-object trajectories generated in Genesis. Results show strong rigidity and physical consistency, generalization beyond training geometries and object counts, and comparable performance from factorized object attention.

  • Experimental setup: The experiments use Genesis-simulated 49-frame trajectories with randomized object counts, sizes, shapes, materials, and initial conditions in a bounded 3D environment.Each scene records initial positions and velocities, ground-truth vertex trajectories, mesh connectivity, and per-vertex rigid or elastic material labels.
  • Physical consistency: PhysiFormer preserves object rigidity over long rollouts, whereas autoregressive baselines diverge through stationary-object motion, boundary escape, and severe shape deformation.The qualitative comparison reports that rigidity is not preserved in ΦAR_ctx1 at t = 10 and that all autoregressive baselines eventually diverge, including TIE r=1.0.
  • Generalization: PHYSIFORMER-L-10k generalises to unseen object geometries and counts, including complex real-world meshes with more vertices than the training primitives.The reported generalization settings extend beyond the geometries and object counts used during training.
  • Inference: PhysiFormer generates plausible motion from initial positions and velocities alone and offers capped inference cost for deformable, contact-rich rollouts after training.The model does not require density, friction, or material parameters as input, according to the experiments discussion.
  • Ablations: Noise scale = 0.1 works best for raw coordinate diffusion, while smaller values hinder generalisation and larger values reduce generation quality.This finding comes from an appendix ablation using smaller DiT-B models trained on 1k data.
  • Ablations: On 10k rigid trajectories, factorized per-object attention has slightly worse MSE and momentum drift ratio but better rigidity loss than alternating full spatial-temporal attention with object-ID embeddings.The reported values are MSE 9.6e-3 vs. 9.1e-3, momentum drift ratio 1.70 vs. 1.53, and rigidity loss 1.9e-5 vs. 4.8e-5.

5 Conclusion

PhysiFormer learns multi-material, multi-object mechanics through full-trajectory diffusion of mesh coordinates in world space. Joint generation improves trajectory accuracy, rigidity preservation, and momentum-based physical consistency over autoregressive baselines while preserving object coherence.

  • 5 Conclusion: PhysiFormer models multi-material, multi-object mechanics as full-trajectory coordinate diffusion in world space.The model is a unified diffusion transformer for future mesh motion.
  • 5 Conclusion: Jointly generating future mesh motion avoids autoregressive error accumulation.Object coherence is preserved through factorised spatial, temporal, and object-level attention.
  • 5 Conclusion: PhysiFormer improves trajectory accuracy, rigidity preservation, and momentum-based physical consistency over autoregressive baselines.

A Method Continued … B Inference Time and Denoising Step Numbers

PhysiFormer extends its DiT architecture with register tokens and standard diffusion-transformer components, while the autoregressive baseline predicts velocities and integrates them into rolled-out positions. Inference uses a default 50-step Heun sampler, with denoising-step tradeoffs evaluated against quality and runtime.

  • A.1 PHYSIFORMER: PhysiFormer prepends Nreg = 16 shared, learnable register tokens to capture global context and potentially remove high-frequency embedding noise.The tokens are replicated across temporal, spatial, object, and vertex dimensions according to the attention block.
  • A.1 PHYSIFORMER: Its DiT blocks use RMSNorm, AdaLN timestep conditioning, gated residuals, QK-normalized non-causal attention, and SwiGLU feed-forward activations.The default PHYSIFORMER mirrors DiT-L with 24 blocks and 16 attention heads per block.
  • A.2 Baseline ΦAR: ΦAR predicts next-timestep vertex velocities from a context window, then recovers positions by integration and iteratively feeds predictions back into the model.Training normalizes positions and velocities using dataset-wide statistics and supervises velocity with SmoothL1 loss.
  • A.2 Baseline ΦAR: The ΦAR_ctx4 variant uses four previous timesteps and rolls out predictions with a sliding window initialized by four ground-truth timesteps.The context window defines the autoregressive dependency horizon.
  • A.2 Baseline ΦAR: Train-time noise injection is used for ΦAR to reduce the train-test domain gap and help correct errors accumulated during autoregressive inference.The tuned noise levels are ϵpos ∼N(0, 0.0082) and ϵvel ∼N(0, 0.082).
  • A.2 Baseline ΦAR: ΦAR is a pre-norm Transformer encoder that tokenizes position and velocity coordinates, uses Fourier features, and masks samples with fewer vertices.Its tokens include a learnable shared base vertex token and temporal embeddings for relative ordering.
  • B Inference Time and Denoising Step Numbers: Inference uses a 50-step Heun sampler by default, and denoising-step numbers are compared with inference time and evaluation metrics on an NVIDIA H100 GPU.The analysis reports that fewer denoising steps can produce outputs of comparable quality, while Table 3 examines the tradeoff explicitly.

C PHYSIFORMER-L-10k Generalization

PHYSIFORMER-L-10k is trained on convex geometries with 1–5 objects per scene, yet generalizes to unseen geometries, object counts, and topologies. It produces physically plausible rigid-body motion beyond these training conditions, including objects with up to 86 vertices.

  • Training scope: PHYSIFORMER-L-10k is trained solely on convex object geometries and scenes containing 1–5 objects.The training distribution uses convex geometries and limits scenes to five objects.
  • Out-of-distribution generalization: PHYSIFORMER-L-10k generalizes to unseen object geometries and object counts not present during training.The evaluation includes novel convex meshes, counts exceeding five objects, and unseen concave geometry.
  • Topology generalization: Up to 86 vertices, PHYSIFORMER-L-10k generates physically plausible motion for unseen convex shapes despite training objects having at most 20 vertices.This demonstrates generalization across object topology beyond the training vertex limit.
  • Geometry and count generalization: Seven objects from seen convex templates and three objects with unseen concave geometry receive physically plausible rigid-body dynamics in the reported settings.Seven objects exceed the training maximum of five.

D PHYSIFORMER Training and Inference Meshes · E Additional Quantitative Analysis

PhysiFormer is trained and evaluated using multiple mesh-template sets, including templates for PHYSIFORMER-L-10k, PHYSIFORMER-B-1k, and PHYSIFORMER. Selected unseen meshes with substantially greater complexity also achieve good inference results.

  • D PHYSIFORMER Training and Inference Meshes: Training uses mesh templates for PHYSIFORMER-L-10k.These templates are shown in Fig. 6(a).
  • D PHYSIFORMER Training and Inference Meshes: Training uses mesh templates for PHYSIFORMER-B-1k.These templates are shown in Fig. 6(a).
  • D PHYSIFORMER Training and Inference Meshes: PHYSFORMER training uses additional mesh templates.The passage identifies templates shown in Fig. 6(a) and (b).
  • D PHYSIFORMER Training and Inference Meshes: Inference includes selected unseen meshes.These meshes are presented separately from the training templates in Fig. 6(c).
  • D PHYSFORMER Training and Inference Meshes: The unseen meshes have a significant increase in complexity.The passage specifically characterizes the selected unseen meshes as substantially more complex.
  • D PHYSFORMER Training and Inference Meshes: The selected unseen meshes still exhibit good PHYSFORMER inference results.This result is reported despite their increased complexity.

E.1 AR Models

The section analyzes autoregressive inference for ΦAR and TIE, showing that rollout errors arise from compounding prediction mistakes. It also examines larger TIE interaction radii but excludes them because they provide only slight metric advantages and depart from the matched neighborhood setting.

  • Inference behavior: ΦAR variants and TIE use autoregressive inference, where previous model outputs become inputs for predicting future timesteps.Table 5 evaluates these models under self-conditioned rollout, reflecting their true inference-time constraints.
  • Inference behavior: Ground truth-conditioned inference produces smaller losses and errors because the model receives in-distribution inputs during prediction.The comparison separates training-setting behavior from self-conditioned rollout behavior.
  • Inference behavior: Autoregressive inference-time failures are attributed to errors compounding during rollout.This explains the gap between ground truth-conditioned evaluation and true self-conditioned inference.
  • Interaction radius: TIE experiments increased the interaction radius to 2.0 and 3.5, with 3.5 effectively connecting every particle through implicit edges.In a 2 × 2 × 2 box, radius 2.0 captures interactions among most particles, while 3.5 connects all particles.
  • Interaction radius: The larger TIE radii were omitted because they offered only slight advantages on certain metrics and did not match the intended small-neighborhood setting.The selected radius was calibrated to particles’ range of motion and matched the TIE configuration.

F Evaluation Metric Discussion · F.1 PHYSIFORMER Chunked Long-Horizon Inference

The evaluation discussion argues that trajectory MSE alone is insufficient for judging physical plausibility, motivating complementary rigidity, momentum, and qualitative measures. It also explores chunked long-horizon inference, which reuses generated states to continue rollouts beyond the 49-frame training length.

  • F Evaluation Metric Discussion: Trajectory MSE against ground truth is used for evaluation, following prior work.
  • F Evaluation Metric Discussion: Deviation from the ground-truth trajectory does not necessarily indicate physical implausibility.
  • F Evaluation Metric Discussion: Rigidity loss, momentum drift ratio, and qualitative visualization provide a more complete assessment of learned physics.Rigidity loss is considered when models are conditioned on rigid materials.
  • F Evaluation Metric Discussion: The discussion calls for improved and more general evaluation metrics for assessing physics learning.
  • F.1 PHYSIFORMER Chunked Long-Horizon Inference: PHYSIFORMER’s inference output length is determined by the 49-frame training-data length.
  • F.1 PHYSIFORMER Chunked Long-Horizon Inference: Chunked rollout uses each previous chunk’s final generated frame as the next initial position and estimates velocity by finite differencing the final two frames.This explores long-horizon inference because only initial position and velocity are required.
  • F.1 PHYSIFORMER Chunked Long-Horizon Inference: Full-horizon rigidity error increases with rollout length.

G Ablations Continued

The ablations examine diffusion noise scale and mesh tokenization, finding that triangle tokens create redundant vertex predictions and poorer performance, especially for object rigidity. The comparison uses a PHYSIFORMER-B-1k model trained on a 1k-sample subset.

  • Noise Scale in Diffusion: Diffusion noise scale is significant for performance, with varying scales evaluated quantitatively in Table 6.The ablation follows the noise-scale setting described in Section 4.8.
  • Mesh Tokenization: Triangle tokenization replaces per-vertex tokens with flattened 9D coordinates for each mesh triangle face, analogous to RenderFormer.Triangle tokens can encode edge geometry, area, and a normal direction, but each vertex appears in multiple triangles.
  • Ablation Setup: The ablation variants use PHYSIFORMER-B-1k with 12 DiT blocks and D = 768, trained on a 1k-sample subset.Table suffixes are omitted for brevity, and the noise scale samples ϵ ∼N(0, I) × ns.
  • Mesh Tokenization: Redundant triangle-based predictions are averaged into one vertex output at inference, yet poorer training-time vertex consistency harms performance, especially object rigidity.Scatter-add and averaging preserve connectivity at inference but do not remove the consistency issue during training.

H Limitations and Future Work

PhysiFormer currently generates 49 frames and performs best with up to 356 vertices because of training-dataset specifications. Future work targets longer generation and spatial compression, while simulator failures and object overlap remain documented inference issues.

  • Current limitations: 49 frames and up to 356 vertices define PhysiFormer’s current generation limits under its training-dataset specifications.The model performs best with up to 356 vertices.
  • Future work: Future work will explore Diffusion Forcing for autoregressive diffusion to expand generation length.The paper also proposes spatial compression using latent encoding or a latent variational autoencoder.
  • Failure cases: Physics-simulator failures arise when boundary contacts are imperfectly resolved, especially at high velocities or with fewer simulation substeps.These failures can allow objects to escape the bounding box.
  • Failure cases: Object overlap is observed during inference.The paper documents overlap examples without quantifying their frequency or impact.
Loading 2606.27364v1…