PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation

Yu Zhang, Xudong Xu, Xingang Pan

arXiv:2610.07609v1cs.CVcs.GR

TL;DR

High-resolution volumetric deformable simulation makes long-horizon prediction difficult, and the suitable predictive paradigm remains uncertain. PhysLDM uses a holistic spatiotemporal VAE and latent diffusion for one-shot trajectory prediction; the paper reports accurate reconstruction and plausible predictions in chaotic regimes, alongside zero-shot generalization and differentiable applications.

  • Problem

    Long-horizon prediction for high-resolution volumetric meshes is challenging, spatiotemporal compression is largely unexplored for mesh-based physics, and regression versus diffusion has not been systematically compared in this setting.

    Source: §1 · p. 2

  • Method

    PhysLDM uses a holistic spatiotemporal VAE that separates rigid motion from deformation, then predicts the trajectory latent from the initial state with diffusion.

    Source: §1 · pp. 2–3

  • Results

    In chaotic dynamics, diffusion predicts physically plausible trajectories while deterministic regression produces damped averages; reconstruction reaches ∼2.48 mm precision at up to 78× token compression.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    A single model trained purely kinematically generalizes zero-shot to unseen datasets and supports differentiable inverse problems and design optimization.

    Source: §1 · p. 3

    The model currently predicts 64-frame trajectories in a single diffusion pass; extending it autoregressively is future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. Long-horizon prediction for high-resolution 3D volumetric meshes is hard: autoregressive methods are susceptible to error accumulation, while direct multi-frame prediction at native resolution is computationally prohibitive. This motivates a compact spatiotemporal latent representation, which is largely unexplored for mesh-based volumetric physics. Meanwhile, it remains unclear whether deterministic regression or generative diffusion is the more appropriate predictive paradigm. To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot volumetric deformable simulation. Its core is a holistic spatiotemporal VAE that avoids the "staircase" artifacts of standard temporal compression (as in common video VAEs), achieving ~2.48 mm reconstruction precision on meter-scale scenes at up to 78x token compression. Based on this reliable latent space, we systematically compare regression and diffusion methods. Our experiments uncover a key modeling insight: complex deformable dynamics are often chaotic, and in this regime deterministic regression tends to produce non-physical averages, whereas diffusion better models their distribution. Accordingly, we employ a latent diffusion model that effectively learns from the chaotic data to generate physically plausible trajectories. Trained purely kinematically on an Objaverse-scale dataset, a single PhysLDM generalizes zero-shot to unseen OOD datasets (GSO and Toys4K). Its differentiability further enables efficient solution of inverse problems and higher-order design optimization. To our knowledge, PhysLDM is the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation.

1 INTRODUCTION

PhysLDM combines a holistic spatiotemporal latent representation with diffusion to predict high-fidelity volumetric deformable trajectories in one pass. The paper reports accurate reconstruction, plausible predictions in chaotic regimes, zero-shot transfer, and differentiable applications.

Motivation

  • High-precision classical solvers are slow and their gradients are hard to obtain, motivating fast, differentiable neural surrogates for physical interaction and physical AI.The introduction situates deformable simulation in graphics, engineering, robotics, and physical AI.Source: §1 · p. 1
  • High-resolution volumetric meshes make long-horizon prediction challenging: autoregressive rollouts can drift, while native-resolution one-shot prediction is computationally prohibitive.The paper identifies spatiotemporal compression for mesh-based volumetric physics as largely unexplored.Source: §1 · p. 2

Main contributions

  • In chaotic deformable dynamics, diffusion models multiple plausible futures, whereas deterministic regression tends to average modes into damped trajectories.Small microscopic solver non-determinism can be amplified into macroscopically different plausible outcomes.Source: §1 · pp. 2–3
  • ∼2.48 mm average geometric reconstruction precision on meter-scale scenes is achieved at up to 78× token compression by decoupling rigid motion from deformation.The holistic temporal compression scheme eliminates staircase artifacts and preserves fine geometric details.Source: §1 · pp. 2–3
  • PhysLDM combines a high-fidelity spatiotemporal VAE with latent diffusion, and the paper reports zero-shot generalization and support for inverse problems and design optimization.The model is trained purely kinematically on an Objaverse-scale dataset and is described as constitutive-model-agnostic.Source: §1 · p. 3

2 RELATED WORK

Prior work spans accurate but costly physics-based solvers and fast neural simulators, while high-fidelity spatiotemporal compression and systematic regression-versus-diffusion comparisons remain limited for volumetric deformable physics.

Physics-based deformable simulation

  • Classical methods can provide high physical fidelity but are costly, while real-time solvers trade away fidelity and differentiable simulators face gradient-computation burdens.PhysLDM is positioned as a fast, end-to-end differentiable neural surrogate learned from high-accuracy solvers.Source: §2 · p. 3

Neural simulators for deformable dynamics

  • Autoregressive neural simulators predict frame by frame and may drift; concurrent one-shot work is limited to low-resolution surface meshes and a small template set.That concurrent work trains on about 356 vertices per scene across 35 meshes.Source: §2 · p. 3

Spatiotemporal latent representations

  • Spatiotemporal autoencoders remain largely unexplored for high-fidelity volumetric deformable simulation, and standard video-VAE temporal compression can produce staircase artifacts.Existing physics autoencoders typically compress only spatially, while video architectures do not transfer directly to deformable physics.Source: §2 · p. 4

Prediction paradigms: regression versus diffusion

  • Regression and diffusion had not been systematically compared for deformable-body simulation; the paper finds diffusion more suitable under chaotic dynamics, where regression may collapse to non-physical averages.Most neural simulation work uses regression, while diffusion has commonly been used for domains such as image, video, and human-motion generation.Source: §2 · p. 4

3 METHODOLOGY

PhysLDM factors volumetric trajectories into rigid motion and canonical deformation, encodes them in a compact joint latent, and predicts that latent from the initial state. Its training separates spatial VAE, spatiotemporal VAE, and diffusion-model stages.

3.1 PROBLEM FORMULATION

  • Each future frame is represented as a rigid transform of the rest geometry plus a canonical deformation, separating pose-independent local deformation from global motion.The initial state includes rest geometry, frame-0 pose, linear and angular velocity, and Young’s modulus; rotations and centroids are obtained by per-frame Kabsch fitting.Source: §3.1 · p. 4

3.2 ARCHITECTURE OVERVIEW

  • The Spatial VAE maps each frame’s canonical mesh to a fixed number of latent tokens, independent of vertex count, and decodes per-vertex displacements from them.The framework overview places this module before spatiotemporal compression and latent prediction.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
  • The ST-VAE compresses the trajectory into one joint latent using holistic temporal tokens for deformation and a dedicated temporal pathway for rigid motion.Its decoder is frozen during diffusion training; the dedicated rigid path avoids capacity competition with deformation tokens.Source: §3.2 · p. 5
  • Conditioned on the initial state, the LDM predicts the full-trajectory latent in one pass, which frozen decoders turn into the full trajectory.The initial mesh is injected through cross-attention, while pose, velocity, and stiffness condition the network through adaptive layer normalization.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 TRAINING

  • Training proceeds from spatial VAE pretraining to ST-VAE training and then flow-matching training of the LDM on the frozen ST-VAE.Supervision is purely kinematic: reconstruction losses use geometry, including vertex positions, velocities, and rigid poses, without constitutive models.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

PhysLDM evaluates holistic spatiotemporal reconstruction and one-shot trajectory prediction for deformable meshes, including ablations and differentiable applications. The results favor diffusion for physically plausible prediction and show that the model supports material estimation and trajectory design.

4 EXPERIMENTS

  • The experiments use GPU FEM-generated trajectories and train the main ST-VAE and LDM results on single-object scenes, with GSO and Toys4K used for OOD evaluation.Source: §4 · p. 6
  • Fixed temporal compression blocks can make deformation hold and then snap between states, so DCC helps assess motion timing when snapshots obscure staircase artifacts.The evaluation also includes GSO and Toys4K OOD sets and meshes beyond the 5,000-vertex training cap.Source: §4 · pp. 6–7
  • 2.48 mm mean geometric reconstruction error on held-out scenes at up to 78× compression is about half the standard baseline’s error.The holistic ST-VAE’s DCC exceeds 0.93 across evaluation sets, while standard temporal compression produces staircase artifacts.Source: Table 1 · p. 7, §4 · p. 7
  • 22.7% volume violations for regression versus 7.9% for the LDM on held-out scenes; across metrics and datasets, the LDM outperforms both baselines.The comparison uses physics-consistency checks because chaotic dynamics make world-space error alone insufficient; the AR baseline also has 80.6 mm world-space error against 28.2 mm for the LDM.Source: Table 2 · p. 8, §4 · p. 8

4.1 ABLATIONS

  • Without the ST-VAE, native-resolution one-shot diffusion runs out of memory at 2,600 vertices on a 140 GiB H200; with the latent, cost is independent of vertex count.The latent also turns sampler noise into slight coherent shape shifts rather than per-vertex jitter.Source: §4.1 · p. 9
  • Routing the rigid token through shared holistic temporal compression causes translation error to explode and reconstructed deformation to shrink drastically.Source: §4.1 · p. 9

4.2 DIFFERENTIABLE APPLICATIONS

  • PhysLDM recovers Young’s modulus within 0.06 dex in all seven cases in 40 Adam steps, taking about 40 seconds per run.The model also supports higher-order trajectory-sensitivity optimization: the optimized throw makes rigid motion 13× more sensitive to stiffness in the ground-truth simulator.Source: §4.2 · p. 9

5 CONCLUSION

PhysLDM combines a holistic spatiotemporal VAE with latent diffusion to predict deformable-body trajectories, producing physically consistent trajectories where deterministic regression produces damped, non-physical averages. The conclusion also identifies 64-frame single-pass prediction as the current scope and autoregressive extension as future work.

5 CONCLUSION

  • PhysLDM predicts physically consistent trajectories in one diffusion pass, while deterministic regression produces damped, non-physical averages in the chaotic regime.The approach is trained purely kinematically and provides end-to-end differentiability for inverse design optimization.Source: §5 · p. 10
  • The model currently predicts 64-frame trajectories in a single diffusion pass; extending it to autoregressive prediction remains future work.Source: §5 · p. 10

A.1 STAGE 1: SPATIAL VAE

  • The spatial VAE encodes each frame into a fixed set of learned latent tokens and decodes vertex displacements, with mesh topology used for refinement.The three training stages take about seven days on eight H200 GPUs; Table 3 distinguishes ST-VAE Small from the Base configuration used for PhysLDM.Source: §A.1 · p. 17, A ARCHITECTURE AND TRAINING DETAILS · p. 17, Table 3 · p. 18

A.2 STAGE 2: ST-VAE

  • The ST-VAE compresses temporal sequences with learned queries, keeps rigid motion on a separate path, and decodes the latent representation back into per-frame latents.The separate rigid path prevents the much larger rigid signal from taking over the shared latent and starving deformation.Source: §A.2 · p. 18

A.3 STAGE 3: LDM

  • The LDM and capacity-matched regression model share conditioning, but the LDM learns a token-weighted velocity objective along a noise-to-latent path.The LDM uses rectified flow and Euler sampling; its loss up-weights the rigid token relative to deformation tokens.Source: §A.3 · p. 18, §A.4 · p. 19

A.4 LOSS DETAILS

  • The ST-VAE objective combines deformation reconstruction across frames, rigid-track losses, and a KL term.The component losses are balanced with learned uncertainty weights, and the KL term uses per-channel free bits.Source: §A.4 · pp. 18–19

B.1 DATA DETAILS

  • Training data comprise 63,397 single-object and 16,078 multi-body trajectories, with held-out scenes and separate GSO and Toys4K OOD evaluations.Training uses Objaverse meshes and 64-frame trajectories under a Stable Neo-Hookean material.Source: §B.1 · p. 19

B.2 EVALUATION SETUP

  • Evaluation uses 300 scenes per set for prediction, with 50 Euler steps for the LDM and per-scene metrics reported as medians.DCC detects staircase artifacts but is amplitude-invariant, so it should be considered alongside other metrics.Source: §B.2 · p. 19, §B.3 · p. 19

B.4 METRICS OF TABLE 2

The metrics assess physical consistency alongside trajectory error, while the staircase and chaos analyses show how temporal compression and impact-sensitive dynamics complicate reconstruction and prediction.

B.4 METRICS OF TABLE 2

  • Physical checks supplement trajectory error with volume, self-intersection, speed and acceleration violations, and deformation-to-rigid kinetic-energy ratio.Violation rates count vertex-frame samples whose predicted speed or acceleration exceeds twice or falls below half the ground-truth value; every model has a kinetic-energy ratio below 1, with regression lowest on all three evaluation sets.Source: §B.4 · p. 20
  • World-coordinate error is only a reference because chaotic dynamics can make the ground-truth trajectory one of several possible outcomes.The metric is the per-vertex Euclidean error averaged over vertices and frames, reported as the median across scenes.Source: §B.4 · p. 21

C.1 SYMPTOM AND DIAGNOSIS

  • Standard compression, its variants, and adaptive compression delay rapid motion and then overshoot, while the holistic ST-VAE follows ground-truth timing and is best or tied on every scene.Figure 5 plots deformation-change curves across five scenes; Figure 6 shows corresponding silhouette changes around impact in one scene.Source: §C.1 · p. 21, Figure 5 · p. 22, Table 4 · p. 22

C.2 CAUSE

  • In the standard baseline, errors occur between accurate latent-frame positions because fixed blocks lose information about when rapid changes happen within each block.The motion lags early and catches up late within blocks; smoothing across block boundaries does not remove the artifact.Source: §C.2 · p. 21

C.3 WHY THE VARIANTS AND ADAPTIVE COMPRESSION DO NOT HELP

  • Velocity and spectral losses leave fixed-block local upsampling intact, while adaptive compression can make steps fewer but steeper and delay rebound and settling.The direct world-coordinate baseline has high DCC but lower precision, so its weakness is precision rather than timing.Source: §C.3 · p. 21

D.1 SETUP; D.2 THE PERTURBATION IS TINY

  • Five runs of a soft racket scene from byte-identical inputs remain practically identical before impact, despite solver nondeterminism and tiny perturbations.The experiment uses the same mesh, initial state, material parameters, and solver configuration; pre-impact differences are at most 0.02 mm after the first step.Source: §D.1 · p. 24, §D.2 · p. 24, D CHAOS IN TRADITIONAL GPU SIMULATORS · p. 24

D.3 AMPLIFICATION AFTER IMPACT; D.4 WHERE THE CHAOS COMES FROM

  • After impact, racket runs diverge by 10%–51% of object size, and controlled perturbations show divergence does not grow with perturbation magnitude.Discrete contact choices can redirect the bounce and final pose; thin, branchy, or non-convex soft objects are more prone to amplification.Source: §D.3 · p. 24, §D.4 · p. 24

D.5 HOW COMMON IS CHAOS ACROSS SCENES?

  • Across 300 held-out scenes, median run differences are 0.5 mm over trajectories and 1.0 mm at the last frame, but about a quarter end over 1% of object size apart.The most divergent scene ends 35% apart; on these divergent scenes, runs agree to sub-millimeter precision until ground contact.Source: §D.5 · pp. 24–25, Table 5 · p. 25

D.6 REGRESSION VERSUS DIFFUSION ON CHAOTIC AND NON-CHAOTIC SCENES

The section tests regression and diffusion on chaotic and non-chaotic scenes, comparing single predictions with multiple diffusion samples. It also describes the reconstruction and prediction baselines used in the wider comparisons.

D.6 REGRESSION VERSUS DIFFUSION ON CHAOTIC AND NON-CHAOTIC SCENES

  • A single diffusion sample improves on regression in every metric on both subsets, outperforming it in 79% of non-chaotic and 65% of chaotic scenes.The subsets contain 229 non-chaotic and 71 chaotic scenes; Table 6 reports their medians over scenes.Source: Table 6 · p. 26, §D.6 · p. 26
  • Best-of-eight sampling lowers median error by 9.3 mm on chaotic scenes versus 3.4 mm on non-chaotic scenes, and the second solver run falls within the sample spread in 99% of chaotic scenes.The best-of-eight figures require ground truth for sample selection and are an upper bound, not a usable predictor.Source: §D.6 · p. 26
  • For a soft key with two distinct solver outcomes, regression predicts an intermediate pose resembling neither, while diffusion samples fall on both sides.Among six scenes with solver runs ending more than 30° apart, a diffusion sample ends within 30° of a solver run in five scenes, versus one for regression.Source: §D.6 · p. 26, Figure 8 · p. 27
  • The evidence supports a paradigm difference: diffusion samples vary with solver-run separation, whereas a deterministic model produces one prediction per initial state.The benefit of best-of-eight selection is 9.3 mm on chaotic scenes and 3.4 mm on non-chaotic scenes with the same model.Source: §D.6 · p. 26

E BASELINES

  • The baselines compare alternative learned representations of the same trajectories and alternative dynamics models given the same initial state.The prediction baselines receive the same initial state as PhysLDM.Source: E BASELINES · p. 27, §E.3 · p. 28

E.1 RECONSTRUCTION BASELINES (TABLES 1 AND 7)

  • Reconstruction baselines include strided temporal compression, velocity- and spectral-loss variants, curvature-based adaptive frame placement, and direct world-coordinate decoding without rigid/deformation factoring.The spectral loss matches deformation-change energy by frequency but does not encode when rapid changes occur; adaptive compression places latent frames non-uniformly along time.Source: §E.1 · pp. 27–28

E.2 FULL RECONSTRUCTION RESULTS (TABLE 7)

  • Table 7 reports the full reconstruction comparison across all six ST-VAEs, including variants and adaptive compression omitted from Table 1.The table covers in-distribution and zero-shot GSO and Toys4K sets, with separate large-mesh subsets.Source: §E.2 · p. 28

E.3 PREDICTION BASELINES (TABLE 2)

  • Prediction baselines include a capacity-matched, one-pass latent regression model and an MGN-style autoregressive graph network that predicts frames step by step.The autoregressive baseline predicts world coordinates and accumulates error over time, drifting from ground truth.Source: §E.3 · p. 28

E.4 FULL PREDICTION RESULTS (TABLE 8)

The full prediction comparison reports results across three evaluation sets and includes six physics checks. It also provides the full reconstruction comparison used as context for prediction results.

E.4 FULL PREDICTION RESULTS (TABLE 8)

  • Table 7 gives the full ST-VAE reconstruction comparison across evaluation sets, including in-range scenes and large meshes beyond the 5,000-vertex training cap.Large-mesh subsets report mean deformation and rigid errors only.Source: Table 7 · p. 29
  • Table 8 compares prediction on the same 300 scenes per evaluation set within the training vertex range across three evaluation sets.It reports the full version of Table 2.Source: Table 8 · p. 29
  • The six checks cover volume, speed, and acceleration violations, self-intersection, kinetic-energy ratio, and world-coordinate error, with ST-VAE reconstruction as a reference row.The table identifies best and second-best among the three prediction models within each block.Source: §E.4 · p. 28, Table 8 · p. 29

F MORE ABLATIONS

The ablations examine the ST-VAE, rigid pathway, spatial-VAE training, and decoder GNN; inference runtime and additional qualitative results are also reported.

F.1 THE ST-VAE

  • Without the ST-VAE, raw-token training runs out of memory at 2,600 vertices, below the dataset median of about 3,400; with it, the LDM costs about 4.4 GiB per trajectory at any resolution.The ST-VAE stage scales by about 8 GiB per 1,000 vertices, and ST-VAE Base trains up to about 14,000 vertices on the same card.Source: §F.1 · p. 30, Table 9 · p. 30

F.2 THE RIGID PATHWAY

  • Routing the rigid token through shared temporal compression raises translation error from 1.8 to 887 mm and shrinks deformation to 28% of its true magnitude; a dedicated pathway reaches 8.3 mm world error.Normalizing the rigid signal barely helps, changing world error from 52.8 to 49.6 mm.Source: §F.2 · p. 30

F.3 SPATIAL VAE PRETRAINING

  • Jointly training both VAEs diverges at the standard learning rate and yields about 5× the geometric reconstruction error at a reduced rate, with weaker deformation and volume loss.The authors attribute the poor result to the changing latent target; pretraining fixes that target first, and freezing or fine-tuning the pretrained spatial VAE gives similar results.Source: §F.3 · pp. 30–31

F.4 THE SPATIAL DECODER GNN

  • The decoder GNN couples neighboring vertices; removing it degrades reconstruction and stretches mesh edges, while moving it to the encoder performs worse on every metric.The authors relate the encoder result to smoothing that removes detail before it reaches the latent.Source: §F.4 · p. 31

G INFERENCE RUNTIME

  • On an H200, the latent model’s cost stays independent of mesh resolution; in bf16, the LDM runs at about 40–80 fps and regression reaches up to 195 fps, using about 5 GiB.The ST-VAE decoder is the only resolution-dependent stage and takes 318–384 ms across 980 to 9,851 vertices; the last three scenes exceed the training cap.Source: G INFERENCE RUNTIME · p. 31, Table 10 · p. 32

H ADDITIONAL QUALITATIVE RESULTS

  • The additional qualitative section extends the main-text figures, lists scene parameters, and includes an extreme case.The passage does not state further outcomes for this section.Source: H ADDITIONAL QUALITATIVE RESULTS · p. 32

H.1 MORE RECONSTRUCTION AND PREDICTION RESULTS

Additional scenes show the reported reconstruction and prediction patterns beyond the main-text examples. The figures include a large, highly detailed soft-body reconstruction and a multi-object prediction scene.

H.1 MORE RECONSTRUCTION AND PREDICTION RESULTS

  • Across additional scenes, reconstruction baselines can look reasonable in static frames despite staircase artifacts, while direct world-coordinate decoding is noisy and regression and autoregression show wrong physics or drift.The prediction figures compare ground truth, diffusion, regression, and the MGN-style autoregressive baseline.Source: §H.1 · p. 32, Figure 9 · p. 33, Figure 10 · p. 34
  • One scene predicts 26 objects from GSO, Toys4K, and held-out Objaverse, with 1,517–9,821 nodes per object; soft objects deform on landing while stiff ones retain their shape.Each object is predicted independently as a 64-frame trajectory from its initial state.Source: §H.1 · p. 32
  • Figures 9 and 10 show reconstruction on a Toys4K cake, a GSO shape-sorter toy, a held-out Objaverse object, and a GSO turboprop; Figure 11 adds bucket, sofa, flower-pot, and plush-dog predictions.Scene parameters are listed in Tables 11 and 12.Source: Figure 9 · p. 33, Table 12 · p. 33, Figure 10 · p. 34
  • For the 9,021-node soft birdcage, reconstruction recovers the collapsed overall shape but distorts and tangles its slender bars, with 9.9 mm mean error versus 5.3 mm for the teaser mug.The cage has 1.8× the 5,000-vertex training cap.Source: Figure 12 · p. 36

I EXTREME CASES ANALYSIS

Extreme thin-feature geometry, rather than mesh size alone, limits reconstruction detail, while the method extends to multi-body scenes without architectural changes. The ST-VAE reconstructs and the LDM predicts whole scenes through collisions and ground contact.

I EXTREME CASES ANALYSIS

  • 9.9 mm mean trajectory reconstruction error on the 9,021-node birdcage is about twice the held-out mean for meshes above the training cap, with distortion localized to thin bars.Despite the distorted bars, the collapsed cage’s overall shape, pose, and motion are recovered.Source: I EXTREME CASES ANALYSIS · p. 36
  • Node count alone is not the primary bottleneck: the 11,621-node teaser mug has a lower 5.3 mm reconstruction error than the birdcage.The birdcage’s many independently deforming thin bars require high-frequency detail that the fixed-size latent cannot preserve all at once.Source: I EXTREME CASES ANALYSIS · p. 36

J MULTI-BODY EXTENSION

  • Multi-body scenes use a single global rigid transform across bodies, while a shared deformation field captures their relative motion, with no ST-VAE or LDM architectural changes.All scene vertices are encoded through the same K query tokens.Source: J MULTI-BODY EXTENSION · p. 37
  • The ST-VAE reconstructs multi-body collisions and ground contacts, and one diffusion pass predicts each scene’s bodies, collisions, and landing behavior end to end.The scenes contain up to three bodies with independently randomized Young’s moduli and initial velocities.Source: J MULTI-BODY EXTENSION · p. 37, Figure 13 · p. 37, Figure 14 · p. 38

K DIFFERENTIABLE APPLICATIONS: SETUP AND RESULTS

The differentiable applications test material recovery and throw design, while the concurrent-model comparison examines memory, training cost, and resolution limits. PhysLDM supports higher-order optimization and fits latent training to vertex counts that exceed the raw-coordinate model’s limits.

K.1 MATERIAL ESTIMATION

  • All seven Young’s moduli are recovered within 0.06 dex in 40 Adam steps, taking about 40 seconds.Starting guesses are up to 1.1 dex from ground truth; DiffIPC needs manual solver tuning and about 26 minutes, while Newton/Warp fails after consuming 49 GB for a 131k-substep tape.Source: §K.1 · pp. 38–39, Figure 15 · p. 39

K.2 BEYOND MATERIAL ESTIMATION: DESIGNING THE THROW

  • Designing a throw increases rigid-motion sensitivity to stiffness by 13× and enables material recovery from a distant initial guess where the original throw stalls.The design is local to E⋆=6·10^4 Pa and averages sensitivity over eight small release-velocity perturbations; it takes 30 Adam steps, or 7 minutes.Source: §K.2 · p. 39, Figure 16 · p. 40

K DIFFERENTIABLE APPLICATIONS: SETUP AND RESULTS

L COMPARISON WITH CONCURRENT WORK

  • The comparison uses a concurrent diffusion transformer that predicts full-resolution vertex-by-frame tokens, so its cost grows with vertex and frame count.The models cannot be directly compared on the full dataset because the raw-coordinate model cannot train at that resolution.Source: L COMPARISON WITH CONCURRENT WORK · p. 41

L.1 MEMORY AND RESOLUTION LIMITS

  • The raw-coordinate DiT-L runs out of memory above 1,000 vertices and DiT-B above 2,750, while neither fits the dataset’s median 3,400-vertex scene.The published-backbone comparison uses batch 1 on a 140 GiB H200; the latent LDM trains at about 4.4 GiB and 27 ms per trajectory across vertex counts.Source: §L.1 · p. 41, L COMPARISON WITH CONCURRENT WORK · p. 41

L.2 TRAINING SETTING AND COST

  • Under matched scenes and GPUs at up to 1,000 vertices, the full three-stage pipeline trains in less time than the concurrent single-stage model.Most of the pipeline’s training time goes to the ST-VAE rather than the LDM.Source: §L.2 · p. 41

L.3 PREDICTION RESULTS COMPARISON

On four held-out scenes, PhysLDM preserves geometric detail that the concurrent model renders noisily, despite both placing objects roughly correctly. Its inference is faster, and its kinematic input representation accommodates diverse energy models and point-cloud data formats.

L.3 PREDICTION RESULTS COMPARISON

  • On four held-out scenes, PhysLDM preserves smooth geometric detail while the concurrent model’s predictions show noisy, rippled or corrugated surfaces.Both models place objects roughly where ground truth places them; the comparison is at the last trajectory frame.Source: §L.3 · p. 41, Figure 17 · p. 42

L.4 INFERENCE COST COMPARISON

  • At about 1,000 vertices, PhysLDM predicts 64 frames in 1.5 s at 50 steps and 0.8 s at 20 steps, roughly 9× and 6× faster than the concurrent model.The LDM denoises a fixed 4,104 latent tokens, while the concurrent model denoises N × 64 vertex tokens.Source: §L.4 · p. 42

M TRAINING ACROSS ENERGY MODELS AND ON REAL CAPTURED TRAJECTORIES

  • Kinematic-only training uses no constitutive model, allowing the same pipeline to learn across energy models and from real trajectories without an explicit energy model.Its vertex-position and graph input also supports surface meshes and point clouds, including cloth and rope.Source: M TRAINING ACROSS ENERGY MODELS AND ON… · p. 43

M.1 DATA

  • The evaluation includes 40 simulated scenes under Stable Neo-Hookean and ARAP energies, plus 22 real captured sequences of soft objects, cloth, and ropes.The captured sequences are point clouds tracked over time, with no mesh or explicit energy model, and involve human manipulation on a table rather than objects falling onto the ground.Source: §M.1 · p. 43

M.2 ZERO-SHOT RESULTS

The pretrained ST-VAE reconstructs unseen energy-model trajectories and real point clouds without adaptation, but zero-shot LDM prediction fails on the real sequences. Fine-tuning the LDM with unchanged training code improves predictions across the new data.

M.2 ZERO-SHOT RESULTS

  • ST-VAE reconstruction errors are 4.5 mm for Stable Neo-Hookean, 4.6 mm for ARAP, and 6.0 mm on real point clouds without adaptation.These tests span an unseen energy model and out-of-domain real captures, including cloth and rope absent from training.Source: §M.2 · pp. 43–44, Table 16 · p. 44
  • Zero-shot LDM error is 37.5 mm on ARAP and 875 mm on real sequences, where the learned falling-object setting mismatches objects resting on a table until hand-manipulated.On Stable Neo-Hookean simulations, prediction error is 23.9 mm; the ARAP kinetic energy ratio drops to 0.79.Source: §M.2 · p. 44

M.3 FINE-TUNING WITH THE UNCHANGED PIPELINE

  • Fine-tuning with the unchanged trainer reduces prediction errors by about two thirds on both explicit energy models, to 7.6 mm and 8.8 mm.For ARAP, the kinetic energy ratio rises from 0.80 to 0.92.Source: §M.3 · p. 44, Table 17 · p. 44
  • On real point clouds, fine-tuning reduces prediction error from 921 mm to 6.2 mm, and the unchanged pipeline also covers cloth and rope.The comparison includes tracked points, ST-VAE reconstruction, pretrained zero-shot predictions, and fine-tuned predictions.Source: §M.3 · p. 44, Figure 18 · p. 45
Loading 2610.07609v1…