Source-linked AI summary
LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splatting
Louen Pottier
TL;DR
Existing video-based scene reconstructions replay motion without interaction, while learned physical simulators require generalized-coordinate data from sensors or simulation. LaGSplat combines a latent dissipative Lagrangian with moving Gaussian primitives to infer interactive dynamics from monocular video, matching observed frequencies across seven real systems and outperforming baselines on a soft robot.
Problem
Video-based scene reconstructions replay recorded motion without interaction, while learned physical simulators require generalized-coordinate data from sensors or simulation.
Method
LaGSplat uses one unsupervised low-dimensional latent state as both the generalized coordinate of a dissipative Lagrangian and the conditioning variable for moving Gaussian primitives.
Results
Across seven real systems, natural frequencies match observed values within a few percent when excited, while LaGSplat is most accurate among five models on a two-segment soft robot.
Takeaways & Limitations
The reconstructed object can receive image- or volume-space forces that transfer through decoder Jacobians into the learned dynamics without retraining.
Takeaways & Limitations
The method applies to systems with few generalized coordinates, smooth kinematics, and dissipative dynamics, excluding contact, impact, topology changes, plasticity, and hysteresis.
Abstract
from arXiv · showhide
We present LaGSplat (Latent Lagrangian Gaussian Splatting), a framework that infers interactive, physics-governed dynamics from one or a few monocular videos. At inference it lets a user push on the filmed object, rigid or deformable, with an external force that was never measured, annotated, or seen during training. This is possible because a low-dimensional latent state $\mathbf{q} \in \mathbb{R}^d$ plays two roles at once: it is the generalised coordinate of a learned dissipative Lagrangian and the conditioning variable of a Gaussian Splatting decoder. The inductive bias of this decoder, whose primitives are explicit points $μ_i(\mathbf{q})$ that move with the object, is what lets a force $f$ applied in the image pull back into a latent generalised force $J(\mathbf{q})^\top f$ and enter the equations of motion, which pixel-space (CNN) or neural-field (NeRF) decoders cannot do. We validate LaGSplat on test cases of increasing difficulty, from rigid to deformable and from autonomous to forced real systems, combining monocular video and sensor measurements. We further demonstrate interactive use: forces of arbitrary magnitude and direction can be applied to the reconstructed object at any time, its response rendered in real time, in 2D or 3D. Assuming a dissipative Euler-Lagrange equation over a few generalised coordinates trades generality for a bounded, plausible response to unseen forces, where an unconstrained predictor diverges.
1 Introduction
LaGSplat combines explicit Gaussian primitives with a learned latent Lagrangian to reconstruct filmed objects as real-time, physics-governed interactive simulations. Its moving primitives provide the Jacobian needed to transport image-space forces into latent generalized forces without retraining.
- Motivation: Unlike time-parameterized dynamic scene reconstructions that replay recorded clips, LaGSplat supports interaction with reconstructed objects.Dynamic scene reconstruction provides real-time photorealistic representations, but time-based parameterization does not permit object interaction.
- Contribution: LaGSplat fills a gap by combining a genuine mechanical law with explicit primitives that move with and render the object.Prior approaches provide either structured dynamics without persistent explicit moving primitives or explicit reconstructions without a genuine mechanical law.
- Method: A low-dimensional latent state q is both the generalized coordinate of a learned dissipative Lagrangian and the conditioning variable of a Gaussian Splatting decoder.An encoder maps frames to q, the latent Lagrangian integrates the equations of motion, and the decoder renders q into 2D or 3D images.
- Interactive forcing: Explicit decoder points μ_i(q) move under latent variations, enabling an image-space force f to become the generalized force J(q)^T f through virtual work.This force enters the same equations of motion without retraining, even though no force was applied in the training videos.
- Contribution: The pipeline converts monocular video into an identified physical model and a real-time interactive simulation rendered in 2D or 3D.It combines explicit Gaussian primitives and differentiable rasterization with a latent Lagrangian and generalized-force dynamics.
2 Related Work
Related work spans dynamic rendering, learned mechanics, video-supervised dynamics, and force-interactive models. LaGSplat combines a learned dissipative Lagrangian over latent coordinates with an explicit material-point Gaussian decoder whose Jacobian transports forces.
- Field overview: The literature separates dynamic novel-view synthesis, analytical mechanics, video-generated physical rendering, and learning dynamics directly from footage of the simulated system.The key distinction is whether video is an output or supervision, and whether the model reproduces the filmed system or a plausible scene of its kind.
- Rendering representations: NeRF-based decoders are Eulerian and provide no material point for force attachment, whereas explicit moving primitives provide positions and Jacobians.Neural networks that warp canonical primitives satisfy the attachment criterion, but direct primitives are preferred for motion-rich content.
- Physics and force control: Prescribed-physics methods such as PhysGaussian [37] [45] [74] and VR-GS [29] use external dynamics, while Force Prompting [18] learns force responses without mechanical structure.Force Prompting [18] requires force-video examples, uses a fixed force vector per generated clip, and is neither real-time nor per-frame controllable.
- Latent mechanics: NeuROK [17] and related latent-mechanics methods share latent integration and Jacobian pullbacks, but their decoders render meshes rather than affine Gaussian primitives.Reduced-order graphics methods also use material-point decoders and pullback structure, while LaGSplat learns the governing law over the learned state.
- LaGSplat’s distinction: LaGSplat replaces time with latent coordinates in higher-dimensional Gaussians, learns a dissipative Lagrangian, and transports arbitrary unseen forces through J^T f.Unlike rigid-only force-interactive systems such as Wang et al. [69], it addresses deformable objects and permits forces to be redefined at any instant.
3 Method
LaGSplat learns a continuous latent state that simultaneously parameterizes dissipative Euler–Lagrange dynamics and a state-conditioned Gaussian Splatting decoder. Its learned kinematics pull image-space forces into latent generalized forces, enabling bounded interactive responses to unseen loading while accounting for observation geometry and force-scale ambiguity.
- Architecture and training: LaGSplat jointly learns an encoder, latent Lagrangian with dissipation, and Gaussian Splatting decoder, then freezes the autoencoder before fitting dynamics to the encoded trajectory.Stage 1 fits appearance and fixes the latent chart; Stage 2 fits the Lagrangian after encoding the clip once.
- Architecture and training: A Lipschitz-constrained encoder regularizes temporal continuity because the dynamics objective differentiates the encoded trajectory to obtain q̇ and q̈.The method uses the continuity regularizer of Zhu et al. [81] rather than relying on greater encoder depth or expressiveness.
- Gaussian Splatting decoder: The decoder tiles the space–state volume (x, y, q), so each scene state is stored once and can be rendered from Euler–Lagrange trajectories, unseen initial conditions, or applied forces.Conditioning Gaussian primitives on q produces state-dependent centers and suppresses primitives far from the queried latent state; rendering supports n=2 video output or n=3 interactive 3D scenes.
- Latent dynamics: The learned latent chart is whitened, while errors are measured in the anisotropic pull-back image metric Ḡ that weights latent directions by their decoded pixel motion.This metric approximates decoded image error to first order without requiring rasterization during evaluation.
- Latent dynamics: Dissipation and a potential with a unique minimum guarantee bounded free dynamics that settles to rest and converges to a loaded equilibrium under maintained force.An ICNN models a single convex basin for the pendulum, rocking chair, and hanging bag; the soft continuum robot instead uses an ICNN composed with an i-ResNet to relax convex level sets while preserving a unique equilibrium.
- Interactive forcing: The Gaussian kinematics Jacobians pull primitive forces back into latent generalized forces, with neighborhood contact aggregation and O(nd) inference cost, enabling unseen loading without network differentiation.The response direction and shape are determined by the learned kinematics, but physical force magnitude remains ambiguous up to one global energy factor κ because (L, D) and (αL, αD) produce identical observed trajectories.
4 Experiments
LaGSplat is evaluated on monocular real-object videos spanning autonomous rigid and deformable systems, plus a continuously actuated soft robot with pressure measurements. Across these tests, it recovers plausible dynamics and uniquely supports image- or world-space external forces at inference through decoder-derived Jacobians.
- Overall dynamics evaluation: On nearly every system, the unsupervised natural-frequency error εfreq is 1–6%, while the one-segment robot reaches 31.4% because its recordings do not excite the natural mode.The double pendulum has εq = 0.947, rising to 1.544 over the full clip, whereas the hanging-bag error increases from 0.150 over two periods to 0.396 over seven.
- Soft continuum robot: LaGSplat remains accurate on the harder two-segment robot, achieving 0.769 × 10−3 with d=4 coordinates and outperforming the comparison models there by ×0.98.The result is the median of five seeds, while all four baselines use d=10; the one-segment recordings primarily reveal a quasi-static pressure-to-shape map rather than inertia or natural frequencies.
- Interactive force application: External image forces enter LaGSplat’s learned dynamics at inference as f_lat = J^T f without retraining, using Jacobians induced by the Gaussian map q 7→μ_i.The decoder pulls forces from image primitives into latent generalized coordinates, while training identifies the autonomous dynamics and actuation map; this capability is absent from the compared entries.
- Interactive force application: On the one-dimensional rainbow rocker, equal forces produce lever-arm-dependent and sign-reversing latent shifts, including Δq = −0.50, −0.76, −0.84 and +1.22 across application points.Background Gaussians with null Jacobians do not transmit force, demonstrating that the response follows object motion rather than arbitrary image content.
- Three-dimensional interaction: With multi-view synthetic supervision, the same transport mechanism operates in 3D: a ray-selected Gaussian receives a world-space force and can move the object toward a new equilibrium at q = +1.52.The demonstrations include pushes along directions absent from the training motion, showing interactive responses beyond the observed trajectory.
5 Discussion
LaGSplat’s physics prior improves structure but limits applicability to observable, smooth, low-dimensional dissipative systems, while its force mechanism remains unvalidated against measured forces. Future work targets force-labeled data, hidden constitutive variables, simulator coupling, and improved latent and geometric modeling.
- Limitations: The Lagrangian prior excludes contact, impact, topology changes, plasticity, and hysteresis, restricting the method to few-coordinate, smooth, dissipative dynamics.The stated boundary comes from the instantiated dynamics rather than the representation itself.
- Limitations: Monocular learning requires image-observable state, so out-of-plane motion needs multiple-view supervision and plasticity requires hidden loading-history information.Frames that coincide visually can encode different plastic histories, making such variables invisible to single-frame supervision.
- Limitations: The central force mechanism is not validated against measured forces because no dataset combines fixed-view video, few degrees of freedom, applied point forces, and resulting deformation.Section 4.5 instead checks Jacobian-derived ratios, signs, sums, and pullback-kernel properties invariant to the energy factor κ.
- Limitations: Appearance-first fitting remains a contested design choice, while hand-selecting latent dimension d can overestimate the minimal state size.Friedl et al. [14] report jointly trained models outperforming sequential training, whereas Zhu et al. [81] report a frozen-encoder approach outperforming jointly trained baselines; Chen et al. [8] report reconstruction degrading beyond the minimal latent width.
- Future work: Future work proposes measured-force datasets, internal variables for plasticity and hysteresis, coupling with external simulators, and improved treatment of per-primitive density.Internal-variable models would require larger d and a multi-frame encoder, while simulator coupling must preserve the force mechanism’s isolation test.
6 Conclusion
LaGSplat learns a shared low-dimensional physical state from video alone, using it as both generalized coordinate and Gaussian decoder condition. Across real systems, it recovers observed dynamics accurately and enables bounded, plausible responses to previously unseen forces without retraining.
- LaGSplat learns an unsupervised physical state q that simultaneously parameterizes a dissipative Lagrangian and conditions explicit moving-point Gaussian primitives.The video itself supplies the training signal; q receives no direct supervision.
- Across seven real systems, learned natural frequencies match video observations within a few percent on every recording that excites them.The systems span one to four degrees of freedom.
- On Krauss et al.’s [35] two-segment soft continuum robot data, LaGSplat is the most accurate of five models while using four coordinates instead of ten.The evaluation uses the authors’ own data, split, and protocol.
- At inference, image or reconstructed-volume forces enter the learned equations through decoder Jacobians without retraining, despite no training force measurements.Force direction and shape follow learned kinematics, while amplitude is determined only up to one global energy factor.
- The Lagrangian prior limits the class of reachable systems but keeps responses to unseen forces bounded and physically plausible.
A Derivations
This appendix derives how learned kinematics transports forces and inertia, and explains the energy balance underlying the bounded-response claim.
- A.1: Learned kinematics transports forces and inertia alike.This derivation is covered in Section A.1.
- A.2: The appendix derives the energy balance behind the bounded-response remark.This analysis is covered in Section A.2.
A.1 Forces and inertia from the learned kinematics
The learned kinematics map latent configurations to primitive positions, thereby determining generalized forces and kinetic energy without additional modeling choices. Force transport is invariant to smooth latent-coordinate changes, while normalized primitive opacities make the configuration-dependent mass matrix vary along trajectories.
- A.1 Forces and inertia from the learned kinematics: The learned configuration-to-primitive map determines both generalized forces and kinetic energy by standard analytical-mechanics rules.It plays the role of each particle’s position, so no further modeling choice is required once the map is fixed.
- A.1 Forces and inertia from the learned kinematics: A force applied to a primitive transports into the canonical latent generalized force through the primitive Jacobian and preserves virtual work.This construction makes force injection into the equations of motion consistent with the energy balance.
- A.1 Forces and inertia from the learned kinematics: The transported force and resulting response are invariant to any smooth reparameterization of the latent coordinates.The Jacobian transformation and virtual-displacement transformation cancel, leaving work unchanged across latent charts.
- A.1 Forces and inertia from the learned kinematics: Normalized latent opacities weight equal-mass primitives to produce a configuration-dependent mass matrix.Because the primitive velocities satisfy dot(μ_i) = J_i dot(q), state dependence enters through the gating weights, so the mass matrix varies along trajectories and carries Coriolis terms even when each J_i is constant.
A.2 Bounded forced response
Under positive-definite inertia, coercive potential, and uniformly positive damping, LaGSplat’s continuous-time dynamics remain bounded and dissipative. Free rollouts converge to the unique potential minimum, while maintained constant forces yield bounded convergence to loaded equilibria; symplectic integration limits discrete drift but not far-out-of-distribution quantitative error.
- Assumptions: The bounded-response guarantee relies on positive-definite inertia, a coercive potential floor, and damping with isotropic floor c0 > 0.These assumptions prevent unbounded position drift while ensuring kinetic and dissipative control.
- Free rollout: Free rollouts are bounded and converge to the unique global potential minimum under coercive potential, positive-definite inertia, and uniformly positive damping, by LaSalle’s invariance principle.Energy decreases as Ė ≤ −c0∥q̇∥2, while the energy sublevel set bounds both q̇ and q.
- Maintained force: Maintained constant forces produce bounded responses that converge to loaded equilibria satisfying ∂V/∂q = f.The loaded energy E_f = T + V − f^Tq cancels injected power, and the quadratic potential floor keeps it coercive.
- Discrete rollout: The symplectic scheme preserves long discrete rollouts against drift, but boundedness does not imply quantitative accuracy far from the training data.The guarantees stated for continuous time therefore constrain response growth without ensuring exact extrapolation.
B Deformation-Field Decoder at Equal Budget
At equal training budget, LaGSplat’s affine latent-to-primitive decoder matches or improves reconstruction while producing materially smoother, more meaningful primitive Jacobians than a learned deformation field. The deformation field’s extra capacity adds no image quality but can corrupt the derivatives required for force application and inertia estimation.
- Decoder parameterization: LaGSplat places primitives in an (n+d)D volume conditioned on q, making each μ_i affine in q with a constant Jacobian, whereas the alternative learns nonlinear q-dependent warps.The deformation family can trace arbitrary curves and potentially keep primitives on object parts throughout motion, unlike straight latent-volume trajectories.
- Reconstruction: At equal reconstruction budget, the deformation field trails LaGSplat on L1 (0.0157 vs 0.0142) and SSIM loss (0.076 vs 0.063), despite 9.8M versus 195k parameters.Both decoders use 15,000 primitives, identical losses, regularizers, optimizer, and schedule; only the parameterization differs.
- The Jacobian field: Figure 11 shows smoother primitive velocities for LaGSplat, with median neighbor differences of 1.4 px/s versus 6.9 px/s for the deformation field.The deformation field also moves primitives over static background and rocker regions, indicating a less meaningful Jacobian despite visually similar renderings.
- Why the Jacobian matters: Because frame reconstruction does not supervise material tracking or primitive velocities, a decoder may fit the clip while yielding meaningless ∂μ_i/∂q for force pullback and inertia estimation.Material tracking emerges in LaGSplat because the same primitives must explain every frame, with small shifts between nearby states preserving their covered matter.
C Implementation Details
LaGSplat uses shared encoder, decoder, and LNN architectures across systems while tuning latent dimension, resolution, primitive count, and run length per system. Training uses two Adam-optimized stages with Gaussian lifecycle management and latent whitening procedures.
- Implementation settings: The same architectures are used on every system, while latent dimension, image resolution, primitive count, and run length are set per system.
- Architectures: The continuity-preserving convolutional autoencoder uses three 12×12 convolutions, two channels, stride 2, ELU activations, 4×4 average pooling, and a linear map to q.It has about 2 000 parameters; the decoder contains 1 000–15 000 Gaussians with 104 to 2 × 10^5 parameters, and the LNN has under 15 000 parameters.
- Training: Both training stages use Adam, with Stage 1 jointly training the encoder and decoder for about 5 × 10^4 optimizer steps.The number of epochs depends on clip size.
- Training: Stage 1 optimizes 0.08 L1 + 0.02 (1 − SSIM) + 0.01 anisotropy while periodically recycling Gaussians whose peak opacity falls below 0.02.Recycling splits overloaded Gaussians along the principal eigenvector of their full (n+d)-dimensional covariance or teleports them to high-error regions; latent whitening uses an eigendecomposition of encoded covariance.