PaperlayerView exact revision on arXiv

Source-linked AI summary

LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splatting

Louen Pottier

arXiv:2608.16324v1cs.CVcs.LG

TL;DR

Learning interactive physical dynamics from video calls for combining visual reconstruction with mechanics without measured generalized coordinates. LaGSplat couples a learned dissipative Lagrangian and a moving-point Gaussian Splatting decoder through a shared low-dimensional latent state. Across seven real systems, it recovers natural frequencies within a few percent on recordings that excite them.

  • Problem

    The research gap is learning interactive physical dynamics directly from video rather than from measured generalized coordinates or simulator data.

    Source: §1 · p. 2

  • Method

    LaGSplat couples a learned dissipative Lagrangian and a moving-point Gaussian Splatting decoder through one unsupervised low-dimensional latent state.

    Source: §1 · p. 2

  • Results

    Seven real systems demonstrate natural-frequency recovery within a few percent of video-observed values on recordings that excite their natural modes.

    Source: §6 · p. 19

  • Takeaways & Limitations

    Users can apply previously unseen forces to reconstructed objects without retraining, with force amplitude determined only up to a global energy factor.

    Source: §6 · p. 19

    The instantiated dynamics assume few generalized coordinates, smooth kinematics, and autonomous or measured-input dissipative motion, excluding contact, impact, topology changes, plasticity, and hysteresis.

    Source: §5 · p. 17

Abstract

from arXiv · show

We present LaGSplat (Latent Lagrangian Gaussian Splatting), a framework that infers interactive, physics-governed dynamics from one or a few monocular videos. At inference it lets a user push on the filmed object, rigid or deformable, with an external force that was never measured, annotated, or seen during training. This is possible because a low-dimensional latent state $\mathbf{q} \in \mathbb{R}^d$ plays two roles at once: it is the generalised coordinate of a learned dissipative Lagrangian and the conditioning variable of a Gaussian Splatting decoder. The inductive bias of this decoder, whose primitives are explicit points $μ_i(\mathbf{q})$ that move with the object, is what lets a force $f$ applied in the image pull back into a latent generalised force $J(\mathbf{q})^\top f$ and enter the equations of motion, which pixel-space (CNN) or neural-field (NeRF) decoders cannot do. We validate LaGSplat on test cases of increasing difficulty, from rigid to deformable and from autonomous to forced real systems, combining monocular video and sensor measurements. We further demonstrate interactive use: forces of arbitrary magnitude and direction can be applied to the reconstructed object at any time, its response rendered in real time, in 2D or 3D. Assuming a dissipative Euler-Lagrange equation over a few generalised coordinates trades generality for a bounded, plausible response to unseen forces, where an unconstrained predictor diverges.

1 Introduction

LaGSplat learns a physical model and Gaussian scene representation from monocular video, enabling real-time interaction with previously unmeasured forces. A shared latent state connects learned Lagrangian dynamics to explicit points that move with the object.

1 Introduction

  • Prior video-based methods use unconstrained dynamics [62], prescribed equations [7, 35, 69], fixed pixels [21] [35], or no reconstruction [7].Dynamic reconstruction [31] [75] replays recorded motion without interaction, whereas physical-system learning [11] [21] [48] requires observed generalized-coordinate trajectories. LaGSplat instead identifies the particular filmed system’s physics from its own video and combines mechanical laws with explicit moving primitives.Source: §1 · p. 2
  • Latent coordinates emerge without state annotations or an external simulator, jointly constrained by photometric reconstruction and an Euler–Lagrange residual.An encoder maps frames to latent states, a latent Lagrangian neural network integrates the learned equations, and the Gaussian decoder renders the resulting states.Source: §1 · p. 2
  • Explicit moving primitives make image-applied forces actionable: their Jacobian maps a force f into the latent generalized force J(q)^T f without retraining.This mapping depends on the learned kinematics rather than the loading, so autonomous training video can determine it without examples of applied forces.Source: §1 · p. 2
  • Force responses are restricted to training-visited latent states, have an unresolved global amplitude scale, and cannot activate motion modes absent from the video.With well-identified dynamics and correctly moving primitives, response direction and shape are correct; one force–displacement measurement calibrates amplitude, otherwise its scale is chosen manually. Unobserved modes lack latent coordinates and behave as infinitely rigid.Source: §1 · p. 2
  • LaGSplat couples a learned dissipative Lagrangian and Gaussian Splatting decoder through one latent state, enabling real-time interactive simulation from monocular video.The latent state serves simultaneously as generalized coordinates and decoder conditioning. Rendering supports 2D, or novel 3D viewpoints with depth or multi-view supervision.Source: §1 · pp. 2–3
  • LaGSplat converges to a stable equilibrium absent from training, unlike a motion predictor without a mechanical prior, while targeting systems with few generalized coordinates.Validation spans a simple pendulum, an oscillating rigid body, a hanging bag, and the pneumatic actuator of Krauss et al. [35], covering rigid/deformable and autonomous/forced systems.Source: §1 · p. 3

2 Related Work

The section distinguishes dynamic view synthesis, learned analytical mechanics, physically plausible video generation, and simulation learned from a filmed system’s own video. LaGSplat adopts the last setting, pairing latent dissipative mechanics with moving Gaussian primitives through which interactive forces enter the dynamics.

2.1 Dynamic novel-view synthesis

  • Latent-conditioned Gaussian primitives supply moving material points, constant position Jacobians, and real-time rasterization, allowing interactive forces to enter in closed form.Dominant NeRF formulations [53] [56] [61] lack material points, although object-outward deformation [22], particle radiance fields [44], and PAC-NeRF [40] provide usable alternatives. LaGSplat replaces time with latent coordinates in Yang et al.’s [75] higher-dimensional Gaussians and rasterizes with gsplat [76]; unlike canonical warping, this preserves affine position maps, while FreeTimeGS [70] also favors directly placed extended-space primitives for motion-rich content.Source: §2.1 · p. 3

2.2 Learned analytical mechanics

  • Learned latent mechanics already supports virtual-work force pullback, but existing models obtain states and forces through instrumentation rather than video-rendered material points.L-LNN transports forces through encoder Jacobians, while Friedl et al. [14] [15] use decoder Jacobians; LaGSplat inherits the transport, not its instrumented provenance. Relevant training difficulties include position-only identifiability [5], non-convex potentials [30], ill-conditioned mass matrices [23, 34], and insufficient long-horizon stability from single-step derivative matching [24] [27] [36].Source: §2.2 · p. 5

2.3 Generating physically plausible video

  • Prescribed-physics video generators can support interventions, but their governing laws are not learned from the filmed object’s observed motion.VR-GS [29] supports sustained real-time interaction, but uses hand-set materials and a static multi-view scan. PhysDreamer calibrates stiffness against diffusion-generated video under prescribed MPM physics, then responds to initial impulses rather than sustained forces, costing about a minute of simulation per rendered second.Source: §2.3 · p. 5
  • Force Prompting [18] learns force-conditioned video without mechanics, requiring force-video training pairs and fixing the force before generation rather than permitting causal interaction.Its generation is neither real-time nor per-frame causal. LaGSplat instead accepts previously unseen forces that can change at any instant, transporting them through material points into the identified dynamics by virtual work.Source: §2.3 · pp. 5–6
  • NeuROK [17] shares latent Euler–Lagrange dynamics and interactive virtual-work forcing, but learns transferable mesh mechanics rather than identifying a system from its own video.Its nonlinear mesh decoder supplies the pullback metric G(q) = J^T J and force Q = J^T F over user-selected regions and time windows, unlike LaGSplat’s affine Gaussian decoder. LaWM [73] instead learns a conservative latent Lagrangian from simulator renders with auxiliary signals when available, and its rollouts remain unforced.Source: §2.3 · p. 6

2.4 Learning and simulating from video supervision

  • Video-supervised simulation identifies the filmed system and extrapolates beyond its recording, but unstructured evolution rules provide neither mechanical force inputs nor guaranteed physical plausibility.Chen et al. [8] discover compact states but predict through regenerated pixels; Zhu et al. [81] impose Lipschitz filters to obtain continuous latents before fitting a free neural ODE. ParticleGS [62], EvoGS [3], GaussianPrediction, and NVFi [39] likewise lack learned energy, inertia, and dissipation structures that would expose an external-effort input.Source: §2.4 · p. 6
  • Structured video-based predecessors constrain mechanics or force access, whereas LaGSplat fixes a reconstruction-derived latent chart before learning a dissipative law with state-dependent inertia.Pixel-HNN [21] is conservative and unforced with a fixed-pixel decoder; Castañeda Garcia et al. [7] fit known scalar equations without reconstruction; Krauss et al. [35] learn linear oscillators driven only through measured pressure channels, whose attention centroids are not generative material points. LaGSplat adopts Lipschitz filters from Zhu et al. [81] and uses M(q) to accommodate chart scaling and curvature; four latent coordinates outperform the ten-dimensional oscillator chain on the two-segment pneumatic robot.Source: §2.4 · pp. 6–7
  • LaGSplat combines video-supervised latent dissipative mechanics, explicit moving Gaussian primitives, and interactive forces, targeting the filmed system rather than zero-shot transfer.The primitive axes carry the same state governed by the equation of motion, coupling force application points to integrated coordinates. Wang et al. [69] also support applied forces, but fit prescribed rigid-body dynamics with predefined poses rather than discovering latent states for rigid and deformable objects.Source: §2.4 · p. 7

3 Method

LaGSplat learns a continuous latent chart with a Gaussian splatting autoencoder, then freezes it to fit dissipative Lagrangian dynamics. Decoder kinematics transport unseen user forces into latent generalized forces, with one calibration measurement needed to establish physical force scale.

3.1 Problem Statement and Overview

  • An unsupervised latent state serves as both generalized coordinate and rendering condition, with appearance training preceding dynamics fitting on the frozen chart.Stage 1 jointly fits the encoder and Gaussian splatting decoder; Stage 2 freezes both, encodes the clip once, and fits the latent Lagrangian and dissipation.Source: §3.1 · p. 7

3.2 Stage 1: Autoencoder and Latent Chart

  • Lipschitz-constrained encoder filters preserve temporal continuity, while whitening and a decoder-derived visibility metric account for unequal observability across latent directions.The continuity regularizer follows Zhu et al. [81]; latent dimension comes from known physical degrees of freedom or the smallest width achieving near-perfect reconstruction. Frozen encoded states are whitened to zero mean and unit covariance, while the trajectory-averaged image-metric pullback weights dynamics fitting and reported errors.Source: §3.2 · pp. 7–8
  • State-conditioned Gaussian splats store revisited states once and render trajectories from unseen initial conditions or applied forces.Conditioning shifts centers affinely, keeps conditional covariance state-independent, and suppresses primitives distant in latent space; photometric training includes continuity and ellipsoid aspect-ratio penalties. Direct 2D output uses monocular supervision, whereas 3D novel-view rendering requires multiview supervision, synthesized in the experiments using SHARP [51].Source: §3.2 · p. 8

3.3 Stage 2: Latent Lagrangian Dynamics

  • Learned positive-definite inertia and damping accompany a convex or invex potential, accommodating curved latent charts and nonconvex energy contours while retaining one equilibrium.The Lagrangian splits kinetic and potential energy; learned mass absorbs chart geometry, while a parameter-free decoder-Jacobian alternative assumes uniformly distributed visible inertia, and reported results use learned mass. An Input-Convex Neural Network [2] handles convex basins; composing it with a learned diffeomorphism yields an invex potential [65], whereas multiple wells would require a locally-convex variant [30].Source: §3.3 · p. 9
  • A positive damping floor ensures bounded, convergent free responses, while concave chamber-volume maps preserve a unique stable equilibrium under nonnegative pneumatic pressures.These stability properties hold before training; free dynamics settles at the potential minimum and maintained loading at its loaded equilibrium. Measured chamber pressures enter as generalized forces through derivatives of the latent chamber-volume map.Source: §3.3 · p. 9
  • Dynamics training minimizes visibility-weighted one-step prediction error, but model selection uses multistep rollout error to address teacher forcing’s long-horizon blind spot.A differentiable velocity-Verlet step predicts the next encoded state; decoder-metric weighting approximates image error without rasterization. An isotropic floor caps the weighting ratio at 1/ρ, preventing barely visible, potentially stiff directions from remaining unconstrained.Source: §3.3 · p. 9

3.4 Interaction and Inference

  • Gaussian-center Jacobians transport unseen user forces into latent generalized forces, with neighborhood contact blending and one measurement calibrating physical force scale.Each primitive’s affine kinematics provides a constant Jacobian, making force pullback O(nd) without network differentiation at inference. Video determines response direction and shape, but a global energy-scale ambiguity requires one known force-response measurement.Source: §3.4 · p. 10
  • 30 or 60 Hz dynamics and above 80 fps rendering on a 16 GB GPU support interaction with force-modified trajectories and editable physical properties.One or two frames initialize state and velocity, with zero initial velocity assumed for one frame; symplectic integration limits long-rollout energy drift [9]. Rendering runs independently of dynamics, and scaling D, T, or V edits damping, inertia, or effective potential.Source: §3.4 · p. 10

4 Experiments

The experiments evaluate LaGSplat’s natural-frequency recovery, soft-robot prediction accuracy, dissipative long-horizon motion, and interactive responses to applied forces. Without force ground truth, interactive tests examine soft-robot deformation and direction-dependent 3D rocker responses governed by learned Gaussian kinematics.

4.1 Protocol

  • Experiments fit dynamics on frozen photometric latent charts for real systems with d = 1–4, without state, force, or torque annotations.Cases span rigid oscillators, a hanging bag, and pressure-driven soft robots [35]; nonrigid dimensions are empirical, and only the rocker receives multi-view 3D supervision. Unreanchored rollouts use frequency error, visibility-weighted latent NRMSE, and image error relative to the autoencoder floor, with latent and image errors averaged over two natural periods.Source: §4.1 · pp. 10–11

4.2 Results across test cases

  • 1–6% natural-frequency error holds across every system except the one-segment robot, whose unexcited natural mode yields 31.4% error.Small frequency and damping errors still accumulate phase drift: hanging-bag latent error rises from 0.150 over two periods to 0.396 over seven. The double pendulum reaches latent error 0.947 over two periods and 1.544 over the whole clip, exposing the limits of identifying chaotic motion from 211 training frames.Source: Table 2 · p. 11, §4.2 · pp. 11–12

4.3 What the Lagrangian prior buys

  • 53% of initial amplitude persists after 1000 seconds for GaussianPrediction, whereas LaGSplat reaches its structurally enforced stable equilibrium within tens of seconds.GaussianPrediction nevertheless leads held-out PSNR by 2.0 dB and achieves 0.9% frequency error versus LaGSplat’s 1.2%; its long-horizon frames remain visually coherent and within the observed latent range. Both underestimate the filmed-window dissipation coefficient: LaGSplat by 29% and GaussianPrediction by 60%.Source: Table 3 · p. 12, §4.3 · p. 12

4.4 Real actuated soft robot

  • Only LaGSplat supports applied forces anywhere in the image; Osc. + ABCD permits d/2 locations, while Koopman baselines cannot accept mechanical forces.Deconvolutional decoders expose no application positions, and only oscillator dynamics mechanically convert latent forces into acceleration; the other oscillator baseline therefore accepts forces only in latent coordinates. Robot comparisons retain the released 32 × 32, 60 fps, contiguous 80/20 protocol, using measured pressures through learned configuration-dependent actuation and published baseline errors over 30 steps and 50 validation trajectories.Source: §4.4 · pp. 13–14
  • 0.769 × 10^-3 pixel MSE makes LaGSplat the most accurate two-segment soft-robot predictor over the 0.5-second benchmark horizon.This median over five seeds uses d = 4 versus the baselines’ d = 10, improving over Osc. + ABCD, Osc. + deconv, Koopman + deconv, and Koopman + ABCD by factors of 8.5, 30, 7.4, and 1.28, respectively. The last margin is narrow; on the weakly excited one-segment robot, LaGSplat ranks last and inertia is poorly identifiable.Source: Table 4 · p. 14, §4.4 · p. 14
  • On the two-segment robot, decoder-induced inertia yields 1.33 × 10^-3 over 0.5 seconds versus learned inertia’s 0.769 × 10^-3, yet ranks ahead of three of four baselines.The parameter-free, decoder-Jacobian-derived inertia assumes uniform density; the comparison fixes the autoencoder, latent chart, objective, and settings while retraining other terms. Frequency error is 6.9% versus learned inertia’s 5.7%, while constant inertia reaches 4.8 × 10^-3 on this test case.Source: §4.4 · p. 14

4.5 Applying forces

  • Unseen image forces enter the learned dynamics through J(q)^T f without retraining, combining with pressure actuation while preserving rendered responses under latent-coordinate changes.Transport sums the per-primitive Jacobian contributions beneath the click, using kinematics already fixed by video training. Because force ground truth and absolute energy calibration are unavailable, the experiments examine equilibrium-response ratios, signs, and sums that remain invariant to the unidentified energy scale.Source: §4.5 · p. 15
  • 8 × 10^-8 relative error supports force superposition on the rocker, alongside lever-arm scaling, sign reversal, and negligible background response.Equal forces produce latent shifts of -0.50, -0.76, and -0.84 with increasing lever arm, then +1.22 across the rotation centre; background clicks yield absolute shifts ≤ 0.007 without segmentation. On the deformable robot, both segments follow left/right forces, and loading only the first also bends the second, although the figure does not distinguish explanations.Source: §4.5 · p. 15
  • Force responses within and between segments arise solely from learned kinematics, without explicit segment knowledge, because users apply forces to selected Gaussians.Quantifying deformation error would require recorded responses to known point forces, which none of the available systems provides.Source: §4.5 · p. 16
  • The two-segment soft robot of Krauss et al. [35] bends under lateral forces and stretches or shortens under axial forces, largely preserving left-right symmetry.Opposed junction/tip forces bend the second segment while the first deflects about one-quarter as much as under the junction force alone, in the opposite direction. Gravity prevents interpreting this response as static balance.Source: Figure 9 · p. 16, §4.5 · p. 16
  • Synthetic SHARP views supervise the 3D decoder; clicking casts a ray to the first intersected Gaussian, where a world-space force is applied.The supervision uses rendered views of lifted frames rather than real multi-camera capture.Source: §4.5 · p. 16
  • Less than 3 × 10^-9 latent-state change occurs under orthogonal 3D forcing, while equal-magnitude vertical forcing yields equilibrium q = +1.52.For the single-degree-of-freedom rocker, the orthogonal force produces J^T f = -3.1 × 10^-17, effectively zero regardless of force magnitude.Source: §4.5 · pp. 16–17, Figure 10 · p. 17

5 Discussion

LaGSplat’s discussion identifies limitations in its dynamical assumptions, video coverage, force validation, and modeling choices, while outlining extensions to richer physics, simulator coupling, control, and cross-system learning.

Limitations

  • The Lagrangian prior restricts LaGSplat to few coordinates, smooth kinematics, and dissipative dynamics that are autonomous or driven by measured inputs.The instantiated dynamics exclude contact, impact, topology changes, plasticity, and hysteresis, although an unstructured motion predictor can cover these behaviors. Alternative low-dimensional evolution equations could retain interaction only if they accept the generalized effort J^T f; contact and plasticity admit suitable formulations, but neither is implemented.Source: §5 · p. 17

Observability, coverage, and designed excitation

  • Video learning requires observable, sufficiently excited states and increasingly extensive recordings as the latent state space grows.Out-of-plane displacement requires real multiple views, and unobserved motion remains unavailable: the rocker cannot tip under a large push because its clip never excites tipping. Gaussians must cover every reachable state to render it, while frames must densely sample that region to identify dynamics; designing persistently exciting inputs or releases could improve coverage.Source: §5 · pp. 17–18

Limitations and measured forces

  • The force mechanism lacks measured-force validation, while one measured poke would determine the model-wide energy scale kappa.Existing tests check Jacobian properties invariant to kappa; PokeFlex [54] supplies logged interactions but would test quasi-static equilibrium rather than forced dynamics. Validation requires a fixed monocular viewpoint, few degrees of freedom, and recorded forces and contact points; masking hands and using instrumented gloves could provide human-manipulation recordings.Source: §5 · pp. 17–18

Density assumptions and material separation

  • Uniform-density primitives can misestimate inertia and incorrectly assign mass and force response to moving shadows.The inertia error was measured on only one system, where it was small; objects with strongly differing material densities may behave differently. Appearance-based material prediction, following Le et al. [37] and Lin et al. [45], could determine relative primitive densities but not kappa, while object–shadow segmentation would exclude shadows from both mass and force calculations.Source: §5 · pp. 17–19

Training and latent dimension

  • Sequential fitting remains untested against joint training in LaGSplat, while latent dimension is manually chosen and can be overestimated.Friedl et al. [14] report an order-of-magnitude decoded-prediction advantage for joint training, whereas Zhu et al. [81] outperform standard jointly trained pairings using a continuous latent representation and frozen encoder; their different encoders and simulated-state versus video settings leave LaGSplat’s choice unresolved. Dimension is selected from physics or a reconstruction-error knee, although Chen et al. [8] report reconstruction degradation before the latent reaches the minimal state dimension.Source: §5 · p. 18

Contact, impact, plasticity, and hysteresis

  • Contact and impact require compatible collision geometry, while plasticity and hysteresis require history-dependent state beyond the current per-frame encoder.A differentiable latent contact model [80] could transport primitive impulses through J^T, unlike the current continuous-state force brush; Wang et al. [69] provide Gaussian collision geometry, but preserving LaGSplat’s anisotropic covariances remains unresolved. Plasticity could add internal variables, free energy, and dissipation following Masi and Stefanou, increasing dimension and requiring frame windows; response-supervised recurrent models offer a route when microscopic internal variables are unobserved.Source: §5 · p. 18

Simulator coupling, control, and multiple systems

  • Proposed extensions include simulator coupling, model-based control, independent multi-object dynamics, and articulated or explicitly constrained systems.Simulator reactions could enter through J^T while primitives return collision geometry without retraining, and identified mass, dissipation, potential, and actuation could support soft-robot control without joint encoders [34] [48]. Independent objects should have uncoupled coordinate blocks and pass a push-isolation test, but current losses do not encourage that structure and their product state space increases coverage demands; Lagrangian graph networks [6] could support articulation, whereas explicit constraints [13] are currently unenforced to accommodate unknown constraint topology.Source: §5 · pp. 18–19

Cross-system priors and shared charts

  • Cross-system kinematic and dynamic priors could reduce recording demands, but require latent coordinates with consistent roles across systems.Priors over kinematics, mass, potential, and dissipation could enable shorter identification clips in the spirit of [17] [73]; coordinate-invariant force transport permits a shared space but does not itself align coordinate meanings. NeuROK [17] learns a shared encoder, while frozen DINOv2 [55] or JEPA-style video encoders [71] could remove per-scene encoder training, provided their features resolve the few motion-bearing degrees of freedom.Source: §5 · p. 18

6 Conclusion

LaGSplat learns a low-dimensional physical state directly from video and uses decoder Jacobians to apply unseen forces without retraining. Its Lagrangian prior restricts applicability while yielding bounded, physically plausible responses.

6 Conclusion

  • A single low-dimensional state replaces time parameterization, serving as both dissipative Lagrangian coordinates and the conditioning variable for a decoder of explicit moving points.The state receives no direct supervision: video alone supplies the training signal.Source: §6 · p. 19
  • Four coordinates suffice for LaGSplat to outperform all four alternatives on Krauss et al.’s [35] two-segment soft continuum robot; published baselines use ten.Evaluation uses the authors’ data, split, and protocol. An inertia derived from decoder kinematics, with no free parameters, still outperforms three compared models.Source: §6 · p. 19
  • Seven real systems spanning one to four degrees of freedom yield learned natural frequencies within a few percent of video-observed frequencies whenever recordings excite them.Source: §6 · p. 19
  • A thousand seconds later, GaussianPrediction still orbits, whereas LaGSplat settles at a stable equilibrium never shown during training.GaussianPrediction is a motion predictor without mechanical structure.Source: §6 · p. 19
  • Unseen forces pass through decoder Jacobians into the learned equations without retraining, despite no forces being applied during training.Forces can act anywhere in the image, or in the reconstructed volume with multi-view supervision. Kinematics determines their direction and shape, but amplitude remains identifiable only up to one global energy factor.Source: §6 · p. 19
  • The Lagrangian prior narrows the supported system class in exchange for bounded, physically plausible responses to unseen forces.Source: §6 · p. 19

A Derivations

This appendix derives how learned kinematics transports forces and inertia, and the energy balance underlying the bounded-response remark.

A.1

  • Section A.1 derives how the learned kinematics transports both forces and inertia.Source: A Derivations · p. 19

A.2

A Derivations

A.1 Forces and inertia from the learned kinematics

The learned primitive kinematics determines both generalized forces and kinetic energy. Force transport preserves virtual work independently of the latent coordinate chart, while presence weighting makes inertia configuration-dependent.

Transport of a force

  • Applied forces pull back through the learned kinematic Jacobian into unique latent generalized forces that preserve virtual work and respect the model’s energy balance.Stage 1 learns an affine position map for each primitive, analogous to particle positions in analytical mechanics. A force f on primitive i becomes J_i^T f.Source: §A.1 · p. 19

Invariance to the latent chart

  • The response to a user force is invariant to smooth changes of latent coordinates, despite those coordinates having no intrinsic physical meaning.Under q = ψ(q̃), the Jacobian becomes J_i Ψ and the generalized force transforms as Ψ^T f_lat, preserving work against the transformed virtual displacement.Source: §A.1 · p. 19

The same map carries the inertia

  • Equal primitive masses weighted by normalized latent opacity induce a configuration-dependent mass matrix and Coriolis terms, even though the kinematic Jacobians are constant.Primitive velocities satisfy μ̇_i = J_i q̇, so kinetic energy follows from the learned position maps. All mass-matrix state dependence comes from the presence weights.Source: §A.1 · pp. 19–20

A.2 Bounded forced response

Under positive-definite inertia, a coercive potential, and strictly positive damping, LaGSplat’s free and constant-force responses remain bounded and settle to equilibria. These continuous-time guarantees do not ensure quantitative accuracy far from training data.

Assumptions

  • The guarantees require positive-definite inertia, a coercive potential enforced by a quadratic floor, and a uniformly positive damping floor.Without coercivity, energy could decrease while its sublevel sets remained unbounded, allowing position to drift.Source: §A.2 · p. 20

Free rollout

  • Every free trajectory remains bounded and converges to the potential’s unique minimum, excluding sustained oscillation independently of training.Energy decreases outside rest, and LaSalle’s invariance principle establishes convergence to zero velocity and a stationary potential. Strong convexity of the floored ICNN potential and the invex construction each ensure a unique minimum.Source: §A.2 · p. 20

Maintained force

  • A constant generalized force produces a bounded response converging to a loaded equilibrium satisfying ∂V/∂q = f.The loaded energy E_f = T + V − f^T q cancels injected power; the quadratic potential floor dominates the linear loading term, preserving coercivity.Source: §A.2 · p. 20

Scope

  • The guarantees concern continuous time; symplectic integration limits long discrete-rollout drift, but boundedness does not imply quantitative accuracy far from training data.Source: §A.2 · p. 20

B Deformation-Field Decoder at Equal Budget

With equal primitive counts and training conditions, the deformation-field decoder uses more parameters but reconstructs fitted frames less accurately than LaGSplat. Its rougher primitive velocities show that similar-looking renderings need not provide meaningful derivatives for force transfer and inertia.

Decoder parameterization

  • LaGSplat conditions primitives in an (n+d)D volume to obtain affine positions and constant Jacobians; the alternative warps n-D primitives through a network following Wu et al..The deformation-field family contains LaGSplat’s parameterization and can represent curved primitive trajectories, but this additional freedom provides no image-quality benefit in this experiment.Source: B Deformation-Field Decoder at Equal… · p. 20

Material tracking

  • Material tracking is unsupervised: LaGSplat’s trajectory constraints encourage primitives to follow matter, whereas deformation fields permit primitives to replace one another without a rendering penalty.LaGSplat’s visible primitives translate along fixed lines in q while latent gating fades them in and out. Deformation-field centers and extents vary freely, allowing one primitive to shrink below a pixel while another grows in its place.Source: B Deformation-Field Decoder at Equal… · p. 20

Reconstruction

  • 0.0157 L1 and 0.076 SSIM term for the deformation field exceed LaGSplat’s 0.0142 L1 and 0.063 SSIM term on fitted frames.Both use 15,000 primitives and identical losses, regularizers, optimizers, and schedules; parameter counts are 9.8M for the deformation field versus 195k for LaGSplat. Evaluation measures capacity spent, not generalization.Source: B Deformation-Field Decoder at Equal… · p. 20

The Jacobian field

  • 6.9 px/s median neighboring-primitive velocity difference for the deformation field contrasts with 1.4 px/s for LaGSplat, despite visually similar renderings.The deformation field also moves primitives over the static background. These derivatives determine both applied-force pullback and inferred inertia, so reconstruction quality alone does not establish their physical usefulness.Source: B Deformation-Field Decoder at Equal… · pp. 20–21, Figure 11 · p. 21

C Implementation Details

LaGSplat uses the same architectures across systems, with system-specific dimensions, image resolutions, primitive counts, and run lengths. Training first jointly fits the encoder and decoder, then fits dynamics on a frozen, whitened latent chart.

Architectures

  • All systems share architectures, while latent dimension, image resolution, primitive count, and run length vary by system.The continuity-preserving convolutional encoder [81] uses three stride-2 convolutions with 12×12 filters, two channels and ELU activations, followed by 4×4 average pooling and a linear latent map, totaling about 2,000 parameters. The decoder contains 1,000–15,000 Gaussians of dimension n + d.Source: C Implementation Details · p. 21
  • Fewer than 15,000 parameters define the LNN, combining a potential ICNN [2], learned mass and damping matrices, and chamber-specific actuation heads.The potential uses [64] softplus layers and, when invexity is required, a two-block width-64 i-ResNet with Lipschitz cap 0.9. Separate [64] MLPs produce the mass-matrix Cholesky factor with SPD floor 0.1 and full damping matrix initialized at 0.2 I with isotropic floor 1; each chamber has an invex ICNN actuation head.Source: C Implementation Details · p. 21

Training

  • About 5 × 10^4 Adam steps jointly train the encoder and decoder in Stage 1, with periodic recycling of low-opacity Gaussians.The loss is 0.08 L1 + 0.02 (1 − SSIM) + 0.01 anisotropy, using a 7×7 SSIM window; Gaussians below peak opacity 0.02 are recycled by splitting overloaded primitives along their principal covariance eigenvector or teleporting to high-error regions. Under multi-view supervision, 65 SHARP renders per frame are computed once and cached.Source: C Implementation Details · p. 21
  • About 5 × 10^4 Adam steps train dynamics alone in Stage 2 using one differentiable Verlet step on clip windows.After Stage 1, covariance-based latent whitening is computed from training frames and frozen with the chart; Stage 2 weights latent position and velocity equally in the visibility metric, with ridge ρ = 0.01. High-resolution rendering retrains only the decoder on the frozen chart.Source: C Implementation Details · p. 21

Related papers

Loading 2608.16324v1…