EMMA: Extracting Multiple physical parameters from Multimodal Data

Farhat Shaikh; Ayan Banerjee; Sandeep Gupta

Source-linked AI summary

EMMA: Extracting Multiple physical parameters from Multimodal Data

Farhat Shaikh, Ayan Banerjee, Sandeep Gupta

arXiv:2605.24047v1May 21, 2026cs.CV

TL;DR

Video-only dynamical inference can miss hidden inputs, implicit effects, and unknown reference invariants, motivating a more complete multimodal approach. EMMA combines cross-modal alignment, an LTC-based physics-informed estimator, and invariant calibration to recover and simulate dynamical models. Across diverse canonical and real systems, it reports accurate recovery with a compact pipeline, while retaining runtime and modality-related limitations.

Problem
Video-based inverse modeling can miss occluded forcing inputs, implicit physical effects, and unknown initial-condition or coordinate-frame invariants needed for physically meaningful dynamical models.
Method
EMMA fuses video, audio, and chart-derived time series in an LTC-based physics-informed estimator that jointly recovers explicit parameters, implicit dynamics, and calibration invariants.
Results
Across diverse canonical systems and real platforms, EMMA delivers accurate multi-parameter recovery and reproduces observed trajectories with high fidelity using a compact pipeline.
Takeaways & Limitations
The recovered parameters are interpretable and executable for simulation, verification, and downstream control using standard sensors and off-the-shelf tooling.
Takeaways & Limitations
EMMA depends on at least one temporally varying modality, may degrade under turbulence because of its linear frequency-speed audio prior, and has higher LTC-based runtime.

Abstract

from arXiv · show

We introduce EMMA, a physics-informed multimodal framework that recovers all identifiable dynamical parameters of a system directly from raw video, audio, and image-based time-series observations. Unlike prior video-only approaches that struggle with occluded states, hidden actuation inputs, or assumptions about known initial conditions and coordinate frames, EMMA performs joint inference of explicit parameters, implicit dynamical components, and calibration invariants within a unified continuous-time model. EMMA leverages a Liquid Time-Constant (LTC) network to learn latent dynamics from heterogeneous modalities while a physics-constrained loss enforces consistency with the governing differential equations. A unified feature pipeline enables consistent alignment across video trajectories, acoustic signatures, and chart-derived measurements, allowing EMMA to estimate parameters under forced, implicit, and multivariate dynamics without requiring segmentation masks, differentiable rendering, or specialized sensors. Across 100+ scenarios including five standard dynamical benchmarks (75 Delfys videos), real-world rover and quadrotor systems with hidden inputs, and simulation-chart case studies spanning biological and chaotic systems, EMMA delivers robust multi-parameter recovery and significantly outperforms existing single-modality and equation-discovery baselines. Our results establish EMMA as a general, scalable solution for physics-consistent model extraction from opportunistic multimodal data. Code and data are available at: https://github.com/ImpactLabASU/EMMA-CVPR2026

1. Introduction

EMMA addresses key gaps in video-based dynamical model extraction by combining synchronized multimodal observations with physics-informed continuous-time inference. It jointly recovers explicit parameters, implicit dynamics, and unknown invariants for forward simulation.

Digital twins of autonomous platforms require dynamical parameters inferred from observable trajectories for testing, simulation, fault diagnosis, and safety-critical decision support.
Video-only inference can become ill-posed when key states or external forcing inputs are occluded, whereas audio may encode hidden actuation such as wheel speed.The rover example contrasts visible wheel pose with unavailable wheel-power commands and acoustically recoverable wheel rotation information.
Existing approaches often omit implicit effects such as frictional drag and assume known initial conditions, coordinate origins, or fixed reference frames.These unmeasured effects may remain observable through nonlinear dependencies among measured states, while real-world camera geometry and origins are generally unknown.
The recovered parameters support physically consistent trajectory simulation without requiring frame reconstruction or segmentation masks.Figure 1 presents the physics-informed digital twin and its forward-simulation validation role.
EMMA fuses video, audio, and chart-derived time series in a physics-informed multivariate LTC estimator that recovers explicit parameters, implicit components, and coordinate-frame invariants.The video pipeline estimates observable states, audio recovers latent actuation, and joint calibration estimates unknown origins and initial conditions.

2. Related work

Related work spans video-based parameter recovery, differentiable-rendering methods, simpler unforced systems, and general equation-discovery frameworks. EMMA’s primary comparison is Delfys because it most closely matches the unsupervised, video-native continuous-time setting.

Delfys is the primary baseline because it performs unsupervised video recovery of physical parameters for known continuous-time ODEs.Its decoder-free design avoids frame prediction and remains stable under intensity or scale variations in real videos.
Differentiable-rendering approaches such as gradSim/rSim and ϕ-SfT rely on known geometry or templates, limiting direct comparability with EMMA.
Table 1 compares related works by whether capabilities are demonstrated, using ✓ and ✗ entries across model-recovery criteria.
PAIG and related methods address simpler, unforced systems with a small number of unknowns.
PySINDy, SINDy-PI, and PINNs are general equation-discovery frameworks that require video-to-state or video-to-field front ends and are not video-native.

3. Method

EMMA combines time-aligned video, audio, and image features with continuous-time latent dynamics and differentiable physics simulation to estimate physical parameters and calibration quantities. Its architecture addresses hidden inputs, implicit dynamics, and heterogeneous observations through an LTC network, nonlinear parameter readout, and physics-informed constraints.

Unified multimodal feature extraction: EMMA processes video, audio, and chart or sensor images through modality-specific pipelines, synchronizing their features into unified temporal representations.Features are temporally interpolated to video timestamps, concatenated into multimodal state vectors, and missing modalities are handled by zero-padding or learned embeddings.
Architecture overview: An LTC network models continuous-time latent dynamics with input-dependent time constants, producing hidden trajectories for multimodal parameter estimation.The network uses hidden states h(t) and adaptive time constants, while a sequence-to-sequence prediction head produces temporally averaged parameter estimates.
Feature extraction robustness: Farneback optical flow achieves comparable accuracy without a pretrained detector, supporting the LTC physics layer rather than a specific visual feature extractor as EMMA’s core contribution.The alternative is evaluated on rover and pendulum systems.
Audio-based forcing recovery: Audio features recover latent actuation information through an approximately linear relation between dominant tonal frequency and rotational speed, with calibration parameters learned by EMMA.Raw audio is converted into aligned spectral features, and the affine frequency-speed mapping is treated as an invariant calibration component.
Parameter estimation: EMMA jointly estimates dynamical parameters, latent forcing or implicit effects, and invariant quantities by mapping measured and hidden dynamics through a nonlinear dense readout.The dense head learns parameter estimates and calibrates invariants; the LTC hidden dynamics provide capacity for multiple implicit components.
Physics-informed training: A differentiable physics simulator injects predicted parameters into the governing ODE and enables end-to-end training against simulated observables.The objective compares observed and simulated trajectories while regularization enforces physical validity such as positivity and parameter bounds.

4. Evaluation

EMMA is evaluated on multimodal physical-parameter extraction across canonical systems, real platforms, and simulation charts. It generally outperforms video and equation-discovery baselines while retaining a compact model.

Evaluation scope: The evaluation covers five canonical physical systems, rover and quadcopter videos, and simulation-chart case studies spanning multiple dynamical settings.Ground truth is obtained from benchmark parameters or datasheets, and chart experiments include F8 Crusader, Lotka–Volterra, Lorenz, HIV therapy, and insulin delivery systems.
Evaluation scope: EMMA estimates multiple parameters from video and charts, with comparisons against PySINDy and ground-truth values where no direct multimodal baseline exists.The chart comparison uses the shared EMMA image pipeline for both EMMA and PySINDy.
Parameter recovery: EMMA consistently reduces parameter error relative to video baselines and PySINDy across benchmark systems, including pendulum, sliding-block, LED-decay, and free-fall settings.Its estimates remain stable across configurations, while PySINDy is affected by derivative noise, occlusion, fractional powers, and framerate variation.
Robustness: EMMA’s physics-constrained continuous-time formulation maintains low-variance estimates and handles irregular sampling and realistic measurement noise.The reported low variance across container sizes is ±0.0004 to ±0.0009, and audio noise down to 5 dB SNR causes less than 1.1% variation in rover parameters.
Implicit dynamics: Under implicit chart dynamics, both EMMA and PySINDy degrade, but EMMA outperforms PySINDy and shows substantially less performance degradation.The implicit condition measures only one state variable while the remaining state variables are unmeasured.
Efficiency: EMMA takes 1.4 × more execution time than Delfys on an NVIDIA RTX Ada 6000 GPU, while using a 107× smaller model.The runtime overhead is attributed to solving ordinary differential equations in the LTC-NN.

5. Conclusions

EMMA couples multimodal observations with an LTC-based estimator, invariant calibration, and cross-modal alignment to recover explicit and implicit dynamics from raw data. The framework achieves accurate recovery with standard sensors, but its scope includes temporal, audio, camera, and runtime constraints.

Conclusion: EMMA recovers explicit parameters and implicit dynamical components from raw video, audio, and images, then reproduces observed trajectories with high fidelity.The recovered parameters are interpretable and executable for simulation, verification, and downstream control.
Conclusion: Its compact pipeline uses invariant calibration, cross-modal alignment, standard sensors, and off-the-shelf tooling across canonical systems and real platforms.The paper positions EMMA as a foundation for learning physical models from opportunistic multimodal data.
Limitations: EMMA depends on at least one temporally varying modality and may be limited by turbulence, severe camera shake, and LTC-based ODE-integration runtime.The stated audio prior is linear frequency-speed, and the paper notes that it may degrade under turbulence.

S1. Ablation Study

The ablation study examines architecture choice, multimodal input, forcing, and implicit dynamics. LTC-NN is especially advantageous for complex forced dynamics, while audio improves parameter estimation under hidden inputs.

Architecture ablation: On the unforced pendulum example, LSTM, GRU, Transformer, and LTC architectures estimate parameters with comparable accuracy.The pendulum estimates two parameters and has no external force input.
Architecture ablation: On the more complex forced rover example, alternative recurrent architectures degrade relative to LTC-NN, which yields accurate results with faster convergence.The rover requires estimating multiple parameters in the presence of an external force.
Multimodal input: Removing audio from the rover experiment demonstrates that multimodal input improves parameter estimation under forced dynamics.The comparison uses otherwise identical video-and-audio versus video-only setups.

S2. Physics Equations

The supplementary equations specify governing dynamics and parameterizations for benchmark and real-world systems, including pendulum, draining fluid, light decay, inclined motion, drag, rover, and quadrotor models.

Pendulum: The pendulum model evolves angle and angular velocity under gravity and damping.Its parameters include pendulum length, gravity, and damping coefficient.
Fluid draining: Fluid height follows an orifice-draining law parameterized by the drainage coefficient K.K depends on discharge coefficient, orifice area, gravity, and tank area.
Light decay: Light intensity is modeled as first-order exponential decay with decay rate γ.The stated parameter range is γ ∈(0.01, 5.0] s−1.
Inclined motion: Inclined-plane motion combines gravitational acceleration along the incline with friction opposing motion.The model varies incline angle α and friction coefficient µ over specified ranges.
Real-world systems: The rover and quadrotor models combine kinematic or rigid-body dynamics with implicit forces and rotor dynamics.Rover parameters include friction, drag, and center-of-mass height; quadrotor audio-inferred parameters include motor gain, time constant, and damping ratio.

S3. Differentiable Trajectory Rollout

EMMA rolls estimated parameters through the continuous physics function using a differentiable fourth-order Runge–Kutta integrator with constrained step sizes and parameters.

Integrator: A differentiable RK4 integrator advances the state using estimated parameters from the LTC network and the continuous physics function.The method is selected for higher-order error control in stiff systems such as the quadrotor.
Integrator: Each RK4 update evaluates four physics slopes and combines them in a weighted state increment.The intermediate evaluations use half-step states before the final full-step update.
Numerical constraints: The simulation step is clamped by the smaller of 0.03 and the inverse frame rate.Intermediate forcing inputs are linearly interpolated, and rollout length is capped at 500 or T steps.
Numerical constraints: Soft clamping enforces a positive lower bound of 10−4 on physical parameters during rollout.This constraint is applied after defining the simulation horizon and forcing interpolation.

S4. Additional Robustness and Ablation Experiments

Additional experiments test EMMA’s robustness to feature extraction, architecture choice, initialization, random seeds, and audio noise, with favorable results across these settings.

Feature extraction: Comparable accuracy from unsupervised Farneback optical flow supports the LTC physics layer’s independence from the object detector.The comparison covers rover and pendulum systems, and Table S4 reports the detector-versus-flow results.
Reproducibility: 9.5% ± 8.9% average error across four measurable rover parameters is reported over five random seeds.The standard deviation quantifies variation across seeds 42–46.
Architecture comparison: Under forcing inputs, LTC outperforms Neural ODE by approximately 25% and CT-GRU by approximately 5% in average parameter error.All three continuous-time architectures perform comparably on the unforced pendulum.
Initialization sensitivity: EMMA achieves <10% error in 5 out of 6 configurations despite 200% expanded parameter bounds and distant initialization.The experiment directly evaluates sensitivity to poor initialization.
Audio noise robustness: Parameter estimates vary by less than 1.1% across injected audio noise levels of 20, 10, and 5 dB SNR.The authors describe the audio pipeline as degrading gracefully under realistic acoustic interference.