Source-linked AI summary
PhysMamba: Selective State Space Models as Learned Articulated Body Simulators
Haochuan Zhang, Sinisa Todorovic
TL;DR
Monocular 3D human motion recovery from video is underconstrained, and existing physics-based approaches can be too slow and difficult to train end-to-end. PhysMamba uses a selective state space model to predict articulated-body states from history, and its results show different accuracy and rollout-stability behavior across observation settings and training protocols.
Problem
Monocular 3D human motion recovery is underconstrained, while physics-based regularizers and optimization can be too slow for real-time inference and difficult to train end-to-end.
Source: §1 · p. 1Method
Results
Mamba2 achieves the best 285d single-step MPJPE among four evaluated architectures, while GRU TF→Ro is the lowest-error long-horizon configuration and Mamba2 TF→Ro diverges frequently.
Source: §5 · p. 4, §4.2 · p. 4Takeaways & Limitations
The authors propose PhysMamba as a differentiable dynamics module for HMR that can regularize pose candidates and forecast motion to warm-start the backbone on occluded frames.
Source: §5 · p. 4The end-to-end HMR loop is unverified, and evaluation is limited to one humanoid morphology and one dataset.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We introduce the first learned articulated body simulator based on a selective state space model (SSM), called PhysMamba. PhysMamba predicts next-frame full-body state from position, rotation, and joint-action history, without velocity inputs. We compare four architectures under partial- and full-observation inputs and three training protocols. The from-scratch rollout training protocol gives Mamba2 strong short- and mid-horizon accuracy under partial observation (s10 = 43 mm, 2/50 diverged), while the two-stage teacher-based rollout protocol stabilizes GRU but fails for Mamba2. With CUDA graph compilation, Mamba2 reaches 0.107 ms per frame (9,334 FPS) on an H100 GPU, within 1.1$\times$ of GRU's un-compiled throughput, adding under 1% latency to a 30 Hz HMR pipeline and enabling integration as a differentiable physics module for video-based mesh recovery.
1. Introduction
Monocular 3D motion recovery is underconstrained, while existing physics-based approaches are too slow for real-time, end-to-end HMR. PhysMamba uses a selective SSM to predict future articulated-body states, with partial-observation training and a two-stage rollout failure revealing distinct accuracy and stability behavior.
1. Introduction
- Monocular 3D motion recovery is underconstrained, and physics-based regularizers are too slow for real-time inference and difficult to train end-to-end.The paper seeks a fast, differentiable model that predicts a next-frame state in one forward pass for use in HMR.Source: §1 · p. 1
- LARP’s fixed 20-frame context cannot adapt to changing dynamics, and extending its horizon increases input dimensionality linearly.LARP predicts next-frame velocities and matches MuJoCo accuracy at approximately 10× throughput.Source: §1 · p. 1
- Mamba2’s selective recurrence provides an O(1) hidden state that represents physical history and is structurally analogous to rigid-body evolution.The recurrence is presented as a natural fit for articulated-body simulation.Source: §1 · p. 1
- PhysMamba is introduced as the first selective-SSM learned articulated-body simulator, predicting future states from current states rather than regressing mesh pose from video.The paper trains it without velocity inputs so it must infer dynamics from position history.Source: §1 · p. 1
- The standard teacher-forcing-to-rollout strategy stabilizes GRU but frequently diverges for Mamba2, motivating SSM-specific rollout training.The authors attribute the instability hypothesis to gradient amplification through Mamba2’s input-dependent transition.Source: §1 · p. 1
2. Related Work
Prior learned simulators use message passing, fully connected networks, or fixed-window and memoryless models, while physics-informed HMR often applies physics post hoc. PhysMamba instead applies a selective SSM to articulated bodies, a domain that combines high-dimensional states, contact changes, and kinematic constraints.
2. Related Work
- Graph neural simulators model rigid-body interactions by message passing, while SuperTrack updates articulated body-part states and accumulates joint-displacement errors over long sequences.LARP reduces this error with displacement-aware features and losses.Source: §2 · p. 1
- Memoryless MLP and fixed-window simulator backbones cannot adapt to dynamics requiring longer context; a selective SSM offers adaptive, unbounded temporal memory.The overview depicts per-frame inputs processed sequentially by Mamba2, with its hidden state accumulating temporal context before next-state prediction.Source: §2 · p. 2, Figure 1 · p. 2
- Physics-informed HMR methods use analytic losses or post-hoc optimization, whereas PhysMamba learns a differentiable dynamics model for direct pipeline use.Prior SSM dynamics work addresses vehicles and drones, not articulated rigid bodies with contact, high-dimensional states, and kinematic-chain constraints.Source: §2 · p. 2
3. Our Approach
PhysMamba formulates articulated-body simulation as sequence-to-sequence next-state prediction from physical states and control actions. Its selective Mamba2 architecture, physics-simulated training data, structured loss, and two-stage training recipe specify the model and its training setup.
3. Our Approach
- The model learns sequence-to-sequence prediction of the next physical state from sequences of states and control actions.The state is a 358-dimensional representation of a 24-body humanoid, and the control action is 69-dimensional.Source: §3 · p. 2
3.1. PhysMamba Architecture
- Full-state input is 430d, while partial-observation input is 285d and withholds velocities for inference from position and rotation history.PhysMamba uses Mamba2 selective blocks with input-dependent transitions and an output head predicting the 358d next state.Source: §3 · p. 2, §3.1 · p. 2
3.2. Training Data
- PHC rollouts in MuJoCo yield 2,048 trajectories and approximately 785k frames at 30 Hz, with each next state produced by physics simulation.The dataset contains 1,847 training and 201 validation trajectories after 40 of 2,088 clips are dropped.Source: §3.2 · p. 2
3.3. Loss Function
- The Mamba2, GRU, and MLP loss combines joint-position MSE, quaternion-rotation distance, weighted velocity MSE, and bone-displacement MSE.MLP-LARP omits the velocity term; the rotation distance is bounded to prevent gradient spikes through SSM recurrence.Source: §3.3 · p. 3
3.4. Two-Stage Training
- Two-stage training first teacher-forces on ground-truth sequences, then fine-tunes autoregressively on the model’s own predictions with truncated backpropagation.The rollout stage warms hidden state on ground-truth frames; GRU uses a reduced learning rate, while lowering Mamba2’s rate accelerated rather than prevented divergence.Source: §3.4 · p. 3
4. Experiments
The experiments compare architectures and training protocols for single-step prediction and autoregressive rollout, including partial-observation and physical-validity evaluations. Mamba2 performs best at short horizons under rollout training, while GRU TF→Ro has the lowest long-horizon error; Mamba2’s two-stage rollout training remains unstable.
4. Experiments
- The evaluation uses 1847 training and 201 validation clips, with single-step MPJPE and 150-step autoregressive rollout metrics.Rollout divergence is any step exceeding 500 mm; physical-validity metrics use a retrain with a root-height channel and contact-conditioned Lfs.Source: §4 · p. 3
- Training uses AdamW, cosine scheduling, batches of eight clips, and z-rotation augmentation; Stage 2 learning rates differ between GRU and Mamba2.Stage 2 uses 10^-4 for GRU and 3×10^-4 for Mamba2.Source: §4 · p. 3
4.1. Main Results
- s10 = 43.1 mm for rollout-trained Mamba2, while GRU TF→Ro has the lowest s150 error at 147 mm; both diverge in 2/50 clips.MLP-LARP rollout reduces but does not eliminate divergence (47/50 clips).Source: Table 3 · p. 3, Figure 2 · p. 3, §4.2 · p. 4
4.2. Analysis
- At 430d, GRU achieves 29.2 mm versus Mamba2’s 37.5 mm; at 285d, Mamba2 achieves 29.2 mm versus GRU’s 36.6 mm.MLP shows no corresponding flip, with 49.4 versus 49.9 mm.Source: Table 2 · p. 3, §4.2 · p. 4
- Mamba2 TF→Ro diverges in 33/50 clips, with gradient norms exceeding 10^16 during Stage 2 despite multiple mitigation attempts.The authors identify SSM-specific rollout training strategies as an open direction.Source: §4.2 · p. 4
- In the v2 retrain with 286d + Lfs, Mamba2 has the lowest foot-slide and PhysCap MPE among three matched-recipe configurations.The contact-conditioned Lfs rollout loss, supervised by the MuJoCo body-contact mask, drives this gap.Source: §4.2 · p. 4
- With CUDA graph compilation, Mamba2 reaches 0.107 ms per frame (9,334 FPS), within 1.1× of GRU’s uncompiled throughput.Both models add under 1% latency to a 30 Hz HMR pipeline.Source: §4.2 · p. 4
5. Conclusion
PhysMamba applies a selective state space model to articulated-body dynamics, with results showing that performance depends on both architecture and training protocol. Its long-horizon behavior and evaluation scope remain important constraints.
5. Conclusion
- PhysMamba is a selective SSM for articulated-body dynamics, and its hidden state integrates physical history more flexibly than memoryless and fixed-window baselines under partial observation.The paper identifies SSM-specific rollout training, including spectral regularization of ¯At and forward-mode gradients, as an open problem.Source: §5 · p. 4
- Mamba2 achieves the best 285d MPJPE among the four architectures at single-step prediction, while GRU has the lowest error at long horizons.The long-horizon result depends on training protocol, and Mamba2's BPTT instability remains unsolved.Source: §5 · p. 4
- PhysMamba is proposed as a fast differentiable dynamics module for video-based human mesh recovery, with hidden-state rollouts forecasting motion intent 5–15 frames ahead.The forecast is described as a warm start for HMR on occluded frames and an anticipatory signal for downstream tasks.Source: §5 · p. 4
- Evaluation uses ground-truth PHC actions, one SMPL 24-body morphology, and AMASS-CMU, leaving closed-loop HMR and cross-morphology or cross-dataset transfer unassessed.Contact is supervised from MuJoCo body-contact labels rather than modeled at mesh-vertex level.Source: §5 · p. 4
Supplementary Material — PhysMamba
The supplement defines and qualifies physical-validity metrics, addresses evaluation and data-generation concerns, and reports a contact-conditioned retrain and action-perturbation analysis.
S1 — Foot-slide metric: definition and caveat
- The joint-level foot-slide proxy averages predicted XY displacement for four foot joints over qualifying contact pairs, and is not directly comparable to SimPoE’s mesh-vertex metric.The joint-centre measure may under-report slide during stationary contact and over-report it during fast contact transitions.Source: S1 — Foot-slide metric: definition and… · p. 5
S2 — PhysCap MPE: definition
- PhysCap MPE measures predicted toe penetration only when the ground-truth toe is in contact, targeting frames where the floor constraint is active.The contact set uses ground-truth labels for the two toe joints.Source: S2 — PhysCap MPE: definition · p. 5
S3 — Contact-mask source
- Contact labels come from MuJoCo rollout data, not AMASS, and the foot-slide loss uses the left- and right-toe entries of the exported contact mask.Each frame carries a 23-dimensional binary contact vector.Source: S3 — Contact-mask source · p. 6
S4 — Response to OpenReview reviewers
- The end-to-end HMR loop remains unverified; evaluation uses ground-truth PHC actions, and cross-dataset evaluation on non-CMU morphologies was deferred.Forty clips were excluded after PHC diverged before the source sequence ended; simulator behavior may depend on the controller source, whose ablation is deferred.Source: S4 — Response to OpenReview reviewers · p. 6
- Mamba2 Rollout is strongest at short and mid horizons, but GRU TF→Ro has lower error at s150: 147 mm versus Mamba2 Rollout’s 184 mm.The supplement identifies long-horizon stability under selective-SSM BPTT as an open problem.Source: S4 — Response to OpenReview reviewers · p. 6
S5 — Physical-validity addendum (v2 retrain)
- Mamba2 has the lowest foot-slide and PhysCap MPE among the three retrained architectures, while removing Lfs leaves penetration on most in-contact frames.The v2 retrain adds root height and a contact-conditioned foot-slide loss supervised by MuJoCo body-contact labels; the v1 main-paper results are unchanged.Source: S5 — Physical-validity addendum (v2… · p. 6
S6 — Body-part perturbation breakdown (v1 ckpts)
- Under large arm perturbations, about 0.2% of Mamba2 trajectories diverge, producing a mean/median error ratio up to 30 000× for the left arm group.GRU’s mean/median ratio remains within 2–3× across groups, attributed to its bounded forget-gate update.Source: S6 — Body-part perturbation breakdown… · p. 6, Table 4 · p. 7