STATERA: Hidden Mass Estimation via Zero-Shot Sim-to-Real Kinematics using Frozen Temporal Tubelets

1 Introduction

• The task is difficult because visual cues and surface trackers can follow visible features rather than the hidden mass, especially under self-occlusion.The geometric centroid is a valid heuristic for uniformly dense objects but fails for asymmetric payloads.Source: §1 · pp. 1–2
• STATERA evaluates motion across temporal sequences with a partially frozen V-JEPA backbone, while spatial-only ablations collapse predictions to the geometric center.The introduction describes this as observing rotational torque and momentum transfer during freefall and impact.Source: §1 · p. 2
• STATERA tracks the unobservable CoM of asymmetric rigid bodies from raw video using temporal features from a partially frozen V-JEPA backbone.The contribution is framed as extracting motion features rather than relying on visible surface cues.Source: §1 · pp. 1–2
• The work identifies settling-state dataset bias: complex phase-aware targets can cause visual-kinematic aliasing, addressed with curriculum label smoothing.The contribution statement describes this as a resolution to the observed bias.Source: §1 · p. 2
• STATERA demonstrates zero-shot transfer on calibrated real-world trajectories and is reported to outperform spatial foundation models, 3D-CNNs, and point trackers under N-CoME.The contribution statement specifies calibrated camera conditions and the Normalized Center of Mass Error metric.Source: §1 · p. 2

Visual Foundation Models and Temporal Representations

• Spatial foundation models excel at semantic and geometric tasks but evaluate isolated frames, whereas V-JEPA predicts missing temporal segments and learns scene transition dynamics.STATERA uses the latter’s pretrained temporal representations for kinematic properties not visible in static spatial embeddings.Source: §2 · pp. 2–3

Intuitive Physics from Video

• Earlier intuitive-physics benchmarks such as PHYRE tested mechanical-event prediction but largely relied on simplified 2D engines or explicit state access.This work shifts the objective to raw real-world video and hidden-mass estimation without engineered state abstractions.Source: §2 · p. 3

Object-Centric Tracking and Inference

• NeRFs, 3D Gaussian Splatting, and Slot Attention reconstruct or disentangle visible surfaces, while their optimization targets do not capture continuous Newtonian momentum constraints.The passage identifies hidden density and asymmetric CoM as internal properties these approaches struggle to infer.Source: §2 · p. 3

Physical Inference and Point Tracking

• Classical system identification typically needs explicit, noise-free state observations, while point-tracking pipelines can fail on tumbling bodies with motion blur and self-occlusion.STATERA’s latent end-to-end temporal approach bypasses surface correspondences for the visually inaccessible internal CoM.Source: §2 · p. 3

3.1 Problem Formulation

• In uniform ballistic free-fall, the internal CoM offset is unobservable; contact dynamics such as impacts and frictional settling provide torques and impulse gradients that reveal inertial parameters.The formulation defines the target as per-frame CoM coordinates over a 16-frame trajectory sampled at 24 FPS.Source: §3.1 · p. 3

3.2 STATERA Architecture

• STATERA processes 16-frame video with a partially frozen V-JEPA backbone, temporal tubelets, and a 1D convolution before decoding spatial outputs.The architecture processes the full sequence offline; its spatial decoder preserves the 24 × 24 patch grid rather than globally pooling it.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Continuous Coordinate Extraction

• Temperature-scaled soft-argmax converts the trained spatial heatmap into a continuous sub-pixel coordinate, avoiding the quantization noise of argmax coordinates.The model trains heatmaps with KL divergence rather than direct coordinate regression.Source: §3.3 · pp. 4–5

3.4 Training Dynamics: Curriculum Label Smoothing via Variance Decay

• The Crescent Target combines a radial prior, angular phase mask, and asymmetric temporal weighting; curriculum variance decay shrinks a broad annulus into a sharper target.The early frames are down-weighted because the mass distribution is unobservable in a static frame, while later tumbling and torque reveal inertial mechanics.Source: §3.4 · p. 5

4.1 The HiddenMass-50K Benchmark

• HiddenMass-50K contains roughly 50,000 valid MuJoCo sequences with asymmetric payloads and camera and depth variation, and evaluation uses a held-out 10,000-sequence validation set.Sixteen-frame sequences at 24 FPS capture high-angular-velocity tumbling dynamics.Source: §4.1 · p. 5, §4 · p. 5

4.2 Evaluation Metrics

• N-CoME normalizes mean coordinate distance by the projected box diagonal, jitter normalizes acceleration error, and Physics Capture measures movement from visual centroid toward hidden mass.The experiments also assess prediction entropy and describe expectation collapse toward the geometric centroid.Source: §4.2 · p. 6

4.3 Physically-Constrained High-Fidelity Ground Truth

• The real-world test comprises 63 disjoint sequences and 1,008 frames, with CoM ground truth calibrated using a fulcrum balance test, ArUco anchoring, and pose estimation.Models receive no prior camera-intrinsic or object-mass information under calibrated camera conditions.Source: §4.3 · p. 6

4.4 Quantitative Results: The metric artifact and disentanglement

• DINOv2 achieves only 2.6% Physics Capture, while STATERA-50K-Sigma collapses near the geometric centroid despite artificially minimizing jitter.The authors interpret these predictions as visual tracking or expectation collapse rather than recovery of hidden Newtonian momentum.Source: §4.4 · pp. 6–7
• 41.0% Physics Capture for STATERA-50K-Crescent makes it the only architecture with substantial positive disentanglement, despite 16.09% N-CoME versus 15.19% for the geometric centroid.The authors attribute the higher absolute error to monocular vector overshoot along the correct kinematic direction.Source: §4.4 · p. 7, Table 2 · p. 8
• Temperature scaling exposes a trade-off between Euclidean accuracy and kinematic phase: τ = 1.25 gives 15.81% N-CoME, while τ = 1.0 gives 16.09% N-CoME and maximizes Physics Capture.The selected τ = 1.0 reaches 40.96% Physics Capture; higher temperatures sacrifice disentanglement for a marginal jitter improvement.Source: §4.4 · p. 8

4.5 Discussions

• Removing the 1D temporal convolution degrades N-CoME to 38.74%, while phase-aware training can produce bimodal predictions by memorizing simulator resting states.The discussion also says ArUco markers serve only as ground-truth anchors, not explicit state inputs.Source: §4.5 · p. 8

• The conclusion calls for separating Euclidean spatial error, temporal kinematic jitter, and physical directional phase in evaluation.It also describes a duality between Crescent’s settling-state convergence and Sigma’s expectation collapse.Source: §5 · p. 9
• Performance may degrade without impact events, and predicted CoM can overshoot object bounds, limiting safe robotic grasping.The authors say tactile or proprioceptive priors are needed for safe grasping.Source: §5 · p. 9
• Future work proposes fiducial conditioning for absolute 3D coordinates, broader textured-object evaluation, and dynamic temporal-mask gating.The authors also propose mapping kinematic embeddings to latent diffusion models for Newtonian momentum in generated video.Source: §5 · p. 9

• The authors release code, procedural dataset-generation and training scripts, and pretrained STATERA weights, including evaluated ablations.The scripts are hosted on GitHub; datasets and model weights are hosted on Hugging Face.Source: A Reproducibility Statement · p. 12
• The public dataset splits include 40K training sequences, a 50K-test-public split, and 1K ablation datasets.The 50K-test-public split includes the 10K simulation test set and the 63-sequence real-world test set.Source: A Reproducibility Statement · p. 12
• Ground-truth CoM coordinates for both test sets are withheld, with evaluations planned through a leaderboard.Source: A Reproducibility Statement · p. 12

B Extended Methodology and Training Details

• The Crescent target places a probability ring around the geometric centroid and applies a Von Mises angular mask toward the true CoM.Its radius is the distance between the visual centroid and true CoM.Source: B Extended Methodology and Training… · p. 12
• The curriculum concentration parameter κ rises as α increases linearly over the 10-epoch funnel schedule.The supplied passage defines κ = 50.0α and α = min(1.0, epoch/funnel_epochs).Source: B Extended Methodology and Training… · p. 12
• A temporal quadratic weighting mask is applied to the sequence loss.Source: B Extended Methodology and Training… · p. 12

B.1 Hardware Requirements and Parameter Efficiency

• Training updates the 2.5D decoder, with about 2.5M parameters, and two adaptation blocks; the pipeline ran in under 120 hours on one RTX 5070 Ti.The backbone is partially frozen; training used mixed-precision gradient accumulation.Source: §B.1 · p. 12

B.2 Compute Constraints

• STATERA reached positive disentanglement and state-of-the-art performance in 10 epochs on one consumer RTX 5070 Ti.Source: §B.2 · p. 12
• Processing 16-frame tubelets through a ViT-L V-JEPA backbone creates latency bottlenecks for high-frequency closed-loop robotic control.The authors say practical deployment will require distilling the temporal priors into lighter, real-time architectures.Source: §B.2 · p. 12
• Across training epochs, Sigma decays to an isotropic impulse while Crescent strengthens its angular mask to optimize for rotational torque direction.Both targets begin as identical broad Gaussian rings at epoch 0.Source: Figure 4 · p. 13

C Extended Metric Formulations

• Physics Capture measures how far predictions move along the true offset vector over the dynamic trajectory.It is defined using a sequence-based Ratio of Means.Source: C Extended Metric Formulations · p. 13

D Real-World Ground Truth Pipeline

• OpenCV SQPnP solves absolute 3D pose with stateful extrinsic tracking, followed by quadratic kinematic filtering and SO(3) alignment with reflection.The alignment resolves chirality differences between MuJoCo Z-Up and OpenCV Y-Down coordinates.Source: D Real-World Ground Truth Pipeline · p. 13
• The calibrated ground-truth pipeline combines pose tracking, kinematic regression, and coordinate-chirality alignment for real-world measurements.The pose is solved with SQPnP and filtered to reduce high-frequency sensor noise.Source: D Real-World Ground Truth Pipeline · p. 13
• Inference uses calibrated ArUco projections to stabilize a single fixed bounding box spanning the object's 16-frame trajectory.This static observation window preserves macro-translation and absolute depth cues.Source: D Real-World Ground Truth Pipeline · p. 13
• The model infers hidden mass from parabolic translation and rotational torque simultaneously, using unconstrained Newtonian momentum.The crop does not dynamically resize the object's pixel-to-meter ratio.Source: D Real-World Ground Truth Pipeline · pp. 13–14
• Standardized camera intrinsics across 63 test sequences preserve a fixed physical pixel-to-meter ratio in the crop.This standardization makes Head B's absolute Z-Depth regression well-posed.Source: D Real-World Ground Truth Pipeline · p. 14

E Extended Architecture Ablations

• Freezing V-JEPA adaptation blocks degrades the 1K anchor to 38.74% N-CoME and 25.76% Capture; removing Z-Depth constraints causes unbounded spatial overshoot.The passage identifies Head B as necessary to anchor 2D extractions to an absolute 3D boundary.Source: E Extended Architecture Ablations · p. 14, Figure 6 · p. 16
• Standard Softmax at τ = 1.0 during training reintroduces the quantization staircase and raises Normalized Kinematic Jitter by roughly 20%.Source: E Extended Architecture Ablations · p. 14
• HMS 53.2 for STATERA-1K-Anchor exceeds DINOv2's 39.4, while VideoMAE v2 reaches only 18.17% Physics Capture despite identical sequence mixing.DINOv2's static patches lack intra-frame momentum; the passage attributes stronger kinematic extraction to V-JEPA's predictive temporal priors and tubelets.Source: E Extended Architecture Ablations · p. 14
• Phase-aware Crescent targets can memorize settling orientation, creating bimodal predictions during aerial bounces and high jitter; curriculum-smoothed Sigma targets avoid aliasing but collapse toward safe centroids.The paper argues for accepting some bimodal Euclidean penalties to retain physical disentanglement.Source: E Extended Architecture Ablations · p. 14, Figure 5 · p. 15
• Scaling the unregularized model to 50K grounds predictions within realistic physical boundaries, reducing spatial error while achieving 40.96% Capture.The 1K-No-Z-Depth model's apparent 36.21% Capture accompanies unusable 45.11% N-CoME from overshoot.Source: E Extended Architecture Ablations · p. 15, Figure 6 · p. 16
• Surviving surface tracks drift toward the CoM because extremal points move faster and self-occlude, creating an apparent tracking signal rather than reliable internal-mass estimation.STATERA's markers are extrinsic anchors only, not state inputs.Source: E Extended Architecture Ablations · pp. 15–16
• 40.96% Capture: STATERA maintains empirical superiority across metrics despite an overshooting Euclidean penalty; TAPIR's 8.37% Capture reflects survivorship bias from occlusion.ArUco markers anchor ground-truth pose but are not provided as explicit state inputs.Source: E Extended Architecture Ablations · pp. 15–16

G Broader Impacts

• Calibrated monocular kinematic tracking could let robots infer internal mass without explicit sensors or manual calibration, but hidden-payload tracking also raises dual-use concerns.The passage calls for ethical review of deployments involving privacy or intent.Source: G Broader Impacts · p. 16

F Benchmark Profile Visualization

• HMS weights Kinematic Accuracy at 25%, Tracking Stability at 25%, and Physics Disentanglement at 50%; Crescent breaks centroid bias and scores highest overall at 59.6.DINOv2 and Sigma show low disentanglement and remain closer to the geometric centroid.Source: F Benchmark Profile Visualization · p. 16, Figure 7 · p. 17