Source-linked AI summary
RAM: Recover Any 3D Human Motion in-the-Wild
Sen Jia, Ning Zhu, Jinqin Zhong, Jiale Zhou, Huaping Zhang, Jenq-Neng Hwang, Lei Li
TL;DR
RAM addresses unstable identity association and discontinuous 3D reconstruction in multi-person monocular videos under occlusion, fast motion, and viewpoint changes. It combines motion-aware tracking, temporal mesh recovery, motion prediction, and gated fusion, achieving state-of-the-art tracking and reconstruction across in-the-wild benchmarks. The framework is intended to support stable, zero-shot multi-person motion recovery in long real-world videos.
Problem
Existing multi-person recovery methods can lose identities and produce fragmented or unstable motion under fast motion, severe occlusion, and viewpoint changes.
Method
RAM combines motion-aware semantic tracking, memory-based temporal mesh reconstruction, future-pose prediction, and gated fusion for robust multi-person recovery.
Results
RAM achieves state-of-the-art tracking and recovery performance across PoseTrack and 3DPW, including 74.4 MOTA on PoseTrack21 and 53.0 MPJPE with 34.1 PA-MPJPE.
Takeaways & Limitations
RAM provides a foundation for zero-shot, stable multi-person motion recovery in long, real-world monocular videos without retraining on PoseTrack.
Takeaways & Limitations
Prior approaches remain limited by dataset-specific tuning and motion matching, which constrain generalization and robustness in-the-wild.
Abstract
from arXiv · showhide
RAM incorporates a motion-aware semantic tracker with adaptive Kalman filtering to achieve robust identity association under severe occlusions and dynamic interactions. A memory-augmented Temporal HMR module further enhances human motion reconstruction by injecting spatio-temporal priors for consistent and smooth motion estimation. Moreover, a lightweight Predictor module forecasts future poses to maintain reconstruction continuity, while a gated combiner adaptively fuses reconstructed and predicted features to ensure coherence and robustness. Experiments on in-the-wild multi-person benchmarks such as PoseTrack and 3DPW, demonstrate that RAM substantially outperforms previous state-of-the-art in both Zero-shot tracking stability and 3D accuracy, offering a generalizable paradigm for markerless 3D human motion capture in-the-wild.
1. Introduction
RAM targets robust, real-time multi-person 3D motion recovery from monocular video, addressing identity instability and fragmented reconstruction under occlusion, fast motion, and viewpoint changes. It combines motion-aware tracking, temporal recovery, and prediction, achieving state-of-the-art performance and faster real-time processing.
- Multi-person 3D motion recovery seeks temporally coherent human meshes in real time for applications including sports analytics, rehabilitation, and virtual content creation.
- RAM performs online monocular-video reconstruction while achieving real-time performance 2–3× faster than previous approaches.
- Existing multi-person methods are sensitive to fast motion, severe occlusion, and viewpoint changes, causing identity switches, lost tracks, and fragmented trajectories.
- RAM achieves state-of-the-art multi-person 3D motion recovery on PoseTrack and 3DPW, including zero-shot tracking stability, reconstruction accuracy, and inference efficiency.
- RAM combines motion-aware semantic tracking, temporal mesh recovery, and prediction to maintain identity continuity and robust reconstruction under occlusion.
2. Related Works
Related work has progressed from appearance-based tracking and single-frame human reconstruction toward unified, memory-aware multi-person video recovery. However, dataset-specific tuning and motion matching can limit generalization and robustness in-the-wild.
- Tracking and Motion Modeling: Modern tracking methods increasingly integrate target representation, association, and memory mechanisms to improve temporal stability.
- Human Motion Reconstruction: Human motion reconstruction has evolved from single-person, single-frame estimation toward multi-person, video-based recovery, but crowded scenes remain challenging.
- RAM Framework: RAM’s overview combines SegFollow, Temporal HMR, a Predictor, and a gated Combiner for tracking, reconstruction, forecasting, and recovery.
- Limitations of Prior Work: Dataset-specific tuning and motion matching limit generalization and robustness in in-the-wild scenarios.
3. Method
RAM is a unified framework for real-time, accurate multi-person 3D human motion reconstruction from monocular videos. Its four modules combine stable tracking, memory-based mesh estimation, future-pose prediction, and gated fusion.
- SegFollow: SegFollow performs motion-aware semantic tracking to maintain identity associations across frames under occlusion and rapid motion.
- T-HMR: T-HMR reconstructs 3D human meshes using temporal context and memory-augmented attention for coherent estimation.
- Predictor: Predictor models motion dynamics from past reconstructions and forecasts future pose states when current observations are unreliable.
- Combiner: Combiner uses learnable gating to fuse predictions and reconstructions into stable SMPL outputs with improved temporal consistency under uncertainty.
3.1. SegFollow Module
SegFollow augments SAM2 with motion-guided association, confidence-gated Kalman updates, and an adaptive temporal buffer. These mechanisms preserve reliable motion state and memory during fast motion, occlusion, and noisy observations.
- Module Design: SegFollow combines SAM2 with a motion-guided selector and temporal buffer to provide explicit motion reasoning and stable long-term tracking.
- Motion-Guided Selection: The selector represents each detection with a bounding-box motion state and compares candidate masks against Kalman predictions for motion consistency.
- Motion-Guided Selection: The highest fused mask score determines the selected association, improving tracking robustness during fast motion or occlusion.
- Confidence-Gated Update: The confidence-gated update increments a reliability counter and updates the Kalman posterior only after sufficient evidence accumulates.
- Temporal Buffer: The temporal buffer uses confidence-modulated exponential averaging to preserve historical memory when motion evidence is uncertain and incorporate current features when reliable.
- Module Outcome: Together, the motion-guided selector and temporal buffer enable zero-shot, occlusion-robust tracking for downstream 3D reconstruction.
3.2. T-HMR
T-HMR reconstructs 3D human meshes by combining current-frame features with selected temporal priors. Its Memory Cache filters temporal features by relevance and consistency, while MemFormer integrates the selected memory into SMPL regression.
- T-HMR: T-HMR uses Memory Cache and MemFormer to inject temporal priors into SMPL regression for coherent 3D mesh reconstruction.Current-frame features alone can produce temporal inconsistency and weak occlusion robustness.
- Memory Cache: The Memory Cache selects the top-k relevant frame features from a temporal window centered on the current frame.It collects ViT features from adjacent frames and forms a memory feature representation.
- Memory Cache: Memory-frame importance combines cross-frame relevance with intra-frame consistency to rank temporal features.The first branch measures current-to-memory dependencies, while the second evaluates consistency among memory frames.
- Memory Cache: The highest-scoring frames form the memory bank, discarding redundant features while retaining cues useful under occlusion and motion blur.This selection also supports efficient MemFormer reasoning.
- MemFormer: MemFormer concatenates a learnable SMPL token with current-frame features, then applies self-attention and temporal cross-attention to integrate memory features.Pooled memory features provide motion-consistent keys and values for temporal reasoning.
3.3. Predictor
The Predictor forecasts the next-frame latent motion representation from a queue of recent reconstructed states. Transformer blocks model motion dynamics, and the prediction serves as a motion-conditioned prior when current visual cues are unreliable.
- Predictor: The Predictor maintains a FIFO queue of recent reconstructed states and feeds it into stacked Transformer blocks to model motion dynamics.The queue is updated online during inference for adaptation to changing motion patterns.
- Predictor: The Predictor forecasts the next-frame latent representation, which the Combiner uses as a motion-conditioned prior for stable reconstruction.This prior supports continuity when current-frame visual cues are unreliable.
3.4. Combiner
The Combiner fuses current T-HMR features with the Predictor’s latent motion prior through a learned gate. It favors observed features when evidence is reliable and shifts toward prediction under occlusion or uncertainty before regressing SMPL parameters.
- Combiner: The Combiner integrates the current T-HMR feature and predicted latent motion prior using a gating vector produced by an MLP.The gated fusion combines reconstructed and predicted representations before SMPL regression.
- Combiner: A weighted interpolation fuses current reconstruction features with the predicted motion prior.The gate controls the contribution of each representation.
- Combiner: The fusion favors T-HMR features under confident observations and shifts toward the predicted prior under occlusion or uncertainty.This design supports stable and consistent motion recovery.
- Combiner: A regression head maps the fused feature to SMPL parameters for next-frame reconstruction.
3.5. Training Objectives
RAM uses three training stages: single-image pretraining, temporal-predictor training, and full-framework fine-tuning for dynamic fusion under occlusion. Occlusion simulation teaches the Combiner to balance visual reconstruction with predicted motion.
- Stage 1: Stage 1 pretrains T-HMR’s image encoder and pose regression module on large batches of single images using multiple SMPL-related objectives.The objectives include losses on 2D projections, pelvis-relative 3D joints, rotations, shape parameters, and prediction confidence.
- Stage 2: Stage 2 trains the temporal Predictor with scheduled sampling on 8-frame and synthetic 3D pose sequences.The Predictor is unrolled over time to forecast pose states through the sequence.
- Stage 3: Stage 3 fine-tunes the full framework for dynamic fusion while keeping the Predictor and RAM components frozen.The training simulates occlusion by randomly masking 60% of human body regions.
- Stage 3: Occlusion-aware fine-tuning encourages the Combiner to rely more on predicted motion when visual cues are incomplete.The strategy trains adaptive balance between reconstruction and prediction under simulated occlusion.
4. Experment
RAM is evaluated across tracking, pose estimation, and 3D recovery benchmarks, including zero-shot sports videos with occlusion, fast motion, and multi-person interaction. Across these settings, it delivers stronger identity stability, accuracy, robustness, and real-time generalization than prior methods.
- Evaluation setup: RAM is evaluated on PoseTrack, COCO, TrackID3x3, Olympic Boxing, and 3DPW, covering 2D/3D estimation, tracking, occlusion, fast motion, and dense interactions.3DPW provides mesh and joint annotations in unconstrained outdoor settings.
- Tracking results: 66.4 HOTA with 15 ID switches on PoseTrack18 demonstrates substantially stronger identity stability than prior methods.On PoseTrack21, RAM reaches 74.4 MOTA, with +6.4 IDF1 and +82% FPS over CoMotion.
- Zero-shot tracking: +78% indoor and +116% outdoor TI-HOTA over CoMotion establishes stronger zero-shot generalization on TrackID3x3.The benchmark uses longer real-world sports videos with frequent occlusions and fast multi-person interactions.
- Ablation: SAM2-only tracking improves outdoor TI-HOTA over CoMotion by +7.7, whereas full RAM achieves a +35.8 gain.The ablation attributes the difference to motion-aware designs that complement semantic initialization with motion priors and temporal modeling.
- 2D pose estimation: RAM achieves 0.93 PCK@0.05 on PoseTrack and 0.89 on COCO, outperforming prior methods in dense, occluded multi-person settings.The reported gains include comparison with CoMotion, which already incorporates temporal cues.
- 3D recovery: 53.0 MPJPE and 34.1 PA-MPJPE on 3DPW are the lowest reconstruction errors among the compared methods.The text attributes improved frame-to-frame consistency to T-HMR’s temporal priors and the combination of reconstruction with predictive motion priors.
- Qualitative results: RAM sustains identity and reconstruction quality in long, dynamic sports videos, achieving real-time tracking and accurate 3D recovery under occlusion and motion blur.Qualitative comparisons report failures and high latency for 4DHumans and CoMotion in these conditions.
5. Conclusion
RAM provides a unified framework for real-time, robust multi-person 3D motion reconstruction from monocular videos. By combining semantic tracking, motion-aware modeling, temporal priors, and prediction, it mitigates identity loss and supports smooth, accurate reconstruction under severe occlusions.
- RAM integrates semantic tracking with motion-aware modeling to mitigate ID switches and tracking loss under occlusion and viewpoint changes.
- Temporal priors and motion prediction enable smooth and accurate reconstruction under severe occlusions.
- Extensive experiments show consistent improvements over prior work in tracking stability, reconstruction quality, and computational efficiency, especially in complex real-world sports scenes.
- RAM provides a foundation for future human-centric motion-understanding research and is intended for extension to human-object recovery.