SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videos

1. Introduction

• Video HMR methods that model temporal information or use tracking are often optimization-based and require annotated video data and carefully designed objectives.The paper identifies these requirements as constraints on scalability and robustness to diverse in-the-wild motion and scene dynamics.Source: §1 · p. 1
• SAM-Body4D is a training-free framework for temporally consistent and robust human mesh recovery from in-the-wild videos.It is designed to scale to multiple people and frames.Source: §1 · p. 2
• Identity-consistent masklets serve as prompts for producing 4D mesh trajectories across complex, dynamic scenes.The masklets carry temporal continuity from the video into the mesh predictions.Source: §1 · pp. 1–2
• Occlusion-aware refinement improves reconstruction quality when body regions are hidden.The method refines masklets to recover missing or corrupted regions and provide fuller visual references for HMR.Source: §1 · pp. 1–2, Figure 1 · p. 2

Human Mesh Recovery

• Image-based HMR includes regression-based methods such as HMR and token-based methods such as TokenHMR.The passage distinguishes direct regression of 3D model parameters from token-based structured prediction.Source: §2 · p. 2
• Feature-level temporal methods such as VIBE and TRAM aggregate frame-wise representations, while 4DHumans combines mesh reconstruction with identity-consistent tracking.The paper notes that these approaches are optimization-based and require annotated video data and designed loss functions.Source: §2 · p. 2
• SAM-Body4D addresses these gaps with training-free temporal coherence from pixel-level human continuity and an occlusion-aware refinement module.This contrasts with approaches that rely on feature- or pose-space temporal modeling and annotated video data.Source: §2 · pp. 2–3

Video Object Segmentation

• SAM-based VOS methods segment visible regions but cannot recover occluded body parts, leaving incomplete cues for downstream HMR and potentially hallucinated geometry.SAM 2 extends segmentation to video with a memory mechanism; SAM 3 (Carion et al., 2025) adds text prompts and an object perceiver.Source: §2 · p. 3

• SAM 3 predicts a video mask sequence from user-defined prompts, combining propagation and detection through a match-and-update step.Propagation transfers mask labels using historical predictions; detection associates prompts with objects in the current frame.Source: §3 · p. 3
• The two SAM 3 modules address different cases: propagation preserves continuity, while detection handles disappearance and reappearance; their integration is reported to outperform SAM 2 and other VOS methods on complex videos.Source: §3 · p. 3
• SAM 3D Body accepts optional encoder prompts, such as 2D keypoints or segmentation masks, to focus on the target human.Source: §3 · p. 3
• Its decoder uses optional keypoint, camera, or MHR prompts to predict full-body tokens, then an MLP maps the first output token to pose, shape, camera-pose, and skeleton parameters.Body and hand are optimized separately and later fused.Source: §3 · p. 3

4.1. Overview

• Three stages connect identity-consistent masklets, occlusion-aware mask refinement, and mask-guided HMR, propagating masklet continuity to reconstructed meshes.The refined masks serve as spatial prompts for mesh prediction.Source: §4.1 · p. 3, Figure 2 · p. 4

4.2. Masklet Generator

• SAM 3 processes prompted target humans across the video to produce spatio-temporally aligned masklets that maintain identity consistency.Other video segmentation models that produce temporally aligned instance masks can also be used.Source: §4.2 · p. 3

4.3. Occlusion-Aware Masklet Refiner

• Diffusion-VAS completes masklets; detected frames are temporally grouped and reprocessed to recover missing pixels and produce refined video and masks.The refined inputs provide full-body cues and improved temporal stability for subsequent HMR.Source: §4.3 · p. 4

4.4. Training-Free Mask-Guided HMR

• Refined masks act as encoder prompts for per-frame mesh prediction, keeping meshes aligned with identities over time without task-specific fine-tuning.The mask-guided HMR pipeline operates training-free.Source: §4.4 · p. 4
• Padding unifies batches with varying numbers of people for joint inference; temporal smoothing and first-visible-frame scale and shape reuse further stabilize trajectories.These operations require no learning and add negligible computational overhead.Source: §4.4 · pp. 4–5

5. Experiments

• Experiments combine SAM 3 for masklet generation, Diffusion-VAS for refinement, and SAM 3D Body for HMR; detection and refinement run at 512 × 1024 resolution.Lower resolutions can improve efficiency with a moderate loss in visual fidelity.Source: §5 · pp. 5–6
• 2× faster processing was reported for a 90-frame, 480 × 854 video with five people and batch size 32 on an NVIDIA A100-SXM4-80GB, without the refiner.With the refiner enabled, runtime and memory use increase depending on occlusion duration and the number of people refined.Source: §5 · p. 6

5.1. Comparison with Vanilla Image-to-Video Extension

• Masklet guidance maintains continuous, identity-preserving mesh trajectories, while vanilla per-frame HMR can miss meshes when detection fails.Missed detections may occur when people are small, motion-blurred, or heavily occluded.Source: §5.1 · p. 6

5.2. Effectiveness of Occlusion-Aware Refinement

• With extensive occlusion, vanilla SAM 3D Body produces distorted, unstable meshes, whereas refinement restores missing regions and preserves plausible pose and body structure.The paper identifies recovery of occluded evidence as a way to mitigate hallucinated predictions in challenging videos.Source: §5.2 · p. 6

6. Conclusion

• SAM-Body4D improves temporal stability and robustness in challenging in-the-wild scenarios without extra training or architectural modification to SAM-3D-Body.Parallel multi-frame inference further supports efficient and scalable deployment.Source: §6 · p. 6