PaperlayerView exact revision on arXiv

Source-linked AI summary

Human3R: Everyone Everywhere All at Once

Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, Gerard Pons-Moll

arXiv:2510.06219v2cs.CV

TL;DR

Human3R addresses the challenge of unified online 4D reconstruction of multiple humans, scenes, and cameras from casually captured monocular video. It extends CUT3R with parameter-efficient visual prompt tuning and achieves competitive or state-of-the-art performance across several tasks while running at 15 FPS with 8 GB memory. Its current scope is limited by reliance on visible head keypoints and proxy SMPL meshes without clothing or appearance.

  • Problem

    Existing human-scene reconstruction approaches rely on multi-stage refinement and heavy dependencies, while annotated datasets for joint human, scene, and camera reconstruction remain limited.

  • Method

    Human3R extends CUT3R with visual prompt tuning to jointly reconstruct multi-person SMPL-X bodies, dense scenes, and camera trajectories in one forward pass.

  • Results

    Across global motion, local mesh, video depth, and camera pose tasks, Human3R achieves superior or competitive performance with 15 FPS inference and an 8 GB memory footprint.

  • Takeaways & Limitations

    Human3R provides a unified real-time baseline for online human-scene reconstruction from casually captured videos.

  • Takeaways & Limitations

    Human3R can fail when heads are not visible or multiple heavily occluded people share a head token, and its proxy SMPL meshes omit clothing and appearance.

Abstract

from arXiv · show

We present Human3R, a unified, feed-forward framework for online 4D human-scene reconstruction, in the world frame, from casually captured monocular videos. Unlike previous approaches that rely on multi-stage pipelines, iterative contact-aware refinement between humans and scenes, and heavy dependencies, e.g., human detection, depth estimation, and SLAM pre-processing, Human3R jointly recovers global multi-person SMPL-X bodies ("everyone"), dense 3D scene ("everywhere"), and camera trajectories in a single forward pass ("all-at-once"). Our method builds upon the 4D online reconstruction model CUT3R, and uses parameter-efficient visual prompt tuning, to strive to preserve CUT3R's rich spatiotemporal priors, while enabling direct readout of multiple SMPL-X bodies. Human3R is a unified model that eliminates heavy dependencies and iterative refinement. After being trained on the relatively small-scale synthetic dataset BEDLAM for just one day on one GPU, it achieves superior performance with remarkable efficiency: it reconstructs multiple humans in a one-shot manner, along with 3D scenes, in one stage, in real-time (15 FPS) with a low memory footprint (8 GB). Extensive experiments demonstrate that Human3R delivers state-of-the-art or competitive performance across tasks, including global human motion estimation, local human mesh recovery, video depth estimation, and camera pose estimation, with a single unified model. We hope that Human3R will serve as a simple yet strong baseline, which can be easily adapted for downstream applications. Code, models and 4D interactive demos are available at https://fanegg.github.io/Human3R/.

1 INTRODUCTION

Human3R targets online 4D reconstruction of people and their surrounding 3D world from visual data, avoiding the multi-stage pipelines and dependencies used by prior methods. It unifies human, scene, and camera reasoning with CUT3R-derived priors, visual prompt tuning, and efficient streaming inference.

  • Motivation: The broader task is motivated by the need to interpret human behavior within its surrounding 3D environment for applications including AR/VR and autonomous navigation.The paper frames joint reconstruction of global human motion and scene geometry as challenging but fundamental.
  • Limitations of prior work: Prior approaches separately reconstruct and refine humans and scenes, requiring hours-long pipelines plus human detection and tracking for top-down mesh regression.These dependencies make multi-person reconstruction slower and more complex.
  • Contribution: Human3R unifies human, scene, and camera reconstruction in one online, all-at-once model running at 15 FPS on an RTX 4090.It replaces separate component models and iterative refinement with a single lightweight pipeline.
  • Method: Human3R extends CUT3R’s spatiotemporal reconstruction priors through parameter-efficient visual prompt tuning while keeping the CUT3R backbone frozen.The approach addresses limited annotated video data by relying on minimal tuning with BEDLAM training data.
  • Method: Human prompts act as SMPL-X queries that use head-keypoint spatial priors, image-token self-attention, and persistent-state cross-attention to produce scene-aware whole-body estimates.The design uses the persistent reconstruction state to connect human estimates with the surrounding scene.

2 RELATED WORKS

Related work spans local mesh recovery, global human motion estimation, and joint human-scene reconstruction. Existing methods commonly rely on optimization, motion or geometric priors, SLAM, or multiple external estimators.

  • Local Human Mesh Recovery: Local human mesh recovery methods estimate parametric body pose and shape from images, landmarks, or sensor trajectories, typically in the camera frame.The literature includes optimization-based fitting and learning-based regression of SMPL-family parameters.
  • Global Human Motion Estimation: Global human motion estimation addresses the ill-posed task of recovering world-grounded humans using motion priors, camera estimation, geometric constraints, or recurrent online reconstruction.Many approaches still depend on external detection, segmentation, depth, or SLAM-related estimators.
  • Human-Scene Reconstruction: Joint human-scene reconstruction methods often optimize camera poses, scenes, and SMPL parameters from multi-view imagery, while newer optimization-free methods seek more direct reconstruction.These approaches connect human reconstruction with scene and camera estimation rather than treating the components independently.

3 METHODS

Human3R extends CUT3R with human-specific prompts to jointly reconstruct multiple world-frame SMPL-X bodies, scenes, and cameras online. Its feed-forward design combines head detection, human priors, recurrent scene context, and parameter-efficient tuning while preserving the frozen reconstruction backbone.

  • Human3R formulation: Human3R estimates multiple SMPL-X bodies, camera poses, and canonical point clouds from each streaming image in real time.Bodies use SMPL-X with 10,475 vertices and 54 joints; camera extrinsics and intrinsics, plus canonical point clouds, are also predicted.
  • CUT3R foundation: CUT3R recurrently updates a fixed-size state containing spatial and temporal scene history, enabling online metric-scale point-map and camera-pose reconstruction.Image tokens and camera tokens interact with the previous state, which is updated with each new observation.
  • One-stage global reconstruction: Human3R fine-tunes CUT3R with visual prompt tuning, keeping the backbone frozen while human-related layers read out global human meshes.Trainable visual prompts are inserted into the decoder input, while the remaining CUT3R parameters stay frozen.
  • Human detection and prompting: Detected head tokens are converted into human prompts that attend to image tokens for whole-body aggregation and to scene state for temporally consistent, scene-aware SMPL-X estimates.The human prompts act as discriminative human-ID queries and predict mesh parameters in parallel for detected people.
  • Human prior: Multi-HMR ViT-DINO features are concatenated with CUT3R head tokens to inject detailed human pose and shape priors, while the human encoder remains frozen.The resulting prompts support fine-grained human reconstruction and training-free human segmentation and tracking.
  • Test-time sequence length adaptation: Test-time training updates CUT3R’s memory state to adapt sequence processing beyond the four-image training context while balancing historical retention with new observations.The state is updated through a fast-weight gradient rule using the current image, camera, and human prompts.

4 EXPERIMENTS

Human3R is evaluated as a unified online system for local and global human reconstruction, scene and camera estimation, and crowded-scene generalization. It achieves strong benchmark performance while retaining one-shot, real-time inference and robustness to several camera, truncation, and occlusion conditions.

  • Local Human Mesh Reconstruction: 10% improvement on both MPJPE and PVE on EMDB-1, with Human3R surpassing multi-stage and one-stage methods across all local-mesh metrics.The comparison includes methods with detection, cropping, or ground-truth intrinsic dependencies, whereas Human3R does not require those additional inputs in the stated setting.
  • Global Human Motion Estimation: 20% lower W-MPJPE and 60% lower RTE than WHAM on EMDB-2, while jointly reconstructing world-space humans, scenes, and camera poses online.Human3R also avoids precomputed detection, segmentation, camera poses, depth, contact, and iterative refinement required by some competing pipelines.
  • Camera Pose and Video Depth Estimation: Human3R+TTT3R improves camera pose and metric-scale video depth estimation over naive TTT3R, while online foundation models trade memory efficiency against long-sequence accuracy.VGGT and StreamVGGT use full attention and can be slow or run out of memory; CUT3R uses low GPU memory but may forget long sequences.
  • Generic 3D Reconstruction: Human3R achieves state-of-the-art human mesh recovery and slightly improves generic 3D reconstruction after human prompt tuning and TTT3R integration.The reported gains support mutual benefits from jointly reasoning about humans and scenes.
  • Generalization to Crowded Scenes: Human3R recovers multiple human meshes in one shot with inference speed independent of crowd density, and generalizes qualitatively to in-the-wild crowds exceeding 10 people.The crowded-scene evaluation is qualitative because common ground-truth benchmarks contain only one or two people.
  • Robustness and Ablations: Human3R remains consistent without camera intrinsics and retains identities through some head occlusions, while severe shared-head-token occlusion can prevent subject differentiation.Human-prior injection also improves fine-grained pose and shape reconstruction, and naive CUT3R+Multi-HMR combinations underperform the unified model for global humans.

5 CONCLUSION

The conclusion presents Human3R as an efficient one-stage approach for 4D human-scene reconstruction. It reports competitive or state-of-the-art performance across human motion recovery and general 3D reconstruction, while identifying important limitations in human representation and visibility.

  • 5 CONCLUSION: Human3R provides a feasible strategy for efficient finetuning and real-time inference through one-stage 4D human-scene reconstruction.The conclusion frames this as the method's central practical contribution.
  • 5 CONCLUSION: Human3R achieves competitive or state-of-the-art performance in human motion recovery and general 3D reconstruction and generalizes to casually captured videos.This conclusion summarizes the reported evaluation scope without restricting the claim to a single benchmark.
  • 5 CONCLUSION: Human3R relies on the head as a discriminative keypoint, can fail when the head is not visible, and represents humans with proxy SMPL meshes lacking clothing or appearance.The authors suggest pixel-aligned body-point prompts and 3DGS anchored on SMPL as directions for addressing these limitations.

A.1 ROBUSTNESS TO TRUNCATION AND OCCLUSION

Human3R is robust to partial head visibility and some severe truncation or occlusion by using adjacent anatomical points, but head visibility remains a limiting factor. Multiple people sharing one head token can still cause missed differentiation or tracking difficulties.

  • Truncation and Occlusion Robustness: Human3R detects people when only a small head portion or adjacent points such as the chest, back, or neck are visible.The method uses cropping augmentation and supervises the visible point closest to the unobserved head center when the head is truncated.
  • Truncation and Occlusion Robustness: Multiple people occupying the same head token can cause head-detection misses, although the method can re-associate subjects once the head or adjacent parts become visible.The shared-token case remains a limitation for differentiating heavily occluded subjects.

A.2 ROBUSTNESS TO NON-FULL BODY CAPTURES

Human3R remains effective on partial-body and challenging multi-person captures, while exposing limitations on dynamic, close-up, and extreme-pose inputs. Its lightweight variants also support real-time or competitive-speed inference across diverse datasets.

  • Non-full-body robustness: Human3R reconstructs consistent scenes and camera poses from upper-body-dominant views, while inferring physically plausible lower bodies grounded on the reconstructed plane.The model uses a learned scene-aware human pose prior to recover unobserved lower-body structure.
  • Limitations: Human3R fails to recover running motion in face-focused partial views, defaulting to a standing pose when visual evidence is missing.The authors attribute this to deterministic regression toward an average pose and suggest generative modeling as future work.
  • Limitations: Performance degrades for fine-grained scene details, unobserved extremities, and extreme out-of-distribution poses such as breakdancing.The authors identify larger training datasets and self-supervised solutions as possible directions.
  • Human mesh projection: Human3R preserves mesh projection accuracy comparable to GVHMR while retaining TRAM-like robustness to multi-person occlusion in crowded scenes.GVHMR is limited to single-person reconstruction, whereas TRAM supports multiple people with slightly lower projection accuracy.
  • Runtime: 13.59–15.87 FPS: the lightweight ViT-S/672 model operates at real-time speed across datasets and image resolutions.The largest ViT-L/896 variant remains at 5.06–5.38 FPS, showing a speed–capacity trade-off.

A.6 HUMAN3R BENEFITS FROM THE 3D AWARENESS OF CUT3R

Human3R benefits from CUT3R’s 3D awareness by improving robustness to missing camera intrinsics and out-of-distribution image aspect ratios. These gains support coherent human recovery while preserving joint scene, camera, and motion performance.

  • Intrinsic independence: Human3R outperforms Multi-HMR even when Multi-HMR receives ground-truth intrinsics, while requiring no intrinsic inputs.Multi-HMR performs substantially worse without ground-truth intrinsics across the reported metrics.
  • OOD robustness: Human3R remains consistently strong under out-of-distribution aspect ratios, whereas Multi-HMR shows sharp degradation in human- and scene-aware metrics.The comparison attributes this robustness to CUT3R’s metric-scale scene context.
  • Overall comparison: Human3R improves camera and scene prediction over CUT3R, local human details over Multi-HMR, and global human reconstruction over naive Multi-HMR+CUT3R combinations.These comparisons cover camera poses, scenes, local mesh recovery, and global human motion within the unified model.

A.8 ABLATION OF CROSS-ATTENTION

Cross-attention between human and scene branches is important for Human3R’s multi-task performance. Removing it harms 3D reconstruction, human mesh recovery, and global human motion estimation.

  • Cross-attention ablation: Removing cross-attention degrades performance across 3D reconstruction, human mesh reconstruction, and global human motion.The ablation supports the role of scene-derived 3D awareness and human–scene representation alignment.

A.9 COMPARISON OF JOINT TRAINING STRATEGIES

Human prompt tuning preserves CUT3R’s prior knowledge more effectively than jointly fine-tuning the decoder on BEDLAM. Full decoder fine-tuning suffers catastrophic forgetting and degrades key tasks.

  • Joint training strategies: Full decoder fine-tuning causes severe degradation in 3D reconstruction and global human motion because CUT3R loses 3D-awareness priors on the relatively small BEDLAM dataset.The result motivates parameter-efficient human prompt tuning rather than unrestricted decoder training.

A.10 ROBUSTNESS TO FEATURE RESOLUTION

Feature resolution affects Human3R’s human detection robustness, especially in crowded scenes, where coarse features can merge nearby instances.

  • A.10 ROBUSTNESS TO FEATURE RESOLUTION: Precision, recall, and F1-score are used to evaluate detection across feature scaling factors from ×1/8 to ×8.The analysis covers a broad range of input feature resolutions and measures validity, coverage, and their harmonic mean.
  • A.10 ROBUSTNESS TO FEATURE RESOLUTION: Token collision at lower resolutions merges multiple nearby human heads, producing false negatives in crowded scenes.The resulting coarse representation fails to distinguish individual instances within the crowded token.
  • A.10 ROBUSTNESS TO FEATURE RESOLUTION: Higher feature resolutions are recommended for robust human detection, particularly when multiple people appear close together.At coarse resolutions, distinct nearby heads can collapse into one feature token, causing missed detections.
  • A.10 ROBUSTNESS TO FEATURE RESOLUTION: Optimization-based SfM and SLAM pipelines are computationally intensive and generally unsuitable for real-time applications.They also struggle with small camera parallax, textureless surfaces, and dynamic elements, while typically producing sparse point clouds.

C TRAINING DETAILS

Human3R is trained by adapting pretrained reconstruction components on BEDLAM, while qualitative evaluation covers human motion, scene reconstruction, and camera poses. The paper also notes unresolved interaction constraints and lower accuracy than strong offline methods.

  • C TRAINING DETAILS: Human3R fine-tunes human-related modules on BEDLAM while freezing the pretrained CUT3R and Multi-HMR encoder weights.BEDLAM contains 3D scene depth and SMPL-X meshes for scenes with 1–10 people; 5,000 sequences are used for training and 1,000 for validation.
  • C TRAINING DETAILS: Qualitative evaluation on EMDB compares Human3R predictions with ground truth for global human motion, 3D scenes, and camera poses.The comparison is reported for EMDB subset 1.
  • C TRAINING DETAILS: Human3R does not yet resolve human interactions and has not matched strong offline methods such as JOSH in reconstruction accuracy.The model can serve as an initialization for applications requiring higher accuracy, where iterative optimization provides stronger constraints on contacts and interpenetration.
Loading 2510.06219v2…