PaperlayerView exact revision on arXiv

Source-linked AI summary

Coherent Human-Scene Reconstruction from Multi-Person Multi-View Video in a Single Pass

Sangmin Kim, Minhyuk Hwang, Geonho Cha, Dongyoon Wee, Jaesik Park

arXiv:2603.12789v2cs.CV

TL;DR

Existing human-scene reconstruction methods largely focus on monocular inputs or require additional modules, preprocessing, or optimization for multi-view settings. CHROMM unifies camera, scene, and multi-human reconstruction in a single pass using integrated priors, scale adjustment, fusion, and geometric association, achieving competitive estimation performance while running over 8× faster than prior multi-view approaches. Its performance may degrade when head regions are severely occluded or not visible.

  • Problem

    Existing approaches largely focus on monocular inputs, while multi-view reconstruction can require external modules, preprocessing, or optimization.

  • Method

    CHROMM integrates Pi3X and Multi-HMR priors with head–pelvis scale adjustment, test-time multi-view fusion, and geometry-based multi-person association.

  • Results

    CHROMM achieves competitive performance in global human motion and multi-view pose estimation while running over 8× faster than prior multi-view approaches.

  • Takeaways & Limitations

    The framework provides unified single-pass reconstruction of cameras, scene geometry, and multiple humans from multi-person, multi-view video without external modules or preprocessing.

  • Takeaways & Limitations

    Performance may degrade when the head region is severely occluded or not visible because CHROMM heavily relies on head tokens.

Abstract

from arXiv · show

Recent advances in 3D foundation models have led to growing interest in reconstructing humans and their surrounding environments. However, most existing approaches focus on monocular inputs, and extending them to multi-view settings requires additional overhead modules or preprocessed data. To this end, we present CHROMM, a unified framework that jointly estimates cameras, scene point clouds, and human meshes from multi-person multi-view videos without relying on external modules or preprocessing. We integrate strong geometric and human priors from Pi3X and Multi-HMR into a single trainable neural network architecture, and introduce a scale adjustment module to solve the scale discrepancy between humans and the scene. We also introduce a multi-view fusion strategy to aggregate per-view estimates into a single representation at test-time. Finally, we propose a geometry-based multi-person association method, which is more robust than appearance-based approaches. Experiments on EMDB, RICH, EgoHumans, and EgoExo4D show that CHROMM achieves competitive performance in global human motion and multi-view pose estimation while running over 8x faster than prior optimization-based multi-view approaches. Project page: https://nstar1125.github.io/chromm.

1 Introduction

CHROMM addresses multi-person, multi-view human-scene reconstruction with a unified single-pass framework that avoids external modules and preprocessing. It combines geometric and human priors with scale adjustment, multi-view fusion, and geometry-based person association, achieving competitive accuracy and over 8× faster runtime than optimization-based approaches.

  • CHROMM jointly reconstructs cameras, scenes, and multiple humans from multi-person, multi-view videos in a single pass without external modules or preprocessing.
  • The framework integrates geometric and human priors from Pi3X and Multi-HMR within a unified architecture.
  • A head–pelvis-based scale adjustment aligns approximate metric-scale scene geometry with metric-scale SMPL meshes.The method compares image and projected-SMPL head–pelvis lengths to adjust the predicted scene scale.
  • A test-time optimization-free fusion strategy aggregates per-view human estimates into a coherent global representation by separately handling view-invariant and view-dependent attributes.Canonical-space pose and body shape are fused directly, while rotation and translation are transformed into a shared world coordinate system.
  • Geometry-based association uses 3D positions and human poses to establish cross-view person correspondences.This avoids relying on preprocessed identities and addresses the unreliability of appearance-based matching for visually similar people.
  • 8× faster runtime is reported alongside competitive performance on global human motion and multi-view human pose estimation.Experiments cover EMDB, RICH, EgoHumans, and EgoExo4D.

2 Related Work

Related work spans optimization-based 3D reconstruction, feed-forward scene and human recovery, world-coordinate motion estimation, and joint human-scene reconstruction. Existing multi-view systems still commonly involve restricted settings, iterative optimization, external modules, or assumed identity correspondences.

  • Structure-from-Motion and Multi-View Stereo estimate cameras and scene geometry through feature matching and bundle adjustment.These classical pipelines are highly accurate but computationally expensive and struggle in dynamic scenes.
  • Data-driven scene reconstruction replaces iterative optimization with feed-forward prediction, progressing from pairwise point maps to direct camera prediction and permutation-equivariant architectures.DUSt3R, MASt3R, VGGT, and Pi3 represent successive approaches described in the paper.
  • Human mesh recovery predicts parametric body-model parameters, with Multi-HMR enabling unified single-shot whole-body reconstruction without bounding-box detection.The related work traces progress from transformer-based HMR to multi-person reconstruction.
  • World-coordinate human-motion methods decouple camera and motion, use motion and contact priors, or introduce gravity-aligned coordinates for globally grounded trajectories.The passage describes SLAHMR, WHAM, GVHMR, and TRAM as examples.
  • Recent human-scene methods combine 3D foundation models with human pose estimation, but UniSH and Human3R operate on monocular inputs.Other approaches cited include JOSH and JOSH3R.
  • Multi-view systems such as HSfM and HAMSt3R address human-scene reconstruction, yet prior approaches may use single frames, iterative optimization, external modules, or assumed cross-view identities.These constraints increase computational cost and system complexity according to the related-work discussion.

3 Method

CHROMM is a unified single-pass network that reconstructs cameras, scene point clouds, and multiple human meshes from multi-person multi-view video. It combines scene and human priors with scale adjustment, multi-view fusion, and geometry-based identity association.

  • CHROMM jointly reconstructs camera parameters, scene point clouds, and human meshes from multi-person multi-view video in a single pass.
  • The architecture combines Pi3X for scene geometry with Multi-HMR for detailed human representation through separate scene-wise and human-wise feature streams.The model avoids early feature fusion because it negatively affects geometric reconstruction performance.
  • Scene Reconstruction: Scene reconstruction predicts per-frame cameras and local 3D point maps, while a metric decoder estimates a global scene scale applied to points and camera translations.Human regions are masked out before forming the static scene point cloud.
  • Human Reconstruction: Human reconstruction detects head patches, fuses human and scene tokens, and uses SMPL decoders to regress human parameters and scene-relative translation.Head-based translation is estimated from a predicted 2D head location and depth recovered by unprojecting with camera intrinsics.
  • Scale Adjustment: The scale-adjustment module uses the image-to-projected-SMPL head–pelvis length ratio to align the scene and metric-scale human meshes.Pelvis detection uses a coarse-to-fine strategy, and the pelvis provides a relatively pose-invariant reference.
  • Multi-View Fusion and Association: Multi-view fusion separates view-invariant pose and shape from view-dependent rotation and translation, then combines estimates in a shared world coordinate system.Cross-view identities are matched using costs based on global 3D position and canonical-space pose cues.

4 Experiments

CHROMM is evaluated on global human motion and multi-view human pose estimation across several datasets, including monocular and multi-view settings. Ablations examine scale adjustment, fusion, and multi-person association, while runtime comparisons assess efficiency.

  • Experimental Setup: CHROMM is evaluated on global human motion estimation and multi-view human pose estimation using datasets with monocular, fixed multi-view, and egocentric video settings.EMDB-2 and RICH support conventional evaluation, while EgoHumans and EgoExo4D assess multi-view performance under a single-frame protocol.
  • Multi-View Human Pose Estimation: Competitive performance is achieved on EgoHumans and EgoExo4D without ground-truth re-identification, iterative optimization, or external modules.The comparison follows a single-timestep protocol used by prior multi-view methods.
  • Runtime Comparison: Over 8× speedup is achieved against HAMSt3R for inference on a single timestep with three people and four views on one NVIDIA V100 GPU.CHROMM avoids additional preprocessing stages and iterative optimization procedures.
  • Ablation Studies: Scale adjustment prevents the performance drop caused by the scale gap between reconstructed scenes and human meshes across diverse scene scales.The ablation is conducted on EMDB-2, which contains a wide range of scene scales.
  • Ablation Studies: Triangulation improves multi-view fusion over averaging translation predictions, while explicit averaging of view-invariant parameters achieves the highest performance.The results support separating view-invariant attributes from view-dependent attributes during fusion.
  • Ablation Studies: Combining pose and positional cues further boosts multi-person re-identification accuracy, especially when positional information alone is insufficient.Pose-only matching has low accuracy and high recall with low precision, indicating frequent over-matching for people with similar poses.

5 Conclusion

CHROMM jointly reconstructs cameras, scene geometry, and multiple humans from multi-person multi-view video in a single pass without external modules or preprocessing. It combines geometric and human priors with scale adjustment, fusion, and geometry-based association, achieving competitive accuracy and over 8× faster motion and pose estimation.

  • 5 Conclusion: CHROMM jointly reconstructs cameras, scene geometry, and multiple humans from multi-person multi-view video in a single pass without external modules or preprocessing.The framework integrates geometric and motion priors from Pi3X and Multi-HMR.
  • 5 Conclusion: The method addresses human-scene scale discrepancy with a head–pelvis ratio, aggregates per-view estimates through fusion, and associates humans using pose and spatial cues.These components are presented as core parts of the unified framework.
  • 5 Conclusion: Over 8× faster performance is reported for global human motion estimation and multi-view pose estimation while maintaining competitive performance.The comparison is against prior multi-view approaches.
  • Limitations and Future Works: Performance may degrade when the head region is severely occluded or not visible because the method heavily relies on head tokens.The authors identify this as a limitation shared with Multi-HMR and Human3R.

Appendix

The appendix provides implementation details and additional experiments supporting CHROMM's claims. It covers training, losses, matching, ablations, scene reconstruction, inference time, and qualitative results.

  • Implementation Details: The appendix describes CHROMM's training setup, loss design, and matching algorithm.
  • Additional Experiments: Additional experiments examine SMPL translation strategies, pelvis detection robustness, two-stage training, and scene reconstruction performance.
  • Additional Results: Supplementary results include inference time on multi-view video and additional human-scene reconstruction results.

A Implementation Details

CHROMM is trained in two stages with frozen foundation-model components, supervised losses, and geometric consistency constraints. Its implementation also includes coarse-to-fine pelvis detection and explicit multi-person association across views.

  • Training Details: Stage 1 freezes the Pi3 encoder and decoder and Multi-HMR encoder while training SMPL decoders and fusion, mask, and pelvis-detection MLPs.Scale adjustment is disabled during the first 10 of 20 epochs, then enabled for the remaining training.
  • Training Details: Stage 2 freezes all modules except the pelvis-detection MLPs and trains them for 10 epochs using single images with head keypoint locations.Training uses AdamW, OneCycleLR, up to 10 people per frame, and four NVIDIA A100 GPUs.
  • Loss Design: Stage 1 combines geometric, parametric, and detection losses supervising 3D vertices, 3D joints, 2D projections, SMPL parameters, masks, heads, and pelvises.The listed losses include vertex and joint reconstruction, parameter supervision, and BCE detection losses.
  • Stage 2 Loss: Stage 2 adds pelvis, reprojection, and Chamfer losses to enforce geometric consistency between predicted depth and visible SMPL vertices.The Chamfer loss is computed without applying the optimal scale factor to predicted depth maps.
  • Association Algorithm: The association algorithm tracks humans within each view using human-token distances and 3D-joint displacement, then associates tracklets across views.Global human identities are assigned after cross-view association using the described matching procedure.

B Additional Experiments

Additional experiments evaluate CHROMM’s translation estimation, pelvis detection, scene reconstruction, training strategy, and runtime. The results support coarse-to-fine pelvis detection, two-stage training, and efficient inference.

  • Translation estimation: CHROMM compares direct translation, direct depth, and depth-residual prediction using the estimated depth map and camera intrinsics.The depth-residual formulation unprojects the predicted residual and leverages Pi3 geometric priors.
  • Pelvis detection comparison: Coarse-to-fine pelvis detection achieves better accuracy than directly regressing pelvis location from the head token.Pelvis error is normalized by the maximum bounding-box width or height.
  • Pelvis detection robustness to occlusions: The model estimates pelvis locations even when surrounding objects occlude the pelvis, and averages detections across views for scale adjustment.Accurate pelvis localization supports reliable scale adjustment.
  • Early fusion impact on scene reconstruction: Scene-depth experiments compare early and late fusion using Abs Rel and thresholded accuracy δ < 1.25 on PROX.Early fusion feeds human tokens with patch tokens into the Pi3X decoder, whereas late fusion leaves scene reconstruction separate.
  • Effect of two stage training: Two-stage training improves global human pose estimation by enabling more accurate scale adjustment.The comparison uses CHROMM trained on BEDLAM alone versus further training on in-the-wild datasets with pelvis prediction.
  • Inference time breakdown: Runtime analysis measures module costs on 100 frames from a three-person, four-view EgoHumans scene across five repetitions on one NVIDIA A100.The reported average includes feature fusion and image encoding into Multi-HMR features.

C Additional Results

Additional qualitative experiments show CHROMM reconstructing humans and scenes in multi-view, monocular, and single-image settings. The paper identifies this as its first capability spanning both multi-view and monocular setups.

  • Multi-view results: CHROMM reconstructs humans and scenes in diverse multi-view settings.The qualitative results are presented in Fig. 7.
  • Monocular and single-image results: CHROMM also handles monocular settings and single images.These qualitative results are shown in Figs. 8 and 9.
  • Scope: The paper describes CHROMM as the first model capable of reconstructing both humans and scenes under multi-view and monocular setups.This claim covers the qualitative settings presented in the additional results.

D Limitations

CHROMM has failure cases involving complex poses, close human interactions, and severely limited head visibility. Extreme zoom-in conditions also challenge reconstruction.

  • Failure cases: CHROMM can fail on complex poses and close human interactions.These failure cases are illustrated in Fig. 10.
  • Head visibility: Performance may degrade when the head is completely invisible or occupies most of an extreme zoom-in image.The limitation is attributed to reliance on head tokens.
Loading 2603.12789v2…