PaperlayerView exact revision on arXiv

Source-linked AI summary

Feed-Forward Multi-view Multi-person Reconstruction with Contrastive Human-Aware 3D Representation

Yuanwang Yang, Buzhen Huang, Zongxuan Ren, Jing Huang, Kun Li

arXiv:2609.00745v1cs.CV

TL;DR

Robust multi-person reconstruction remains difficult when cameras are uncalibrated and occlusions make bottom-up 2D matching unreliable. The paper introduces a unified human-aware 3D space with spatial contrastive learning for joint calibration, association, and reconstruction, achieving strong reconstruction and camera-estimation results across challenging datasets. Its initialization depends on VGGT, whose errors under large viewpoint gaps, low texture, and difficult depth discontinuities can affect downstream quality.

  • Problem

    Existing bottom-up multi-view reconstruction relies on accurate calibration and explicit cross-view matching, which are challenged by severe occlusions and ambiguous observations.

  • Method

    The method lifts multi-view observations into a unified human-aware 3D space and applies spatial contrastive learning to align same-instance features while separating different instances.

  • Results

    The method achieves strong global reconstruction accuracy, competitive or superior local pose accuracy, and consistently lower camera pose errors across EgoHumans and OcMotion.

  • Takeaways & Limitations

    A shared 3D representation supports robust cross-view association, instance reasoning, and joint camera-scene inference under complex occlusions.

  • Takeaways & Limitations

    The framework relies on VGGT initialization, whose imperfect predictions may limit reconstruction under large viewpoint gaps, low-texture regions, or challenging depth discontinuities.

Abstract

from arXiv · show

Multi-view human reconstruction has been extensively studied under simplified settings, yet robust and efficient multi-person reconstruction in unconstrained environments remains challenging. Existing bottom-up methods often rely on accurate camera calibration and explicit cross-view matching, and therefore struggle with severe occlusions and ambiguities. We propose a new top-down paradigm that maintains a unified, instance-centric human-aware 3D space, enabling simultaneous camera calibration, cross-view association, and human reconstruction via cross-modal contrastive learning. Observations from multiple views are lifted and fused into this shared 3D space, where geometric structure, visual appearance, and human-centric semantic cues are jointly encoded at the instance level. We further introduce a spatial contrastive learning strategy that aligns 3D features corresponding to the same human instance across different views and modalities while separating different instances. This enables correspondence reasoning, semantic aggregation, and instance discrimination to be performed natively in 3D, improving cross-view consistency and robustness under severe occlusions. Finally, structured human body models are recovered in a feed-forward manner by regressing SMPL parameters from instance-level 3D human tokens. Extensive experiments demonstrate robust, accurate, and efficient multi-view human reconstruction in challenging real-world scenarios.

1 Introduction

Existing multi-view human reconstruction is fragile in complex real-world settings because bottom-up pipelines depend on accurate calibration, cross-view matching, and reliable 2D observations. The paper instead proposes a unified human-aware 3D space for simultaneous calibration, association, and reconstruction.

  • Traditional systems become fragile under imperfect setups, heavy occlusions, and large-scale scenes because they rely heavily on accurate camera calibration.
  • Bottom-up methods extract 2D cues, establish cross-view correspondences, and fuse observations into a 3D representation.
  • Camera inaccuracies can propagate into global geometric inconsistencies, while occlusions can corrupt correspondence estimation and feature fusion.
  • The paper advocates reasoning directly within an explicit shared 3D representation rather than fusing fragmented, view-dependent 2D predictions.
  • The proposed top-down paradigm jointly encodes geometry, appearance, and human-centric semantics, using spatial contrastive learning to improve consistency and instance discriminability.

2 Related Work

Related work addresses unconstrained cameras, multi-view human pose, and feed-forward reconstruction, but existing approaches trade robustness against computation or remain dependent on view-based reasoning. This paper instead uses a unified implicit 3D embedding space for joint human and scene modeling.

  • Unconstrained deployments involve unknown, unsynchronized, or distorted cameras and sparse wide-baseline viewpoints, coupling camera and human inference under noisy cues.
  • Bottom-up pose pipelines are sensitive to 2D detection noise and often fail under heavy occlusion or multi-person interaction.
  • Voxel-based methods improve robustness by aggregating multi-view features in 3D, but require substantial computation and memory.
  • Feed-forward geometry predictors avoid expensive test-time optimization by directly predicting cameras and dense geometric outputs in a single forward pass.
  • The proposed framework constructs a unified implicit 3D embedding space that jointly models pose, appearance, and spatial relationships for robust multi-human SMPL reconstruction.

3 Method

The framework explicitly maintains and optimizes a unified 3D feature space as an intermediate representation, rather than reasoning directly from per-view image features. This shared space jointly supports camera calibration, cross-view association, and human reconstruction in a feed-forward pipeline.

  • The method encodes geometric structure, semantic information, and human identity within a shared 3D space.
  • Camera calibration, cross-view association, and human reconstruction are addressed jointly in the feed-forward pipeline.

3.1 Human-aware Unified 3D Feature Space Initialization

The method initializes a compact, view-agnostic 3D feature space from uncalibrated images using geometry-aware predictions, confidence-weighted aggregation, and continuous 3D points. Human semantics, Gaussian attributes, and rendering-based supervision are then decoded from this shared representation.

  • 3.1.1 Construction of 3D Feature Space: Each 3D token stores a 3D location and learned feature embedding, forming the basic view-consistent representation.
  • 3.1.1 Construction of 3D Feature Space: Image features and confidence weights are lifted into 3D using predicted depth and camera parameters from a pretrained geometry-aware model.
  • 3.1.1 Construction of 3D Feature Space: Confidence-weighted soft aggregation combines neighboring points across views into a compact representation while reducing noisy and redundant observations.
  • 3.1.1 Construction of 3D Feature Space: Continuous 3D-point aggregation avoids fixed voxel discretization and produces view-agnostic tokens anchored to spatial structure.
  • 3.1.2 Human-aware 3D Feature Decoding: A shared latent feature decodes Gaussian density, scale, rotation, spherical-harmonic appearance, and human-related logits.
  • 3.1.2 Human-aware 3D Feature Decoding: Human confidence is defined at the 3D level, enabling consistent cross-view reasoning while jointly encoding geometry, appearance, and semantics.
  • 3.1.3 Initial 3D Space Training: Gaussian rendering and human-mask supervision train appearance and human confidence, with regularization stabilizing scales, opacity, and local confidence smoothness.

3.2 3D Feature Enhancement with Spatial Contrastive Learning

The method enhances a shared 3D feature space by selectively sampling multi-view semantic cues and applying spatial contrastive supervision for instance-aware, view-consistent representations.

  • 3.2.1 Human Semantics Sampling: High-confidence human tokens are selectively enhanced with pose, appearance, and geometry features sampled from reprojected semantic fields.Tokens outside the selected human region retain their original geometric features.
  • 3.2.1 Human Semantics Sampling: Visibility-aware reprojection accepts samples inside image boundaries whose projected depth agrees with the predicted depth map.The camera projection uses estimated camera parameters, and valid samples are aggregated across views with visibility-based confidence weights.
  • 3.2.2 Cross-modal Feature Fusion: A residual MLP fusion preserves the original 3D representation while injecting semantic information independently and in parallel for each token.This design supports scalable refinement while retaining geometric stability.
  • 3.2.3 3D Instance Embedding and Spatial Contrastive Learning: Spatial contrastive learning combines intra-view, cross-view, and 3D-space supervision to separate different people and align same-person embeddings.The three domains respectively improve local separability, cross-view identity consistency, and global 3D representation stability.
  • 3.2.3 3D Instance Embedding and Spatial Contrastive Learning: Semantic contrastive supervision aligns features for the same identity across valid views, producing more view-invariant and spatially coherent representations.The resulting stable semantic features benefit subsequent SMPL regression.

3.3 SMPL Parameter Regression from Human Tokens

The framework pools confidence-weighted person tokens from refined 3D features and directly regresses SMPL parameters for each detected instance without iterative fitting.

  • 3.3.1 Human Token Pooling: Refined 3D tokens, locations, human confidences, and pseudo-instance labels provide the inputs for person-specific regression.Background tokens use label 0, while positive labels identify person instances.
  • 3.3.1 Human Token Pooling: Each person descriptor concatenates token features with coordinates and is pooled by confidence-weighted averaging over tokens assigned to that person.The weighting suppresses noisy or floating low-confidence points.
  • 3.3.1 Human Token Pooling: The accumulated token weight acts as a validity indicator for masking persons that are not reliably observed.This supports instance validity handling during training.
  • 3.3.2 Feed-forward SMPL parameter regression: A small MLP regresses pose, shape, translation, and optionally scale from each pooled person token.The outputs include 6D pose parameters, shape coefficients, global translation, and a positive scale for coordinate-system ambiguity.
  • 3.3.2 Feed-forward SMPL parameter regression: The regression outputs define an SMPL mesh per person for rendering, silhouette supervision, and instance-level evaluation without iterative optimization.Training combines 2D reprojection, 3D joint alignment, silhouette consistency, and retained photometric supervision.

4 Experiments

Experiments evaluate reconstruction and camera estimation on real-world multi-view benchmarks featuring multi-person interaction, viewpoint variation, and severe occlusion.

  • 4.1 Datasets: EgoHumans contains synchronized multi-camera sequences with two to four people across indoor and outdoor scenes with frequent occlusions and irregular camera placement.Its varied layouts and viewpoints test unconstrained multi-person reconstruction.
  • 4.1 Datasets: OcMotion evaluates single-person motion under severe object-induced occlusions and incomplete, view-dependent observations.The dataset emphasizes robustness to dynamic occlusion and missing visual evidence.
  • 4.2 Metrics: Reconstruction is evaluated with CA-MPJPE, GA-MPJPE, and PA-MPJPE, which measure camera-aligned, group-layout, and per-person pose accuracy.These metrics distinguish global configuration accuracy from articulated pose accuracy.
  • 4.2 Metrics: Camera quality is measured with AE, s-TE, s-CCA@10, and AUC@10, where lower errors and higher consistency scores indicate better estimation.AE and s-TE assess rotation and translation, while the consistency metrics use a fixed angular threshold.
  • 4.2 Metrics: Table 1 compares multi-view human reconstruction accuracy on OcMotion.The table targets a severe-occlusion benchmark setting.

4.3 Results

The method achieves strong reconstruction and camera-estimation performance across challenging datasets and occlusion settings. Qualitative results further show coherent, consistent multi-view reconstructions.

  • The method achieves strong global reconstruction accuracy across OcMotion and EgoHumans, including severe occlusion settings.OcMotion evaluates severe object-induced occlusions, while EgoHumans distinguishes subjects occluded in more than 50% of views.
  • The method consistently achieves substantially lower camera pose errors across all metrics on EgoHumans and OcMotion.High s-CCA@10 and AUC@10 further indicate strong cross-view consistency.
  • Qualitative reprojection comparisons show accurate and consistent reprojections across views, unlike baseline-specific misalignments or inconsistent reconstructions.HSfM often misaligns in the image plane, while Multi-HMR lacks explicit 3D reasoning across viewpoints.
  • Recovered scene geometry and SMPL bodies are visually coherent and physically plausible across multiple views.Figure 5 visualizes both reconstructed scenes and recovered human bodies on EgoHumans and OcMotion.

4.4 Runtime and Memory Analysis

The feed-forward framework avoids iterative test-time optimization, with runtime scaling sub-linearly as input views increase. It also offers lower runtime than HSfM while maintaining comparable or better memory efficiency.

  • The framework’s computational cost is primarily determined by input frames rather than iterative optimization or test-time refinement.All components are implemented in a feed-forward regression manner.
  • Runtime increases with more input views but grows sub-linearly rather than strictly proportionally.The scene reconstruction stage scales with views, while segmentation gradually stabilizes as spatial density converges.
  • The method achieves substantially lower runtime than HSfM while maintaining comparable or higher memory efficiency as input views increase.This comparison indicates a favorable accuracy–efficiency trade-off for multi-view reconstruction.

4.5 Cross-view Association Evaluation

The evaluation measures cross-view identity consistency using AIDP and finds that the proposed method outperforms both HSfM and a pose-based baseline. Spatial contrastive learning provides the central mechanism for robust identity alignment.

  • AIDP evaluates precision of subject association by averaging pairwise identity-matching accuracy across view pairs.The metric directly measures consistency of instance-level correspondence in multi-view settings.
  • The method achieves the highest AIDP score, outperforming both HSfM and the VGGT-based Pose-Aware ReID baseline.The baseline uses confidence-weighted pose descriptors and nearest-neighbor matching.
  • Spatial contrastive learning explicitly enforces instance-level consistency in the unified 3D feature space, improving cross-view identity alignment.Geometry- or pose-based heuristics provide useful cues but remain limited under challenging multi-view conditions.

4.6 Ablation Study

Ablations show that unified 3D representation, reprojection enhancement, and spatial contrastive learning each contribute to reconstruction quality and robustness. Additional analyses show robustness to geometry initialization and improved human-awareness over 2D mask fusion.

  • Ablation Study: Removing the unified 3D feature space significantly degrades reconstruction accuracy under occlusion and viewpoint variation.Per-view 2D features make cross-view association of the same individual more difficult.
  • Ablation Study: Disabling reprojection enhancement produces inferior reconstruction quality because pose-related image features are no longer injected into the 3D representation.The resulting optimization is more difficult and convergence is less stable.
  • Ablation Study: Removing spatial contrastive learning contaminates 3D features through segmentation noise and view-dependent errors, reducing robustness under occlusion and clutter.The ablation aggregates features from 2D instance segmentation without enforcing 3D instance consistency.
  • Geometry Initialization: Performance remains highly consistent when replacing VGGT initialization with AnySplat, with only marginal differences across evaluation metrics.The result indicates robustness to the choice of geometry initialization.
  • 3D Human Awareness: 2D segmentation errors lifted into 3D create ray-like background leakage and inconsistent identities, whereas shared 3D learning yields spatially consistent human awareness.The learned confidence produces compact, coherent human regions that support later human-guided modules.
  • 3D Human Awareness: Learned 3D human confidence improves precision and F1 score over 2D mask fusion while maintaining competitive recall and achieving higher PR-AUC.2D mask fusion has high recall but extremely low precision because of background leakage.

5 Conclusion

The framework provides feed-forward multi-view multi-person reconstruction through a shared 3D representation, with spatial contrastive learning supporting separation and cross-view association under occlusion. Its current scope is static reconstruction and depends on geometry-aware initialization, while future work targets dynamic settings and improved initialization robustness.

  • Conclusion: The feed-forward framework jointly encodes geometry, semantics, and human identity in a shared 3D representation for multi-view multi-person reconstruction.It initializes the 3D space with a geometry-aware prior and 3D Gaussian representation, then recovers structured human models from instance-aware 3D tokens.
  • Conclusion: Spatial contrastive learning improves 3D feature discriminability and consistency, enabling multi-person separation and cross-view association despite occlusion and partial visibility.
  • Conclusion: Extensive experiments demonstrate competitive or superior human reconstruction accuracy while significantly improving camera estimation quality.
  • Limitations and future work: The current formulation focuses on static reconstruction and does not explicitly model temporal evolution, motivating future work on temporal token updates for 4D reconstruction.Suggested mechanisms include token propagation, merging, and splitting within the same 3D Gaussian representation.
  • Limitations and future work: The framework relies on VGGT for geometry-aware initialization, whose limitations in large viewpoint gaps, low-texture regions, and depth discontinuities can affect downstream reconstruction quality.The representation can nevertheless be integrated with alternative or improved geometry foundation models.
Loading 2609.00745v1…