PaperlayerView exact revision on arXiv

Source-linked AI summary

4D Human-Scene Reconstruction from Low-Overlap Captures

Minhyuk Hwang, Sangmin Kim, Seunguk Do, Daneul Kim, Jaesik Park

arXiv:2607.09125v1cs.CV

TL;DR

Sparse, low-overlap cameras make high-fidelity 4D reconstruction difficult, especially with multiple interacting people and occlusions. StudioRecon decouples backgrounds and humans, using diffusion-synthesized views and geometry-driven human initialization, then harmonizes the composition with motion-adaptive enhancement; it achieves state-of-the-art results across diverse real-world datasets.

  • Problem

    In-the-wild studio capture requires reconstructing multiple interacting humans from sparse, low-overlap, uncalibrated cameras with frequent occlusions.

  • Method

    StudioRecon decouples backgrounds and humans, using diffusion-synthesized novel views for background supervision, geometry-driven multi-view body estimation, and motion-adaptive diffusion enhancement.

  • Results

    StudioRecon achieves state-of-the-art reconstruction results across diverse real-world datasets and supports novel camera trajectories and actor replacement.

  • Takeaways & Limitations

    The decoupled approach brings high-fidelity 4D human-scene capture from sparse, low-overlap cameras closer to practical deployment.

  • Takeaways & Limitations

    Dynamic objects such as basketballs or props are not handled and may disappear from rendered outputs.

Abstract

from arXiv · show

Existing volumetric capture of dynamic human performance achieves high fidelity with dense camera arrays. However, in real-world scenarios, only a handful of low-overlap cameras are available, which degrades the output quality and leaves large areas unobserved. Recent 4D reconstruction methods have focused on low-overlap settings, yet they still produce noticeable artifacts in under-observed regions. Video diffusion models have emerged as another option, but they show geometrically inconsistent results for humans. To address these limitations, we propose StudioRecon, a pipeline that reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and humans. We densify background supervision by synthesizing hundreds of camera-controlled novel views with a video diffusion model. We also robustly initialize deformable Gaussian humans with cross-view identity association and triangulated multi-view keypoint fitting. Finally, our recursive enhancement module with motion-adaptive consistency injection harmonizes the composed output, thereby further avoiding remaining artifacts. We achieve state-of-the-art novel view synthesis across four real-world datasets and demonstrate applications such as novel trajectory rendering and human replacement.

1 Introduction

StudioRecon targets high-fidelity 4D human-scene reconstruction from sparse, uncalibrated, low-overlap cameras in challenging in-the-wild environments. It decouples background and human reconstruction, using complementary priors, geometry-driven human estimation, and diffusion-based enhancement to reduce artifacts and improve consistency.

  • Motivation: StudioRecon addresses in-the-wild capture settings with few uncalibrated, low-overlap cameras, multiple interacting people, and frequent occlusions.Professional volumetric systems instead require dozens to hundreds of cameras in controlled environments.
  • Decoupled reconstruction: StudioRecon decouples backgrounds and humans because their reconstruction benefits from different priors under low-overlap constraints.The approach avoids entanglement that can cause artifacts in joint reconstruction methods.
  • Background reconstruction: For backgrounds, a camera-controlled video diffusion model synthesizes hundreds of novel views from sparse inputs to provide dense supervision for background Gaussians.This addresses insufficient view coverage and the lack of photometric supervision in never-observed regions.
  • Human reconstruction: For humans, geometry-driven multi-view SMPL estimation associates identities across views and triangulates 2D keypoints to obtain 3D body geometry.The method combines spatial and pose affinity for cross-view human correspondence.
  • Recursive enhancement: A single-step diffusion model removes static Gaussian artifacts and improves background-human concordance, while motion-adaptive consistency injection addresses temporal flickering.Separate rendering can otherwise produce unnatural compositions and artifacts in under-observed regions.

2 Related Work

Related work progresses from dense multi-view and RGB-D capture toward neural and Gaussian-based 4D reconstruction, while human-scene methods jointly model people and environments. Diffusion methods provide video priors but struggle with temporal consistency and incomplete coverage, motivating StudioRecon’s dense-view supervision and efficient consistency enhancement.

  • 4D Scene Reconstruction: Traditional 4D reconstruction uses dense multi-view setups or RGB-D sensors, including free-viewpoint video systems and Fusion4D.These approaches model spatio-temporal scene movements from substantially denser observations.
  • 4D Scene Reconstruction: Neural rendering extended 4D reconstruction through planar factorization, multi-view feature aggregation, and 4D neural voxels, while 3DGS offers a faster alternative.The cited progression follows advances from NeRF-based representations toward Gaussian splatting.
  • Human-Scene Reconstruction: Human-scene reconstruction jointly recovers humans and static scenes from video for applications including pose editing and novel view synthesis.Prior work explored disentangled monocular representations and online methods combining camera tracking, pose estimation, and reconstruction.
  • Diffusion-Based Enhancement: Diffusion video enhancement faces temporal coherence problems from stochastic denoising, causing flicker and texture drift despite warping and temporal-module remedies.Training-free methods also adapt image diffusion through latent warping.
  • Diffusion-Based Enhancement: StudioRecon addresses missing view coverage and costly, flicker-prone enhancement by synthesizing dense background views with video diffusion and applying single-step motion-adaptive consistency injection.The proposed design targets artifact-free free-viewpoint rendering while avoiding multi-step diffusion costs.

3 Method

StudioRecon reconstructs dynamic multi-human scenes from sparse, low-overlap, uncalibrated cameras by separating background and human reconstruction and combining view synthesis with geometric human priors. Its four-stage pipeline densifies background supervision, estimates cross-view human geometry, optimizes separate Gaussian representations, and harmonizes the composite output.

  • Problem setting: The method targets in-the-wild studio capture with sparse, low-overlap cameras, multiple interacting subjects, occlusions, and no assumption of pre-calibrated cameras.Sparse viewpoints leave large scene regions unobserved, while limited observations and self-occlusions make dynamic human reconstruction ill-posed.
  • Pipeline overview: The pipeline has four stages: sparse-to-dense view synthesis, multi-view human pose estimation, decoupled Gaussian reconstruction, and recursive enhancement.These stages respectively address insufficient viewpoint coverage, human geometric ambiguity, source-specific supervision, and remaining compositing artifacts.
  • Sparse-to-dense view synthesis: Video diffusion synthesizes novel views along SLERP-and-linear camera trajectories, while human masks exclude synthesized human regions during background Gaussian optimization.The synthesized views use first-frame images, depth maps, and target poses as inputs.
  • Multi-view human pose estimation: Cross-view identities are associated using calibrated-camera geometry, spatial proximity, and pose similarity before triangulating 2D keypoints for robust 3D SMPL fitting.The affinity weights spatial similarity at 0.9 and pose similarity at 0.1; triangulation uses Huber loss to handle occlusions and detection noise.
  • Decoupled Gaussian reconstruction: Backgrounds and dynamic humans are reconstructed separately because backgrounds benefit from dense views, whereas humans benefit from temporal supervision and SMPL constraints under partial observation.The resulting scene is composed from static background and dynamic human Gaussians, but may retain floaters, blur, pose artifacts, and texture or lighting variation.

4 Experiment

StudioRecon is evaluated on sparse 360° and 180° camera configurations, consistently outperforming baselines while improving background quality and human reconstruction. Ablations, robustness tests, and applications further demonstrate the benefits and practical scope of its components.

  • Experimental Setup: Experiments use four datasets with 4 training cameras and 4 intermediate-angle evaluation cameras across 360° and 180° coverage, comparing against three sparse-view baselines.Each sequence contains 121 frames with 1–3 people performing dynamic activities.
  • Quantitative Comparison: StudioRecon consistently outperforms all baselines across metrics, with particularly large LPIPS improvements when widely spaced cameras leave scene regions poorly captured.Dense view synthesis addresses insufficient viewpoint coverage that limits the baselines.
  • Qualitative Comparison: Qualitative results show clean backgrounds and robust human reconstructions, whereas baselines exhibit background artifacts and unstable human geometry.StudioRecon combines dense synthesized views for scene modeling with explicit SMPL-guided human representation.
  • Ablation Studies: The hybrid cross-view association achieves 97.8% accuracy with zero false positives, outperforming spatial-only association at 93.3% and pose-only association at 81.4%.Spatial proximity fails with unreliable depth, while pose similarity fails when humans perform similar actions.
  • Ablation Studies: Dense view synthesis provides +2.4 PSNR and a 36% LPIPS reduction, while single-step diffusion enhancement adds a 27% LPIPS reduction despite a minor PSNR/SSIM drop.The enhancement’s pixel misalignment reflects generative details that differ from ground truth.
  • Ablation Studies: Motion-adaptive consistency injection reduces warp error by 23%, improving temporal coherence and sharpness while slightly increasing LPIPS.It blends warped previous outputs into the current input to reduce frame-to-frame flickering, especially in static regions.
  • Applications and Limitations: The representation supports arbitrary camera trajectories and human replacement by independently rendering novel paths and compositing a new identity with transferred SMPL pose.Demonstrated trajectories include dolly zoom and oscillating motion; limitations remain for fine details, dynamic objects, and time-varying shadows.

5 Conclusion · Appendix Overview

StudioRecon reconstructs high-fidelity 4D human scenes by decoupling background and human priors, enabling novel camera trajectories and actor replacement. The appendix documents preprocessing, pose estimation, Gaussian reconstruction, and diffusion-based rendering enhancement.

  • 5 Conclusion: StudioRecon is a pipeline for high-fidelity 4D human-scene reconstruction in in-the-wild studio capture.
  • 5 Conclusion: Its key insight is that backgrounds and humans benefit from different priors.
  • 5 Conclusion: Diffusion models synthesize dense supervision for backgrounds, while parametric body models constrain human geometry from limited views.
  • 5 Conclusion: The decoupled approach enables high-quality reconstruction and applications including novel camera trajectories and actor replacement.
  • Appendix Overview: The appendix provides additional implementation details and results for StudioRecon.
  • Appendix Overview: Appendix A describes preprocessing with segmentation, depth estimation, and view synthesis.
  • Appendix Overview: Appendix B details human pose estimation with cross-view association and SMPL fitting.
  • Appendix Overview: Appendices C and D provide optimization configurations for background and human Gaussian reconstruction, while Appendix E explains diffusion-based rendering enhancement.

A Preprocessing · B Human Pose Estimation · C Background Reconstruction

StudioRecon preprocesses four-camera inputs with actor segmentation, geometric estimation, and 481 synthesized views, then estimates consistent human poses through cross-view association and robust triangulation. Background Gaussians are optimized over synthesized views with weighted supervision and iterative refinement.

  • A Preprocessing: The pipeline uses N=4 input cameras, SAM3 actor segmentation, Pi3 camera and depth estimation, and GEN3C synthesis of L=481 novel views.Novel views use 1280×704 resolution, guidance scale 1.0, and spherical-linear camera-pose interpolation.
  • B Human Pose Estimation: CoMotion provides per-view actor detection, tracking, SMPL pose and shape estimates, and 2D keypoints.
  • B Human Pose Estimation: Cross-view identities are matched with the Hungarian algorithm using spatial and pose affinities at reference timestep t_0=0.Spatial and pose affinity weights are w_p=0.9 and w_θ=0.1, with a 0.3m valid-match threshold.
  • B Human Pose Estimation: 3D joints and vertices are obtained through pelvis unprojection and robust DLT triangulation with Huber loss.SMPL fitting runs 30 iterations with β regularization 10.0, while bone-length scale alignment uses L-BFGS for 100 iterations.
  • C Background Reconstruction: Background Gaussians are optimized for 7,000 iterations over the synthesized views using spherical-harmonic degree 3.The loss combines L1 reconstruction, SSIM, LPIPS, and density regularization with weights 0.7, 0.3, 0.1, and 0.1, respectively.
  • C Background Reconstruction: Position learning rate decays from 1.6 × 10^-4 to 1.6 × 10^-6, with densification every 500 iterations through iteration 3,000.Densification uses a gradient threshold of 0.0002.
  • C Background Reconstruction: Views near original cameras receive up to 3× higher loss weight through cosine falloff, prioritizing ground-truth supervision over interpolated views.The boost magnitude is controlled by λ_gt=2 and the weighting depends on angular distance to the nearest original camera.
  • C Background Reconstruction: At iteration 3,500, the method adds 240 views with ±15% height oscillation, refines them using Difix3D+, and assigns them loss weight 0.7.

D Human Reconstruction

StudioRecon reconstructs humans by jointly optimizing Gaussian attributes and SMPL pose parameters under photometric, perceptual, silhouette, density, and temporal-smoothness objectives. Linear blend skinning and a temporal deformation network model articulated and time-varying motion before rendering.

  • Optimization: Human Gaussians are optimized for 10,000 iterations on four input videos with spherical harmonics degree 2.Densification terminates at iteration 1,500.
  • Optimization: The loss combines L1, SSIM, LPIPS, mask, density-regularization, and pose-regularization terms with specified weights, while jointly optimizing Gaussian attributes and SMPL pose parameters.Pose regularization uses λ=50.0 to enforce temporal smoothness; the position learning rate decays from 1.0 × 10−3 to 2.0 × 10−6.
  • Linear Blend Skinning: Linear blend skinning uses SMPL weights computed from a 643-voxel grid.The skinning weights support articulated human deformation.
  • Temporal Deformation Network: Temporal deformation begins at iteration 3,000 with an 8-layer MLP containing 256 hidden units and a skip connection at layer 4.The network predicts per-Gaussian residuals for temporal deformation.
  • Temporal Deformation Network: The deformation network predicts color and opacity residuals from positional encodings of spatial coordinates and normalized time, then adds them to base Gaussian attributes before rendering.The encodings use multires=10, yielding 63D spatial and 21D temporal features.

E Recursive Enhancement Module

StudioRecon’s recursive enhancement module enforces temporal consistency through motion-adaptive reference injection during inference. It further smooths confidence transitions with Gaussian blur.

  • Consistency Injection: Temporal consistency uses motion-adaptive EMA injection from the previous three frames, with α=0.3 and decay factor 0.5.Inference uses per-frame reference mode with single-step denoising at timestep t=200.
  • Consistency Injection: Injection strength is determined by per-pixel RGB warp error using a threshold τ_e=30 on a 0–255 scale.Pixels with low warp error receive full injection.
  • Consistency Injection: A Gaussian blur with kernel size 11 smooths the confidence map and reduces sharp transitions.

F Runtime Analysis

StudioRecon is designed for offline processing, with video-diffusion view synthesis as the main computational bottleneck but substantial multi-GPU parallelism. For a typical 121-frame scene with 1–3 actors, end-to-end rendering takes approximately 130 minutes on one GPU or 60 minutes with eight GPUs.

  • Bottleneck: View synthesis with GEN3C is the primary runtime bottleneck, but processing trajectory segments independently reduces this stage to approximately 13 minutes with 8 GPUs.The bottleneck arises from the computational cost of video diffusion models.
  • Runtime Scaling: Human reconstruction scales linearly with actor count at approximately 3 minutes per additional human, while other stages remain largely unaffected.Background and human reconstruction can run in parallel after preprocessing completes.
  • Parallelization: Preprocessing and view synthesis support multi-GPU processing, and segmentation, depth estimation, and pose estimation can execute concurrently.The pipeline is designed for offline processing, with diffusion-based rendering enhancement adding per-frame computation to the final output.
  • End-to-End Time: For a typical scene with 121 frames and 1-3 actors, end-to-end rendering takes approximately 130 minutes on a single GPU or 60 minutes with 8 GPUs.These times measure production of the final rendered video.

G Additional Results

The additional results characterize the extreme low-overlap setting and validate StudioRecon’s pose initialization, camera augmentation, and artifact suppression. Across datasets, explicit geometric constraints and decoupled reconstruction outperform video-diffusion alternatives for human consistency and novel-view quality.

  • Camera Configuration: Neighboring cameras in prior sparse-view protocols overlap by 28°, whereas StudioRecon’s views observe different scene regions with 70° separation and heavy self-occlusion.This comparison spans EgoHumans, Harmony4D, Mobile Stage, and SelfCap.
  • Augmented Camera Trajectory: The augmented trajectory adds 240 interpolated viewpoints to 481 synthesized poses, including ±15% scene-radius height variation and compensating pitch rotation.These elevated and lowered viewpoints provide additional supervision for undersampled regions such as floors and ceilings.
  • Cross-View Identity Association: Cross-view identity association achieves 97.8% accuracy across 8 scenes by matching 3D-lifted detections using spatial proximity and SMPL pose similarity.The Hungarian algorithm assigns correspondences across four low-overlap cameras.
  • Enhancement on Baseline Methods: Recursive enhancement reduces LPIPS by 20–31% across baseline methods, but produces modest PSNR and SSIM gains and remains below the full pipeline’s PSNR 20.44 and LPIPS 0.198.The module suppresses floaters and blur, while diffusion enhancement cannot recover geometry lost during reconstruction.
  • Baselines with Dense GEN3C Supervision: Dense GEN3C supervision improves baseline background coverage, but StudioRecon reaches 18.08 versus 17.14 FG PSNR with STG+GEN3C using 30× less diffusion.GEN3C-supervised humans also exhibit pose inconsistency, confirming that dense diffusion alone cannot replace SMPL-guided geometric constraints for dynamic regions.
  • GEN3C vs. Ours (Rendering Quality): StudioRecon surpasses GEN3C on held-out cameras with PSNR/SSIM/LPIPS 20.13/0.656/0.215 versus 19.05/0.612/0.329, despite using GEN3C only at t=0.The decoupled pipeline avoids propagating GEN3C’s geometric inconsistencies into dynamic human regions.

H Robustness Analysis

The robustness analysis shows that multi-view pose refinement, temporal smoothing, and mask dilation each prevent distinct reconstruction artifacts when their components are removed.

  • Multi-View Pose Refinement: Removing multi-view pose refinement misaligns Gaussians with human regions, producing blurry or semi-transparent body parts.The refinement uses cross-view triangulation and 3D-to-3D SMPL fitting to correct initial per-view estimates.
  • Mask Dilation: Without 21-pixel mask dilation, residual human pixels enter background Gaussians and cause ghosting near moving human boundaries.Dilation fully excludes human regions; without it, tight masks leave human content baked into the static background.

I Limitations

The method does not reconstruct dynamic objects or accessories held by actors because SMPL models only the human body surface. Shadows baked into the static background at t=0 also remain fixed as humans move, causing incorrect shadow behavior.

  • Dynamic objects: Dynamic objects held by actors are not reconstructed because they lie outside the SMPL body model.SMPL represents only the human body surface and does not model held objects or accessories.
  • Shadow artifacts: Shadows baked into the static background at t=0 remain fixed instead of following human motion, resulting in incorrect shadows.This artifact appears at later timesteps as humans move.
Loading 2607.09125v1…