PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, Yebin Liu

arXiv:2105.00261v2cs.CV

TL;DR

Sparse-view multi-person capture remains difficult because prior methods rely on restrictive capture setups or struggle with severe occlusions. DeepMultiCap combines attention, SMPL priors, and temporal fusion; it reports state-of-the-art performance with partial observations and contributes data for evaluation.

  • Problem

    Prior capture systems often require dense professional camera setups, templates, or RGB-D sensors, while implicit-function methods struggle with severe occlusions in multi-person scenes.

    Source: §1 · p. 1

  • Method

    DeepMultiCap combines an attention-aware multiview network with SMPL priors for occluded regions and temporal fusion for moving-character video.

    Source: §1 · p. 2

  • Results

    The method reports state-of-the-art performance with partial observations in each view and reports generalization capacity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MultiHuman contributes 150 high-quality interactive-scene scans, categorized by occlusion level and person count, for evaluating multi-person capture systems.

    Source: §1 · p. 2

    The method assumes calibrated cameras, accurate segmentation, and well-fitted SMPL; large preprocessing errors can cause failures.

    Source: §7 · p. 8

Abstract

from arXiv · show

We propose DeepMultiCap, a novel method for multi-person performance capture using sparse multi-view cameras. Our method can capture time varying surface details without the need of using pre-scanned template models. To tackle with the serious occlusion challenge for close interacting scenes, we combine a recently proposed pixel-aligned implicit function with parametric model for robust reconstruction of the invisible surface areas. An effective attention-aware module is designed to obtain the fine-grained geometry details from multi-view images, where high-fidelity results can be generated. In addition to the spatial attention method, for video inputs, we further propose a novel temporal fusion method to alleviate the noise and temporal inconsistencies for moving character reconstruction. For quantitative evaluation, we contribute a high quality multi-person dataset, MultiHuman, which consists of 150 static scenes with different levels of occlusions and ground truth 3D human models. Experimental results demonstrate the state-of-the-art performance of our method and the well generalization to real multiview video data, which outperforms the prior works by a large margin.

1. Introduction

DeepMultiCap targets sparse-view multi-person capture, where prior systems often depend on templates, RGB-D sensors, or single-person settings, and implicit-function methods struggle with severe occlusion. It combines spatial attention, parametric body priors, and temporal fusion, and contributes the MultiHuman dataset for evaluation.

1. Introduction

  • Earlier lightweight capture systems rely on pre-scanned templates, RGB-D sensors, or single-person reconstruction, while dense-camera systems can require expensive professional setups.Source: §1 · p. 1
  • PIFu and PIFuHD can lose detail with average-pooled multiview features and deteriorate when severe occlusion removes observations.Source: §1 · pp. 1–2
  • The framework combines spatial attention with SMPL priors to reconstruct fine details and complete bodies under close interactions and occlusions.For video, temporal fusion weights signed distance fields across time to improve consistency.Source: §1 · p. 2
  • The MultiHuman dataset contains 150 high-quality scans of interactive scenes, addressing the shortage of data for evaluating multi-person capture systems.Scenes include one to three people and are categorized by occlusion level and person count.Source: §1 · p. 2
  • The method reports state-of-the-art performance with partial observations in each view by leveraging human shape and pose priors to resolve occlusion ambiguities.Source: §1 · p. 2
  • The contribution includes an attention-aware module for fine-grained multiview detail and temporal fusion to reduce inconsistencies in moving-character video reconstruction.Source: §1 · p. 2

2. Related Work

Prior work spans single-view geometry reconstruction, multi-view motion and surface capture, and attention-based vision models. The section highlights gaps in multi-person occlusion handling and detailed geometry capture.

2. Related Work

  • Single-view reconstruction methods estimate shape through silhouettes, depth, templates, or implicit representations, but remain difficult to apply accurately to multi-person scenes.Source: §2 · p. 2
  • Multi-view motion-capture systems predict skeletons rather than detailed geometry, while geometry methods face limits in texture mapping, topology changes, rapid motion, and occlusion handling.Source: §2 · p. 2
  • Attention mechanisms have been used for visual tasks including multiview stereo and hand-pose estimation, with prior work reporting robustness to severe occlusions.Source: §2 · p. 3

3. Overview

The system reconstructs each person from segmented multiview images and corresponding SMPL models, combining the per-person results directly. Its preliminary discussion describes pixel-aligned implicit reconstruction and the limitations of simple feature averaging across views.

3. Overview

  • Segmented multiview images and corresponding SMPL models enter the system, which reconstructs people separately and combines their results directly.SMPL-X fitting uses multiview-video 3D keypoints; segmentation tracks characters using SMPL projection maps.Source: §3 · p. 3

3.1. Preliminary

  • A pixel-aligned implicit function represents a surface through an occupancy field and conditions its prediction on image features and a projected 3D point.The image feature, projected coordinate, and camera-space depth are inputs used to fit the implicit function.Source: §3.1 · p. 3
  • PIFuHD adds a fine-level network using high-resolution images and predicted normal maps to refine the coarse prediction with more detail.Source: §3.1 · p. 3
  • Mean pooling of multiview features can lose detail or collapse when views disagree because of depth variation and occlusions.Source: §3.1 · p. 3

4. Single Person Reconstruction

DeepMultiCap uses coarse-to-fine implicit reconstruction, attention to aggregate multiview features, and SMPL information to support robust reconstruction; video outputs can be temporally fused.

4. Single Person Reconstruction

  • The reconstruction pipeline uses coarse-to-fine pixel-aligned inference, combining multiview attention with SMPL conditioning and optional temporal fusion for video.Source: §4 · pp. 3–4

4.1. Attention-aware Multi-view Feature Fusion

  • Mean pooling can smooth details and fail when visible regions shift across views; self-attention weights view features to retain salient covisible geometry.Source: §4.1 · p. 4

4.2. Embedded with Parametric Body Model

  • SMPL semantic features and rendered global normal maps guide visible-part identification, compensating for occluded image regions and improving reconstruction robustness.Source: §4.2 · p. 5

4.3. Temporal Fusion

  • For video, reconstructed vertices are warped between frames using SMPL-based skinning, then SDFs are mean-pooled over a sliding window of h=3.Source: §4.3 · p. 5

5. Extend to Multi-person Reconstruction

Multi-person reconstruction processes each individual separately and trains the network to remain robust to occlusions in interactive scenes. Handcrafted occlusions are added to training data by projecting other people onto masks in rendered images.

  • Each individual is reconstructed separately, with robustness to interactive-scene occlusions identified as the key training challenge.Source: §5 · p. 5
  • Training adds handcrafted occlusions by randomly projecting other people onto masks in rendered images, simulating cases from no occlusion to heavy occlusion.Source: §5 · p. 5
  • The training set draws on 1,700 single-human models from Twindom and THuman2.0 [61], whose high-quality geometry supports photorealistic image rendering.Source: §5 · p. 5

6. Dataset and Experiment

The section evaluates DeepMultiCap on MultiHuman and real-world multiview data, examines its components and occlusion robustness, and states limitations tied to its SMPL reference and clothing.

6.1. MultiHuman Dataset

  • MultiHuman contains 150 static scenes with 278 characters across one-to-three-person scenes, with models of about 300,000 triangles and photorealistic texture.The scenes are categorized by person count and occlusion level, including closely interactive pairs and three-person scenes.Source: §6.1 · p. 6

6.2. Evaluation

  • On ZJU-Mocap, DeepMultiCap outperforms DeepVisualHull, PIFuHD, and Neural Body; on MultiHuman, it remains competitive as occlusions intensify and reconstructs detailed humans in close interactions.The MultiHuman evaluation compares against PIFu, PIFuHD, and PaMIR; Figure 6 presents examples across single-person, occluded, and interacting scenes.Source: Figure 5 · p. 6, §6.2 · p. 6, Figure 6 · p. 7

6.3. Ablation Study

  • Temporal fusion further improves reconstruction consistency on real-world sequences, while the method maintains high accuracy as occlusions increase and generates smooth results when body parts are invisible.Figure 8 contrasts original per-frame results with results polished by temporal fusion; Figure 7 illustrates close-interaction cases with invisible body parts.Source: §6.3 · pp. 7–8, Figure 7 · p. 7, Figure 8 · p. 8

6.4. Limitations

  • The SMPL-based method is limited to human reconstruction, can be unstable for loose clothing such as wind coats, and may produce artifacts when the SMPL input is inaccurate.The authors report successful reconstruction of tight dresses, but instability for loose clothing.Source: §6.4 · p. 8, Figure 10 · p. 8

7. Discussion and Future Works

The system assumes well-conditioned preprocessing, tolerates small noise, and may fail when preprocessing errors are large; future work targets a simpler, more applicable pipeline.

7. Discussion and Future Works

  • Large preprocessing errors could lead to failure, although the system remains robust against small noise in real-world cases.The stated preprocessing conditions are calibrated cameras, accurate segmentation, and well-fitted SMPL.Source: §7 · p. 8
  • Future work could simplify the pipeline through autonomous calibration, implicit human templates, and lightweight networks for real-time inference.The authors state these directions would make the system more applicable.Source: §7 · p. 8
  • The method's reconstruction assumes calibrated cameras, accurate segmentation, and well-fitted SMPL.Source: §7 · p. 8

Related papers

Loading 2105.00261v2…