PaperlayerView exact revision on arXiv

Source-linked AI summary

Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors

Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, Yebin Liu

arXiv:2105.01859v2cs.CV

TL;DR

Real-time volumetric capture of complex human scenarios remains challenging with lightweight setups because sparse consumer RGBD sensors provide noisy and incomplete observations. Function4D combines dynamic sliding fusion with detail-preserving deep implicit functions, and reports detailed, temporally continuous capture across challenging scenarios using very sparse consumer RGBD sensors.

  • Problem

    Real-time capture of complex human scenarios remains challenging because lightweight systems must handle topology changes, occlusions, fast motion, and noisy consumer RGBD observations.

  • Method

    Function4D combines dynamic sliding fusion for topology-consistent temporal fusion with detail-preserving deep implicit functions for detailed RGBD-based surface reconstruction.

  • Results

    Function4D achieves detailed and temporally continuous volumetric capture under severe topology changes, human-object manipulations, and multi-person interactions using very sparse consumer RGBD sensors.

  • Takeaways & Limitations

    The paper presents a lightweight, high-fidelity, real-time capture system aimed at applications including consumer-level holographic communications, online education, and gaming.

Abstract

from arXiv · show

Human volumetric capture is a long-standing topic in computer vision and computer graphics. Although high-quality results can be achieved using sophisticated off-line systems, real-time human volumetric capture of complex scenarios, especially using light-weight setups, remains challenging. In this paper, we propose a human volumetric capture method that combines temporal volumetric fusion and deep implicit functions. To achieve high-quality and temporal-continuous reconstruction, we propose dynamic sliding fusion to fuse neighboring depth observations together with topology consistency. Moreover, for detailed and complete surface generation, we propose detail-preserving deep implicit functions for RGBD input which can not only preserve the geometric details on the depth inputs but also generate more plausible texturing results. Results and experiments show that our method outperforms existing methods in terms of view sparsity, generalization capacity, reconstruction quality, and run-time efficiency.

1. Introduction

Function4D targets real-time human volumetric capture with very sparse consumer RGBD sensors, addressing challenging motions, topology changes, and occlusions. It combines dynamic sliding fusion with detail-preserving implicit functions to produce detailed, temporally continuous reconstructions.

  • Motivation: Real-time volumetric capture matters for applications including telecommunications, education, and entertainment but remains difficult for complex human-centric scenarios.Challenges include complex shapes, fast motions, and changing topologies such as human-object manipulations and multi-person interactions.
  • Motivation: High-end systems can reconstruct high-quality volumes but require dense camera rigs, custom lighting, complicated setups, and professional studios.Some systems use up to 100 cameras, limiting their practicality outside controlled environments.
  • Proposed System: Function4D uses as few as 3 consumer RGBD sensors while targeting human-object manipulations, clothing changes, fast motions, and multi-person interactions.The system is presented as a lightweight alternative to dense-camera capture systems.
  • Proposed System: Dynamic sliding fusion restricts tracking and fusion to a temporal window, producing noise-eliminated, topology-consistent, and temporally continuous fusion results.This design reduces dependence on accurate long-term non-rigid tracking under topology changes and occlusions.
  • Proposed System: Detail-preserving implicit functions use RGBD information to reconstruct detailed surfaces while preserving geometric details from depth inputs.The framework explicitly encodes truncated projective SDF values and uses attention for multi-view feature aggregation.
  • Contributions: The paper contributes a publicly available dataset containing 500 high-resolution scans of varied poses and clothing.The dataset is intended to stimulate future research.

2. Related Work

Prior work spans multi-view stereo, template-based capture, non-rigid fusion, and learning-based human reconstruction. These approaches achieve strong results but commonly depend on controlled studios, fixed templates, high-end hardware, tracking, or computationally expensive models.

  • Volumetric capture from multi-view stereo: Multi-view stereo systems combine cues such as silhouettes, stereo, shading, and cloth priors for human reconstruction.State-of-the-art systems may use up to 100 cameras, custom gradient lighting, and extensive mesh-processing pipelines.
  • Volumetric capture from multi-view stereo: Controlled multi-camera studios and substantial computational resources limit multi-view stereo systems in living spaces.Their pipelines can include point-cloud generation, mesh construction, simplification, tracking, and UV mapping.
  • Non-rigid Fusion: Single-view non-rigid fusion methods incorporate geometric, articulated, skeletal, parametric, inertial, and learned priors to improve reconstruction.They remain prone to tracking failure in invisible areas, while sparse multi-view systems can produce incomplete and noisy reconstructions under severe topology changes.
  • Learning-based 3D Human Reconstruction: Learning-based reconstruction methods can recover humans from partial observations but often depend on parametric body models, multi-scale 3D convolutions, or costly post-processing.These dependencies impede real-time reconstruction performance for methods such as PaMIR, IFNet, and IPNet.

3. Overview

The Function4D pipeline has two main stages: dynamic sliding fusion followed by deep implicit surface reconstruction. It fuses neighboring RGBD frames, rerenders synchronized views, and reconstructs the final surface from the rerendered data.

  • Dynamic Sliding Fusion: Dynamic sliding fusion fuses neighboring synchronized multi-view RGBD frames to generate noise-eliminated and temporally continuous fusion results.The fused surfaces are then rerendered from the original camera viewpoints.
  • Deep Implicit Surface Reconstruction: Detail-preserving implicit functions reconstruct the final geometry and color from the rerendered multi-view RGBD images.The overview identifies this as the second major pipeline stage after fusion.

4. Dynamic Sliding Fusion (DSF)

Dynamic Sliding Fusion augments current observations within a short temporal window rather than completing surfaces, maintaining topology consistency while reducing noise and tracking-error accumulation.

  • 4. Dynamic Sliding Fusion (DSF): Dynamic sliding fusion produces more complete, noise-eliminated, topology-consistent, and temporally continuous results under severe topology changes and occlusions.Its design restricts fusion and tracking to a sliding window instead of relying on long-term fusion.
  • 4. Dynamic Sliding Fusion (DSF): DSF combines topology-aware node initialization, non-rigid surface tracking, and observation-consistent TSDF fusion.The pipeline initializes a current-frame graph, tracks deformation to the succeeding frame, and selectively fuses observations.
  • 4. Dynamic Sliding Fusion (DSF): A one-frame-delay sliding window fuses frame t with frames t-1 and t+1, limiting topological changes and long-term tracking-error accumulation.Only the succeeding frame requires new non-rigid tracking because deformation from the preceding frame has already been tracked.
  • 4.1. Topology-aware Node Graph Initialization: The current node graph is initialized by warping the preceding live graph, deleting nodes far from current observations, and sampling nodes for newly observed surfaces.This process preserves a node graph whose topology matches current observations while covering surfaces absent from the warped graph.
  • 4.2. Non-rigid Surface Tracking: Non-rigid registration minimizes data-fitting and local-as-rigid-as-possible regularization energies between the reference surface and succeeding-frame depth observations.A Gauss-Newton solver with Preconditioned Conjugate Gradient solves the nonlinear optimization efficiently on GPU.
  • 4.3. Observation-consistent TSDF Fusion: Observation-consistent TSDF fusion updates a voxel only when its interpolated tracking error is below δe and valid current-observation voxels lie within radius 3.Node errors are computed from correspondence residues and interpolated to voxels using K-Nearest-Neighbors with K = 4.

5. Deep Implicit Surface Reconstruction

The method combines fused RGBD surfaces with detail-preserving implicit functions to reconstruct complete, detailed geometry and plausible color from sparse observations.

  • 5. Deep Implicit Surface Reconstruction: Dynamic sliding fusion produces less noisy fused surfaces, but sparse inputs and occlusions leave them incomplete.The implicit reconstruction stage addresses this incompleteness.
  • 5. Deep Implicit Surface Reconstruction: The implicit reconstruction pipeline re-renders multi-view RGBD images from fused surfaces before reconstructing a complete textured surface.Re-rendering also reduces noise and aligns projection parameters with the training data.
  • 5. Deep Implicit Surface Reconstruction: GeoNet and ColorNet infer detailed geometry and color from multi-view RGBD images using related network architectures.GeoNet uses depth-based features, while ColorNet aggregates multi-view features for color inference.
  • 5.2. Detail-preserving Implicit Functions: Truncated PSDF values preserve geometric details from depth inputs and reduce ambiguity from global depth values.The feature is computed from query-point depth, projected image coordinates, sampled depth, and truncation bounds.
  • 5.2. Detail-preserving Implicit Functions: Without PSDF features, visible regions become over-smoothed; without truncation, depth variations can transfer to invisible regions and create ghost artifacts.The ghost-arm artifact is reported in the comparison using untruncated PSDF.
  • 5.2.2 Multi-view Feature Aggregation: ColorNet uses attention-based multi-view aggregation to account for inter-view feature correlations and produce more plausible color inference than direct averaging.The mechanism can weight a viewpoint more strongly when a query point is visible there but occluded elsewhere.

6. Results

The system is evaluated for real-time reconstruction under challenging motion, topology changes, sparse sensing, and multi-view comparison settings. Results show temporally continuous reconstruction and improved geometry quality relative to comparison methods, while GPU implementation targets efficiency.

  • 6. Results: The system reconstructs temporally continuous results under severe topological changes, human-object manipulation, and multi-person interactions.The reported scenarios include the challenging cases shown in Figures 1 and 8.
  • 6.1. Real-time Implementation: The GPU runtime pipeline uses TensorRT mixed-precision inference and depth-based voxel filtering followed by coarse-to-fine octree reconstruction.The reconstruction proceeds from resolution 64^3 to 256^3.
  • 6. Results: Figure 8 evaluates temporal reconstruction on a fast-dancing subject with skirt deformations and severe hair topology changes.The caption characterizes the results as high-quality and temporally continuous.
  • 6.2. Dataset and Training: Training uses 500 high-quality scans with varied poses, clothing, and human-object interactions, rendered into 60 views with synthesized time-of-flight depth noise.The rendered images have resolution 512×512.
  • 6.3. Comparisons: Compared with Motion2Fusion and Multi-view PIFu, the method is qualitatively evaluated using outputs ordered as the proposed method, Motion2Fusion, and Multi-view PIFu.The comparison targets sparse, low-frame-rate consumer RGBD inputs.
  • 6.3. Comparisons: Explicit truncated-PSDF encoding yields accurate reconstruction and is reported as orders of magnitude faster than IPNet, whose reconstruction takes approximately 80 seconds.The passage also attributes Multi-view PIFu degradation to missing depth and IPNet limitations to SMPL initialization dependence.
  • 6.4. Ablation Studies: Dynamic sliding fusion produces more complete and noise-eliminated reconstructions than the non-sliding alternative under inconsistent multi-view depth inputs.The comparison is motivated by challenging hair motion.

7. Conclusion

Function4D combines dynamic sliding fusion with detail-preserving deep implicit functions for detailed, temporally continuous real-time capture from sparse consumer RGBD sensors. Fully occluded surfaces and textures, and observations missing from materials such as black hair, remain challenging.

  • Function4D uses very sparse consumer RGBD sensors for real-time volumetric capture in challenging scenarios.The system addresses severe topological changes, human-object manipulations, fast motions, and multi-person interactions.
  • Dynamic sliding fusion produces topology-consistent and temporally continuous volumetric fusion results.The method restricts tracking and fusion to a sliding window and eliminates noise.
  • Detail-preserving deep implicit functions support high-quality surface reconstruction while preserving geometric details in visible regions.The functions are used for final surface reconstruction after sliding fusion.
  • Limitations and Future Work: Fully occluded regions remain difficult because current deep implicit functions mainly perform per-frame independent reconstruction.The paper suggests using temporal observations in deep implicit functions as future work.
  • Limitations and Future Work: Materials such as black hair can remove depth observations and severely deteriorate the current system.
Loading 2105.01859v2…