PaperlayerView exact revision on arXiv

Source-linked AI summary

Wild2Avatar: Rendering Humans Behind Occlusions

Tiange Xiang, Adam Sun, Scott Delp, Kazuki Kozuka, Li Fei-Fei, Ehsan Adeli

arXiv:2401.00431v2cs.CV

TL;DR

Rendering humans from monocular videos is difficult when obstacles block the camera’s view, and many methods assume unobstructed scenes. Wild2Avatar separates occlusion, human, and background components and uses objectives for decoupling and completeness. The authors report higher-fidelity results with fewer occlusion artifacts and more complete bodies than HSR on synthesized videos.

  • Problem

    Many human-rendering methods assume ideal, unobstructed scenes, while in-the-wild monocular videos may show humans partially blocked by objects.

    Source: §1 · p. 1

  • Method

    Wild2Avatar separates occlusion, human, and background components and uses rendering objectives to decouple them and support complete human reconstruction.

    Source: §1 · p. 1

  • Results

    On synthesized videos, Wild2Avatar exhibits significantly higher fidelity, fewer occlusion artifacts, and more complete bodies than HSR.

    Source: §4.4 · p. 8

  • Takeaways & Limitations

    The authors describe Wild2Avatar as a practical and effective solution for high-quality reconstruction of human geometry and appearance under occlusions.

    Source: §5 · p. 10

    Wild2Avatar’s occlusion decoupling uses precomputed binary human segmentation masks, and both it and Vid2Avatar rely on accurate pose estimates.

    Source: §5 · p. 10

Abstract

from arXiv · show

Rendering the visual appearance of moving humans from occluded monocular videos is a challenging task. Most existing research renders 3D humans under ideal conditions, requiring a clear and unobstructed scene. Those methods cannot be used to render humans in real-world scenes where obstacles may block the camera's view and lead to partial occlusions. In this work, we present Wild2Avatar, a neural rendering approach catered for occluded in-the-wild monocular videos. We propose occlusion-aware scene parameterization for decoupling the scene into three parts - occlusion, human, and background. Additionally, extensive objective functions are designed to help enforce the decoupling of the human from both the occlusion and the background and to ensure the completeness of the human model. We verify the effectiveness of our approach with experiments on in-the-wild videos.

1 INTRODUCTION

Wild2Avatar addresses the difficulty of rendering humans from monocular videos when obstacles block the camera’s view. It separates occlusion, human, and background components and evaluates the approach on challenging in-the-wild videos.

1 INTRODUCTION

  • In-the-wild monocular videos can show humans moving behind objects, while many existing methods assume unobstructed views in ideal scenes.Occlusion also leaves limited supervision for appearance, geometry, and pose, and can result in incomplete bodies and visual artifacts.Source: §1 · p. 1
  • Wild2Avatar combines photometric, scene decomposition, occlusion decoupling, and geometry completeness objectives to separate components and support complete human rendering.The scene is rendered as three parts, with optimization objectives designed to decouple the occlusion from the human and background.Source: §1 · p. 1
  • Wild2Avatar divides the scene into occlusion, human, and background components to render occluded humans.The method focuses on reconstructing the human rather than accurately rendering irrelevant occlusions or the background.Source: §1 · p. 1
  • The authors evaluate Wild2Avatar on 14 challenging occlusion-intensive in-the-wild videos and report its effectiveness for rendering occluded humans.The paper describes the method as building on the authors’ earlier ICCV-2023 OccNeRF publication.Source: §1 · pp. 1–2

2 RELATED WORK

Prior work covers monocular human rendering, scene decomposition, and neural occlusion handling, but the cited approaches have limitations in occluded settings. Wild2Avatar extends the authors’ earlier OccNeRF work with revised methods and additional evaluations.

2.1 3D Human Modeling and Rendering

  • Monocular human-rendering methods use NeRFs and body priors, while occlusions blocking the camera’s view remain a difficulty for otherwise effective methods.Recent approaches also use 3D Gaussian splatting for efficient human rendering.Source: §2.1 · p. 2

2.2 Rendering via Scene Decomposition

  • Scene-decomposition methods model and aggregate components separately; Vid2Avatar separates dynamic humans from static backgrounds, while HSR models physical interactions.Vid2Avatar’s high-fidelity renderings with minimal artifacts inspire this work.Source: §2.2 · p. 2

2.3 Occlusion Handling

  • Occlusion-handling methods have constraints: OCC-NeRF and NeuRay require multiple views, while HOSNeRF is robust only to very small occlusions.HOSNeRF is also constrained by COLMAP to dynamic scenes.Source: §2.3 · p. 2

2.4 Connection to Our Previous Work

  • This paper extends the authors’ ICCV-2023 OccNeRF work with revised scene modeling and human-rendering methods and evaluations on seven more challenging in-the-wild videos.The authors also report experiments on robustness to training-image count and generative ability, and describe improved fidelity with fewer artifacts and floaters.Source: §2.4 · p. 2

3 METHODS

Wild2Avatar combines human, occlusion, and background rendering for occluded monocular video, using scene parameterization and objectives aimed at separating components and completing human geometry.

3.1 Preliminaries and Background

  • The human is represented as an SDF surface in canonical space, with posed ray samples deformed between observation and canonical spaces for rendering.The surface is the zero level set of the learned signed-distance function.Source: §3.1 · p. 3

3.2 Occlusion-aware Scene Parameterization

  • Wild2Avatar replaces the human–background split with sequential occlusion→human→background regions, placing potential obstacles between the camera and an inner sphere.The inner sphere has radius r < R; its radius is chosen so every camera ray intersects it, using binary search during preprocessing.Source: §3.2 · pp. 3–4
  • The unbounded background uses inverted-sphere coordinates, while occlusion samples use the inner sphere; one scene network renders both with normalized inputs.The shared network is conditioned on different per-frame latent codes, and normalization places both coordinate sets on the unit sphere.Source: Figure 2 · p. 3, §3.2 · p. 4

3.3 Optimization Objectives

  • The occlusion-decoupling objective encourages obstacle density where the human is occupied but not visible, reducing the tendency to absorb obstacle textures into the human.The method uses a segmentation mask and volume-rendered human occupancy to define this target.Source: §3.3 · p. 5
  • A separate completeness regularizer pulls near-surface ray samples toward the implicit surface, and Wild2Avatar optimizes the objectives together with scale weights.The near-surface set is determined by distances to a human mesh initialized from SMPL and updated during optimization.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments evaluate Wild2Avatar on OcMotion, real-world monocular videos, and synthesized occlusions, using dataset-provided or estimated camera and human parameters. Implementation details specify per-video NeRF optimization and shared segmentation-mask supervision for Wild2Avatar and Vid2Avatar.

4.1 Datasets

  • Six challenging OcMotion sequences were selected from 48 videos, with 100 frames per sequence used for training and dataset-provided camera matrices, poses, and SMPL parameters.Frame-wise binary human masks came from the Segment Anything Model.Source: §4.1 · p. 5
  • Three real-world videos were sampled at 150 frames each, with camera matrices, poses, and SMPL parameters estimated by SLAHMR.Because these videos lack ground-truth poses, their evaluations also test robustness to inaccurate estimated priors.Source: §4.1 · p. 5
  • Five NeuMan videos were augmented with static, textured box occluders to create synthesized occlusions, bringing the evaluation total to 14 videos.The occluders were 3D-consistent with the background; these experiments include moving cameras and long-horizon occlusions.Source: §4.1 · p. 5
  • Most evaluations use stationary-camera videos because occluded moving-camera footage is scarce; one in-the-wild video provides a moving-camera evaluation.Wild2Avatar can also be applied to moving-camera setups.Source: §4.1 · p. 5

4.2 Implementation Details

  • Human NeRFs are optimized per video, and Wild2Avatar and Vid2Avatar use the same binary human segmentation mask for fairness.Training samples 512 rays within the human bounding box; inference uses 128 ray samples, versus 64 during training.Source: §4.2 · pp. 5–6

4.3 Evaluations

The evaluation compares Wild2Avatar with Vid2Avatar, OccNeRF, and HSR using qualitative views and metrics for visible appearance, completeness, and perceived quality. The protocols include OcMotion, in-the-wild, and synthesized-video settings, with OccNeRF tested at two training-set sizes.

4.3 Evaluations

  • Wild2Avatar is compared mainly with Vid2Avatar and also with OccNeRF at 100 and 500 training frames; HSR comparisons cover two synthesized videos.All methods use the same binary human masks, and OccNeRF’s original setup requires more than 500 frames for stable training while Wild2Avatar and Vid2Avatar require about 100.Source: §4.3 · pp. 6–7, Figure 7 · p. 7, Figure 8 · p. 8
  • Qualitative comparisons include Vid2Avatar on three in-the-wild monocular videos and HSR on two synthesized occluded videos; occluded-human novel views are unavailable in both figure sets.Table 1 covers quantitative comparisons on three datasets, and Table 2 covers the synthesized videos.Source: Figure 6 · p. 7, Figure 7 · p. 7, Table 1 · p. 8
  • NeuMan is unsuitable for stationary-camera scenes, while the official HOSNeRF implementation is limited to six paper videos and fails when occlusion is disconnected from the human.NeuMan requires three GPUs and 175 hours of training, HOSNeRF four GPUs and 118 hours, and Wild2Avatar one 24GB GPU and 48 hours.Source: §4.3 · p. 7
  • Evaluation measures visible-human PSNR, completeness by IoU against ground-truth SMPL silhouettes, and user-rated occluded-human quality from 25 out-of-field participants.For synthesized videos, PSNR and SSIM are computed between humans in unoccluded raw videos and rendered humans at input views.Source: §4.3 · pp. 7–8

4.4 Results

Across the reported comparisons, Wild2Avatar recovers occluded human parts more consistently than Vid2Avatar and produces higher-fidelity results than HSR on synthesized videos. Quantitative results are on par for visible parts but favor Wild2Avatar for body geometry and occluded-part quality.

4.4 Results

  • Wild2Avatar consistently outperforms Vid2Avatar in the reported real-world in-the-wild videos.On OcMotion, both methods reconstruct visible human parts with high fidelity and without obvious artifacts or floaters.Source: §4.4 · p. 8
  • Against OccNeRF trained on 100 or 500 frames, Wild2Avatar’s comparison notes that OccNeRF can recover occluded parts but is prone to floaters, artifacts, and unexpectedly twisted bodies.The reported defects are associated with lower rendering quality.Source: §4.4 · p. 8
  • Wild2Avatar’s visible-part rendering is on par with Vid2Avatar, while its body geometry and occluded-part rendering quality are consistently better.On OcMotion, Vid2Avatar struggles to recover occluded body parts and sometimes renders occlusions as human appearance; Wild2Avatar decouples them and recovers consistent human appearances.Source: §4.4 · p. 8
  • On two synthesized occluded videos, Wild2Avatar produces higher-fidelity renderings with fewer occlusion artifacts and more complete bodies than HSR.The reported comparisons also support performance on videos with moving cameras.Source: §4.4 · p. 8

4.5 Geometry Reconstructions

Volumetric rendering produces both human appearance and underlying geometry, which the paper visualizes through surface normals overlaid on input frames. The reported reconstructions show strong geometry recovery from occluded videos.

4.5 Geometry Reconstructions

  • Volumetric rendering of human avatars yields both their appearance and their underlying geometry.The paper overlays reconstructed human surface normals on input frames to show geometry.Source: §4.5 · p. 8
  • Reconstructed human surface normals are overlaid on input frames as a visualization of the recovered geometry.Source: §4.5 · p. 8
  • The reported results show that Wild2Avatar excels at reconstructing human geometry from occluded videos.Source: §4.5 · p. 8

4.6 Ablation Studies

Ablations show that the proposed scene parameterization, occlusion decoupling loss, and completeness loss each address distinct failure modes in occluded-human reconstruction. Joint optimization can also refine inaccurate predicted SMPL poses toward reconstruction quality comparable to ground-truth poses.

Impact of occlusion-aware parameterization and Locc

  • Removing the occlusion-aware parameterization leaves artifacts from data scarcity, while removing Locc prevents full recovery of occluded human regions.These comparisons are shown in the first and second columns of Figure 11.Source: §4.6 · p. 8

Impact of Lcomp

  • Removing Lcomp can degenerate 2D renderings and weaken consistency between human geometry and the SMPL mesh, risking incorrect poses.The pose-related effect is shown in the third column of Figure 11.Source: §4.6 · p. 9

Impact of optimizing inaccurate SMPL jointly with Wild2Avatar

  • Joint optimization progressively corrects inaccurate predicted SMPL poses, achieving reconstruction quality comparable to ground-truth SMPL parameters.The comparison uses OcMotion videos and is highlighted in Figure 13.Source: §4.6 · p. 9

4.7 Visualizations of Scene Decomposition

Wild2Avatar separately models and renders occlusion, human, and background components, while its three-stage parameterization supports plausible rendering of unobserved human parts. The paper also visualizes progressive correction of inaccurate predicted limb rotations through joint optimization.

4.7 Visualizations of Scene Decomposition

  • Wild2Avatar models human and background/occlusion in separate neural fields and renders the three scene parts individually.Artifact-free rendering of background and occlusions is outside this work's scope.Source: §4.7 · p. 9

4.8 Generative Ability

  • Wild2Avatar renders complete, reasonable human geometry even without a clear view of the back, unlike methods assuming every body part is visible in some frame.The qualitative comparison with Vid2Avatar varies the amount of training imagery showing the full back.Source: §4.8 · p. 9, Figure 12 · p. 10
  • Joint optimization progressively corrects inaccurate limb rotations in predicted SMPL parameters.Figure 13 compares joint training with predicted parameters against training with ground-truth parameters.Source: Figure 13 · p. 10

5 DISCUSSION AND CONCLUSION

The discussion identifies reliance on semantic priors, segmentation masks, and accurate pose estimates as constraints on occluded-human reconstruction. The conclusion presents Wild2Avatar as a high-fidelity approach for challenging occluded in-the-wild videos, while noting the optimization cost of modeling occlusions.

Discussion

  • Without semantic priors for occluders, fully unsupervised occlusion-human decoupling is impractical; the proposed Locc instead uses precomputed binary human segmentation masks.Both Vid2Avatar and Wild2Avatar rely on accurate pose estimates, and inaccurate initialization can produce poor or incomplete rendering.Source: §5 · p. 10
  • Modeling occlusions increases inference time and slows optimization, while future work may add pose correction or use more efficient rendering methods.The proposed future directions include efficient NeRF variants or Gaussian splatting.Source: §5 · p. 10

Conclusion

  • The conclusion states that Wild2Avatar decouples human, background, and obstacles to produce complete, artifact-free renderings and achieve high-fidelity state-of-the-art results on challenging in-the-wild videos.Source: §5 · p. 10

6 BIOGRAPHY

The biographies describe the authors’ academic and industry affiliations and research backgrounds. They include interests spanning 3D computer vision, healthcare, movement science, neuroscience, and machine learning.

Biographies

  • Tiange Xiang’s research focuses on 3D computer vision and AI for healthcare, while Adam Sun is interested in 3D computer vision for scene understanding and medical applications.Source: §6 · p. 11
  • Scott Delp’s work spans movement science and digital health, and Kazuki Kozuka’s research interests include visual understanding through machine learning.Source: §6 · p. 11
  • Fei-Fei Li’s roles include co-directing Stanford’s Human-Centered AI Institute, while Ehsan Adeli applies AI expertise to healthcare and neuroscience.Source: §6 · p. 11
Loading 2401.00431v2…