PaperlayerView exact revision on arXiv

Source-linked AI summary

MonoHuman: Animatable Human Neural Field from Monocular Video

Zhengming Yu, Wei Cheng, Xian Liu, Wayne Wu, Kwan-Yee Lin

arXiv:2304.02001v1cs.CVcs.GR

TL;DR

Monocular-video avatar animation remains difficult because prior methods can overfit to observed poses, while multi-view capture requires specialized setups. MonoHuman combines shared bidirectional deformation with keyframe correspondence features, and reports high-fidelity rendering that outperforms state-of-the-art methods. Its results include challenging novel poses, though synthesis depends on accurate pose and mask annotations and training remains case-specific.

  • Problem

    Prior monocular-video deformation methods can overfit to observed poses, making generalization to unseen poses difficult, while multi-view methods require carefully collected videos and controlled studios.

    Source: §1 · pp. 1–2

  • Method

    MonoHuman combines shared bidirectional deformation, with shared skeletal motion and separate non-rigid motions, and keyframe correspondence search to guide rendering.

    Source: §1 · p. 2

  • Results

    MonoHuman renders high-fidelity results and outperforms state-of-the-art methods, including under challenging novel pose settings.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework synthesizes avatars at novel viewpoints and poses using monocular video, with keyframe appearance features helping render natural appearance, including currently invisible regions.

    Source: §1 · p. 2

    Synthesis relies on accurate pose and mask annotations, and the method is trained in a case-specific manner.

    Source: §5 · p. 9

Abstract

from arXiv · show

Animating virtual avatars with free-view control is crucial for various applications like virtual reality and digital entertainment. Previous studies have attempted to utilize the representation power of the neural radiance field (NeRF) to reconstruct the human body from monocular videos. Recent works propose to graft a deformation network into the NeRF to further model the dynamics of the human neural field for animating vivid human motions. However, such pipelines either rely on pose-dependent representations or fall short of motion coherency due to frame-independent optimization, making it difficult to generalize to unseen pose sequences realistically. In this paper, we propose a novel framework MonoHuman, which robustly renders view-consistent and high-fidelity avatars under arbitrary novel poses. Our key insight is to model the deformation field with bi-directional constraints and explicitly leverage the off-the-peg keyframe information to reason the feature correlations for coherent results. Specifically, we first propose a Shared Bidirectional Deformation module, which creates a pose-independent generalizable deformation field by disentangling backward and forward deformation correspondences into shared skeletal motion weight and separate non-rigid motions. Then, we devise a Forward Correspondence Search module, which queries the correspondence feature of keyframes to guide the rendering network. The rendered results are thus multi-view consistent with high fidelity, even under challenging novel pose settings. Extensive experiments demonstrate the superiority of our proposed MonoHuman over state-of-the-art methods.

1. Introduction

MonoHuman targets animatable, free-viewpoint avatars from monocular video, addressing the difficulty of generalizing to unseen poses. It combines bidirectional deformation with keyframe correspondence features to render high-fidelity results across novel poses and viewpoints.

  • Prior deformation methods can overfit seen poses, while other approaches face slow root-finding, frame-dependent weights, or reliance on imperfect body templates.Learning an accurate, generalizable deformation field remains an open problem.Source: §1 · pp. 1–2
  • The framework aims to improve generalization by defining deformation weights in canonical space, making them pose-independent, and unifying forward and backward deformation.Direct appearance reference from input observations is also identified as helpful for rendering fidelity.Source: §1 · p. 2
  • MonoHuman uses one shared skeletal motion with separate residual non-rigid motions to model backward and forward deformation.The shared basis encourages general rigid transformation weights, while residual motions retain pose-dependent deformation expressiveness.Source: §1 · p. 2
  • Its Forward Correspondence Search module queries appearance features from sparse keyframes to guide rendering, including for regions invisible in the current frame.Source: §1 · p. 2
  • MonoHuman reconstructs and animates an avatar using a monocular video as supervision, with free-viewpoint synthesis and explicit pose control.Source: §1 · p. 2
  • Extensive experiments report that MonoHuman renders high-fidelity results and outperforms state-of-the-art methods.Source: §1 · p. 2

2. Related Work

Related work spans human capture and rendering from multi-view or monocular data, as well as image-based rendering. MonoHuman is positioned against methods whose capture requirements, deformation generalization, or renderable view range impose constraints.

Human Performance Capture

  • Multi-view and depth-camera capture support human reconstruction, but these methods are largely limited to indoor multi-camera setups with accurate calibration and restricted capture volumes.Source: §2 · p. 2

Human Rendering from Monocular Video

  • Monocular dynamic-human methods use deformable NeRFs and motion priors, yet depth ambiguity and insufficient pose observations can lead deformation to overfit and produce floating artifacts in novel views.Source: §2 · p. 2

Human Animation

  • Prior animation methods can struggle to generalize beyond observed poses; approaches using root-finding to learn deformation fields are highly time-consuming.Source: §2 · p. 3

Image-based rendering

  • Image-based rendering can produce realistic novel views, but light-field interpolation has a limited view range and proxy-geometry methods are sensitive to geometry accuracy.Learned image-based rendering methods use features from multiple views; NeuRay additionally predicts point visibility to input views.Source: §2 · p. 3

3. Our Approach

MonoHuman represents the avatar in canonical space, couples forward and backward deformation, and uses keyframe correspondences to guide rendering. The approach combines deformation consistency, selected observations, and canonical-space volume rendering.

3.1. Preliminaries and Problem Setting

  • The neural field maps posed observation-space points into one canonical space, where it predicts color and density for volume rendering.The backward deformation takes the body pose and observation-space point as input and outputs a canonical-space point.Source: §3.1 · p. 4

3.2. Shared Bidirectional Deformation Module

  • Shared Bidirectional Deformation uses canonical-space motion weights for forward and backward warps, with separate non-rigid motions and a consistency loss that penalizes only errors above a threshold.The module is designed to make the deformation field pose-independent; the threshold is used to avoid over-regularization.Source: §3.2 · p. 4

3.3. Forward Correspondence Search Module

  • The observation bank selects front- and back-view keyframes by pelvis orientation, keeps the closest-pose pairs, then chooses the pair with the greatest texture-map complementarity.Forward deformation projects canonical-space points into keyframe images, where sampled features and colors are blended to guide rendering.Source: §3.3 · pp. 5–6

3.4. Volume Rendering and Network Training

  • In canonical space, the rendering network combines a point with its correspondence feature to predict color and density; training uses MSE, LPIPS, and deformation-consistency losses.The predicted field is rendered with volume rendering.Source: §3.4 · p. 6

4. Experiments

The experiments evaluate MonoHuman on ZJU-MoCap and internet videos, including novel-view and novel-pose synthesis, qualitative challenge poses, and module ablations. Results favor MonoHuman across most reported metrics and show the modules’ roles in rendering and deformation.

4.1. Dataset and Preprocessing

  • ZJU-MoCap evaluation uses six subjects; internet videos use PARE-estimated camera and pose and RVM segmentation, with keyframes selected for the method.The study compares novel views and poses using camera 1 training data, separating seen and unseen poses for the pose evaluation.Source: §4.1 · p. 6, §4.3 · p. 7

4.3. Quantitative Evaluation

  • About 19.6% and 9.46% LIPIS improvement in novel-view and novel-pose settings, respectively, accompanies MonoHuman’s lead on most metrics in both settings.PSNR improvement over HumanNeRF is not significant and is slightly lower for subject 387 in novel-pose testing.Source: Table 1 · p. 7, Table 2 · p. 7, §4.3 · p. 7

4.4. Qualitative Evaluation

  • MonoHuman retains detail in extreme internet-video views and produces more realistic results than HumanNeRF on generated challenge poses, including squatting, jumping, punching, and back-flips.HumanNeRF shows artifacts in extreme-pose examples, including incorrect arm deformation in punch and back-flip motions.Source: §4.4 · p. 7, Figure 6 · p. 8

4.5. Ablation Study

  • Both modules contribute roughly equally in novel-view ablations; deformation consistency matters for novel poses, while correspondence guidance matters in both settings and depends on accurate deformation.Incorrect deformation also affects the correspondence features searched by FCS, worsening generated results.Source: §4.5 · p. 8

5. Discussion

MonoHuman renders view-consistent, high-fidelity avatars at novel poses, including challenging poses. Its synthesis depends on accurate pose and mask annotations, and its training is case-specific.

5. Discussion

  • MonoHuman is reported to render view-consistent, high-fidelity avatars at novel poses, including challenging poses.The discussion describes challenge poses generated with Human Motion Diffusion Model [45].Source: §5 · p. 8
  • Synthesis relies on accurate pose and mask annotations, and training is case-specific; generalizing across people remains future work.Source: §5 · p. 9

A. Network Architecture

The network architecture blends correspondence features with one MLP and uses a second MLP to produce rendered color and density.

A.1. Blend MLP

  • A three-layer MLP takes concatenated correspondence features f_i and f_j as input and outputs their blend weight W.Source: Figure A1 · p. 11

A.2. Rendering Network

  • The 16-layer rendering MLP uses positional encoding γ(x) in its first eight layers, then combines their output with blended feature F to produce RGB.The first eight layers output 36 dimensions, including density σ; the second eight output the final color.Source: Figure A2 · p. 11

B.1. Ablation Study on Shared Bidirectional Deformation Module

The appendix ablation illustrates how the consistent loss affects deformation, including in arm regions, and visualizes its regularizing behavior.

B.1. Ablation Study on Shared Bidirectional Deformation Module

  • Without consistent loss, arm deformation bends and produces obvious artifacts; with the loss, deformation in those regions is more accurate.Source: §B.1 · pp. 11–12, Figure A3 · p. 11
  • Points flagged by the consistent loss decrease over iterations, illustrating that it detects incorrect deformation and regularizes shared deformation weights.The visualization marks points with loss values ≥ 0.05 in red.Source: §B.1 · p. 12, Figure A4 · p. 12

B.2. Ablation Study on Forward Correspondence Search Module

Forward Correspondence Search improves cloth color and texture accuracy by providing correspondence features to the rendering process.

B.2. Ablation Study on Forward Correspondence Search Module

  • Correspondence features produce more accurate cloth color and texture; removing them tends to create unnatural cloth textures.The comparison isolates the contribution of the Forward Correspondence Search module.Source: §B.2 · p. 12

B.3. Ablation Study on Sequence Length

Generated quality does not change linearly with training sequence length; the module helps preserve realistic results with only 19 training frames.

B.3. Ablation Study on Sequence Length

  • At 19 training frames, the module helps retain more realistic results, while quality changes nonlinearly with sequence length.The evaluation samples 380, 76, 38, or 19 frames and tests novel views using 22 unseen cameras.Source: §B.3 · p. 12

C. Evaluation details

Evaluation uses held-out cameras for novel-view synthesis and held-out frames for novel-pose synthesis. Conditioning blend weights on view direction improves some training-frame results but produces artifacts on novel views.

C. Evaluation details

  • Novel-view evaluation uses 242 frames from training-set poses across 22 unseen cameras; novel-pose evaluation uses 184 held-out frames across cameras.Training uses Set A, while Set B supplies synchronous frames for novel-pose evaluation.Source: C. Evaluation details · p. 12

D. Condition on view direction

  • View-direction conditioning can improve training-frame results, including in reflective areas, but produces artifacts in novel views and weakens generalization.Figure A6 compares blend weights conditioned on view direction with weights without that condition.Source: Figure A6 · p. 12, D. Condition on view direction · p. 12
  • Table A1 presents an ablation study comparing novel-view synthesis results across different training-frame lengths, with LPIPS reported as LPIPS × 10^3.Source: Table A1 · p. 13

E. More results

MonoHuman retains details in novel-pose results, including challenge poses, and outperforms existing methods on most reported metrics across novel-view and novel-pose settings.

E. More results

  • MonoHuman retains detail in novel poses, while NeuralBody produces vague, noisy images and HumanNeRF shows black-line artifacts; it also retains detail on MDM challenge poses.The qualitative comparisons cover ZJU-MoCap novel poses and challenge sequences generated by MDM [45].Source: E. More results · p. 13, Figure A7 · p. 14, Figure A8 · p. 15
  • MonoHuman outperforms existing methods on most metrics in both novel-view and novel-pose synthesis, with stronger LPIPS gains than PSNR gains over HumanNeRF.PSNR improvement over HumanNeRF is not significant and is slightly lower for subject 387 in novel-pose evaluation; NeuralBody can score well despite poor visual quality.Source: E. More results · p. 13

Related papers

Loading 2304.02001v1…