PaperlayerView exact revision on arXiv

Source-linked AI summary

HUMBI: A Large Multiview Dataset of Human Body Expressions

Zhixuan Yu, Jae Shin Yoon, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu, Hyun Soo Park

arXiv:1812.00281v3cs.CV

TL;DR

Human appearance modeling needs data covering diverse people and viewpoints rather than detailed scans of each target subject. HUMBI provides such data through synchronized multiview capture of five body-expression categories, 3D mesh reconstruction, and canonical-atlas representation. It outperforms existing limited-view datasets and improves further when combined with them.

  • Problem

    Generalizable modeling of pose- and view-specific human appearance is difficult because existing models rely on detailed scans of individual subjects.

    Source: §1 · p. 1

  • Method

    HUMBI captures gaze, face, hand, body, and garment expressions from 772 diverse subjects using 107 synchronized HD cameras and represents them with 3D meshes and a canonical atlas.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    HUMBI outperforms existing limited-view datasets and is complementary to them, with combined training substantially improving models.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    HUMBI provides a multiview benchmark and data resource for learning more generalizable human reconstruction, recognition, and appearance models.

    Source: §5 · p. 9

Abstract

from arXiv · show

This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. The goal of HUMBI is to facilitate modeling view-specific appearance and geometry of gaze, face, hand, body, and garment from assorted people. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and physical condition. With the multiview image streams, we reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance using their canonical atlas. We demonstrate that HUMBI is highly effective in learning and reconstructing a complete human model and is complementary to the existing datasets of human body expressions with limited views and subjects such as MPII-Gaze, Multi-PIE, Human3.6M, and Panoptic Studio datasets.

1. Introduction

HUMBI addresses the need for a generalizable model of human appearance by providing extensive multiview data across many subjects and body expressions. Its evaluations show that HUMBI outperforms limited-view datasets and complements them when combined.

  • Existing pose- and view-specific models often rely on detailed scans of each target subject, motivating a dataset that generalizes across assorted people without massive per-subject scanning.Source: §1 · p. 1
  • HUMBI targets generalizable human appearance modeling by spanning 107 synchronized HD-camera views and 772 distinctive subjects across diverse demographics and natural clothing.The dataset covers gaze, face, hand, body, and garment expressions.Source: §1 · pp. 1–2
  • HUMBI-trained CNNs quantitatively outperform counterpart models trained on existing datasets with limited views and subjects.The reported analysis evaluates view-invariant 3D pose learning.Source: §1 · p. 2
  • Combining HUMBI with existing datasets substantially improves trained models, demonstrating complementarity rather than simple replacement.Source: §1 · p. 2
  • Its five expression categories and high-fidelity 3D mesh reconstructions support view-specific appearance through a canonical atlas.The dataset is described as complete, dense, natural, diverse, and fine.Source: §1 · p. 2

2. Related Work

Prior datasets address gaze, face, hand, body, or garment modeling with varied capture settings, annotations, and representations. HUMBI instead spans total-body geometry and appearance across many subjects using a dense multiview camera array.

Existing datasets

  • Existing human-expression datasets commonly focus on individual modalities, with gaze, face, hand, body, and garment resources differing in scale, capture conditions, and supervision.Examples include controlled or in-the-wild gaze, 3D face models, synthetic and real hand data, multiview body capture, and cloth reconstruction.Source: §2 · p. 2

Body

  • Prior body datasets capture scripted or socially occluded behaviors, while parametric models and image correspondences support dense reconstruction from scans or single images.Source: §2 · p. 2

HUMBI approach

  • HUMBI differs by jointly spanning geometry and appearance of total-body expressions from distinctive subjects with a dense camera array.Its terascale multiview data are intended to support generalization of pose- and view-specific appearance.Source: §2 · p. 3

3. HUMBI

HUMBI is a multiview dataset representing gaze, face, hand, body, and garment expressions from diverse subjects using dense synchronized camera capture and reconstructed 3D representations.

3. HUMBI

  • HUMBI captures 772 distinctive subjects with 107 synchronized HD cameras and includes gaze, face, hand, body, and garment expressions.The subjects span varied gender, age, skin color, clothing, and physical condition, while the camera system combines broad and frontal views.Source: §3 · p. 3, Figure 3 · p. 4

3.1. Gaze

  • Gaze is represented by a unit 3D vector in a moving head coordinate system and by view-specific eye appearance mapped to a canonical atlas.The eye representation includes normalized patches and an appearance map whose viewpoint-dependent variance is especially visible in the iris region.Source: §3.1 · pp. 3–4

3.2. Face

  • Face geometry uses a 3D blend-shape model aligned from 68 facial keypoints and multiview images, with appearance projected onto a canonical atlas.View-dependent variation around salient landmarks such as the eyes, eyebrows, nose, and mouth motivates view-specific appearance modeling.Source: §3.2 · p. 4

3.3. Hand

  • Hand geometry uses the MANO model aligned from 21 keypoints, while a canonical atlas represents viewpoint-dependent hand appearance.Subject-specific hand shape is estimated across frames using reconstructed keypoints and maximum-likelihood shape inference.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

3.4. Body

  • Body reconstruction combines body keypoints with a segmentation-derived occupancy map, then projects multiview pixels onto a canonical body atlas.The occupancy map also provides semantic labels for the head, torso, arms, and legs, while the SMPL model supplies the body mesh.Source: §3.4 · p. 5, Figure 5 · p. 6

3.5. Garment

  • Garment geometry is reconstructed by deforming manually matched cloth topologies from the body mesh and occupancy map, with Laplacian regularization.The garment representations cover multiple shirt and pants styles, and multiview stereo is complementary to the occupancy map.Source: §3.5 · pp. 5–6

4. Evaluation

HUMBI is evaluated for generalizability, diversity, and reconstruction accuracy across gaze, face, hand, body, and garment tasks. Across these evaluations, HUMBI generally generalizes better than limited-view datasets and complements them when combined.

4. Evaluation

  • HUMBI outperforms existing datasets in cross-data human reconstruction and improves performance further when combined with them.The evaluation covers single-view gaze, face, hand, and body reconstruction, plus garment reconstruction accuracy.Source: §4 · p. 6

4.1. Gaze

  • 5.98° average gaze bias and 24.61° average variance show broad, relatively balanced gaze coverage across yaw and pitch.HUMBI has the smallest average bias and second-largest average variance among the compared datasets; Figure 6 visualizes the distributions.Source: §4.1 · p. 6, Figure 6 · p. 6, Table 3 · p. 7

4.2. Face

  • 3DDFA+HUMBI improves face-mesh prediction over 3DDFA by 2.8 pixels and over HUMBI by 4.9 pixels, demonstrating dataset complementarity.The evaluation uses reprojection error in pixels for monocular 3D face mesh prediction.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. Hand

  • HUMBI improves cross-data hand performance by 0.04–0.12 AUC, while ObMan+HUMBI reduces mesh error by 0.3 and 1.7 pixels versus ObMan and HUMBI.Hand keypoint evaluation uses AUC over 0–20 mm PCK error; mesh evaluation compares real and synthetic data.Source: §4.3 · pp. 7–8, Table 5 · p. 7, Table 6 · p. 8

4.4. Body

  • HUMBI exceeds Human3.6M and MPI-INF-3DHP by 0.023 and 0.064 AUC for body pose, while remaining complementary for body reconstruction.Combining HUMBI increases performance by 0.057 and 0.078 AUC, but HUMBI alone is weak for body mesh prediction because of limited pose diversity.Source: §4.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8

4.5. Garment

  • Garment reconstruction remains near-optimal with fewer cameras: density at 90 cameras and accuracy at 60 cameras are approximately equivalent to 107 cameras.Density is measured by multiview-stereo points, while accuracy is closest-point distance to the reconstructed garment surface.Source: §4.5 · p. 8, Figure 8 · p. 8

5. Discussion

HUMBI spans human appearance along both viewpoint and subject diversity. The dataset is intended to support photorealistic appearance modeling and broader multiview reconstruction and recognition benchmarks.

5. Discussion

  • HUMBI targets general appearance modeling by pushing simultaneously toward extensive viewpoint coverage and diverse subjects.The dataset is described as first-of-its-kind in spanning these two extremes.Source: §5 · p. 9
  • HUMBI can support photorealistic rendering for telepresence and serve as a multiview benchmark for generalizable human reconstruction and recognition.These are stated as opportunities enabled by the dataset’s breadth.Source: §5 · p. 9

Appendices

The appendix provides additional details of HUMBI.

Appendices

A. Multi-camera System

HUMBI uses a reconfigurable 107-camera system to capture diverse body expressions and reconstruct gaze, face, hand, body, and garment geometry from synchronized multiview images. Its reconstruction combines keypoint triangulation with parametric models and appearance, occupancy, correspondence, and regularization objectives.

A. Multi-camera System

  • 107 synchronized GoPro cameras capture 772 subjects performing gaze, face, hand, body, and garment motions in a reconfigurable dodecagon stage.The cameras record at 60Hz, with synchronization error up to 15 ms; 69 cameras cover body and cloth while 38 face the frontal hemisphere for face and gaze.Source: A. Multi-camera System · p. 12

B.1. 3D Keypoint Reconstruction

  • 3D keypoints are triangulated from synchronized undistorted images using RANSAC, reprojection-error refinement, geometric constraints, and visibility reasoning.The method uses detected face, hand, and body keypoints, while enforcing constraints such as left-right body symmetry.Source: §B.1 · p. 12

B.2. Gaze

  • Gaze is represented in a moving coordinate system defined from facial keypoints, with gaze direction and head-cylinder geometry projected onto images.The representation includes coordinate axes, an eye region, a face region, and the projected head cylinder.Source: §B.2 · p. 12, Figure 9 · p. 12

B.3. Face

  • Face reconstruction fits a 3D morphable model to multiview keypoints and appearance while estimating shape, expression, texture, reflectance, pose, and illumination.The appearance model combines texture with Lambertian illumination and spherical harmonics, and subject shape is inferred across reconstructed frames.Source: §B.3 · pp. 12–13

B.4. Hand / B.5. Body / B.6. Garment

  • Hand, body, and garment reconstruction use MANO, SMPL, and an in-house garment mesh with fitting objectives that combine measurements and priors or smoothness constraints.Hand fitting regularizes pose and shape; body fitting uses keypoints, occupancy, and a subject-aware shape prior; garment fitting uses correspondence, Chamfer, and Laplacian terms.Source: §B.4 · p. 13, §B.5 · p. 14

C. Training Mesh Prediction Network

The mesh prediction network regresses body-model parameters from a single image by pairing a neural encoder with pretrained face, body, and hand model decoders. Training updates the regression network using reprojection error.

C. Training Mesh Prediction Network

  • A single-image network predicts face, body, or hand model parameters and camera viewpoint using an encoder with pretrained 3DMM, SMPL, or MANO decoders.Only the regression network is newly trained, while the body-model decoders use pretrained weights.Source: C. Training Mesh Prediction Network · p. 14

D.1. Mesh Prediction Results

HUMBI is evaluated for monocular face, hand, and body mesh prediction using a recent CNN, with qualitative results reported across external and HUMBI test settings. The supplied passages identify the evaluation figures but do not state a quantitative winner or comparison.

D.1. Mesh Prediction Results

  • A recent CNN is evaluated for single-view face, hand, and body mesh prediction, with qualitative results shown on HUMBI and external test data.The referenced figures cover face, hand, and body predictions under different dataset combinations, including UP-3D and HUMBI Body.Source: §D.1 · p. 14, Figure 12 · p. 14, Figure 15 · p. 15

D.2. Garment Reconstruction Accuracy

Garment reconstruction accuracy is measured by the 2D Chamfer distance between annotated and reprojected garment boundaries across camera views. Side views have the largest reported silhouette error.

D.2. Garment Reconstruction Accuracy

  • 11 pixels side-view silhouette error exceeds the 7.5-pixel frontal and 8-pixel rear errors for representative top and bottom garments.Errors are measured using the Chamfer distance between annotated and reprojected garment boundaries across camera views.Source: §D.2 · p. 15, Figure 16 · p. 15

Related papers

Loading 1812.00281v3…