Source-linked AI summary
Harvesting Multiple Views for Marker-less 3D Human Pose Annotations
Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, Kostas Daniilidis
TL;DR
Human-pose ConvNets need abundant labels, but 3D groundtruth is difficult to collect beyond restrictive settings. The paper harvests annotations by combining generic per-view 2D predictions with calibrated multi-view geometry and skeletal structure. The resulting labels support personalization and single-view 3D training without 3D groundtruth, while the multi-view estimator achieves state-of-the-art benchmark results.
Problem
Human-pose learning requires abundant labeled data, while reliable 3D groundtruth is difficult to collect outside restrictive motion-capture settings.
Method
The method combines generic ConvNet 2D heatmaps from calibrated views with 3D camera geometry and human-body structure to harvest reliable 3D annotations.
Results
The harvested annotations personalize a 2D pose ConvNet, train a single-view 3D ConvNet from scratch without 3D groundtruth, and support state-of-the-art multi-view results.
Takeaways & Limitations
Automatically harvested annotations provide abundant labels for pose-related learning tasks while exploiting multi-view information.
Takeaways & Limitations
Training uses no dataset groundtruth but assumes a calibrated multi-view camera setup and skeletal articulation constraints.
Abstract
from arXiv · showhide
Recent advances with Convolutional Networks (ConvNets) have shifted the bottleneck for many computer vision tasks to annotated data collection. In this paper, we present a geometry-driven approach to automatically collect annotations for human pose prediction tasks. Starting from a generic ConvNet for 2D human pose, and assuming a multi-view setup, we describe an automatic way to collect accurate 3D human pose annotations. We capitalize on constraints offered by the 3D geometry of the camera setup and the 3D structure of the human body to probabilistically combine per view 2D ConvNet predictions into a globally optimal 3D pose. This 3D pose is used as the basis for harvesting annotations. The benefit of the annotations produced automatically with our approach is demonstrated in two challenging settings: (i) fine-tuning a generic ConvNet-based 2D pose predictor to capture the discriminative aspects of a subject's appearance (i.e.,"personalization"), and (ii) training a ConvNet from scratch for single view 3D human pose prediction without leveraging 3D pose groundtruth. The proposed multi-view pose estimator achieves state-of-the-art results on standard benchmarks, demonstrating the effectiveness of our method in exploiting the available multi-view information.
1. Introduction
The paper addresses scarce human-pose annotations with a geometry-driven method that harvests reliable 2D and 3D labels from calibrated multi-view imagery. It combines generic 2D ConvNet predictions with camera and skeletal constraints, then demonstrates the annotations in personalization, single-view 3D training, and multi-view estimation.
- Abundant labeled data is often unavailable, motivating automatic annotation collection for human pose estimation.
- 3D pose annotation is difficult to gather outside restrictive MoCap settings, limiting variation in users, clothing, and environments.
- A generic 2D pose ConvNet produces per-view confidence heatmaps that are backprojected into a common discretized 3D space.
- A 3D pictorial-structures model combines multi-view evidence with tree-structured skeletal relations and uses marginalized joint posteriors to identify reliable annotations.
- The harvested labels personalize a pretrained 2D pose ConvNet and train a single-view 3D pose ConvNet from scratch without available 3D groundtruth.
- The multi-view pose estimator achieves state-of-the-art results on standard benchmarks, supporting its use of available multiview information.
2. Related work
Related work addresses pose-data scarcity through synthetic or cross-source data, personalization through subject-specific adaptation, and pose estimation through ConvNet and pictorial-structure models. This paper instead harvests annotations from multi-view imagery while retaining image statistics similar to those found in the wild.
- Prior data-scarcity methods generate synthetic examples, combine body parts, or integrate annotations and motion-capture data from different sources.
- The paper's applications include personalized 2D pose prediction and single-view 3D pose prediction trained from harvested annotations.
- Unlike synthetic-data or missing-data workarounds, this approach gathers automatically annotated images whose statistics resemble in-the-wild data.
- Existing 2D pose systems evolved from appearance models with structural constraints toward end-to-end ConvNets that regress coordinates or heatmaps.
- Single-view 3D pose work combines image-based discriminative methods with expressive 3D priors, including coordinate, autoencoder, heatmap, and nearest-neighbor approaches.
- Multi-view 3D pose methods extend pictorial structures to calibrated viewpoints, with earlier approaches combining cross-view 2D reasoning and triangulation.
3. Technical approach
The approach combines generic 2D ConvNet heatmaps with calibrated multi-view geometry and human-skeleton constraints to infer 3D poses and select reliable annotations. These annotations support subject-specific 2D adaptation and single-view 3D pose training.
- 3.1. Generic ConvNet: A generic stacked-hourglass ConvNet supplies noisy single-view 2D human-pose predictions as joint heatmaps.The architecture is treated as interchangeable in principle, although the paper adopts stacked hourglasses with intermediate supervision.
- 3.2. Multi-view optimization: The method backprojects heatmap evidence from calibrated cameras into a common discretized 3D space for multi-view inference.The 3D solution space is a 64 × 64 × 64 grid within a bounding volume around the subject.
- 3.2. Multi-view optimization: A 3D pictorial-structures model combines camera-based likelihoods with tree-structured skeletal relations and infers joint marginals using sum-product dynamic programming.The model represents each joint directly in R3, while pairwise terms constrain limb lengths within tolerance ε = 1.
- 3.3. Annotation selection: The mean of each joint’s marginal distribution yields a pose estimate, while marginal uncertainty provides a cue for selecting reliable annotations.The determinant of a joint’s marginalized 3D covariance matrix can serve as a confidence measure.
- 3.4. Personalization: Selected 3D annotations are projected into images to fine-tune a generic 2D ConvNet for subject-specific appearance adaptation.Training targets are per-joint 2D Gaussian heatmaps centered on annotation locations, with σ = 1 pixel.
- 3.5. 3D pose ConvNet training: The harvested annotations also supervise a ConvNet trained from scratch to predict single-view 3D joint coordinates.The output is a 3N-dimensional vector containing x, y pixel coordinates and z metric depth relative to the pelvis, trained with L2 regression.
4. Empirical evaluation
The evaluation tests the approach on multiview pose estimation, 2D personalization, and single-view 3D training across KTH Multiview Football II and Human3.6M. Results show strong multiview accuracy, improved personalized 2D predictions, and state-of-the-art-level 3D performance using automatically harvested annotations.
- Datasets: The evaluation uses KTH Multiview Football II and Human3.6M, covering challenging outdoor and large-scale indoor multiview pose settings.KTH evaluation uses PCP for 3D part localization, while Human3.6M reports average 3D joint error.
- Implementation: No groundtruth from either evaluation dataset is used for training; experiments rely on a generic MPII-trained 2D ConvNet and calibrated camera geometry.The 3D pose ConvNet is trained from scratch rather than using a pretrained model.
- Multi-view pose estimation: On KTH, the proposed method outperforms methods using specialized 2D detectors while using only a generic 2D ConvNet.Table 1 reports 3D PCP with two- and three-camera setups.
- Multi-view pose estimation: On Human3.6M, the approach nearly halves the error of Zhou et al.'s state-of-the-art single-view method, while naive per-view averaging yields only minimal improvement.The naive combination reaches 103.10mm versus 113.01mm for the corresponding single-view approach; Zhou et al. [47] use video.
- “Personalizing” 2D pose ConvNet: Fine-tuning improves 2D PCP across all KTH body parts, with larger gains on the more challenging second sequence.The refinement is especially beneficial when the original detector fails.
- “Personalizing” 2D pose ConvNet: On Human3.6M, selected high-confidence multiview annotations outperform using all annotations, while heatmap-based bootstrapping is only marginally helpful on average.Benefits are greater on the most challenging actions: Purchases, Sitting, and Sitting Down.
- Training a 3D pose ConvNet: A ConvNet trained from scratch on harvested annotations achieves performance on par with the state-of-the-art despite ignoring available 3D groundtruth.The experiment focuses on Human3.6M because KTH is small-scale.
5. Summary
The paper presents automatic 3D annotation harvesting from calibrated multi-view imagery and demonstrates its use for personalization and single-view 3D pose learning. The approach performs on par with current state-of-the-art while using automatically harvested annotations instead of provided groundtruth.
- Automatically generated annotations improve adaptation of a generic 2D pose ConvNet to subject-specific appearance.The paper demonstrates this personalization on KTH Multiview Football II and Human3.6M.
- A ConvNet trained for single-view 3D pose estimation performs on par with current state-of-the-art using only harvested annotations.The training ignores the datasets' provided 3D groundtruth.
- The annotation setup is proposed for outdoor environments where MoCap systems and depth sensors are not applicable.The intended use is collecting 3D annotations for in-the-wild images.