Source-linked AI summary
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild
Dominik Kulon, Riza Alp Güler, Iasonas Kokkinos, Michael Bronstein, Stefanos Zafeiriou
TL;DR
Monocular 3D hand reconstruction lacks broad in-the-wild RGB supervision, motivating a weakly supervised approach. The paper combines automated YouTube mesh-data generation with an image encoder, mesh-convolutional decoder, and direct reconstruction loss, achieving 50% lower pose error in the in-the-wild scenario. The authors conclude that simple encoder-decoder systems and automated data collection can support strong mesh and pose reconstruction without acquiring extensive 3D ground truth.
Problem
In-the-wild monocular 3D hand reconstruction lacks a substantial RGB dataset with 3D annotations; the only cited non-laboratory collection has fewer than 3,000 manually annotated 2D samples.
Method
The method generates weakly supervised 3D mesh annotations from YouTube videos and trains an image encoder with a mesh-convolutional decoder using a direct mesh reconstruction loss.
Results
50% lower hand pose estimation error is achieved than the currently best model on the in-the-wild scenario.
Takeaways & Limitations
A simple encoder-decoder architecture with automated data collection can obtain superior mesh reconstruction and pose estimation performance while adapting to non-laboratory domains.
Takeaways & Limitations
The pose prior does not model pairwise dependencies between joint angles, which the authors leave to future work.
Abstract
from arXiv · showhide
We introduce a simple and effective network architecture for monocular 3D hand pose estimation consisting of an image encoder followed by a mesh convolutional decoder that is trained through a direct 3D hand mesh reconstruction loss. We train our network by gathering a large-scale dataset of hand action in YouTube videos and use it as a source of weak supervision. Our weakly-supervised mesh convolutions-based system largely outperforms state-of-the-art methods, even halving the errors on the in the wild benchmark. The dataset and additional resources are available at https://arielai.com/mesh_hands.
1. Introduction
The paper targets monocular 3D hand reconstruction in unconstrained environments with weak supervision from YouTube videos. It combines automated mesh data generation, direct mesh losses, and mesh convolutions to outperform prior methods on pose and reconstruction tasks.
- Weak supervision: The authors curate 50,000 training annotations with 3D hand meshes from YouTube videos covering diverse subjects and tasks.The dataset is generated by fitting a hand model to 2D keypoint detections while respecting a rotation-angle prior.
- Results: 50% lower hand pose estimation error is achieved than the currently best model in the in-the-wild scenario.The system also outperforms previous methods in controlled environments without dataset-specific tuning or overfitting.
- Dataset: The released dataset contains 50,000 meshes aligned with frames selected from over a hundred YouTube videos.
- Method: The method uses a simple mesh-reconstruction loss that trains neural networks without intermediate supervision.The end-to-end objective enforces reconstruction of a hand mesh aligned with the image.
- Architecture: Spatial mesh convolutions based on neighborhood ordering outperform spectral methods and SMPL-based models for hand reconstruction.
- Results: The single-shot encoder-decoder architecture largely outperforms state-of-the-art methods on pose estimation and mesh reconstruction tasks.
2. Background
The paper situates its RGB-only hand pose and mesh reconstruction approach among keypoint, deformable-model, and geometric deep-learning methods. Its architectural contribution is an image-to-mesh system using spatial mesh convolutions and image-coordinate mesh losses.
- Task scope: RGB-only hand pose estimation is more challenging than estimation using a depth sensor, while mesh recovery additionally estimates subject-specific shape.
- Hand pose estimation: Prior hand-pose methods commonly regress 3D pose from estimated 2D keypoints, while other approaches use variational latent spaces or spectral graph convolutions.
- Hand mesh recovery: MANO-based hand mesh methods regress rotation, shape, and camera parameters, often from images and heatmaps, to reconstruct deformable hand models.
- Proposed method: The proposed approach computes loss directly on meshes with points localized in image coordinates and leverages spatial mesh convolutions.
- Comparison: The paper reports an architecturally simpler approach that significantly outperforms prior works on mesh reconstruction and pose estimation.
- Geometric deep learning: Spatial convolutions define local charting for mesh learning, whereas spectral approaches express convolutions in the frequency domain.
3. YouTube Hands-in-the-Wild Dataset
The paper addresses the lack of diverse in-the-wild 3D hand data by fitting MANO to OpenPose detections from YouTube videos, producing weakly supervised mesh annotations for training.
- Fewer than 3,000 manually annotated 2D hand samples existed for monocular 3D reconstruction in non-laboratory settings.
- The fitting pipeline lifts OpenPose 2D landmarks into 3D MANO meshes by optimizing reprojection, bone-length, and regularization objectives.
- MANO represents each hand with 778 surface vertices and 16 joints, using shape and pose parameters mapped through a differentiable model.
- Joint angles are constrained to convex combinations of 64 k-means Euler-angle cluster centers, while global orientation remains unrestricted.
- The resulting angle prior restricts poses to plausible configurations and supports unseen poses, but does not model pairwise dependencies between joints.
- The YouTube collection processes videos, extracts frames, runs OpenPose, fits MANO, filters samples by confidence and reprojection error, and selects diverse data.The training set uses 102 videos and at most 500 qualifying samples per video; validation and test subjects do not overlap with training subjects.
4. Hand Reconstruction Network
The reconstruction network encodes a hand crop and directly decodes an image-aligned mesh using spatial spiral convolutions, hierarchical upsampling, and a direct mesh loss.
- The system directly reconstructs a mesh in image coordinates from an image encoding using a spatial convolutional mesh decoder.
- Spiral operator: Spiral neighborhoods impose an ordered clockwise traversal of adjacent vertices, with a fixed spiral length for each neural-network layer.
- Topology hierarchy: The decoder hierarchy halves the vertex count at each stage, using quadric-error vertex contractions and barycentric interpolation for upsampling.
- Architecture: A ResNet-50 encoder maps the cropped hand image to a 64-parameter latent vector, which the spiral decoder maps to the output mesh.
- Training: Training uses an L1 vertex reconstruction term and an edge-length preservation term, with no explicit pose-estimation loss.
- Training: The network is trained with Adam for 150 epochs, using learning-rate decay after epochs 90 and 120 and augmentation with random crops and transformations.
5. Evaluation
The evaluation spans challenging in-the-wild, rendered, controlled, and mesh-annotated benchmarks, showing strong performance across pose estimation and mesh reconstruction. Ablations further examine decoder design, training data, iterative fitting, and inference speed.
- Benchmark results: MPII results halve the Pose Error of the leading MANO-based method and improve AUC by 0.21 points.
- Benchmark results: The system outcompetes existing methods by a significant margin on the RHD 3D pose estimation benchmark.
- Benchmark results: The system achieves superior performance on FreiHAND for 3D pose estimation and mesh reconstruction.
- Decoder ablations: Spatial mesh convolution outperforms the spectral approach, while end-to-end image-coordinate mesh losses outperform canonical-frame pretraining with camera estimation.
- Robustness and efficiency: The network performs much better on the whole MPII dataset than on the filtered high-confidence subset, while inference reaches 60 FPS.On the filtered subset, it performs slightly worse than iterative fitting; a single fitting sample takes 110 seconds on GPU or 70 seconds on CPU.
- Dataset and method ablation: Including the YouTube dataset during training gives the best results across dataset combinations, and the contributed data collection system improves MANO-based performance.
6. Conclusion
The paper concludes that a simple encoder-decoder with an appropriate objective can perform strongly on mesh reconstruction and pose estimation. Automated YouTube data collection supports adaptation to non-laboratory domains and motivates further weakly- and self-supervised human modeling.
- A simple encoder-decoder architecture obtains superior performance on mesh reconstruction and pose estimation with an appropriately defined objective function.
- Automated data collection from YouTube adapts the system to non-laboratory domains and improves results on common benchmarks.
- The findings motivate further investigation of mesh generative models and weakly- and self-supervised methods that reduce the need for 3D ground truth.