PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-based 3D Hand Reconstruction via Self-Supervised Learning

Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, Junsong Yuan

arXiv:2103.11703v1cs.CV

TL;DR

Single-view 3D hand reconstruction is challenging because of depth ambiguity, and existing methods often depend on costly 3D annotations. S2HAND learns pose, shape, texture, and viewpoint from images with noisy detected 2D keypoints, reporting performance comparable to fully supervised methods. The authors also report lower accuracy for extreme poses, severe occlusion, and extreme viewpoints.

  • Problem

    Single-view hand reconstruction is difficult because of varied configurations and depth ambiguity, while 3D annotations used by many methods are expensive to obtain.

    Source: §1 · pp. 1–2

  • Method

    S2HAND estimates pose, shape, texture, and camera viewpoint using image information and detected 2D keypoints, with a 2D–3D consistency loss linking the representations.

    Source: §1 · p. 2, §3.4.2 · p. 6

  • Results

    The self-supervised method achieves comparable performance with recent fully supervised methods while using fewer supervision data.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The experiments demonstrate the feasibility of training a 3D hand reconstruction network without manual annotations, using outputs from a 2D keypoint detector.

    Source: §1 · p. 2

    Accuracy is lower for extreme poses, severe occlusion, and extreme viewpoints, partly because of poor single-view 2D keypoint supervision; texture modeling is inaccurate under complex skin reflections.

    Source: §5 · p. 8

Abstract

from arXiv · show

Reconstructing a 3D hand from a single-view RGB image is challenging due to various hand configurations and depth ambiguity. To reliably reconstruct a 3D hand from a monocular image, most state-of-the-art methods heavily rely on 3D annotations at the training stage, but obtaining 3D annotations is expensive. To alleviate reliance on labeled training data, we propose S2HAND, a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. Specifically, we obtain geometric cues from the input image through easily accessible 2D detected keypoints. To learn an accurate hand reconstruction model from these noisy geometric cues, we utilize the consistency between 2D and 3D representations and propose a set of novel losses to rationalize outputs of the neural network. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations. Our experiments show that the proposed method achieves comparable performance with recent fully-supervised methods while using fewer supervision data.

1. Introduction

Single-view 3D hand reconstruction is difficult because hand configurations vary and depth is ambiguous, while 3D training annotations are costly. The paper proposes a self-supervised approach using image information and noisy detected 2D keypoints, with reported performance comparable to fully supervised methods.

  • The approach uses geometric cues from unlabeled images and image texture to reduce reliance on costly manual annotations for shape and texture modeling.Source: §1 · p. 2
  • Aligning projected 3D joints with detected 2D keypoints alone can yield invalid poses and cannot resolve scale ambiguity, so the model embeds pose and size priors.Source: §1 · p. 2
  • A trainable 2D keypoint estimator and 2D–3D consistency loss link the 2D and 3D branches, while detection items receive different supervision weights.Source: §1 · p. 2
  • S2HAND predicts 3D pose, shape, texture, and camera viewpoint without ground-truth annotations, using outputs from a 2D keypoint detector.Source: §1 · p. 2
  • The self-supervised method achieves comparable performance to previous fully supervised methods on challenging datasets.Source: §1 · p. 2

2. Related Work

Related work spans parametric model-based recovery, methods using limited supervision, and self-supervised reconstruction. The paper distinguishes its approach by training hand recovery without ground-truth annotations or extra 2.5D or 3D data.

  • Prior model-based methods use parametric hand representations, but many rely on 3D annotations; this work instead demonstrates an annotation-free framework.Source: §2 · pp. 2–3
  • Limited-supervision approaches use depth, biomechanical constraints, 2D fitting, or automatically collected meshes, whereas this method tests noisy 2D supervision without extra 2.5D or 3D data.Source: §2 · p. 3
  • Earlier self-supervised reconstruction methods assume symmetry, use 2D silhouettes for restricted categories, rely on depth, or address face reconstruction rather than hands.Source: §2 · p. 3
  • Prior hand texture work uses shading cues or PCA from textured scans, while this paper models texture through self-supervision without extra data.Source: §2 · p. 3
  • The authors present their network as the first self-supervised 3D hand reconstruction method in the field [27].Source: §2 · p. 4

3. Method

The method separately encodes hand geometry and texture, decodes a textured 3D hand, and trains the model with image-space, photometric, regularization, and 2D–3D consistency losses.

3. Method

  • An end-to-end autoencoder maps a hand image to pose, shape, texture, and camera viewpoint, while image-space representations and losses train the reconstruction.The method generates multiple 2D representations for its training objectives.Source: §3 · p. 4

3.1. Deep Hand Encoding

  • An EfficientNet-b0 backbone encodes the input into geometry parameters for pose, shape, scale, rotation, and translation, and texture parameters for vertex color and lighting.The geometry and texture codes are denoted x and y, respectively.Source: §3.1 · p. 4

3.2. Model-based Hand Decoding

  • MANO recovers a 778-vertex mesh and 21 joints from pose and shape parameters, then estimated scale, rotation, and translation place them in camera coordinates.The hand texture is represented by per-vertex RGB values, with ambient and directional lighting.Source: §3.2 · p. 4

3.3. Represent Hand in 2D

  • The method represents hands through projected 3D joints, a rendered silhouette and color image, and independently estimated 2D joints.A stacked hourglass network with integral pose regression estimates the 2D joints; this branch is optional at inference.Source: §3.3 · p. 4

3.4. Training Objective

  • Training combines 3D-branch, 2D-branch, and 2D–3D consistency losses, with the latter two used when the 2D estimator is applied.Weights balance the three loss terms.Source: §3.4 · pp. 4–5

3.4.1 Losses of the 3D Branch

  • The 3D branch combines detected-keypoint location and bone-orientation alignment, rendered-image pixel and SSIM consistency, and shape, texture, scale, and feasible bone-angle regularizers.The scale regularizer constrains hand size to help address monocular scale ambiguity.Source: §3.4.1 · p. 5, Figure 3 · p. 5

3.4.2 2D-3D Consistency

  • The 2D–3D consistency loss links projected and estimated 2D joints, letting 3D structure inform the 2D estimator while estimated joints provide cues for 3D reconstruction.The 2D estimator is optional during inference.Source: §3.3 · p. 4, §3.4.2 · p. 6

4. Experiments

The experiments benchmark S2HAND on FreiHAND and HO-3D, then analyze its performance against prior methods and under different losses and supervision choices.

4.1. Datasets and evaluation metrics

  • Evaluation uses FreiHAND and HO-3D, measuring 3D joints and meshes with position-error, AUC, and mesh F-score metrics.The datasets contain real images; HO-3D features hands interacting with objects and larger occlusions than FreiHAND.Source: §4.1 · p. 6

4.2. Implementation

  • Training uses a pretrained EfficientNet-b0 and Adam, with separate branch training followed by whole-network fine-tuning when the 2D estimator is used.Training on FreiHAND takes around 36 hours on two NVIDIA Tesla V100 GPUs.Source: §4.2 · p. 6

4.3. Comparison with State-of-the-art Methods

  • On FreiHAND, annotation-free S2HAND outperforms on all metrics, matches comparably, and uses less supervision than.[37] is slightly better on pose but uses RHD synthetic 3D annotations and FreiHAND 2D ground truth.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4.2 Comparison of Different 2D Keypoint Sets

  • 2D-3D consistency fine-tuning reduces 2D MPJPE by 5.4% for the 2D branch and 9.3% for the projected 3D branch.The curves indicate improvement for the 3D branch across all distance thresholds.Source: Figure 4 · p. 7, §4.4.2 · pp. 7–8

4.3. Comparison with State-of-the-art Methods

  • On HO-3D, S2HAND reduces MPVPE by 14% versus 's hand branch and obtains comparable joint and shape estimates to.[18] reports slightly better shape estimates, potentially due to multi-frame refinement and mesh supervision.Source: Table 3 · p. 7, §4.3 · p. 7

4.4.3 Comparison with GT 2D Supervision

  • Self-supervised training outperforms weakly supervised ground-truth 2D supervision on both datasets, with gains of 4.9% on FreiHAND and 1.0% on HO-3D.The authors attribute the smaller HO-3D gain to greater occlusion and poorer 2D detections, and note that unlabeled image quality matters.Source: Table 5 · p. 8, §4.4.3 · p. 8

4.4.1 Effect of Each Component

  • Adding Eori reduces MPJPE and MPVPE by 19.5%; adding the 2D branch and 2D-3D consistency reduces them by a further 4%.Ephoto slightly improves pose and shape estimation.Source: §4.4.1 · p. 7, Table 4 · p. 8

5. Discussion

The method has lower accuracy in extreme poses, severe occlusion, and extreme viewpoints, partly because single-view 2D keypoint detections provide poor supervision; complex skin reflections are also modeled inaccurately.

5. Discussion

  • Accuracy drops in extreme poses, severe occlusion, and extreme viewpoints, partly due to poor supervision from single-view 2D keypoint detection.The authors also report inaccurate texture modeling under complex skin reflections, possibly because those reflections are difficult to simulate with a simple illumination representation and coarse hand mesh.Source: §5 · p. 8
  • Figure 6 presents failure cases alongside the input image, projected joints, and 3D reconstruction.Source: Figure 6 · p. 8
  • Complex skin reflections are modeled inaccurately with a simple illumination representation and coarse hand mesh.The authors also report that texture modeling does not markedly improve shape reconstruction, possibly because the hand model is not meticulous enough and skin-reflection simulation is inaccurate.Source: §5 · p. 8

6. Conclusion

The conclusion presents a self-supervised network that learns hand pose, shape, texture, illumination, and camera viewpoint from images and outputs a textured mesh and 3D joints. The authors report performance comparable to state-of-the-art fully supervised methods.

6. Conclusion

  • The network learns pose, shape, texture, illumination, and camera viewpoint from unlabeled hand images, then decodes them into a textured mesh and 3D joints.Source: §6 · p. 8
  • Projecting the predicted mesh and joints into 2D image space enables end-to-end learning.Source: §6 · p. 8
  • The authors report accurate single-view 3D hand reconstruction under noisy 2D keypoint supervision and performance comparable to state-of-the-art fully supervised methods.Source: §6 · p. 8

Appendix

The appendix outlines supplementary details on statistical regularization, implementation and architecture, camera-coordinate comparisons with a fully supervised setting, and additional visualizations.

Appendix

  • Section A provides details on statistical regularization terms.Source: Appendix · p. 11
  • Section B covers implementation and architecture details.Source: Appendix · p. 11
  • Section C compares results in camera coordinates with a fully supervised setting.Source: Appendix · p. 11
  • Section E presents additional visualizations.Source: Appendix · p. 11

A. Statistical Regularization

The appendix regularizes texture, scale, and skeleton plausibility. Its skeleton prior penalizes out-of-range joint rotations rather than all departures from an average pose, which the authors report produces better performance.

Texture Regularization

Scale Regularization

Skeleton Regularization

  • Table 6 lists the minimum and maximum joint-angle values used by the skeleton regularizer.Source: Table 6 · p. 11
  • The proposed skeleton prior penalizes invalid poses according to how far joint rotations exceed their feasible ranges, rather than penalizing every departure from the average pose.The authors report better performance than the average pose prior, which can yield reasonable projected joints despite invalid 3D hand configurations.Source: A. Statistical Regularization · p. 11

B. Implementation and Architecture Details

The implementation uses an encoder-decoder network with separate heads for hand geometry and appearance. The appendix also specifies training loss weights and dataset-specific image preparation.

B. Implementation and Architecture Details

  • For HO-3D, training crops use 2D keypoints and testing crops use provided bounding boxes; FreiHAND training uses no augmentation.The appendix also lists the loss weighting factors used in the experiments.Source: B. Implementation and Architecture… · p. 12
  • EfficientNet-b0 encodes each 224 × 224 image, and separate heads estimate pose, shape, translation, rotation, and scale.The network also uses intermediate, higher-resolution features to predict texture and scene lighting.Source: B. Implementation and Architecture… · p. 12
  • Higher-resolution features feed convolutional layers and heads that estimate hand texture and scene lighting.The architecture uses an intermediate feature map of size 56 × 56 × 32 for this branch.Source: B. Implementation and Architecture… · p. 12

C. Comparison to using GT 3D as Supervision

The comparison evaluates raw camera-coordinate outputs without Procrustes alignment, contrasting self-supervised and fully supervised training. On FreiHAND, S2HAND outperforms MANO-CNN under both settings, with a smaller reported degradation from full to self-supervision.

C. Comparison to using GT 3D as Supervision

  • The evaluation reports raw camera-coordinate outputs without Procrustes alignment and does not use the testing sets’ provided scale or depth information.The FSL comparison adds a 3D joint loss with weight 100, alongside ground-truth 2D keypoints with equal confidences.Source: C. Comparison to using GT 3D as… · p. 12
  • MPJPE is 0.06cm lower for S2HAND than MANO-CNN under FSL and 2.18cm lower under SSL on FreiHAND.The comparison uses unaligned results on the FreiHAND testing set.Source: Table 9 · p. 12, C. Comparison to using GT 3D as… · p. 12
  • MPJPE increases 22.1% for S2HAND and 46.2% for MANO-CNN when supervision drops from FSL to SSL.The authors describe their approach as more robust to the self-supervised setting.Source: C. Comparison to using GT 3D as… · p. 12

D. More Comparison of Different 2D Keypoint Sets

The appendix compares 2D keypoint predictions and projections using per-joint and per-frame error measures. It reports that 2D-3D consistency improves AUC in both branches, while relative performance depends on the error measure.

D. More Comparison of Different 2D Keypoint Sets

  • Per-frame-maximum error measures the fraction of frames within a maximum 2D MPJPE threshold, making it more sensitive to outliers.The appendix visualizes this measure in Figure 8 and reports AUCs for the three 2D error measures in Table 10.Source: D. More Comparison of Different 2D… · p. 13
  • 2D-3D consistency improves AUC for both the predicted 2D branch and the projected 3D branch, and both outperform OpenPose.AUC is evaluated over 0 to 50 pixels.Source: D. More Comparison of Different 2D… · p. 13
  • Predicted 2D results outperform projected 3D results on per-joint and per-frame-mean error, while projections perform better on per-frame-maximum error.The authors attribute the latter pattern to MANO constraints eliminating 2D outliers.Source: D. More Comparison of Different 2D… · p. 13

E. Qualitative Results

Qualitative visualizations show projected joints and meshes alongside meshes from multiple views on FreiHAND and HO-3D. HO-3D occlusion is associated with less accurate texture estimates, while sequence examples show consistent shape and texture despite single-image inference.

E. Qualitative Results

Related papers

This paper cites

Loading 2103.11703v1…