PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal Deep Variational Hand Pose Estimation

Adrian Spurr, Jie Song, Seonwook Park, Otmar Hilliges

arXiv:1803.11404v1cs.CV

TL;DR

Estimating 3D hand pose from images is difficult because hand articulation and image ambiguities complicate monocular recovery. The paper extends the VAE framework to learn a unified cross-modal latent space, which outperforms state of the art on RGB tasks, matches specialized depth methods, and supports consistent multimodal generation.

  • Problem

    Full 3D hand-pose estimation from monocular RGB images is challenging because of dexterity, symmetries, occlusions, lighting variation, self-similarity, and inaccurate scale.

  • Method

    The method re-derives the VAE variational lower bound to train encoder-decoder pairs across modalities in a single cross-modal latent space.

  • Results

    The approach outperforms state-of-the-art methods on RGB datasets, is comparable to specialized depth methods, and generates physically plausible multimodal hand configurations.

  • Takeaways & Limitations

    The learned latent space enables direct 3D joint-posterior estimation and consistent generation across RGB, 2D, and 3D modalities.

  • Takeaways & Limitations

    Generated images are legible but blurry, while producing high-quality natural images remains a separate research topic.

Abstract

from arXiv · show

The human hand moves in complex and high-dimensional ways, making estimation of 3D hand pose configurations from images alone a challenging task. In this work we propose a method to learn a statistical hand model represented by a cross-modal trained latent space via a generative deep neural network. We derive an objective function from the variational lower bound of the VAE framework and jointly optimize the resulting cross-modal KL-divergence and the posterior reconstruction objective, naturally admitting a training regime that leads to a coherent latent space across multiple modalities such as RGB images, 2D keypoint detections or 3D hand configurations. Additionally, it grants a straightforward way of using semi-supervision. This latent space can be directly used to estimate 3D hand poses from RGB images, outperforming the state-of-the art in different settings. Furthermore, we show that our proposed method can be used without changes on depth images and performs comparably to specialized methods. Finally, the model is fully generative and can synthesize consistent pairs of hand configurations across modalities. We evaluate our method on both RGB and depth datasets and analyze the latent space qualitatively.

1. Introduction

Monocular RGB hand-pose estimation is difficult because hand articulation, occlusion, lighting, scale, and self-similarity make full 3D recovery challenging. The paper addresses this with a unified cross-modal latent space that supports estimation and generation across modalities.

  • Full 3D hand-pose estimation from monocular RGB images is challenging because of dexterity, symmetries, occlusions, lighting variation, and inaccurate scale.
  • Prior work includes depth-based tracking and RGB methods that detect hands and 2D keypoints before lifting them into 3D without an explicit hand model.
  • Bio-mechanical evidence motivates a lower-dimensional valid-pose subspace because many hand degrees of freedom are not independently articulated.
  • The paper learns a single unified latent space by extending the VAE framework to train across multiple modalities, placing similar poses near one another regardless of input modality.
  • The approach outperforms the state-of-the-art in RGB-to-3D estimation and 2D-to-3D lifting, while also supporting many-to-many mappings and consistent multimodal synthesis.

2. Related Work

Related work spans tracking, per-frame classification, optimization-based fitting, and deep generative models for hand pose. The paper distinguishes its approach through theoretically grounded cross-modal training of one latent space rather than separate modality-specific spaces.

  • Hand-pose approaches include temporal tracking, per-frame classification, discriminative regression, and hybrid initialization followed by model fitting and tracking.
  • Wan et al. learn separate latent spaces for 3D joints and depth maps, then require a mapping function between them.
  • This paper instead derives a VAE-based cross-modal training scheme that jointly learns a single latent space across modalities.
  • The same architecture can be applied to depth images without modification, reaching parity with state-of-the-art depth-based methods.
  • For RGB images, the method supports either RGB or 2D-keypoint inputs and directly decodes 3D hand poses, outperforming the cited prior method in both settings.

3. Method

The method learns a smooth, continuous cross-modal latent space for hand poses using a VAE-derived objective, allowing multiple input modalities to reconstruct hand configurations in the same or different modalities. This shared space supports 3D pose estimation, cross-modal generation, and unsupervised or semi-supervised training.

  • Motivation: Natural hand motion occupies a low-dimensional subspace, motivating a latent representation that is smooth, continuous, and consistent for 3D pose estimation.The method uses data-driven structure rather than relying only on direct geometric or physical constraints.
  • Cross-modal architecture: Encoders map 2D keypoints, RGB, or depth images into a low-dimensional latent space representing physically plausible hand poses, while decoders reconstruct any selected modality.The architecture focuses on estimating the 3D joint posterior but remains fully generative across modalities.
  • Generated samples: The learned manifold is smooth and continuous, supporting generated cross-modal samples such as novel pairs of 3D poses and natural-hand images.Generated images are reported as legible but blurry, making high-quality natural-image synthesis a remaining limitation.
  • Objective: The cross-modal training objective extends the VAE variational lower bound by combining posterior reconstruction with KL regularization for a shared latent space.The encoder produces Gaussian mean µ and variance σ2, samples z, and the decoder reconstructs the target representation.
  • Objective: Input and target modalities can be decoupled through the shared embedding z, enabling RGB-to-3D training and simultaneous same-modality reconstruction.For example, q(z|xRGB) encodes an RGB image while p(x3D|z) reconstructs a 3D joint configuration.
  • Training procedure: Iteratively training encoder-decoder pairs keeps every modality in the same embedding space and permits multiple cross-modal reconstruction pathways.The RGB-and-3D example uses separate encoders and decoders while preserving one shared latent representation.

4. Experiments

The experiments evaluate cross-modal VAE variants for 3D hand pose estimation from 2D keypoints, RGB, and depth across public datasets. Results show strong RGB and lifting performance, comparable depth performance, effective semi-supervision, and coherent multimodal generation.

  • Experimental setup: The model is evaluated on 2D keypoints, RGB images, and depth images, always predicting at least the 3D hand configuration.Experiments use combinations of modalities during training and assess the resulting cross-modal latent space.
  • Experimental setup: Experiments use STB and RHD for RGB and keypoint inputs, and ICVL, NYU, and MSRA for depth inputs.STB contains 18k images, while RHD provides 42k training and 2.7k evaluation images with varied viewpoints and poses.
  • Variant comparison: Variant 3 performs best for RHD 2D-to-3D lifting and STB RGB-to-3D regression, while Variant 1 is best for RHD RGB-to-3D.Individual performance differences are generally minor, and shared latent-space training does not harm performance.
  • Depth-to-3D estimation: On ICVL, NYU, and MSRA, the RGB-designed architecture performs comparably to recent depth-specific methods after changing only the input to one channel.The comparison uses PCF curves for 3D joint estimation from depth input.
  • Semi-supervised learning: With increasing unlabeled data, the semi-supervised variant improves prediction accuracy by up to 22% on STB.The experiment compares supervised Variant 1 with semi-supervised Variant 3 using median EPE.
  • Generative capabilities: Interpolated latent-space samples decode into physically plausible, modality-consistent hand poses with fingers moving synchronously.The latent-space walk uses two real RGB images as endpoints and synthetic multimodal reconstructions between them.
  • Generative capabilities: Generating labeled data from the smooth latent space is identified as a potential benefit, but fully exploring this use remains future work.The paper does not fully evaluate this data-generation application.

5. Conclusion

The approach estimates 3D hand poses from RGB and depth images using a cross-modal latent space derived from the variational lower bound. Its generative formulation also produces consistent hand configurations across modalities and supports direct estimation of the 3D joint posterior.

  • 5. Conclusion: The method trains multiple encoder-decoder pairs to shape a joint cross-modal latent representation for 3D hand pose estimation.The formulation supports RGB and depth inputs while reconstructing configurations across modalities.
  • 5. Conclusion: The approach outperforms state-of-the-art methods on public RGB datasets and is at least comparable to specialized methods on depth data.
  • 5. Conclusion: The fully generative model produces consistent hand configurations across modalities and enables direct estimation of the 3D joint posterior.

7. Supplementary

The supplementary material details implementation and qualitative analyses of the cross-modal latent space. Cross-training yields a coherent multimodal embedding, while interpolation and sampling produce consistent, realistic pose reconstructions across RGB, 2D, and 3D modalities.

  • Implementation: The supplementary experiments use PyTorch, ADAM with learning rate 10^-4, and batch size 64.
  • Implementation: RGB, depth, and 2D-to-3D variants use modality-specific preprocessing or architectures while sharing the cross-modal modeling framework.Depth changes only the input-channel count relative to RGB, while 2D-to-3D uses linear-ReLU layers.
  • Latent space consistency: Cross-trained RGB, 3D-joint, and 2D-joint samples form a single dense, approximately Gaussian t-SNE cluster, unlike separately trained modalities.The comparison identifies cross-training as vital for learning a multimodal latent space.
  • Latent space consistency: Interpolated latent samples decode into aligned RGB images, 2D keypoints, and 3D skeletons, with intermediate triplets being synthetic.Each column-triplet originates from the same latent-space point.
  • Additional figures: The model predicts realistic poses for difficult, self-occluded RHD examples and produces similar qualitative results for depth images.STB poses are simpler and are often predicted very accurately.
  • Influence of model capacity: 8.0 mean EPE is achieved with a ResNet-50 depth encoder, compared with 8.1 for DeepPrior++ and 8.5 for the original architecture.The supplementary comparison attributes the difference primarily to model capacity.
Loading 1803.11404v1…