Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation

Chengde Wan; Thomas Probst; Luc Van Gool; Angela Yao

Source-linked AI summary

Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation

Chengde Wan, Thomas Probst, Luc Van Gool, Angela Yao

arXiv:1702.03431v2Feb 11, 2017cs.CV

TL;DR

3D hand-pose estimation from depth images depends on large annotated datasets, while realistic unlabeled depth data is easier to collect. The paper crosses a pose VAE and depth GAN through a shared latent space and jointly trains generator and discriminator networks. Evaluations on three benchmarks report comparable or better state-of-the-art accuracy, with CPU execution at about 90 FPS.

Problem
3D hand-pose estimation is challenging and current deep-learning methods rely heavily on large annotated depth-image corpora, although real unlabeled depth maps are easier to collect.
Method
A VAE models hand poses and a GAN models corresponding depth maps in a shared latent space, while a multi-task discriminator learns pose posteriors using labeled, unlabeled, and synthesized samples.
Results
On three challenging benchmarks, the method achieves results comparable to or better than state of the art with a simple architecture.
Takeaways & Limitations
The crossed architecture combines GAN-based generalization with pose constraints learned by the VAE and supports semi-supervised training from unlabeled depth data.
Takeaways & Limitations
The model assumes a one-to-one mapping between a depth map and hand pose while neglecting sensor noise.

Abstract

from arXiv · show

State-of-the-art methods for 3D hand pose estimation from depth images require large amounts of annotated training data. We propose to model the statistical relationships of 3D hand poses and corresponding depth images using two deep generative models with a shared latent space. By design, our architecture allows for learning from unlabeled image data in a semi-supervised manner. Assuming a one-to-one mapping between a pose and a depth map, any given point in the shared latent space can be projected into both a hand pose and a corresponding depth map. Regressing the hand pose can then be done by learning a discriminator to estimate the posterior of the latent pose given some depth maps. To improve generalization and to better exploit unlabeled depth maps, we jointly train a generator and a discriminator. At each iteration, the generator is updated with the back-propagated gradient from the discriminator to synthesize realistic depth maps of the articulated hand, while the discriminator benefits from an augmented training set of synthesized and unlabeled samples. The proposed discriminator network architecture is highly efficient and runs at 90 FPS on the CPU with accuracies comparable or better than state-of-art on 3 publicly available benchmarks.

1. Introduction

The paper addresses data-hungry 3D hand-pose estimation by crossing a VAE for poses with a GAN for depth maps in a shared latent space. Joint multi-task learning uses labeled, unlabeled, and synthesized data, while the method achieves competitive real-time performance.

3D hand-pose estimation from single depth images is difficult because of local self-similarity, self-occlusions, and reliance on large annotated datasets.
Accurate annotation is difficult and time-consuming, while synthetic data can differ from real depth images because natural poses and sensor noise are hard to model.
Unlabeled real-hand depth maps are easy to collect, but semi-supervised hand-pose estimation has received very little attention.
The proposed dual generative model uses a VAE for hand poses and a GAN for corresponding depth maps, linked through a shared latent space.A shared latent point can decode into both a 3D pose and a depth map.
The discriminator jointly learns latent-space distances, real-versus-synthesized depth classification, and hand-pose posterior estimation from shared input features.The generator is updated through discriminator gradients, while the discriminator uses labeled, unlabeled, and synthesized samples.
The method is evaluated on three challenging benchmarks and achieves results comparable to or better than state of the art with a simple architecture.

2. Related Works

Related work spans deep generative modeling, hand-pose estimation, shared latent manifolds, and bidirectional GANs. The paper distinguishes its neural, end-to-end shared-space model from prior methods focused on classification, handcrafted features, or non-parametric latent-variable models.

GANs and VAEs provide efficient approximations for learning tractable generative models of otherwise difficult unlabeled-image distributions.
Prior semi-supervised extensions of VAEs and GANs focused on classification and assumed multimodal latent distributions, unlike continuous hand-pose regression.
Bidirectional GANs enabled posterior estimation, while the proposed network learns the entire latent posterior rather than only a subvector.
Modern hand-pose methods use CNNs for end-to-end discriminative estimation, but typically require many labeled examples and rarely exploit unlabeled depth maps.
Earlier shared-manifold approaches used GPLVMs, whereas this work uses neural generative models trained jointly with posterior estimation end to end.

3. Preliminaries

The paper introduces VAEs and GANs as tractable generative models for unlabeled data, using a VAE for hand-pose configurations and a GAN for depth maps. These models learn low-dimensional representations while supporting reconstruction and realistic synthesis.

VAEs and GANs provide efficient approximations for learning tractable generative models of unlabeled images.
A VAE encodes observations into latent variables and decodes samples while regularizing the latent distribution toward a prior and reconstructing the input.The paper describes a typical Gaussian prior zy ∼ N(0, I) and a loss combining reconstruction error with KL divergence.
The VAE models hand-pose configurations in a low-dimensional space while maintaining high reconstruction accuracy and simplifying shared-space learning.
A GAN generator maps random noise to depth-map samples, while its discriminator distinguishes real samples from synthesized ones.
GAN training uses discriminator gradients rather than explicit pixel-wise reconstruction loss, enabling realistic depth synthesis and latent representations with linear semantics.

4. Method

The method links pose and depth generative models through a shared latent space, then jointly trains posterior, smoothness, and GAN tasks using labeled, synthesized, and unlabeled depth maps. This design updates the generator and discriminator together to improve use of available data and latent-space regularity.

System overview: The framework combines pose and depth generative networks, pre-trains them separately, learns a latent-space mapping, and then fine-tunes the complete network end-to-end.
System overview: The render route maps a pose through the VAE and GAN to synthesize a corresponding depth map, while other routes estimate the posterior and enforce generator smoothness.
System overview: Under a one-to-one pose–depth assumption, either latent space can serve as a shared reference for linking the two generative models.
Learning the shared latent space: The latent mapping uses corresponding observed and synthesized depth images to define a proxy reconstruction loss despite lacking paired latent vectors.The mapping is learned as zy = Ali(zx), and the rendered depth map is compared with the observed depth image.
Learning the shared latent space: The aligned 23-dimensional latent spaces allow the mapping to be fitted and realistic samples generated from few labeled pose–depth pairs.
Learning the posterior: The posterior learner uses labeled, synthesized, and unlabeled depth maps, with shared convolutional layers supporting posterior, smoothness, and GAN tasks.
Learning the posterior: The smoothness task estimates latent differences between depth maps and regularizes both discriminator and generator using synthesized samples and labeled reconstructions.
Learning the posterior: Generator and discriminator parameters are updated together during multi-task training, while the generator loss and discriminator loss are jointly defined.

5. Experiments

Experiments on three public datasets evaluate semi-supervised learning, multi-task learning, and comparisons with state-of-the-art methods. The approach consistently outperforms the evaluated baselines while achieving real-time CPU inference.

Experimental setup: Experiments use NYU, MSRA, and ICVL, which differ in noise, pose diversity, viewpoint changes, and train-test motion patterns.NYU evaluates 14 of 36 annotated joints for fair comparison.
Experimental setup: 90.9 FPS is achieved on an Intel 3.40 GHz i7 CPU, with mean joint error and threshold-based frame accuracy as evaluation metrics.The reported runtime is 11 ms per image.
Semi-supervised learning: At m=2% labeled data, both the GAN-pretrained baseline and the semi-supervised method outperform training from scratch.The GAN-pretrained baseline loses its advantage over training from scratch when m ≥5%.
Semi-supervised learning: The proposed method consistently outperforms both posterior-estimation baselines, indicating benefits from synthesized and unlabeled samples for generalization.Increasing labeled data does not monotonically reduce baseline error, partly because correlated frames add limited information and may increase overfitting.
Contribution of multi-task learning: Multi-task learning outperforms direct posterior estimation in both semi-supervised and fully supervised settings.Removing either the smoothness loss L_smo or GAN loss L_gan reduces performance relative to the full approach.
Comparison with state-of-the-art: Compared with six state-of-the-art methods, the approach is generally comparable or better, with performance varying by dataset and error threshold.It outperforms prior methods by a large margin on NYU, exceeds hierarchical methods above 35 mm on MSRA, and is 2% better than ICVL refinement at thresholds ≥30 mm.

6. Conclusion

The paper estimates hand pose through the shared latent space of depth maps and hand poses using crossed VAE and GAN models. Experiments on three public datasets show improved performance over previous methods while enabling semi-supervised learning from unlabeled data.

Conclusion: The method estimates the posterior of a shared latent space linking depth maps and hand pose parameters.A VAE models hand poses, while a GAN models depth-image distributions.
Conclusion: The crossed generative architecture combines GAN generalization with pose constraints learned by the VAE for discriminative pose estimation.The complete network is trained end-to-end through a mapping between the two latent spaces.
Conclusion: The architecture naturally supports learning from unlabeled data and extends semi-supervised GANs to real-valued structured prediction.This addresses sparse annotation in hand pose estimation.
Conclusion: Evaluation on three publicly available datasets demonstrates consistently better performance than previous state-of-the-art methods.The conclusion attributes the advantages to the GAN and VAE components together with semi-supervised learning.