Source-linked AI summary
Disentangling Latent Hands for Image Synthesis and Pose Estimation
Linlin Yang, Angela Yao
TL;DR
RGB hand-pose estimation and image synthesis are challenging because appearance varies with factors such as pose and background, while RGB data can be ambiguous. The paper proposes a disentangled VAE that separates these factors and enables controlled image synthesis and pose estimation. It reports realistic, controllable synthesis and pose-estimation accuracy competitive with state of the art.
Problem
RGB hand images vary with factors such as pose, viewpoint, and background, while projection and diverse backgrounds make pose estimation ambiguous and accurately labeled real datasets are small.
Source: §1 · pp. 1–2Method
The dVAE learns a disentangled cross-modal latent representation with factors associated with observed variables, enabling explicit control over pose and image content.
Source: §1 · p. 2, §3.2 · p. 3Results
The dVAE synthesizes highly realistic hand images with control over pose and background content, and its RGB pose estimation is competitive with state of the art.
Source: §5 · pp. 7–8Takeaways & Limitations
The model supports both controllable hand-image synthesis and 3D hand-pose estimation, including use of unlabeled and weak labels.
Source: §1 · p. 2, §5 · p. 8The synthesis factors are assumed to be independent, an assumption the authors consider valid for hand images but plan to relax.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Hand image synthesis and pose estimation from RGB images are both highly challenging tasks due to the large discrepancy between factors of variation ranging from image background content to camera viewpoint. To better analyze these factors of variation, we propose the use of disentangled representations and a disentangled variational autoencoder (dVAE) that allows for specific sampling and inference of these factors. The derived objective from the variational lower bound as well as the proposed training strategy are highly flexible, allowing us to handle cross-modal encoders and decoders as well as semi-supervised learning scenarios. Experiments show that our dVAE can synthesize highly realistic images of the hand specifiable by both pose and image background content and also estimate 3D hand poses from RGB images with accuracy competitive with state-of-the-art on two public benchmarks.
1. Introduction
RGB hand-pose estimation is difficult because projection and diverse backgrounds create ambiguities, while accurately labeled real datasets are small. The paper introduces a disentangled VAE for controllable hand-image synthesis and pose estimation, with representations that separate image factors and support different tasks.
1. Introduction
- RGB hand-pose estimation faces projection and background ambiguities, and accurately labeled real datasets remain small compared with synthesized datasets.These challenges motivate strong kinematic or image priors.Source: §1 · p. 1
- Figure 1 depicts a VAE representation of RGB hand images conditioned on labeled factors such as pose, viewpoint, and image content.Red lines denote variational approximations; black lines denote the generative model.Source: Figure 1 · p. 1
- Standard GAN and VAE latent spaces offer little control over human-interpretable factors, often allowing only one average image per pose.The paper seeks to separate pose, viewpoint, scene context, and background while retaining discriminative pose estimates.Source: §1 · pp. 1–2
- The dVAE uses a cross-modal VAE baseline and a factor-based latent space to learn disentangled representations of hand poses and images.The framework supports learning factors from independent or confounding label inputs.Source: §1 · p. 2
- The dVAE handles RGB hand-image synthesis, pose transfer, and 3D pose estimation from RGB images.It provides explicit control over factors of variation and multiple degrees of freedom for hand-image synthesis.Source: §1 · p. 2
2. Related Works
Prior RGB hand-pose work includes discriminative and generative approaches, while related disentanglement methods often depend on specialized architectures or additional weak labels. The dVAE instead jointly learns a single latent space across modalities and explicitly models image factors.
2.1. Hand Pose Estimation
- Much progress in hand-pose estimation has focused on depth-image inputs [6] [8] [11] [18] [19] [32] [33] [35]; most methods treat depth as 2D pixels, while a few recent approaches use 3D points and/or voxels.Source: §2.1 · p. 2
- Earlier RGB hand-pose methods used restricted pose sets or simplified hand representations, while later work applied CNNs and generative models [21] [16] [25].Generative approaches also made it feasible to generate RGB images from given hand poses [16] [25].Source: §2.1 · p. 2
- The dVAE jointly learns one latent space across modalities, with more stable learning and better convergence properties than alternating VAE training [25].Explicitly disentangling image factors also supports synthesizing multiple images for the same pose, unlike one-to-one modality mappings [32] [25].Source: §2.1 · p. 2
2.2. Disentangled Representations
- Some disentangled-representation methods require specially designed layers and losses, limiting architectural flexibility [24] [28] [29].Other VAE approaches require weak labels such as grouping or pairwise similarities, which can be difficult to obtain for continuous pose and viewpoint [2] [13].Source: §2.2 · p. 2
- Compared with a conditional-dependency approach whose network complexity grows exponentially with the number of factors, the dVAE learns interpretable representations without additional weak labels.Its shared latent space is produced by images and their corresponding factors.Source: §2.2 · p. 3
3. Methodology
The method builds on a cross-modal VAE, then separates observed factors in a shared latent space and embeds image inputs into it. The resulting framework supports controlled hand-image synthesis and RGB pose estimation, including semi- and weakly supervised settings.
3.1. Cross Modal VAE
- A cross-modal VAE encodes modalities x and y through a shared latent z, assuming x and y are conditionally independent given z.For hand pose estimation, x can be an image and y the hand skeleton, allowing pose decoding from an encoded image.Source: §3.1 · p. 3
3.2. Disentangled VAE
- The fully specified dVAE decomposes z into factors associated with observed variables, learns their encoders and decoders, then embeds the image encoder while keeping decoders fixed.The objective uses hyperparameters to trade off latent-space capacity and reconstruction accuracy.Source: §3.2 · pp. 3–4
- When a factor such as image background is not independently labeled, the dVAE represents it with zu and trains it to be disentangled from the observed factor.For the pose case, zu is encoded from x; random samples of zu are used while minimizing the difference between p(y1|zy1, zu) and p(y1|zy1).Source: §3.2 · p. 4
- For multiple input modalities sharing z, the method first disentangles x, y1, and y2, then embeds the additional modality into the same latent space.The additional modality is encoded with qφˆx(z|ˆx).Source: §3.2 · p. 4
3.3. Applications
- Image synthesis varies 3DPose and representative tag-image content through their encoders to generate images with specified poses and backgrounds.The pose decoder can also evaluate the pose error of a synthesized image.Source: §3.3 · p. 5
- For pose estimation, RGB images are embedded in a shared latent space to infer 3DPose, CPose, and viewpoint; training can also use unlabeled or weakly labeled data.The semi-supervised setting uses labeled and unlabeled pose-related data, while the weakly supervised setting can use images with weak labels such as viewpoint.Source: §3.3 · p. 5
4. Experimentation
The experiments test whether disentangled representations support image synthesis, pose transfer, and monocular RGB hand-pose estimation. Results show controlled latent changes and pose-estimation performance comparable to prior methods on two benchmarks.
4.1. Implementation details
- Image encoding uses ResNet-18, image decoding follows DC-GAN, and pose encoding and decoding use six fully connected layers with 512 hidden units.Training uses ADAM with learning rate 10^-4 and batch size 32; latent dimensions are 64 overall and 32 for each sub-latent, with β adjusted by task.Source: §4.1 · pp. 5–6
4.2. Datasets & Evaluation
- Pose estimation is evaluated on STB and RHD using mean EPE and PCK AUC.STB uses a 15k/3k training/test split; RHD has 42k training and 2.7k testing images.Source: §4.2 · p. 6
4.3. Synthesizing Images
- Interpolating 3DPose with content fixed, or content with pose fixed, produces smooth, consistent synthesized images and reconstructed poses.Figure 4 shows pose interpolation in its first two rows, reconstructed skeletons in the third, and content interpolation with pose fixed in the fourth.Source: §4.3 · p. 6, Figure 4 · p. 6
- Recombining pose from one image with content from other images transfers hand poses accurately while maintaining the tag content.Figure 6 shows the source pose and tag-image content alongside the transferred images.Source: §4.3 · pp. 6–7, Figure 6 · p. 7
- With additional zu, background content changes smoothly while 3DPose stays well-fixed, even without specific tag images for training.Figure 5 displays generated images and corresponding reconstructed poses during content interpolation.Source: §4.3 · p. 7, Figure 5 · p. 7
4.4. 3D hand pose estimation
- The dVAE outperforms on CPose and 3DPose and is comparable to [25] and most other methods, except depth-augmented.[3] has higher PCK AUC by 0.038 on RHD and 0.03 on STB; these results are not directly comparable because uses depth images as additional training data.Source: §4.4 · p. 7
- With additional images and viewpoint labels, mean EPE improves by up to 7.5%; with CPose, viewpoint, and 3DPose labels, it improves by up to 3.5%.The experiments vary the fraction of labeled RHD images from 5% to 100%.Source: §4.4 · p. 7
5. Conclusion
The conclusion presents a dVAE for disentangled hand-image and pose representations, reporting competitive pose estimation and controllable hand-image synthesis. It notes that the approach assumes independent factors and calls for more challenging RGB pose-estimation benchmarks.
5. Conclusion
- The dVAE learns disentangled representations of hand poses and images and synthesizes realistic hand images with control over pose and background.The conclusion identifies independence between factors as an assumption to relax in future work.Source: §5 · pp. 7–8
- Figure 7 reports 3D PCK for RHD and STB and mean EPE in millimeters for both datasets.Figure 8 presents input images alongside CPose and 3DPose ground truth and predictions.Source: Figure 7 · p. 8, Figure 8 · p. 8
- Figure 9 compares mean EPE for the semi-supervised and weakly-supervised settings.Source: Figure 9 · p. 8
- The model is competitive with state of the art for hand pose estimation and can leverage unlabelled and weak labels.The authors note that STB performance is saturated and encourage collecting more challenging RGB benchmarks.Source: §5 · p. 8