Source-linked AI summary
Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision
Hsiao-Yu Fish Tung, Adam W. Harley, William Seto, Katerina Fragkiadaki
TL;DR
The paper addresses the difficulty of learning image-to-factor or image-to-image mappings from unlabelled data. AIGNs combine feedback from differentiable rendering with distribution matching to unpaired ground-truth factors, and they improve supervised alternatives while supporting controllable biased facial transformations. Across tasks, the framework uses priors to adapt to test-set statistics and steer outputs toward selected characteristics.
Problem
Learning mappings from images to latent factors or other images from unlabelled data, or exploiting unlabelled data to improve supervised models, remains elusive.
Method
AIGNs combine feedback from rendering predictions with distribution matching between predicted factors and an unpaired collection of ground-truth factors.
Results
AIGNs outperform supervised alternatives in 3D pose and structure-and-egomotion tasks and produce controllable biased facial transformations through super-resolution and inpainting.
Takeaways & Limitations
Curating the ground-truth memories lets AIGNs steer rendered faces toward selected ages, genders, identities, or facial features without paired supervision.
Abstract
from arXiv · showhide
Researchers have developed excellent feed-forward models that learn to map images to desired outputs, such as to the images' latent factors, or to other images, using supervised learning. Learning such mappings from unlabelled data, or improving upon supervised models by exploiting unlabelled data, remains elusive. We argue that there are two important parts to learning without annotations: (i) matching the predictions to the input observations, and (ii) matching the predictions to known priors. We propose Adversarial Inverse Graphics networks (AIGNs): weakly supervised neural network models that combine feedback from rendering their predictions, with distribution matching between their predictions and a collection of ground-truth factors. We apply AIGNs to 3D human pose estimation and 3D structure and egomotion estimation, and outperform models supervised by only paired annotations. We further apply AIGNs to facial image transformation using super-resolution and inpainting renderers, while deliberately adding biases in the ground-truth datasets. Our model seamlessly incorporates such biases, rendering input faces towards young, old, feminine, masculine or Tom Cruise-like equivalents (depending on the chosen bias), or adding lip and nose augmentations while inpainting concealed lips and noses.
1. Introduction
AIGNs learn latent factors from images by combining rendered-prediction feedback with distribution-matched priors, using unpaired collections of ground-truth factors. The framework supports 3D perception and biased image transformation through task-specific differentiable renderers.
- Motivation: AIGNs use priors to infer hidden or unobserved factors, such as occluded limbs or camera motion from paired frames.The paper frames these predictions as interpretations guided by priors on how the world works.
- Core idea: AIGNs map images to latent factors using rendering feedback and distribution matching against stored ground-truth factors.The stored collection need not correspond directly to the current inputs, enabling unpaired supervision.
- Architecture: Task-specific differentiable renderers connect predictions back to the input space for human pose, structure from motion, super-resolution, and inpainting.Examples include camera projection, camera motion, downsampling, and masking.
- Biased image transformation: Biasing the ground-truth collection steers super-resolution and inpainting toward chosen facial attributes, including age, gender, identity, lip size, and nose size.The demonstrated outputs include older, younger, feminine, masculine, and Tom Cruise-like faces.
- Empirical scope: Across extensive experiments, AIGN performance consistently improves over supervised alternatives by adapting self-supervisedly to test-set statistics.The model is described as applying the same unsupervised framework across training and test phases.
2. Adversarial Inverse Graphics Networks
AIGNs combine reconstruction through task-specific differentiable renderers with adversarial matching to unpaired collections of plausible latent factors. The framework is applied across 3D pose, structure from motion, super-resolution, and inpainting.
- AIGN generators predict latent factors, render them back into the input space, and use discriminators to compare predictions with unrelated ground-truth memories.The memories may be a disordered set of labels, while renderers remain differentiable, parameter-free, and task specific.
- The training objective combines reconstruction and adversarial losses, with β controlling their relative weight.Because paired ground truth is not required, both losses can be used during training and testing.
- 3D human pose estimation: For 3D human pose, the generator predicts 2D heatmaps, basis weights, focal length, and rotation angles before projecting reconstructed 3D keypoints into 2D.The 3D shape uses a PCA basis with 60 retained components, while adversarial priors regularize the predicted shape.
- Structure from Motion: For structure from motion, separate networks estimate camera motion and per-pixel scene depth, then a renderer transforms and reprojects the resulting 3D point cloud.The motion discriminator enforces statistical camera-motion realism, while a second discriminator uses unrelated ground-truth depth maps.
- Image-to-image translation: For super-resolution and inpainting, AIGNs reconstruct inputs through downsampling or masking while discriminators compare generated outputs with unrelated complete images.Biasing the super-resolution memories toward selected face categories produces corresponding appearance changes, while inpainting uses complete face memories.
3. Experiments
Experiments evaluate AIGNs for 3D pose, structure from motion, and biased face transformation, using paired, self-supervised, and unpaired settings. Across these tasks, adversarial priors improve predictions, prevent divergence in monocular reconstruction, and transfer curated biases into image transformations.
- 3D human pose estimation: The 3D pose experiments use Human3.6M annotations and compare AIGNs with supervised predictors and baselines.The evaluation reports 3D reconstruction error in millimeters, organized by activity.
- 3D human pose estimation: AIGN outperforms the baselines, especially when given ground-truth 2D keypoints.Adversarial priors also support finetuning on new unlabeled data by maintaining anthropomorphic and plausible poses.
- Structure from Motion: The SfM study evaluates depth with log-depth L1 error and camera motion using distance, rotation-angle, angular-translation, and magnitude-translation errors.It includes training from scratch with reconstruction and adversarial losses, plus supervised pretraining followed by self-supervised finetuning.
- Structure from Motion: In self-supervised structure from motion, models without depth adversarial priors diverge because monocular reconstruction has scale ambiguity.Adversarial priors constrain scene scale, depth-like appearance, and camera-motion plausibility; depth smoothness provides an intermediate but still divergent result.
- Structure from Motion: Combining supervised pretraining with self-supervision yields lower camera-motion error than the geometric baseline on Virtual KITTI.The pretrained-only model performs worse than the geometric baseline, while the combined model improves without diverging.
- Image-to-image translation: Biased super-resolution and inpainting use curated discriminator memories to transform faces toward selected age, gender, identity, lip, or nose attributes.The super-resolution comparison reports better fidelity and visual detail than Attribute2Image.
4. Conclusion
AIGNs combine rendering feedback with adversarial priors for weakly supervised 2D-to-3D lifting and image-to-image translation. They outperform supervised alternatives in 3D tasks and support biased creative image editing.
- AIGNs combine rendering feedback with data-driven priors on latent semantic factors imposed through adversarial networks.
- AIGNs outperform previous supervised models without adversarial priors in 3D human pose estimation and 3D structure and egomotion extraction.
- Biasing discriminator priors for inpainting and super-resolution produces creative image editing with greater transformation fidelity and visual detail than prior supervised variational autoencoders.
- The framework suggests that priors, like features, can be learned rather than manually engineered.
A. Parametric vs. non-parametric decoders
The paper contrasts learned parametric decoders with non-parametric, domain-specific renderers. An MNIST inpainting experiment shows that parametric decoding can hide inaccurate latent predictions, whereas parameter-free rendering avoids this problem.
- A parametric decoder reverses predictions back into the input space but avoids requiring a hand-engineered target-to-input mapping.
- In MNIST inpainting, an encoder maps partially obscured digits to hallucinated digits, and a decoder reconstructs the original input.
- Parametric decoding can undo incorrect digit predictions, allowing the encoder to learn an inaccurate latent space while still recovering the input.
- Parameter-free rendering avoids the reconstruction loophole demonstrated by the parametric encoder-decoder.
B. Additional experiments and details
The additional-experiments section supplies implementation context and illustrates digit inpainting behavior. The MNIST figure shows accurate input recovery despite many incorrect latent digit predictions.
- The supplementary material provides implementation details, architecture descriptions, training details, and additional experimental results.
- The digit-inpainting experiment uses an encoder-decoder architecture with fully connected layers.
- Many digit predictions are incorrect even though the recovered inputs are accurate.
B.1. 3D human pose estimation from static images
The static-image pose system predicts 3D pose parameters from 2D body-joint heatmaps using adversarial priors. Supplementary figures show its architecture, predicted MPII poses, and related Structure-from-Motion components.
- 3D human pose estimation from static images: The pose generator takes 2D body-joint heatmaps and predicts weights for shape bases, rotation, translation, and focal length.
- 3D human pose estimation from static images: The model uses 60 PCA shape-basis dimensions out of 96 and relies on adversarial priors rather than PCA to regularize 3D shape prediction.
- 3D human pose estimation from static images: The model generalizes well to unseen MPII images without further self-supervised fine-tuning.
- 3D human pose estimation from static images: Figure 9 presents generator and discriminator architectures for single-image 3D human pose estimation.
- 3D human pose estimation from static images: Figure 10 displays predicted 3D human poses from supplied ground-truth 2D keypoints.
- 3D human pose estimation from static images: Figure 11 presents generator and discriminator architectures for Structure from Motion, with dashed lines marking skip connections.
B.2. Structure from Motion
The structure-from-motion system uses separate generator networks for optical flow, depth, and camera motion, with discriminators tailored to depth and egomotion realism. Training is stabilized by conditionally updating generators based on discriminator losses.
- Three encoder-decoder convolutional generators solve for optical flow, depth, and camera motion.
- The egomotion discriminator is a three-layer fully connected network that takes rotation and translation matrices as input.Its hidden layers contain 128, 128, and 64 neurons, respectively, with batch normalization and leaky ReLU activations.
- Stabilizing training: The generator is updated only when discriminator loss falls below a threshold, and may be updated twice when the generator remains relatively weak.
B.3. Image Super-Resolution
The super-resolution architecture uses residual convolutional blocks followed by deconvolution layers, while comparisons evaluate gender and age transformations against prior methods. Additional figures show the model's transformation outputs.
- The super-resolution generator applies an initial convolution, n residual blocks, and two deconvolution layers to produce the final image.Each residual block uses two convolutional layers, batch normalization, and a ReLU activation after the first normalization layer.
- The model is compared with Attribute2Image and Unsupervised Image Translation for gender and age transformations.The comparisons use code provided by the respective authors, while additional figures show the model's gender and age results.
- Figure 12 presents female-to-male, male-to-female, and young-to-old transformations.
B.4. Inpainting
The inpainting architecture combines separate encodings of an occluded image and its mask before decoding the concatenated representation. Additional results demonstrate biased inpainting for age transformation and larger lips.
- The inpainting generator separately convolves the occluded image and mask, concatenates their outputs, and decodes them through three deconvolutional layers.
- Figure 14 presents age transformation from old to young using AIGN.
- Additional biased-inpainting results show transformations producing bigger lips.