Source-linked AI summary
Deep Convolutional Inverse Graphics Network
Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, Joshua B. Tenenbaum
TL;DR
The paper tackles the lack of automatically learned, semantically interpretable graphics codes and 3D rendering engines from images. It introduces DC-IGN, a variational convolutional encoder-decoder with transformation-focused training, and reports disentangled representations that support controlled pose, lighting, and other image variations.
Problem
Existing inverse-graphics approaches had not been shown to automatically produce semantically interpretable graphics codes and learn a 3D rendering engine to reproduce images.
Method
DC-IGN uses a deep convolutional and de-convolutional variational encoder-decoder trained with SGVB, plus mini-batches of active and inactive transformations to encourage disentangled graphics-code groups.
Results
The learned decoder functions as an approximate 3D rendering engine, allowing independent manipulation of scene variables such as lighting, pose, and shape to generate new images.
Takeaways & Limitations
The paper demonstrates that static images can train a deep inverse graphics network with interpretable, disentangled representations showing invariant and equivariant characteristics.
Takeaways & Limitations
The SGVB formulation is restricted to continuous latent variables, while real-world scenes may contain an unknown number of objects moving into and out of frame.
Abstract
from arXiv · showhide
This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that learns an interpretable representation of images. This representation is disentangled with respect to transformations such as out-of-plane rotations and lighting variations. The DC-IGN model is composed of multiple layers of convolution and de-convolution operators and is trained using the Stochastic Gradient Variational Bayes (SGVB) algorithm. We propose a training procedure to encourage neurons in the graphics code layer to represent a specific transformation (e.g. pose or light). Given a single input image, our model can generate new images of the same object with variations in pose and lighting. We present qualitative and quantitative results of the model's efficacy at learning a 3D rendering engine.
1 Introduction
The paper addresses the open problem of learning representations that are interpretable and disentangled with respect to meaningful visual transformations. DC-IGN combines an encoder-decoder model with a transformation-focused training procedure to learn graphics codes that support image re-rendering.
- Motivation: Choosing an optimal image representation remains open despite progress in hierarchical deep visual representations.Existing work characterizes useful properties such as invariance, meaningfulness, abstraction, and disentanglement, but does not settle the best representation for each task.
- Motivation: Inverse graphics offers a model in which compact graphics codes encode transformations such as location, pose, lighting, texture, and shape.These codes are designed to represent real-data transformations compactly and interpretably.
- Research gap: Existing inverse-graphics approaches had not automatically produced semantically interpretable graphics codes or learned a 3D rendering engine from images.Earlier methods generally used latent-parameter inference or optimization, while one encoder-decoder approach approximated a 2D rendering function.
- Approach: DC-IGN uses a convolutional encoder-decoder trained with SGVB to learn interpretable graphics codes for transformations including out-of-plane rotation and lighting.The encoder produces latent variables, while the decoder reconstructs images from those variables.
- Approach: The training procedure uses mini-batches with active and inactive transformations to encourage selected graphics-code neurons to represent specific transformations without target values.The stated objective remains reconstruction quality rather than supervised graphics-code labels.
- Results: Given a single face image, the model can regenerate the image with different pose and lighting and provides qualitative and quantitative evidence of learning a 3D rendering engine.The procedure is intended to produce a disentangled representation in which transformation-specific latent groups can be manipulated.
2 Related Work
Related work established probabilistic generative models and encoder-decoder approaches for visual representation learning, but DC-IGN extends this direction toward interpretable, disentangled graphics codes without requiring graphics-code labels or relying primarily on hand-crafted rendering engines.
- Generative representation learning: RBM-based generative models and DC-IGN differ in training: DC-IGN uses back-propagation with data reconstruction and a variational-bound objective.The comparison concerns the optimization and objective used to learn the representation.
- Variational models: SGVB learns generative models with continuous latent variables using an encoder to approximate the posterior and a decoder for stochastic reconstruction.DC-IGN extends SGVB with many convolutional encoder and de-convolutional decoder layers.
- Supervision: A supervised CNN graphics-code generator requires ground-truth graphics-code labels and therefore cannot be directly applied to image interpretation tasks.DC-IGN instead targets interpretable graphics codes through its own training procedure.
- Encoder-decoder models: Compared with earlier feature-learning encoder-decoder work, DC-IGN is distinguished by its end-to-end convolutional and de-convolutional variational formulation rather than layer-wise training.The cited comparison identifies layer-wise training as a difference in the earlier model.
- Inverse graphics: Many inverse-graphics methods rely on hand-crafted rendering engines, whereas DC-IGN is presented as learning interpretable and disentangled representations needed for a meaningful 3D graphics engine.The cited discussion identifies transforming autoencoders as an exception to the hand-crafted-engine pattern.
3 Model
DC-IGN learns a factored graphics representation in which latent variables correspond to pose, lighting, and intrinsic properties. Its training procedure uses transformation-specific minibatches, clamping, and modified gradients to encourage equivariance to one active transformation and invariance to the others.
- Model structure: The DC-IGN encoder captures a distribution over graphics codes, while its decoder learns to reconstruct images from latent variables representing pose, light, and shape.The model uses a factored latent representation to approximate a 3D graphics engine.
- Representation: The representation targets semantically interpretable variables, including face azimuth, face elevation, light-source azimuth, and intrinsic properties such as identity and shape.The first three variables are treated as extrinsic; intrinsic properties occupy the remaining latent variables.
- Transformation-specific training: Training minibatches vary only one transformation at a time, or vary intrinsic properties while holding the three extrinsic variables fixed.The selected batch type determines which latent is trained to capture the active variation.
- Transformation-specific training: For each minibatch, the outputs of nonselected encoder neurons are replaced by their batch averages before decoding, while reconstruction error is backpropagated using SGVB.The selected latent remains variable, whereas inactive latents are clamped to suppress unrelated variation.
- Transformation-specific training: Gradients for clamped neurons are replaced by their differences from the batch mean, directing inactive representations toward invariance while the selected neuron receives the reconstruction signal.This procedure forces the decoder to explain the batch’s variation through the selected neuron and trains the encoder to place that information there.
- Learned behavior: The resulting latent variables show strong equivariance with corresponding generating parameters, and the decoder can rerender an input image by independently varying pose or lighting latents.Pose latents are swept while other latents remain fixed, and the light latent is varied while the remaining latents are clamped.
4 Experiments
Experiments evaluate DC-IGN on generated 3D faces and varied chairs, testing whether disentangled latents support transformation prediction and novel-view rendering from single images.
- 4.1 3D Face Dataset: The model was trained on about 12,000 batches of procedurally generated faces varying identity, pose, and lighting.Each batch contained 20 faces with random variations in shape, texture, pose, or lighting.
- 4.1 3D Face Dataset: Figure 6 evaluated transformation prediction across datasets varying light, azimuth, and elevation, with all three encoder configurations reasonably predicting transformations from static images.The figure also suggests a switch node that separately processes left- and right-profile azimuth.
- 4.1 3D Face Dataset: DC-IGN learned an approximate rendering engine that independently varies lighting or other scene latents while preserving the remaining encoded factors.A static test image is encoded into latents such as light, pose, and shape, which can then be manipulated before decoding.
- 4.1 3D Face Dataset: The learned representation quantitatively encoded pose and light on smooth linear manifolds, although azimuth showed a discontinuity at 0°.Inferred and ground-truth transformation values were compared on random test examples; the discontinuity appeared for azimuth in Figure 6(a).
- 4.2 Chair Dataset: For chairs, azimuth was assigned to a single disentangled latent z1, and test reconstruction achieved an MSE of 2.7722 × 10^-4.Other image variation was represented by z[2,200], and images were grayscale at 150 × 150 pixels with values in [0, 1].
- 4.2 Chair Dataset: The network re-rendered previously unseen chairs across angles, producing smooth transitions for some examples but keyframe-like outputs for others.Changing one latent can require conjecturing about unseen components such as arms or seat shape.
5 Discussion
The discussion concludes that DC-IGN can learn interpretable, disentangled graphics codes from static images, while identifying architectural and latent-variable limits for scaling to more complex scenes.
- 5 Discussion: DC-IGN learned interpretable graphics-code representations from static images using a convolutional/de-convolutional variational autoencoder trained end-to-end.The proposed training procedure was intended to force disentangled and interpretable representations, demonstrated through invariant and equivariant characteristics on 3D faces.
- 5 Discussion: Scaling to many object categories and complex scenes may require deeper architectures and spatio-temporal convolutions that exploit motion.These directions are presented as likely important for handling larger category sets and complicated object transformations.
- 5 Discussion: The current SGVB formulation is restricted to continuous latent variables, whereas scenes may contain an unknown number of objects moving into and out of frame.The paper suggests discrete distributions or recurrent extensions as possible responses to this limitation.