Source-linked AI summary
Discovering Hidden Factors of Variation in Deep Networks
Brian Cheung, Jesse A. Livezey, Arjun K. Bansal, Bruno A. Olshausen
TL;DR
The paper asks how deep networks can represent variation beyond classification labels without discarding factors such as style or identity. It augments autoencoders with supervised and cross-covariance regularization, finding disentangled and manipulable factors across several image datasets, including extrapolation of hidden variation in labels.
Problem
Supervised representations may discard variation such as style or pose, motivating methods that preserve and factor variation beyond classification signals.
Method
The method divides autoencoder outputs into observed and latent variables and adds discriminative and cross-covariance penalties to the reconstruction objective.
Results
The networks discover digit style, facial identity, and identity-specific attributes, generate manipulated instances, and extrapolate continuous variation hidden in discrete labels.
Takeaways & Limitations
Standard deep feedforward networks can learn higher-order factors of variation beyond observed labels without explicitly defining higher-order interactions.
Takeaways & Limitations
The balance between classification performance and separation depends on the chosen multipliers β and γ.
Abstract
from arXiv · showhide
Deep learning has enjoyed a great deal of success because of its ability to learn useful features for tasks such as classification. But there has been less exploration in learning the factors of variation apart from the classification signal. By augmenting autoencoders with simple regularization terms during training, we demonstrate that standard deep architectures can discover and explicitly represent factors of variation beyond those relevant for categorization. We introduce a cross-covariance penalty (XCov) as a method to disentangle factors like handwriting style for digits and subject identity in faces. We demonstrate this on the MNIST handwritten digit database, the Toronto Faces Database (TFD) and the Multi-PIE dataset by generating manipulated instances of the data. Furthermore, we demonstrate these deep networks can extrapolate `hidden' variation in the supervised signal.
1 Introduction
The paper addresses how to preserve and disentangle variation beyond classification-relevant features. It proposes a simpler feedforward approach for representing class-irrelevant factors and evaluates the learned factors across image datasets.
- Supervised representations can make classes linearly separable but may discard variations such as style or pose.
- Autoencoders preserve input variation through reconstruction, but require explicit mechanisms to factor different sources apart.
- Earlier bilinear approaches separate factors multiplicatively but require learning an approximate tensor of three-way unit combinations.
- The proposed training scheme seeks to separate factors using standard nonlinearities in feedforward neural networks rather than bilinear interactions.
- Across MNIST, TFD, and Multi-PIE, the learned latent factors correspond to digit style, facial identity, and identity-specific attributes.
2 Semi-supervised Autoencoder
The semi-supervised autoencoder divides its representation into observed variables for prediction and latent variables for reconstruction. Reconstruction, supervised prediction, and XCov regularization jointly encourage complementary, disentangled factors.
- The latent variables must preserve dataset variation not represented by class labels, such as digit style, slant, and width.
- Learning: The objective combines squared-error reconstruction, cross-entropy supervision, and an unsupervised cross-covariance cost.
- Learning: XCov penalizes cross-covariance between observed and latent activations across examples in a batch to disentangle their representations.
- Learning: For unlabeled data, the supervised-cost multiplier β is set to zero; increasing β favors classification, whereas increasing γ favors factor separation.
3 Experimental Results
The experiments evaluate the trained autoencoders on datasets of increasing complexity, using MNIST’s handwritten digits and its standard train, validation, and test split.
- The models are evaluated on three datasets of increasing complexity using ADADELTA and standard backpropagation.
- MNIST contains 60,000 training and 10,000 test images, with the training set split into 50,000 training and 10,000 validation samples.
Toronto Faces Database
The Toronto Faces Database contains labeled facial-expression images, while the broader Multi-PIE dataset supplies supervised pose and illumination variation for face modeling.
- Toronto Faces Database: TFD contains 102,236 grayscale 48x48 face images, including 4,178 labeled examples across seven expressions.The seven expressions are anger, disgust, fear, happy, sad, surprise, and neutral.
- Toronto Faces Database: TFD’s five train, validation, and test folds are disjoint in identity, preventing identity overlap across partitions.
- Multi-PIE Dataset: Multi-PIE contains 754,200 color images of 337 subjects recorded across camera poses, illumination conditions, and facial expressions.The dataset originally used 15 camera poses, 19 illumination conditions, and varied expressions; overhead-camera images were discarded.
- Multi-PIE Dataset: Multi-PIE preprocessing uses pose-specific weak registration, 48x48 grayscale crops, and subject-disjoint training, validation, and test splits.The resulting partitions contain 528,060 training, 65,000 validation, and 60,580 test examples.
- Multi-PIE Dataset: Multi-PIE has more complex variation than MNIST or TFD, including body visibility and registration-induced changes in head position and scale.
3.2 Model Performace
The additional regularization has negligible impact on classification performance, while the MNIST model uses a two-dimensional latent variable for visualization.
- 3.2 Model Performace: The additional regularization negligibly affects classification performance across convolutional, maxout, and dropout architectures.MNIST and TFD classification results are reported as comparable to previously published results.
- 3.2 Model Performace: The MNIST architecture uses z ∈ R2 so the learned latent variables can be visualized directly.The latent variable distribution is approximately isotropic Normal with mean 0 and standard deviation .35.
Visualizing Latent Variables
Decoder traversals show that the latent variables preserve digit style while class labels change, supporting a class-invariant representation of handwriting variation.
- Visualizing Latent Variables: Varying one latent coordinate while fixing the other to zero and conditioning on each class label preserves digit style across classes.This pattern suggests the network learned a class-invariant latent representation.
- Visualizing Latent Variables: The center of latent space produces canonical MNIST style, whereas moving away from it generates increasingly stylized and less probable digits.
- Visualizing Latent Variables: The same latent organization is reliably reproduced without XCov when the latent dimensionality is relatively small.The authors report that the network naturally prefers this representation for variation absent from the supervised signal.
Moving From Latent Space to Image Space
The decoder’s Jacobians connect latent and hidden-unit perturbations to image changes, revealing how digit-style transformations are represented and progressively localized through the network.
- Decoder Jacobians measure how perturbing a hidden unit changes the output image.The analysis computes output-image derivatives with respect to hidden activations to characterize each unit’s transformation.
- Figure 3 varies one latent coordinate at a time while fixing the other and the class label, generating controlled digit-style changes.The figure also shows the test-set distribution of the two latent variables.
- The latent-variable Jacobians mirror Figure 3’s transformations, supporting a smooth latent space that controls digit style.As z2 approaches 2σ, a slanted style arises through a gaborlike filter applied to vertically oriented digit parts.
- The first hidden layer has a peaked singular spectrum, while the next layer is more degenerate and contributes increasingly localized, less semantically meaningful effects.For most digits, the leading first-layer component creates a template digit and remaining components make small style adjustments.
3.3 Generating Expression Transformations
On TFD, the latent representation encodes subject identity separately from expression, allowing expression changes while preserving facial identity in previously unseen faces.
- The latent representation z encodes subject identity, a major variation not represented by expression labels.The decoder changes expression while preserving identity for faces never seen during training.
- Replacing an encoded expression label with a new expression label changes facial expression while leaving other facial features largely intact.The model initializes {ŷ, z} from a test example and feeds the decoder {ŷ′, z}.
- With high-dimensional z, the XCov penalty improves reconstruction and prevents expression-label variation from leaking into the latent representation.The penalty is less necessary when z has dimensionality below 10, but training convergence becomes more difficult with that bottleneck.
3.4 Extrapolating Observed Variables
The decoder extrapolates facial-expression structure beyond the observed label range, producing exaggerated and abstract opposite expressions from extreme inputs.
- Scaling expression variables from -5 to 5 produces greatly exaggerated expressions and abstract negative counterparts beyond the softmax range.The natural encoder output range is 0 to 1, whereas the decoder receives values outside that range.
- Extreme opposite expressions reverse features such as open versus closed eyes and smiling versus frowning mouths.These opposites remain consistent across expression labels and other abstract facial features.
- The decoder learns a smooth, semantically sensible expression space for values absent from the class labels.This extrapolation is obtained by manipulating observed variables beyond those produced by the encoder’s softmax activation.
3.5 Manipulating Multiple Factors of Variation
Multi-PIE experiments manipulate camera pose and illumination while preserving identity-related attributes, and interpolate between neighboring observed conditions.
- 3.5 Manipulating Multiple Factors of Variation: Multi-PIE uses separate observed factors for camera pose and illumination, encoded by two softmax layers in a substantially deeper nine-layer network.The deeper architecture reflects the increased complexity of the images.
- 3.5 Manipulating Multiple Factors of Variation: Changing camera pose while fixing illumination and latent variables preserves subject illumination and identity-related attributes.Generated images retain shirt color, hair style, and skin tone, although they are blurry.
- 3.5 Manipulating Multiple Factors of Variation: Changing illumination while fixing camera position generates illumination transformations across the test-set faces.Figure 8 displays the transformations to the right of the source samples, with ground-truth lighting shown for the first face in each block.
- 3.5 Manipulating Multiple Factors of Variation: Linearly interpolating observed labels between neighboring camera positions produces interpolated camera and lighting conditions.This supports continuity in the class-label representation.
4 Conclusion
Adding supervised cost and unsupervised XCov to an autoencoder enables standard feedforward networks to disentangle transformations and represent higher-order variation. The decoder generates novel image manipulations, while the networks can extrapolate continuous variation hidden in discrete labels.
- A supervised cost combined with an unsupervised cross-covariance penalty enables autoencoders to disentangle transformations using standard feedforward components.
- The decoder implicitly generates novel image manipulations across multiple sets of transformation variables.
- Deep feedforward networks learn higher-order variation beyond observed labels without explicitly defining higher-order interactions.
- The networks learn a continuum of higher-order factors in both latent and observed variables and extrapolate continuous variation hidden in discrete class labels.