Source-linked AI summary
AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, Jiebo Luo
TL;DR
Limited labeled data motivates unsupervised visual representation learning. AET trains representations by predicting sampled image transformations from encoded features, and the authors report improved unsupervised performance closer to fully supervised counterparts.
Problem
Limited labeled data motivates learning visual feature representations in an unsupervised fashion.
Source: §1 · p. 1Method
AET learns representations by reconstructing sampled image transformations from features of original and transformed images, rather than auto-encoding the data themselves.
Source: §1 · p. 2Results
AET substantially improves state-of-the-art unsupervised performance and narrows the fully supervised gap; on CIFAR-10, AET-project's error rate was 7.82% versus 7.2% for fully supervised NIN.
Source: §1 · p. 2, §4.1.3 · p. 5, §5 · p. 9Takeaways & Limitations
The framework accommodates varied transformation forms, including image warping and parametric or non-parametric transformations.
Source: §1 · p. 2The experiments focus on parameterized transformations, leaving GAN-induced and non-parametric transformations for future study partly because available GANs lack high-resolution fine-grained image quality.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
The success of deep neural networks often relies on a large amount of labeled examples, which can be difficult to obtain in many real scenarios. To address this challenge, unsupervised methods are strongly preferred for training neural networks without using any labeled data. In this paper, we present a novel paradigm of unsupervised representation learning by Auto-Encoding Transformation (AET) in contrast to the conventional Auto-Encoding Data (AED) approach. Given a randomly sampled transformation, AET seeks to predict it merely from the encoded features as accurately as possible at the output end. The idea is the following: as long as the unsupervised features successfully encode the essential information about the visual structures of original and transformed images, the transformation can be well predicted. We will show that this AET paradigm allows us to instantiate a large variety of transformations, from parameterized, to non-parameterized and GAN-induced ones. Our experiments show that AET greatly improves over existing unsupervised approaches, setting new state-of-the-art performances being greatly closer to the upper bounds by their fully supervised counterparts on CIFAR-10, ImageNet and Places datasets.
1. Introduction
The paper introduces Auto-Encoding Transformations (AET), which learns representations by predicting image transformations rather than reconstructing data. AET is intended to capture visual structure and its changes across a broad range of transformations, addressing the need for unsupervised learning when labeled data are limited.
1. Introduction
- Limited availability of labeled data constrains deep networks, motivating unsupervised feature learning for visual understanding tasks.The introduction notes that training deep networks often requires large amounts of labeled data.Source: §1 · p. 1
- AET trains an encoder to predict sampled image transformations from the learned representations of original and transformed images, rather than reconstructing input data as AED does.The framework depends on representations retaining enough information about image visual structures to decode the transformation.Source: Figure 1 · p. 1, §1 · p. 2
- AET explores how feature representations change under transformations, capturing both static visual structures and their dynamics.The encoder is trained to extract features that contain sufficient information about visual structures to decode the input transformation.Source: Figure 1 · p. 1, §1 · p. 2
- AET can use transformations ranging from simple image warping to parametric and non-parametric forms, without restricting their form.The introduction also says the framework can instantiate GAN-induced transformations.Source: §1 · p. 2
- The authors report that AET outperforms other unsupervised models and brings their performance closer to fully supervised counterparts.This is presented as an experimental finding across CIFAR-10, ImageNet, and Places in the paper context.Source: §1 · p. 2
2. Related Work
Prior unsupervised representation methods use data reconstruction, GAN-derived features, or self-supervised objectives; AET differs from auto-encoding data by predicting transformations directly from encoded features.
Auto-Encoders
- Auto-encoders learn features by reconstructing inputs, while AET instead minimizes discrepancy between input and output transformations.Variants include probabilistic feature assumptions in variational auto-encoders [15], denoising reconstruction, and cross-channel reconstruction; transforming auto-encoders still use the AED reconstruction objective.Source: §2 · pp. 2–3
Generative Adversarial Nets
- GANs represent data through noise coordinates on a learned manifold, often using mutually inverse generators and encoders [6] [8], and focus on generating the data distribution rather than one-to-one reconstruction.Regularized GANs’ generalization to unseen data under a Lipschitz assumption suggests expressive image representations [24] [2] [6] [8].Source: §2 · p. 3
Self-Supervised Representation Learning
- Self-supervised methods derive training signals from data, including patch relations [5], jigsaw puzzles [19], counting [20], rotations [10], surrogate classes, image transforms, and video self-motion [1].Surrogate-class features may over-discriminate visually similar images, and training is more expensive because each example forms an individual class.Source: §2 · p. 3
3. AET: The Proposed Approach
AET trains an encoder and decoder to infer sampled image transformations from the encoded features of original and transformed images. The approach supports parameterized, GAN-induced, and non-parameterized transformations.
3.1. The Formulation
- AET jointly trains an encoder and decoder to estimate a sampled transformation from the representations of an image and its transformed version.Training minimizes a loss between the sampled transformation and its estimate over sampled data and transformations.Source: §3.1 · p. 3
3.2. The AET Family
- Parameterized transformations, including affine and projective mappings, can be estimated through their parameters, with matrix differences capturing geometric changes.The paper recommends parameter-based loss for these transformations, finding image-based loss less effective unless many images are sampled.Source: §3.2 · pp. 3–4
- GAN-induced transformations use sampled noise to alter image semantics, enabling AET to learn from changes such as facial attributes or bedroom layouts.A decoder predicts the noise parameter from the encoded original and transformed images.Source: §3.2 · p. 4
- For non-parameterized transformations, AET can measure differences across transformed random images and approximate the transformation with a parameterized estimate.The approximation need not recover the input transformation exactly; the objective is to learn a useful representation within the chosen parameterized space.Source: §3.2 · p. 4
- The paper focuses on parameterized transformations for fair comparisons without extra models or auxiliary approximations, leaving GAN-induced and non-parameterized variants for future study.The authors note that GAN-induced transformations depend on generated-image quality, which they describe as limited for fine-grained high-resolution details.Source: §3.2 · p. 4
4. Experiments
Experiments evaluate AET representations on CIFAR-10, ImageNet, and Places through classification protocols. AET performs strongly against unsupervised baselines, approaches supervised performance on CIFAR-10, and transformation-prediction loss tracks classification performance on CIFAR-10 and ImageNet.
4.1.1 Architecture and Implementation Details
- CIFAR-10 AET uses shared-weight NIN branches for original and transformed images, concatenating fourth-block features into a 384-d vector to predict affine or projective transformations.AET-project performs better than AET-affine; affine and projective transformations are sampled using the ranges described for the CIFAR-10 experiments.Source: §4.1 · p. 4, §4.1.1 · pp. 4–5
4.1.2 Evaluation Protocol
- CIFAR-10 evaluation trains classifiers on features from the frozen first two convolutional blocks and also tests model-free KNN on pooled features.The classifiers include fully connected and convolutional models; KNN assesses features without training a labeled classifier.Source: §4.1.2 · p. 5
4.1.3 Results
- On CIFAR-10, AET-project with a convolutional classifier reaches 7.82% error versus 7.2% for fully supervised NIN with four convolutional blocks.The paper describes this as nearly matching the supervised counterpart.Source: §4.1.3 · p. 5, Table 1 · p. 5
- On CIFAR-10, AET-project consistently has the lowest errors across the tested classifiers and KNN settings, including evaluation without labeled examples.Against the same-network RotNet baseline, AET beats it with fully connected and convolutional classifiers; the paper also reports a significant lead over ExemplarCNN.Source: §4.1.3 · pp. 5–6, Table 2 · p. 6, Figure 3 · p. 6
4.2. ImageNet Experiments
- On ImageNet, AET beats the compared unsupervised models, including BiGAN, and narrows the supervised accuracy gap by 33% on Conv4 and 19% on Conv5.The gaps fall from 9.7% to 6.5% on Conv4 and from 15.7% to 12.7% on Conv5.Source: Table 3 · p. 6, §4.2 · p. 7
- With linear classifiers on ImageNet, AET again obtains the best accuracy among the compared unsupervised models.This evaluation trains a 1,000-way linear classifier on features from different convolutional layers.Source: §4.2 · p. 7, Table 4 · p. 7
4.3. Places Experiments
- On Places, AET outperforms other unsupervised models in most cases, while Counting performs slightly better on Conv1 and Conv2.The features are pretrained on ImageNet and evaluated with a logistic-regression classifier trained using Places labels.Source: §4.3 · p. 8, Table 5 · p. 8
4.4. Analysis of Predicated Transformations
- On CIFAR-10 and ImageNet, transformation-prediction loss trends align with classification error and Top-1 accuracy, supporting transformation prediction as a surrogate for representation quality.Examples also show AET decoding transformations from encoded image features.Source: §4.4 · pp. 8–9, Figure 4 · p. 8, Figure 5 · p. 8
5. Conclusions
AET trains unsupervised representations by predicting randomly sampled transformations rather than reconstructing data, encouraging encodings to retain visual structure from original and transformed images. The paper reports substantial improvements over prior state-of-the-art performances and a narrowed gap to fully supervised counterparts.
5. Conclusions
- AET substantially improves over state-of-the-art performances and significantly narrows the gap with fully supervised counterparts.Source: §5 · p. 9
- AET introduces unsupervised neural-network training by estimating randomly sampled transformations at the output, in contrast to conventional AED.Source: §5 · p. 9
- Predicting transformations encourages the encoder to retain sufficient information about visual structures in original and transformed images.Source: §5 · p. 9
- The framework incorporates a wide variety of transformations.Source: §5 · p. 9