Source-linked AI summary
Learning Invariant Representations with Local Transformations
Kihyuk Sohn, Honglak Lee
TL;DR
Learning invariant representations that remain robust to variability in high-dimensional data is an important problem. The paper introduces a transformation-invariant feature-learning framework based on linear transformations and probabilistic max pooling, then reports strong results across image and phone classification tasks. It also identifies boundary effects from zero-padded squared transformation matrices as a limitation.
Problem
Learning representations invariant to transformations remains important for robust generalization from high-dimensional data, including images and speech.
Method
The framework incorporates linear transformations into unsupervised feature-learning algorithms, using transformed weights and probabilistic max pooling in a transformation-invariant RBM.
Results
The method significantly outperforms baseline RBMs on matched MNIST transformations, performs competitively on CIFAR-10 and STL-10, and achieves state-of-the-art TIMIT phone classification.
Takeaways & Limitations
The proposed framework handles predefined translation, rotation, and scaling transformations and shows strong classification performance across visual and phone tasks.
Takeaways & Limitations
Zero-padded squared transformation matrices produced boundary effects in filter visualizations and often significantly worsened classification performance.
Abstract
from arXiv · showhide
Learning invariant representations is an important problem in machine learning and pattern recognition. In this paper, we present a novel framework of transformation-invariant feature learning by incorporating linear transformations into the feature learning algorithms. For example, we present the transformation-invariant restricted Boltzmann machine that compactly represents data by its weights and their transformations, which achieves invariance of the feature representation via probabilistic max pooling. In addition, we show that our transformation-invariant feature learning framework can also be extended to other unsupervised learning methods, such as autoencoders or sparse coding. We evaluate our method on several image classification benchmark datasets, such as MNIST variations, CIFAR-10, and STL-10, and show competitive or superior classification performance when compared to the state-of-the-art. Furthermore, our method achieves state-of-the-art performance on phone classification tasks with the TIMIT dataset, which demonstrates wide applicability of our proposed algorithms to other domains.
1. Introduction
The paper develops a general framework for learning representations invariant to local transformations by incorporating linear transformations into unsupervised feature learning. Experiments report strong performance across image and phone classification tasks.
- Motivation: Invariant representations are motivated by the need for robustness to variability in high-dimensional data such as images and speech.Such representations are described as supporting good generalization with few labeled examples.
- Contribution: The paper introduces transformation-invariant feature learning by incorporating linear transformation operators into feature learning algorithms.The framework targets local translation, rotation, and scaling, and extends beyond RBMs to autoencoders and sparse coding.
- Contribution: Transformation-invariant RBMs represent inputs using transformed weights and obtain invariant features through probabilistic max pooling.The hidden units are pooled over the set of transformations.
- Experiments: The method significantly outperforms baseline RBMs on MNIST variations when model transformations match transformations underlying the data.The experiments also report greater robustness to a wide range of local transformations.
- Experiments: The method achieves competitive performance on CIFAR-10 and STL-10 visual recognition tasks and state-of-the-art performance on TIMIT phone classification.The TIMIT result is presented as evidence of applicability beyond visual recognition.
2. Preliminaries
The preliminaries introduce the restricted Boltzmann machine as the paper’s main example of an unsupervised feature-learning model. They describe its structure, probabilities, training objective, and extensions beyond binary inputs.
- Scope: The paper uses RBMs as its main example while noting extensions to other feature-learning algorithms later.The stated extensions include autoencoders and sparse coding.
- Restricted Boltzmann machine: An RBM is a bipartite undirected graphical model with visible and hidden layers.The preliminary formulation assumes binary-valued visible and hidden units.
- Restricted Boltzmann machine: RBM parameters include weights, hidden biases, visible biases, and a normalization factor determined by those parameters.The weights and biases are assigned to the visible-hidden connections and the two layers, respectively.
- Inference and training: Because RBMs have no within-layer connections, exact inference is tractable and block Gibbs sampling can be performed efficiently.The conditional probabilities are used for this sampling procedure.
- Inference and training: The RBM parameters are trained by minimizing negative log-likelihood with stochastic gradient descent.Contrastive divergence approximates the otherwise intractable exact gradient.
- Scope: The real-valued-input RBM can be formulated straightforwardly beyond the binary-input case presented for space reasons.The preliminary discussion explicitly limits its displayed treatment to binary-valued input variables.
3. Learning Transformation-Invariant Feature Representations
The framework learns invariance to local linear transformations by integrating transformation operators into feature-learning models and pooling across transformed responses. TIRBM provides the main example, with extensions to autoencoders and sparse coding.
- Transformation framework: The framework models local translation, rotation, and scaling transformations as linear operators incorporated into feature-learning algorithms.The transformation operator maps D1-dimensional inputs to D2-dimensional outputs through a linear matrix T.
- Transformation-invariant RBM: TIRBM represents input data using shared filter weights and their transformed versions for a specified set of transformation matrices.Its energy function is defined over visible units, filter weights, and hidden units indexed by filters and transformations.
- Transformation-invariant RBM: Probabilistic max pooling activates at most one transformed hidden unit per filter, producing pooled features invariant to the modeled linear transformations.When an input is transformed, TIRBM searches for a matching transformed filter and pools the corresponding hidden responses into z_j.
- Transformation-invariant RBM: Parameter sharing increases the effective number of filters by a factor of S without significantly increasing parameters, while pooling yields invariant representations.The comparison is with a regular RBM; transformed filters are generated from shared weights.
- Generating transformation matrices: Transformation matrices can represent shifts directly and image rotations or scalings through bilinear interpolation, with pre-computation and sparsity enabling efficient computation.The paper first presents one-dimensional transformations and states that the construction extends to two-dimensional image transformations.
- Extensions to other methods: The framework extends beyond RBMs to autoencoders and transformation-invariant sparse coding.The sparse-coding formulation constrains hidden activations analogously to the TIRBM softmax constraint and can be optimized using alternating updates with Orthogonal Matching Pursuit.
4. Related Work
Prior work learned invariant features through hand-crafted descriptors, noise robustness, convolutional translation invariance, and topographic maps. The paper positions its approach as addressing transformations beyond translation with compact, parameter-sharing models.
- Hand-crafted invariant descriptors: Hand-crafted descriptors such as SIFT can provide rotation- or scale-invariant features but require domain-specific knowledge and substantial manual design.These descriptors are presented as successful computer-vision tools, while their construction is not automatic from sensory data.
- Unsupervised robust representations: Denoising autoencoders learn robust features by reconstructing original data from representations of randomly perturbed inputs.The perturbations include adding noise or randomly zeroing selected coordinates.
- Translation invariance: Convolutional learning methods encourage shift-invariance, and convolutional deep belief networks obtain local translation-invariant representations through convolution and probabilistic max pooling.The cited CDBN is composed of multiple layers of convolutional RBMs and pooling operations.
- Transformation-based Boltzmann models: The paper distinguishes its models from contemporary transformation-equivariant Boltzmann machines by their motivation in local transformation invariance rather than global equivariance.Both approaches infer matching filters using linear transformation matrices, but the stated motivation differs.
- Topographic maps: Compared with topographic filter maps, the paper claims a more compact model with fewer parameters and greater pattern diversity at equal parameter counts.Its compactness comes from factoring filter weights from their transformations.
5. Experiments
The experiments evaluate transformation-invariant feature learning on transformed digits, natural images, and speech, using TIRBMs and related models. Across these settings, incorporating transformations improves representations and classification performance, while local transformations produce more diverse filters.
- Experimental setup: The framework translates filters across receptive fields or scales them while sharing parameters, using local transformation pooling to produce K-dimensional invariant features.The extracted feature dimension remains K rather than K × S.
- Experimental setup: Zero-padded squared transformation matrices caused boundary effects in filter visualizations and often significantly worsened classification performance.
- Handwritten digit recognition: Sparse TIRBMs consistently outperformed sparse RBMs across the MNIST transformation datasets, with lower error rates than published stacked denoising autoencoder results on mnist-rot and mnist-rot-back-image.The cited stacked denoising autoencoder error rates were 9.53% and 43.75%, respectively.
- Natural-image representation learning: On natural images, TIRBMs learned diverse diagonal and horizontal edge patterns, whereas sparse RBMs learned many similar shifted vertical edges.The natural-image filters used 14×14 patches, with translations, rotations, and two-level scale transformations.
- Object recognition: Each single transformation type improved CIFAR-10 accuracy over sparse RBMs, and combining transformations improved performance further; 4,000 filters achieved 82.2%.The 82.2% result was reported as better than previously published single-layer and deep-network results.
- Object recognition and phone classification: Transformation-enhanced models improved STL-10 accuracy and TIMIT phone classification, including 58.7% on STL-10 and a 1% gain over MFCC features on TIMIT.Using 512 TIRBM features also beat the best published TIMIT phone-classification result cited by the paper.
6. Conclusion and Future Work
The paper proposes feature-learning algorithms invariant to predefined transformations and reports strong classification performance for such transformations. It identifies automatically learning transformation matrices as future work for potentially more robust representations.
- The proposed feature-learning algorithms achieve invariance to a set of predefined transformations, including translation, rotation, and scaling.
- The implementation used K = 256 with a two-sided encoding function based on positive and negative weight matrices [W, −W].
- Experiments show that learning invariant features for these transformations leads to strong classification performance.
- Future work will learn transformation matrices from data automatically to potentially obtain more robust feature representations.