PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Discrete Representations via Information Maximizing Self-Augmented Training

Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, Masashi Sugiyama

arXiv:1702.08720v3stat.MLcs.LG

TL;DR

Unsupervised discrete representation learning needs compact, interpretable codes that preserve application-relevant similarities, but deep networks require careful regularization to learn intended invariances. IMSAT combines augmentation-based end-to-end invariance with information maximization, and experiments report state-of-the-art results for clustering and unsupervised hash learning.

  • Problem

    Unsupervised discrete representation learning must produce compact, interpretable representations with intended application-specific invariance, while deep networks are highly complex and their targets are unconstrained.

  • Method

    IMSAT combines Self-Augmented Training with Regularized Information Maximization, encouraging predictions to remain invariant under data augmentation while maximizing information-theoretic dependency between inputs and representations.

  • Results

    IMSAT achieves state-of-the-art performance for clustering and unsupervised hash learning on benchmark datasets.

  • Takeaways & Limitations

    Direct end-to-end invariance regularization can produce intended discrete representations for clustering and hash learning using deep neural networks.

  • Takeaways & Limitations

    The hash-learning approximation ignores interactions among hash bits beyond pairwise interactions.

Abstract

from arXiv · show

Learning discrete representations of data is a central machine learning task because of the compactness of the representations and ease of interpretation. The task includes clustering and hash learning as special cases. Deep neural networks are promising to be used because they can model the non-linearity of data and scale to large datasets. However, their model complexity is huge, and therefore, we need to carefully regularize the networks in order to learn useful representations that exhibit intended invariance for applications of interest. To this end, we propose a method called Information Maximizing Self-Augmented Training (IMSAT). In IMSAT, we use data augmentation to impose the invariance on discrete representations. More specifically, we encourage the predicted representations of augmented data points to be close to those of the original data points in an end-to-end fashion. At the same time, we maximize the information-theoretic dependency between data and their predicted discrete representations. Extensive experiments on benchmark datasets show that IMSAT produces state-of-the-art results for both clustering and unsupervised hash learning.

1. Introduction

IMSAT learns compact, interpretable discrete representations by combining augmentation-based invariance with information maximization in deep neural networks. The method targets clustering and unsupervised hash learning while scaling through stochastic optimization.

  • Motivation: Unsupervised discrete representation learning maps similar and dissimilar data to correspondingly similar and dissimilar discrete representations.Its applications include clustering and unsupervised hash learning.
  • Motivation: Deep networks model nonlinear decision boundaries and scale well, but require regularization because unconstrained representations may fail to exhibit application-specific invariance.Weight decay encourages global smoothness rather than necessarily imposing the intended invariance.
  • Method: IMSAT uses data augmentation to encourage neural-network predictions for augmented inputs to remain close to predictions for the original inputs.This directly imposes the invariance specified by the augmentation.
  • Method: IMSAT combines Self-Augmented Training with Regularized Information Maximization to maximize dependency between inputs and mapped outputs while regularizing the mapping.The method uses a deep network and end-to-end regularization, and can be optimized with SGD.
  • Contributions: The paper adapts IMSAT to clustering and hash learning and reports state-of-the-art performance on several benchmark datasets.These adaptations extend the method beyond the original clustering setting.

2. Related Work

Prior clustering and hashing methods trade off linear modeling limits, nonlinear-method scalability, and the availability of unsupervised deep hashing approaches. IMSAT addresses these gaps by directly imposing augmentation invariance on discrete representations for clustering and hash learning.

  • Clustering and hashing: K-means, Gaussian mixtures, iterative quantization, and minimal-loss hashing model only linear boundaries, limiting their fit to nonlinear data structures.Kernel and spectral methods model nonlinearity but are difficult to scale.
  • Deep clustering: Deep neural networks have been applied to clustering through learned features, cluster assignments, and generative models with Gaussian-mixture priors.These approaches represent recent deep-learning directions in clustering.
  • Deep hashing: Most deep hashing work has been supervised, while relatively few studies have addressed unsupervised hash learning.Semantic hashing pioneered compact binary representations with stacked RBM models.
  • Augmentation: Data augmentation has regularized supervised and semi-supervised neural networks, and has also been used to model invariance in unsupervised representation learning.IMSAT builds on this line of work while targeting discrete representations.
  • IMSAT distinction: Unlike prior unsupervised augmentation work, IMSAT directly imposes invariance on discrete representations rather than surrogate classes or continuous representations.The resulting representations are intended for direct use in clustering and hash learning.

3. Method

IMSAT extends information maximization to learn multi-dimensional discrete representations with deep neural networks, while using self-augmentation to impose application-specific invariance. Its clustering and hash-learning adaptations combine information objectives with representation-level regularization and scalable approximations.

  • Information maximization: IMSAT learns multi-dimensional discrete representations by maximizing mutual information between inputs and outputs while regularizing a multi-output probabilistic classifier.The representation has M dimensions, with each dimension taking values from its own finite set; the classifier is modeled by a deep neural network.
  • Self-Augmented Training: Self-Augmented Training penalizes dissimilarity between predictions for original and augmented inputs, directly imposing invariance on learned representations.The augmentation can be stochastic or deterministic and may be designed for the application, such as image rotations, scaling, and translations.
  • Self-Augmented Training: Local perturbations encourage local invariance and tend to place decision boundaries in low-density regions of the data distribution.The perturbation is intended to be small enough not to alter the data point’s meaning; random and adversarial perturbation variants are discussed.
  • Clustering: For clustering, IMSAT uses marginal and conditional entropy terms: the former encourages uniform cluster sizes, while the latter encourages unambiguous assignments.A specified class prior can replace the uniform distribution when prior knowledge about cluster sizes is available.
  • Hash learning: For hash learning, IMSAT maps data to D-bit codes and approximates joint mutual information using individual-bit and pairwise interaction terms.The approximation avoids the exponential computation of the full joint distribution but ignores interactions among hash bits beyond pairs.
  • Hash learning: The hash-learning approximation separates regularization, information maximization for each bit, and redundancy removal, while the objective is decomposed for mini-batch SGD.The scaling procedure decomposes the approximated mutual information into conditional entropy, marginal entropy, and pairwise output mutual information.

4. Experiments

Experiments evaluate IMSAT for clustering and hash learning on benchmark datasets, using fixed or broadly shared hyper-parameters. IMSAT performs strongly across clustering datasets, benefits from affine augmentation and larger networks, and achieves competitive hash-learning performance.

  • Clustering: IMSAT was evaluated for clustering on eight benchmark datasets alongside RIM variants, K-means, DEC, and denoising-autoencoder plus K-means.The experiments targeted clusters corresponding well with ground-truth categories; pre-trained residual-network features were used for STL, CIFAR10, and CIFAR100.
  • Clustering: IMSAT (VAT) performed well across datasets and outperformed Linear RIM, Deep RIM, and Linear IMSAT (VAT) on most datasets.The authors attribute this pattern to deep networks with end-to-end SAT regularization, while noting that linear classifiers may not model the intended invariance flexibly enough.
  • Clustering: IMSAT (VAT) consistently outperformed IMSAT (RPT), suggesting VAT was an effective regularization method in these unsupervised experiments.
  • Omniglot: Adding affine distortion to VAT significantly improved Omniglot clustering accuracy and produced assignments invariant to affine distortion.The combined augmentation was intended to capture invariance in handwritten character recognition without supervision.
  • Hash learning: Hash-learning experiments used mAP, precision at N = 500, and Hamming look-up at radius r = 2, averaged over ten trials.MNIST and CIFAR10 used 1000 randomly sampled queries, with the remaining data forming the gallery set.
  • Hash learning: IMSAT with 400-400 networks achieved competitive hash-learning performance on MNIST and CIFAR10, while 200-200 networks significantly improved IMSAT performance.The reported tendency was similar for 32-bit codes; Deep RIM benefited little from larger networks under weight-decay regularization.

5. Conclusion & Future Work

The paper concludes that IMSAT learns unsupervised discrete representations by combining information maximization with end-to-end augmentation-based invariance. It identifies structured data and suitable augmentations as future extensions.

  • 5. Conclusion & Future Work: IMSAT is an information-theoretic method for unsupervised discrete representation learning with deep neural networks.
  • 5. Conclusion & Future Work: The experiments show that neural-network predictions can directly acquire intended invariance to data augmentation through end-to-end regularization.
  • 5. Conclusion & Future Work: Future work includes applying IMSAT to structured data such as graphs or sequential data with appropriate augmentations.
  • Related methods: Unlike denoising auto-encoders, IMSAT regularizes the predicted representations directly rather than relying on decoder flexibility to model augmentation invariance.

B. Penalty Method and its Implementation

The implementation replaces IMSAT’s constrained objective with a penalty objective and progressively increases the penalty coefficient until the constraint is satisfied. Mini-batch optimization uses an upper bound on the exact KL divergence.

  • B. Penalty Method and its Implementation: The penalty method introduces µ and converts the constrained objective into an unconstrained objective for optimization.
  • B. Penalty Method and its Implementation: Increasing µ to the smallest feasible µ* yields a solution equivalent to the constrained optimization under the stated penalty-method condition.
  • B. Penalty Method and its Implementation: In experiments, µ was increased through λ, 2λ, 4λ, 6λ, and subsequent values until the constraint was satisfied.
  • B. Penalty Method and its Implementation: Convexity of KL divergence and Jensen’s inequality show that minimizing the mini-batch KL upper bound can satisfy the exact KL constraint.

D. Implementation Detail

The implementation uses mini-batches, He-style Gaussian weight initialization, layer-specific scales, and zero biases. These settings define the reported training procedure.

  • D. Implementation Detail: Training used mini-batches of 250 samples and 50 epochs for each dataset.
  • D. Implementation Detail: Weights were initialized independently from Gaussian distributions with standard deviation scale × 2/fanin, using scales 0.1-0.1-0.0001 across layers.All bias terms were initialized to 0.

E. Datasets Description

The experiments use image, digit, character, street-number, and document datasets, with preprocessing tailored to each data type. Visual datasets receive learned feature extraction or resizing, while text datasets use tf-idf features.

  • MNIST contains handwritten digit images whose pixel values were linearly transformed to [-1, 1].
  • Omniglot contains 100 character types sampled from four alphabets, with 20 examples per character and images down-sampled to 21-by-21 pixels.
  • STL images are 96-by-96 pixels, while CIFAR10 and CIFAR100 images are 32-by-32 pixels with 10 and 100 object classes, respectively.
  • SVHN uses 960-dimensional GIST features, while Reuters and 20news use tf-idf representations after dataset-specific text preprocessing.

F. Affine Distortion for the Omniglot Dataset

Omniglot augmentation uses stochastic affine distortion to generate transformed views for IMSAT experiments. The transformation combines random scaling, translation, rotation, and shearing, with Figure 3 illustrating examples.

  • Omniglot data points were augmented with stochastic affine distortion applied after down-sampling, following an implementation based on scikit-image.
  • Scaling independently varies x and y by factors drawn uniformly from [0.8, 1.2].
  • Translation independently varies x and y by offsets drawn uniformly from [−0.4, 0.4].
  • Rotation samples θ uniformly from [−10°, 10°], while shearing samples ρx and ρy uniformly from [−0.3, 0.3].
  • Figure 3 shows examples of the random affine distortion used in the experiments.

H. Experimental Results on Hash Learning with 32-bit Hash Codes

For 32-bit hash codes, IMSAT with larger networks achieved competitive performance on both datasets, and its performance improved substantially with moderately larger networks, unlike Deep RIM.

  • IMSAT with the largest 400-400 network sizes exhibited competitive 32-bit hash-learning performance on both datasets.
  • IMSAT improved significantly with 200-200 networks, whereas Deep RIM did not improve much when its networks were enlarged.

I. Comparisons of Hash Learning with Different Regularizations and Network Sizes Using Toy Dataset

Toy spiral experiments compare IMSAT with VAT regularization against Deep RIM with weight decay across increasing hidden-layer sizes. IMSAT models increasingly complicated decision boundaries as network dimensionality grows, whereas Deep RIM does not adapt similarly.

  • The results suggest that IMSAT’s benefit from large networks in benchmark datasets is related to its ability to model local invariance.
  • The toy dataset consists of a spiral with 300 data points per arc, and experiments vary hidden-layer sizes among 5-5, 10-10, and 20-20.
  • IMSAT uses VAT regularization with ϵ = 0.3, while Deep RIM uses weight decay with decay rate 0.0005.
  • IMSAT (VAT) models complicated decision boundaries as network dimensionality increases, but Deep RIM boundaries do not adapt to the data’s non-linearity.
Loading 1702.08720v3…