PaperlayerView exact revision on arXiv

Source-linked AI summary

Associative Compression Networks for Representation Learning

Alex Graves, Jacob Menick, Aaron van den Oord

arXiv:1804.02476v2cs.NEcs.LGstat.ML

TL;DR

High-level concepts can be costly to encode for individual data points, motivating a dataset-wide view of compression. ACNs condition each code’s prior on a neighboring code, and the paper reports informative representations even with autoregressive decoders, including codes useful for clustering and classification.

  • Problem

    Associating a high-level concept with each individual example can outweigh its compression benefit, motivating ways to associate concepts collectively across the dataset.

    Source: §1 · p. 2

  • Method

    ACNs condition each code’s prior on a neighboring latent code and use proximity-based ordering to model local code variations.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    ACN representations retain meaningful high-level information with autoregressive decoders and can support clustering, linear classification, and traversals through latent and data space.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper presents ACNs as a dataset-wide direction for representation learning, with joint latent and data spaces that can be traversed by daydream sampling.

    Source: §5 · p. 8

    Code staleness could matter more on larger datasets, and nearest-neighbor search could become computationally prohibitive.

    Source: §3 · p. 4

Abstract

from arXiv · show

This paper introduces Associative Compression Networks (ACNs), a new framework for variational autoencoding with neural networks. The system differs from existing variational autoencoders (VAEs) in that the prior distribution used to model each code is conditioned on a similar code from the dataset. In compression terms this equates to sequentially transmitting the dataset using an ordering determined by proximity in latent space. Since the prior need only account for local, rather than global variations in the latent space, the coding cost is greatly reduced, leading to rich, informative codes. Crucially, the codes remain informative when powerful, autoregressive decoders are used, which we argue is fundamentally difficult with normal VAEs. Experimental results on MNIST, CIFAR-10, ImageNet and CelebA show that ACNs discover high-level latent features such as object class, writing style, pose and facial expression, which can be used to cluster and classify the data, as well as to generate diverse and convincing samples. We conclude that ACNs are a promising new direction for representation learning: one that steps away from IID modelling, and towards learning a structured description of the dataset as a whole.

1. Introduction

The introduction argues that powerful autoregressive decoders can ignore VAE codes, while attaching high-level concepts to individual examples can be costly. ACNs instead use latent relationships to organize dataset-wide compression, aiming to preserve informative codes without weakening the decoder.

1. Introduction

  • Powerful autoregressive decoders can ignore VAE codes, and proposed fixes add hand-tuned choices or often yield worse log-likelihoods than purely autoregressive models.The paper explains that stochastic codes may be more costly than simply ignoring them when the decoder is typically deterministic.Source: §1 · pp. 1–2
  • ACNs condition each code’s prior on a neighboring latent code, so nearby codes can be transmitted consecutively using a prior for local rather than global variation.The neighbor is selected from the K nearest codes; as K approaches dataset size N, the procedure recovers an unconditional prior.Source: §1 · p. 2
  • The optimal dataset ordering minimizes the sum of transmission-step KL costs, and the authors report that K-nearest-neighbor sampling approximates this ordering for a suitable K.The sequential procedure has the receiver re-encode each decoded example and use its encoding statistics to condition the prior for the next code.Source: §1 · p. 2
  • ACNs optimize transmission of the ordered dataset rather than the expected cost of transmitting one randomly selected data point, making their loss not directly comparable to IID VAE losses.The introduction argues that dataset relationships, such as grouping voices by gender, can make high-level features useful for compression.Source: §1 · p. 3
  • Because the decoder cannot access relationships between examples, the authors recommend a powerful decoder; they report that the resulting codes capture salient features useful for downstream tasks.The paper presents dataset-level relations and within-example dependencies as a separation that naturally encourages latent information.Source: §1 · p. 3

2. Background: Variational Auto-Encoders

The background describes VAEs as encoder-decoder generative models with a stochastic latent code. Their loss combines reconstruction cost and a KL term, while the prior is constant across inputs.

2. Background: Variational Auto-Encoders

  • A VAE encoder maps observable data x to a distribution q(z|x), samples a latent vector z, and a decoder uses z to form a reconstruction distribution r(x|z).Source: §2 · p. 3
  • The VAE loss adds the expected negative log-likelihood under r(x|z) to the KL divergence from prior p(z) to q(z|x).Source: §2 · p. 3
  • The prior p is constant for every input x, whether it is a simple Gaussian or a more complex autoregressive distribution.Source: §2 · p. 3

3. Associative Compression Networks

ACNs condition each code’s prior on a neighboring datum’s encoding distribution and add that KL cost to the usual reconstruction cost. Restricting neighbors to unique codes yields a sequential compression procedure.

3. Associative Compression Networks

  • ACNs condition each code’s prior on a neighboring datum’s encoding distribution and add that KL cost to the usual VAE reconstruction cost.The neighbor code is selected from the K nearest codes in latent space.Source: §3 · p. 3
  • Independent mixture priors for each latent dimension encourage multimodal but independent representations, while local conditioning can make the marginal prior flexible even when local priors are simple.The intended independence is described as potentially encouraging disentanglement.Source: §3 · p. 3
  • Restricting each code’s neighbors to unused codes enables sequential compression; with K = 1, the procedure is a greedy nearest-neighbor heuristic for a Euclidean travelling-salesman tour.That route upper-bounds the optimal KL cost, although the shortest Euclidean tour need not minimize KL cost.Source: §3 · pp. 3–4
  • Training maintains a separate code vector for each training example, selects nearby codes, and optimizes the encoder, prior, and decoder using the ACN loss.Codes are updated when their corresponding data are sampled, so neighbor searches can use somewhat stale codes.Source: §3 · p. 4
  • Code staleness made no discernible learning difference at tested update frequencies, but could matter more on larger datasets, where KNN search could also become prohibitive.The reported experiments found KNN computation inexpensive relative to network activation.Source: §3 · p. 4

3.1. Unconditional Prior

  • ACNs lack an unconditional prior by default, so an unconditional prior can be fitted to training-code samples after training.This provides a way to draw unconditional samples from the model.Source: §3.1 · p. 4

3.2. Sampling

  • Sampling includes unconditional draws, stochastic reconstructions, and daydreams that iteratively encode data, sample from a code-conditioned prior, generate data, and re-encode.Reconstructions use mean codes to reduce sampling noise, assuming an autoregressive decoder.Source: §3.2 · p. 4

3.3. Test Set Evaluation

  • Test evaluation uses separate tours for simplicity, but this may inflate KL costs when the test set is much smaller and its wider code spacing miscalibrates the prior.The alternative discussed is compressing training and test data together in a single tour.Source: §3.3 · pp. 4–5

4. Experimental Results

Across four image datasets, ACNs use gated PixelCNN decoders while learning codes that support compression, classification, and high-level image structure. Results include lower ordered MNIST coding cost, improved linear classification over pixels on CIFAR-10 and ImageNet, and represented facial attributes on CelebA.

4. Experimental Results

  • Experiments use a Gated PixelCNN decoder across four image datasets, with convolutional encoders and a learned conditional prior; the encoding distribution is a unit-variance Gaussian.The latent dimensionality is 16 for binarized MNIST and 128 for the other datasets.Source: §4 · p. 5

4.1. Binarized MNIST

  • The ordered ACN cost upper bound is 7 nats per MNIST image below the unordered cost, against 8.21 nats per image to specify test-set ordering.The authors interpret this as using most of the available “free bits” for latent information; the ordered cost remains an upper bound.Source: §4.1 · p. 5, Table 2 · p. 5
  • ACN codes cluster MNIST digit classes and capture writing style, while reconstructions preserve digit features and daydream samples morph across rotation, stroke width, and digits.Unconditional ACN samples show a subtle quality improvement over Gated PixelCNN samples.Source: Figure 3 · p. 6, Figure 4 · p. 6, Figure 5 · p. 6

4.2. CIFAR-10

  • CIFAR-10 ACN codes achieve 55.3% linear-classification accuracy versus 38.4% for pixels, with reconstructions showing occasional object parts and samples improving in coherence over baseline.The setup uses a VGG-style convolutional encoder and a gated-residual decoder.Source: §4.2 · p. 7, Figure 7 · p. 7, Figure 8 · p. 7

4.3. ImageNet

  • ImageNet ACN codes reach 18.5% top-1 and 40.5% top-5 linear-classification accuracy, versus 3.0% and 9.0% for pixels, respectively.Reconstructions suggest codes represent composition, colour, background, and setting; daydream sampling shows continuous transitions, while sample quality is less clearly distinct from Gated PixelCNN.Source: §4.3 · p. 7, Figure 10 · p. 8, Figure 9 · p. 8

4.4. CelebA

  • CelebA codes represent high-level attributes including gender, pose, lighting, face shape, and expression, with a slight sample-quality advantage over the baseline.Reconstructions leave specific image details to the decoder.Source: §4.4 · pp. 7–8, Figure 12 · p. 8, Figure 13 · p. 8

5. Conclusion

ACNs condition latent priors on associated codes and learn high-level representations that remain informative with autoregressive decoders. The representations support clustering and linear classification, while daydream sampling traverses the learned joint latent and data space.

5. Conclusion

  • ACNs condition the latent prior on associated codes, forming a new type of variational autoencoder.Source: §5 · p. 8
  • The learned representations contain meaningful high-level information that is not diminished by autoregressive decoders.Source: §5 · p. 8
  • The representations can cluster and linearly classify data, and the learned joint latent and data space can be traversed through daydream sampling.The paper suggests these representations may be useful for other cognitive tasks.Source: §5 · p. 8

A. CIFAR-10

CIFAR-10 nearest-neighbour comparisons contrast high-level clustering in ACN code space with superficial matches in pixel space. The supplied tables cover test-set compression results and ACN costs.

A. CIFAR-10

  • Table 4 presents CIFAR-10 test-set compression results.Source: Table 4 · p. 11
  • Table 5 presents CIFAR-10 test-set ACN costs.Source: Table 5 · p. 11
  • ACN-code nearest neighbours often cluster by high-level features such as object class and figure composition, unlike pixel-space neighbours, which tend to match background colour and favour blurry images.Source: Figure 14 · p. 11

B. ImageNet

The supplied ImageNet materials include 32x32 test-set compression results, ACN costs, reconstructions, and daydream samples.

B. ImageNet

Related papers

Loading 1804.02476v2…