PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Autoencoder for Deep Learning of Images, Labels and Captions

Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, Lawrence Carin

arXiv:1609.08976v1stat.MLcs.LG

TL;DR

The paper addresses learning image representations when many images lack labels or captions. It proposes a VAE coupling a DGDN decoder, CNN recognition model, and label or caption generators, achieving competitive results across tasks while enabling semi-supervised and unsupervised learning.

  • Problem

    Most available images lack labels or captions, motivating models that can learn image features from associated and unassociated images.

  • Method

    A VAE uses a DGDN decoder, a CNN recognition model for latent-code posteriors, and Bayesian SVM or RNN models for labels or captions.

  • Results

    The model achieves competitive state-of-the-art performance across several tasks, including novel semi-supervised results, while the recognition model reaches up to 400x test-time speedup with comparable accuracy to Gibbs sampling and MCEM.

  • Takeaways & Limitations

    The framework supports jointly learned image, label, and caption models and extends CNN learning to semi-supervised and image-only unsupervised settings.

Abstract

from arXiv · show

A novel variational autoencoder is developed to model images, as well as associated labels or captions. The Deep Generative Deconvolutional Network (DGDN) is used as a decoder of the latent image features, and a deep Convolutional Neural Network (CNN) is used as an image encoder; the CNN is used to approximate a distribution for the latent DGDN features/code. The latent code is also linked to generative models for labels (Bayesian support vector machine) or captions (recurrent neural network). When predicting a label/caption for a new image at test, averaging is performed across the distribution of latent codes; this is computationally efficient as a consequence of the learned CNN-based encoder. Since the framework is capable of modeling the image in the presence/absence of associated labels/captions, a new semi-supervised setting is manifested for CNN learning with images; the framework even allows unsupervised CNN learning, based on images alone.

1 Introduction

The paper introduces a VAE that combines a DGDN image decoder with a CNN recognition model to learn from images with or without labels or captions. This framework supports semi-supervised and unsupervised CNN learning while retaining fast CNN-based inference.

  • Semi-supervised learning: The framework addresses the prevalence of images without labels or captions by enabling semi-supervised learning from both associated and unassociated images.A generative image model accounts for unlabeled or uncaptioned images.
  • Inference: The CNN recognition model provides a faster alternative to the original DGDN inference procedures, which relied on Gibbs sampling or MCEM.The original DGDN required relatively expensive latent-feature inference.
  • Core framework: The proposed VAE uses a DGDN as image decoder and a CNN recognition model to approximate the posterior distribution of latent DGDN parameters.The CNN-based encoder is designed for fast test-time inference.
  • Multimodal modeling: The model links shared latent codes to Bayesian SVM label models or RNN caption models, with all parameters learned jointly.Images can be modeled together with associated labels or captions.
  • Architectural distinction: Unlike prior convolutional VAEs with deterministic pooling and unpooling, the proposed DGDN-CNN model infers stochastic unpooling maps by maximizing a variational lower bound.This distinguishes the model's decoder from related architectures.
  • Reported contributions: The paper reports accuracy comparable to Gibbs sampling and MCEM, up to 400x faster test-time recognition, semi-supervised classification results, and extensive image-captioning experiments.The contributions span image modeling, classification, and caption generation.

2 Variational Autoencoder Image Model

The image model is a multilayer DGDN decoder driven by latent feature codes and stochastic unpooling. A bottom-up CNN encoder produces posterior distributions over the decoder's latent codes and unpooling maps.

  • Image decoder: The decoder generates each image from top-layer activation maps that serve as the latent code and feed a top-down multilayer deconvolutional process.The decoder is represented as pα(X|s, z), where s contains top-layer features and z contains unpooling maps.
  • Image decoder: Convolving activation maps with dictionary elements and summing the resulting tensors reconstructs the image's modeled signal E(X(n)).At the bottom layer, the reconstruction is combined with independent zero-mean Gaussian noise with precision α0.
  • Stochastic unpooling: Stochastic unpooling partitions activation maps into px × py blocks and samples a one-hot location within each block to place pooled activations.The unpooling map is drawn from a uniform multinomial prior, producing sparse higher-resolution activation slices.
  • Multilayer model: The decoder can be replicated across L > 2 layers, with dictionary elements and precision α0 treated as model parameters and the latent code assigned a standard normal prior.The prior is p(s) = N(0, I).
  • CNN encoder: The encoder processes images bottom-up with convolutional filters, stochastic pooling, and additional convolutional layers to form feature tensors.Spatial feature maps are aligned and stacked after convolution, while pooling is performed stochastically.
  • CNN encoder: MLPs transform pooled CNN features into posterior pooling statistics and the mean and variance functions for the latent DGDN code.These components define qφ(s, z|X), the recognition-model distribution over codes and unpooling maps.
  • CNN encoder: The CNN encoder parameters comprise convolutional filter banks and MLP parameters, enabling fast testing while modeling a posterior distribution over decoder parameters.The same recognition model supports semi-supervised and unsupervised CNN learning.

3 Leveraging Labels and Captions

The framework links top-layer DGDN image features to generative models for labels and captions. Labels use Bayesian one-versus-all SVMs, while captions are generated sequentially by an RNN.

  • Generative Model for Labels: Bayesian SVM: C one-versus-all binary SVM classifiers map top-layer DGDN image features to image labels.The classifiers use the same image code s_n from the top DGDN layer.
  • Generative Model for Labels: Bayesian SVM: The SVM objective combines hinge-loss error penalization with regularization controlling classifier complexity.The tuning parameter γ controls the trade-off between these terms.
  • Generative Model for Labels: Bayesian SVM: The Bayesian SVM pseudo-likelihood has a location-scale Gaussian-mixture representation using latent variables λ_n.λ_n controls the mixture components' means and variances, enabling data augmentation for Bayesian inference.
  • Generative Model for Labels: Bayesian SVM: The label classifier is analogous to the fully connected parameters atop a traditional CNN, with a softmax likelihood available as an alternative.The paper compares SVM- and softmax-based classifiers in its experiments.
  • Generative Model for Captions: Captions are represented as sequences of one-hot vocabulary vectors and generated word by word with an RNN conditioned on top-layer DGDN features.The first word is generated from the image code, after which hidden states are recursively updated until the end-sentence symbol.
  • Generative Model for Captions: Each caption word is embedded through a learned matrix, while the RNN uses learned weight matrices to produce distributions over words.The transition function can use LSTM or GRU units; experiments use GRU because it performed slightly better.

4 Variational Learning of Model Parameters

Variational learning jointly trains the image encoder, DGDN decoder, and label or caption generator using labeled or captioned and uncaptioned images. At test time, predictions average the generative output over latent codes inferred efficiently by the recognition model.

  • Model integration: The joint model combines the image encoder q_φ(s,z|X), image decoder p_α(X|s,z), and caption generator, replacing the caption model with a Bayesian SVM for labels.This setup supports both captioning and labeling within the same image-generative framework.
  • Variational objective: The parameters {φ, α, ψ} are learned by minimizing a variational lower bound for each captioned image.The lower-bound objective contains terms for the image and associated caption.
  • Variational objective: Setting ξ to zero recovers the variational lower bound for a single uncaptioned image.The full dataset objective separates captioned images D_c from uncaptioned and unlabeled images D_u.
  • Optimization: Monte Carlo integration approximates expectations in the objective, while stochastic gradient descent and variance reduction techniques optimize the parameters.The optimization is performed with respect to φ, ψ, and α.
  • Variational objective: ξ is scaled by image size, caption or label size, and the labeled or captioned fraction of each mini-batch to assign equal weight to data points.When ξ equals 1, the objective recovers the exact variational lower bound.
  • Test-time inference: At test time, caption or label inference averages predictions over latent codes sampled from q_φ(s|X⋆).The recognition model approximates p(s|X) and enables fast inference of the image representation.

5 Experiments

Experiments evaluate classification, semi-supervised learning, and image captioning across standard benchmarks. The CNN-based recognition model matches sampling-based accuracy while providing substantially faster testing, and joint latent modeling improves captioning.

  • Benchmark Classification: The proposed recognition model achieves accuracy commensurate with Gibbs sampling and MCEM while providing up to 400x faster test-time inference.The comparison uses classification error and testing time on benchmark datasets.
  • Semi-Supervised Classification: Using 10 labeled images per MNIST class, the semi-supervised model achieves a test error of 1.49 and performs best with more labeled images.The unsupervised setting also produces image representations evaluated with a separate transductive SVM.
  • Semi-Supervised Classification: ImageNet 2012 experiments vary the balanced labeled-image proportion from 1% to 100% and compare against supervised AlexNet and GoogLeNet baselines.Training is repeated 10 times with different unlabeled-image sets; the supervised baselines use only labeled data.
  • Semi-Supervised Classification: Replacing the Bayesian SVM with softmax worsens results by about 1% below 30% labeled data and about 0.5% above 30%.The reported gap suggests the semi-supervised framework contributes more than the SVM's discriminative power.
  • Image Captioning: The unsupervised two-step model outperforms comparable VggNet and GoogLeNet feature-extractor baselines on image captioning.Those baselines require labeled ImageNet images, whereas the proposed deep model is trained using uncaptioned images.
  • Image Captioning: The joint captioning model improves average BLEU scores by nearly 10% over the two-step model, with the largest semi-supervised gains on Flickr8k and Flickr30k.The captioning evaluation covers Flickr8k, Flickr30k, and MS COCO using BLEU, METEOR, CIDEr, and PPL metrics.

6 Conclusions

The paper couples a CNN-based DGDN recognition model with Bayesian SVM and RNN models for labels and captions, respectively. The resulting variational framework supports semi-supervised and unsupervised CNN learning and achieves competitive results across tasks.

  • Conclusions: A CNN-based recognition model approximates DGDN latent variables and couples them with Bayesian SVM and RNN models for labels and captions.The model is learned with a variational autoencoder setup.
  • Conclusions: The framework leverages images without labels or captions for semi-supervised learning and supports unsupervised CNN learning from images alone.Its GPU-based implementation achieves competitive results on several tasks, including novel semi-supervised results.

A Semi-supervised Results on ImageNet 2012

Table 4 reports semi-supervised classification accuracy on the ImageNet 2012 validation set.

  • Table 4 measures semi-supervised classification accuracy (%) on ImageNet 2012 validation data.

B Model Architecture and Initialization

The experiments use dataset-specific image architectures with convolutional dictionaries and pooling, while parameters are randomly initialized without layer-wise pretraining.

  • MNIST and CIFAR-10 use 28 × 28 grayscale and 32 × 32 RGB images, respectively.
  • The image models use two layers with 8 × 8 and 6 × 6 dictionary elements, 3 × 3 pooling, and 30 and 80 dictionary elements.
  • Image-model parameters are randomly initialized, with no layer-wise pretraining.
  • Captioning RNNs use orthogonal recurrent initialization, uniform non-recurrent weights in [-0.01,0.01], zero biases, and 512 hidden units.

C.1 Image Captioning

The image-captioning objective is optimized through variational gradients, Monte Carlo expectations, variance reduction, and reparameterized latent sampling.

  • The captioning model optimizes a variational lower bound whose gradients are taken with respect to decoder and encoder parameters.
  • Because the model contains real and binary latent variables, variance-reduction techniques are used to control high estimator variance.
  • The latent representation is reparameterized as s = µφ( ˜C(L)) + ϵ(σφ( ˜C(L)) with ϵ ∼ N(0, I).
  • The caption likelihood term is combined with the image-model variational terms and weighted by ξ.
  • The encoder gradient uses the same image-model gradient form identified in prior work.
  • Monte Carlo sampling approximates the expectation terms in the variational objective.

C.2 Image Classification

The image-classification objective treats pseudo-labels as latent variables and derives gradients using variational expectations, reparameterization, and stacked convolutional features.

  • The classification model defines a variational lower bound while treating each pseudo-likelihood label ℓ_n ∈ {1, . . . , C} as latent.
  • The classification derivation largely follows the image-captioning objective and focuses on gradients with respect to β.
  • The gradient with respect to β is obtained using the same reparameterization trick as the image-captioning model.
  • The encoder means and variances are formed by stacking spatially aligned features across the K2 second-layer dictionary elements.
Loading 1609.08976v1…