Learning Wake-Sleep Recurrent Attention Models

• Hard attention reduces computation by selecting discrete glimpse locations, whereas soft attention examines every image location through weighted averaging.Source: §1 · p. 1
• Training stochastic hard attention is difficult because posterior expectations are intractable and stochastic gradient estimates have high variance.Source: §1 · p. 1
• WS-RAM uses importance sampling with a label-conditioned inference network, reweighted wake-sleep training, and control variates to improve inference and reduce gradient variability.The inference network proposes glimpse locations using the object category label, while control variates target stochastic-gradient variance.Source: §1 · pp. 1–2
• WS-RAM achieves similar performance to variational training while greatly reducing training time across translated and scaled MNIST classification and image caption generation.Source: §1 · p. 2

• Attention-based neural networks have been applied to classification, tracking, translation, caption generation, and image generation, while improving efficiency and interpretability.Source: §2 · p. 2
• Stochastic hard attention requires expectations over posterior saccade distributions, creating an intractable inference problem related to training deep generative models.Source: §2 · p. 2
• Inference networks approximate posterior distributions, and wake-sleep methods train generative and inference networks using alternating objectives.Source: §2 · p. 2
• Reweighted wake-sleep replaces mean-field inference with importance sampling, while NVIL and variational autoencoders address related stochastic-gradient or inference problems.Source: §2 · p. 2
• The literature commonly calls an inference network a recognition network, but this paper avoids that term to prevent confusion with image classification.Source: §2 · p. 2

• Given an image, WS-RAM stochastically selects sequential glimpses and then predicts a distribution over the target category or caption.The first glimpse uses a low-resolution input, and later glimpses depend on information from earlier observations.Source: §3 · p. 3
• The two-layer prediction network outputs each saccade for computing the next glimpse and separates low-resolution input from class prediction to prevent attention-bypassing local optima.Source: §3 · p. 3
• The inference network receives the class label and the prediction network’s top-layer representation to estimate the next-saccade posterior and guide training-time sampling.It acts as a teacher for the attention network by predicting where to look when the class is correctly identified.Source: §3 · p. 3
• Localized or coarse glimpse mappings can be efficient and need not be differentiable, but this flexibility prevents standard backpropagation.Source: §3 · p. 3

4.2 An improved lower bound on the log-likelihood

• The variational baseline averages log-likelihoods across actions, disproportionately penalizing bad glimpses and motivating direct optimization of the marginal likelihood.For example, the variational objective treats probabilities 0.01 and 0.02 for the correct class as substantially different even when their practical outcomes may be similar.Source: §4.2 · pp. 4–5
• WS-RAM directly maximizes the marginal log-likelihood using normalized importance sampling, yielding a lower bound at least as accurate as the earlier variational objective.The estimator is biased but consistent, and the proposed bound is shown to be at least as accurate as F through Jensen’s inequality.Source: §4.2 · p. 5

4.3 Training an inference network

• The inference network uses image observations and class labels to propose informative glimpse sequences, helping when the prior has little overlap with the posterior early in training.It is trained to reduce the KL divergence between its recognition distribution and the attention model’s posterior, reusing importance weights for its gradient estimate.Source: §4.3 · pp. 5–6

4.4 Control variates

• Control variates reuse the same samples and importance weights as the gradients, producing much lower-variance estimates without additional bias beyond importance sampling.Their effectiveness depends on correlation with the stochastic gradients, and the resulting estimates are reported to have substantially lower variance.Source: §4.4 · p. 6
• Figure 2 compares VAR, WS-RAM, and ablations using training error, gradient-estimate variance, and effective sample size over thousands of updates.The ablations remove the inference network or control variates, with the prior used as the proposal when the inference network is removed.Source: Figure 2 · p. 7

Translated scaled MNIST

• The classification experiment compares WS-RAM with a variational baseline using equal sample counts and also evaluates ablations removing the inference network or control variates.The task uses transformed MNIST images and reports classification error after training; the supplied passages do not include the table’s numerical values.Source: §5.1 · p. 7, Table 1 · p. 8

Flickr8K caption generation

• Training behavior on Flickr8K is also examined through negative log-likelihood during the first 10,000 updates.The supplied figure caption identifies the metric and training interval but does not state the curves’ numerical outcomes.Source: Figure 4 · p. 8, §5.2 · p. 8

Translated scaled MNIST

• Control variates improved both gradient-estimation variance and effective sample size, whereas the inference network reduced gradient-estimation variance but not ESS.The ESS analysis compares WS-RAM with and without its inference network and control variates.Source: §5.1 · pp. 7–8

Exploration heuristics

• Without exploration heuristics, the variational method collapsed to predicting one glimpse scale, while WS-RAM did not require the heuristics.The heuristics corrected the variational method’s local-minimum behavior during training.Source: §5.1 · p. 8, Figure 3 · p. 8

Flickr8K caption generation

• The caption-generation experiment evaluates WS-RAM against the variational method on Flickr8K using attention over convolutional representations at both selected layers and locations.Selecting a convolutional layer functions as selecting an attention scale because max-pooling changes representation resolution.Source: §5.2 · p. 8, Table 2 · p. 8

6 Conclusions

• WS-RAM combines reweighted wake-sleep posterior approximation [9] with control variates to reduce gradient variability and accelerate stochastic attention training.The conclusion reports these benefits for invariant handwritten-digit recognition and image caption generation.Source: §6 · p. 8