Unsupervised Learning of Visual Structure using Predictive Generative Networks

1 INTRODUCTION

• Prediction errors provide continual feedback, while successful prediction requires a robust internal model of the world.The paper frames prediction as a possible framework for unsupervised learning.Source: §1 · p. 1
• Predicting high-dimensional video frames is difficult, and pixel-space MSE can blur uncertain predictions or penalize slight deformations.Earlier work had success in language and lower-dimensional motion tasks, while Ranzato et al. (2014) discretized image patches for prediction.Source: §1 · p. 1
• The CNN-LSTM-deCNN model is trained end-to-end for future-frame prediction using MSE, with an adversarial loss also implemented.The paper applies the architecture to bouncing balls and rotating computer-generated faces, combining weighted MSE and adversarial loss for face prediction.Source: §1 · p. 2
• Predictive models trained on rotating faces learn latent object structure and outperform reconstruction-trained controls on static-image classification.The MSE PGN scores highest across training-set sizes; the AL/MSE PGN also beats the controls.Source: §1 · p. 2

2 RELATED WORK

• Temporal continuity methods learn transformation invariances or robust representations from persistent causes in sequences (Földiák, 1991; Wiskott & Sejnowski, 2002).Deep-learning work has also brought temporal coherence into costs or linearized transformations across video frames (Goroshin et al., 2015a; Wang & Gupta, 2015; Goroshin et al., 2015b).Source: §2 · p. 2
• Relational feature learning models time series by representing transformations between frames.Michalski et al. (2014) inferred a transformation from two frames and applied it to predict a third, reporting a benefit over reconstruction training.Source: §2 · p. 2
• Palm (2012) used “Predictive Encoder” for an autoencoder trained to predict future rather than reconstruct current input.Preliminary experiments found Gabor-like filters where traditional autoencoders failed to learn useful representations.Source: §2 · p. 2

3 PREDICTIVE GENERATIVE NETWORKS

• The CNN embeds successive frames, the LSTM processes their sequence, and its final hidden state feeds a deCNN that produces a predicted image.The prediction is configured to match the input image size.Source: §3 · p. 3
• The LSTM uses a memory cell controlled by input and forget gates, with an output gate controlling its hidden state.The model uses 1568 LSTM units for bouncing balls and 1024 for rotating faces; σ denotes the elementwise logistic sigmoid.Source: §3 · p. 3
• For rotating faces, the deCNN uses a fully connected layer and two upsampling-convolution stages; the bouncing-balls model omits the fully connected layer.The face model’s final layer uses a saturating nonlinearity set at the maximum pixel value.Source: §3 · p. 3
• The adversarial discriminator is conditioned on the input sequence and uses separate parameters from the generator in the best-performing setup.It encodes the proposed frame, concatenates that encoding with its LSTM state, and passes the result through a three-layer MLP.Source: §3 · pp. 3–4
• For the adversarial objective, the generator is trained to maximize log D rather than minimize log(1 − D), which tends to saturate.The discriminator is trained to distinguish true frames from generated ones, while the generator attempts to fool it.Source: §3 · p. 4
• The generator’s weighted objective combines MSE and adversarial loss, and MSE remains useful for stabilizing and accelerating training.The weighting is controlled by λ.Source: §3 · p. 4

4.2 ROTATING FACES

• The face test uses unique generated faces rotating vertically, with five 150x150 input frames and 4,000 training clips.Source: §4.2 · pp. 4–5

4.1 BOUNCING BALLS

• On bouncing balls, PGNs compare favorably with the Deep Temporal Sigmoid Belief Network and RTRBM/SRTRBM variants on average squared one-step-ahead error per frame.The example sequence repeatedly predicts one step ahead using the prior ten frames as input.Source: §4.1 · p. 4, Table 1 · p. 5, Figure 2 · p. 5

4.2 ROTATING FACES

• AL/MSE λ was chosen qualitatively because Parzen log-likelihood was considered deficient for high-dimensional images; warm starts were used.Source: §4.2 · p. 5
• For unseen rotating faces, both losses recover angle and basic shape, while weighted AL/MSE yields detailed eyes and ears that MSE blurs or omits.Higher AL weighting exaggerates occasional identity inconsistencies in otherwise realistic predictions.Source: §4.2 · p. 5

5 EXPLORING LATENT REPRESENTATION LEARNING

• Both predictive models decode latent variables better than the reconstruction autoencoder, despite their visually distinct predictions.Their decoding performance is similar for the initial angle, rotation speed, and first four principal components.Source: §5 · p. 6, Table 2 · p. 6
• Decoding of all latent variables improves during training, with speed and initial angle learned quickly while some principal components initially decode worse.The MSE model’s random initial weights already permit fairly good decoding; principal components are learned more slowly.Source: §5 · pp. 6–7, Figure 4 · p. 7
• Training aligns feature-vector variance with the generative model’s latent variables without increasing overall feature-vector length.The visualizations compare initial and trained representations and track projections along regression-coefficient directions for PC1 and rotation speed.Source: §5 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8
• Linear changes along principal-component directions generate realistic faces, especially with AL/MSE, and generally track the underlying face model.The authors note that the principal components do not have precise semantic meanings, though changes are visible in features such as cheeks and jaw lines.Source: §5 · p. 7, Figure 7 · p. 8
• The classification test uses separate training and testing angles, evaluating whether the learned representation tolerates transformations.Source: §5 · p. 8
• On static-face classification, MSE PGN scores highest at every training-set size, while AL/MSE PGN remains ahead of the reconstruction controls.The controls include reconstruction models trained on dynamic or static stimuli and fully connected autoencoders.Source: §5 · p. 8, Figure 8 · p. 9

• Predictive-loss representations outperform comparable-complexity models on static-image classification, especially when classifying new faces from few example views.Source: §6 · p. 9
• The CNN-LSTM-deCNN models predict future frames from bouncing-ball physics through out-of-plane rotations of previously unseen computer-generated faces.Source: §6 · p. 9
• Experiments use highly simplified artificial worlds with known stimulus generators and few data degrees of freedom; scaling across real-world object transformations remains future work.Source: §6 · p. 9