Source-linked AI summary
Learning to Linearize Under Uncertainty
Ross Goroshin, Michael Mathieu, Yann LeCun
TL;DR
Although supervised deep features have succeeded and representations can transfer across tasks, principled unsupervised learning of broadly useful features remains elusive. The paper trains generative frame-prediction models to learn locally linearized video representations while addressing uncertainty; its Deep Architecture 3 has the best overall L2 prediction error and most visually realistic images among the evaluated architectures.
Problem
Supervised deep features have succeeded, and learned representations can transfer across tasks, but principled unsupervised learning of generically useful features remains an open question.
Source: §1 · p. 1Method
The paper trains generative frame-prediction models to learn locally linearized video representations, using phase-pooling and input-nondeterministic latent corrections to address transformations and uncertainty.
Source: §1 · pp. 1–2, §3 · p. 2, §3.1 · p. 3Results
Deep Architecture 3 achieved the best overall L2 prediction error and generated the most visually realistic images.
Source: §4.2 · p. 8Takeaways & Limitations
Stacking these architectures is proposed as future work to linearize more complex features over longer temporal scales.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Training deep feature hierarchies to solve supervised learning tasks has achieved state of the art performance on many problems in computer vision. However, a principled way in which to train such hierarchies in the unsupervised setting has remained elusive. In this work we suggest a new architecture and loss for training deep feature hierarchies that linearize the transformations observed in unlabeled natural video sequences. This is done by training a generative model to predict video frames. We also address the problem of inherent uncertainty in prediction by introducing latent variables that are non-deterministic functions of the input into the network architecture.
1 Introduction
The paper asks whether broadly useful representations can be learned without supervision, and proposes training deep networks to linearize temporal trajectories through frame prediction. It addresses blurry predictions from uncertainty and the tendency to copy the latest frame by adding latent variables and predicting in linearized feature space.
1 Introduction
- The paper asks whether a generically useful feature representation exists and what principles could enable learning it, building on successful supervised feature learning and cross-task transfer.The introduction notes that representations learned for one task can transfer to others.Source: §1 · p. 1
- Temporal coherence in video offers weak supervision: treating videos as trajectories on a data manifold motivates learning representations that locally linearize those trajectories.The paper assumes deep convolutional networks can model natural data and learn a local parametrization.Source: §1 · p. 1
- Frame prediction serves as a proxy for representation learning, addressing blurry averages under uncertainty with latent variables and copying the latest frame with prediction in linearized feature space.The introduction identifies both failure modes as obstacles to learning useful representations.Source: §1 · pp. 1–2
- The paper reviews prior work, introduces its architecture and phase-pooling and uncertainty methods, evaluates them on simple datasets, and closes with future directions.Source: §1 · p. 2
2 Prior Work
The work builds on capsule representations and video-based feature learning, while moving toward unsupervised discovery of unknown latent states. It is most closely related to prior unsupervised frame-prediction work, but uses prediction as a proxy for learning representations.
2 Prior Work
- Inspired by capsule units, the work seeks an unsupervised representation of unknown latent states with locally stable “what” and locally linear, equivariant “where” components.The authors state that their representation is theoretically very similar to the capsule representation.Source: §2 · p. 2
- The approach assumes adjacent video samples are latent-space neighbors and future samples can be predicted by linear operations in that space.Source: §2 · p. 2
- Unlike video methods using a pretrained network [13] or tracker supervision [14], this work is unsupervised and most closely related to frame-prediction method [12].The architecture was inspired by and.Source: §2 · p. 2
3 Learning Linearized Representations
The method trains frame-wise codes to vary more linearly over time while retaining predictive information. Phase pooling separates stable feature magnitude from linearly varying position, and latent corrections address uncertain future frames.
3 Learning Linearized Representations
- The encoder is trained to produce codes with lower average local curvature than the input, while a decoder reconstructs predicted frames and an explicit curvature penalty guards against uninformative codes.Prediction extrapolates linearly in code space; prediction error alone may not enforce low-curvature codes because the decoder can map different codes to the same image.Source: §3 · p. 2
3.1 Phase Pooling
- Phase pooling separates feature presence (magnitude) from local position (phase), which can vary linearly as a feature moves.Pooling across features as well as spatial locations can locally linearize arbitrary transformations when those features have a topology.Source: §3.1 · pp. 3–4
- Soft pooling makes the max and argmax channels differentiable: magnitude is an activation expectation, while phase estimates local coordinates under the softmax distribution.The phase vector's dimension corresponds to the topology dimension of the pooling neighborhood; un-pooling places magnitudes at positions specified by phases.Source: §3.1 · pp. 3–4
3.2 Addressing Uncertainty
- Latent variables δ account for unpredictable aspects of future frames, including alternative likely outcomes, by correcting the extrapolated code during training.Their dimension is constrained to be much smaller than the code's, and at test time δ can be selected by sampling from an estimated training-set distribution.Source: §3.2 · pp. 4–5
4 Experiments
The experiments move from shallow features trained on transformed natural-image movies to deep NORB models, using decoded code interpolation and partially predictable sequences to examine linearization and uncertainty.
4.1 Shallow Architecture Trained on Natural Data
- On simulated movie triplets from YouTube videos [4], shallow phase-pooling learned ordered feature groups that tracked local translations; overlapping feature pooling yielded smoother transitions and more complex transformations.Source: §4.1 · p. 6, Figure 3 · p. 7
4.2 Deep Architecture trained on NORB
- For NORB rotations, decoded linear interpolations test whether codes linearize transformations while the decoder synthesizes images.The Siamese-encoder experiment generated intermediate frames by linearly interpolating codes.Source: §4.2 · pp. 6–7, Figure 4 · p. 7
- Deep Architecture 2's fixed code-space extrapolation produced more dataset-like images and lower L2 prediction error than Deep Architecture 1.Source: §4.2 · p. 7
- With phase-pooling, Deep Architecture 3 achieved the best overall L2 prediction error and most visually realistic images; Figure 5 contrasts regularized and uncertainty-trained interpolations.Its pooled feature vector has dimension 192, compared with 4096 in the previous architectures.Source: Figure 5 · p. 8, §4.2 · p. 8
- Under skipped-frame uncertainty, Equation 7 partly restored interpolation sharpness, and δ predicted skip variable s with 94% test accuracy.Source: §4.2 · p. 8
5 Discussion
The paper proposes a loss and architecture for learning locally linearized video features, uses nondeterministic latent variables to address video uncertainty, and identifies longer-term feature stacking as future work.
5 Discussion
- The proposed loss and architecture learn locally linearized features from video.Source: §5 · p. 8
- Nondeterministic functions of the input serve as latent variables for coping with inherent uncertainty in video.Source: §5 · p. 8
- Future work will explore stacking these architectures to linearize more complex features over longer temporal scales.Source: §5 · p. 8