Source-linked AI summary
Folded Recurrent Neural Networks for Future Video Prediction
Marc Oliu, Javier Selva, Sergio Escalera
TL;DR
Future video prediction must handle variable content and multiple feasible futures while controlling recurrent error propagation. The paper introduces folded recurrent networks built from bijective GRUs with shared encoder-decoder states. The approach outperforms state-of-the-art prediction results on MMNIST and UCF101 and achieves competitive KTH results with lower memory and computational cost.
Problem
Future video prediction faces variable video content, multiple feasible futures for a given past sequence, and error propagation through recurrent predictions.
Source: §1 · p. 2Method
The paper uses bijective GRUs to build recurrent auto-encoders with shared encoder-decoder states and a bidirectional mapping between layer inputs and outputs.
Source: §1 · p. 2, §3.1 · p. 5Results
The fRNN outperforms state-of-the-art prediction results on MMNIST and UCF101 and achieves competitive KTH results with 2 and 3 times less memory usage and computational cost than the best-scored approach.
Source: §5 · p. 15Takeaways & Limitations
The shared-state topology lets the encoder or decoder run independently for encoding or prediction, avoiding re-encoding predictions and reducing computational cost.
Source: §1 · p. 2, §5 · p. 15
Abstract
from arXiv · showhide
Future video prediction is an ill-posed Computer Vision problem that recently received much attention. Its main challenges are the high variability in video content, the propagation of errors through time, and the non-specificity of the future frames: given a sequence of past frames there is a continuous distribution of possible futures. This work introduces bijective Gated Recurrent Units, a double mapping between the input and output of a GRU layer. This allows for recurrent auto-encoders with state sharing between encoder and decoder, stratifying the sequence representation and helping to prevent capacity problems. We show how with this topology only the encoder or decoder needs to be applied for input encoding and prediction, respectively. This reduces the computational cost and avoids re-encoding the predictions when generating a sequence of frames, mitigating the propagation of errors. Furthermore, it is possible to remove layers from an already trained model, giving an insight to the role performed by each layer and making the model more explainable. We evaluate our approach on three video datasets, outperforming state of the art prediction results on MMNIST and UCF101, and obtaining competitive results on KTH with 2 and 3 times less memory usage and computational cost than the best scored approach.
1 Introduction
Future video prediction can use unlabelled data for a range of video-related tasks, but it must handle variable content and multiple possible futures. The paper proposes a shared-state recurrent auto-encoder to address model cost and prediction error propagation.
1 Introduction
- Future video prediction can leverage unlabelled data for tasks including action recognition, task planning, weather prediction, optical flow estimation, and new view synthesis.The introduction motivates the task in part through its unsupervised-learning potential.Source: §1 · pp. 1–2
- Video prediction must capture content variability while managing recurrent error propagation under multiple feasible futures for the same input sequence.Uncontrolled propagation can produce blurry predictions that affect subsequent frames.Source: §1 · p. 2
- The proposed recurrent auto-encoder shares states between encoder and decoder using bijective GRUs, reducing computation and memory while mitigating error propagation.The paper also presents stratified representations, a noisy identity mapping, and layer removal for model analysis.Source: §1 · p. 2
2 Related work
Prior future-prediction work uses recurrent auto-encoders and varied strategies for representing space, time, motion, and possible futures. The paper situates its approach among these architectures and prediction methods.
Building Blocks
- Future-frame models commonly use auto-encoders, with CNNs for spatial processing and LSTMs or convolutional recurrent layers for temporal dynamics.Some approaches condition the transformation on actions or other information to reduce the space of possible futures.Source: §2 · p. 3
Bridge connections
- Video Ladder Networks and Recurrent Ladder Networks use bridge connections, while the proposed alternative shares recurrent states between encoder and decoder.The passage attributes higher parameter and memory requirements, greater computation, and reduced explainability to skip connections compared with state sharing.Source: §2 · p. 3
Prediction atom
- Prediction architectures either generate pixels directly or predict motion or predefined features that are then used to generate frames.Examples include optical flow, convolutional kernels, affine transforms, and human pose vectors.Source: §2 · p. 4
Loss and GANs
- Some methods use GANs alongside traditional losses to select among possible futures and promote realistic, coherent sequences.The passage notes that L2 or MSE losses tend to average possible futures.Source: §2 · p. 4
Disentangled Motion/Content and Feedback Predictions
- Some approaches represent motion and content separately or predict them jointly, while recurrent models use their own predictions as inputs when generating beyond the trained horizon.Feedback can magnify mistakes over time; the passage describes fRNN as allowing encoder and decoder to run independently any number of times, unlike the cited single-layer recurrent auto-encoder.Source: §2 · p. 4
3 Proposed method
The proposed fRNN builds an encoder and decoder from bijective GRU layers that share recurrent states. This topology supports stratified representations, lower resource use, and sequential prediction without repeatedly re-encoding predictions.
3 Proposed method
- Stacked bijective GRU layers form an fRNN with shared encoder-decoder states, stratified information, lower resource requirements, and mitigated error propagation.The shared-state design also supports layer removal for analyzing model behavior.Source: §3 · pp. 4–5
3.1 Bijective Gated Recurrent Units
- A bijective GRU uses a second set of gates to define an inverse mapping, treating the input as a state updated using the forward function's output.The paper names this double mapping a bijective GRU.Source: §3.1 · p. 5
3.2 Folded Recurrent Neural Network
- In the folded recurrent auto-encoder, the decoder updates shared states so predictions need not be fed back through the encoder-decoder, and errors at one layer do not propagate into deeper layers.Compared with bridge-connected alternatives, the bGRU has fewer weights; those alternatives require both encoder and decoder at each step, doubling memory and increasing computation by a factor of 2.88 to 3.Source: Figure 1 · p. 5, §3.2 · p. 6
3.3 Training Folded RNNs
- Training first passes g ground-truth frames through the encoder, then applies the decoder p times to produce predictions using only half the memory requirements.Applying the decoder without further ground-truth inputs encourages learning video dynamics and prevents learning an identity model.Source: §3.3 · pp. 6–7
4 Experiments
Experiments compare ten-frame forecasts across MMNIST, KTH, and UCF101, then assess qualitative rollouts and layer-removal behavior. fRNN leads on MMNIST and UCF101, with competitive, more stable KTH results.
4.1 Data and evaluation protocol
- Tests span MMNIST, KTH, and high-variability UCF101, with 10 input frames used to predict the next 10 under MSE, PSNR, and DSSIM.Source: §4.1 · p. 7
4.2 Methods
- Baselines include last-frame prediction and RLadder, which substitutes bridge connections for state sharing and nearly triples computational cost.Source: §4.2 · p. 8
4.3 Quantitative analysis
- fRNN leads on MMNIST and UCF101; on KTH it is comparable to Villegas et al. and more stable over time, while RLadder scores best at higher resource cost.Source: §4.3 · pp. 8–10, Table 2 · p. 9
4.4 Qualitative analysis
- Across qualitative rollouts, MMNIST digits stay sharp and bounce correctly, KTH blur plateaus after three predictions, and UCF static regions remain sharp while backgrounds recover after occlusion.Source: §4.4 · pp. 10–11
4.5 Representation stratification analysis
- Removing bGRU layers exposes progressively specialized dynamics: early layers retain linear motion, while deeper layers group digit parts and encode coherent whole-digit bouncing.Source: §4.5 · pp. 12–15
5 Conclusions
Folded Recurrent Neural Networks use bijective GRUs to share encoder–decoder states, reducing recurrent-autoencoder costs while supporting competitive video prediction and layer-based analysis of learned representations.
- It outperformed state-of-the-art results on MMNIST and UCF101, and was competitive on KTH with 2 and 3 times less memory and computational cost than the best-scored approach.Source: §5 · p. 15
- Bijective GRUs pass information horizontally between encoder and decoder states, so only the encoder or decoder must run for input encoding or prediction.This lowers computational and memory cost compared with equivalent recurrent auto-encoders.Source: §5 · p. 15
- The model can limit and recover from blur by preventing its propagation from low- to high-level dynamics.Source: §5 · p. 15
- Layer removal demonstrates stratified representations: removing a layer eliminates its behaviors while leaving lower-level behaviors intact, supporting topology optimization and model explainability.Source: §5 · p. 15