Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning

• Pixel-level future-frame prediction is challenging because video motion has both high-dimensionality and long-term dynamics.Source: §1 · p. 1
• Existing RNN cell states mainly capture relations within the current sequence, making long-term context difficult to infer from limited motion such as a short walking clip.Earlier methods that simplify motion also did not consider long-term frame dynamics.Source: §1 · p. 1
• LMC-Memory uses two-phase alignment learning to store context from long sequences and match it to short inputs, enabling recall of long-term motion context.The phases store long-term context and then match short-term sequences with stored contexts.Source: §1 · p. 2
• Memory query decomposition represents overall motion as low-dimensional local dynamics, allowing suitable context to be recalled for each input part individually.The paper says this can boost alignment between input and stored long-term context.Source: §1 · p. 2

2.1. Video Prediction

• Earlier video-prediction methods address deterministic changes, but many rely on input-sequence frame relations and struggle to encode long-term dynamics from limited inputs.Prior work also explored stochastic modeling of multiple futures, whereas this paper focuses on deterministic changes.Source: §2.1 · p. 2

2.2. Memory Network

• Memory-augmented networks have been used in several vision tasks; this work applies external memory to storing and recalling long-term motion context from limited inputs.Figure 1 depicts a lower path recalling context from LMC-Memory and an upper recurrent path predicting frames using that context.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

3.1. Motion Context-Aware Video Prediction

• The framework uses difference frames to query external LMC-Memory for long-term context, then supplies that context alongside the input sequence for recurrent future-frame prediction.The memory path recalls context while the prediction path generates each next frame; generated frames feed subsequent predictions.Source: §3.1 · pp. 3–4
• A ConvLSTM processes spatial features over time, while attention refines the embedded memory context before the decoder generates the next frame.The cell state and memory feature determine channel-wise attention; the refined context is combined with the ConvLSTM output state.Source: §3.1 · pp. 3–4

3.2. LMC-Memory with Alignment Learning

• Alignment learning alternates between storing context from long-term sequences and training short-term matching while keeping the stored memory fixed.In the matching phase, only short-term sequences are used at inference; during training, other network weights are optimized to predict future frames using recalled memory.Source: §3.2 · pp. 4–5
• The long-term encoder divides motion features into local parts and uses each local feature as an individual memory query.The encoder extracts context from difference frames before decomposing its feature into local queries.Source: §3.2 · p. 4
• LMC-Memory represents stored contexts as memory items and uses cosine-similarity softmax addressing to weight slots for each local query.Each local memory output is a context feature for a corresponding spatial location.Source: §3.2 · p. 4

4.1. Datasets

• The evaluation uses Moving-MNIST, KTH Action, and Human 3.6M, covering synthetic digits and natural human-action videos.The datasets contain 10,000/5,000 training/test sequences for Moving-MNIST, six action types for KTH, and 17 action scenarios for Human 3.6M.Source: §4.1 · p. 5

4.2. Implementation

• Experiments use 10 input frames and predict 30 frames for Moving-MNIST or 40 frames for KTH and Human 3.6M, with a 100-slot memory.The model uses four-layer ConvLSTMs and Adam with a learning rate of 0.0002.Source: §4.2 · p. 5

4.3. Evaluation

• On Moving-MNIST and KTH, the proposed method outperforms other state-of-the-art methods, especially in long-term prediction and LPIPS.The KTH comparison also reports faster inference than other methods.Source: Table 1 · p. 6, §4.3 · p. 6, Table 2 · p. 7
• For KTH inputs beginning mid-motion, the proposed predictions preserve leg shape and long-term trajectories, unlike the other methods.The qualitative results use 10 input frames.Source: Figure 4 · p. 7, §4.3 · p. 7
• On Human 3.6M, the proposed method outperforms other state-of-the-art methods for both 40-frame prediction and the last 10 frames.Its qualitative predictions capture long-term direction changes, while other methods show a person disappearing at the corner.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Ablation Study

• Ablations show that both LMC-Memory and local motion-context queries contribute to last-10-of-40-frame performance, with the full model strongest on LPIPS.The authors report that locally manipulated queries boost memory effects and that LMC-Memory adds marginal computational cost.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Memory Addressing

• Memory-addressing vectors are more similar for long- and short-term sequences of the same action than for different actions.The authors interpret this as alignment of memory features belonging to similar actions.Source: §4.5 · p. 8, Figure 7 · p. 8

• The method outperforms state-of-the-art methods with sophisticated RNNs, with the difference especially notable in long-term conditions.Source: §5 · p. 8
• LMC-Memory with alignment learning stores long-term contexts from training data and recalls suitable context from limited inputs.Source: §5 · p. 8
• Memory query decomposition separates overall motion into low-dimensional dynamics to address motion-context high dimensionality.Source: §5 · p. 8
• The proposed method’s effectiveness is analyzed quantitatively and qualitatively.Source: §5 · p. 8