Source-linked AI summary
Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, Philip S Yu
TL;DR
Spatiotemporal prediction must handle complex higher-order non-stationarity, while prior recurrent transitions have limited ability to model it. The paper proposes MIM, which uses differential hidden-state signals and cascaded recurrent memories to model non-stationary and approximately stationary dynamics. MIM networks achieve state-of-the-art results across four synthetic and real-world prediction tasks.
Problem
Existing spatiotemporal prediction methods do not fully exploit differential features, and their relatively simple transitions inadequately model high-order non-stationarity.
Method
MIM replaces the forget-gate transition with cascaded memory modules that use differential information between neighboring hidden states and can be stacked layer by layer.
Results
MIM networks achieve state-of-the-art results on multiple prediction tasks spanning one synthetic dataset and three real-world datasets.
Takeaways & Limitations
Stacking MIM blocks may gradually stationarize spatiotemporal processes and make their deterministic components more predictable.
Takeaways & Limitations
Prior work had not thoroughly considered high-order non-stationarity in video dynamics, and radar data include long-tail pixel-value distributions.
Abstract
from arXiv · showhide
Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. From Cramer's Decomposition, any non-stationary process can be decomposed into deterministic, time-variant polynomials, plus a zero-mean stochastic term. By applying differencing operations appropriately, we may turn time-variant polynomials into a constant, making the deterministic component predictable. However, most previous recurrent neural networks for spatiotemporal prediction do not use the differential signals effectively, and their relatively simple state transition functions prevent them from learning too complicated variations in spacetime. We propose the Memory In Memory (MIM) networks and corresponding recurrent blocks for this purpose. The MIM blocks exploit the differential signals between adjacent recurrent states to model the non-stationary and approximately stationary properties in spatiotemporal dynamics with two cascaded, self-renewed memory modules. By stacking multiple MIM blocks, we could potentially handle higher-order non-stationarity. The MIM networks achieve the state-of-the-art results on four spatiotemporal prediction tasks across both synthetic and real-world datasets. We believe that the general idea of this work can be potentially applied to other time-series forecasting tasks.
1. Introduction
Spatiotemporal prediction must capture complex spatial and temporal non-stationarity, including higher-order local trends. The paper proposes MIM, which uses differential hidden-state information and cascaded memories to model these dynamics.
- Motivation: Natural spatiotemporal processes combine changing spatial dependencies, temporal dependencies, and deterministic non-stationary trends.These properties appear in tasks such as traffic-flow and weather prediction.
- Motivation: Precipitation forecasting requires modeling diverse higher-order trends across local radar regions.Different regions can exhibit different variation trends over time.
- Research gap: Previous CNN recursions and RNN state transitions provide limited mechanisms for learning complicated non-stationary variations.PredRNN forget gates were reported to be saturated for about 80% of timestamps during precipitation forecasting.
- Method: MIM replaces the LSTM-style forget gate with two embedded long short-term memories that use differential information between neighboring hidden states.The design targets both higher-order non-stationary and approximately stationary components of spatiotemporal dynamics.
- Method and results: Stacking MIM blocks may gradually stationarize spatiotemporal processes and make them more predictable.The authors report state-of-the-art results across multiple prediction tasks, including one synthetic and three real-world datasets.
2. Related Work
Prior work models spatial correlations, temporal dependencies, and stochastic variation, but high-order non-stationarity remains insufficiently addressed. MIM instead uses higher-order differencing and recurrent memory transitions to target this gap.
- Time-series foundations: ARIMA-style forecasting transforms non-stationary processes toward stationarity through differencing before extrapolating the signal.The paper draws inspiration from this signal-and-noise separation strategy.
- Spatiotemporal modeling: Spatiotemporal non-stationarity is more complex than low-dimensional time-series non-stationarity because neighboring pixel distributions vary across space and time.Such processes can be decomposed into deterministic and stochastic components.
- Prior neural methods: Earlier CNN and RNN approaches learn spatial correlations and temporal dependencies, but some sequence models capture only temporal variation.Convolutional recurrent transitions were introduced to unify spatial and temporal modeling.
- Research gap: High-order non-stationarity was not thoroughly considered by prior video-prediction methods with relatively simple temporal transitions.Those transitions were controlled by recurrent gates or feed-forward-network recursion.
- Stochastic modeling: Variational-autoencoder methods model stochastic video dynamics and increase diversity, but are difficult to evaluate and require many runs.This paper instead focuses on making the deterministic component more predictable through stationarization.
3. Methods
MIM replaces simple recurrent forget-gate transitions with cascaded memory modules that use differences between hidden states to model non-stationary and stationary dynamics. Stacking these blocks and connecting their recurrent states is intended to capture higher-order non-stationarity in spatiotemporal sequences.
- Memory In Memory Blocks: MIM replaces the simple temporal forget gate with two cascaded recurrent memory modules for non-stationary and approximately stationary variations.MIM-N captures non-stationary variation from differences between consecutive hidden representations, while MIM-S processes its differential features with outer temporal memory.
- Memory In Memory Blocks: MIM applies differencing in memory transitions rather than all recurrent signals to avoid information loss from over-differencing.The design specifically limits differencing while retaining the differential information needed for non-stationarity modeling.
- Memory In Memory Blocks: MIM-S adaptively reuses the original memory when differential features vanish and overwrites it more when non-stationary dynamics are prominent.Its gating mechanism decides whether to trust the original memory or focus on the differential dynamics supplied by MIM-N.
- Memory In Memory Networks: Stacking multiple MIM blocks is intended to capture higher orders of non-stationarity and gradually stationarize the spatiotemporal process.The architecture delivers hidden states needed to generate differential features and facilitate non-stationarity modeling.
- Memory In Memory Networks: Differencing adjacent hidden states provides differential features for MIM-N, rather than differencing raw images directly.The bottom ST-LSTM encodes local spatiotemporal variation into hidden representations before temporal differencing.
- Memory In Memory Networks: The network combines one ST-LSTM with multiple MIMs and uses diagonal hidden-state, horizontal memory-cell, and zigzag state-transition paths.The recurrent paths deliver hidden states for differential modeling and propagate C, N, S, and M memories through the network.
4. Experiments
Experiments evaluate MIM across synthetic and real-world spatiotemporal prediction tasks, using common training settings and quantitative, qualitative, ablation, and sensitivity analyses. MIM generally achieves stronger prediction quality, with benefits attributed to its cascaded memory design and handling of non-stationary dynamics.
- Common settings: Experiments use four datasets, with four layers comprising one ST-LSTM and three MIM blocks in the common setup.Each MIM block uses 64 feature channels; models are trained with ℓ2 loss and ADAM at a learning rate of 0.001.
- Moving MNIST: MIM produces less blurry Moving MNIST predictions than other models, especially when occlusions create high-order non-stationarity.The standard setting predicts 10 future frames from 10 observed frames, and evaluation uses SSIM, MSE, and MAE.
- Moving MNIST: 16%: MIM increases the generated-image sharpness score over PredRNN on Moving MNIST.MIM outperforms its ST-LSTM baseline, while MIM* also outperforms its Causal LSTM baseline, supporting a generic memory-transition mechanism.
- Moving MNIST: The complete MIM block performs best when its MIM-N and MIM-S modules are interlinked, while removing either module reduces performance.Both ablated variants significantly improve over PredRNN in MSE and MAE, with the model without MIM-N performing slightly better than the model without MIM-S.
- Moving MNIST: Three MIM blocks perform best on Moving MNIST, balancing inadequate modeling with the training difficulty of excessively deep recurrent models.Models with 2, 3, and 4 MIM blocks were evaluated.
- Memory analysis: MIM mitigates saturated forget gates and better uses long-term variations by separating short-term tendencies from longer-term memory handling.For MIM, saturation is measured using |T_t/C_t−1| < 0.1, whereas other models use f_t < 0.1.
- TaxiBJ Traffic Flow: MIM achieves the best qualitative performance on TaxiBJ, showing the lowest prediction–ground-truth difference intensities across predicted frames.TaxiBJ inputs contain four historical traffic-flow images and predict the next four images.
- Radar Echo: MIM consistently outperforms other models on radar-echo MSE and CSI, including thresholds associated with severe weather conditions.The forecasting horizon covers 10 timestamps at six-minute intervals, or the next hour; lower MSE and higher CSI indicate better results.
5. Conclusions
The paper frames high-order spatiotemporal non-stationarity as varying pixel-intensity trends and reduces their order through differencing. It proposes stacked MIM blocks with cascaded recurrent modules and reports state-of-the-art prediction performance across four datasets.
- High-order non-stationarity is formulated as high-order polynomials over statistically varying pixel-intensity trends.
- Differencing operations reduce the order of non-stationary polynomials, following an idea from time-series analysis.
- Stacked MIM blocks use differential inputs layer by layer to model complicated spatiotemporal variations.Each MIM block uses two cascaded recurrent modules for non-stationary and approximately stationary components.
- MIM achieves state-of-the-art prediction performance on four datasets spanning synthetic and real-world spatiotemporal tasks.