Source-linked AI summary
Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction
Vincent Le Guen, Nicolas Thome
TL;DR
Generalist unsupervised video prediction needs to represent complex visual dynamics even though physical laws rarely apply directly in pixel space. PhyDNet learns a semantic latent space with separate PDE-driven and residual branches, using PhyCell’s prediction-correction mechanism. It outperforms state-of-the-art methods on four generalist datasets and reports benefits from both physical prediction and disentanglement.
Problem
Generalist video prediction requires modeling complex dynamics without semantic labels, while pixel-space physical laws are rarely applicable and full visual content cannot be described by physics alone.
Method
PhyDNet jointly learns a latent space with a two-branch architecture that separates PDE dynamics from residual information, using PhyCell for prediction-correction dynamics in latent space.
Results
PhyDNet outperforms state-of-the-art methods on four generalist datasets, while ablations show gains from both disentanglement and PDE-constrained prediction.
Takeaways & Limitations
PhyDNet extends PDE-constrained prediction beyond fully observed physical phenomena in pixel space and supports missing-data and long-term-forecasting contexts.
Takeaways & Limitations
The approach assumes a conceptual latent space H in which physical dynamics and residual factors are linearly disentangled.
Abstract
from arXiv · showhide
Leveraging physical knowledge described by partial differential equations (PDEs) is an appealing way to improve unsupervised video prediction methods. Since physics is too restrictive for describing the full visual content of generic videos, we introduce PhyDNet, a two-branch deep architecture, which explicitly disentangles PDE dynamics from unknown complementary information. A second contribution is to propose a new recurrent physical cell (PhyCell), inspired from data assimilation techniques, for performing PDE-constrained prediction in latent space. Extensive experiments conducted on four various datasets show the ability of PhyDNet to outperform state-of-the-art methods. Ablation studies also highlight the important gain brought out by both disentanglement and PDE-constrained prediction. Finally, we show that PhyDNet presents interesting features for dealing with missing data and long-term forecasting.
1. Introduction
Unsupervised video prediction must represent complex raw-data dynamics without semantic labels, while pixel-space physics rarely describes generalist videos. PhyDNet addresses this by learning a latent space that disentangles physical dynamics from residual visual information and constrains prediction with PDEs.
- Motivation: Unsupervised video prediction seeks to forecast future frames from previous ones without semantic labels, making representation of raw-data dynamics challenging.The task is relevant to weather forecasting, autonomous driving, reinforcement learning, robotics, and action recognition.
- Motivation: PDE-based physical constraints are effective when dynamics are well described in the input space, but that assumption rarely holds for generalist videos at the pixel level.PhyDNet therefore targets a learned semantic latent space rather than directly constraining pixel predictions.
- Approach: PhyDNet learns an unsupervised latent space H that disentangles physical dynamics from complementary factors such as appearance, details, and texture.The model maps input videos into H, where future frames can be predicted from separated representations.
- Approach: A two-branch sequence-to-sequence model combines a PDE-informed PhyCell branch for physical dynamics with a ConvLSTM branch for residual information.The physical and residual representations are summed before decoding into future frames.
- Approach: PhyCell is a recurrent physical cell that discretizes a broad class of PDEs in H using a prediction-correction paradigm inspired by data assimilation.The architecture is designed to support robust training with missing data and long-term forecasting.
- Results: Experiments report that PhyDNet outperforms state-of-the-art methods on four generalist datasets, with both disentanglement and physical prediction contributing to optimal performance.The paper identifies this as the first physically constrained model to demonstrate these capabilities on four generalist datasets.
2. Related work
Prior unsupervised video-prediction methods use deep architectures, adversarial or stochastic training, geometric constraints, optical flow, and disentanglement. Existing approaches face limits in long-term evolution, general real-world forecasting, or direct applicability of physical equations, motivating PhyDNet’s latent-space design.
- Deep video prediction: Unsupervised multi-step video prediction uses only video data and no manual semantic supervision, with deep sequence models forming a major approach.Prior architectures include sequence-to-sequence LSTMs, recurrent neural networks, and 2D/3D convolutional networks.
- Existing strategies: Adversarial training, stochastic models, geometric constraints, optical flow, and disentanglement have been explored to improve or simplify video prediction.These methods target sharper predictions, constrained image generation, or lower-dimensional representations.
- Existing strategies: Geometric-transformation and optical-flow methods are effective for short-term prediction but degrade quickly as video content evolves.More complex models and memory about dynamics are required for longer-term forecasting.
- Disentanglement: Object-centric and relational disentanglement methods can represent basic physical laws but are evaluated mainly in controlled settings and are not suited to general real-world video forecasting.The cited examples include interactions represented with graphs and laws such as drift, gravity, and springs.
- Physics and PDEs: Physics-informed approaches successfully model phenomena such as climate when equations describe the underlying input-space dynamics, but this assumption is uncommon for generalist videos.Prior work includes PDE discovery, neural PDE approximations, and methods dedicated to specific PDEs.
- Deep Kalman filters: Deep state-space models incorporate Kalman-filter ideas by modeling dynamics in learned latent spaces to handle unobserved phenomena.Examples include the Kalman variational autoencoder and the Recurrent Kalman Network.
3. PhyDNet model for video forecasting
PhyDNet learns a latent space that separates physical dynamics from residual information, using a two-branch architecture and a physically constrained recurrent cell for video forecasting.
- 3.1 PhyDNet disentangling architecture: PhyDNet learns a latent space H where physical dynamics and residual factors are disentangled for accurate future-frame prediction.The physical branch models PDE dynamics, while the residual branch learns unknown phenomena from data.
- 3.1 PhyDNet disentangling architecture: A convolutional encoder maps frames into H, where parallel recurrent branches separately process physical and residual representations before decoding combined predictions.The encoder, decoder, and recurrent block are trained end-to-end for multi-step sequence-to-sequence prediction.
- 3.2 PhyCell: PhyCell models physical dynamics with a PDE-based predictor and a data-driven correction term that incorporates observed input into the latent state.The physical predictor combines spatial derivatives, while the correction uses a learned gate to balance predicted dynamics and embedded observations.
- 3.2.1 Discrete PhyCell: PhyCell discretizes the continuous PDE with forward Euler and can be used as an atomic recurrent cell or stacked to build physically constrained RNNs.Its two-step formulation separates physically constrained prediction from input correction, following a prediction-correction paradigm inspired by data assimilation.
- 3.2.1 Discrete PhyCell: Convolutional filters approximate spatial differential operators and are constrained by a kernel moment loss, while learned gates implement the correction gain.The gate can suppress input influence or reset latent dynamics toward the observed input depending on its value.
- 3.3 Training: PhyDNet is designed to improve robustness when inputs are unreliable, including multi-step forecasting and missing-data settings.The paper states that its gain over standard recurrent models increases in these contexts.
4. Experiments
Experiments across four datasets show that PhyDNet consistently outperforms general and specialized baselines, while ablations support both PDE-constrained prediction and disentanglement. Additional analyses show improved qualitative forecasts, learned dataset-specific dynamics, and robustness to long-term prediction and missing inputs.
- State-of-the-art comparison: PhyDNet outperforms all baselines on all four datasets, including specialized DDPAE and advection-diffusion models.The reported MSE gains over general RNNs are 17 points for Moving MNIST, 6 for Human 3.6, 3 for SST, and 1 for Traffic BJ.
- State-of-the-art comparison: PhyDNet produces accurate future frames across datasets, while DDPAE fails to forecast Moving MNIST frames properly.The qualitative results cover digits, traffic flow, sea-surface temperature phenomena, and human motion.
- Ablation Study: A 1-layer PhyCell outperforms a 3-layer ConvLSTM while using 270,000 versus 3 million parameters.The MSE gains are 50 points for Moving MNIST, 8 for Human 3.6, 7 for SST, and equivalent results for Traffic BJ.
- Ablation Study: Adding disentanglement to PhyCell yields further MSE gains of 25 points for Moving MNIST, 7 for Traffic BJ and SST, and 5 for Human 3.6.The results indicate that physical modeling alone does not cover all information required for general video prediction.
- Ablation Study: Physical regularization improves PhyDNet by 5 MSE points on Moving MNIST, but disabling it improves standalone PhyCell by 7 points.The authors conclude that physical constraints must be incorporated so both branches cooperate.
- Learned physical dynamics: PhyDNet learns dataset-specific PDE dynamics through coefficients combining partial derivatives, with low differential orders dominant for Moving MNIST.The learned coefficients indicate predominantly advective behavior consistent with piecewise-constant digit translation.
- Unreliable inputs: PhyDNet degrades approximately linearly up to 80-step forecasting, and its performance gap over DDPAE increases as missing data rises from 10% to 50%.At 80 steps, PhyDNet reaches similar MSE to DDPAE at 20 steps; with missing inputs, PhyCell relies on latent dynamics by setting K_t = 0.
5. Conclusion
PhyDNet disentangles prior dynamical knowledge from other factors of variation, enabling PDE-constrained prediction beyond fully observed physical phenomena in pixel space. It outperforms state-of-the-art methods on four generalist datasets and introduces a recurrent physical cell that decouples prediction from correction.
- PhyDNet disentangles prior dynamical knowledge from residual factors required for video prediction.
- PhyDNet applies PDE-constrained prediction beyond fully observed physical phenomena in pixel space.
- PhyDNet outperforms state-of-the-art methods on four generalist datasets.
- The recurrent physical cell models PDE dynamics while decoupling prediction from correction.