Source-linked AI summary
UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation
Daikun Liu, Xin Zhan, Teng Wang, Xiaoping Wang, Changyin Sun
TL;DR
Single-frame future-scene prediction can lack reliable motion cues, and prior approaches may require histories or structured priors while often omitting explicit motion fields. UniDynamics uses an event-RGB pair with diffusion to predict future RGB, depth, and flow, and reports higher-quality, more coherent predictions on VKitti2 and DSEC, particularly under high-speed motion blur.
Problem
Single-frame extrapolation lacks reliable motion cues, while existing approaches may require long histories or control priors and often omit explicit motion-field modeling.
Source: §1 · p. 2Method
UniDynamics conditions diffusion on an event-RGB pair, aligning event cues with RGB and using a dynamics space to interactively model depth, flow, and appearance.
Source: §1 · pp. 2–3Results
Experiments on VKitti2 and DSEC report higher-quality, more spatiotemporally coherent future-scene generation and perception, particularly under high-speed motion blur.
Source: §1 · p. 3Takeaways & Limitations
Event cues offer an alternative motion prior for single-frame extrapolation, while joint RGB, depth, and flow prediction targets physically plausible, coherent future scenes.
Source: §1 · p. 2, §5 · p. 15
Abstract
from arXiv · showhide
We propose UniDynamics, a diffusion-based framework for future 4D dynamic scenes (RGB, depth, and optical flow) generation from a single event-RGB pair, without requiring long histories or control priors as in existing methods, while explicitly modeling future motion fields. The core idea is to leverage event streams to offer an alternative motion prior for single-RGB extrapolation, and to enforce geometric and motion constraints throughout generation via multimodal modeling. Specifically, we design an Event Latent Enhancement (ELE) module to align and enhance event latents into diffusion-injectable conditioning features, providing robust initial motion priors and reliable texture/structure cues. We further introduce a Perceptual Dynamics Space (PDS) embedded in the multi-scale U-Net, which decouples and adaptively interacts depth and flow while continuously feeding back constraints to appearance features, improving geometric-motion consistency for physically plausible and spatiotemporally coherent prediction. Experiments on VKitti2 and DSEC demonstrate state-of-the-art performance, producing high-quality, temporally coherent, and 4D-consistent future predictions, especially under challenging high-speed motion blur.
1 Introduction
UniDynamics uses an event-RGB pair to predict future appearance, depth, and flow, addressing the limits of single-frame extrapolation without motion priors. Its event conditioning and joint multimodal modeling target coherent predictions, with reported gains on VKitti2 and DSEC, especially under high-speed motion blur.
1 Introduction
- Single-frame future extrapolation can lack reliable motion priors, while motion blur degrades appearance cues and many methods omit explicit motion-field modeling.Event streams provide continuous motion observations without long histories and remain informative under fast-motion blur.Source: §1 · p. 2
- Event Latent Enhancement aligns and enhances event features against the RGB latent, producing diffusion-compatible conditions with initial momentum and texture or boundary cues.The RGB latent serves as an appearance reference for cross-attention alignment.Source: §1 · pp. 2–3
- UniDynamics conditions diffusion on a current event-RGB pair to jointly predict future RGB, depth, and optical flow.It treats events as complementary to RGB and uses a shared latent space for appearance, geometry, and motion.Source: §1 · pp. 2–3
- Perceptual Dynamics Space decouples depth and flow and feeds their constraints back to appearance features to improve structural stability and spatiotemporal coherence.Task-specific adapters and gated interactions reduce multi-task interference and flow-induced structural perturbations.Source: §1 · p. 3
- Experiments on VKitti2 and DSEC report higher-quality, more spatiotemporally coherent future-scene generation and perception, particularly under high-speed motion blur.The reported results cover synthesized event-based VKitti2 and zero-shot generalization studies on DSEC.Source: §1 · p. 3
2 Related Work
Prior driving-scene prediction methods add structured conditions or use vision-only histories, while event-based and event-RGB methods have their own limitations. UniDynamics is motivated by combining event and RGB cues with explicit motion-field prediction for coherent future scenes.
RGB-based Future Scene Prediction Methods
- Driving-focused generative models add conditions such as actions, text, trajectories, or other priors, while vision-only approaches may have lower fine-grained fidelity or require long histories.Both patterns can increase complexity, adaptation cost, memory, or latency in deployment settings.Source: §2 · pp. 3–4
Event-based Future Scene Prediction Methods
- Event-only predictors can break down in static or low-texture regions, while DESSERT's long-horizon prediction suffers from error accumulation; these methods also lack explicit motion-field modeling.Event-RGB complementarity motivates jointly modeling 4D scenes.Source: §2 · p. 4
Event-RGB-fusion-based Vision Tasks
- Event-RGB fusion has improved current or past depth and flow perception, but the cited approaches do not target future prediction.Prior fusion techniques include recurrent fusion, cross-modal attention, and event-RGB extensions of flow estimators.Source: §2 · p. 4
3 UniDynamics
UniDynamics uses event cues to condition diffusion from a single event–RGB pair, then jointly predicts future RGB, depth, and optical flow in a shared latent space. Its ELE module supplies event-based motion and appearance cues, while PDS models depth and flow and feeds their constraints back into RGB generation.
3.1 Preliminaries
- SVD provides the diffusion backbone, but single-frame future extrapolation remains ill-posed without initial motion cues, limiting temporal coherence.UniDynamics predicts future RGB alongside depth and optical flow from a current frame and its corresponding event stream.Source: §3.1 · p. 5, §3.2 · p. 5
3.2 Overview
- A shared image-VAE latent represents appearance, geometry, and motion; the diffusion backbone generates RGB latents while depth and flow are refined through U-Net interactions.The shared representation avoids extra modality-specific encoders and supports cross-modal alignment.Source: §3.2 · pp. 5–6
3.3 Event Latent Guided Motion Injection
- ELE aligns event features with RGB features and enhances the event latent with RGB appearance, yielding diffusion-compatible motion and texture cues.Events are treated as a soft initial-momentum prior rather than a hard condition, helping avoid entanglement with global scene evolution and providing guidance when RGB is blurred.Source: §3.3 · pp. 6–7
3.4 Perceptual Dynamics Space for Depth-Flow Modeling
- PDS decouples depth and flow into task-specific adapters, enables gated bidirectional exchange, and feeds geometry and motion cues into U-Net spatial and temporal layers, respectively.Depth adapters use 2D convolutions for spatial structure, while flow adapters use 3D convolutions for spatiotemporal dependencies.Source: §3.4 · pp. 7–8
3.5 Training and Inference
- Training jointly generates pseudo-history, current, and future RGB-depth-flow outputs using weighted generation and perception losses; inference iteratively denoises from RGB, event, and noise latents.The same VAE encoder and decoder are shared across modalities, and the loss includes latent RGB, depth, flow, and event terms plus depth scale-shift-invariant and flow L1 losses.Source: §3.5 · p. 9
4 Experiments
UniDynamics predicts future RGB, depth, and optical flow from event–RGB input, with experiments spanning synthetic E-VKitti2 and zero-shot real-world datasets. It reports coherent, high-fidelity predictions, including under motion blur, and ablations support the roles of event cues and decoupled depth–flow modeling.
4.1 Datasets and Metrics
- Evaluation uses E-VKitti2 for training and testing, with zero-shot tests on DSEC, MVSEC, and M3ED; it measures RGB quality and coherence, depth accuracy, and flow accuracy.Inputs use a 100 ms event window split into 15 bins; inference predicts nine future RGB frames with corresponding depth and optical flow.Source: §4.1 · p. 10, §4.2 · p. 10
4.3 Main Results
- UniDynamics achieves the lowest E-VKitti2 FID (39.1) and FVD (223.6), with more accurate depth and lower-error, more robust optical flow.It is the only compared method supporting simultaneous future RGB, depth, and optical-flow generation.Source: §4.3 · pp. 11–12, Table 1 · p. 11
- Under simulated high-speed motion blur, event signals provide texture and structural cues that compensate for blurred RGB, supporting robust future prediction.The authors report that single blurred RGB input makes UniFuture variants and UniDynamics without events perform poorly across generation and perception metrics.Source: §4.3 · p. 12, Table 2 · p. 13
- On zero-shot real-world datasets, UniDynamics improves FVD and perception substantially, though its FID is slightly inferior to UniFuture-Depth on DSEC and MVSEC.The paper also reports better generation quality on M3ED and overall more coherent, accurate forecasts across domains.Source: §4.3 · p. 13, Table 3 · p. 14
4.4 Ablation Study
- Ablations show that removing events degrades FID from 39.1 to 46.8 and FVD from 223.6 to 577.1, while removing depth or flow harms complementary geometry or motion constraints.Removing ELE or PDS task adapters and gated interaction also degrades generation and perception; the authors attribute this to weaker event-cue injection or entangled geometry-motion features.Source: §4.4 · pp. 14–15, Table 4 · p. 15
5 Conclusion
UniDynamics uses an event–RGB pair to generate future RGB, depth, and optical flow, combining ELE event cues with PDS depth–flow decoupling. Experiments on synthetic and real datasets demonstrate high-quality, temporally coherent appearance prediction and accurate geometric and motion perception.
5 Conclusion
- UniDynamics conditions diffusion on an event–RGB pair to generate future RGB, depth, and optical flow.ELE supplies initial momentum and appearance guidance, while PDS decouples depth and flow and couples them to U-Net denoising.Source: §5 · p. 15