Source-linked AI summary
Toward Physically Consistent Driving Video World Models under Challenging Trajectories
Jiawei Zhou, Zhenxin Zhu, Lingyi Du, Linye Lyu, Lijun Zhou, Zhanqian Wu, Hongcheng Luo, Zhuotao Tian, Bing Wang, Guang Chen, Hangjun Ye, Haiyang Sun, Yu Li
TL;DR
Existing driving world models are limited by safe, nominal training data and often fail on physically inconsistent or rare challenging trajectories. PhyGenesis rectifies potentially invalid conditions, generates physics-consistent multi-view videos, and trains on real-world plus CARLA-derived extreme scenarios. The paper reports improved visual fidelity and physical consistency, particularly under challenging trajectories, while its trajectory representation starts from potentially physics-violating 2D inputs.
Problem
Existing driving world models struggle with physics-violating trajectories and rare dynamics because they lack physical feasibility reasoning and are mostly trained on safe, nominal driving data.
Method
PhyGenesis uses a Physical Condition Generator to rectify arbitrary trajectories into physically feasible 6-DoF motions, then a Physics-Enhanced Video Generator trained with heterogeneous real-world and CARLA data.
Results
PhyGenesis outperforms existing methods in visual fidelity and physical consistency, particularly under challenging trajectory conditions.
Takeaways & Limitations
PhyGenesis offers a building block for more reliable simulation of safety-critical collisions and off-road behaviors in simulation-driven autonomous-driving evaluation and safety testing.
Takeaways & Limitations
The framework accepts 2D trajectories that can violate physical constraints, so its stated task begins from potentially flawed trajectory inputs rather than only physically feasible conditions.
Abstract
from arXiv · showhide
Video generation models have shown strong potential as world models for autonomous driving simulation. However, existing approaches are primarily trained on real-world driving datasets, which mostly contain natural and safe driving scenarios. As a result, current models often fail when conditioned on challenging or counterfactual trajectories-such as imperfect trajectories generated by simulators or planning systems-producing videos with severe physical inconsistencies and artifacts. To address this limitation, we propose PhyGenesis, a world model designed to generate driving videos with high visual fidelity and strong physical consistency. Our framework consists of two key components: (1) a physical condition generator that transforms potentially invalid trajectory inputs into physically plausible conditions, and (2) a physics-enhanced video generator that produces high-fidelity multi-view driving videos under these conditions. To effectively train these components, we construct a large-scale, physics-rich heterogeneous dataset. Specifically, in addition to real-world driving videos, we generate diverse challenging driving scenarios using the CARLA simulator, from which we derive supervision signals that guide the model to learn physically grounded dynamics under extreme conditions. This challenging-trajectory learning strategy enables trajectory correction and promotes physically consistent video generation. Extensive experiments demonstrate that PhyGenesis consistently outperforms state-of-the-art methods, especially on challenging trajectories. Our project page is available at: https://wm-research.github.io/PhyGenesis/.
1 Introduction
Existing driving world models struggle with physically inconsistent or rare challenging trajectories because they lack trajectory-feasibility reasoning and physics-rich training. PhyGenesis addresses both issues through trajectory rectification, physics-enhanced video generation, and heterogeneous real-plus-CARLA training data.
- Challenges: Existing models often produce severe artifacts and structural failures when imperfect simulator- or planner-generated trajectories violate physical constraints.These models largely behave as condition-to-pixel translators without explicit physical reasoning.
- Challenges: Safe, nominal real-world training data leaves prior models unable to generate realistic dynamics for rare events such as collisions and off-road departures.The resulting videos can remain physically inconsistent even when input trajectories are feasible.
- Approach: PhyGenesis combines a Physical Condition Generator with a Physics-enhanced Video Generator to handle trajectory feasibility and physically consistent multi-view synthesis.The first module resolves physical conflicts before the second synthesizes high-fidelity videos.
- Training data: The framework is trained on a heterogeneous physics-rich dataset combining real-world driving logs with extreme CARLA scenarios.CARLA supplies diverse events such as collisions and off-road departures that provide supervision for complex object–environment interactions.
- Approach: The Physical Condition Generator converts arbitrary trajectory inputs into feasible 6-DoF vehicle motions using counterfactual trajectory rectification training.This training equips the model with physical priors for resolving physics-violating trajectories.
2 Related Work
Existing driving world models typically specialize in either nominal scenarios or high-risk synthesis rather than covering both. PhyGenesis bridges this gap with physical trajectory correction, physics-enhanced video generation, and combined real-world and simulation-derived extreme data.
- Research gap: Prior driving world models generally handle either nominal scenarios or high-risk synthesis, but rarely both.PhyGenesis is presented as a framework that bridges these two capabilities.
- PhyGenesis: PhyGenesis bridges the gap through a Physical Condition Generator and a Physics-Enhanced Video Generator trained on real-world and simulation-derived extreme data.The combined design targets both trajectory feasibility and physically consistent video generation.
3 Method
PhyGenesis combines heterogeneous real and simulated multi-view data with a two-stage pipeline that rectifies potentially invalid trajectories before generating physically consistent videos. Its physical condition generator models 6-DoF motion using spatial, agent, map, and time-wise components, while PE-MVGen synthesizes conditioned multi-view outputs.
- Heterogeneous Multi-view Data: The heterogeneous dataset combines nominal real-world driving data with simulated physically challenging scenarios, including collisions and off-road departures.The simulated data are collected with CARLA and sensor configurations aligned to nuScenes.
- Physical Condition Generator: PhyGenesis rectifies potentially physics-violating 2D trajectories into physically plausible 6-DoF motions before video generation.The 6-DoF representation includes x, y, z, pitch, yaw, and roll, capturing extreme vertical and rotational changes that 2D coordinates miss.
- Physical Condition Generator: Spatial cross-attention grounds agent tokens in perspective-view features, agent self-attention models vehicle interactions, and map cross-attention improves off-road awareness.The agent-interaction layer is designed to help resolve overlapping and penetration conflicts.
- Physical Condition Generator: The time-wise output head combines step-specific temporal embeddings with a temporal convolutional network to capture abrupt collision dynamics.Compared with a standard MLP, it produces an instantaneous velocity drop at impact rather than a gradual decrease.
- Physical Condition Generator: The physical model uses weighted trajectory distance loss that emphasizes collision or off-road windows and agents involved in those events.The event-window and physical-agent weights focus optimization on critical physical moments.
- Physics-Enhanced Multi-view Video Generator: PE-MVGen encodes multi-view clips and projected 3D layouts, reshapes views into the spatial axis, and uses shared self-attention to model cross-view dependencies.It is adapted from Wan2.1 and conditions generation on camera-view control images derived from 3D agent boxes and map polylines.
4 Experiment
PhyGenesis is evaluated across nominal, physics-violating, and ground-truth trajectory settings using real and simulated driving data. It achieves strong physical consistency and visual quality, while ablations show benefits from trajectory rectification and heterogeneous physics-rich co-training.
- Experimental Setup: The evaluation uses nuScenes, CARLA Ego, and CARLA ADV, with 150 clips sampled per test split and CARLA videos translated into nuScenes style for fair comparison.nuScenes uses nominal or stress-test conditions, while CARLA includes physically violating trajectories.
- Performance of PhyGenesis: PhyGenesis achieves the best visual quality and physical consistency across datasets, with the largest gains on physically challenging CARLA sets.Physics-violating inputs often cause deformation and penetration in prior methods.
- Performance of PhyGenesis: Under the nuScenes stress test, PhyGenesis remains more physically consistent with unchanged first-frame conditions and corrupted, out-of-distribution trajectories.The stress test scales ego speed and retains collision cases.
- Physics-Enhanced Video Generator: With ground-truth trajectories, prior baselines still struggle with collisions and off-road events, whereas PE-MVGen improves visual quality, physical consistency, and condition following through physics-rich co-training.The comparison reports visual realism, physical consistency, and controllability error.
- Ablation Study: The Physical Condition Generator consistently reduces 6-DoF trajectory error and mitigates penetration and deformation artifacts under physics-violating conditions.A qualitative example shows an invalid trajectory rectified into a guardrail collision followed by a stop.
- Ablation Study: On CARLA ADV, heterogeneous co-training reduces FVD from 89.83 to 77.83 and increases preference from 0.13 to 0.53.The study attributes sharper, more physically coherent dynamics to training with physics-rich CARLA data.
5 Conclusion
PhyGenesis combines trajectory-feasibility handling with physics-enhanced video generation to produce high-fidelity, physically consistent multi-view driving videos. It performs particularly well under challenging trajectories and supports simulation of safety-critical events.
- Conclusion: PhyGenesis outperforms existing methods in visual fidelity and physical consistency, particularly under challenging trajectory conditions.The framework targets more reliable simulation of collisions and off-road behaviors.
- Conclusion: The framework aligns planner- or simulator-provided trajectories with physically consistent visual world modeling for simulation-driven evaluation and safety testing.Its stated scope includes safety-critical events such as collisions and off-road behaviors.
A User Study Setting
The user studies compare videos generated by different baselines under identical conditions and convert participant selections into normalized human-preference scores. Separate studies use sampled scenes and a fixed response count across participants.
- User Study Protocol: Participants select the video with the highest visual quality and may choose an uncertain option.An uncertain response distributes credit equally among all compared videos.
- Scoring: A selected video receives 1 point, while an uncertain response gives each of N videos 1/N point; the final score divides total points by comparisons.The questionnaire is shown in Figure 11.
- Sampling and Responses: The studies are conducted separately using videos from five scenes, with each preference score based on 150 responses from 30 participants.Across all studies, the authors collect 1,050 responses.
B Style Transfer Model Used for Video Generation
The style-transfer model converts CARLA videos into the nuScenes visual style for fair baseline comparison. It conditions generation on per-frame depth and video-level text while training entirely on nuScenes data.
- Purpose and Architecture: The transfer model translates CARLA clips into nuScenes appearance because comparison baselines are primarily trained on nuScenes.Its architecture is shown in Figure 12.
- Conditions: Generation is conditioned on per-frame depth maps and detailed video descriptions without using the initial frame.Depth Anything V2 extracts depth, while Qwen2.5-VL produces the video description.
- Training: The model is trained entirely on nuScenes data to learn its appearance characteristics more faithfully before translating CARLA videos at inference.The transfer model follows the rectified-flow framework and uses a flow-matching objective.
C Implementation Details of Different Baselines
The evaluation standardizes video length while accommodating each baseline’s inference limits and records differing resolution and conditioning inputs.
- All methods are evaluated on 33-frame videos, with UniMLVG requiring two autoregressive stages because it generates at most 19 frames per pass.The second UniMLVG pass uses frame 14 as its conditioning frame for frames 14–33.
- PhyGenesis generates at 448 × 800, while MagicDriveV2 and DiST-4D use 424 × 800 and UniMLVG uses 256 × 448.
- PhyGenesis, MagicDriveV2, and UniMLVG condition generation on the initial RGB frame, whereas DiST-4D additionally uses its initial-frame depth map.The DiST-4D depth map is obtained through its official preprocessing pipeline.
D Physical-Challenging Scenario Construction in CARLA Ego and CARLA Adv
The CARLA construction preserves standard route and scene context while perturbing one vehicle to produce challenging behaviors centered on either the ego or a nearby adversarial agent.
- CARLA rollouts preserve the original map, traffic, weather, and background behaviors while perturbing one designated vehicle on a valid predefined route.The perturbation is based on the Bench2Drive routing setup.
- Three equally likely perturbation modes vary lateral offset, target speed, or both, producing behaviors from mild deviations to aggressive maneuvers.The fixed target speed in one mode is 10 m/s.
- CARLA Ego: CARLA Ego perturbs the ego vehicle after a 24-step, 2-second warm-up, then applies a smoothed lateral route offset and sampled target speed.The simulator runs at 12 Hz, and the modified route is tracked after warm-up.
- CARLA Adv: CARLA Adv spawns a nearby non-ego vehicle within approximately 15 meters, applies the same perturbation after warm-up, and centers the challenging behavior on that agent.
- Collisions are sensor-detected and categorized by object type, while off-road events are identified when the perturbed vehicle enters sidewalk or shoulder regions.After an event, data collection continues for 48 additional frames.
E Weighting Design and Ablation Study of λevent and λagent
The weighting design emphasizes supervision near physical events and on event-relevant agents, while ablations indicate limited sensitivity to the selected hyperparameters.
- The temporal weight λevent increases loss emphasis near critical events such as collisions or off-road departures within a forward event window.For each event timestep te, the window spans [max(0, te − 1), min(T − 1, te + 10)].
- Within each event window, the temporal weight starts at λevent and exponentially decays across time.
- When event windows overlap, the design uses the maximum weight at each timestep and assigns weight 1 outside all event windows.This concentrates supervision near event onset while retaining standard supervision elsewhere.
- The agent weight λagent emphasizes agents selected from event annotations, distinguishing static-object collisions from collisions involving dynamic participants.
- Different λevent and λagent values have limited performance effects, suggesting robustness while retaining focused supervision around critical events and agents.