Source-linked AI summary
BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generation
Jiaqi Wang, Zhuo Zhang, Haining Guan, Tingguang Zhou, Haowen Cui, Zhongyang Zhu, Yulong Zheng, ChuanYe Wang, Xuefeng Chen, Zhen Yang, Tianchen Deng, Feiyang Tan, Hangning Zhou, Bo Dai, Lixia Shen, Xiwu Chen, Xiyang Wang, Jiajun Zhu
TL;DR
Existing world simulators struggle to produce behaviorally plausible responses from surrounding agents, making interactive self-improvement data unrealistic and imbalanced. BehaviorWorldGen uses BehaviorFlow to inject interpretable behavior controls into structured multi-agent rollouts, which are rendered and paired with corrected trajectories for action-model refinement. The framework demonstrates consistent improvements across world generation, scene extrapolation, and policy refinement, especially in difficult interactive scenarios.
Problem
Current simulators mainly condition on ego actions and cannot model responsive surrounding-agent behaviors, limiting interaction fidelity and long-tail scenario coverage.
Method
BehaviorWorldGen uses BehaviorFlow, a meta-action-conditioned traffic-flow model, to generate interaction-consistent rollouts that connect action models with world simulators through structured trajectories.
Results
Experiments demonstrate consistent improvements in world generation, scene extrapolation, and policy refinement, with the largest benefits in difficult interactive scenarios.
Takeaways & Limitations
Controllable behavior-aware generation provides targeted interactive training scenarios while remaining compatible with diverse action models and world simulators.
Takeaways & Limitations
Image-only supervision leaves weakly observed or occluded regions geometrically underconstrained, particularly near dynamic objects.
Abstract
from arXiv · showhide
Modern driving action models are increasingly improved in a self-improvement loop, where a learned world simulator imagines future observations and the resulting data is fed back to refine the action model. However, the bottleneck of this loop lies in the simulators' inability to generate behaviorally plausible responses by surrounding agents, making generated data both unrealistic in interaction and imbalanced in distribution. We introduce BehaviorWorldGen, a framework that closes the loop between action models and world simulators through controllable behavior-aware structured world generation. Its core component is BehaviorFlow, a meta-action-conditioned traffic-flow model that injects interpretable behavior controls and jointly generates multi-agent rollouts. BehaviorFlow realizes the specified agent behaviors while allowing surrounding vehicles to respond to the ego and to one another. The resulting rollouts are rendered by a world simulator into realistic multi-view observations, which are paired with corrected interaction-aware trajectories for action-model refinement. Since BehaviorWorldGen uses structured trajectories as the interface between its modules, it is compatible with diverse action models and world simulators. Experiments on world generation, scene extrapolation, and policy refinement demonstrate consistent improvements, with the largest benefits concentrated on difficult interactive scenarios.
1 Introduction
BehaviorWorldGen addresses a core self-improvement bottleneck: current simulators cannot generate responsive surrounding-agent behavior, producing distorted interaction feedback and poor long-tail coverage. It introduces BehaviorFlow and an outer-loop framework that controllably generates interactive rollouts for action-model refinement.
- Current simulators replay or generate surrounding-agent futures independently of ego actions, preventing behavioral response modeling.
- This limitation distorts interaction feedback, making potentially unsafe plans appear safe and passing the distortion into action-model refinement.
- Undirected self-improvement mostly reproduces routine driving, leaving coordinated long-tail interactions underrepresented and training data imbalanced.
- BehaviorFlow injects interpretable meta-actions, such as lane keeping, lane changing, or turning, to control agent lifecycles and high-level behaviors.
- Conditioned on these controls, BehaviorFlow jointly generates ego and surrounding-agent rollouts in which vehicles respond to the ego and one another.
- BehaviorWorldGen evaluates action-model failures, derives target meta-actions, renders corrected rollouts into observations, and feeds them back for refinement.
2 Related Works
Related work spans action-conditioned generative and reconstruction-based world simulators, action-model families, and traffic-flow models. These lines establish the components that BehaviorWorldGen combines for interactive driving refinement.
- Generative world models synthesize future street-view or multi-view observations from structured scene conditions such as maps, boxes, and camera poses.
- Reconstruction-based approaches explicitly represent scenes with 3D Gaussian Splatting to render novel views during driving simulation.
- Action models map observations and navigation intent to ego behavior, including end-to-end planners, generative trajectory models, and vision-language-action models.
- Traffic-flow models generate multi-agent motion, with prior work ranging from controllable rule-based systems to data-driven and diffusion-based approaches.
3 Methodology
BehaviorWorldGen connects action models, BehaviorFlow, and world simulators through structured trajectories, using controllable multi-agent rollouts to correct interaction failures and generate training observations. Its world-model branch supports multimodal generation through a four-stage training scheme, while BehaviorFlow provides explicit, temporally aligned behavior controls for interaction-consistent rollouts.
- 3.1 Overview: The framework places BehaviorFlow in an outer loop between action models and world simulators, exchanging structured trajectories while keeping modules counterpart-agnostic.The action model proposes and diagnoses ego trajectories; BehaviorFlow generates corrected rollouts, and the simulator renders observations for policy refinement.
- 3.2 World Simulator: BehaviorWorldGen supports either an action-conditioned world model or a reconstruction-based 3D Gaussian Splatting pipeline as its world simulator.The action-conditioned model offers broad scene controllability, whereas the reconstruction-based pipeline provides high-fidelity novel-view rendering.
- 3.2.1 Action-Conditioned World Model: The action-conditioned world model generates future multi-view RGB videos and LiDAR rangemaps from multimodal observations, HD maps, and scene text.Its DiT architecture encodes modalities into latents, injects HD-map latents as spatial conditions, and jointly processes RGB and LiDAR tokens.
- 3.2.1 Action-Conditioned World Model: The world model uses four training stages: LiDAR VAE training, bidirectional model training, few-step distillation, and causal model training.The final stages address generation efficiency and the need for real-time autoregressive, long-horizon closed-loop simulation.
- 3.3 Action Model: ChainFlow-VLA separates trajectory prediction into proposal generation and VLM-guided residual refinement rather than directly generating trajectories.This action-model design is used as one policy branch for validating the improvement loop.
- 3.4 BehaviorFlow: BehaviorFlow jointly generates agent lifecycles, frame-level meta-actions, and states, allowing explicit control over scene composition and behavior.Frame-level controls stay temporally aligned with state transitions, and unconstrained variables are generated jointly to produce interaction-consistent multi-agent rollouts.
4 Experiments
Experiments show that the action-conditioned world model generates temporally coherent, controllable multimodal observations, while BehaviorWorldGen improves action-model performance across architectures, especially on difficult failure cases.
- 4.1.1 Action-Conditioned World Model: The causal model improves short-horizon video quality and inference efficiency, achieving nearly 70× acceleration over the bidirectional model.It achieves better FID/FVD, remains temporally coherent across views, and follows HD-map conditions closely.
- 4.1.1 Action-Conditioned World Model: Thirty-second autoregressive videos maintain stable visual quality and plausible scene evolution under HD-map conditions.The stability supports reliable visual observations for closed-loop simulation.
- 4.1.1 Action-Conditioned World Model: Generated LiDAR remains temporally consistent while preserving static-background and moving-vehicle structure.The resulting point clouds provide geometrically meaningful observations for LiDAR-dependent applications such as 3DGS scene extrapolation.
- 4.1.1 Action-Conditioned World Model: The model controls rare objects, atmospheric density, environmental styles, and vehicle trajectories while preserving scene coherence.Animals move plausibly, fog changes smoothly, weather variants preserve scene structure, and edited trajectories produce specified vehicle motion.
- 4.1.2 3DGS Scene Extrapolation: Adding generated LiDAR yields more stable geometry and improves dynamic-object visibility in extrapolated regions, where image-only supervision is underconstrained.The image-and-LiDAR variant suppresses ground-level floaters and maintains geometric reliability beyond the observed region.
- 4.2 Policy Refinement: BehaviorWorldGen improves action-model performance across VLA and end-to-end planning settings, with larger gains concentrated on low-PDMS failure scenarios.It improves safety and planning metrics, reduces catastrophic outcomes, preserves comfort, and recovers meaningful performance where the baseline nearly collapses.
5 Conclusion
BehaviorWorldGen closes the loop between action models and world simulators through controllable behavior-aware structured world generation. Its structured trajectory interface supports diverse policies and rendering mechanisms, while experiments demonstrate benefits for world generation, scene extrapolation, and planning.
- BehaviorFlow generates interaction-consistent multi-agent rollouts by controlling agent lifecycles and high-level behaviors through meta-actions.The framework uses these rollouts to model responsive surrounding-agent behavior rather than conditioning mainly on ego actions.
- BehaviorWorldGen renders generated rollouts into realistic observations and pairs them with corrected interaction-aware trajectories for policy refinement.
- Structured trajectories serve as the interface between action models and world simulators, supporting different policy architectures and rendering mechanisms.
- Experiments on world generation, 3DGS scene extrapolation, and end-to-end planning demonstrate the effectiveness of controllable behavior generation for action-model improvement.
Author List
The project credits identify contributors across generation and reconstruction world models, action modeling, BehaviorFlow, advising, and project leadership.
- Generation World Model contributors are Zhuo Zhang, Jiaqi Wang, Haowen Cui, Chuanye Wang, Zhongyang Zhu, and Yulong Zheng.
- Haining Guan is credited for the Reconstruction World Model, while Tingguang Zhou and Zhen Yang are credited for the Action Model.
- Jiaqi Wang, Xuefeng Chen, and Haining Guan contributed to BehaviorFlow; Lixia Shen, Bo Dai, Hangning Zhou, and Feiyang Tan were project advisors.
- Jiajun Zhu, Xiyang Wang, and Xiwu Chen are credited as project leaders.
A 3D Gaussian Scene Reconstruction
The reconstruction system extends OmniRe’s multi-representation scene decomposition for closed-loop simulation by separately modeling static environments and dynamic actors in a unified scene graph.
- The reconstruction system builds on OmniRe by representing static environments and dynamic actors separately within a unified scene graph.
- Three adaptations support closed-loop simulation: RoGS-based road modeling, object-level reconstruction and refinement of dynamic vehicles, and MVSA-based geometric completion.
A.1 Road-Surface Reconstruction
Conventional 3DGS can distort road geometry under lateral viewpoint shifts because photometric optimization may move road primitives away from the physical surface. RoGS constrains road reconstruction with structured 2D Gaussian primitives and trajectory-informed initialization.
- A.1 Road-Surface Reconstruction: Photometric-only 3DGS may shift road primitives away from the physical surface, causing lane lines to blur or distort under lateral viewpoint shifts.This geometric error is critical when the ego vehicle deviates from its logged trajectory during closed-loop simulation.
- A.1 Road-Surface Reconstruction: Figure 17 depicts road reconstruction under lateral viewpoint shifts, the setting where geometric inaccuracies become pronounced.
- A.1 Road-Surface Reconstruction: RoGS represents roads with dense 2D Gaussian primitives on a regular bird’s-eye-view grid without relying on LiDAR.
- A.1 Road-Surface Reconstruction: The recovered ego trajectory determines the grid extent and initializes each primitive’s height and orientation from the nearest ego pose.
- A.1 Road-Surface Reconstruction: RoGS fixes horizontal positions during optimization while learning heights with local smoothness regularization.
A.2 Dynamic-Vehicle Reconstruction and Refinement
MVSAM3D reconstructs tracked vehicles independently as object-centric 3D Gaussian models, using carefully selected multiview observations to recover complete geometry and appearance. The aligned vehicles are inserted with their identities and trajectories, then refined from coarse structure to view-dependent details and shadows.
- Object-centric reconstruction: Independent object-centric 3D Gaussian models address incomplete vehicle reconstructions caused by limited viewpoints and frequent occlusion.Observations are selected across cameras and timestamps using visibility, projected object size, segmentation quality, and viewpoint diversity.
- Object-centric reconstruction: Diverse vehicle categories, shapes, and appearances are covered by the reconstructed asset collection.Representative reconstructions demonstrate the resulting coverage across vehicle assets.
- Track-aligned insertion: Canonical orientation and dimensions are aligned to the track’s average 3D bounding-box size before insertion with the original track ID and motion trajectory.
- Coarse-to-fine refinement: A three-stage coarse-to-fine schedule optimizes appearance, pose, scale, view-dependent details, opacity, and finally the ground-contact shadow.Low-frequency appearance, pose, and residual scale are optimized first; higher-order spherical harmonics and opacity are refined after pose and scale are fixed.
A.3 Rear-View Geometry Completion with MVSA
Forward-facing solid-state LiDAR leaves rear-view regions without initial geometric observations, degrading reconstruction quality. MVSA-based depth completion supplies filtered multiview geometry for both Gaussian initialization and depth supervision, resolving this rear-view cold start.
- Motivation: Predominantly forward-facing solid-state LiDAR creates an unobserved rear-view region whose missing initialization severely degrades reconstruction.The method introduces MVSA-based visual depth completion to provide dense geometric initialization and supervision without relying on LiDAR.
- Depth completion: MVSA predicts dense depth from calibrated multiview observations, while a pose-aware consistency filter discards estimates lacking sufficient cross-view agreement.Depths are back-projected into 3D, projected into neighboring source views, and compared with source-view predictions.
- Depth completion: Filtered depths initialize rear-view background Gaussian primitives and constrain Gaussian positions as dense reconstruction supervision.These complementary uses compensate for missing LiDAR points and resolve the initial rear-view geometric cold-start problem.
A.4 Camera Appearance and Environment Modeling
Camera-aware appearance correction and sky modeling complement structured road and vehicle representations to improve consistency across viewpoints. Representative reconstructions span varied driving environments and provide a high-fidelity foundation for closed-loop simulation.
- Camera appearance modeling: A learnable bilateral grid compensates for camera-dependent ISP and exposure differences to improve cross-camera appearance consistency.
- Environment modeling: A learnable cubemap represents the distant sky and is composited with the Gaussian scene during rendering.
- Reconstruction results: Overall reconstruction results include elevated panoramic novel views and reconstructed seven-camera views across diverse driving environments.The figure presents results for urban roads, highways, dense traffic, and nighttime scenes.
- Reconstruction results: The combined representation improves geometric consistency, scene completeness, and rendering quality across observed and novel viewpoints.The resulting high-fidelity, geometrically complete reconstructions provide a reliable foundation for closed-loop simulation.