BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generation

1 Introduction

• Action-model self-improvement loops use learned world simulators to imagine futures and feed resulting data back through reinforcement learning or supervised fine-tuning (Li et al., 2026b; Guo et al., 2026; Liu et al., 2026).BehaviorWorldGen is reported to improve diverse action models and remain compatible with different world simulators, enabling refinement on rare interactive scenarios.Source: §1 · pp. 1–2
• Current simulators replay surrounding agents or generate them as a by-product, leaving their futures fixed with respect to ego actions and unable to model plausible responses.This limitation affects both generative world models and reconstruction-based simulators.Source: §1 · pp. 1–2
• Without reactive agents and targeted multi-agent behaviors, simulator feedback can make risky plans appear safe and leaves long-tail interactions underrepresented in refinement data.The resulting data are unrealistic in interaction and imbalanced in distribution, contributing to the difficulty of learning rare interactive scenarios.Source: §1 · p. 2
• BehaviorWorldGen introduces BehaviorFlow, which uses injectable meta-actions to control agent lifecycles and behaviors while jointly generating vehicles that respond to the ego and one another.The explicit controls can orchestrate rare interactions, focusing data generation on scenarios where the current action model is weakest.Source: §1 · p. 2
• The outer loop evaluates an action model’s predicted ego trajectory, derives target meta-actions for failures, renders BehaviorFlow rollouts, and pairs observations with corrected interaction-aware trajectories.Structured trajectories connect the modules, allowing the framework to accept any upstream action model and downstream world simulator.Source: §1 · p. 2

2.1 World Simulator

• World simulators use action-conditioned generation for controllable future observations or explicit Gaussian-scene reconstruction for novel-view rendering and interactive environments.The passage describes both multi-view and closed-loop generation and reconstruction approaches that render camera and LiDAR data.Source: §2.1 · pp. 2–3

2.2 Action Model

• Driving action models include unified end-to-end planners, generative trajectory policies, vision-language-action models, and policies paired with action-conditioned world models.These lines of work encompass planning-oriented scene representations, multimodal trajectory modeling, language-guided action generation, and joint policy–world-model improvement.Source: §2.2 · p. 3

2.3 Traffic Flow Model

• Traffic-flow research progresses from controllable but unrealistic rule-based models to data-driven rollouts, test-time-controlled diffusion, autoregressive joint prediction, and meta-action-conditioned trajectories.Meta-actions make high-level behavior injectable while preserving physically feasible motion.Source: §2.3 · p. 3

3.1 Overview

• The loop diagnoses an action model’s trajectory, prompts BehaviorFlow for corrections, renders the resulting rollout, and pairs observations with corrected trajectories for policy refinement.The modules exchange structured trajectories and standard training data, keeping each module agnostic to its counterparts.Source: §3.1 · pp. 3–4

3.2 World Simulator

• The simulator can use an action-conditioned world model to imagine scenes beyond recorded logs or 3D Gaussian Splatting for high-fidelity novel-view rendering.BehaviorFlow rollouts extend the reconstruction pipeline beyond its original spatial boundary.Source: §3.2 · p. 4

3.2.1 Action-Conditioned World Model

• The HD-map-conditioned world model jointly generates multi-view RGB and LiDAR; its four-stage training progresses from a LiDAR VAE and bidirectional generation to few-step distillation and causal long-sequence inference.Causal training supports autoregressive inference for closed-loop simulation.Source: §3.2.1 · pp. 4–7

3.2.2 3D Gaussian Splatting

• BehaviorFlow rollouts and an action-conditioned world model generate multimodal observations that extend 3D Gaussian reconstructions beyond logged coverage for rendering along deviated trajectories.The synthesized camera observations and LiDAR point clouds are appended to the original log for joint reconstruction.Source: §3.2.2 · p. 7

3.3 Action Model

• The loop supports both a vision-language-action model and an end-to-end planner as the policy being improved.The implementations are ChainFlow-VLA and DiffusionDrive, respectively.Source: §3.1 · p. 4, §3.3 · p. 7

3.3.1 Vision-Language-Action Model

• ChainFlow-VLA generates kinematically consistent trajectory proposals autoregressively, then uses VLM semantics to guide residual diffusion refinement of each proposal.This refinement-based design differs from approaches that use VLM representations to directly generate final trajectories.Source: §3.3.1 · p. 8

3.3.2 End-to-End Planner

• DiffusionDrive initializes trajectory generation from clustered training-data anchors perturbed by truncated noise, then uses a cascaded diffusion decoder to refine and select a high-confidence plan.The anchor-centered prior shortens denoising while retaining plausible driving trajectories.Source: §3.3.2 · p. 8

3.4 BehaviorFlow

• BehaviorFlow jointly generates agent lifecycles, frame-aligned meta-actions, and states, enabling controlled behaviors and interaction-consistent reactions from surrounding agents.Its ENTER, ALIVE, and EXIT lifecycle labels preserve agent identities while allowing traffic composition to change.Source: §3.4 · p. 9
• Lifecycle and meta-action constraints can be specified for arbitrary agent–timestep pairs, changing traffic composition or intended behavior, respectively.After either intervention, BehaviorFlow regenerates all active agents jointly, allowing unconstrained agents to respond to the modified scene.Source: §3.4 · p. 10
• BehaviorFlow targets scenarios where the action model performs poorly, using diagnosed failures to constrain participants and generate diverse, interaction-consistent scenes.Examples include collisions, unsafe gap selection, failed yielding, and insufficient route progress; the edits create alternative agent compositions and behavioral interactions that passive logs may not capture.Source: §3.4 · p. 10
• Generated rollouts are combined with scene context to create multi-view layouts, rendered into sensor observations, and paired with interaction-aware trajectories for action-model training.This process converts identified failures into controllable training examples for subsequent policy refinement.Source: §3.4 · p. 10

4.1.1 Action-Conditioned World Model

• The causal model improved FID/FVD over the bidirectional model with nearly 70× acceleration, sustained stable 30-second video, and generated temporally consistent, structured LiDAR.Short video generation covered 93 frames at 10 FPS; videos remained coherent across views and followed HD-map conditions, while long rollouts remained visually stable under those conditions.Source: §4.1.1 · p. 11, Figure 4 · p. 12, Figure 5 · p. 13
• Animal insertion preserved coherent motion and surrounding scenes, while fog density and weather style could be controlled without changing the underlying scene structure.Fog control smoothly reduced visibility and contrast as density increased; style transfer generated cloudy, nighttime, rainy, foggy, and snowy variants while retaining road geometry, layout, and object identities.Source: §4.1.1 · p. 11, Figure 7 · p. 14, Figure 8 · p. 15
• Trajectory edits produced the specified lane changes, overtaking, and straight-turn conversions while preserving scene context, enabling plausible counterfactual observations.The four edits were lane changing, overtaking, straight-to-turn, and turn-to-straight; vehicle motion followed the edited trajectory while road geometry and surrounding content remained consistent.Source: Figure 10 · p. 16, §4.1.1 · pp. 16–17, Figure 11 · p. 17

4.1.2 3DGS Scene Extrapolation

• Both extrapolation-augmented 3DGS variants preserved higher visual quality beyond observed coverage, while adding LiDAR stabilized geometry and improved dynamic-object visibility.Within the observed region, reconstruction quality was comparable; the original-only variant deteriorated beyond it, and image-only supervision left weakly observed or occluded geometry underconstrained.Source: Figure 14 · p. 18, §4.1.2 · p. 19

4.2.1 Vision-Language-Action Model

• BehaviorWorldGen gave ChainFlow-VLA the highest PDMS among compared VLA approaches and raised ReCogDrive PDMS to 87.3 (+0.8), with component gains and unchanged comfort.Three action models were trained on 85K NAVSIM training samples augmented with 16K synthetic samples; the ReCogDrive reproduction omitted its original reinforcement-learning stage.Source: Table 2 · p. 19, §4.2.1 · p. 20, Table 3 · p. 20

4.2.2 End-to-End Planner

• DiffusionDrive PDMS rose from 87.7 to 88.6 (+0.9 points), with component gains led by EP (+0.7), DAC and TTC (+0.6 each), and unchanged comfort.The evaluation also found gains in safety-related performance without sacrificing trajectory smoothness.Source: §4.2.2 · pp. 20–21, Table 4 · p. 21
• On low-PDMS scenarios, BehaviorWorldGen improved planning across all three intervals, with the strongest recovery from near-zero PDMS in the [0, 0.15) interval.Gains were concentrated on difficult cases: DAC and EP improved most in the two hardest intervals, TTC stood out in [0.3, 0.45), and NC improved across intervals while comfort remained unchanged.Source: Table 5 · p. 21, §4.2.2 · p. 21

4.3 BehaviorFlow Results

• BehaviorFlow generated distinct following and emergency-avoidance responses, as well as alternative collision-free yielding orders, from the same scene contexts.In the intersection variants, either the oncoming vehicle or ego yielded; across both scenarios, agents realized specified maneuvers while adapting coherently to one another.Source: Figure 15 · p. 22, Figure 16 · p. 22, §4.3 · pp. 22–23

• BehaviorFlow uses meta-action-conditioned controls over agent lifecycles and high-level behaviors to generate interaction-consistent multi-agent rollouts.Source: §5 · p. 23
• Unlike simulators that mainly condition on ego actions, BehaviorWorldGen models responsive surrounding-agent behaviors to address distorted interaction feedback and limited safety-critical long-tail coverage.Source: §5 · p. 23
• Structured trajectories connect action models and simulators; generated rollouts are rendered into realistic observations and paired with corrected interaction-aware trajectories for policy refinement.Source: §5 · p. 23
• The structured interface supports different policy architectures and rendering mechanisms, while experiments demonstrate effectiveness in world generation, 3DGS scene extrapolation, and end-to-end planning.Multimodal image and LiDAR extrapolation further extends the reliable operating range of reconstruction-based simulators.Source: §5 · p. 23

A 3D Gaussian Scene Reconstruction

• The system builds on OmniRe’s separate Gaussian representations for static environments and dynamic actors, rendered together in a unified scene graph.Source: A 3D Gaussian Scene Reconstruction · p. 28
• Its three adaptations are RoGS-based road modeling, object-level reconstruction and refinement of dynamic vehicles, and MVSA-based geometric completion beyond forward-facing LiDAR coverage.Source: A 3D Gaussian Scene Reconstruction · p. 28

A.1 Road-Surface Reconstruction

• Conventional 3DGS can drift road primitives from the physical surface to reduce image-space errors, blurring or distorting lane lines under lateral shifts.This is a critical failure for closed-loop simulation when the ego vehicle deviates from its logged trajectory.Source: §A.1 · p. 28
• Under lateral viewpoint shifts, RoGS preserves sharp, geometrically consistent lane markings, while 3DGS increasingly distorts them away from the observed trajectory.Figure 17 compares road reconstruction under lateral viewpoint shifts.Source: §A.1 · p. 28, Figure 17 · p. 28
• RoGS uses dense 2D Gaussian primitives on a regular bird’s-eye-view grid, fixes horizontal positions, and optimizes heights with local smoothness regularization.The ego trajectory recovered from camera poses and extrinsics sets the grid extent; the nearest ego pose initializes primitive height and orientation, while other attributes are learned from image observations.Source: §A.1 · p. 28

Dynamic-Vehicle Reconstruction and Refinement

• MVSAM3D reconstructs each valid vehicle as an independent object-centric 3D Gaussian model to recover geometry and appearance from limited, often occluded views.For each track, RGB crops and masks are selected across cameras and timestamps using visibility, projected size, segmentation quality, and viewpoint diversity.Source: §A.2 · p. 28
• The reconstructed asset collection covers diverse vehicle categories, shapes, and appearances.Representative object-centric reconstructions are shown in Figure 18.Source: §A.2 · p. 29, Figure 18 · p. 29
• Each vehicle’s canonical orientation and dimensions are aligned to its track’s average 3D bounding-box size before insertion with its original track ID and motion trajectory.This alignment addresses the reconstructed object’s lack of reliable metric scale.Source: §A.2 · p. 29
• After the base scene converges, inserted vehicles undergo coarse-to-fine refinement of alignment, appearance, and ground-contact shadows while other scene components remain fixed.The schedule first optimizes low-frequency appearance, pose, and residual three-axis scale; then fixes pose and scale while refining higher-order spherical harmonics and opacity; finally it fixes the vehicle representation and refines only its shadow.Source: §A.2 · p. 29

A.3 Rear-View Geometry Completion with MVSA

• Missing initial LiDAR observations in rear-view regions severely degrade 3DGS reconstruction by leaving geometry without reliable initialization.This is a problem for platforms equipped with predominantly forward-facing solid-state LiDAR.Source: §A.3 · p. 29
• MVSA predicts dense depth from calibrated multi-camera views, and a pose-aware consistency filter rejects estimates lacking agreement across neighboring views.The filter back-projects each reference-view depth, projects it into source views, and compares it with their depth predictions.Source: §A.3 · p. 29
• Filtered depths both initialize background Gaussian primitives in the rear view and provide dense supervision that constrains Gaussian positions during reconstruction.These two uses compensate for missing LiDAR points and resolve the initial rear-view geometric cold start.Source: §A.3 · p. 29

A.4 Camera Appearance and Environment Modeling

• A learnable bilateral grid compensates for camera-dependent ISP and exposure differences, while a learnable cubemap represents distant sky for compositing with the Gaussian scene.The bilateral grid improves cross-camera appearance consistency.Source: §A.4 · p. 29
• Figure 19 shows elevated panoramic novel views and reconstructed seven-camera views for diverse driving environments.Source: Figure 19 · p. 30

A.5 Summary

• Combining structured roads, object-centric vehicle reconstruction, MVSA-based rear-view completion, camera-aware appearance correction, and sky modeling improves geometric consistency, scene completeness, and rendering quality.The improvements apply across observed and novel viewpoints.Source: §A.5 · p. 30
• The system produces high-fidelity, geometrically complete driving-scene reconstructions across urban roads, highways, dense traffic, and nighttime scenes.Fig. 19 presents representative overall reconstruction results across these environments, which the paper describes as a reliable foundation for closed-loop simulation.Source: §A.5 · p. 30