PaperlayerView exact revision on arXiv

Source-linked AI summary

HelloWorld: Towards Practical Applications of Generative Driving World Models

Fan Lu, Hanshi Wang, Zijing Wang, Quan Feng, Zhi Wang, Shijie Chen, Xianming Zeng, Yujian Zhang, Jiazhe Wang, Xin Zha, Kai Wang, Zhijie Zhao, Lin Zhu, Tianyi Yang, Yucheng Xu, Tao Ji, Haodong Zhang, Zhipeng Zhang, Peixi Peng, Guang Chen, Xingliang Liu, Lei Yang, Jianyun Xu

arXiv:2609.28931v1cs.CV

TL;DR

Driving logs contain only encountered trajectories and traffic configurations, limiting direct access to counterfactual driving observations. HelloWorld combines controllable, multi-sensor generation with causal sequential modeling; its first-frame-conditioned post-trained model reduces rotation error from 5.534° to 2.027° and Sim(3)-aligned translation error from 1.880 to 1.352 against Cosmos transfer 2.5.

  • Problem

    Driving logs contain only encountered trajectories and traffic configurations, while practical world models must jointly generalize, enable intentional scene changes, generate multi-sensor observations, and support tractable repeated inference.

    Source: §1 · p. 1

  • Method

    HelloWorld progressively specializes heterogeneous-video visual and motion priors with structured driving supervision, using block-causal self-generated-context alignment and synchronized multi-view, LiDAR, and few-step generation.

    Source: §1 · pp. 2–3, §2.4 · p. 5

  • Results

    Against Cosmos transfer 2.5, the first-frame-conditioned post-trained model reduces rotation error from 5.534° to 2.027° and Sim(3)-aligned translation error from 1.880 to 1.352.

    Source: §5.2 · p. 19

  • Takeaways & Limitations

    HelloWorld provides a unified framework for scalable driving data generation and interactive simulation beyond recorded logs.

    Source: §1 · pp. 1, 3

    HelloWorld prescribes ego trajectories but does not model low-level vehicle dynamics, policy feedback, or reactive traffic, so its generated observations are not a validated closed-loop simulator.

    Source: §7 · p. 35

Abstract

from arXiv · show

Driving world models provide a promising route toward scalable counterfactual data generation and interactive simulation beyond recorded driving logs. Realizing this potential requires a system that can generalize across diverse scenes, respond faithfully to prescribed controls, generate coherent multi-sensor observations, and operate efficiently under repeated inference. We present \textbf{HelloWorld}, a 2B driving world model system designed around these requirements. HelloWorld progressively specializes broad visual and motion priors from heterogeneous video data into controllable driving generation using ego pose, HD maps, and 3D boxes. A block-causal generation interface, together with adaptation to self-generated context, aligns the model with sequential simulation. The system further supports synchronized seven-camera RGB generation and conditional LiDAR synthesis, and is distilled toward few-step inference for efficient deployment. Experiments evaluate visual quality, control fidelity, cross-view consistency, robustness under repeated generation, inference efficiency, and LiDAR synthesis. Together, HelloWorld provides a unified framework for scalable driving data generation and interactive simulation.

1 Introduction

Recorded driving logs cannot show how scenes would evolve under unobserved ego trajectories or traffic configurations, motivating generative world models for counterfactual data generation and interactive simulation. HelloWorld addresses this as a system-level challenge by combining broad video priors, explicit controls, sequential multi-sensor generation, and efficient inference.

Introduction

  • HelloWorld is a 2B driving world model that integrates broad video priors, structured driving supervision, controllable generation, multi-sensor observations, and efficient inference for data generation and interactive simulation.Its stated objective is to generate controllable observations beyond recorded trajectories and scene configurations while retaining the temporal and computational properties needed for repeated simulation.Source: §1 · pp. 2–3
  • HelloWorld extends each simulation step to synchronized seven-camera RGB and conditional LiDAR generation, using cross-view communication and geometry-aware cross-modal interaction.The LiDAR branch preserves valid returns and metric structure without requiring RGB and LiDAR to share a native representation.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Consistency training and distribution matching distill HelloWorld toward few-step generation to reduce repeated sampling cost while preserving its control and sequential-generation interfaces.The overview describes distilling a 20-step causal teacher into a four-step student.Source: Figure 1 · p. 2, §1 · p. 3
  • Progressive specialization transfers broad visual and motion priors into controllable driving generation, using ego pose, HD maps, and 3D boxes to alter camera motion and scene configuration.Heterogeneous general and fleet video establish pose-conditioned dynamics before calibrated multi-view data and structured driving supervision are introduced.Source: §1 · pp. 2–3
  • Block-causal generation and adaptation to self-generated context align HelloWorld with sequential simulation and reduce the gap between recorded-history training and model-generated inference histories.The model generates frames jointly within blocks, restricts later blocks to available observations, and uses context corruption and self rollout to improve robustness.Source: §1 · pp. 2–3

2 Related Work

HelloWorld builds on broad video-model priors and specializes them for driving through metric pose control, synchronized multi-view generation, and a causal interface spanning training and deployment. The related work frames its LiDAR path and capability positioning as further parts of this system.

2.1 Video Foundation and Interactive World Models

  • HelloWorld starts from Cosmos-Predict2.5 and transfers broad video priors to metric pose control and synchronized seven-view generation.Unlike interactive systems centered on first-person streams and scale-free motion, HelloWorld targets fixed vehicle-camera geometry; this does not itself establish a validated driving closed loop.Source: §2.1 · p. 3

2.2 Driving Video Generation and Conditioning

  • HelloWorld separates dense layout control from metric 6-DoF ego pose and uses one block-causal interface across training stages.Pose and layout may describe the same future, but neither must pass through the other’s encoder; the system combines Plücker pose features, a switchable layout tower, and a calibrated seven-camera rig.Source: §2.2 · p. 4

2.3 Multi-View Consistency and Single-to-Multi-View Transfer

  • HelloWorld learns appearance and pose response from single-view and flattened fleet video before inserting adjacency-restricted cross-view attention for synchronized data.The added branch copies self-attention query, key, value, and normalization weights, but its zero output projection makes its residual contribution zero at initialization; this does not establish whole-model parity or retention after fine-tuning.Source: §2.3 · p. 4

2.4 Causal and Long-Horizon Video Generation

  • HelloWorld aligns chunk-causal diffusion to generated history using per-chunk noise schedules, context corruption, and Self-Rollout Alignment.Completed chunks remain context rather than being jointly denoised again; HelloWorld shares a high-level causal-training sequence with LingBot-World-Infinity but instantiates it for chunk-causal driving observations and explicit cache-boundary training, without establishing superiority over a matched retrofit.Source: §2.4 · pp. 4–5

2.5 Structured Control Architectures

  • HelloWorld encodes pose through Plücker features and AdaLN, while a VACE-style tower handles maps and boxes; joint, pose-only, and control-only batches train each mode.Thus, dropping either pose or structured control at inference is a trained operating mode, not an architectural modification.Source: §2.5 · p. 5

2.6 Few-Step Generation

  • HelloWorld combines teacher-forced consistency training with self-forcing distribution matching so capability acquisition, generated-history alignment, and compression share a causal interface.The section notes that small per-window distribution shifts can accumulate when each generated window becomes the next context.Source: §2.6 · p. 5

2.7 LiDAR Representation, Conditional Generation, and Rectification

  • For LiDAR, HelloWorld adapts a video-VAE representation with return validity and geometric constraints, synthesizes LiDAR from synchronized RGB, and rectifies decoded range images along original rays.Its conditional generator combines native-grid token packing, azimuth and coverage ray features, latent anchoring, and geometric supervision through a frozen decoder; rectification is reusable across reconstruction and generation, but gains need not transfer unchanged.Source: §2.7 · p. 6

2.8 Capability Positioning

  • The capability positioning distinguishes independently controllable metric ego pose from low-dimensional actions or ego motion implicit in projected layouts.Table 1 focuses on core-generator properties; the supplied passages do not provide its cell values.Source: §2.8 · p. 6, Table 1 · p. 6

3 Data Construction

HelloWorld builds its training data from broad pose-supervised video, then progressively narrower synchronized multi-view and structured-control pools. Its preparation, balancing, pose normalization, and governance procedures make these sources usable for controllable driving generation.

3.1 Data Sources and Supervision Availability

  • The training inventory combines seven-camera fleet logs with general-purpose video, while synchronized calibration and structured annotations are available only in progressively smaller fleet subsets.Fleet logs include metric ego pose, and an annotated subset adds HD maps and 3D boxes; general-purpose trajectories are recoverable but scale-free. Fleet data contribute 35.1% of Stage-1 training windows.Source: §3.1 · p. 7, Table 2 · p. 8

3.2 Conversion Pipeline

  • Preparation converts pose-supervised videos into temporal windows with captions and trajectories, adding calibration and synchronization checks for multi-view data and rendered, validated annotations for structured control.Quality checks include duplicate removal, visual-quality and pose/motion validation, stream synchronization and overlap, and alignment between control geometry and RGB frames.Source: §3.2 · p. 8

3.3 Captioning and Semantic Annotation

  • Segments receive six VLM caption variants and structured scene tags, but Stage 1 uses only the three non-behavior captions and selects annotations nearest the segment center.For synchronized multi-view training, the front-wide view receives the complete caption while other cameras receive view-specific prefixes.Source: §3.3 · pp. 8–9

3.4 Selection, Balancing, and Quality Gates

  • 497,773 clips were selected, raising long-tail share uniformly by 3.74×; pose-based window balancing then produced 1,510,454 balanced 29-frame windows.Clip-level selection addresses scenario diversity, while segment-level quotas balance turn-angle and speed categories; high-acceleration windows bypass quotas.Source: §3.4 · p. 9

3.5 Pose Representation and Scale Unification

  • Fleet poses are converted to a shared camera convention and dense six-channel Plücker raymaps, with sequence-relative translation scaling for non-metric trajectories.A binary metric-scale indicator distinguishes fleet trajectories from scale-free reconstructed ones, allowing both to share the pose-conditioning contract.Source: §3.5 · p. 10

3.6 Data Governance

  • Governance anonymizes fleet faces and plates, keeps training and evaluation sequences disjoint, and treats licenses, geographic reporting, and condition provenance as constraints.The separate LiDAR tokenizer dataset contains 50,000 29-frame segments; calibration, rendering semantics, and adapter versions are recorded in experiment condition manifests.Source: §3.6 · p. 10

4 Method

HelloWorld progressively turns broad video priors into a controllable driving world model, then aligns it with sequential generation and distills it for efficient inference. It extends synchronized multi-view RGB generation with conditional LiDAR synthesis.

4.1 Overview and Problem Formulation; 4.2 Generative Backbone and Factorized Conditioning

  • HelloWorld predicts sensor observations from prescribed trajectories and scene configurations rather than inferring vehicle actions or propagating vehicle dynamics.Its RGB generator uses latent rectified flow with block-causal temporal attention and factorized conditioning for camera view, ego pose, scene layout, and text.Source: §4.2 · p. 12

4.3 Causal Observation Modeling

  • Self-Rollout Alignment trains later chunks on model-generated histories, while inference commits only completed chunks to a bounded KV cache.Context corruption also broadens training histories with noise, appearance degradation, and spatial distortions; rollout generation is separated from optimization to avoid backpropagating through the full trajectory.Source: §4.3 · p. 13

4.4 Broad Pose-Conditioned World Modeling

  • Stage 1 learns pose-conditioned RGB generation from heterogeneous video and fleet clips, flattening fleet camera streams into independent samples before later multi-view specialization.Dense Plücker raymaps encode camera trajectories; a metric-scale flag distinguishes metric fleet motion from scale-normalized reconstructed trajectories, and a null-pose representation distinguishes missing pose from a stationary camera.Source: §4.4 · p. 13

4.5.1 Synchronized Multi-View Modeling

  • Synchronized multi-view specialization adds residual attention between calibrated neighboring cameras at the same latent time, initialized as an inactive branch and then learned through full-network fine-tuning.This introduces cross-view interaction without assuming pixel-level alignment or dense communication among all camera pairs.Source: §4.5.1 · p. 14

4.5.2 Explicit Scene Control

  • Scene control keeps ego pose and scene configuration as separate interfaces, using rendered HD maps and 3D boxes to inject learned residuals into the RGB backbone.Training first freezes the multi-view backbone while fitting the control tower, then fine-tunes on joint, pose-only, and control-only examples; controls are re-rendered when the viewpoint changes.Source: §4.5.2 · p. 14

4.6.1 Consistency Initialization and Training; 4.6.2 Self-Forcing Distribution Matching

  • The few-step student is initialized with discretized consistency training, extended to continuous-time consistency, and then trained on its own generated distribution using self-forcing distribution matching.Consistency training begins before distribution matching; generated histories use the same chunk-sequential KV-cache interface as inference.Source: §4.6 · p. 15, §4.6.1 · p. 15, §4.6.2 · pp. 15–16

4.6.3 Few-Step Sequential Inference

  • At inference, the student follows a descending noise schedule, commits each completed chunk to the KV cache, and retains the teacher’s text, pose, scene-control, and cross-view interfaces.Cross-view communication is evaluated at each student step during synchronized multi-view generation.Source: §4.6.3 · p. 16

4.7 Multi-Sensor Extension: Conditional LiDAR Generation

  • The LiDAR extension learns a geometry-preserving representation and generates LiDAR conditioned on fixed multi-camera RGB observations.Its VAE separates range from return validity and uses Noman and Haar losses to address flying points and depth-boundary artifacts; conditional diffusion adds geometric supervision through a frozen decoder, with gradients updating the generator but not the decoder. An offline radial refinement network adjusts existing returns along sensor rays.Source: §4.7 · p. 16, §4.7.1 · p. 16, §4.7.2 · p. 17

5 Experiments · 5.1 Single-View Video Generation · 5.2 Multi-View Video Generation

HelloWorld is evaluated for trajectory adherence and perceptual quality in single-view generation, then for trajectory control, cross-view consistency, and perceptual quality in multi-view generation. The experiments also include LiDAR generation, few-step distillation, and controllability under different conditioning inputs.

5 Experiments

  • The experiments assess generation quality, geometric consistency, and controllability across staged single-view and multi-view video generation, with additional evaluations of LiDAR synthesis, few-step distillation, and conditioning inputs.The single-view evaluation examines Stage 1; synchronized multi-view generation is evaluated at Stages 2 and 3.Source: §5 · p. 17

5.1 Single-View Video Generation

  • HelloWorld has the lowest rotation and Sim(3)-aligned translation errors on both single-view benchmarks, reducing Lingbot World 2.0’s errors by 36.2%/9.9% at 29 frames and 50.1%/6.6% at 100 frames.HelloWorld’s metric-scale translation errors on real-world driving cases are 0.6830 m and 5.5857 m; without corresponding baseline measurements, these do not establish comparative superiority.Source: §5.1 · p. 18, Table 3 · p. 18, Table 4 · p. 18
  • Single-view perceptual means are 0.8328 and 0.8229 for HelloWorld versus Lingbot World 2.0’s 0.8411 and 0.8416, despite HelloWorld’s higher motion smoothness on both benchmarks.HelloWorld also has higher dynamic degree at 29 frames and higher subject and background consistency at 100 frames; its dynamic-degree score is 0.9200 on both, while lower imaging-quality scores remain a limitation.Source: §5.1 · p. 18, Table 5 · p. 19, Table 6 · p. 19

5.2 Multi-View Video Generation

  • With first-frame conditioning, the post-trained multi-view model beats Cosmos transfer 2.5 on rotation, aligned translation, and metric-scale translation errors, reducing them by 63.4%, 28.1%, and 44.8%, respectively.Its errors are 2.027° rotation, 1.352 aligned translation, and 4.904 m metric-scale translation, compared with Cosmos’s 5.534°, 1.880, and 8.886 m.Source: §5.2 · p. 19, Table 7 · p. 20
  • Post-training improves orientation and aligned trajectory accuracy over mid-training, but increases metric-scale translation error; first-frame conditioning improves all three pose metrics for both stages.For the post-trained model, first-frame conditioning reduces rotation error from 3.334° to 2.027°, aligned translation from 1.844 to 1.352, and metric-scale translation from 8.485 m to 4.904 m.Source: §5.2 · pp. 19–20, Table 7 · p. 20
  • Post-training lowers cross-view CSE by 32.5% with first-frame conditioning and 34.9% without; even its best result, 4.180, remains above the real-video reference of 1.730.CSE measures epipolar agreement, so depth discrepancies that preserve the epipolar constraint may go undetected.Source: §5.2 · pp. 19–20, Table 8 · p. 20
  • All four multi-view HelloWorld settings exceed Cosmos transfer 2.5’s six-dimension VBench mean, but first-frame conditioning raises the mid-trained mean from 0.7791 to 0.7882 and lowers the post-trained mean from 0.7901 to 0.7754.With matched first-frame conditioning, post-training improves subject consistency, background consistency, and imaging quality over Cosmos, but lowers aesthetic quality and dynamic degree while leaving motion smoothness nearly unchanged.Source: §5.2 · p. 20, Table 9 · p. 21
  • At short horizons, HelloWorld records FID 7.75 and FVD 41.08, reducing FAR-Drive’s values by 35.0% and 50.4%; long-horizon results trade higher FID for lower FVD versus HorizonDrive.For long-horizon generation, HelloWorld reports FID 17.81 and FVD 88.10, versus HorizonDrive’s 13.82 and 92.99. FID measures frame-level fidelity, while FVD measures video-level distributional similarity.Source: §5.2 · pp. 20–21, Table 10 · p. 21

5.3 Controllability Analysis

HelloWorld follows counterfactual ego-pose controls and generates varied environmental and rare traffic scenes while preserving coherent geometry, traffic layout, or scene semantics. Explicit pose conditioning also improves scale consistency over the compared alternatives.

Pose Control

  • Under the same scene prefix, counterfactual left- and right-turn pose sequences produce matching viewpoint transitions while retaining realistic geometry and traffic dynamics.These sequences prescribe futures that differ from the observed motion.Source: §5.3 · p. 21, Figure 4 · p. 22
  • The model follows unconventional lane-change poses across a roadside green belt or road barrier, changing viewpoint while preserving coherent scene structure.The interventions enter a non-motorized lane or the opposite-direction lane.Source: §5.3 · p. 22, Figure 5 · p. 22

Scale Control with Pose Conditioning

  • With pose conditioning, Layout + Pose stays closer to ground truth than control-video-only and Cosmos transfer 2.5, reducing scale drift and improving geometric fidelity.Control video alone provides rough layout guidance but its estimated trajectory departs from ground truth.Source: §5.3 · pp. 22–23, Figure 6 · p. 23

Environment Control

  • Across snowy, rainy, foggy, night-and-rainy, golden-hour, and blue-hour conditions, generated scenes show condition-specific appearance while largely preserving road structure, traffic layout, and semantics.Night-and-rainy generation combines nighttime illumination with rainy-weather effects.Source: §5.3 · p. 23, Figure 7 · p. 24, Figure 8 · p. 25

Special Scene Control

  • Conditioned on scene-level semantic descriptions, the model generates plausible rare traffic events while maintaining coherent geometry, traffic layout, and visual realism.Examples include accidents, abnormal road occupancy, and uncommon traffic participants, supporting generation of long-tail and safety-critical scenarios.Source: §5.3 · p. 23, Figure 9 · p. 26

5.4 Few-step Distillation

HelloWorld distills its 20-step rectified-flow teacher into a 4-step sampler using dCM followed by DMD. DMD improves pose and cross-view consistency beyond the teacher on the matched first-frame condition while reducing inference cost.

5.4 Few-step Distillation

  • The two-stage pipeline trains a 4-step student by applying dCM to the 20-step RF teacher, then DMD score-matching with the frozen RF teacher and fake-score network under causal KV rollout.At test time, the student uses CFG=1 and a single conditional forward without a CFG branch.Source: §5.4 · p. 26
  • With the first-frame condition, DMD is best on Rot, Trans, Trans*, and CSE; without it, DMD still beats the teacher on those metrics (Rot 2.311°, Trans 1.034, Trans* 3.193, CSE 4.223).The comparison separates rotation and cross-view error from translation.Source: §5.4 · p. 26, Table 11 · p. 27, Table 12 · p. 27
  • dCM distillation increases rotation and CSE errors; translation changes vary, with Trans* improving in both conditions and Trans worsening without a first frame.With a first frame, dCM changes rotation from 2.027° to 2.568° and CSE from 4.180 to 5.201; without one, they change from 3.334° to 4.619° and 4.892 to 6.399.Source: §5.4 · p. 26, Table 11 · p. 27, Table 12 · p. 27
  • DMD raises mean VBench above the teacher to 0.7906 / 0.7938 from 0.7754 / 0.7901 and leads dynamic degree at 0.9900 / 1.0000.Subject and background improve only without a first frame; aesthetic and imaging quality without a first frame remain highest for the teacher, while motion smoothness stays near 0.98 for every row.Source: Table 13 · p. 27, §5.4 · p. 27
  • At 720p, the 4-step DMD student cuts DiT ODE time from 305.3 s to 30.8 s (9.91×), sampling time from 357.7 s to 77.8 s (4.60×), and end-to-end time by 2.98×.Memory is essentially matched; VAE encoding and decoding take about 32 s on both sides.Source: §5.4 · p. 27, Table 14 · p. 28

5.5 LiDAR Reconstruction and Conditional Generation

The section evaluates LiDAR reconstruction and RGB-conditioned LiDAR generation, reporting that temporal RRN improves several measures but not every geometric measure. It also presents qualitative reconstructions and generated LiDAR examples.

5.5 LiDAR Reconstruction and Conditional Generation

  • Range MAE uses jointly valid rays, while Ray-Align and Static-Align diagnose ray stretching and motion-assisted temporal agreement rather than complete coverage or camera–LiDAR correspondence.Static-Align uses ego-motion compensation and reference-derived transforms; neither structural diagnostic alone establishes correct geometry or semantic alignment with RGB.Source: §5.5 · p. 28

5.5.1 Comparison with Cosmos1

  • Temporal RRN improves all five reported measures over Cosmos1 under the common three-camera output-region protocol.Cosmos1 has less radial streaking than the unrectified generator by Ray-Align, despite higher geometric errors and weaker temporal agreement; within the pipeline, RRN increases Chamfer from 1.0134 to 1.0426 m. The output region is equalized, but input information, training budget, and architecture are not.Source: Table 15 · p. 29, §5.5.1 · p. 29

5.5.2 LiDAR VAE reconstruction and rectification

  • Temporal RRN lowers VAE MAE by 5.2%, Chamfer by 7.1%, and edge prediction-to-reference distance by 29.7% versus the unrectified VAE.Static-Align rises from 82.15% to 89.74% (+7.59 percentage points), while Warp Chamfer falls from 0.4668 to 0.3668 m. These gains do not imply better return coverage, and the separately trained models do not isolate temporal input alone.Source: Table 16 · p. 29, §5.5.2 · p. 29

5.5.3 RGB-conditioned LiDAR generation

  • Ray-Align falls from 0.4366 to 0.2822 and Static-Align rises from 67.41% to 83.10% (+15.69 percentage points) with temporal RRN, while Chamfer remains worse than the generator.These are full-panorama RGB-to-LiDAR generation results using the same saved generator outputs, with rectification applied after generation.Source: Table 17 · p. 30, §5.5.3 · p. 30

5.5.4 Qualitative results

  • Figures 10–12 show VAE reconstructions, synchronized generated RGB and LiDAR outputs, and clear-morning and light-fog LiDAR examples conditioned on generated RGB.These are qualitative examples: the altered-weather scenes lack paired target scans, and the stills do not establish long-horizon consistency or controlled weather effects.Source: §5.5.4 · pp. 30, 33, Figure 10 · p. 31, Figure 11 · p. 32

5.6 Super-Resolution Refinement

HelloWorld applies super-resolution to generated frames to improve local visual fidelity while preserving the global scene layout and object structure.

5.6 Super-Resolution Refinement

  • Super-resolution restores high-frequency details and sharper structural boundaries than bicubic interpolation, clarifying lane markings, vehicle fronts, headlights, and license-plate regions.Figure 13 highlights sharper boundaries and richer local detail in zoomed-in lane-marking and vehicle regions.Source: §5.6 · p. 33, Figure 13 · p. 34
  • The enhancement preserves global scene layout and object structure, making generated videos more suitable for high-resolution visualization and downstream analysis.Source: §5.6 · p. 33

6 Applications

HelloWorld combines controllable counterfactual editing of recorded scenes with interactive observation generation under updated ego motion, extending driving simulation beyond replaying recorded logs.

6.1 Special-Scene Editing

  • Special-scene editing preserves road geometry, ego trajectory, and structured context while changing semantic or object-level conditions to introduce a target event.This tests scene-consistent counterfactual editing while maintaining geometric and contextual coherence.Source: §6.1 · p. 34
  • HelloWorld inserts long-tail traffic events—including pedestrian and motorbike conflicts—while preserving scene layout and multi-view alignment, supporting safety-oriented augmentation and stress testing.Examples include a pedestrian stepping out from behind a parked bus and pedestrians and a motorbike crossing ahead.Source: §6.1 · p. 34

6.2 Closed-Loop Simulation

  • Coupled with Alpamayo 1.5, HelloWorld iteratively generates multi-view observations along newly predicted ego trajectories instead of replaying the recorded sequence.Alpamayo predicts the next trajectory from generated observations; HelloWorld conditions its next temporal chunk on committed observation history, updated ego motion, and available scene conditions.Source: §6.2 · p. 35
  • Block-causal generation and self-generated-context adaptation support repeated closed-loop execution, while the few-step student reduces each interaction step’s sampling cost.Both the action model and world model progressively operate beyond the recorded trajectory, with the world model conditioning on its generated history and newly predicted ego motion.Source: §6.2 · p. 35

7 Limitations

HelloWorld’s scope is bounded by its observation interface, available supervision, and finite history. Its generated observations have important limits in online causality, modeled dynamics, long-horizon behavior, and geometric coverage.

Limitations

  • The model’s scope is bounded by its observation interface, available supervision, and finite history.Source: §7 · p. 35

Observation causality and external dynamics

  • Block-causal observation generation does not ensure strict online causality because some control-tower paths are bidirectional; prescribed ego trajectories omit vehicle dynamics, policy feedback, and reactive traffic.Generated observations do not constitute a validated closed-loop simulator.Source: §7 · p. 35

Finite memory and conflicting conditions

  • A bounded KV cache provides no persistent scene memory for loop closure or revisited streets, while text, pose, and layout conditions can conflict without a guaranteed priority rule.Long-horizon evaluation must assess condition adherence, visibility changes, and memory-boundary behavior as well as visual smoothness.Source: §7 · p. 35

Compression and observation coverage

  • Few-step inference can change per-chunk quality and subsequent-history distributions, so favorable short-window trade-offs do not establish equivalent long-horizon behavior.Image refinement also adds a separate quality-and-latency trade-off.Source: §7 · p. 35
  • Native RGB resolution is 480 × 832, and layout response is constrained by annotation gaps in unlabeled or occluded regions.Image refinement adds a separate quality-and-latency trade-off.Source: §7 · p. 35

Geometric extension

  • LiDAR quality depends on RGB inputs and calibration; recorded-RGB success does not establish generated-RGB quality, and compression can lose thin structures, boundaries, or rare returns.A fleet-mean conditioning rig is not equivalent to per-clip calibration, and recorded- versus generated-camera conditioning creates different input distributions.Source: §7 · p. 35

8 Conclusion · A HelloWorld Team · Project Leader

HelloWorld organizes driving observation generation around a shared block-causal transition, progressively adding multi-view communication and structured scene control. It targets reliable repeated execution and few-step sampling, while extending the observation interface to conditional LiDAR synthesis.

8 Conclusion

  • A supervision-driven curriculum progressively adds calibrated multi-view communication and structured scene control to a pose-conditioned video model.Source: §8 · p. 35
  • Generated-history alignment addresses the states encountered during repeated execution.Source: §8 · p. 35
  • Consistency and distribution-matching distillation target few-step sampling of the same block-causal transition.Source: §8 · p. 35
  • Capability retention is evaluated across appearance coverage, cross-view geometry, condition response, horizon, and execution cost.Source: §8 · p. 36
  • Conditional LiDAR synthesis extends the observation interface through explicit return validity, geometric compression, and native-grid multiview conditioning.Source: §8 · p. 36
  • Persistent memory, fully causal conditioning, and policy-coupled dynamics remain directions for extending the observation model.Source: §8 · p. 36

Related papers

Loading 2609.28931v1…