PaperlayerView exact revision on arXiv

Source-linked AI summary

World4Scorer: Outcome-Grounded World Modeling for Autonomous Driving

Jieyuan Pei, Meiyi Lu, Sining Ang, Yubo Zhao, Zhangyi Hu, Mingwei Xu, Haokai Ding, Wei Li, Zihan You, Jianwei Zheng, Li Yu, Yifeng Pan, Ji Tao, Rongjunchen Zhang, Yan Wang

arXiv:2609.36438v1cs.RO

TL;DR

Generate-and-select driving planners must rank alternatives, but logged future observations cover only the executed trajectory. World4Scorer trains a trajectory-conditioned predictor with candidate outcomes and an observed-future anchor, achieving strong results across driving and manipulation evaluations.

  • Problem

    Driving logs contain future observations only for the executed trajectory, leaving predictions for alternative plans unconstrained even though those predictions inform selection.

    Source: §1 · p. 1

  • Method

    World4Scorer uses a trajectory-conditioned predictor to produce candidate states, trains them with simulator outcomes, and anchors the shared predictor with the executed trajectory’s observed future.

    Source: §1 · pp. 1–2

  • Results

    World4Scorer achieves state-of-the-art NAVSIM-v2 performance and strong closed-loop driving results, while outcome-based scoring improves manipulation with a frozen world model.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The findings support grounding predictive states in decision outcomes.

    Source: §5 · p. 9

Abstract

from arXiv · show

Autonomous driving requires choosing a safe and efficient plan as surrounding traffic evolves. Generate-and-select planners propose multiple trajectories and score them for execution, and they have outperformed representative direct-prediction baselines on NAVSIM. Their scorer must compare plans that were never executed. Driving logs record the future of only the executed trajectory, so matching the logged future can leave predictions for the alternatives unconstrained; a simulator, in contrast, can label the outcome of every candidate. We introduce World4Scorer, which builds the scorer as a trajectory-conditioned JEPA-style predictor: it predicts a state for each candidate and reads the candidate's scores from that state. Simulator outcome labels supervise the states of all candidates, and the observed future of the executed trajectory anchors the predictor to real scene evolution. Because one predictor produces every candidate's state, the anchor can constrain shared parameters used to score unexecuted plans, while the future itself is needed only during training. Generated candidates mostly score well, so a scene-matched bank adds low-scoring plans to the outcome supervision; framewise choices can conflict, so inertial re-ranking keeps consecutive selections consistent. World4Scorer achieves state-of-the-art NAVSIM-v2 performance and a strong adapted-system result on closed-loop Bench2Drive. With the LeWM world model and planning budget fixed, outcome-based scoring also improves manipulation planning on the OGBench-Cube benchmark.

1 INTRODUCTION

World4Scorer addresses the supervision gap in candidate scoring by training a trajectory-conditioned predictor with outcomes for alternatives and an observed-future anchor. It achieves strong results across driving and manipulation evaluations.

1 INTRODUCTION

  • Generate-and-select planners need to rank alternatives, but driving logs record future observations only for the executed plan, leaving alternative predictions unconstrained.The scorer must choose among plans by balancing safety, progress, and comfort.Source: §1 · p. 1
  • Simulator outcomes supervise every candidate’s state, while the executed trajectory’s observed future anchors the shared predictor to real scene evolution.Figure 1 contrasts outcome-label routing with the observed-future anchor in World4Scorer.Source: §1 · p. 1, Figure 1 · p. 2
  • World4Scorer makes a trajectory-conditioned JEPA-style predictor the scorer, reading each candidate’s outcome from its predicted state.A visual readout predicts the executed trajectory’s future features, while shared predictor parameters are trained by candidate outcomes and the observed future.Source: §1 · p. 2
  • World4Scorer achieves state-of-the-art NAVSIM-v2 performance and leads compared methods on NAVSIM-v1, with strong adapted-system results on Bench2Drive and improved manipulation planning.The manipulation result uses outcome-based scoring with a frozen world model.Source: §1 · p. 2

2 RELATED WORK

Prior work spans direct trajectory prediction, candidate-based driving planners, and predictive representations for planning. World4Scorer follows candidate scoring while training its scorer as a predictor with both simulator outcomes and observed futures.

End-to-end autonomous driving

  • World4Scorer follows the generator–scorer framework of DrivoR but makes the scorer a predictor trained with simulator labels and anchored by observed futures.Related driving methods include fixed-vocabulary, confidence-based, coarse-to-fine, and evaluator-supervised candidate selection.Source: §2 · p. 2

Predictive representations for planning

  • Predictive representations support trajectory generation and candidate evaluation; World4Scorer focuses on combining executed-future observations with outcomes for every candidate.Drive-JEPA instead uses video-JEPA pretraining to provide predictive planning features.Source: §2 · p. 2

Overview of World4Scorer

  • Figure 2 shows 64 proposals scored before inertial re-ranking and shared-predictor training with executed-future observations and outcomes for 64 candidates plus 16 bank trajectories.The visual readout supervises the executed trajectory, while outcome labels cover generated and bank candidates.Source: Figure 2 · p. 3

3 World4Scorer

World4Scorer treats candidate scoring as decision-focused world modeling: latent states carry candidate outcomes rather than full futures. The shared predictor is trained with outcomes for candidate plans and the observed future of the executed plan.

3.1 OVERVIEW

  • At inference, DINOv2 with LoRA encodes four camera views, a transformer proposes 64 trajectories, and the scorer evaluates their states before re-ranking by prior-plan compatibility.Training uses winner-take-all imitation, candidate outcome labels, and a logged-trajectory query for observed-future prediction through the same predictor.Source: §3.1 · p. 3

3.2 SHARED PREDICTION AND SCORING

  • Candidate trajectories become query tokens processed with scene tokens by a shared predictor; score heads combine each predicted state with the ego state to predict six measures.Detaching candidate coordinates blocks scoring gradients from reaching generator outputs, while the scoring loss still trains the query encoder, predictor, score heads, and scene features.Source: §3.2 · p. 3

3.3 OUTCOME-GROUNDED PREDICTIVE LEARNING

  • Simulator outcomes supervise every candidate state, while an observed-future target supervises the executed trajectory and constrains shared predictor parameters used for candidate scoring.The visual target is a frozen DINOv2 embedding of the observed front view two seconds later; at deployment, prediction uses current observations and candidate trajectories only.Source: §3.3 · p. 4
  • Future constraints can remove local disagreement over candidate choice, but narrow comparisons only along directions coupled to the executed-query readout.The figure illustrates that this constraint can resolve a disagreement while allowing different predictions elsewhere.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4 OUTCOME SUPERVISION ACROSS CANDIDATES

  • Because generated plans mostly score well, a scene-matched perturbed bank adds low-scoring plans, including unsafe cases; training samples 16 bank trajectories per step.The bank loss is down-weighted to 0.5 to limit its influence on shared features while retaining supervision outside the generated candidate set.Source: §3.4 · p. 5, Figure 4 · p. 5

3.5 INERTIAL RE-RANKING

  • For NAVSIM-v2, inertial re-ranking checks candidate compatibility with the preceding plan using controller rollouts and acceleration, jerk, yawrate, and yaw-acceleration profiles.It uses only the previous plan, current candidates, and controller rollouts at deployment; it adds no learned parameters or training loss.Source: §3.5 · p. 5

4 EXPERIMENTS

The experiments test World4Scorer on NAVSIM and Bench2Drive, examine its supervision and candidate-specific states, and evaluate outcome-based scoring for manipulation. Results include state-of-the-art NAVSIM-v2 performance and gains in the reported Bench2Drive and OGBench-Cube evaluations.

4.1 SETUP

  • The NAVSIM evaluation uses all 12,146 test scenes and official simulator metrics, with the main checkpoint selected on the validation split.World4Scorer uses DrivoR’s training split, encoder, generator, and candidate count.Source: §4.1 · p. 6

4.2 • DRIVING PERFORMANCE (RQ1)

  • World4Scorer reaches 93.0 EPDMS on NAVSIM-v2 and 94.0 PDMS on NAVSIM-v1, the best results reported for those evaluations.Without re-ranking, it reaches 91.4 EPDMS, 0.7 points above DrivoR’s 90.7; re-ranking adds 1.6 EPDMS points. The complete-planner comparison uses each planner’s own generator.Source: §4.2 · p. 7
  • On Bench2Drive, the adapted planner scores 73.27 Driving Score and 46.36% success over 220 routes; its published-reference comparison uses unmatched training data and input interfaces.In two illustrated routes, Ours completes both without lane departures or collisions, while VAD leaves its lane in one and collides in the other.Source: §4.2 · p. 7, Table 3 · p. 7, Figure 6 · p. 7

4.3 ■SUPERVISION AND RE-RANKING (RQ2)

  • Observed-future anchoring raises PDMS from 93.01 to 93.96, while a current-frame target gives 92.70 in the reported single-seed comparison.The runs share architecture, seed, bank supervision, and last-epoch evaluation; the gap is consistent with, but does not isolate, the targets’ temporal content.Source: §4.3 · p. 7
  • Shuffling bank outcome labels lowers ordering accuracy from 64.2% to 55.6% and selected PDM from 0.728 to 0.647; shuffling all labels yields 51.5% and 0.461.The controls preserve scene trajectories and label distributions while breaking their correspondence. With candidates and weights fixed, inertial re-ranking raises extended comfort from 77.0 to 92.8 and yields 1.6 EPDMS points, while progress changes from 90.0 to 89.8.Source: §4.3 · pp. 7–8

4.4 ♦TRAJECTORY-SELECTION ANALYSIS (RQ3)

  • Figure 7 illustrates reassignment changing choices in twelve scenes; Figure 8’s query-conditioned image readouts are qualitative because only the logged trajectory has an observed future.The separately trained reconstruction network uses 6,000 logged current/future pairs; changing the query changes the decoded viewpoint.Source: Figure 7 · p. 8, §4.4 · pp. 8–9, Figure 8 · p. 8
  • Reassigning candidate states lowers PDMS from 94.19 to 79.73 averaged over 100 shuffles; replacing states with the scene mean gives 71.10.Every shuffle lowers PDMS, and the table caption specifies the 12,146-scene, 64-candidate evaluation and that drops are relative to Own state.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 ▲TRANSFER TO MANIPULATION (RQ4)

  • On OGBench-Cube, outcome-based scoring raises LeWM success from 68.91% to 73.64% across 11 evaluation seeds, a paired gain of 4.73 points with 95% CI [+2.18, +7.27].The comparison keeps LeWM’s implementation, world model, and planning budget fixed; each seed uses 50 paired episodes sharing initial states and goals.Source: §4.5 · p. 9, Table 5 · p. 9

5 CONCLUSION

The conclusion presents World4Scorer as a trajectory-conditioned predictor trained with both simulator outcomes and the observed future. The reported findings span driving, state-selection interventions, and manipulation with a frozen world model.

5 CONCLUSION

  • World4Scorer uses a trajectory-conditioned predictor trained by candidate outcomes and anchored by the observed future.The conclusion reports state-of-the-art NAVSIM-v2 performance, strong closed-loop driving results, and improved manipulation planning with a frozen world model.Source: §5 · p. 9
  • Controlled experiments report contributions from correctly paired outcome labels and realized-future targets, while state interventions show selection depends on candidate-specific states.These findings support grounding predictive states in decision outcomes.Source: §5 · p. 9
  • Outcome-based scoring also improves manipulation planning when the world model is frozen.This is the conclusion’s stated cross-domain scope for the method’s reported result.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies the benchmarks, evaluators, implementation details, protocols, and planned code release relevant to reproducing the work.

REPRODUCIBILITY STATEMENT

  • The experiments use public benchmarks and their official evaluators: NAVSIM-v1, NAVSIM-v2, Bench2Drive, and OGBench-Cube.The statement locates benchmark protocols in Appendices D.2 and E.5.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix A.2 and Table 7 give the training configuration, while Appendix A.3 reports model size, compute, and latency.The configuration includes data split, losses and weights, optimizer, schedule, batch size, and checkpoint selection.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The statement says code will be released upon publication and identifies appendices covering simulator labels, re-ranking, and fixed-pool intervention procedures.Appendix B.1 specifies simulator labels; Section 3.5 and Appendix A.2 specify re-ranking; Appendix D.1 gives permutation seeds and bootstrap procedures.Source: REPRODUCIBILITY STATEMENT · p. 10

USE OF AI

This material describes the authors’ use of AI tools and outlines appendix contents on implementation, analyses, and notation.

USE OF AI

  • AI tools assisted with prose editing, LaTeX, figure code, analysis scripts, and cluster tooling; the authors state they determined the scientific claims, design, controls, and interpretation.The authors also state that they verified the manuscript and take responsibility for its content.Source: USE OF AI · p. 10

APPENDIX

  • The appendix provides implementation details and further analyses of prediction and planning, organized across five appendices.Its outline covers notation and cost, simulator labels, visual targets, state reassignment, and evaluation protocols.Source: APPENDIX · p. 14

A SETUP AND REFERENCE

  • Appendix A documents notation, training setup, and computational cost, and its notation reference table covers method and analysis terms.Table 6 notes that time indices are omitted when a scene is fixed and that Section 3.2 defines six predicted score components.Source: A SETUP AND REFERENCE · p. 14, Table 6 · p. 14

A.2 TRAINING AND STATE INTERVENTIONS

The appendix specifies training losses and candidate-state interventions, simulator-based outcome-label construction, evaluation controls, and compute requirements. It describes how candidate outcomes are measured and how the NAVSIM objectives are isolated for comparison.

A.2 TRAINING AND STATE INTERVENTIONS

  • Generated, executed, and bank queries share predictor P; generated candidates receive losses against logged four-second and spline-resampled five-second trajectories.The logged trajectory is time-warped to five seconds by cubic-spline resampling.Source: §A.2 · p. 14
  • Scene-mean replacement or fixed-point-free state shuffling tests candidate states while holding the checkpoint, features, pool, score heads, and aggregation fixed.The intervention uses the epoch-23 checkpoint.Source: §A.2 · p. 15
  • Inertial re-ranking carries the prior selection forward and compares 0.1-second LQR rollouts over their shared 3.5-second window.Validation selects the reported NAVSIM checkpoint, and the test split is excluded from training, validation, and checkpoint selection.Source: §A.2 · pp. 15–16

A.3 COMPUTE AND PARAMETER SCALE

  • World4Scorer has 35.52M parameters and 349.9 billion operations excluding fused attention; generation and scoring take 1.4 of 106.5 ms per pass.Including attention raises the operation count to 927.1 billion; latency figures use fp32 with TF32 disabled, while fp16 runs 4.6× faster.Source: §A.3 · p. 16, Table 8 · p. 16

B EVALUATION PROTOCOLS

  • The NAVSIM-v1 and NAVSIM-v2 objectives are compared with the model and candidate pools fixed to isolate the scoring rule.Source: B EVALUATION PROTOCOLS · p. 16

B.1 SIMULATOR-BASED TRAINING LABELS

  • Simulator labels track each candidate through a four-second LQR-controlled ego-state sequence and measure map compliance, collision avoidance, progress, time-to-collision, and comfort.The scene cache supplies map and occupancy information; labels depend on the trajectory and scene, not other candidates.Source: §B.1 · pp. 16–17
  • The v1 training-label scorer combines progress, time-to-collision, and comfort with the deployed weights, while collision avoidance and drivable-area compliance scale progress and the aggregate score.Driving-direction compliance is computed but has zero v1 weight; changing generated candidates require online labels for all N candidates per sample.Source: §B.1 · p. 17

B.2 DIFFERENCES BETWEEN NAVSIM EVALUATION OBJECTIVES

NAVSIM-v1 and NAVSIM-v2 often select different candidates, with extended comfort accounting for most of their utility gap. The latent-target analysis finds persistent scene content alongside measurable, scene-specific future information.

B.2 DIFFERENCES BETWEEN NAVSIM EVALUATION OBJECTIVES

  • 43% of scenes have different NAVSIM-v1 and NAVSIM-v2 selections on the same frozen candidate pools, with within-scene utility rank correlation 0.69.Removing extended comfort raises correlation to 0.96 and reduces argmax disagreement to 3%.Source: §B.2 · p. 17
  • 98% of the utility gap between v1 and v2 choices is attributed to extended comfort; the v2-optimal candidate gains 50.4 EC points and loses 2.7 progress points.Driving direction accounts for 3% of the gap, while progress, time-to-collision, and comfort together account for −1%.Source: §B.2 · p. 17
  • In 38% of scenes, no candidate is both top-quartile in progress and EC-passing; the share reaches 57% in the most dynamic quarter.Within-scene progress–EC correlation is +0.07.Source: §B.2 · p. 18
  • The recomputed v1 selections differ from benchmark results only on 39 fp32 ties, changing mean PDMS by 6 × 10−7.Recomputed EC matches the benchmark on all 10,040 adjacent pairs; the benchmark omits EC for 2,106 frames without valid adjacent pairs.Source: §B.2 · p. 18

C.1 GEOMETRY OF THE LATENT PREDICTION TARGET

  • Current and +2.0-second DINOv2 features have mean cosine 0.94, while cross-scene pairing reduces it to 0.649, matching the random-pairing baseline.True and shuffled pairs are distinguishable at AUC 0.981, indicating the true-pair advantage is scene-specific.Source: §C.1 · p. 18

C.2 VISUALIZING PREDICTED FEATURES

The section visualizes trajectory-conditioned future features and examines how candidate states and supervision constrain candidate comparisons. Its analyses show that states reflect candidate outcomes and that added supervision can stabilize local selection comparisons.

C.2 VISUALIZING PREDICTED FEATURES

  • The 384-dimensional visual readout predicts a front-view DINOv2 feature 2.0 seconds ahead, while separate reconstruction networks combine predicted features with the current image’s patch grid.The reconstruction networks are for visualization only; the logged trajectory supplies the observed future target.Source: §C.2 · pp. 18–19
  • Changing the queried trajectory shifts decoded viewpoints while preserving scene structure, but alternative-query image accuracy cannot be measured without matching observations.Additional scenes show road openings and visible building surfaces shifting with the query, with subtler differences between nearby plans.Source: §C.2 · p. 19, Figure 11 · p. 20
  • At guidance 2, the observed future’s pooled vector improves regional patch cosine by +0.005 to +0.041, while selected-query whole-grid cosine is 0.681 versus 0.701 without the vector.The readout’s reconstruction depends on both the pooled vector and current spatial grid; guidance 1 yields whole-grid cosine 0.707.Source: §C.2 · p. 19
  • The 30-scene gallery orders scenes by current-to-future feature change and contrasts predicted-feature reconstructions with a decoder ceiling and observed futures.The decoder ceiling isolates decoder quality; larger viewpoint changes leave multiple spatial layouts unresolved after pooling and prediction.Source: §C.2 · p. 21, Figure 12 · p. 21

C.3 CANDIDATE-STATE GEOMETRY

  • 80.2% of a candidate’s five nearest states share its outcome group, versus 76.8% for nearest trajectories and 52.0% after shuffling labels.State agreement exceeds the shuffled value in all 128 scenes and trajectory agreement in 98; the four-scene figure shows candidate-state geometry.Source: §C.3 · pp. 21–22, Figure 13 · p. 22

C.4 SUPERVISION GEOMETRY AND CANDIDATE COMPARISONS

  • A fitted candidate remains the unique winner throughout the local perturbation set when its score margin exceeds the maximal first-order change against every other candidate.This is a local sensitivity result, not a claim about optimization trajectories or global planning performance.Source: §C.4 · pp. 23–24
  • With exact first-order output preservation, a candidate comparison is fixed when its gradient lies in the row space of the stacked observation Jacobians.The unconstrained directions are the intersection of the visual- and outcome-readout Jacobians’ null spaces.Source: §C.4 · p. 23
  • Additional outcome labels reduce comparison sensitivity when they constrain directions left unresolved by existing supervision.The result characterizes local sensitivity under a perturbation budget, not global performance after training.Source: §C.4 · p. 24

D.1 CANDIDATE-STATE INTERVENTIONS WITH FIXED POOLS

Fixed-pool interventions test whether candidate-state assignments matter for selection, while a separate manipulation experiment adds outcome-based scoring to LeWM. Reassignment disrupts driving selection, and outcome scoring improves cube-placement success with the world model and search budget fixed.

D.1 CANDIDATE-STATE INTERVENTIONS WITH FIXED POOLS

  • 14.47 points is the average selected-PDMS loss across 100 random state reassignments versus matched states.Reassignment means range from 77.44 to 81.99, averaging 79.73; the paired interval for the loss is [13.65, 15.29].Source: Table 10 · p. 24, §D.1 · p. 24
  • State reassignment changes the selected candidate on 100% of scenes and reduces ordering accuracy to 49.9%, approximately chance.The intervention preserves states but changes their assigned candidates; the mean-state control gives every candidate the same score.Source: §D.1 · p. 24, Table 10 · p. 24
  • Reassigning DrivoR scorer-attention outputs raises regret by +14.11 points over its matched 5.64 and reduces ordering accuracy from 72.5% to 50.2%.This control indicates state assignment matters even without future prediction.Source: §D.1 · p. 25

D.2 OUTCOME-BASED SCORING INSIDE THE LEWM PLANNER

  • The LeWM integration adds a learned outcome score to standardized negative goal distance when selecting sequences during CEM search.The world model and search budget remain the same; the learned score is weighted by λcube = 0.3.Source: §D.2 · p. 25
  • Outcome-based scoring raises OGBench-Cube success from 68.91% to 73.64% over 550 paired episodes, with the LeWM world model and planning budget fixed.The gain is +4.73 percentage points; success improves on 9 seeds, ties on 1, and decreases on 1.Source: §D.2 · p. 25

E.1 EXTENDED-COMFORT FAILURE TAXONOMY

The extended-comfort analysis identifies why consecutive selections fail and evaluates inertial re-ranking, history controls, and an adapted closed-loop Bench2Drive system. Results include reduced plan discontinuity, gains with fixed candidate pools, and six completed Bench2Drive routes.

E.1 EXTENDED-COMFORT FAILURE TAXONOMY

  • Score–continuity conflicts account for 71.3% of 2,717 framewise extended-comfort failures, while 0.7% are near-ties and 88.7% of pool-limited failures occur in the most dynamic quarter.After inertial re-ranking, 91.7% of residual failures have no compatible candidate; DrivoR’s 7.3% incompatible pairs cap its pool’s EC at 92.66. Human plans pass the check on 90.1% of adjacent pairs and 16.5% of pool-limited pairs.Source: §E.1 · p. 26, Table 11 · p. 26

E.2 CHANGES IN CONSECUTIVE SELECTIONS

  • With planner weights and candidates fixed, inertial re-ranking changes 1,939 pairs from EC failure to pass and 18 from pass to failure, reducing heading jitter by 32% on repaired pairs.It also reduces lateral discrepancy by 21% and consecutive-plan L2 distance by 11%; among changed selections, distance to the logged plan drops from 1.87 to 1.51 m.Source: §E.2 · p. 26

E.3 RE-RANKING WITH FIXED CANDIDATE POOLS

  • With DrivoR’s weights and candidates fixed, inertial re-ranking raises EPDMS from 90.66 to 92.27; varying Coverage’s state weight eight-fold moves EPDMS by at most 0.07 points.The gain over framewise selection persists for λstate ∈ {0.25, 0.5, 1, 2}.Source: §E.3 · p. 26

E.4 HISTORY CORRUPTION CONTROLS

  • Using the correct previous plan improves EPDMS by 2.96 points over wrong-log history and 1.01 points over two-step-old history; copying the previous plan scores below framewise selection.The controls use the same fixed Coverage pool and scores, changing only the history supplied to the EC check.Source: §E.4 · pp. 26–27, Table 12 · p. 27

E.5 CLOSED-LOOP EVALUATION ON BENCH2DRIVE

  • Bench2Drive deployment retains route commands until the vehicle passes a route node and re-ranks candidates by route agreement within 30 m, combined with learned utility.The route-blind checkpoint is adapted with a route-point input 20 m ahead; the same coordinate conversion is used in training and deployment.Source: §E.5 · pp. 27–28
  • All six shown Bench2Drive routes complete with Driving Score and Route Completion both equal to 100 for Ours, though published baselines were not retrained under matched conditions.The evaluation uses an adapted system trained on the official 1,000-clip subset.Source: §E.5 · p. 27, Figure 14 · p. 28

Related papers

Loading 2609.36438v1…