PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaWM: Adaptive World Model based Planning for Autonomous Driving

Hang Wang, Xin Ye, Feng Tao, Chenbin Pan, Abhirup Mallik, Burhaneddin Yaman, Liu Ren, Junshan Zhang

arXiv:2501.13072v2cs.ROcs.AI

TL;DR

Pretrained world-model reinforcement learning can degrade when distribution shifts separate a new driving task from pretraining. AdaWM identifies whether model or policy mismatch dominates and selectively finetunes that component; CARLA experiments report improved routing success rate and time-to-collision.

  • Problem

    Distribution shifts between pretraining and new tasks can cause significant performance degradation during online finetuning of pretrained world-model policies.

    Source: §1 · p. 2

  • Method

    AdaWM identifies the dominant model or policy mismatch, then selectively applies low-rank updates to the dynamics model or policy.

    Source: §4 · p. 10

  • Results

    CARLA experiments show AdaWM improves routing success rate and time-to-collision performance.

    Source: §4 · pp. 10–11

  • Takeaways & Limitations

    The authors emphasize choosing an efficient and robust finetuning strategy for challenging real-world tasks.

    Source: §4 · p. 11

    The theoretical analysis assumes a specified latent RNN dynamics form and bounded Frobenius norms for weight matrices W, U, and V.

    Source: §2.1 · p. 5

Abstract

from arXiv · show

World model based reinforcement learning (RL) has emerged as a promising approach for autonomous driving, which learns a latent dynamics model and uses it to train a planning policy. To speed up the learning process, the pretrain-finetune paradigm is often used, where online RL is initialized by a pretrained model and a policy learned offline. However, naively performing such initialization in RL may result in dramatic performance degradation during the online interactions in the new task. To tackle this challenge, we first analyze the performance degradation and identify two primary root causes therein: the mismatch of the planning policy and the mismatch of the dynamics model, due to distribution shift. We further analyze the effects of these factors on performance degradation during finetuning, and our findings reveal that the choice of finetuning strategies plays a pivotal role in mitigating these effects. We then introduce AdaWM, an Adaptive World Model based planning method, featuring two key steps: (a) mismatch identification, which quantifies the mismatches and informs the finetuning strategy, and (b) alignment-driven finetuning, which selectively updates either the policy or the model as needed using efficient low-rank updates. Extensive experiments on the challenging CARLA driving tasks demonstrate that AdaWM significantly improves the finetuning process, resulting in more robust and efficient performance in autonomous driving systems.

1 INTRODUCTION

Pretraining can accelerate autonomous-driving learning, but distribution shifts can make pretrained models and policies perform poorly during online finetuning. AdaWM diagnoses the dominant mismatch and selectively finetunes the model or policy; CARLA experiments report improved performance across new tasks.

1 INTRODUCTION

  • Distribution shifts between pretraining and new tasks can cause significant performance degradation, despite pretraining’s aim of accelerating learning and improving performance.Learning a world model and planning policy from scratch can be prohibitively time-consuming in complex driving environments.Source: §1 · p. 2
  • In right-to-left turns under similar traffic, model-only finetuning recovers faster because the shift primarily challenges the dynamics model.Figure 1 compares alternate, model-only, and policy-only strategies; all initially degrade, but model-only recovers significantly faster.Source: §1 · p. 2, Figure 1 · p. 2
  • AdaWM identifies whether model or policy mismatch dominates, then selectively finetunes that component using efficient low-rank updates.The dynamics model uses a LoRA-based approach, while the policy is decomposed into a weighted convex ensemble of sub-units whose weights are updated.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • On CARLA tasks, AdaWM shows superior routing success rate and time-to-collision performance while mitigating finetuning drops across new tasks.The reported results support the importance of identifying and addressing the dominant mismatch during finetuning.Source: §1 · p. 3
  • Prior world-model approaches emphasize pretraining, while existing driving systems lack a unified dynamic-adaptation framework or use single-focus finetuning strategies.AdaWM is reported to outperform supervised methods without finetuning and RL approaches with rigid adaptation strategies.Source: §1 · pp. 3–4

2 ADAWM: ADAPTIVE WORLD MODEL BASED PLANNING

The method learns a latent dynamics model and planning policy, analyzes how model and policy mismatches contribute to performance gaps, and adapts finetuning to the dominant mismatch. Its updates target either the model or policy using low-rank parameterizations.

2 ADAWM: ADAPTIVE WORLD MODEL BASED PLANNING

  • Offline pretraining encodes observations and history into latent states, learns recurrent dynamics to predict futures, then trains a planning policy using model rollouts and value.The dynamics model can also predict rewards and episode termination signals.Source: §2 · p. 4

2.1 IMPACT OF MISMATCHES ON PERFORMANCE DEGRADATION

  • The theoretical analysis uses a specified latent RNN dynamics form and assumes bounded weight matrices, bounded actions, and Lipschitz policy and reward functions.The hidden-state mapping uses the current state and action; a second mapping converts the hidden state to the next state representation.Source: §2.1 · p. 5
  • The performance gap compounds shifted-task state-prediction errors and policy suboptimality; prediction error separates task shift from pretrained-model error.The analysis attributes these effects to differences between current and pretraining states, and to prediction error of the pretrained RNN on pretraining tasks.Source: §2.1 · pp. 5–6
  • AdaWM estimates model and policy mismatch with total variation distances, updating the dynamics model when its mismatch exceeds a threshold relative to policy mismatch, and otherwise updating the policy.This simplified criterion follows the theoretical analysis while reducing computational complexity.Source: §2.1 · p. 6

2.2 ADAWM: MISMATCH IDENTIFICATION AND ALIGNMENT-DRIVEN FINETUNING

  • AdaWM identifies the dominant mismatch and selectively updates either the dynamics model or policy, updating low-rank model parameters or policy ensemble weights.For the model, only the base-vector weight is updated; for the policy, only the weights of its convex-combination sub-units are updated.Source: §2.2 · p. 7

3 EXPERIMENTAL RESULTS

CARLA experiments compare AdaWM with baseline algorithms and finetuning strategies across increasingly difficult driving tasks. AdaWM outperforms the reported baselines, while its mismatch-guided updates align model and policy and remain robust across a range of C values.

3 EXPERIMENTAL RESULTS

  • CARLA evaluation uses BEV observations across four increasingly difficult tasks, comparing AdaWM with offline VAD, UniAD, and DreamerV3 using TTC and SR.Tasks vary route, traffic density, and town; pretraining uses CARLA leaderboard v2 and Bench2Drive tasks.Source: §3 · pp. 7–8, §3.1 · p. 8

3.1.1 EVALUATION RESULTS

  • Across CARLA tasks AdaWM outperforms pretrained-only counterparts, VAD, and UniAD; on LTD03 it scores TTC 1.92 and SR 0.70 versus DreamerV3’s TTC 0.15 and SR 0.35.Table 2 summarizes the impact of finetuning across the CARLA tasks.Source: Table 2 · p. 8, §3.1.1 · pp. 8–9

3.2 ABLATION STUDIES

  • Model-only finetuning increases policy mismatch, policy-only finetuning increases dynamics-model mismatch, while AdaWM aligns both factors during finetuning.Figure 4 compares mismatch changes across the strategies.Source: §3.2 · p. 9, Figure 4 · p. 10

3.1.1 EVALUATION RESULTS

  • On ROM03, AdaWM reaches TTC 2.05 and SR 0.82, versus model-only TTC 0.95 and SR 0.60 and policy-only TTC 0.46 and SR 0.72.The reported comparison also finds AdaWM consistently better than model-only, policy-only, and alternate finetuning.Source: §3.1.1 · p. 9, Table 3 · p. 10

3.2 ABLATION STUDIES

  • Performance remains strong for C values from 2 to 50, whereas C = 100 degrades results across tasks and very small C yields insufficient dynamics-model updates.Figure 5 shows AdaWM prioritizing model or policy updates according to which mismatch is more dominant.Source: §3.2 · pp. 9–10, Table 4 · p. 10, Figure 5 · p. 10

4 CONCLUSION

AdaWM addresses finetuning performance drops by identifying whether model or policy mismatch dominates and selectively updating the corresponding component. CARLA experiments report improvements in route success rate and time-to-collision.

4 CONCLUSION

  • AdaWM identifies the dominant mismatch and selectively applies low-rank updates to the dynamics model or policy.The two components are mismatch identification and alignment-driven finetuning.Source: §4 · p. 10
  • CARLA experiments report improved route success rate and time-to-collision with AdaWM.The conclusion names these outcomes but supplies no numerical values.Source: §4 · pp. 10–11
  • Future research avenues include testing AdaWM beyond autonomous driving and extending it to multi-agent settings.The conclusion presents both as directions for future research.Source: §4 · p. 11

ETHICS STATEMENT

The statement identifies safety, pretraining-data bias, decision transparency, and societal impacts as ethical considerations for autonomous driving deployment.

ETHICS STATEMENT

  • Real-world safety, unintended pretraining-data biases, and decision transparency are identified as ethical considerations.The statement says these should be considered in system operation and finetuning.Source: ETHICS STATEMENT · p. 11
  • The statement identifies public safety and employment as societal implications requiring ongoing attention.Source: ETHICS STATEMENT · p. 11
  • AdaWM is framed as addressing performance degradation from distribution shifts in autonomous driving.The statement links this focus to safer and more reliable decision-making in dynamic environments.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The reproducibility statement promises documentation, replication instructions, hyperparameters, and experimental settings for CARLA experiments. It says these resources are intended to support replication and application to other tasks or environments.

REPRODUCIBILITY STATEMENT

A PROOF OF UPPER BOUND OF PREDICTION ERROR

The proof bounds prediction error by separating distribution shift from pretrained-model generalization error, then bounding the resulting terms. It also combines bounds on model-related and policy-related contributions to the performance gap.

A PROOF OF UPPER BOUND OF PREDICTION ERROR

  • Lemma 1 bounds prediction error at rollout step k with probability at least 1 − δ under Assumptions 1 and 2.The proof decomposes the error into a distribution-shift term and pretrained RNN generalization error before bounding them.Source: A PROOF OF UPPER BOUND OF PREDICTION… · p. 14
  • The proof derives an upper bound for the non-stationary part of prediction error and then uses Lipschitz properties to bound error propagation.The resulting propagation bound is applied recursively across rollout steps.Source: A PROOF OF UPPER BOUND OF PREDICTION… · pp. 14–15
  • The RNN generalization-error bound assumes training and testing data share a distribution.The proof further states that its setting uses data from the same simulation platform and assumes uniform input distribution.Source: A PROOF OF UPPER BOUND OF PREDICTION… · p. 15

B PROOF OF THEOREM 1

  • Theorem 1's proof bounds the performance-gap terms separately, then combines them into an overall bound.The first term involves modeling error and policy sub-optimality; the second is relevant only to modeling error.Source: B PROOF OF THEOREM 1 · p. 16

B.1 DERIVATION OF C1 AND C2

The derivation separates the bound on performance degradation into dynamics-model and policy mismatch terms, then uses their relative size to identify the dominant mismatch and select an update.

B.1 DERIVATION OF C1 AND C2

  • The bound separates dynamics-model mismatch from policy mismatch, and the larger term identifies the dominant source of performance degradation.The section derives criteria for updating the model or policy according to which mismatch dominates.Source: §B.1 · pp. 16–17
  • Update the dynamics model when the model-error term exceeds its threshold relative to policy mismatch; update the policy when policy mismatch dominates.The corresponding rules are stated in terms of Ê_P, Eπ, C1, and C2.Source: §B.1 · p. 17

C EXPERIMENTS DETAILS

The experiments use CARLA tasks with BEV observations, discrete acceleration and steering actions, and a reward built from driving objectives. The section also distinguishes a world model’s broader scope from a dynamics model’s focus on state transitions.

C EXPERIMENTS DETAILS

  • The vehicle state is represented by a 128 × 128 bird’s-eye-view semantic segmentation image.Environmental observations include sensor information about objects and their geographical context.Source: C EXPERIMENTS DETAILS · p. 17
  • At each time step, the discrete action space combines acceleration choices [−2, 0, 2] with steering choices [−0.6, −0.2, 0, 0.2, 0.6].Source: C EXPERIMENTS DETAILS · p. 17
  • The reward is a weighted sum of six factors covering safety, comfort, travel time, velocity, orientation, and waypoint following.The velocity factor penalizes speeding beyond 5 m/s when the leading vehicle is too close.Source: C EXPERIMENTS DETAILS · p. 17
  • The CARLA benchmark configuration lists five tasks: signalized and non-signalized junction turns, plus a NoScenario route, across Town03 and Town12.The evaluation metric is measured while the agent follows predetermined waypoints.Source: C EXPERIMENTS DETAILS · p. 17

D TERMINOLOGY

  • A world model represents observations and rewards as well as state transitions, whereas a dynamics model specifically predicts state evolution from state and action.Source: D TERMINOLOGY · p. 19

E WORLD MODEL TRAINING

The training setup uses a Dreamer v3-style RSSM world model and actor-critic learning, with BEV representations constructed from privileged CARLA information. Bench2Drive supplies diverse driving scenarios, and the visualization comparison reports smaller prediction errors for AdaWM than the other finetuning approaches.

E WORLD MODEL TRAINING

  • The world model uses a Dreamer v3-style encoder-decoder and RSSM, with a large model for all experiments.The setup also restates Dreamer v3 hyperparameters in Table 6.Source: E WORLD MODEL TRAINING · p. 19, Table 6 · p. 20
  • CARLA privileged information, including location and map topology, is used to construct the BEV representations.Source: E WORLD MODEL TRAINING · p. 19
  • The RSSM sequence model updates its hidden state from previous state, latent state, and action, while its predictors estimate latent state, reward, continuation, and observation.Source: E WORLD MODEL TRAINING · p. 20
  • World-model parameters are trained end-to-end on prediction, dynamics, and representation losses with weights 1, 0.5, and 0.1, respectively.The loss is applied to batches of observations, actions, rewards, and continuation flags.Source: E WORLD MODEL TRAINING · p. 20
  • Actor-critic training uses a 16-step prediction horizon, and bootstrapped λ-returns integrate predicted rewards and values beyond that horizon.The discount factor is γ = 0.997.Source: E WORLD MODEL TRAINING · p. 21

E.1 TRAINING DATASET: BENCH2DRIVE

  • Bench2Drive contains 2 million annotated frames from 10,000 clips across 44 interactive scenarios, 23 weather conditions, and 12 CARLA towns.Its shorter scenarios support granular evaluation and are described as suitable for reinforcement-learning applications.Source: §E.1 · p. 21

F VISUALIZATION OF MODEL PREDICTION

G SUPPLEMENTARY EXPERIMENTS

The supplementary experiments list five additional CARLA scenarios and provide impact tables for those scenarios. The visualizations also include predictions from model-only, policy-only, and no-finetuning settings.

G SUPPLEMENTARY EXPERIMENTS

  • Five additional CARLA scenarios are listed: HighwayCutIn, HighwayExit, YieldToEmergencyVehicle, BlockedIntersection, and VehicleTurningRoutePedestrian.They span Town11, Town12, and Town13.Source: G SUPPLEMENTARY EXPERIMENTS · p. 22
  • The supplementary prediction figures show 65-step results for model-only finetuning, policy-only finetuning, and no finetuning.Source: Figure 8 · p. 22, Figure 9 · p. 23, Figure 10 · p. 23
  • Tables 7 and 8 report the impact of finetuning across the highway-cut-in and highway-exit tasks and the yield, blocked-intersection, and vehicle-turning tasks.The supplied captions identify the task scope but do not state table values.Source: Table 7 · p. 24, Table 8 · p. 24

G.1 EXPERIMENTS RESULTS

AdaWM improves success rates over baseline methods across highway, emergency-response, blocked-intersection, and pedestrian scenarios. Its alignment-driven finetuning also outperforms policy-only and model-only updates in the reported strategy comparisons.

G.1 EXPERIMENTS RESULTS

  • SR rises from 0.33 to 0.73 in HC13, 0.52 to 0.89 in HE12, 0.15 to 0.52 in YE12, 0.42 to 0.59 in BI12, and 0.58 to 0.88 in VT11.The reported comparisons are against baseline methods UniAD, VAD, and DreamerV3.Source: §G.1 · p. 24
  • In highway scenarios HC13 and HE12, Time-To-Collision metrics show similar improvements alongside the reported Success Rate gains.Source: §G.1 · p. 24
  • In HE12, AdaWM reaches SR 0.89 versus 0.72 for policy-only and 0.70 for model-only finetuning; in VT11, it reaches 0.88 versus 0.76 and 0.44, respectively.Table 9 compares different finetuning strategies.Source: §G.1 · p. 25, Table 9 · p. 25

Related papers

Loading 2501.13072v2…