PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveMind: Mind-Evolving Belief Tracking for Closed-Loop Autonomous Driving

Yan Shi, Kehua Chen, Jiarui Qi, Yihong Tang, Haichen Liao, Yinhai Wang

arXiv:2610.32069v1cs.AI

TL;DR

MLLM-based drivers often reason from a single interpretation at each frame, despite ambiguity and changing observations. DriveMind tracks multiple weighted hypotheses and updates them over time; on Bench2Drive, belief tracking improves DS from 68.50 to 81.37 and SR from 41.36% to 54.09%.

  • Problem

    Existing MLLM-based drivers generate a single rationale or trajectory at each inference step, although partial and changing observations can leave several scene interpretations plausible.

    Source: §1 · p. 1

  • Method

    DriveMind maintains a weighted set of natural-language scene hypotheses, updates them with new observations, and pairs slow MLLM belief tracking with fast diffusion-based trajectory generation.

    Source: §1 · p. 2

  • Results

    Belief tracking improves Bench2Drive DS from 68.50 to 81.37 and SR from 41.36% to 54.09%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported gains are largest on long-tail scenarios requiring intent disambiguation, and experiments confirm improved performance and robustness in ambiguous scenarios.

    Source: §1 · p. 2, §6 · p. 10

    The prompted VLM update lacks an explicit probabilistic state-transition model, and normalized weights express relative support rather than calibrated probabilities.

    Source: §4.1 · pp. 4–5

Abstract

from arXiv · show

Recent multimodal large language models (MLLMs) can perceive a traffic scene, reason about it, and propose a driving action within a single model. However, existing MLLM-based drivers reason from scratch at each frame and commit to a single explanation, discarding earlier hypotheses about the scene. When the scene is ambiguous, the agent locks in on the wrong interpretation and acts on it before evidence accumulates. In this paper, we propose DriveMind, an MLLM-based driving framework that keeps several hypotheses about the scene alive at the same time and updates their weights as new frames arrive. Inspired by particle filtering, DriveMind propagates each hypothesis through a VLM, scores it against the current observation, and resamples to maintain diversity. On Bench2Drive, adding belief tracking to the same driving pipeline improves the Driving Score (DS) from 68.50 to 81.37 and the Success Rate (SR) from 41.36% to 54.09%. The largest gains occur on ambiguous scenarios such as unsignalized intersections (+20.0 DS) and oncoming-traffic interactions (+20.5 DS). Beyond the overall metrics, DriveMind also improves fine-grained interactive driving abilities such as merging, emergency braking, and give-way.

1 Introduction

MLLM-based drivers can commit to one scene interpretation before later frames resolve ambiguity, and closed-loop actions may reinforce an initial mistake. DriveMind carries multiple weighted hypotheses forward and updates them as observations arrive.

1 Introduction

  • Single-rationale MLLM drivers can commit too early under ambiguity, and their resulting actions may reinforce an initial misinterpretation through later observations.A slowing vehicle or a pedestrian near the curb can each support multiple plausible interpretations.Source: §1 · pp. 1–2
  • DriveMind improves Bench2Drive DS from 68.50 to 81.37 and SR from 41.36% to 54.09% against DriveMind without belief tracking.On NAVSIM, belief tracking improves PDMS from 78.2 to 81.0.Source: §1 · p. 2
  • DriveMind maintains and updates a weighted set of natural-language scene hypotheses as new frames arrive.The figure contrasts one rationale per timestep with multiple hypotheses propagated over time, with darker color indicating higher weight.Source: §1 · p. 2, Figure 1 · p. 2
  • Its asynchronous fast–slow architecture uses a slow MLLM for belief tracking and a fast diffusion controller for real-time trajectory generation.The controller conditions trajectories on the latest belief.Source: §1 · p. 2
  • Belief tracking improves performance and robustness on Bench2Drive and NAVSIM, with the largest gains on long-tail scenarios requiring intent disambiguation.The introduction names unsignalized intersections as one such scenario, with a +20.0 DS gain.Source: §1 · p. 2

2 Related Work

Prior autonomous-driving systems use language models across scene understanding, decision-making, and planning, while uncertainty methods largely assess static outputs. DriveMind focuses on temporal belief updates over competing scene interpretations.

2.1 Autonomous Driving with LLM

  • Language-model driving systems span scene understanding, decision-making, and motion planning, while some also use memory and reflection to accumulate experience.DriveMind instead carries forward and reweights multiple competing scene interpretations as observations arrive.Source: §2.1 · pp. 2–3

2.2 Uncertainty in Reasoning and Prediction

  • Existing uncertainty methods assess static responses or final predictions, whereas classical planners maintain temporal beliefs over predefined state spaces.DriveMind uses VLM-generated natural-language hypotheses to represent scene interpretations without relying on predefined state and behavior representations.Source: §2.2 · p. 3
  • Figure 2 depicts a slow VLM maintaining a multi-hypothesis belief state while a fast diffusion planner generates trajectories asynchronously.The figure also shows hypothesis propagation and reweighting as observations arrive.Source: Figure 2 · p. 3

3 Problem Formulation

The paper formulates driving as sequential decision-making under partial, changing observations and represents open-ended scene reasoning with a finite weighted set of hypotheses. The belief state approximates the posterior and conditions trajectory generation.

3 Problem Formulation

  • At each timestep, the agent receives an observation, executes an action, and the environment transitions to a new observation.The driving policy is expressed by marginalizing over hypotheses conditioned on the available history.Source: §3 · p. 3
  • Because the natural-language reasoning space is open-ended, selecting one hypothesis forces early commitment and can make recovery difficult if it is wrong.DriveMind instead uses a finite weighted set of K hypotheses as an approximation to the posterior over reasoning hypotheses.Source: §3 · pp. 3–4
  • The belief state is propagated through closed-loop interaction and serves as the conditioning signal for trajectory generation.The weighted-set approximation is substituted into the policy's marginalization over hypotheses.Source: §3 · p. 4

4 Methodology

DriveMind runs a slow VLM reasoner that updates a multi-hypothesis belief and a fast diffusion controller that generates trajectories from the latest belief. The reasoner propagates, scores, and resamples hypotheses, while the controller aggregates them into a belief context for motion generation.

4 Methodology

  • The VLM reasoner updates the belief around 2 Hz, while the diffusion controller generates trajectories around 10 Hz and reuses the latest belief between updates.The reasoner is trained with maneuver and score-prediction supervision.Source: §4 · p. 4, §4.2 · pp. 5–6

4.1 High-Level Reasoner: Multi-Hypothesis Belief-State Propagation

  • The reasoner predicts updated hypotheses from prior hypotheses, the previous action, and the current observation, then reweights them by observation consistency.The VLM’s consistency scores are rescaled to [0,1], and normalized weights represent relative support rather than calibrated probabilities.Source: §4.1 · pp. 4–5
  • When effective sample size falls below αK, DriveMind resamples by weight and regenerates near-duplicate hypotheses to preserve diverse interpretations.It identifies near-duplicates by embedding similarity above ρ = 0.9, retains the higher-weighted hypothesis, and regenerates the other.Source: §4.1 · p. 5

4.2 Low-Level Controller: Belief-Conditioned Trajectory Generation

  • The diffusion controller conditions waypoint generation on a weight-aggregated embedding of all hypotheses and visual evidence, rather than only the top-ranked hypothesis.It generates a short-horizon sequence of eight ego-frame waypoints and executes it in a receding-horizon manner.Source: §4.2 · p. 5

4.3 Closed-loop Inference and Uncertainty Tracking

  • In representative scenarios, belief entropy starts near log K and decays as scenes become less ambiguous, while Night Accident retains persistent ambiguity and diversity.The reported analysis also tracks the dominant hypothesis weight and effective sample size across scenarios.Source: §4.2 · p. 6, Figure 3 · p. 6, §4.3 · p. 6

5 Experiments

Experiments evaluate DriveMind on Bench2Drive and compare it with a baseline using the same VLM and diffusion controller but a single scene interpretation. Results show gains in overall driving performance and challenging interaction scenarios, while ablations examine hypothesis tracking, scoring, and runtime.

5.1 Benchmarks

  • Bench2Drive evaluation follows the official protocol across 220 short routes of approximately 150 m, reporting closed-loop metrics and a five-ability breakdown.The abilities are merging, overtaking, emergency braking, give-way, and traffic-sign handling.Source: §5.1 · p. 6

5.2 Results

  • DS rises from 68.50 to 81.37 (+12.87) and SR from 41.36% to 54.09% (+12.73 percentage points) over DriveMind w/o belief.DriveMind also reaches the highest multi-ability mean score in Table 1, at 58.17%.Source: Table 1 · p. 7, §5.2 · p. 7
  • Opposite Direction gains +20.5 DS and Unsignalized Intersection gains +20.0 DS; SR is unchanged at 13.3% in Opposite Direction and falls from 12.0% to 4.0% in Signalized Intersection.The largest DS gains occur where decisions depend on uncertain interactions with other agents.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • DriveMind improves DS and SR in all six selected challenging scenarios; Parking–Crossing Pedestrian rises from 53.2 to 100.0 DS and from 20% to 100% SR.Merger into Slow Traffic (V2) gains 35.2 DS, and Hazard at Side Lane (Two Ways) gains 33.4 DS.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 Ablation Studies

  • With K=2, DS is 70.00 and SR is 48.00%; with K=4, they reach 81.37 and 54.09%, versus 68.50 DS and 41.36% SR for the single-interpretation baseline.K=2 initializes going-straight and stopping hypotheses; K=4 adds deceleration and lane change, so count and initial maneuver coverage both vary.Source: §5.3 · p. 8, Table 4 · p. 9
  • VLM-based weighting achieves 93.0% maneuver prediction accuracy, versus 77.9% for uniform weighting and 64.4% for random weighting.The lower entropy of VLM-based weights (0.711 vs. 1.386 for uniform weighting) indicates more concentrated weights, not calibrated confidence.Source: Table 5 · p. 9, §5.3 · pp. 9–10
  • Action disagreement and unique maneuver predictions plateau around K≈4, while effective hypotheses grow sublinearly; the four-category vocabulary limits interpretation of this saturation.The study uses K=4 in its main experiments.Source: Figure 5 · p. 9, §5.3 · p. 9
  • At K=4, belief updates take 120.0 ms and the controller takes 45.0 ms; asynchronous refresh reduces average control-step latency from 177.0 ms synchronous to 65.5 ms at s=4.The reported latencies are measured on a single NVIDIA RTX A6000 GPU.Source: Table 6 · p. 10, §5.3 · p. 10

6 Conclusion and Limitations

DriveMind tracks and reweights multiple natural-language hypotheses as new observations arrive, with experiments confirming improved performance and robustness in ambiguous scenarios. Its fixed hypothesis count and update frequency, and its focus on ego-vehicle maneuvers, remain limitations.

6 Conclusion and Limitations

  • DriveMind propagates and reweights multiple hypotheses over time, reducing premature commitment under ambiguity.Its asynchronous fast–slow architecture separates VLM-based belief tracking from diffusion-based trajectory generation.Source: §6 · p. 10
  • Bench2Drive experiments confirm improved performance and robustness in ambiguous scenarios.Ablations support multi-hypothesis belief tracking, and runtime analysis demonstrates the efficiency of asynchronous inference.Source: §6 · p. 10
  • The hypothesis count K and update frequency are fixed, though adapting them to scene complexity or belief entropy could improve efficiency and responsiveness.Source: §6 · p. 10
  • The belief state focuses on ego-vehicle maneuvers rather than predictions of other agents’ future intentions.Including those intentions could enable more explicit reasoning about multi-agent interactions.Source: §6 · p. 10

A Implementation Details

DriveMind couples a high-level VLM reasoner with a low-level diffusion controller, then trains the controller using belief contexts precomputed by the frozen reasoner.

A.1 Model Architecture

  • The high-level VLM generates natural-language hypotheses, predicts maneuver distributions, and scores hypotheses for reweighting.Qwen3VL-2B-Instruct is the backbone, using a ViT-L/14 visual encoder at 448 × 448 resolution.Source: §A.1 · p. 14
  • The DDPM-based low-level controller uses a Transformer backbone to generate short-horizon waypoint trajectories conditioned on belief context and visual history.Source: §A.1 · p. 14

A.2 Training the Low-Level Diffusion Controller

  • Controller training uses visual feature history and belief context to generate a continuous future waypoint trajectory.Source: §A.2 · p. 14
  • After training the reasoner, the method freezes it and caches belief contexts, visual features, and expert waypoints for controller learning.Source: §A.2 · p. 14
  • The controller is trained with a conditional diffusion denoising objective, with noisy inputs formed by adding Gaussian noise to expert trajectories at a diffusion step.Source: §A.2 · p. 14
  • At inference, the reasoner updates beliefs and predicts maneuvers while the controller uses the latest belief context to synthesize trajectories online.Source: §A.2 · p. 14

A.3 Training Details

Training separates VLM reasoner fine-tuning from diffusion-controller learning; inference propagates and reweights hypotheses, manages diversity, and generates belief-conditioned trajectories.

A.3 Training Details

  • The VLM is fine-tuned on VQA and action-prediction samples at a 1:1 sampling ratio, using maneuver-prediction and hypothesis-score losses.The controller is subsequently trained on cached belief contexts and visual features to denoise expert waypoint trajectories.Source: §A.3 · p. 14

A.4 Closed-loop Inference Algorithm

  • Inference initializes K equally weighted hypotheses, then iteratively propagates hypotheses and reweights them against evidence.When effective sample size falls below αK, the algorithm resamples hypotheses and regenerates near-duplicates; the controller forms a belief context and samples a trajectory for receding-horizon execution.Source: §A.4 · p. 15
  • Near-duplicate hypotheses are identified by cosine similarity of normalized VLM text embeddings, and the lower-weighted member is regenerated to satisfy the similarity threshold.Weights are reset after the diversity-maintenance step.Source: §A.4 · p. 15
  • Inference uses the current belief context to generate a trajectory by reverse diffusion, executes it receding-horizon, and records uncertainty and effective sample size in a trace.Source: §A.4 · p. 15

A.5 Diversity Maintenance Visualization

  • When belief concentration makes naive resampling repeatedly select the same high-weight hypothesis, regeneration creates semantically distinct alternatives consistent with the current observation.Source: §A.5 · p. 15

B Formal Definitions of Evaluation Metrics

The evaluation metrics cover safety, efficiency, comfort, and similarity to human driving behavior. Bench2Drive uses route completion and infraction penalties for its closed-loop measures, while NAVSIM uses a multi-component planning score.

B Formal Definitions of Evaluation Metrics

B.1 Bench2Drive (CARLA-based) Metrics

  • When effective sample size falls below αK, DriveMind resamples high-weight hypotheses and regenerates near-duplicates to maintain semantic diversity consistent with the observation.Source: Figure 6 · p. 16
  • Bench2Drive Driving Score is the product of Route Completion and Infraction Penalty.Route completion is measured per route, while the penalty factor decreases for infractions such as collisions and red-light violations.Source: §B.1 · p. 16
  • Bench2Drive Success Rate is the ratio of routes completed without any infractions.Source: §B.1 · p. 16
  • Infraction penalties are multiplied when multiple infractions occur; listed weights are 0.50 for pedestrian or vehicle collision, 0.65 for static-layout collision, and 0.70 for red-light violation.Source: §B.1 · p. 16

B.2 NAVSIM (PDM-based) Metrics

  • NAVSIM uses the official Predictive Driver Model Score to summarize closed-loop planning quality across collision avoidance, drivable-area compliance, time-to-collision safety, comfort, and route progress.Source: §B.2 · p. 16
  • NAVSIM score components include no-collision score, drivable-area compliance, ego progress, time-to-collision safety, and comfort.Source: §B.2 · p. 16

C NAVSIM Results

On NavSim, DriveMind reaches 81.0 PDMS, slightly below the strongest listed planners, while belief tracking provides a small positive gain. The appendix also details how hypotheses are propagated, scored, and mapped to maneuvers.

C NAVSIM Results

  • 81.0 PDMS is below TransFuser (84.0) and UniAD (83.4), while adding belief tracking raises PDMS from 78.2 to 81.0.The gain is small but consistent; DAC rises from 88.9 to 90.6 and EP from 75.0 to 76.1, while safety-related metrics remain unchanged. The authors attribute the smaller NavSim benefit to logged observations that do not change in response to planned trajectories, unlike interactive Bench2Drive.Source: C NAVSIM Results · p. 17, Table 8 · p. 17

D Prompt and Maneuver Space

  • Each particle’s previous hypothesis, previous executed action, and current camera observation are used by the VLM to generate an updated belief.A second VLM pass checks each updated hypothesis against the current observation and a reference action, producing a consistency score rescaled to [0, 1].Source: D Prompt and Maneuver Space · pp. 17–18
  • DriveMind represents actions with four maneuvers: STOP for emergency braking, DECELERATE for cautious slowing, GO_STRAIGHT for unimpeded progress, and LANE_CHANGE for lateral maneuvers.LANE_CHANGE includes overtaking and intersection branch selection.Source: D Prompt and Maneuver Space · p. 18
  • Reference maneuver labels prioritize LANE_CHANGE from lateral navigation commands or steering, then identify STOP and DECELERATE from speed, braking, or should_brake, assigning remaining frames GO_STRAIGHT.Source: D Prompt and Maneuver Space · p. 18

E Additional Qualitative Results and Failure Cases

Qualitative cases show belief weights shifting with evolving evidence, concentrating on cautious actions in uncertain scenes and on forward progress when the lane clears. DriveMind can still fail in dense interactions, sudden fast-object appearances, and prolonged occlusions when updates are delayed or visual evidence is insufficient.

Failure Cases

  • Dense multi-agent interactions, sudden fast-moving objects, and prolonged occlusions remain challenging when belief updates are delayed or visual evidence is insufficient.An occluded agent appearing between slow reasoning updates can leave the controller temporarily relying on an outdated belief context.Source: E Additional Qualitative Results and… · p. 18

Additional Qualitative Results

  • In intersection and foggy-pedestrian cases, beliefs shift toward caution: deceleration and stopping receive weights of 0.55 and 0.45 at the intersection, while foggy-scene deceleration reaches 0.99.Source: Figure 7 · p. 18, Figure 8 · p. 19
  • When a parking-exit lane remains clear, go-straight reaches 0.99; in a dark, wet scene with an emergency vehicle, deceleration becomes the highest-weight action.Source: Figure 9 · p. 20, Figure 10 · p. 20

Related papers

Loading 2610.32069v1…