PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving

Zhengxu Tang, Xiaozhou Zhang, Guofeng Cui, Ziyu Gong, Zi Wang, Yunfei Shi, Ruifeng Deng, Chengzhi Qi, Ke Chen, Sachin Patil, Tianjun Xiao, Langechuan Liu, Pichao Wang

arXiv:2609.01659v1cs.CVcs.CLcs.RO

TL;DR

Autonomous-driving reasoning must bridge scene interpretation and continuous trajectory generation while remaining explicit enough for supervision and auditing. This survey organizes 130 method papers around four intermediate-representation families and synthesizes their trade-offs and open challenges. It concludes that useful driving CoT requires representations grounded in the world, coupled to actions, efficient, and verifiable in closed-loop safety-critical systems.

  • Problem

    Driving models need a reasoning layer that bridges scene interpretation and trajectory generation while remaining explicit enough to supervise and audit.

  • Method

    The survey proposes a representation-centered framework and organizes 130 method papers into four representation families and 13 subtypes.

  • Results

    The synthesis identifies distinct trade-offs: language is inspectable, visual-spatial reasoning is more grounded, latent-dynamic reasoning is closer to control, and externalized reasoning expands knowledge and cooperation.

  • Takeaways & Limitations

    Future driving systems should combine representations that are grounded, causally linked to behavior, efficient, and monitored in closed-loop driving.

  • Takeaways & Limitations

    The corpus may omit very recent work, and assigning each method a primary representation simplifies hybrid systems using multiple representations.

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning powers generative models by eliciting intermediate steps before producing an answer. In autonomous driving, the answer is a continuous action. Thus its reasoning must share the same spatiotemporal structure as the physical world. This survey studies the resulting shift from textual CoT to action-grounded reasoning. Surveying 171 papers, including 130 method papers and 41 benchmarks, datasets, surveys, and analysis papers, we propose a representation-centered taxonomy that treats the form of the intermediate state as the organizing axis. We systematize the 130 methods into four categories: language-based, visual-spatial, latent-dynamic, and externalized reasoning, further divided into 13 subtypes tied to distinct regions of interests. Our synthesis shows that the open frontier of reasoning in driving agents lies in intermediate representations that can be grounded in the real world, coupled to real-time action, and verified under safety-critical systems. Project page: https://github.com/tangzhengxu/awesome-av-cot.

1 Introduction

Autonomous-driving CoT must bridge scene interpretation and trajectory generation while remaining explicit enough to supervise and audit. The survey shifts attention from textual chains to action-grounded intermediate representations that connect reasoning to behavior.

  • Driving models need a reasoning layer between scene interpretation and trajectory generation that is flexible, explicit, supervisable, and auditable.
  • Unlike language tasks, driving CoT produces trajectories and must account for geometry, motion, traffic rules, real-time operation, and closed-loop interaction.Verbose or unstable reasoning traces can compound errors under these constraints.
  • Action-grounded reasoning representations organize driving-relevant information in identifiable intermediate structures before a final decision, including language, visual, latent, retrieved, tool-based, and multi-agent forms.
  • The survey compares representations by how artifacts appear between observation and driving-relevant output, how they connect downstream, and what evidence can evaluate them.
  • The taxonomy organizes driving CoT around the intermediate representation rather than restricting reasoning to visible text.

2 From Textual CoT to Action-Grounded Representations

Autonomous driving changes CoT because its outputs are physical actions rather than textual answers. Intermediate reasoning therefore must preserve spatial and temporal grounding while operating under real-time, closed-loop constraints.

  • Driving outputs include trajectories, control commands, maneuver decisions, and planner-ready interventions rather than textual answers.
  • Driving reasoning must be grounded in metric geometry, lane topology, map priors, object locations, occlusion, motion, interaction intent, and traffic rules.A plausible chain can be unsafe when it mislocates a pedestrian, ignores a blind spot, or misunderstands right-of-way.
  • These pressures motivate intermediate forms beyond text, including visual states, occupancy structures, spatial evidence, and other action-grounded representations.
  • Real-time and closed-loop constraints make slow or unstable reasoning costly because vehicle actions change subsequent observations and errors compound through interaction.

3 Representation-Centered Taxonomy

The survey organizes driving CoT by the form of its decision-relevant intermediate artifact. It covers four representation families and their design trade-offs across interpretability, grounding, action coupling, latency, and verification.

  • The corpus grows rapidly after 2025, while language-based methods remain dominant and visual-spatial, latent-dynamic, and externalized methods form substantial branches.
  • The taxonomy assigns a primary family according to how the intermediate artifact is consumed at the reasoning-to-output interface, not its serialization or provenance.
  • Language-Based Representations: Language-based reasoning offers inspectable scene descriptions, object analyses, risk explanations, plans, reflections, and compressed traces, but descriptions alone may not affect the final trajectory.
  • Language-Based Representations: Procedural language structures reasoning across perception, prediction, planning, and control, making it more action-aware but potentially rigid.
  • Language-Based Representations: Reflective and compressed language methods trade additional error correction or explicit supervision against inference latency and cost.
  • Visual-Spatial Representations: Visual-spatial reasoning uses future frames, occupancy maps, crops, masks, boxes, and trajectory sketches to preserve geometry and temporal structure more directly than text.
  • Visual-Spatial Representations: Visual evidence introduces a selection risk: missing the true risk factor can make downstream reasoning confidently wrong.
  • Latent-Dynamic Representations: Tokenized latent-dynamic reasoning uses identifiable codes or latent positions supervised, decoded, or consumed for future dynamics, actions, or intermediate reasoning, but semantic alignment remains open.

4 Cross-Category Analysis

The four reasoning categories occupy different trade-offs among human interpretability, action coupling, grounding, coverage, and latency. The survey frames representation choice as a system-design decision requiring explicit triggering, interfaces, validation, and fallback behavior.

  • Language-based reasoning is easiest to inspect, whereas latent-dynamic reasoning is closer to control but harder to verify.
  • Visual-spatial reasoning improves grounding but can be costly or incomplete, while externalized reasoning expands knowledge and cooperation but depends on interfaces and external sources.
  • Interpretability versus action coupling: A dual-representation design can pair compact internal control states with faithful external traces for monitoring and explanation.
  • Grounding versus coverage: Robust systems should combine perception-grounded evidence, learned dynamics, and external knowledge to balance grounding, coverage, and distribution-shift risks.
  • Reasoning depth versus latency: Adaptive reasoning should allocate fast control, retrieval, simulation, verification, or communication according to uncertainty, risk, and conflict.
  • From representation choice to system design: System design should specify when reasoning triggers, where artifacts affect decisions, how artifacts are validated, and what fallback applies when they are unreliable or late.

5 Evaluation Landscape

Existing benchmarks evaluate separate fragments of driving reasoning, but evaluation remains fragmented across scene understanding, planning, and closed-loop behavior. The survey therefore treats metrics as source-specific evidence rather than directly comparable measures of reasoning quality.

  • QA and explicit-reasoning datasets test scene description, question answering, and structured reasoning traces.
  • Spatial grounding and robustness benchmarks assess object localization, relational understanding, and stability under perturbations.
  • Planning benchmarks evaluate motion decisions, collision avoidance, route, and progress, while closed-loop benchmarks test reliability when actions alter future observations.
  • Evaluation remains fragmented because QA rarely tests action consequences, planning rarely inspects intermediate reasoning, and closed-loop tests rarely assess faithfulness or evidence reliability.
  • Exact-match, trajectory displacement, collision, closed-loop, and composite metrics each capture limited or benchmark-specific aspects of driving reasoning.

6 Open Challenges

Open challenges center on ensuring that intermediate reasoning is causally coupled to actions, computationally adaptive, and safe under closed-loop interaction. Verification must distinguish grounding, action-faithfulness, and closed-loop safety.

  • Faithful action coupling: Faithful action coupling requires interventions that change reasoning representations without creating implausible states and produce behavior changes at the artifact’s abstraction level.
  • Adaptive reasoning budget: Adaptive reasoning should spend more budget under high uncertainty, risk, or conflict and less when actions are obvious.
  • Closed-loop safety evaluation: Closed-loop evaluation must measure collision and route completion alongside temporal reasoning stability, recovery from incorrect states, and robustness under distribution shift.
  • Closed-loop safety evaluation: Monitoring should detect abnormal reasoning, including missing risks, impossible futures, irrelevant retrievals, failed tool calls, and conflicting agent messages.
  • Grounding, action-faithfulness, and closed-loop safety are distinct: a grounded artifact may be ignored, while a relied-upon erroneous artifact may yield unsafe action.

7 Conclusion

Driving reasoning has expanded beyond text because it is action-oriented, perception-based, and latency-constrained. The survey concludes that trusted driving CoT requires representations that are grounded, behaviorally connected, efficient, and monitored in closed loop.

  • Language-based, visual-spatial, latent-dynamic, and externalized representations support inspection, grounding, action coupling, knowledge, and cooperation in different ways.
  • No single representation suffices; future systems should combine representations that can be executed and trusted in safety-critical driving.

Limitations

The survey’s scope and taxonomy clarify the shift toward action-grounded reasoning, while highlighting uneven coverage of cognitive targets and important boundaries on interpretation.

  • Scope and corpus: The survey may miss very recent work because autonomous-driving CoT and foundation-model research is evolving rapidly.Its corpus reflects publicly available literature at the time of writing.
  • Taxonomy: Assigning each method to one primary reasoning representation simplifies hybrid systems that combine multiple representations.Alternative categorizations may be reasonable when pipelines rely on language, visual grounding, latent planning, retrieval, tools, or communication.
  • Scope and corpus: The survey excludes foundation-model driving works whose intermediate states do not explicitly serve a decision-relevant reasoning role.This boundary focuses analysis on intermediate representations connecting observation and action.
  • Cognitive coverage: Perception-description reasoning can identify scene elements while still failing to produce a safe trajectory.Thus describing a pedestrian, vehicle, or lane boundary is necessary but not sufficient for action-grounded CoT.
  • Cognitive coverage: Spatial reasoning remains a major weakness because language-only reasoning poorly preserves metric structure needed for driving decisions.Relevant structure includes distances, lanes, drivable areas, occlusions, right-of-way geometry, and relative motion.
  • Cognitive coverage: Rules, norms, risk, and social negotiation are the least developed cognitive targets despite their importance for driving reasoning.The survey identifies ambiguous right-of-way, rare traffic rules, causal failure analysis, and uncertainty-aware risk reasoning as areas needing greater emphasis.

A.3 The Action Shift: From Talking to Doing

Driving CoT is shifting from textual explanation toward trajectory-oriented and executable action representations, but outputting an action does not ensure that reasoning causally determines it. The survey therefore emphasizes faithfulness, deployment constraints, and evaluation of reasoning-action coupling.

  • From Talking to Doing: Driving CoT is shifting from textual explanations and high-level commands toward waypoints, trajectories, and planner-compatible action representations.The five output forms span meta-actions, bird’s-eye-view waypoints, continuous trajectories, low-level controls, and closed-loop policies.
  • Weak Coupling Patterns: Parallel generation can make reasoning post-hoc: text and trajectories may share visual features while the trajectory remains independent of the generated explanation.This creates a risk that coherent reasoning rationalizes an action rather than determining it.
  • Weak Coupling Patterns: Training-only reasoning improves efficiency by compressing or removing verbose traces at inference, but it complicates verification of whether causal and semantic benefits are preserved.The deployment motivation is that verbose language reasoning is too slow for real-time driving, while hidden-state internalization is harder to audit.
  • Weak Coupling Patterns: QA or explanation success does not guarantee safe planning, and independently rewarded reasoning and trajectories can contradict one another without consistency constraints.Reward-driven shortcuts may ignore reasoning when trajectories alone are rewarded, while linguistic rewards can be satisfied through format manipulation or hallucinated content.
  • Faithfulness: Trajectory output alone does not establish faithful reasoning, because intermediate traces may be removed, corrupted, or replaced without changing the trajectory.The survey distinguishes producing driving actions from producing reasoning that is action-faithful and causally coupled to closed-loop behavior.
  • Deployment Constraints: Driving CoT must balance reasoning depth with latency, temporal stability, distribution-shift robustness, and fallback safety rather than simply maximizing reasoning length.The survey identifies adaptive triggering, reasoning efficiency, and closed-loop stability as evaluation priorities under realistic deployment constraints.

C.1 Evaluation Focus

The evaluation landscape spans reasoning, grounding, planning, and closed-loop driving, but these benchmarks inspect different parts of the reasoning-to-action pipeline. The survey therefore advocates representation-conditional evaluation that tests artifact quality, action relevance, faithfulness, and operational costs.

  • Reasoning and QA: Reasoning and QA benchmarks assess scene description, structured questions, traffic rules, or step-wise explanations, primarily for language-based representations.They include DriveLM, Reason2Drive, DriveLMM-o1, AD2-Bench, DriveQA, WOMD-Reasoning, DriveCombo, and AgentDrive.
  • Spatial grounding and robustness: Spatial-grounding benchmarks test whether intermediate representations identify correct objects, locations, and relations before action.NuScenes-SpatialQA, VLADBench, DVBench, DRAMA-X, and RoboDriveVLM also probe intent prediction, safety-critical perception, and robustness under corruption.
  • Planning and action: Planning benchmarks connect reasoning representations to trajectory, action, or instruction-conditioned outputs, but often report final planning metrics without evaluating intermediate reasoning.Examples include nuScenes, nuPlan, NAVSIM, CoVLA, DriveAction, doScenes, and AutoDriDM.
  • Closed-loop: Closed-loop benchmarks measure route completion, driving score, infractions, and safety outcomes, yet usually omit intermediate reasoning correctness, stability, and faithfulness.Bench2Drive, Bench2ADVLM, and Bench2Drive-VL are closest to deployment-oriented evaluation.
  • Coverage gaps: Most evaluation matrix cells remain empty because benchmark families separately assess reasoning, grounding, actions, or closed-loop behavior.Latent-dynamic representations are barely tested for grounding or faithfulness, while externalized artifacts lack consistent measures of reliability, delay, and conflict handling.
  • Implications: The survey proposes representation-conditional evaluation using intervention-based faithfulness plus artifact-specific grounding and operational-cost measures.Interventions should preserve artifact validity and assess behavioral change at the artifact’s abstraction level; reporting should include latency, retrieval calls, tool invocations, and communication rounds.

D Reported Results by Benchmark

The reported-result tables aggregate scores from surveyed method papers while warning that metric groups should not be interpreted as a single leaderboard.

  • Reported results: Reported scores are aggregated by benchmark family and metric type, and different metric groups should not be compared as one leaderboard.The tables summarize results reported by surveyed method papers.

nuScenes

Table 7 reports nuScenes results grouped by metric type, covering planning error, collision, generation/occupancy, and closed-loop metrics. Reported values vary across representation and evaluation settings, including open-loop validation and HugSim closed-loop evaluation.

  • Planning error: Planning-error results include L2 Avg 0.22 for Compressed L2 Avg on nuScenes open-loop validation and L2 Avg 0.28 under ST-P3 metrics.Other reported planning values range from L2 Avg 0.23 to 1.8 across methods and protocols.
  • Safety: Collision results include Rollout-based Collision Avg 0.16 and Predictive Collision Avg 0.17 on nuScenes validation.These entries use open-loop planning or ST-P3 metric settings.
  • Generation / occupancy: Predictive occupancy-generation results report mIoU_f 36.3 and VPQ_f 25.1 on nuScenes with 2s generation and occupancy evaluation.The reported setting uses inflated GMO and flow forecasting.
  • Generation / occupancy: Future-frame generation results report Predictive FID 6.9 for Vista and Predictive FID 15.8 for Drive-WM on nuScenes validation.Vista is evaluated for prediction fidelity, while Drive-WM uses multiview video generation.
  • Closed-loop evaluation: VERDI reports 359 ms latency alongside COM 0.954, DAC 0.944, NC 0.659, TTC 0.469, and Rc 0.354 in HugSim closed-loop evaluation.These are reported as overall metrics for the compressed reasoning setting.

NAVSIM

NAVSIM results report aggregate and component metrics across navtest, open-loop, closed-loop, and pseudo-closed-loop settings. The values are organized by metric type and configuration rather than a single comparable ranking.

  • EPDMS: NAVSIM v2 EPDMS results range from 43.7 to 50.9 in Navhard or leaderboard-style evaluations, while standard navtest results include EPDMS values from 85.2 to 89.9.The reported settings include pseudo-closed-loop, two-stage test, private test, combined ensemble, and alternative configurations.
Loading 2609.01659v1…