PaperlayerView exact revision on arXiv

Source-linked AI summary

AnchorReasoning: A Visual Grounding and Causal Reasoning Dataset in Long-Tail Autonomous Driving Scenarios

Zhipeng Bao, Wenjie Zhao, Tianle Zhu, Haohua Que, Chence Yang, Geng Yuan, Qianwen Li

arXiv:2609.28366v1cs.CVcs.AI

TL;DR

Driving datasets provide limited supervision for connecting decision-critical visual evidence with reasoning and planning. AnchorReasoning combines visually grounded reasoning chains with curriculum training and an object-size-aware grounding metric. Across evaluated backbones, VG-CoT improves trajectory and reasoning outcomes while using fewer reasoning tokens and less inference time.

  • Problem

    Existing driving datasets provide limited supervision for connecting decision-critical visual evidence with reasoning and planning.

    Source: §1 · p. 1

  • Method

    AnchorReasoning organizes visual grounding, element implications, action rationale, and action and trajectory planning into VG-CoT, with curriculum SFT and an object-size-aware grounding metric.

    Source: §1 · pp. 2–3

  • Results

    Across models, VG-CoT reduces 5-s ADE by 7.84 and FDE by 11.86, improves RFS Frame by 1.66 and RFS Cluster by 1.70, and reduces inference latency by 0.32 s/frame.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The results demonstrate the value of visually grounded, decision-focused supervision for VLM reasoning and planning in long-tail autonomous driving.

    Source: §7 · p. 10

    Annotations cover only the front-left, front, and front-right camera views and lack explicit depth supervision.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) offer a promising approach to long-tail autonomous driving, but existing driving datasets provide limited supervision for connecting decision-critical visual evidence with reasoning and planning. We introduce AnchorReasoning, a visually grounded reasoning dataset built on WOD-E2E, containing 416,119 annotated frames and 395,379 decision-critical elements across four major categories and 19 fine-grained types. Each frame is organized as a visually grounded chain-of-thought (VG-CoT) that links decision-critical element identification and localization, element attributes and implications, driving-action rationale, and action and trajectory planning. We further develop a curriculum supervised fine-tuning strategy that progressively learns these hierarchical capabilities, together with an object-size-aware grounding metric for evaluating localization quality. Experiments across eight general-purpose, embodied-AI, and AV-specific backbones show that VG-CoT supervision improves grounded reasoning and trajectory prediction. Across models, 5-s ADE and FDE decrease by 7.84 and 11.86, while RFS Frame and Cluster improve by 1.66 and 1.70. These gains are achieved with 18.5 fewer reasoning tokens and 0.32 s/frame lower inference latency on average, demonstrating the value of visually grounded, decision-focused supervision for VLM reasoning and planning in long-tail autonomous driving.

1 INTRODUCTION

Existing driving annotations offer limited supervision for connecting decision-critical visual evidence to driving decisions. AnchorReasoning addresses this with visually grounded reasoning chains, curriculum training, and a grounding metric for long-tail driving.

1 INTRODUCTION

  • Existing scene descriptions give limited fine-grained, object-centric supervision and little explicit guidance for translating observations into driving decisions; most CoT annotations also lack decision-critical visual grounding.The introduction notes that this can leave reasoning distracted or weakly supported by scene observations.Source: §1 · pp. 1–2
  • AnchorReasoning organizes each frame into a VG-CoT linking decision-critical element grounding and implications with action rationale and action and trajectory planning.The dataset provides decision-relevant visual grounding, fine-grained element attributes and semantics, and structured reasoning annotations.Source: §1 · p. 2
  • The curriculum SFT follows VG-CoT’s hierarchy to progressively train visual grounding, element implications, action rationale, behavior planning, and trajectory generation.The strategy is designed to leverage the dataset’s hierarchical supervision.Source: §1 · p. 2
  • The comparison distinguishes graph-organized QA from grounded QA and marks nuReasoning as partially grounded because its spatial grounding is not propagated into decision reasoning or trajectory generation.Graph QA links perception, prediction, and planning questions; grounded QA associates question-answer pairs with visual regions or objects.Source: §1 · p. 2
  • The object-size-aware grounding metric uses scale-adaptive point-to-mask distances to evaluate localization quality across elements of different sizes.The paper presents it as a consistent measure of localization quality.Source: §1 · p. 3

2 RELATED WORK

Related driving datasets progress from language descriptions to targeted question-answering and structured reasoning. These approaches provide different forms of scene, object, and reasoning supervision.

Language Description and Explanation

  • Language-supervised datasets describe driving behavior and explain action decisions; BDD-X provides action descriptions and justifications, while DRAMA localizes risk-relevant objects and gives hazard explanations.Source: §2 · p. 3

Question Answering

  • VQA datasets target scene questions, covering object properties and spatial relations in NuScenes-QA, broader tasks in LingoQA, ranked decision-relevant participants in Rank2Tell, and dependency-linked perception, prediction, and planning in DriveLM.Source: §2 · p. 3

Chain-of-Thought Reasoning

  • Reason2Drive and DriveLMM-o1 provide reasoning chains across driving tasks, while nuReasoning adds spatial, decision, and counterfactual reasoning for real-world long-tail scenarios.Source: §2 · p. 3

3 ANCHORREASONING DATASET

AnchorReasoning builds a visually grounded reasoning dataset on WOD-E2E, linking decision-critical evidence to driving reasoning and planning. Its schema, human-in-the-loop annotation pipeline, and dataset statistics define the scope and structure of that supervision.

3.1 OVERVIEW

  • 416,119 frames and 395,379 decision-critical elements across 2,492 clips anchor a WOD-E2E dataset for long-tail VLM training and evaluation.Annotations connect grounding, scene understanding, causal reasoning, action, and trajectory prediction.Source: §3.1 · pp. 3–4

3.2 VG-COT ANNOTATION SCHEMA

  • VG-CoT combines scene context and traffic events with grounded elements, their attributes and implications, then integrates those cues into action and trajectory planning.Elements span vehicles, vulnerable road users, obstacles, and traffic-control elements; attributes include state, intention, and impact rank.Source: §3.2 · pp. 4–5

3.3 HUMAN-IN-THE-LOOP ANNOTATION AND QUALITY CONTROL

  • A hybrid pipeline combines human judgments, VLM assistance, segmentation, and rules, with cross-annotation review and consistency checks for quality control.Annotators retain at most five decision-critical elements per frame; richer interactions use GPT-5.2 for implications and planning, while simpler scenes use locally deployed Qwen3-VL.Source: §3.3 · pp. 5–6

3.4 DATASET STATISTICS

  • The dataset includes approximately 395K elements across 4 categories and 19 types, with 34-token mean implications, 32-token mean rationales, and 4 longitudinal plus 10 lateral actions.Grounded locations span three-camera images and vary spatially across element classes.Source: §3.4 · p. 6

4 CURRICULUM-BASED SUPERVISED FINE-TUNING

The curriculum progressively trains perception and grounding before full reasoning and planning. Stage-dependent supervision and a joint chain-and-trajectory objective structure learning across those capabilities.

Stage I: Perception and Grounding

  • Stage I progresses from scene context, traffic events, and element presence to element identification, grounding, impact ranking, and attribute understanding.Locked fields remain in the target structure but are masked from the loss and contribute no gradient.Source: §4 · p. 6

Stage II: Reasoning and Planning

  • Stage II supervises the complete VG-CoT chain, freezes the visual encoder, and adds trajectory-only supervision for a subset of samples.Sampling also increases for frames with longitudinal action transitions and rare element states.Source: §4 · p. 6

Training Objective

  • Curriculum-aware token weights gate fields by stage and upweight rare attribute values and trajectory waypoints that deviate more from constant-velocity motion.The weighting combines a stage-dependent supervision gate, field base weights, and target informativeness adjustments.Source: §4 · p. 6
  • The objective combines VG-CoT chain loss and trajectory loss, with λ set to 1 and independent normalization of the two terms.Stage I optimizes only the chain loss, whereas Stage II jointly optimizes both objectives.Source: §4 · p. 7

5 EXPERIMENTS

The experiments compare VG-CoT supervision with domain-adapted baselines across eight general-purpose, embodied-AI, and AV-specific models, evaluating scene and element understanding, reasoning, trajectory prediction, and efficiency. VG-CoT improves reasoning and trajectory metrics while reducing average output length and latency.

5.1 MODELS

  • Eight evaluated backbones span general-purpose VLMs, embodied-AI models, and AV-specific models, including driving-specialized models capable of joint reasoning and trajectory prediction.The set includes Qwen, Cosmos-Reason, Alpamayo, AutoVLA, and Impromptu-VLA models.Source: §5.1 · p. 7

5.2 EVALUATION METRICS

  • Evaluation covers scene and element semantics, trajectory prediction, and inference cost, with scale-adaptive point-to-mask matching for grounding and GPT-5.5 assessment of implications and action rationales.Grounding measures include element recall, rank-1 recall, element type accuracy, and point-in-mask rate; Table 2 covers scene-level semantics and decision-critical elements.Source: §5.2 · pp. 7–8, Table 2 · p. 8

5.3 RESULTS

  • Across model families, element type and traffic-control-content recall are consistently higher than element state and intention recall.The paper attributes this pattern to dynamic behavior understanding being more difficult than recognizing element identity and traffic-control semantics from single-frame input.Source: §5.3 · p. 9, Table 3 · p. 9
  • VG-CoT reduces average output length by 18.5 tokens and latency by 0.32 s/frame while improving trajectory accuracy; its non-grounded ablation performs worse across trajectory metrics for three backbones.The reported trajectory pattern is lower ADE/FDE and higher RFS metrics with VG-CoT supervision.Source: Table 3 · p. 9, §5.3 · p. 9, Table 4 · p. 9
  • VG-CoT improves Element Implication and Driving Action Rationale across all model families, with Element Implication scoring higher than multi-factor rationale.The authors report that interpreting an individual element's effect is easier than consolidating multiple factors into a final decision.Source: §5.3 · p. 9, Table 3 · p. 9

6 LIMITATIONS AND FUTURE WORK

Current annotations cover only three forward-facing camera views and lack explicit depth supervision; future work will extend view coverage and incorporate depth estimates.

6 LIMITATIONS AND FUTURE WORK

  • Annotations cover front-left, front, and front-right views, leaving side and rear views unannotated, and do not provide explicit depth supervision.Future work proposes full-surround perception and depth estimates to strengthen spatial understanding and reasoning.Source: §6 · p. 9

7 CONCLUSION

AnchorReasoning provides visually grounded supervision connecting decision-critical elements with reasoning and action and trajectory planning, and introduces an object-size-aware grounding metric. Across eight backbones, VG-CoT supervision improves grounded reasoning and trajectory prediction, with explicit visual grounding benefiting trajectory prediction.

7 CONCLUSION

  • AnchorReasoning supervises decision-critical elements, their attributes and implications, driving-action rationale, and action and trajectory planning through VG-CoT.The paper also introduces curriculum supervised fine-tuning and an object-size-aware grounding metric.Source: §7 · p. 9
  • Across models, VG-CoT reduces 5-s ADE by 7.84 and FDE by 11.86 on average, improves RFS Frame by 1.66 and RFS Cluster by 1.70, and reduces latency by 0.32 s/frame with fewer reasoning tokens.The conclusion states that visual-grounding ablations confirm explicit grounding benefits trajectory prediction.Source: §7 · pp. 9–10
  • Generative AI tools assisted annotation, data cleaning and reformatting, code work, and language editing; authors reviewed and verified outputs and take responsibility for the final work.Only the introduced annotations are redistributed; users must obtain the original Waymo data separately under its license.Source: AI USE STATEMENT · p. 10, ETHICS STATEMENT · p. 10

A.1 DATASET COMPOSITION AND WOD-E2E SPLITS

AnchorReasoning annotates WOD-E2E frames and decision-critical elements, with annotations emphasizing high-impact elements and joint lateral-longitudinal action structure. The appendix also describes annotation examples and automatic scene-context annotation tools.

A.1 DATASET COMPOSITION AND WOD-E2E SPLITS

  • AnchorReasoning adds semantic and reasoning annotations to decoded WOD-E2E frames without changing sensor data or dataset splits; it contains 416,119 annotated frames and 395,379 decision-critical elements.The training split includes 415,663 frames from 2,037 segments; validation annotations cover 456 frames with rater-feedback trajectories, and the test split is not annotated.Source: §A.1 · p. 12, Table 5 · p. 12

A.2 IMPACT-RANK DISTRIBUTION

  • The impact-rank distribution is concentrated at the highest ranks: 67.9% of elements are rank 1 and 25.3% rank 2, with 93.2% in the first two ranks.The annotation principle retains elements that materially influence the current driving decision rather than labeling every visible object.Source: §A.2 · p. 12, Figure 8 · p. 13

A.3 ACTION-PLAN DISTRIBUTION

  • Lateral and longitudinal actions are not independent: the same lateral maneuver can pair with different longitudinal responses depending on constraints and interaction state.Routine lane-keeping dominates the distribution, while more complex lateral maneuvers form a long tail.Source: §A.3 · p. 12, Figure 9 · p. 13

A.4 ANNOTATION EXAMPLES

  • Representative annotation examples progressively connect scene context and grounded element attributes to implications, frame-level reasoning, and the final action plan.Figures 10–12 show examples of complete visual and textual annotations.Source: §A.4 · p. 12, Figure 10 · p. 14, Figure 11 · p. 15

B AUTOMATIC ANNOTATION TOOLS

  • Automatic annotation tools are checked against a manually annotated subset before large-scale deployment, measuring human–automatic agreement rather than absolute accuracy.The paper notes that human labels may themselves be ambiguous.Source: B AUTOMATIC ANNOTATION TOOLS · p. 17

B.1 SCENE CONTEXT

  • Qwen3-VL-8B-Instruct predicts five scene-context fields from images and a predefined taxonomy, then video-level rules stabilize context labels across frames.Reliability is evaluated on 7,450 manually annotated frames; the supplied passages specify the evaluation set but no agreement values.Source: §B.1 · p. 17, Table 6 · p. 17

B.2 TRAFFIC-LIGHT STATE

Sections B.2–B.6 describe automated traffic-light, object-location, and vehicle-type annotation, followed by reasoning-chain generation and action–trajectory consistency checks.

B.2 TRAFFIC-LIGHT STATE

  • 97.81% overall agreement was achieved for traffic-light states across three manually annotated partitions containing 11,318 matched instances.Yellow recall was 0.42 on p19 and 0.59 on p20, so red and green predictions are used automatically while yellow labels receive human annotation.Source: §B.2 · pp. 17–18
  • Relaxing the yellow threshold improves recall but produces many false yellow predictions from red lights, making the adopted configuration the more reliable overall trade-off.Three threshold configurations were evaluated on p19.Source: §B.2 · p. 18, Table 9 · p. 18

B.3 OBJECT LOCATION

  • 98.42% agreement was achieved between automatic directional-prefix labels and human annotations across 4,439 aligned objects.Five manually defined spatial regions are reused across frames; objects are assigned by greatest bounding-box overlap, with a bottom-center cue when the top two overlap scores differ by no more than 0.05.Source: §B.3 · p. 18

B.4 VEHICLE TYPE

  • Vehicle-type classification agreed with human labels on 3,681 of 3,720 vehicles, or 98.95%.Disagreements cluster around visually similar categories, especially cars, pickup trucks, and large van-like vehicles.Source: §B.4 · pp. 18–19

B.5 REASONING CHAIN GENERATION

  • Reasoning-chain generation first interprets each decision-critical element independently, then integrates its implications and impact rank into a frame-level rationale and action plan.The plan specifies short-horizon longitudinal and lateral actions from the AnchorReasoning action vocabulary; scenes are routed by complexity and generated annotations are inspected and corrected.Source: §B.5 · p. 19

B.6 ACTION–TRAJECTORY CONSISTENCY CHECK.

  • Action–trajectory consistency checks map annotated plans and future trajectories into shared longitudinal and lateral categories, then flag contradictions for manual review.The longitudinal categories are accelerate, decelerate, cruise, and stop; lateral categories are left turn, right turn, or straight when specified.Source: §B.6 · p. 19

C TRAINING AND EVALUATION DETAILS

Training and evaluation use NVIDIA B200 GPUs, distributed training, memory and attention optimizations, and full-parameter fine-tuning.

C.1 HARDWARE AND IMPLEMENTATION DETAILS

  • Training uses two NVIDIA B200 GPUs with distributed data parallelism and DeepSpeed ZeRO Stage 2.All training and evaluation experiments use NVIDIA B200 GPUs.Source: §C.1 · p. 19
  • bfloat16 precision, FlashAttention-2, and gradient checkpointing are used for training.Gradient checkpointing is applied to reduce memory consumption.Source: §C.1 · p. 20
  • All trainable components are fine-tuned with full parameters, without LoRA or other parameter-efficient adaptation methods.Source: §C.1 · p. 20

C.2 TRAINING CONFIGURATION

Training uses a shared multimodal input representation and a two-stage SFT curriculum that progresses from scene and element understanding to the complete VG-CoT output.

C.2 TRAINING CONFIGURATION

  • Each sample uses a three-camera panorama, textual ego-motion history of 16 waypoints over the preceding 4 s, and high-level navigation intent.No temporal image sequence, LiDAR, HD map, or auxiliary prediction head is used.Source: §C.2 · p. 20
  • Domain-adapted baselines train on original WOD-E2E data without AnchorReasoning annotations, retaining each model’s original objective and output format.Their visual input is replaced with a single-frame panorama, and targets retain native reasoning, action, or trajectory representations rather than VG-CoT.Source: §C.2 · p. 20
  • Stage I SFT teaches scene semantics and decision-critical element understanding, while Stage II extends the best Stage-I checkpoint to the complete VG-CoT chain.Stage II covers ego-state understanding, implications, rationale, action planning, motion, and future trajectory; Table 10 summarizes optimization settings.Source: §C.2 · p. 20, Table 10 · p. 20

C.3 EVALUATION METRICS

The evaluation defines metrics for semantic understanding, grounding, reasoning, and trajectory prediction, and illustrates both failure modes and successful multi-element planning cases.

C.3 EVALUATION METRICS

  • Semantic metrics cover scene-context accuracy, traffic-event F1, element presence and attribute recall, sign-content recall, and red/green confusion.Context Acc. is macro-averaged across five scene-context dimensions; lower R/G Conf. is better.Source: §C.3 · p. 20
  • Grounding uses point-to-mask distance and scale-adaptive one-to-one matching, processing ground-truth elements by impact rank and counting unmatched items as misses or false positives.Point-in-Mask Rate requires predicted points to fall on segmented objects, not merely within their bounding boxes.Source: §C.3 · p. 21
  • GPT-5.5 judges element implications and driving rationales for correct element interpretation, influence on ego driving, and integration of scene factors.The complete judge prompts are shown in Figures 19 and 20.Source: §C.3 · pp. 21–22
  • Trajectory evaluation uses a common 5 s, 4 Hz grid, reporting ADE at 1, 3, and 5 s, FDE at 5 s, and Waymo RFS Frame and Cluster scores.RFS ranges from 0 to 10, with higher values indicating better agreement with human-rated driving trajectories.Source: §C.3 · p. 22

D.1 FAILURE CASES

  • Single-frame input makes motion-dependent state and intention difficult to infer, while semantic recognition can fail to align with accurate visual localization.The reported failures also include omitted critical elements, potentially related to missing geometric cues such as distance and ego-vehicle dimensions.Source: §D.1 · p. 22

D.2 SUCCESSFUL CASES

  • A successful multi-object example grounds a backing vehicle, construction vehicle, and pedestrian, then integrates their effects into a keep-lane, decelerate plan with a trajectory close to ground truth.Another example connects a parked bus’s lateral-space constraint and an oncoming vehicle’s effect on moving left to a coherent rationale and action plan.Source: §D.2 · p. 22

E PROMPTS

Loading 2609.28366v1…