Source-linked AI summary
FocusDrive: Reasoning with Visual Focus for Autonomous Driving
Zhiyuan Liu, Zehong Ke, Yuanxin Tian, Hao Cheng, Jinhao Li, Yining Xing, Yanbo Jiang, Zhenhua Xu, Wenhao Yu, Jianqiang Wang
TL;DR
Driving rationales can describe a response without explicitly tying the relevant object to its image region. FocusDrive pairs object descriptions with patch references and uses focus annotations for planning; experiments report competitive gaze prediction and planning, including improvement over text-based CoT.
Problem
Driving reasoning expressed in natural language can leave the correspondence between an object description and its specific image region implicit.
Source: §1 · p. 2Method
FocusDrive pairs a decision-relevant object description with a generated image-patch reference and uses human-refined driving-focus annotations to train planning reasoning.
Source: §1 · p. 2Results
FocusDrive demonstrates competitive gaze prediction and end-to-end planning, and improves over text-based CoT on NAVSIM with focus annotations from a small subset of training scenes.
Source: §1 · pp. 2–3Takeaways & Limitations
The results support explicit visual focus as a link between modeling where drivers look and reasoning about how to act on what is seen.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Driving decisions depend on both where to focus and how to act on what is seen. Effective driving reasoning must establish which objects matter, where they are, and how they inform the intended action. Text-based rationales can describe a driving response while leaving its correspondence to specific visual evidence implicit. Visual focus provides a concrete starting point for this connection by identifying what matters in the scene and where it is. We propose FocusDrive, a structured multimodal reasoning framework that organizes end-to-end planning around explicit visual focus. It pairs descriptions of decision-relevant objects with image-patch references, bringing explicit visual focus into the reasoning that generates driving plans and trajectories. We first assess this focus representation through driver gaze prediction, then investigate its role in planning reasoning using driving-focus annotations within existing NAVSIM training scenes. Experiments on W3DA and NAVSIM demonstrate competitive gaze prediction and end-to-end planning performance, with FocusDrive improving over text-based chain-of-thought. These results support visual focus as an effective link between scene understanding and driving action.
1 INTRODUCTION
Driving reasoning needs to connect the objects a model considers with their locations in the image and the response they inform. FocusDrive makes that connection explicit, supports learning human-selected focus through NAVSIM annotations, and evaluates focus in gaze prediction and planning.
1 INTRODUCTION
- Prior work connects driver attention to language by describing attended objects and reasons, motivating FocusDrive’s sequence of identifying and locating a scene element before choosing a response.Earlier work also modeled spatial attention distributions and used predicted attention in learned driving control.Source: §1 · p. 1
- Language-based driving CoT describes what matters and why, but leaves the correspondence between object descriptions and specific image regions implicit.Image-region references and other spatial multimodal representations offer ways to connect language reasoning with image structure.Source: §1 · pp. 1–2
- FocusDrive pairs a decision-relevant object description with a generated image-patch reference, then reasons about a driving response and generates an action plan and future waypoints.A lightweight adapter converts patch scores into dense attention heatmaps for gaze prediction.Source: §1 · p. 2
- Driving-focus annotations in NAVSIM use risk assessment and VLM screening to propose scenes and objects, which an annotator refines into human-selected visual focus for planning supervision.The annotations pair object descriptions with spatial references before subsequent reasoning and action targets.Source: §1 · pp. 2–3
- FocusDrive shows competitive gaze prediction and end-to-end planning, with visual-focus reasoning improving planning over text-based CoT using annotations from a small subset of NAVSIM training scenes.The evaluation covers W3DA gaze prediction and NAVSIM planning.Source: §1 · pp. 2–3
2 RELATED WORK
Related work spans driver-attention prediction and reasoning in driving vision-language-action models. These lines move from spatial attention estimates toward language explanations and from scene interpretation toward action generation.
2 RELATED WORK
- Driver-attention research ranges from convolutional saliency maps to driving-specific models using temporal context, critical situations, and semantic scene structure.Predicted attention has also been explored as an input to learned driving control.Source: §2 · p. 3
- Language-guided gaze methods connect attention maps with semantic content by predicting attended objects and explanations or combining object-level gaze with global prediction.Examples include LLada, FSDAM, and DualGaze-VLM.Source: §2 · p. 3
- Driving VLA research uses language for scene understanding and planning, connects reasoning to action generation, and adapts reasoning to driving situations.Other approaches enrich intermediate representations with future visual goals, world-model supervision, latent reasoning, dynamics tokens, or spatial trajectory prompts.Source: §2 · p. 3
3 METHOD
FocusDrive structures driving reasoning around selected visual focus, linking object descriptions and image locations to gaze prediction and planning. Its methods combine human-reviewed focus annotations, visual-patch references, supervised learning, and reward-based fine-tuning.
3.1 TASK FORMULATION
- Gaze prediction maps a driving image and task prompt to a dense gaze map supervised by recorded driver gaze.The task studies global attention prediction on W3DA, using language and spatial annotations to relate attended objects to image regions.Source: §3.1 · p. 3
3.2 DRIVING-FOCUS ANNOTATIONS
- Driving-focus annotations pair an object description with a human-selected image location in existing NAVSIM training scenes.Risk cues and VLM assessments screen candidate scenes and objects; human annotators choose the final focus and mark its location.Source: §3.2 · p. 4
3.3 REPRESENTING VISUAL FOCUS
- Image-patch references use spatial embeddings for grid cells, with each reference prototype serving as both an output choice and its token input embedding.A projector transforms patch embeddings into prototypes in the VLM hidden space.Source: §3.3 · p. 4
3.4 FROM VISUAL REFERENCES TO GAZE PREDICTION
- For gaze prediction, patch scores retain spatial correspondence, then an adapter reshapes and decodes them into a dense map.The adapter is trained jointly with the VLM using language-modeling loss for attention text and references, plus binary cross-entropy and KL divergence for gaze maps.Source: §3.4 · p. 5
3 METHOD
- FocusDrive uses an object description paired with an image-patch reference to ground subsequent driving plans and future waypoints.The reference supplies spatial grounding and image-conditioned visual information for the action plan.Source: §3 · p. 3, §3.5 · p. 5
3.5 FROM VISUAL FOCUS TO END-TO-END PLANNING
- Planning completions place a visual-focus span before an action plan and waypoint text, connecting selected objects to intended behavior and future motion.Scenes without focus annotations still provide plan and waypoint supervision.Source: §3.5.1 · pp. 5–6
3.5.1 SUPERVISED LEARNING OF FOCUS-BASED REASONING
- Supervised learning first warms up visual-reference tokens on object descriptions and patch references, then trains planning sequences with token-level cross-entropy.Warm-up uses broader object-reference annotations without action-plan or waypoint targets; planning SFT includes focus spans when annotated, followed by plans and waypoint text.Source: §3.5.1 · pp. 5–6
3.5.2 REINFORCEMENT FINE-TUNING WITH GRPO
- GRPO compares sampled completions for the same observation and reinforces those with better driving outcomes, using trajectory PDMS as reward.The loss covers reasoning and waypoint tokens, while KL regularization constrains deviation from the frozen SFT policy.Source: §3.5.2 · p. 6
4 EXPERIMENTS
Experiments evaluate FocusDrive’s gaze prediction and NAVSIM planning, then test which components of explicit visual focus contribute to planning performance. FocusDrive performs competitively across gaze scenarios and outperforms text-based CoT in planning comparisons.
4.1 EXPERIMENTAL SETUP
- The evaluation covers W3DA gaze prediction and NAVSIM Navtest planning, using PDMS and EPDMS for complementary aspects of safety, compliance, comfort, and progress.NAVSIM planning uses Navtrain for training and 12,146 Navtest scenarios for evaluation.Source: §4.1 · pp. 6–7
4.2 MAIN RESULTS
- FocusDrive achieves the strongest KLdiv, CC, and AUC results across Normal, Safety, and Accident gaze scenarios, and the highest SIM in Normal and Accident.Its largest correlation gain over DualGaze-VLM occurs in safety-critical scenes.Source: §4.2 · p. 7, Table 1 · p. 7
- 91.1 PDMS and 91.1 EPDMS: FocusDrive leads the explicit CoT planners compared on NAVSIM Navtest.The prose attributes the PDMS result to strong collision avoidance and road compliance, and reports leading NC and TTC plus tied-highest TLC for EPDMS.Source: §4.2 · p. 7, Table 2 · p. 8, Table 3 · p. 8
4.3 ABLATION STUDIES AND ANALYSIS
- Against text CoT, the complete focus formulation improves SFT by 1.3 PDMS and 1.4 EPDMS, with gains of 0.9 and 1.0 after reinforcement fine-tuning.The formulation adds focus supervision and structures reasoning around an object description, image-patch reference, and driving plan.Source: §4.3 · pp. 7–8, Table 4 · p. 9
- Patch references exceed Text-2D by 0.4 points on both metrics without warm-up and by 0.6 points with warm-up; warm-up adds 0.3 PDMS and 0.2 EPDMS.Both representations use focus supervision on 13.1% of Navtrain scenes and planning supervision on the full training set.Source: §4.3 · p. 8, Table 5 · p. 9
- Image-conditioned patch prototypes improve PDMS and EPDMS by a further 0.3 points each over learned reference-ID embeddings, and the combined configuration scores highest among SFT variants.The prototype supplies a visual representation of the selected region in the current image in addition to location information.Source: §4.3 · pp. 8–9, Table 5 · p. 9
- Figure 3 illustrates two scenes where text CoT receives an NC score of 0 or a TTC score of 0, while FocusDrive avoids the corresponding penalty.The figure pairs scene images and visual-focus distributions with FocusDrive reasoning and predicted trajectories.Source: Figure 3 · p. 9, §4.3 · p. 9
5 CONCLUSION
FocusDrive makes visual focus explicit in driving reasoning by pairing object descriptions with image-patch references. The conclusion connects strong gaze-prediction results with NAVSIM evidence for incorporating explicit focus into driving reasoning.
5 CONCLUSION
- FocusDrive pairs object descriptions with generated patch references to connect decision-relevant objects to their visual representations within planning.The conclusion frames this as connecting where drivers attend with how to act on what is seen.Source: §5 · p. 9
- Strong gaze-prediction results provide a foundation for modeling focus, while NAVSIM experiments demonstrate benefits from incorporating explicit focus into driving reasoning.Source: §5 · p. 9
- The authors report using AI tools for development, literature checking, drafting, and discussion, with final responsibility for the paper’s content retained by the authors.AI-proposed annotation objects were reviewed by a human annotator, who selected locations and revised or removed annotations.Source: AI USE STATEMENT · p. 10
A.1 VISUAL REFERENCE IMPLEMENTATION
The visual-reference implementation projects visual backbone embeddings into the language model’s hidden space and maps annotated locations to patch-grid cells. Reference tokens use image-specific prototypes as outputs and input embeddings.
A.1 VISUAL REFERENCE IMPLEMENTATION
- The projector applies layer normalization followed by a low-rank residual map with bias-free down- and up-projection layers.Source: §A.1 · p. 14
- The projector bottleneck width is r = 64.Source: §A.1 · p. 14
- Annotated locations map to the containing cell on the processed image’s patch grid, ordered left to right and top to bottom.The grid is 34 × 60 for planning and 18 × 32 for gaze prediction; box annotations use their normalized center.Source: §A.1 · p. 14
A.2.1 MODEL AND TRAINING CONFIGURATION
The gaze model learns from annotated attention text, patch references, and gaze maps, then converts patch-level scores into dense gaze predictions. Evaluation uses several complementary gaze metrics.
A.2.1 MODEL AND TRAINING CONFIGURATION
- Training pairs each frame with annotated attention explanations and a reference sampled from grid cells inside the annotated object box.The completion is trained with language-modeling and heatmap losses.Source: §A.2.1 · p. 14
- The gaze adapter normalizes reference scores across patches, reshapes them onto the grid, and uses a five-layer convolutional readout to produce a gaze map.The readout architecture repeats its middle block three times and uses bilinear upsampling.Source: §A.2.1 · p. 15, Figure 4 · p. 15
- Evaluation reports KLdiv, CC, SIM, AUC-J, and AUC-B, with AUC variants differing in how non-fixation negatives are selected.AUC-J uses all non-fixation pixels; AUC-B uses uniformly sampled pixels matched in count to fixation pixels.Source: §A.2.1 · p. 15
A.2.2 PROMPT AND OUTPUT FORMATS
The gaze prompt supplies the front-view image and scene description, then asks for one visual-reference token identifying the primary safety-critical focus. The explanation follows in a plan span.
A.2.2 PROMPT AND OUTPUT FORMATS
- The gaze task uses the same image-and-instruction user message for training and evaluation, while only the assistant continuation differs.The prompt asks the model to locate its primary attention focus and explain it.Source: §A.2.2 · pp. 15–16
- When a safety-critical object needs attention, the required focus span contains exactly one visual-reference token, followed by the attention explanation in a plan span.Training completions place the focus and explanation inside a reasoning block, with the token index sampled from the annotated box.Source: §A.2.2 · p. 16
- At test time, a fixed scaffold ends immediately before the reference position and supplies no reference index or other spatial information.This selects the same pre-reference decoding position supervised during training.Source: §A.2.2 · p. 16
A.2.3 QUALITATIVE RESULTS
The appendix illustrates gaze and planning behavior, describes how driving-focus annotations and output formats are constructed, and reports training and evaluation procedures. The qualitative examples show focus distributions alongside images and trajectories.
A.2.3 QUALITATIVE RESULTS
- FocusDrive aligns dominant gaze responses with ground truth across selected scenes, preserving separated multi-region attention where comparison models broaden or miss regions.Examples also show compact responses in single-target scenes, without ground-truth focus references or attention explanations.Source: §A.2.3 · pp. 16–17
A.3.1 DRIVING-FOCUS ANNOTATIONS
- Vehicles make up 84.43% of selected focus objects, and 89.2% of focus-annotated examples have plans involving deceleration.Annotations cover 13.1% of Navtrain, with planning supervision retained across the full training set; focus labels also include risk scenes without observed braking.Source: §A.3.1 · p. 17
A.3.2 PROMPT AND OUTPUT FORMATS
- Planning prompts request a plan and eight future waypoints, with an optional single patch reference when a safety-critical object needs attention.Text-2D expresses the selected location as ordinary text coordinates, while text CoT uses language reasoning and the same waypoint targets.Source: §A.3.2 · p. 18
A.3.3 TRAINING CONFIGURATION
- Planning variants receive three SFT epochs; a group-relative reinforcement fine-tuning stage samples 16 completions per scene and excludes groups with reward standard deviation below 0.05.Evaluation generates one deterministic trajectory per scene without stochastic sampling, best-of-N selection, or reranking.Source: §A.3.3 · pp. 18–19
A.3.4 QUALITATIVE RESULTS
- Additional planning examples pair front-camera images with focus distributions, BEV trajectories, and generated focus and action plans.Heatmaps display the softmax distribution over patch-reference logits on the patch grid, using per-scene color scales.Source: Figure 6 · p. 20