PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic

Songyan Zhang, Wenhui Huang, Zhan Chen, Chua Jiahao Collister, Qihang Huang, Chen Lv

arXiv:2512.01830v2cs.CV

TL;DR

Data-driven and SFT-based driving approaches face generalization limits, while open-ended scene reasoning is difficult to reward and current RFT work concentrates on verifiable tasks. OpenREAD combines CoT-based cold-start training with end-to-end GRPO fine-tuning and an LLM critic; experiments report improvements in reasoning and planning, with state-of-the-art benchmark performance.

  • Problem

    SFT-based driving models have limited reasoning generalization, and open-ended scene understanding is difficult to reward compared with verifiable downstream tasks.

    Source: §1 · pp. 1–2

  • Method

    OpenREAD uses CoT-based SFT followed by end-to-end GRPO fine-tuning across knowledge and planning data, with Qwen3-8B evaluating open-ended responses.

    Source: §1 · p. 2, §3.3.4 · p. 5

  • Results

    Joint end-to-end RFT substantially improves upstream reasoning and downstream planning, and OpenREAD achieves state-of-the-art performance on reasoning and planning benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The LLM-as-critic mechanism makes GRPO applicable to open-ended knowledge learning alongside verifiable driving behaviors.

    Source: §1 · p. 2

    The scale of RFT training datasets remains limited because multiple generations from OpenREAD and Qwen3-LLM incur computational costs.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recently, two-stage fine-tuning strategies, e.g., acquiring essential driving knowledge through supervised fine-tuning (SFT) and further enhancing decision-making and planning via reinforcement fine-tuning (RFT), have shown strong potential in advancing the knowledge-driven autonomous driving (AD) paradigm. However, the learning nature of SFT still limits the generalization of reasoning, thereby constraining the full potential of driving performance. Meanwhile, current RFT approaches are primarily applied to downstream tasks, since scene understanding is an open-ended problem where corresponding rewards are difficult to quantify. To address these limitations, we propose OpenREAD, an OPEN-ended REasoning reinforced vision-language model (VLM)-based autonomous driving (AD) framework that enables end-to-end RFT across the full spectrum from high-level reasoning to low-level trajectory planning. Specifically, we begin by constructing large-scale Chain-of-Thought (CoT) annotations on open-source driving-related knowledge datasets, and employ the powerful Qwen3 large language model (LLM) as the critic in RFT to quantify reasoning quality for open-ended questions during reward modeling. Extensive experiments confirm that joint end-to-end RFT yields substantial improvements in both upstream and downstream tasks, enabling OpenREAD to achieve state-of-the-art performance on reasoning and planning benchmarks.

1. Introduction

The introduction identifies limits in data-driven and SFT-based driving models, including generalization and fixed-answer constraints. OpenREAD applies end-to-end reinforcement fine-tuning to open-ended reasoning and planning, and the reported experiments show gains in both upstream reasoning and downstream planning.

1. Introduction

  • Purely data-driven driving approaches face overfitting, limited long-tail and cross-domain generalization, and interpretability challenges.The introduction illustrates the risk with a stop sign shown on an advertisement screen misleading a vehicle.Source: §1 · p. 1
  • Prior VLM approaches often rely on fixed SFT answer patterns, while existing RFT work focuses mainly on verifiable tasks such as meta-decision making or trajectory planning.Open-ended driving questions can have multiple contextually reasonable answers.Source: §1 · p. 2
  • OpenREAD uses Qwen3-8B as a critic for open-ended responses and combines semantic-similarity and average-displacement-error rewards for knowledge and planning.It begins with SFT on reasoning annotations, then performs end-to-end RFT with GRPO.Source: §1 · p. 2
  • Joint end-to-end RFT substantially improves upstream reasoning and downstream planning, with OpenREAD reaching state-of-the-art performance on reasoning and planning benchmarks.The framework applies RFT across knowledge learning and trajectory planning.Source: §1 · p. 2

2. Related Works

Related work spans unified end-to-end driving models, VLMs that bring language and semantic reasoning into driving, and reinforcement fine-tuning for multimodal reasoning. The cited studies also identify open-ended reasoning and motion prediction as connected challenges.

2.1. End-to-End Autonomous Driving

  • End-to-end systems unify perception, prediction, and planning, but the cited work notes remaining difficulty capturing high-level semantics and complex interactions in diverse or unseen scenarios.Examples include DriveTransformer [22], ParaDrive, and DiffusionDrive [33].Source: §2.1 · p. 2

2.2. LLMs and VLMs for Autonomous Driving

  • VLM-based driving research adds semantic reasoning and multimodal inputs, while newer systems connect reasoning to motion prediction and planning [58] [71].Examples include SimLingo [44], ORION [11], AutoVLA [72], and ReCogDrive [30]; OmniDrive supplies video QA and counterfactual reasoning annotations.Source: §2.2 · pp. 2–3

2.3. Reinforcement Fine-Tuning for LLMs and VLMs

  • Multimodal RFT studies use reward-driven optimization to refine reasoning, and the related-work discussion describes it as promising for joint scene reasoning and downstream driving decisions.The cited methods include Vision Think [65], Visual RFT [37], Vision-R1 [18], and Reason-RFT [50].Source: §2.3 · p. 3

3. Methodolgy

OpenREAD uses Qwen3-VL-8B to process task-specific prompts with image sequences and generate answers with an explicit thinking process. Its training proceeds from CoT-based SFT cold start to GRPO-based RFT.

3.1. Overview of OpenREAD

  • OpenREAD is built on Qwen3-VL-8B and generates answers with an explicit thinking process from images and task-specific prompts.Knowledge tasks use five sequential front-view images; planning uses four sequential images plus a current-timestep LiDAR bird’s-eye-view image.Source: §3.1 · p. 3
  • Training has two stages: cold-start SFT on CoT-annotated data, followed by RFT with GRPO.The stages are presented as the framework’s fine-tuning process.Source: §3.1 · p. 3
  • The image and prompt inputs support both open-ended driving-knowledge learning and trajectory planning.The knowledge input is a sequence of front-view images, while planning also includes a LiDAR bird’s-eye-view image.Source: §3.1 · p. 3

3.2. Cold Start

Cold start builds driving-related reasoning annotations from LingoQA and reformatted OmniDrive data, then uses CoT supervision to train answer generation. The annotations cover scenario understanding, action analysis, trajectory planning, and counterfactual trajectory analysis.

3.2.1. CoT Data Generation

  • The cold-start data includes 7K manually post-processed LingoQA CoT annotations generated with GPT-4.1 from images, questions, and reference answers.The process begins with 100 manually annotated examples used to derive a target annotation prompt.Source: §3.2.1 · pp. 3–4
  • Reformatted OmniDrive annotations provide 24K trajectory-planning and 7K counterfactual trajectory-analysis CoT samples.Planning rationales describe a high-level meta action; answers provide six future trajectory points over three seconds at 2 Hz.Source: §3.2.1 · p. 4
  • For counterfactual analysis, the model judges whether a supplied pseudo-trajectory is reasonable, a task described as supporting planning understanding and spatial perception.Source: §3.2.1 · p. 4

3.2.2. SFT with CoT Data

  • SFT predicts target answer tokens autoregressively from visual tokens and textual-instruction embeddings, then applies cross-entropy against the ground truth.The visual tokens are produced from image inputs by the vision encoder and projector.Source: §3.2.2 · p. 4

3.3. RFT with LLM-as-Critic

OpenREAD uses GRPO to optimize group-relative response rewards, combining an LLM-based evaluator and embedding similarity for open-ended knowledge with trajectory rewards in joint training.

3.3.1. Preliminary on GRPO Algorithm

  • GRPO samples groups of candidate responses, computes normalized group-relative advantages, and optimizes the current policy with a clipped objective.The advantage compares each sample’s reward with the group’s mean and standard deviation.Source: §3.3.1 · p. 4

3.3.2. Open-ended Knowledge Rewards

  • Qwen3 scores predicted/reference semantic matches on a 0-to-1 scale, thresholded at 0.5, and a 0.9/0.1 weighted combination adds embedding similarity.The embedding reward uses cosine similarity between predicted and reference responses.Source: §3.3.2 · pp. 4–5

3.3.3. Trajectory Planning Reward

  • Trajectory reward uses average L2 errors for each of three seconds across six future points sampled at 2 Hz, with temperatures 0.7, 0.7, and 0.3.The passage identifies the three-second horizon and per-second average distances as the basis for the reward.Source: §3.3.3 · p. 5

3.3.4. Joint Training

  • RFT jointly trains trajectory planning and driving knowledge, combining their rewards with a format reward that constrains reasoning and final-answer tags.The prescribed tags are <think>...</think> and <answer>...</answer>.Source: §3.3.4 · p. 5

4. Experiment

The experiments compare OpenREAD’s reasoning and planning against baselines and test how joint driving-knowledge learning, reward design, and data scale affect performance.

4.1. Implementation Details

  • The implementation uses Qwen3-VLM-8B, CoT-annotated LingoQA and OmniDrive data for cold start, then LoRA and AdamW for both SFT and RFT.Cold start uses 7K LingoQA, 9K counterfactual trajectory analysis, and 24K trajectory-planning examples; RFT samples 6K planning and 5K mixed reasoning examples.Source: §4.1 · pp. 5–6

4.2. Evaluation Metric

  • Driving-knowledge evaluation uses Lingo-Judge for LingoQA and GPT-score for counterfactual analysis, while NuScenes planning uses ST-P3 L2 distance and collision rate across 1s, 2s, and 3s.CIDEr, BLEU, and Meteor measure similarity between generated explanations and reference descriptions.Source: §4.2 · p. 6

4.3. RFT for Knowledge-Enhanced Planning

  • Planning-only RFT yields limited gains and worsens average collision rates at 2s and 3s; the authors report that this highlights the need to extend driving-related knowledge.The planning-only RFT stage uses 6K randomly sampled examples after SFT and cold start.Source: §4.3 · p. 6
  • With counterfactual trajectory data, RFT reduces L2 distance from 0.45m to 0.43m and collision rate from 0.17% to 0.13% versus SFT.The study uses planning data alongside counterfactual samples in cold start and RFT.Source: §4.3 · p. 6, Table 1 · p. 6

4.4. State-of-the-Art Benchmark

  • OpenREAD is competitive with VLMs trained on comparable data, has the lowest collision rate in NuScenes’ final two seconds, and achieves the highest Lingo-Judge accuracy among specialist models.EMMA achieves the lowest L2 distance error, which the authors attribute to its extensive internal training data.Source: §4.4 · p. 7, Table 2 · p. 7, Table 3 · p. 8

4.5.1. Reward for Driving Knowledge Learning

  • A 7K CoT cold start raises Lingo-Judge accuracy from 51.4% to 65.0%; Qwen3-LLM judgment and weighted embedding similarity then balance accuracy and response style.The reward ablation reports that embedding similarity alone can miss contradictions, while the combined rewards improve accuracy and preserve concise expression.Source: §4.5.1 · pp. 7–8, Table 4 · p. 8

4.5.2. Data Scale for Knowledge Learning

  • After cold start, GRPO raises Lingo-Judge accuracy from 65.0% to 68.4% with 10K additional data, while SFT uses 50K data to move from 65.6% to 68.0%.The authors report consistent gains as training data increases across SFT, cold-start, and RFT stages.Source: §4.5.2 · p. 8, Table 5 · p. 8

5. Conclusion and Limitations

OpenREAD jointly applies RFT to knowledge learning and trajectory planning, using Qwen3-LLM as a critic for open-ended knowledge; the authors report state-of-the-art performance in trajectory planning and knowledge evaluation. They also note that computational costs keep the RFT training dataset scale limited.

5. Conclusion and Limitations

  • OpenREAD achieves state-of-the-art performance in both trajectory planning and knowledge evaluation.The authors attribute this to reinforcement of open-ended knowledge through GRPO with Qwen3-LLM as critic.Source: §5 · p. 8
  • OpenREAD uses RFT jointly for knowledge learning and trajectory planning, with Qwen3-LLM as the critic in the RL loop.The authors state that this enables effective reinforcement of open-ended knowledge through GRPO.Source: §5 · p. 8
  • Generation costs for OpenREAD and Qwen3-LLM keep the datasets used for RFT training limited.The authors plan to scale up the knowledge data in future research.Source: §5 · pp. 8–9

6. Visualization on the Trajectory Planning

The trajectory-planning visualizations show OpenREAD planning differently across an intersection, wet-road following, a parking gate, and a nighttime front-car scenario.

6. Visualization on the Trajectory Planning

  • At an intersection, OpenREAD performs less-conservative planning.Figure 5 illustrates this planning behavior.Source: Figure 5 · p. 13
  • On a wet road, OpenREAD follows safely and consistently.Figure 6 illustrates the following behavior.Source: Figure 6 · p. 13
  • At a parking gate, OpenREAD recognizes the gate and plans cautiously.Figure 7 illustrates the recognition and planning behavior.Source: Figure 7 · p. 13
  • At night, OpenREAD plans a slow trajectory to avoid collision with the front car.Figure 8 illustrates this scenario.Source: Figure 8 · p. 13

Related papers

Loading 2512.01830v2…