Source-linked AI summary
LongTail Driving Scenarios with Reasoning Traces: The KITScenes LongTail Dataset
Royden Wagner, Omer Sahin Tas, Jaime Villa, Felix Hauser, Yinzhe Shen, Marlon Steiner, Dominik Strutz, Carlos Fernandez, Christian Kinzig, Guillermo S. Guitierrez-Cabello, Hendrik Königshof, Fabian Immel, Richard Schwarzkopf, Nils Alexander Rack, Kevin Rösch, Kaiwen Wang, Jan-Hendrik Pauls, Martin Lauer, Igor Gilitschenski, Holger Caesar, Christoph Stiller
TL;DR
Rare-scenario decision-making remains a challenge for self-driving systems, especially beyond perception-level generalization. The paper introduces a long-tail driving dataset and benchmark combining multimodal inputs, instructions, expert multilingual reasoning traces, and evaluations of semantic coherence and multiple maneuvers. Few-shot prompting consistently improves results over zero-shot baselines, while CoT prompting can hurt open-source models.
Problem
Decision-making in long-tail driving scenarios remains challenging despite improvements in perception-level generalization.
Method
The paper introduces a dataset and benchmark combining multi-view driving data, high-level instructions, multilingual expert reasoning traces, semantic-coherence evaluation, and the multi-maneuver score.
Results
Few-shot prompting and few-shot CoT consistently improve results over zero-shot baselines across evaluated models and measures.
Takeaways & Limitations
The dataset supports studying domain-grounded and multilingual reasoning for instruction-driven decision-making in long-tail driving scenarios.
Takeaways & Limitations
CoT prompting worsens results for open-source models, possibly because their pretrained reasoning traces differ from the driving traces used here.
Abstract
from arXiv · showhide
In real-world domains such as self-driving, generalization to rare scenarios remains a fundamental challenge. To address this, we introduce a new dataset designed for end-to-end driving that focuses on long-tail driving events. We provide multi-view video data, trajectories, high-level instructions, and detailed reasoning traces, facilitating in-context learning and few-shot generalization. The resulting benchmark for multimodal models, such as VLMs and VLAs, goes beyond safety and comfort metrics by evaluating instruction following and semantic coherence between model outputs. The multilingual reasoning traces in English, Spanish, and Chinese are from domain experts with diverse cultural backgrounds. Thus, our dataset is a unique resource for studying how different forms of reasoning affect driving competence. Our dataset is available at: https://hf.co/datasets/kit-mrt/kitscenes-longtail
1 Introduction
The paper addresses long-tail decision-making by introducing a dataset that combines driving data, high-level instructions, and multilingual reasoning traces. It also proposes evaluation tools for multiple maneuvers and semantic alignment, and finds that few-shot prompting improves brittle zero-shot planning.
- Long-tail decision-making remains challenging despite advances in perception-level generalization, motivating domain-grounded reasoning for rare driving scenarios.
- The dataset couples synchronized six-view video and stitched 360◦ frames with high-level instructions and expert reasoning traces in English, Chinese, and Spanish.
- The multi-maneuver score rates safety, comfort, and instruction-following across multiple possible futures rather than a single expert trajectory.
- Zero-shot planning in long-tail scenarios is brittle, whereas few-shot prompting improves planning and highlights the need for domain-grounded reasoning.
- The benchmark evaluates semantic coherence by measuring whether reasoning-trace actions match predicted trajectories.
2 Related work
Prior self-driving datasets and benchmarks provide strong visual coverage but limited support for behavioral generalization, rich instructions, and reasoning evaluation in rare scenarios. This work positions its benchmark as a complement through long-tail scenario coverage, language-guided decision-making, and multi-maneuver evaluation.
- Existing multi-sensor datasets support visual generalization across sensors and regions but primarily target perception rather than planning.
- The benchmark complements prior datasets by integrating multi-view video, high-level instructions, and expert reasoning traces for long-tail decision-making.
- End-to-end driving benchmarks commonly evaluate against one expert trajectory, while non-reactive and reactive simulation approaches have distinct limitations.
- Waymo Open E2E covers rare long-tail scenarios but lacks video data, reasoning traces, and semantic-coherence evaluation.
- Chain-of-thought reasoning extends language models with intermediate steps, while VLMs and VLAs condition on visual inputs and can ground reasoning in executable actions.
- Prior datasets use varied language annotations, including graph-based Q&A or auto-generated captions, and may evaluate trajectories against single expert trajectories.
3 Dataset
The dataset targets long-tail driving by collecting rare and challenging scenarios across diverse environments, then pairing multi-view video with detailed instructions and expert reasoning traces. Its data pipeline provides 360° visual coverage, stitched frames, and multilingual, trajectory-grounded annotations for studying context-aware driving decisions.
- 3.1 Scenarios: Data collected across urban, suburban, and highway environments emphasizes rare events, including adverse weather, road closures, accidents, construction zones, and intersections.Routes were adjusted and filtered to obtain scenarios that diverge from nominal data distributions.
- 3.1 Scenarios: Scenario distributions are approximately equal across the dataset splits.Figure 2 reports scenario-type frequencies as percentages.
- 3.2 Multi-view videos and frame-wise stitching: The dataset provides 360° multi-view video from six viewing angles, with raw and pinhole image formats and frame-wise stitched views.Stitching uses section-wise blends of homography and identity transformations to generate panoramic views.
- 3.3 High-level instructions: High-level instructions specify intended maneuvers, with common commands including driving straight, turning, lane use, and detailed overtaking actions.Some instructions intentionally cannot be executed because of oncoming traffic, obstacles, or the ego vehicle being overtaken.
- 3.4 Reasoning traces: Domain experts produce reasoning traces in English, Chinese, and Spanish by answering five questions about observations and trajectory-grounded steering and acceleration actions.Questions cover the next 0 s to 3 s and final 2 s of a 5-second future, while responses may be verbal or written.
4 Metrics
The paper introduces semantic coherence to measure whether reasoning-trace actions match planned-trajectory actions, and MMS to evaluate multiple maneuvers using safety, comfort, similarity, and instruction-following.
- 4.1 Semantic coherence between model outputs: Semantic coherence measures whether driving actions described in reasoning traces match actions in planned or predicted future trajectories.Rocchio classification compares embeddings of reasoning-trace segments with reference embeddings for all taxonomy-defined driving actions.
- 4.1 Semantic coherence between model outputs: Classification-based coherence remains robust to synonyms that can produce divergent BLEU scores, indicating semantic rather than lexical alignment.The resulting accuracy measures whether reasoning-trace actions semantically align with the final planned trajectory.
- 4.2 Multi-maneuver score: Each scenario supplies three reference trajectories: expert-like, wrong-speed, and instruction-neglect examples, alongside manually labeled crash trajectories.Wrong-speed references modify average speed by ±20%, while instruction-neglect references represent reasonable trajectories that violate the high-level instruction.
- 4.2 Multi-maneuver score: MMS ranks planned trajectories against reference trajectories across five maneuver categories while incorporating comfort penalties for excessive jerk or tortuosity.The metric is designed to cover multiple maneuvers, potential crashes, instruction-following, and comfort.
- 4.2 Multi-maneuver score: MMS subtracts a comfort penalty CP ∈{0, 1, 2} from category scores and reduces the score when jerk exceeds the reference by more than 44% or tortuosity by at least 6%.These thresholds are based on empirical standard-deviation-to-mean ratios computed from expert trajectories.
- 4.2 Multi-maneuver score: Trajectory similarity uses velocity-dependent lateral and longitudinal thresholds, assigning full similarity within both thresholds and the lower axis-wise similarity otherwise.The final MMS combines similarity with category-specific rules, including crash, off-road, and comfort handling.
5 Experiments
The experiments evaluate MMS, model planning under zero-shot and few-shot prompting, and coherence between reasoning traces and trajectories. Results show MMS aligns better with DrivingScore than L2 errors, while few-shot methods improve open-source models but raw CoT can hurt unless paired with kinematic decoding.
- 5.1 Relationship between MMS, L2 errors, and closed-loop DrivingScores: MMS correlates more strongly with DrivingScore than L2 errors and penalizes trajectories that swerve heavily.The reported Pearson correlations are 0.59 for MMS and −0.45 for L2 versus DrivingScore.
- 5.2 End-to-end driving evaluation: Do models generalize to our data?: Zero-shot evaluation shows closed-source and classic end-to-end models outperform open-source VLMs, with Gemini 3 Pro achieving the highest MMS overall.The evaluation plans a 5 s future trajectory from visual context, past trajectory, and a high-level instruction.
- 5.2 End-to-end driving evaluation: Do models generalize to our data?: Few-shot and few-shot CoT prompting significantly improve open-source model performance, but CoT alone worsens results relative to plain few-shot prompting.The authors relate this degradation to differences between driving reasoning traces and reasoning patterns in pretraining and instruction-tuning.
- 5.2 End-to-end driving evaluation: Do models generalize to our data?: Performance is strongest at night and weakest on snow, intersection, and specifically selected scenarios.The authors associate snow and specifically selected cases with difficulty, and intersections with more viable trajectories and weaker instruction following.
- 5.2 End-to-end driving evaluation: Do models generalize to our data?: Kinematic conversion of driving actions described in CoT traces yields the best results for open-source models.The model predicts discrete, speed-dependent acceleration and steering actions, which a kinematic bicycle model converts into a future trajectory.
- 5.3 Semantic coherence between model outputs: Reasoning-trace and trajectory coherence is low, with average scores from 0.27 to 0.51 across acceleration and steering intervals.The authors report that actions in reasoning traces often do not match planned trajectories, indicating frequent hallucinated traces or unreasonable trajectories.
6 Conclusion and discussion
The paper presents a long-tail driving resource combining multimodal scenarios, instructions, and expert reasoning traces, then identifies improvements from in-context prompting. It positions the dataset as a basis for studying reasoning, generalization, and interpretability in driving.
- Conclusion: The dataset provides long-tail driving scenarios with multi-view videos, high-level instructions, and human-labeled reasoning traces.The reasoning traces are multilingual and support evaluation of semantic coherence and multimodal driving behavior.
- Conclusion: Few-shot examples and few-shot CoT produce consistent improvements over zero-shot baselines.The conclusion summarizes these prompting results alongside measurements of semantic coherence and driving multimodality.
- Discussion: The dataset supports future studies of reinforcement-learning fine-tuning, reasoning styles and languages, world models, human-like reasoning, and interpretability.The authors specifically connect interpretability with understanding mechanisms behind actions, debugging, and model development.
7 Supplementary Material: LongTail Driving Scenarios with Reasoning Traces
The supplementary material documents action mappings, validation results, qualitative scenarios, and prompting formats. It shows validation trends consistent with the test set and specifies how zero-shot, few-shot, CoT, and kinematic prompts are constructed.
- 7.1 Action mapping: The action-to-kinematics mapping converts discrete driving actions into acceleration values and steering angles.This mapping is used in the few-shot CoT kinematic inference setting.
- 7.2 Results on our validation set: On the validation set, closed-source models lead zero-shot MMS and L2 performance, while open-source models surpass them in MMS with few-shot prompting, especially with kinematic modeling.The supplementary results are described as similar to the main-paper test results.
- 7.3 Qualitative results: The qualitative examples cover heavy-rain intersection turns, lane changes, snow, crashes, road closures, and combinations of rare conditions.Figure 5 compares expert, wrong-speed, instruction-neglect, and Qwen3-VL trajectories; Figure 6 presents selected adverse-weather and long-tail scenes.
- 7.4 Zero-shot and few-shot prompts: Zero-shot prompts provide visual input, past trajectory, driving instruction, and a task description before requesting future waypoints.The prompt format uses a 5 Hz trajectory representation and a 5 s prediction horizon.
- 7.4 Zero-shot and few-shot prompts: Few-shot prompting adds three example prompts and trajectories, while few-shot CoT augments those examples with reasoning traces.The kinematic variant adds reasoning traces but removes explicit future trajectories, requiring action-based trajectory generation.
- 7.4 Zero-shot and few-shot prompts: The kinematic prompt restricts outputs to five acceleration commands and four steering commands within a fixed XML-like structure.The required fields include situational awareness, interval-specific actions, and natural-language reasons.