PaperlayerView exact revision on arXiv

Source-linked AI summary

doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Driving

Parthib Roy, Yash Tandon, Marcus Blennemann, Giovanni Tapia Lopez, Angel Martinez-Sanchez, Mohan M. Trivedi, Ross Greer

arXiv:2609.38028v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Language-enabled driving datasets largely focus on localized interactions, while passenger intent can persist across stages and evolving scenes. doPlan adds extended-context human instructions and evaluates four language-conditioned driving models; language sensitivity does not reliably correspond to requested-direction behavior, and associated maneuvers often fall beyond short prediction horizons.

  • Problem

    Language-enabled driving work largely studies localized interactions, leaving passenger intent that persists across stages of an evolving drive comparatively underexplored.

    Source: §I · p. 1

  • Method

    doPlan pairs free-form passenger instructions with extended nuPlan driving contexts and evaluates four language-conditioned driving models.

    Source: §I · p. 2

  • Results

    Language sensitivity does not reliably translate into requested-direction behavior, and the maneuvers associated with instructions often lie beyond short prediction windows.

    Source: §I · p. 2, §VII · p. 7

  • Takeaways & Limitations

    The findings point to planning that retains unresolved passenger goals, grounds them as scenes evolve, and tracks them across multiple stages.

    Source: §VIII · p. 8

    doPlan does not distinguish referenced information known from prior context from information visible at the evaluation time point.

    Source: §VI · p. 7

Abstract

from arXiv · show

Autonomous vehicles interacting with passengers through natural language must reason beyond immediate commands. Passenger intent may span multiple stages of behavior, depend on future events, refer to surrounding agents or landmarks, and remain relevant as driving conditions evolve. Existing language-enabled driving datasets largely focus on short, localized interactions, leaving these longer-horizon forms of passenger intent comparatively underexplored. We introduce doPlan, to our knowledge the first publicly available, human-annotated real-world dataset designed to study passenger language as persistent task context. Built on nuPlan, doPlan contains 5,154 human-written passenger instructions spanning 169.1 hours of cumulative instruction-aligned context over 50.9 hours of unique driving, with annotation windows ranging from 30.0 to 508.8 s. The annotations capture immediate, deferred, event-conditioned, persistent, and multi-stage passenger intent. The dataset, annotation interface, and supporting resources are publicly available at https://github.com/Mi3-Lab/doPlan. We evaluate four language-conditioned driving models and find that sensitivity to passenger language does not reliably translate into behavior consistent with the requested direction. More broadly, among 2,161 examples with a matched future maneuver, the first associated maneuver occurs a median of 24.6 s after the evaluation point, and only 9.8% occur within the models' common 5 s prediction horizon. These findings highlight the need to connect persistent passenger intent with successive planning decisions. doPlan provides a setting for studying how unresolved goals can be retained, grounded in evolving scenes, and tracked across multiple stages, including how a planner determines when a future goal becomes relevant to the current plan.

I. INTRODUCTION

Passenger instructions can encode goals that remain pending as driving conditions evolve, while prior language-enabled driving work largely centers on localized or short-horizon interactions. doPlan introduces extended passenger-instruction annotations and evaluates whether current driving models use them consistently with the requested direction.

I. INTRODUCTION

  • Persistent instructions require systems to retain unresolved intent, ground it in new observations, and determine when it should influence current action.The driving examples include holding a lane until construction is passed, then pursuing a later lane-change goal.Source: §I · pp. 1–2
  • Prior language-enabled driving work largely studies localized scenes or short-horizon interactions, leaving instructions that persist across multiple stages comparatively underexplored.doScenes uses fixed 12-second clips, leaving longer, multi-stage passenger intent largely outside its scope.Source: §I · p. 2
  • doPlan provides 5,154 human-written instructions across 169.1 hours of instruction-aligned context and 50.9 hours of unique driving, using 30.0–508.8-second windows.Its instructions include ordered maneuvers, deferred actions, and behavior that persists until a condition is satisfied.Source: §I · p. 2
  • Four evaluated driving models show that language sensitivity does not reliably produce requested-direction behavior, while associated maneuvers often lie beyond short prediction windows.The findings motivate treating passenger language as persistent task context rather than a one-shot perturbation to the next trajectory.Source: §I · p. 2

II. RELATED WORK

Prior work applies language to driving perception and behavior, but generally studies bounded scenes or planning episodes. doPlan adds passenger instructions to continuous nuPlan contexts, extending the temporal scope of this work.

II. RELATED WORK

  • Driving datasets support perception, forecasting, scene understanding, and continuous motion planning, while doPlan adds free-form passenger instructions to nuPlan contexts.The cited datasets include KITTI, Waymo Open Dataset, nuScenes, and nuPlan.Source: §II · p. 2
  • Language-enabled driving work spans explanations, advice, scene understanding, and instruction-conditioned behavior, but generally uses bounded scenes or planning episodes.Longer temporal structure has also been studied in navigation and robotics; doScenes uses fixed 12-second driving clips, whereas doPlan uses 30.0–508.8-second contexts.Source: §II · p. 2

A. Dataset Source and Annotation Interface

  • The annotation interface presents eight synchronized camera views plus a heading-aligned map and compass to provide visual and route context.The map and compass show road geometry, upcoming turns, cardinal direction, and nearby landmarks.Source: A. Dataset Source and Annotation… · p. 2

B. Segment Sampling Strategy

doPlan samples variable-length intervals from continuous driving sequences and elicits free-form passenger instructions for the behavior shown. Its protocol allows multi-stage language and contextual references, while referential labels remain coarse categories.

B. Segment Sampling Strategy

  • Intervals are sampled from continuous nuPlan trajectories with durations drawn uniformly from 30 seconds to the sequence duration and start times drawn uniformly from the remaining interval.Sampling does not preselect maneuver or scenario categories; overlaps are allowed, and the 30-second minimum supports extended instruction generation.Source: B. Segment Sampling Strategy · p. 3

C. Instruction Annotation Protocol

  • The taxi-test prompt asks annotators what they would tell a driver to produce the observed behavior, allowing single-clause, multi-stage, or no-instruction submissions.Instructions have no predefined template or command vocabulary; no-instruction submissions are retained for accounting but excluded from the passenger-instruction set.Source: C. Instruction Annotation Protocol · p. 3
  • The sampled window defines the context shown to annotators, not when the passenger spoke; instructions may draw on route or environmental knowledge beyond a single visible frame.Such context can include landmarks and upcoming turns.Source: C. Instruction Annotation Protocol · p. 3
  • Nineteen licensed drivers annotated the data independently, and doPlan does not assume a unique ground-truth instruction for each observed trajectory.Annotators could not see previous annotations associated with a segment.Source: C. Instruction Annotation Protocol · p. 3

D. Referential Labels

  • Referential labels categorize environmental references but do not provide fine-grained word-to-object or bounding-box grounding.The labels and their dataset distribution are summarized in Table III.Source: D. Referential Labels · p. 3

IV. DATASET ANALYSIS

The dataset analysis reports the number of instruction submissions and the driving context they cover, then organizes comparisons by temporal coverage, instruction structure, and referential labels.

IV. DATASET ANALYSIS

  • 5,154 of 6,549 submissions contain passenger instructions, spanning 169.1 hours of instruction-aligned context and 50.9 hours of unique driving across 988 nuPlan clips.The other 1,395 submissions contain no instruction text.Source: §IV · p. 3

A. Scale and Temporal Coverage

  • Table I compares doScenes and doPlan in scale and temporal coverage, framing doPlan as an extension of the shorter-term interaction setting.The supplied passages do not provide the table’s cell values.Source: A. Scale and Temporal Coverage · p. 3, Table I · p. 3
  • Instruction-bearing windows span a broad range of temporal horizons, and Table II groups instruction structure by annotation-window duration.The supplied passages do not report the distribution values or the group-level structure results.Source: A. Scale and Temporal Coverage · p. 3, Table II · p. 4
  • Table III summarizes the distribution of doPlan’s referential labels.The supplied passages identify the table’s subject but do not provide its category values.Source: Table III · p. 4

B. Temporal Structure and Instruction Complexity

Longer annotation windows are associated with more complex, multi-step instructions and frequent references to the driving environment. A retrieval validation finds that additional driving context improves retrieval of the matching instruction.

B. Temporal Structure and Instruction Complexity

  • The temporal-structure analysis groups the 5,154 instruction-bearing annotations into five approximately equal groups by annotation-window duration.It measures mean words per instruction, multi-sentence share, and a multi-step proxy.Source: B. Temporal Structure and Instruction… · p. 4
  • Across shortest- and longest-duration annotation windows, mean instruction length increased from 9.8 to 27.8 words, multi-sentence share from 24.2% to 52.3%, and multi-step-proxy share from 28.8% to 62.6%.Source: B. Temporal Structure and Instruction… · p. 4

C. Referential and Linguistic Structure

  • 84.1% of instructions are labeled static, dynamic, or both, indicating references to agents, landmarks, or other elements of the driving environment.Static references are most common, followed by instructions containing both static and dynamic references.Source: C. Referential and Linguistic Structure · p. 4

D. Temporal Context Validation

  • R@1 rose from 13.9% (first 5 s) to 23.1% (full window), while R@5 rose from 38.2% to 54.2% and R@10 from 52.2% to 68.6%.Duration-only, text-only, and frozen CLIP baselines achieved 2.7%, 1.3%, and 3.5% R@1, respectively, versus 23.1% for the learned matcher.Source: Table IV · p. 4, D. Temporal Context Validation · p. 4
  • The video–instruction matcher compares visual features from front-camera frames with text features to rank matching and mismatched pairs.It is used to test whether longer windows contain instruction-relevant information beyond the initial seconds.Source: D. Temporal Context Validation · p. 4
  • For each sequence, the matcher ranks 100 human-written instructions: the associated instruction and 99 drawn from other nuPlan source logs.Instructions from the same source log are excluded from the negative set to avoid treating potentially valid annotations of overlapping context as incorrect.Source: D. Temporal Context Validation · p. 4

V. MODEL EVALUATION AND DIAGNOSTIC ANALYSIS

Across four models, the corresponding passenger instruction does not consistently improve trajectory alignment, and models do not reliably distinguish it from unrelated or counterfactual instructions. The evaluation compares language conditions using a shared 5 s prediction horizon.

V. MODEL EVALUATION AND DIAGNOSTIC ANALYSIS

  • Models were evaluated zero-shot with native sensor configurations at the earliest point with required sensor history and future trajectory; this point was the annotation-window start in 85.0% of cases.Otherwise, the evaluation point occurred 0.2–4.6 s later, with a median offset of 2.8 s; it is not the time at which the passenger would have spoken.Source: §V · pp. 4–5
  • OpenEMMA improved trajectory alignment by 1.1%, while AutoVLA and Alpamayo 1.5 worsened by 4.5% and 6.1%; Alpamayo 2.0 ADE increased 26.4%, from 1.451 m to 1.834 m.Correct instructions produced ADE values similar to unrelated and counterfactual instructions across all four models.Source: Table V · p. 5, §V · p. 5
  • The evaluation used 4,334 of 5,154 instructions, with 4,315 available for direct cross-model comparison; 19 were excluded because AutoVLA lacked valid action-format outputs.The subset reflects limited shared compute; Alpamayo 2.0 required two NVIDIA H200 GPUs and four generations per example.Source: §V · p. 5
  • Each model was tested with no instruction, the corresponding passenger instruction, an unrelated instruction, and a rule-based counterfactual requesting different behavior.Predicted trajectories were standardized to ten 0.5 s intervals over the longest horizon shared by all four models: 5 s.Source: §V · p. 5

A. Navigation Guidance and Language Sensitivity

Stronger navigation guidance increases Alpamayo 2.0’s sensitivity to passenger wording, but that sensitivity does not reliably produce motion in the requested direction. Most matched turns occur beyond the models’ 5 s horizon, limiting what immediate directional compliance can establish.

A. Navigation Guidance and Language Sensitivity

  • Trajectory separation rises from 0.219 to 0.679 m across guidance settings, reaching 0.533 m at the released default w = 3.0.The measure captures how much counterfactual wording changes the predicted trajectory, not whether either trajectory is more correct.Source: A. Navigation Guidance and Language… · p. 5, Table VI · p. 5

B. Directional Instruction Compliance

  • Only 4.4% of matched requested turns begin within the common 5 s prediction horizon, so most are beyond the trajectory available for evaluation.The evaluation therefore restricts directional-compliance analysis to matched turns beginning within that horizon.Source: B. Directional Instruction Compliance · pp. 5–6

A. Navigation Guidance and Language Sensitivity

  • A same-scene left–right instruction swap separates Alpamayo 2.0’s trajectories by 0.95 m at w = 1.0 and 3.84 m at w = 4.0.At higher guidance, the counterfactual trajectory shifts toward the left lane while the original bends more sharply right.Source: Figure 3 · p. 6

B. Directional Instruction Compliance

  • At w = 3.0, passenger instructions increase requested-side response by only 1.3 percentage points over no instruction, from 48.0% to 49.3%.No corresponding increase is observed at w = 1.0.Source: B. Directional Instruction Compliance · p. 6
  • Counterfactual directional consistency is absent in all 387 sampled predictions, though mechanically reversed directions may be infeasible in some scenes.The authors treat this test as a diagnostic rather than a complete measure of executable compliance.Source: B. Directional Instruction Compliance · p. 6

C. Temporal Alignment of Future Maneuvers

The matched maneuver typically occurs well after the models’ 5 s prediction horizon. Models vary in how early directional wording affects their trajectories, but an early response is not by itself evidence of premature execution.

C. Temporal Alignment of Future Maneuvers

  • Among 2,161 annotations with aligned logs and a detected maneuver, the event occurs a median of 24.6 s after evaluation, and only 9.8% fall within the 5 s horizon.Restricting the analysis to shorter annotation windows yields the same qualitative result, though the offset depends partly on window construction.Source: C. Temporal Alignment of Future… · p. 6
  • The temporal analysis aligns 4,312 annotations to nuPlan drives and detects the earliest supported action from ego-pose logs and map data; 2,161 have valid alignment and a detected onset.Supported events include turns, sustained stops, and straight intersection traversals.Source: C. Temporal Alignment of Future… · p. 6
  • AutoVLA’s directional response remains positive even when the matched human turn is more than 60 s away; OpenEMMA and Alpamayo 1.5 show little consistent response, while Alpamayo 2.0 shows a smaller effect.The analysis compares original and direction-reversed instructions and groups response by time to the recorded turn.Source: C. Temporal Alignment of Future… · pp. 6–7, Table VII · p. 7
  • An early directional response is not proof of premature execution: road curvature, lane positioning, or preparation may appropriately shape motion before a turn.The authors treat the experiment as a diagnostic of temporal grounding, not proof that future intent is being executed too soon.Source: C. Temporal Alignment of Future… · p. 7

VI. LIMITATIONS

The authors identify annotation and timing boundaries that matter when interpreting doPlan labels and benchmark offsets. The dataset does not independently revalidate every retained instruction or distinguish prior knowledge from information visible at evaluation.

VI. LIMITATIONS

  • Retained instructions are not independently revalidated in a second manual-review round, so some annotator variation may remain by design.doPlan does not assume a unique ground-truth instruction for an observed driving behavior.Source: §VI · p. 7
  • Annotators may replay the full segment, so instructions can reference information not visible at evaluation; doPlan does not distinguish prior knowledge from visible information.The protocol also does not annotate natural instruction-utterance or actionability time.Source: §VI · p. 7
  • Reported temporal offsets are relative to the selected evaluation time point, not measured instruction-to-action delays.The dataset does not annotate when an instruction was naturally uttered or became actionable.Source: §VI · p. 7

VII. DISCUSSION

Using passenger language is more than responding when wording changes: a planner must interpret the requested behavior and decide when it should affect the plan. Long-horizon instruction following also requires tracking intent and evaluating whether it is realized as the drive evolves.

VII. DISCUSSION

  • Greater sensitivity to passenger language does not reliably correspond to behavior in the requested direction.The discussion contrasts Alpamayo 2.0’s increased sensitivity under stronger navigation guidance with directional-compliance results.Source: §VII · p. 7
  • A multi-stage instruction can shift from constraining current behavior to a pending goal, then change status as relevant events occur.The discussion describes instruction parts as pending, active, completed, or invalidated as the scene evolves.Source: §VII · p. 7
  • Evaluating long-horizon instruction following requires more than trajectory error, including event completion, stage order, anticipation, and safe motion.The discussion notes that little immediate response may be appropriate for future maneuvers, while lane positioning may begin earlier.Source: §VII · p. 7
  • Long-horizon planning problems include anticipating future maneuvers, retaining unresolved goals, grounding event conditions, and deferring or revising actions.The discussion frames these as temporal grounding, persistent intent and progress, event-conditioned execution, and deferral and revision.Source: §VII · pp. 7–8
  • doPlan preserves free-form passenger language as a setting for studying intent across stages and temporal horizons, rather than prescribing one task or evaluation protocol.Passengers naturally use goals, references, preferences, and contingencies rather than precise waypoints or control commands.Source: §VII · p. 8

VIII. CONCLUSION

doPlan is a human-annotated dataset for studying passenger instructions over extended, evolving driving contexts. Evaluations show that language sensitivity is not equivalent to following the requested direction, while associated maneuvers often lie beyond current prediction horizons.

VIII. CONCLUSION

  • Four contemporary language-conditioned driving models show that sensitivity to passenger language is not the same as following it, and associated maneuvers typically fall outside current prediction horizons.The conclusion states that neither gap is addressed by better trajectory regression alone.Source: §VIII · p. 8
  • doPlan contains 5,154 free-form passenger instructions spanning 169.1 hours of cumulative context across 50.9 hours of unique annotated driving.The instructions capture intent spanning multiple stages, depending on future events, and remaining relevant beyond the next predicted trajectory.Source: §VIII · p. 8
  • doPlan provides a setting to develop models that retain unresolved goals, reason over multi-stage instructions, and ground references as scenes evolve.It also supports distinguishing immediate from future intent and evaluating behavior when a requested action may not yet be relevant.Source: §VIII · p. 8

Related papers

Loading 2609.38028v1…