PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Driven Interactive Traffic Trajectory Generation

Junkai Xia, Chenxin Xu, Qingyao Xu, Chen Xie, Yanfeng Wang, Siheng Chen

arXiv:2405.15388v1cs.AIcs.RO

TL;DR

Prior language-driven trajectory generation methods do not account for interactive traffic dynamics, limiting the scope of language-based control. InteractTraj maps language into interaction-aware codes and decodes them into interactive trajectories. The authors report improved trajectory metrics on WOMD and nuPlan and greater user preference than the baseline.

  • Problem

    Prior language-driven trajectory generation methods do not model interactions among multiple traffic participants, although such interactions matter for representing complex traffic dynamics.

    Source: §1 · pp. 1–2

  • Method

    InteractTraj encodes language as interaction, vehicle, and map codes and uses interaction-aware feature aggregation to generate trajectories.

    Source: §1 · p. 2, §4.1 · p. 3, §4.2 · p. 4

  • Results

    InteractTraj reports 15.4%/18.7% average ADE/FDE improvement on WOMD, 17.1%/20.4% on nuPlan, and 47.5% higher average user preference than the baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results cover interactive trajectory generation under language commands involving overtaking, merging, yielding, and following.

    Source: §1 · p. 2

    The work generates trajectories only for vehicles, and map generation is limited by the map library.

    Source: §6 · p. 10

Abstract

from arXiv · show

Realistic trajectory generation with natural language control is pivotal for advancing autonomous vehicle technology. However, previous methods focus on individual traffic participant trajectory generation, thus failing to account for the complexity of interactive traffic dynamics. In this work, we propose InteractTraj, the first language-driven traffic trajectory generator that can generate interactive traffic trajectories. InteractTraj interprets abstract trajectory descriptions into concrete formatted interaction-aware numerical codes and learns a mapping between these formatted codes and the final interactive trajectories. To interpret language descriptions, we propose a language-to-code encoder with a novel interaction-aware encoding strategy. To produce interactive traffic trajectories, we propose a code-to-trajectory decoder with interaction-aware feature aggregation that synergizes vehicle interactions with the environmental map and the vehicle moves. Extensive experiments show our method demonstrates superior performance over previous SoTA methods, offering a more realistic generation of interactive traffic trajectories with high controllability via diverse natural language commands. Our code is available at https://github.com/X1a-jk/InteractTraj.git

1 Introduction

The introduction motivates flexible language control of traffic trajectories and identifies interactive dynamics as a gap in prior generation methods. InteractTraj maps language descriptions through interaction-aware numerical codes to interactive trajectories, with reported gains over previous methods.

1 Introduction

  • Driving simulation can generate data across diverse conditions at reduced cost, and trajectory data is a key component of those simulations.Source: §1 · p. 1
  • Earlier control methods used predefined signals, while LCTGen translated language into scenarios but generated individual trajectories without modeling interactions among participants.The introduction notes that traffic jams involving many vehicles cannot be accurately generated when interactions are absent.Source: §1 · pp. 1–2
  • InteractTraj converts language commands into interaction, vehicle, and map codes, then decodes them into interactive trajectories using interaction-aware encoding and feature aggregation.Interaction codes represent relative position and distance; the decoder combines vehicle interactions with environmental map data.Source: §1 · p. 2, Figure 1 · p. 2
  • 15.4%/18.7% average ADE/FDE improvement on WOMD and 17.1%/20.4% average ADE/FDE improvement on nuPlan are reported, alongside 47.5% higher average user preference than the baseline.The paper also reports realistic generation for commands involving overtaking, merging, yielding, and following.Source: §1 · p. 2

2 Related Work

Related work covers traffic trajectory generation, motion prediction, and language models for multimodal generation. Across these areas, the paper identifies controllability and interaction-aware language-driven generation as relevant challenges.

2.1 Traffic Trajectory Generation

  • Rule-based and learning-based trajectory methods face limits in realism, controllability, or reliance on costly past trajectories, while language-driven methods including LCTGen lack interaction awareness.TrafficGen controls vehicle count on a blank map; CTG uses a loss function for user controls, and CTG++ converts queries into a loss function.Source: §2.1 · pp. 2–3

2.2 Motion Prediction

  • Motion prediction methods progressed from physics-based models to RNN/LSTM, diffusion, and transformer approaches, with recent multi-agent work emphasizing interaction and relational reasoning.Trajectory generation is described as creating realistic scenarios that can supplement prediction-model development and evaluation.Source: §2.2 · p. 3

2.3 Large Language Models and Their Multimodal Applications

  • LLMs have supported text-conditioned image and video generation, and this work uses GPT-4 to convert language into codes describing interactions, vehicles, and maps.Source: §2.3 · p. 3

3 Problem Statement

The problem statement formulates language-driven traffic trajectory generation as producing realistic trajectories that match a language description. The output represents vehicle positions across a sequence of timesteps.

3 Problem Statement

  • Given a language description L, the goal is to generate traffic trajectories S that are realistic and match that description.Source: §3 · p. 3
  • The output S represents trajectories for N vehicles over T timesteps, with each vehicle represented by a sequence of positions.Source: §3 · p. 3
  • Each vehicle position s_i^t denotes its 2D location at timestep t.Source: §3 · p. 3

4 Methodology

InteractTraj maps natural-language descriptions to map, vehicle, and interaction codes, then decodes them into interactive trajectories. Its decoder combines map and interaction information in vehicle features, while training reconstructs scenarios from codes extracted from ground-truth trajectories.

4.1 Architecture Overview

  • The framework uses an LLM-based encoder to convert language into map, vehicle, and interaction codes, which a decoder transforms into interactive traffic trajectories.The codes represent environment-map, individual-driving, and vehicle-interaction information.Source: §4.1 · pp. 3–4

4.2 Language-to-Code Encoder

  • Prompted LLMs produce the three code types; interactions encode discretized relative direction and distance, while vehicle codes pair global trajectory type with detailed states.Vehicle types include stopping, straight driving, turning, and lane changes.Source: §4.2 · pp. 4–5

4.3 Code-to-Trajectory Decoder

  • The decoder extracts map, vehicle, and interaction features, adds map context to interaction and vehicle features, then fuses interactions into vehicle features for trajectory generation.The resulting vehicle features retain both map and interaction information.Source: §4.3 · pp. 5–6

4.4 Training

  • Because paired language–scenario data are lacking, training extracts codes from ground-truth scenarios and reconstructs them using trajectory and relative-distance losses.The relative-distance loss supervises distances between vehicles and the ego vehicle.Source: §4.4 · p. 6

4.5 Discussion

  • Unlike LCTGen, InteractTraj combines interaction codes and prompts, interaction-aware aggregation, and mixed-scale vehicle codes for global trajectory type and detailed movement.The mixed-scale design is intended to align trajectories with high-level intentions as well as precise positional changes.Source: §4.5 · pp. 6–7
  • Table 1 reports significantly improved trajectory realism on WOMD and nuPlan, with lower values indicating better performance.The supplied caption gives no metric values.Source: Table 1 · p. 7

5 Experiments

Experiments evaluate InteractTraj on WOMD and nuPlan using reconstruction metrics, interaction-type comparisons, language-conditioned studies, and ablations. Results favor InteractTraj over prior methods and show benefits from its design choices.

5.1 Dataset and Baseline

  • The evaluation uses WOMD and nuPlan, compares TrafficGen and LCTGen, and samples trajectories at 1-second intervals to obtain five timesteps.Source: §5.1 · p. 7, §5.2 · p. 7

5.3 Evaluation Metric

  • Realism is measured with mADE, minADE, mFDE, minFDE, scenario collision rate, and Hausdorff distance; collision rate counts scenarios with vehicle collisions.Source: §5.3 · p. 7

5.4 Reconstruction-based Evaluation

  • Reconstruction evaluation derives conditional inputs from ground-truth trajectories rather than the LLM, and InteractTraj’s inputs include interaction information absent from prior methods.Source: §5.4 · p. 8
  • InteractTraj reduces mADE/mFDE/HD by 15.4%/18.7%/16.9% versus SoTA methods, while InteractTraj(w/o I) still outperforms previous models.Source: §5.4 · p. 8
  • Across overtaking, converging, yielding, and following scenarios, InteractTraj significantly reduces mADE and mFDE relative to LCTGen.Source: §5.4 · p. 8

5.5 Language-Conditioned Evaluation

  • For overtaking, merging, yielding, and following commands, InteractTraj better matches described interactions than independently generated agent trajectories; Figure 5 compares interaction types.Source: Figure 5 · p. 8, §5.5 · p. 9
  • In user studies, 73.7% of preference responses favored InteractTraj on average, and 72.4% judged its scenarios to fulfill interaction descriptions, versus 31.5% for LCTGen.Source: §5.5 · p. 9, Figure 6 · p. 9

5.6 Ablation Study

6 Conclusion

InteractTraj converts language descriptions into multi-level codes and generates trajectories through attention-based information aggregation. Experiments show alignment with language descriptions and real-life scenario distributions, while the work remains limited to vehicle trajectories and map-library-bounded map generation.

6 Conclusion

  • InteractTraj converts language descriptions into multi-level codes and generates trajectories through attention-based information aggregation.The conclusion identifies this as the model’s core idea.Source: §6 · p. 10
  • Experiments show that InteractTraj reproduces real-life scenario distributions and generates scenarios aligned with language descriptions.Source: §6 · p. 10
  • The work generates trajectories only for vehicles, and map generation is limited by the map library.Future work would extend generation to other traffic participants and enable more flexible map generation.Source: §6 · p. 10

Appendix

The appendix provides additional experiment settings, metric definitions, prompts, module-design details, and user-study examples.

Appendix

  • The appendix adds experiment settings, evaluation metric definitions, complete language-to-code prompts, trajectory-analyzing module details, and user-study examples.Source: Appendix · p. 14

A Additional Experiment Settings

Experiments use WOMD and nuPlan, compare against TrafficGen and LCTGen, and report dataset, baseline, and implementation settings.

A Additional Experiment Settings

  • WOMD uses 68,000 training and 2,500 testing scenarios, while nuPlan uses 82,122 training and 20,756 testing scenarios.Each scenario retains at most 384 lanes and 32 vehicles; generated trajectories span 5 seconds at 10 fps (T=50).Source: A Additional Experiment Settings · p. 14
  • TrafficGen and LCTGen are the two controllable trajectory-generation baselines, with adjustments to standardize vehicle counts and prediction lengths for fair comparison.TrafficGen generates trajectories from empty maps without vehicle past states; LCTGen is described as the latest open-sourced language-guided approach.Source: A Additional Experiment Settings · p. 14
  • The model uses discretized distances and speeds, sampled trajectories, MCG lane-feature blocks, transformer decoding, cross-attention fusion, and an MLP trajectory generator.The specified design uses 5 MCG layers, a 2-layer 4-head transformer, 8-head cross-attention, and a 512-dimensional MLP; 100 epochs take about 12 hours on 4 RTX 4090 GPUs.Source: A Additional Experiment Settings · p. 14

B Evaluation Metric Details

Evaluation compares predicted and ground-truth trajectories using path-distance metrics and collision rate.

B Evaluation Metric Details

  • The six metrics are mADE, minADE, mean final displacement error, minFDE, Hausdorff distance, and collision rate.The first five measure projected-path gaps from ground truth; collision rate reflects scenario rationalization, and lower values indicate closer trajectories to ground truth.Source: B Evaluation Metric Details · p. 14

Symbol Definition

The section defines scenario and trajectory symbols and evaluation metrics, then specifies prompt rules for converting language descriptions into vehicle, map, and interaction codes.

Symbol Definition

  • G denotes test scenarios, NG their count, and T the timesteps; each scenario contains ground-truth and predicted trajectories for its agents.The trajectory sets are represented as S_g and cS_g, respectively.Source: Symbol Definition · p. 14
  • mADE, mFDE, minADE, and minFDE measure average or minimum trajectory-point and final-destination errors across agents in a scenario.The definitions distinguish errors over all predicted points from errors at the final destination.Source: Symbol Definition · pp. 14–15
  • HD measures distance between predicted and ground-truth trajectories as point sets, while SCR counts collisions when agents’ bounding-box IoU exceeds threshold δ at every timestep.The section defines IoU as intersection over union between two objects.Source: Symbol Definition · p. 15

C LLM Prompt Details

  • Vehicle codes encode relative position and distance, speed trends, and one of six actions: stop, straight, turn left or right, or change lanes left or right.Speed is represented over five seconds at one-second intervals, using nine grades spanning 0–20 m/s.Source: C LLM Prompt Details · p. 16
  • Map codes represent lane counts in four directions, distance to an intersection, and the ego vehicle’s lane; interaction codes track relative distance and position over time.Interaction codes use two five-value sequences, with neighboring values one second apart.Source: C LLM Prompt Details · pp. 16–17

D Trajectory Analyzing Module

The trajectory analyzing module derives decoder inputs from ground-truth scenes by discretizing map, interaction, and vehicle attributes into codes. It assigns trajectory types using motion patterns, with default handling for missing or outlier trajectories.

D Trajectory Analyzing Module

  • For training, the module analyzes each ground-truth scene into map, vehicle, and interaction codes, which the decoder uses to regenerate the scene.The extracted code dimensions match those produced by the language-to-code module.Source: D Trajectory Analyzing Module · p. 19
  • Interaction codes sample each non-ego vehicle’s trajectory every second and discretize its relative distance and position with respect to the ego vehicle.Relative position is encoded by a spatial region centered on the ego vehicle; relative distance uses fixed intervals.Source: D Trajectory Analyzing Module · p. 19
  • Vehicle codes discretize relative position and speed trends and classify trajectories as stop, straight, left or right turn, or left or right lane change.The classification rules use heading and lateral displacement relative to lane width for turns and lane changes.Source: D Trajectory Analyzing Module · pp. 19–20
  • Trajectories that remain within lane width and vary direction within a proper range are classified as straight; missing or outlier trajectories receive a default type and are masked.Stopping, turning, and lane-change categories are determined by predefined thresholds.Source: D Trajectory Analyzing Module · pp. 19–20

E Additional User Study Details

The additional user studies compare language-conditioned generation or assess whether generated scenarios match described interactions. Their examples illustrate left turns, coordinated lane changes, yielding, and merging.

E Additional User Study Details

  • The second study asks 28 participants to judge whether scenarios match 50 descriptions emphasizing representative interactions, yielding 1,400 responses.For each description, participants could answer positively or negatively for either model’s generated scenario.Source: E Additional User Study Details · pp. 20–21
  • The first study compares InteractTraj with LCTGen on 40 language commands, collecting 1,240 choices from 31 participants about which trajectory better fits each description.GPT-4 generated the descriptions, with interaction-event frequencies balanced as much as possible.Source: E Additional User Study Details · p. 20
  • Illustrative scenarios depict a vehicle slowing and turning left, following cars adjusting to a lane change, a sedan yielding to an ambulance, and a car slowing to let a motorcycle merge.Source: Figure 7 · p. 20, Figure 8 · p. 21, Figure 9 · p. 21
Loading 2405.15388v1…