Source-linked AI summary
Language Prompt for Autonomous Driving
Dongming Wu, Wencheng Han, Yingfei Liu, Tiancai Wang, Cheng-zhong Xu, Xiangyu Zhang, Jianbing Shen
TL;DR
Language-prompt driving research is constrained by scarce paired prompt-instance data, especially for fine-grained 3D objects. The paper introduces NuPrompt, a multi-object 3D, multi-view, multi-frame benchmark and a prompt-conditioned trajectory task, then provides PromptTrack as an end-to-end baseline. PromptTrack achieves strong reported performance on NuPrompt, including 0.259 AMOTA and motion-prediction results over eight frames.
Problem
Scarcity of 3D instance-text pairs limits language-prompt research in self-driving, while existing datasets leave fine-grained multi-object 3D correspondence insufficiently addressed.
Method
The paper constructs NuPrompt and formulates prompt-based driving as tracking and forecasting 3D object trajectories from language, using PromptTrack with prompt-visual fusion and prompt reasoning.
Results
PromptTrack achieves 0.259 on AMOTA, reports ADE 2.17 and FDE 2.21 over eight-frame motion prediction, and runs at 7.7 FPS with a VOV backbone on one Nvidia A100 GPU.
Takeaways & Limitations
NuPrompt supplies object-centric language supervision for 3D driving scenes and supports evaluation of prompt-based tracking and motion prediction across views and frames.
Abstract
from arXiv · showhide
A new trend in the computer vision community is to capture objects of interest following flexible human command represented by a natural language prompt. However, the progress of using language prompts in driving scenarios is stuck in a bottleneck due to the scarcity of paired prompt-instance data. To address this challenge, we propose the first object-centric language prompt set for driving scenes within 3D, multi-view, and multi-frame space, named NuPrompt. It expands nuScenes dataset by constructing a total of 40,147 language descriptions, each referring to an average of 7.4 object tracklets. Based on the object-text pairs from the new benchmark, we formulate a novel prompt-based driving task, \ie, employing a language prompt to predict the described object trajectory across views and frames. Furthermore, we provide a simple end-to-end baseline model based on Transformer, named PromptTrack. Experiments show that our PromptTrack achieves impressive performance on NuPrompt. We hope this work can provide some new insights for the self-driving community. The data and code have been released at https://github.com/wudongming97/Prompt4Driving.
1 Introduction
Language prompts have expanded visual-task flexibility, but driving research lacks sufficient 3D instance-text pairs and fine-grained object references. NuPrompt addresses this gap with a multi-object benchmark and a prompt-conditioned trajectory prediction task supported by PromptTrack.
- 3D instance-text pair scarcity limits the replication of language-prompt progress in self-driving scenarios.
- Existing driving datasets typically support single-object, single-image expressions or scene-level understanding rather than fine-grained multi-object 3D correspondence.
- NuPrompt introduces a prompt-based driving task that tracks and forecasts 3D object trajectories from a specified language prompt.
- 40,147 object-prompt pairs in NuPrompt align language descriptions with multiple object trajectories across 3D, multi-view, and multi-frame driving space.
- PromptTrack is an end-to-end baseline that adds prompt-visual cross-attention and a prompt reasoning branch to a camera-only 3D tracker.
2 Related Work
Language-prompt research in driving builds on nuScenes-based benchmarks and visual-language grounding datasets, while the annotation pipeline combines language elements into generated descriptions.
- Talk2Car is a pioneering autonomous-driving benchmark built on nuScenes that uses language prompts to support human-oriented driving-system interaction.
- The language-prompt annotation procedure collects language elements with referent objects, combines selected elements, and uses an LLM to generate descriptions.
- RefCOCO/+/g grounds visual regions in images using succinct, unambiguous natural-language expressions and later work supports expressions referring to unlimited target objects.
3 Dataset Overview
NuPrompt extends nuScenes with visual language prompts grounded in surrounding 3D, multi-view, and temporal driving scenes. The dataset combines semi-automatic annotation, generated prompts, and statistics describing referent objects and the PromptTrack benchmark pipeline.
- Dataset scope: NuPrompt focuses on visual images from six overlapping cameras, providing surrounding 360° 3D coverage across driving scenes.The source environments include pedestrian streets, public roads, and highways under varied weather and illumination conditions.
- Data collection and annotation: The labeling pipeline associates language elements with 3D bounding boxes, combines attributes, and uses an LLM to generate natural-language prompts.Language elements include object attributes such as colors, actions, locations, and classes.
- Prompt and referent statistics: NuPrompt prompts describe diverse object appearances, locations, and motions, while referents can cross camera views in surrounding 3D space.Common descriptions include colors, relative locations, and motion words such as walking, moving, and crossing.
- Benchmark protocol: PromptTrack fuses visual features with language embeddings, applies Transformer decoding with past and future reasoning, and predicts prompt-referred tracks.The architecture uses cached historical queries, predicted positions for cross-frame propagation, and binary classification for prompt reasoning.
4 Method
PromptTrack extends a query-based camera-only 3D tracker to use language prompts for tracking and forecasting referred objects across multi-frame, multi-view inputs. It combines prompt-aware visual processing, query propagation, past and future reasoning, and prompt-specific grounding within an end-to-end framework.
- PromptTrack overview: PromptTrack adapts PF-Track to track prompt-described objects from multi-frame, multi-view images using an end-to-end framework.The framework modifies query-based PF-Track to accept prompt input while retaining its tracking and forecasting structure.
- Prompt-aware feature processing: Cross-modal attention combines flattened visual features and encoded linguistic features to produce prompt-aware visual feature maps.The resulting features receive 3D position embeddings before interaction with 3D queries in Transformer decoder layers.
- Query construction: Track queries propagate across frames, while fixed detection queries capture newly born objects and provide the decoder's combined query set.The method uses 500 fixed queries and relies only on them for the first frame, which has no preceding frame.
- Past and future reasoning: Past reasoning integrates historical and cross-object information to refine queries and boxes, while future reasoning predicts long-term trajectories and updates positions for changing ego-coordinates.Past reasoning uses cached queries from previous frames; future reasoning predicts trajectories for subsequent frames and adjusts reference positions as the ego-car moves.
- Prompt grounding: Prompt reasoning uses refined queries to predict the probability that each output embedding corresponds to a prompt-referred object.This probability is produced by a two-layer MLP and is trained with a focal prompt reasoning loss.
- Instance matching and loss: One-to-one matching assigns tracking queries to propagated ground truth and fixed queries to new-born objects before classification, box, motion, and prompt losses are calculated.Fixed-query matching follows a bipartite graph matching procedure, while the loss combines focal classification, L1 box regression, L1 motion prediction, and focal prompt reasoning terms.
5 Experiment
Experiments on NuPrompt evaluate PromptTrack against adapted camera-only tracking baselines, motion prediction, fusion and prompt configurations, and qualitative tracking behavior. PromptTrack reaches 0.259 AMOTA, while ablations show the importance of both reasoning branches and prompt-visual fusion.
- Main Results: 0.259 AMOTA is achieved by PromptTrack, outperforming other counterparts across the majority of metrics.The comparison uses modified existing camera-only tracking models because no prior method directly targets the prompt-based prediction task.
- Main Results: ADE of 2.17 and FDE of 2.21 are obtained when PromptTrack forecasts motion over eight frames.PromptTrack with a PETR detector is the only model evaluated for motion prediction because single-frame detectors cannot effectively test object motion.
- Main Results: 7.7 FPS is achieved by PromptTrack with a VOV backbone on one Nvidia A100 GPU over the validation set.
- Ablation Studies: Neglecting either past or future reasoning diminishes performance, supporting the inclusion of both reasoning branches.
- Ablation Studies: The prompt-visual fusion strategy is effective, while bidirectional fusion and using prompts as queries do not yield notable performance gains.The evaluated variants include removing prompt fusion, adding visual-enhanced language features, and adding prompt embeddings to decoder queries.
- Qualitative Results: PromptTrack accurately detects and tracks prompt-referred targets across different views and varying object numbers in qualitative examples.
6 Conclusion
The paper presents NuPrompt for language-guided 3D perception in autonomous driving and introduces prompt-driven trajectory prediction with PromptTrack. Experiments on NuPrompt verify the effectiveness of the proposed algorithm.
- NuPrompt is presented as the first large-scale language prompt set designed specifically for 3D perception in autonomous driving.
- The paper defines a prompt-driven task that tracks objects and forecasts their motion using a language prompt as a semantic cue.
- PromptTrack is an end-to-end prompt-based tracking model with prompt reasoning modification on PF-Track.
- Experiments on NuPrompt verify the effectiveness of the proposed algorithm.