Source-linked AI summary
Talk2Car: Taking Control of Your Self-Driving Car
Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, Marie-Francine Moens
TL;DR
The paper studies how an autonomous vehicle can ground a passenger’s natural-language command in a shared urban visual scene by identifying the referred object. It introduces Talk2Car, a command-annotated driving dataset, and evaluates strong models, finding a challenging task with promising results while noting scope limitations and room for further research.
Problem
The paper addresses grounding a passenger’s natural-language command to an object in an autonomous-driving scene before the vehicle executes the requested action.
Method
The authors create Talk2Car by adding free-form commands and object references to nuScenes videos, then evaluate several state-of-the-art object-referral models.
Results
The task is challenging but yields promising results: MAC performs best on nearly all tasks, while simple baselines perform poorly.
Takeaways & Limitations
Talk2Car provides a benchmark for joint language and visual understanding of natural-language commands in self-driving-car scenes.
Takeaways & Limitations
The dataset’s executable vehicle paths are deferred to a later version, and available 3D information was neglected for comparison with existing models.
Abstract
from arXiv · showhide
A long-term goal of artificial intelligence is to have an agent execute commands communicated through natural language. In many cases the commands are grounded in a visual environment shared by the human who gives the command and the agent. Execution of the command then requires mapping the command into the physical visual space, after which the appropriate action can be taken. In this paper we consider the former. Or more specifically, we consider the problem in an autonomous driving setting, where a passenger requests an action that can be associated with an object found in a street scene. Our work presents the Talk2Car dataset, which is the first object referral dataset that contains commands written in natural language for self-driving cars. We provide a detailed comparison with related datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref and CLEVR-Ref. Additionally, we include a performance analysis using strong state-of-the-art models. The results show that the proposed object referral task is a challenging one for which the models show promising results but still require additional research in natural language processing, computer vision and the intersection of these fields. The dataset can be found on our website: http://macchina-ai.eu/
1 Introduction
The paper frames natural-language control of autonomous vehicles as a visual grounding problem: identifying the object a passenger’s command refers to. It introduces Talk2Car to support this task and evaluates models under challenging language, visual, and speed conditions.
- Motivation: Providing passengers a way to communicate with an autonomous vehicle could address concerns about limited control, hesitation, and requests for situation-specific feedback.The introduction gives parking in the shade as an example of a passenger request.
- Problem: The task maps an image I and command C to the referred region R, before planning and executing the requested vehicle action.The paper restricts the task to commands referring to one targeted object.
- Challenges: Commands can require disambiguating objects through modifiers, spatial relationships, coreference, long sentences, and distant visual targets.The paper notes that some commands contain ambiguous or implicit referrals and difficult language or visual context.
- Contribution: Talk2Car is presented as the first object referral dataset with free natural-language commands for self-driving cars.It adds textual commands to urban visual scenes to stimulate research on grounding commands in visual space.
- Evaluation: The study evaluates models on object disambiguation, difficult language and visual context, and prediction speed for real-life autonomous-vehicle settings.These evaluation dimensions are motivated by the need to ground passenger commands before vehicle action.
2 Related Work
Related work covers object referral, human-robot command grounding, visual question answering, and referral datasets. Talk2Car differs by grounding free-form driving commands rather than conventional object descriptions.
- Object Referral: Object referral retrieves the correct image region from a language expression, commonly using region proposals followed by language-vision matching.Prior methods include attention, global-context, similarity-based, and modular-network approaches.
- Grounding in Human-Robot Interaction: Human-robot interaction research grounds commands in visual environments, including compositional graphical models, object-picking systems, and clarification questions.The paper excludes one computationally complex model from evaluation because of prediction-time cost.
- Visual Question Answering: Visual question answering combines image features with question encodings, while MAC uses multiple reasoning steps and is therefore included in the evaluation.The related-work discussion connects MAC’s reasoning structure to the object referral setting.
- Object Referral Datasets: Talk2Car differs from ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscapes-Ref, and CLEVR-Ref because it uses commands rather than descriptions in driving scenes.The prior datasets include both real-world and computer-generated images, with RefCOCOg containing longer expressions than RefCOCO and RefCOCO+.
3 Dataset
Talk2Car extends nuScenes with crowd-authored natural-language commands linked to selected objects in urban driving videos. The resulting dataset contains 11 959 commands and uses video-level, city-aware, and scene-condition-aware splitting, while omitting executable paths in the current version.
- Base Dataset: Talk2Car builds on nuScenes, which contains 1000 20-second videos from Boston and Singapore across weather and times of day.nuScenes provides approximately 1.4 million images, multiple sensors, and 3D bounding boxes for 23 object classes.
- Collection and Annotation: Workers watched each 20-second front-camera video, selected an interesting frame and pre-annotated object, then entered any command that defined a followable car path.Commands were linked to objects already annotated in nuScenes.
- Collection and Annotation: Two independent reviewers manually verified every command and corresponding bounding box, with bonuses encouraging diverse and meaningful commands.Five workers were hired per video and could enter multiple commands per frame.
- Dataset Statistics: 11 959 commands cover 850 nuScenes training videos, averaging 11.01 words per command and 14.07 commands per video.Commands are distributed 55.94% in Boston and 44.06% in Singapore.
- Scope: The current dataset version does not include the path the car should follow when executing each command.The authors state that these paths will be added in a later version.
- Dataset Splits: The train, validation, and test sets contain 70%, 10%, and 20% of samples, respectively, with videos kept within a single split.The split also preserves city and scene-condition distributions.
4 Datasets Comparisons
Talk2Car differs from prior object-referral datasets through realistic car-centric imagery, unconstrained natural-language commands, and multiple sensing modalities. Its smaller annotation count is paired with reviewed expressions and challenges involving spatial reasoning and language understanding.
- Talk2Car contains fewer natural-language expressions than prior datasets, but its expressions are high quality through a double-review system.
- Talk2Car uses realistic urban settings from car-centric viewpoints rather than curated pedestrian-view imagery.
- Talk2Car ranks third in average expression length and permits unrestricted command content, increasing linguistic complexity and diversity.
- The dataset combines RADAR, LIDAR, and video, modalities largely absent from prior object-referral datasets.
- Its free-language commands include irrelevant side information and require co-reference, entity, relational, attribute, and target-object understanding.
- Talk2Car commands can require spatial reasoning, including cases where a person's spatial description conflicts with the object's actual image location.
5 Application of the State-of-the-Art Models and their Evaluation
The paper evaluates seven models for referring-object detection on Talk2Car using IoU0.5, inference speed, and challenging test subsets. MAC performs best on nearly all tasks, while spatial information, region-proposal quality, object-category alignment, and command length substantially affect performance.
- Overall Evaluation: Seven models are evaluated on Talk2Car using IoU0.5, inference speed, parameter count, and increasingly challenging test subsets.Predictions are correct when the intersection-over-union between predicted and ground-truth regions exceeds 0.5.
- Overall Results: MAC performs best on nearly all tasks and significantly outperforms the similar STACK-NMN model.The comparison covers both the standard test split and challenging subsets.
- Region-Proposal Methods: SCRC often outperforms OSM, indicating that incorporating spatial information benefits region-proposal-based referral.SCRC augments local region representations with global image information and spatial configuration features.
- Overall Results: Simple baselines RS, BOBB, and RNM perform poorly, demonstrating the difficulty of object referral in Talk2Car’s realistic driving scenes.The challenging subsets include distant objects, unusually short or long commands, and scenes with multiple same-category objects.
- Results and Discussion: Object-category alignment is essential: RNM improves over random selection, while longer commands and greater object ambiguity reduce performance for most models.With ground-truth bounding boxes, RNM reaches 54% IoU, and the models struggle as same-category ambiguity increases.
- Inference Speed: Non-region-proposal systems are roughly twice as fast as region-proposal systems because they encode the image once instead of aligning every proposed region with the command.This speed advantage reflects a trade-off between proposal-level alignment and whole-image reasoning.
- Region Proposal Quality: Pretraining region proposals on all training-video keyframes raises OSM performance from 35.31 to 40.78% and SCRC performance from 38.70 to 41.15%.The gains show that proposal quality is important for region-proposal-based systems.
- Word Embeddings: Pre-trained GloVe embeddings have no effect or lower IoU, while ELMO and BERT provide only minor improvements for some models.The authors associate this with the need to discriminate closely related object words such as “car” and “truck.”
6 Conclusions and Future Work
Talk2Car provides a challenging first dataset for natural-language object referral in shared urban scenes. The paper identifies extensions involving 3D information, multiple objects, paths, and dialogue.
- Talk2Car contains natural-language commands referring to objects in visual urban scenes visible to both passengers and self-driving cars.
- Experiments with strong object-referral models produced promising results, while existing methods still need improvement for joint language-vision processing.
- Neglected 3D information could support richer spatial grounding and action execution in 3D.
- The current dataset permits referral to only one object and lacks car-path annotations and dialogues for ambiguous commands.
- Future versions are intended to add 3D-related annotations, paths, and dialogues, although the first version already offers a challenging benchmark.
A Reproducibility
The reproducibility appendix discloses parameters for the mentioned models and excludes two models that use only random strategies.
- The section reports model parameters for reproducibility purposes.
- RNM and RS are the only mentioned models omitted from the parameter disclosures.
- RNM and RS are omitted because they apply a random strategy.
A.1 OSM
The OSM appendix describes region proposal generation, visual and language encoders, optimization settings, and a condition for ignoring a loss term.
- The proposal detector is initialized from ImageNet-pretrained weights and trained with warm-up, batch size 32, and learning-rate reductions after 80,000 and 100,000 iterations.
- OSM uses 64 region proposals extracted by a single-shot detection model.
- OSM encodes local regions with ImageNet-pretrained ResNet-18 features and sentences with a one-layer bidirectional GRU of hidden size 512.
- OSM is optimized with stochastic gradient descent using learning rate 1e-3, momentum 0.9, and weight decay 1e-4.
- The loss term is ignored when no region proposal has mean intersection over union larger than 0.5.
A.2 SCRC
The SCRC appendix specifies separate visual and recurrent context encoders alongside the language model and reuses the earlier optimization scheme.
- SCRC generates 64 region proposals per image using the previously described single-shot detection model.
- Separate ImageNet-pretrained ResNet-18 models produce local and global visual features.
- SCRC uses a bidirectional GRU with 512 hidden units for language and unidirectional 512-unit GRUs for local and global recurrent context.
- The recurrent units are initialized randomly rather than using the original paper’s captioning-task pretraining.
- SCRC reuses the optimization scheme described in section A.1.
A.3 STACK-NMN
STACK-NMN resizes Talk2Car images to 512×512 and extracts 32×32×1024 feature maps using ResNet-101 truncated at its fourth channel.
- Images are resized from 1600×900 to 512×512 before ResNet-101 feature extraction.
- The ResNet-101 model is cut off at its fourth channel, producing a 32 × 32 × 1024 tensor per image.
- STACK-NMN sets its internal FeatH and FeatW parameters to 32.These parameters represent the feature-map height and width.
A.4 MAC
The MAC-related pipeline uses Talk2Car visual features and converts reasoning attention into a bounding-box prediction, while the BOBB procedure searches candidate boxes using training-set spatial statistics.
- MAC uses the pre-Softmax attention map from the last reasoning read unit to predict the bounding-box center cell and its offsets.A fully connected layer predicts the cell, while a convolutional layer predicts offsets relative to the center.
- The BOBB algorithm finds a training-derived bounding box using the spatial distribution of training-set objects.The resulting box is [0, 435, 445, 325] in [x1, y1, w, h] format.
- Figure 4 presents object-location heatmaps separately for the training, validation, and test sets.
- BOBB evaluates candidate boxes by counting training ground-truth boxes whose IoU exceeds a specified threshold, retaining candidates with larger counts.