PaperlayerView exact revision on arXiv

Source-linked AI summary

Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving

Tushar Choudhary, Vikrant Dewangan, Shivam Chandhok, Shubham Priyadarshan, Anushka Jain, Arun K. Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, K. Madhava Krishna

arXiv:2310.02251v2cs.CVcs.RO

TL;DR

Talk2BEV addresses the limits of task-specific autonomous-driving perception by combining general-purpose LVLMs with BEV maps. It builds language-enhanced maps for broad scene reasoning without task-specific training and introduces a benchmark for evaluation across autonomous-driving questions.

  • Problem

    Existing autonomous-driving perception systems rely on predefined object categories and task-specific models, motivating integration of general-purpose language capabilities with BEV representations.

    Source: §I · p. 1

  • Method

    Talk2BEV augments BEV maps with aligned image-language features from pretrained LVLMs and uses them as context for LLM-based scene queries.

    Source: §I · p. 1, §III · p. 2

  • Results

    Talk2BEV supports visual and spatial reasoning, traffic-interaction prediction, and decision-making across a broad class of autonomous-driving scenarios.

    Source: §I · p. 2, §V · p. 5, §VI · p. 7

  • Takeaways & Limitations

    The zero-shot framework and Talk2BEV-Bench support flexible LVLM use and broader evaluation of language-based autonomous-driving scene understanding.

    Source: §I · p. 2, §VI · p. 7

Abstract

from arXiv · show

Talk2BEV is a large vision-language model (LVLM) interface for bird's-eye view (BEV) maps in autonomous driving contexts. While existing perception systems for autonomous driving scenarios have largely focused on a pre-defined (closed) set of object categories and driving scenarios, Talk2BEV blends recent advances in general-purpose language and vision models with BEV-structured map representations, eliminating the need for task-specific models. This enables a single system to cater to a variety of autonomous driving tasks encompassing visual and spatial reasoning, predicting the intents of traffic actors, and decision-making based on visual cues. We extensively evaluate Talk2BEV on a large number of scene understanding tasks that rely on both the ability to interpret free-form natural language queries, and in grounding these queries to the visual context embedded into the language-enhanced BEV map. To enable further research in LVLMs for autonomous driving scenarios, we develop and release Talk2BEV-Bench, a benchmark encompassing 1000 human-annotated BEV scenarios, with more than 20,000 questions and ground-truth responses from the NuScenes dataset.

I. INTRODUCTION

Talk2BEV addresses the challenge of integrating general-purpose language and vision-language capabilities with BEV representations for broad autonomous-driving scene understanding. It introduces a language-enhanced BEV framework and Talk2BEV-Bench for evaluating object- and scene-level reasoning.

Motivation

  • Existing autonomous-driving systems use task-specific models, while Talk2BEV asks how LLM capabilities can be integrated with traditional BEV scene representations.The motivating scope includes contextualizing object interactions and unforeseen scenarios, beyond recognizing fixed object categories.Source: §I · p. 1

Talk2BEV

  • Talk2BEV augments BEV maps with aligned image-language features so LVLMs can support visual reasoning, spatial understanding, and decision-making.The framework uses the enhanced map as visual context for answering scene questions and reasoning about potential future scenarios.Source: §I · pp. 1–2
  • The framework requires no BEV-specific or vision-language training or finetuning, supporting flexible deployment across domains, tasks, and newer pretrained models.Its contribution is explicitly based on existing pretrained LLMs and LVLMs.Source: §I · p. 2

Contributions

  • Talk2BEV is presented as the first system to augment BEV maps with language for general-purpose visuolinguistic reasoning in autonomous-driving scenarios.The contribution targets a broad class of road-scene queries rather than a single predefined task.Source: §I · p. 2

Talk2BEV-Bench

  • Talk2BEV-Bench evaluates LVLMs for autonomous driving with human-annotated ground truth spanning object attributes, semantics, visual reasoning, spatial understanding, and decision-making.The benchmark is designed to support objective evaluation and further research in LVLM-based autonomous driving.Source: §I · p. 2

II. RELATED WORK

Related work spans LVLM benchmarking, 3D vision-language scene understanding, and language-grounded autonomous driving. Talk2BEV differs by combining pretrained LLMs and LVLMs with BEV maps without training or finetuning, while introducing a broader benchmark.

Large Vision Language Models

  • Prior LVLM benchmarks use question-response curation or ranking protocols, and Talk2BEV adopts the objective ranking methodology of SEEDBench.The related work identifies objectivity as a challenge in benchmark construction.Source: §II · p. 2

3D Vision-Language Models

  • 3D vision-language research covers object localization, scene captioning, visual question answering, and point-cloud or multi-view integration.Prior autonomous-driving systems include language grounding, temporal referring-object tasks, and point-cloud language encoders.Source: §II · p. 2

Vision-Language Models for Autonomous Driving

  • Talk2BEV offers zero-shot scene comprehension by blending state-of-the-art LLMs and LVLMs with BEV maps, while requiring no training or finetuning.It also introduces Talk2BEV-Bench for broader BEV-based autonomous-driving scene-understanding evaluation.Source: §II · p. 2

III. TALK2BEV

Talk2BEV enhances a BEV map with pretrained vision-language features and uses an LLM to interpret queries against that representation. Its pipeline estimates the map, extracts object metadata, and generates responses to user questions.

BEV Representation

  • Talk2BEV represents the scene as a top-view, multi-channel BEV grid encoding semantic information, with the ego-vehicle at the map origin.In this work, the vehicle class supplies LVLM features, while the road class is used for visualization.Source: §III · p. 2
  • The framework obtains BEVs from multi-view RGB images and LiDAR point clouds using off-the-shelf BEV prediction approaches.The BEV is centered on the ego-vehicle, which serves as the coordinate origin.Source: §III · pp. 2–3

Pipeline

  • The three-phase pipeline estimates a BEV from onboard sensors, creates aligned image-language object features and metadata, then prompts an LLM to answer user queries.The resulting language-enhanced map combines object data with geometric information from the BEV.Source: §III · p. 3

A. Language Enhanced Maps

Language-enhanced maps associate each BEV object with geometric measurements and textual descriptions derived from aligned image regions. Segmentation and LVLM captioning provide object- and scene-level metadata that are stored with the map.

Map Representation

  • For each object, the map records BEV displacement, area, object description, and background description.Descriptions can include vehicle type, color, utility, indicator status, and displayed text.Source: A. Language Enhanced Maps · p. 3

BEV-Image Correspondence

  • The pipeline localizes BEV objects in camera views by selecting nearby LiDAR points and projecting them into the camera frame with an inverse homography.These projections identify the image region corresponding to each BEV object.Source: A. Language Enhanced Maps · p. 3

Language-enhancement

  • Point-prompted segmentation produces tight bounding boxes whose crops are passed through LVLM visual encoders to obtain image-language features and descriptions.The generated metadata are added to the BEV as text entries, covering both object-level and scene-level details.Source: A. Language Enhanced Maps · p. 3

B. Response Generation

Talk2BEV supports free-form, multiple-choice, and spatial reasoning queries, using structured responses and spatial-operator calls for geometric questions.

  • Talk2BEV handles free-form text, multiple-choice, and spatial reasoning queries.Free-form and spatial queries emulate the end-user interface, while MCQs support objective evaluation.Source: B. Response Generation · p. 3
  • The LLM returns inferred query, achievability, spatial-function usage, and an explanation in JSON format.This organizes outputs into key-value pairs and exposes intermediate reasoning steps.Source: B. Response Generation · p. 4
  • Spatial queries trigger primitive-operator API calls for locations, distances, and orientations rather than direct answer generation.The operators enable joint reasoning over objects that may not be visible in the same camera.Source: B. Response Generation · p. 4

C. Implementation Details

Talk2BEV uses fixed-resolution BEV generation and multiple LVLMs, while Talk2BEV-Bench provides human-verified NuScenes question-answer data for diverse evaluation tasks.

C. Implementation Details

  • Talk2BEV generates 200 × 200 BEV grids at 0.5m resolution using Lift-Splat-Shoot.Ground-truth evaluation maps use the same grid dimensions and resolution.Source: C. Implementation Details · p. 4
  • BLIP-2, MiniGPT-4, and InstructBLIP-2 provide vision-language features for object descriptions.BLIP-2 uses Flan5XXL, while the other two use Vicuna-13b language decoders.Source: C. Implementation Details · p. 4
  • Spatial operators are defined over BEV objects or the ego-vehicle to support geometric reasoning.Object-indexed operators use object IDs, while operators without IDs operate on the ego-vehicle.Source: Table I · p. 4

IV. THE TALK2BEV-BENCH BENCHMARK

  • Talk2BEV-Bench contains 1,000 NuScenes scenes and more than 20,000 human-verified question-answer pairs.Questions cover object attributes, counting, visual reasoning, decision making, and spatial reasoning.Source: §IV · p. 4
  • The benchmark pipeline extracts ground-truth BEV maps, refines captions with human annotators, and uses GPT-4 to generate and validate MCQs.Each question has multiple answer choices with one correct answer.Source: §IV · p. 4

A. Ground-truth language-enhanced maps

The benchmark constructs language-enhanced ground-truth maps from object and scene captions, then evaluates four visual and spatial understanding dimensions with category-specific metrics.

A. Ground-truth language-enhanced maps

  • Object captions combine dense foreground descriptions, recognized text, and background scene context verified by human annotators.The resulting information includes fine-grained object details, signs, barriers, weather, time of day, and other scene elements.Source: A. Ground-truth language-enhanced maps · p. 5

B. Question Generation and Evaluation Metrics

  • The evaluation covers instance attributes, instance counting, visual reasoning, and spatial reasoning.These categories address object properties, query-based counting, general visual understanding, and location, distance, or orientation.Source: B. Question Generation and Evaluation… · p. 5
  • Each scene receives 20 questions, with five generated questions per evaluation dimension.The benchmark reports accuracy for multiple-choice categories, Jaccard index for set-valued spatial outputs, and distance errors for distance outputs.Source: B. Question Generation and Evaluation… · p. 5

V. RESULTS

Talk2BEV supports broad visual and spatial understanding, benefits from explicit spatial operators, and uses zero-shot LVLM substitution; qualitative results show intent-aware caution in a parking scenario.

V. RESULTS

  • Talk2BEV addresses broad visual and spatial understanding tasks through language-enhanced maps.The system also supports predicting risky driving maneuvers and recourse in challenging NuScenes scenarios.Source: §V · p. 5
  • Explicit access to primitive spatial operators significantly improves spatial reasoning performance.The operators let the model invoke geometric computations instead of producing spatial answers directly.Source: §V · p. 5

A. Quantitative Results

  • Talk2BEV uses a vehicle’s parking light and position to infer that advancing is unsafe while it reverses into a parking spot.The prediction aligns with the vehicle’s future activity from t = 0 to t = 3s.Source: Figure 7 · p. 6
  • MiniGPT-4 achieves the best average performance across different question types, while its L(O) map achieves the best accuracy for instance counting.Source: A. Quantitative Results · p. 6
  • BEV prediction errors have only a minor 3% impact on performance.Instance attributes and visual reasoning are more sensitive to LVLM caption quality than other question categories.Source: A. Quantitative Results · p. 6

B. Qualitative Results

Talk2BEV combines qualitative scene reasoning with explicit spatial operators to support interactive autonomous-driving queries. Spatial operators substantially improve quantitative spatial-reasoning performance over direct LLM reasoning.

B. Qualitative Results

  • Talk2BEV uses a free-form dialogue to infer that a vehicle reversing into a parking spot may obstruct the ego-vehicle’s planned 20 m advance.The prediction aligns with the vehicle’s observed future activity from t = 0 to t = 3s.Source: B. Qualitative Results · p. 6
  • The qualitative evaluation also compares multiple LVLMs on multiple-choice questions from Talk2BEV-Bench.The comparison concerns four questions involving a police car and a construction vehicle in an interchange scene.Source: B. Qualitative Results · p. 6

C. Impact of Spatial Operators.

  • 58% higher Jaccard index and 0.09 m lower distance error result from adding spatial operators to Talk2BEV over direct GPT-4 reasoning.Without spatial operators, Talk2BEV still outperforms Random but often produces large spatial-reasoning errors.Source: C. Impact of Spatial Operators · p. 6, Table III · p. 6

D. Performance across Object Categories

Performance varies across object categories and language-enhanced-map configurations. Smaller two-wheelers are harder to back-project accurately, whereas larger vehicles generally perform better.

D. Performance across Object Categories

  • Two-wheelers, including bicycles and motorcycles, consistently perform worse because their smaller BEV segmentations magnify errors from minor position inconsistencies.The category-wise evaluation spans BLIP-2, InstructBLIP-2, MiniGPT-4, and LSS and GT BEV variants.Source: D. Performance across Object Categories · p. 6, Table IV · p. 7
  • Larger vehicles, including trucks and construction vehicles, consistently outperform cars in most cases because larger BEV segmentations enable more accurate back projections.The qualitative figure includes a construction vehicle and compares LVLM-generated captions across object-focused questions.Source: D. Performance across Object Categories · pp. 6–7, Figure 8 · p. 7
  • Talk2BEV evaluates category-wise multiple-choice performance across different LVLM-built language-enhanced maps and BEV variants.Table IV reports results for vehicle categories using qmcq queries.Source: Table IV · p. 7

VI. CONCLUSION

Talk2BEV provides a language interface to BEV maps for diverse autonomous-driving tasks and introduces a benchmark for evaluating LVLMs in these settings. The authors also emphasize safety and alignment research before deployment in safety-critical systems.

VI. CONCLUSION

  • Talk2BEV supports visual and spatial reasoning, prediction of unsafe traffic interactions, and plotting recourse across autonomous-driving tasks.The system draws on recent LLM and LVLM advances to provide this broad task interface.Source: §VI · p. 7
  • Talk2BEV-Bench evaluates subsequent LVLM research for autonomous-driving applications, while deployment in safety-critical stacks still requires safety and alignment research.The conclusion presents benchmarking and pre-deployment safety research as complementary priorities.Source: §VI · p. 7
Loading 2310.02251v2…