PaperlayerView exact revision on arXiv

Source-linked AI summary

Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models

Xinpeng Ding, Jinahua Han, Hang Xu, Xiaodan Liang, Wei Zhang, Xiaomeng Li

arXiv:2401.00988v1cs.CV

TL;DR

Language-based driving research often covers limited tasks and omits temporal and multi-view information. The paper introduces NuInstruct and BEV-InMLLM, which integrates instruction-aware BEV features into MLLMs. BEV-InMLLM outperforms state-of-the-art methods by 9% on various tasks.

  • Problem

    Existing language-based driving research covers only subsets of autonomous-driving tasks and often omits temporal and multi-view information.

    Source: §1 · p. 2

  • Method

    The paper creates NuInstruct with SQL-generated instruction-response pairs and integrates instruction-aware BEV features into MLLMs through BEV-InMLLM.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    9%: BEV-InMLLM outperforms state-of-the-art methods on various tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Ablations find that MV-MLLM enhances multi-view tasks and BEV-InMLLM is vital for most tasks, emphasizing spatial information.

    Source: §1 · p. 3

    The current dataset lacks traffic-light information and tasks related to 3D object detection.

    Source: §6 · p. 9

Abstract

from arXiv · show

The rise of multimodal large language models (MLLMs) has spurred interest in language-based driving tasks. However, existing research typically focuses on limited tasks and often omits key multi-view and temporal information which is crucial for robust autonomous driving. To bridge these gaps, we introduce NuInstruct, a novel dataset with 91K multi-view video-QA pairs across 17 subtasks, where each task demands holistic information (e.g., temporal, multi-view, and spatial), significantly elevating the challenge level. To obtain NuInstruct, we propose a novel SQL-based method to generate instruction-response pairs automatically, which is inspired by the driving logical progression of humans. We further present BEV-InMLLM, an end-to-end method for efficiently deriving instruction-aware Bird's-Eye-View (BEV) features, language-aligned for large language models. BEV-InMLLM integrates multi-view, spatial awareness, and temporal semantics to enhance MLLMs' capabilities on NuInstruct tasks. Moreover, our proposed BEV injection module is a plug-and-play method for existing MLLMs. Our experiments on NuInstruct demonstrate that BEV-InMLLM significantly outperforms existing MLLMs, e.g. around 9% improvement on various tasks. We plan to release our NuInstruct for future research development.

1. Introduction

Existing language-based driving research covers only subsets of autonomous-driving tasks and often lacks temporal and multi-view information. The paper introduces NuInstruct and BEV-InMLLM to address these gaps, reporting improved benchmark performance.

1. Introduction

  • Existing benchmarks cover partial task sets, and datasets such as [11] [48] often use single-view images without temporal or multi-view information.The paper notes that safe decisions can require awareness beyond the front view, such as of an overtaking vehicle on the left.Source: §1 · p. 2
  • NuInstruct contains 91K multi-view video-QA pairs across 17 subtasks, which demand holistic information such as temporal, multi-view, and spatial details.Its SQL-based generation method is designed around the logical progression of driving tasks.Source: §1 · p. 2
  • The SQL-based generation process retrieves instruction responses from a scene-information database using task-specific SQLs designed around driving logic [16].The progression includes perception, prediction, risk assessment, and planning with reasoning, with human or GPT-4 verification to remove erroneous pairs.Source: §1 · p. 2
  • BEV-InMLLM integrates instruction-aware BEV features with MLLMs to combine temporal, multi-view, and spatial information for autonomous-driving tasks.The BEV injection module is presented as plug-and-play for existing MLLMs and more resource-efficient than training a BEV extractor from scratch with visual-language data.Source: §1 · pp. 2–3
  • 9%: BEV-InMLLM outperforms state-of-the-art methods on various tasks, while ablations find MV-MLLM enhances multi-view tasks and BEV-InMLLM is vital for most tasks.The authors attribute the latter finding to the importance of spatial information.Source: §1 · p. 3

2. Related Work

Prior language-driving work spans grounding, perception, risk description, action interpretation, and planning, but often addresses only part of driving or uses incomplete information. The paper positions BEV-InMLLM as a model combining temporal, multi-view, and spatial data.

2. Related Work

  • Many language-driving datasets and models cover only part of autonomous-driving tasks and use incomplete information; the proposed dataset targets tasks requiring temporal, multi-view, and spatial information.The cited work includes language grounding, temporal object detection and tracking, multi-view question answering, risk-object descriptions, and planning.Source: §2 · p. 3
  • The SQL-based data-generation design follows the logical flow of autonomous-driving tasks [16], retrieving responses from a scene-information database with task-specific SQLs.“Planning w/ R” denotes planning with reasoning.Source: Figure 2 · p. 3
  • DriveGPT4 is limited to single-view inputs, while Talk2BEV lacks temporal dynamics and an end-to-end framework; BEV-InMLLM is presented as combining temporal, multi-view, and spatial data.The paper frames this combination as supporting reliable decisions.Source: §2 · p. 3

3. NuInstruct

NuInstruct is built from NuScenes keyframes through SQL-generated driving instructions, then characterized by task and view statistics and evaluated with task-specific metrics.

3.1. Instruction Data Generation

  • SQL tasks follow perception → prediction → risk → planning, with sampled annotated keyframes feeding subtask functions and prompts that retrieve database responses before human or GPT-4 verification.For example, the status subtask derives future status from distances across previous, current, and future frames.Source: §3.1 · pp. 3–4

3.2. Data Statistics

  • 91,355 instruction-response pairs were generated from 11,850 keyframes sampled across 850 NuScenes videos, spanning four tasks and 17 subtasks.The tasks are Perception, Prediction, Risk, and Planning with Reasoning.Source: §3.2 · p. 4
  • Responses require multiple views; Figure 4 shows relatively even view distributions for perception and prediction, while risk and reasoning draw mainly on front and front-side views.The figure summarizes task proportions, view-dependent response counts, and task-level view percentages.Source: §3.2 · pp. 4–5, Figure 4 · p. 5

3.3. Evaluation Protocols

  • Different tasks use different evaluation metrics; MAE scores are extracted with a regular expression.Table 2 summarizes the metrics used for the tasks.Source: §3.3 · p. 5, Table 2 · p. 5
  • Videos are split into training, validation, and test sets at 7.5:1.5:1.5, with the best validation model reported on the test set.Source: §3.3 · p. 5

4. Method

The method extends MLLMs to process multi-view videos and language instructions, then injects BEV representations for autonomous-driving understanding. Its architecture combines multi-view visual features with instruction-aware BEV tokens before passing the enhanced representation to the LLM.

4. Method

  • Section 4 presents the framework, first defining its inputs and outputs, then introducing MV-MLLM and BEV-InMLLM for multi-view video processing and panorama understanding.Figure 5 depicts the base multi-view MLLM and the BEV injection module.Source: §4 · p. 5, Figure 5 · p. 6

4.1. Preliminaries

  • The model takes multi-view videos and a language instruction as inputs, and generates a corresponding language response.Each video is a sequence of frames from one camera view; the instruction and response are represented as language tokens.Source: §4.1 · p. 5

4.2. Multi-View MLLM

  • MV-MLLM processes each view with a vision encoder and connection module, then uses a multi-view Q-Former to aggregate visual semantics across views.Only the multi-view Q-Former is trainable; its output is linearly projected and fed to the LLM.Source: §4.2 · p. 6

4.3. BEV-Injected MLLM

  • BEV-InMLLM uses an instruction-aware BEV Q-Former to select BEV information relevant to the instruction, then fuses those tokens with multi-view features through cross-attention.The enhanced features contain temporal multi-view cues and spatial-aware BEV information; the BEV feature extractor stays frozen while the Q-Former and injection module are trainable.Source: §4.3 · p. 6

5. Experiments

Across NuInstruct tasks, the proposed modules improve evaluation metrics; temporal and multi-view information particularly benefits risk and planning, while BEV helps distance- and position-sensitive tasks.

5. Experiments

  • The three base MLLMs are BLIP-2, MiniGPT-4, and Video-LLaMA; image-centric models use an ST-Adapter, and pretrained weights are frozen during training.Training uses three uniformly sampled frames per video and runs for 20 epochs.Source: §5 · p. 7, §5.1 · p. 7

5.1. State-of-the-art Comparison

  • 5% on risk and 6% on planning are the reported gains from adding temporal and multi-view information; BEV benefits distance- and position-sensitive tasks.Source: §5.1 · p. 7

5.2. Ablation Study

  • The ablation study groups subtasks by temporal, multi-view, spatial, and holistic information needs, with holistic tasks requiring all three information types.The full model is compared with versions lacking video input, multi-view input, or BEV, and with the MiniGPT-4 baseline.Source: Table 4 · p. 7, Table 5 · p. 7, §5.2 · p. 8

5.2.1 Effect of different proposed modules

  • Removing video or multi-view input degrades tasks dependent on those cues, while BEV information is important for most autonomous-driving tasks.Source: Table 5 · p. 7, §5.2.1 · p. 8

5.2.2 Analysis of BEV Injection Module

  • Instruction tokens improve spatial and holistic performance by 1.4; increasing BEV queries from 16 to 32 adds 0.6 on spatial performance, while 64 brings only a small further improvement.A stronger BEV extractor performs better; RGB-and-lidar BEVFusion performs best, but the study uses RGB images for efficiency. Injecting instruction-aware BEV features into multi-view visual semantics yields a 0.7% holistic-task improvement over injecting them into the multi-view Q-Former output.Source: §5.2.2 · p. 8, §5.1 · p. 8

6. Conclusion

The paper presents NuInstruct and BEV-InMLLM as contributions to language-based autonomous driving, and reports empirical efficacy on NuInstruct. The dataset currently lacks traffic-light information and 3D object-detection tasks.

6. Conclusion

  • NuInstruct contains 91K multi-view video-instruction-response pairs across 17 subtasks, created using a SQL-based method.Source: §6 · p. 8
  • BEV-InMLLM integrates instruction-aware BEV features into MLLMs to enhance processing of temporal, multi-view, and spatial details.The paper describes the BEV injection module as a plug-and-play enhancement and reports empirical efficacy on NuInstruct.Source: §6 · p. 8
  • NuInstruct lacks traffic-light information and tasks related to 3D object detection, which the authors plan to address in future work.Source: §6 · p. 9

Supplementary Material

Figure 6 illustrates the scene database and task SQLs, distinguishing table mappings from task derivations.

Supplementary Material

  • Figure 6(a) depicts the scene database, while Figure 6(b) depicts task SQLs.Source: Figure 6 · p. 12
  • Red dashed lines indicate mappings between different database tables.Source: Figure 6 · p. 12
  • Blue lines indicate derivations among task SQLs.Source: Figure 6 · p. 12

7. More details about NuInstruct

The supplementary material details NuInstruct's linked scene database, task SQL sets, and evaluation metrics.

7.1. Scene Database

  • The scene database comprises scene, frame, ego-information, and instance-information tables linked through one-to-many mappings.Scene and frame records connect to frame-level ego and instance details; the tables include pose, velocity, road, camera, and object information.Source: §7.1 · p. 12, Table 7 · p. 13

7.2. Task SQLs

  • Four SQL sets—perception, prediction, risk, and planning with reasoning—generate instruction-response pairs, with higher-level sets inheriting from lower-level task results.Prediction SQLs build on perception; risk SQLs use prediction and perception; planning with reasoning inherits from prediction and risk.Source: §7.2 · pp. 12–13

7.3. Evaluation Details

  • Evaluation uses MAE, accuracy, MAP, and BLEU, selected according to each task's output and purpose.Closest, Status, and Same Road use accuracy; risk tasks use MAP, and planning with reasoning is evaluated with BLEU as a caption-like task.Source: §7.3 · p. 13

8. More Qualitative Examples

The qualitative examples span all 17 subtasks, illustrating how the model handles perception, prediction, risk, and planning tasks. The examples include both task outputs and SQL procedures for deriving them.

Overview

  • The examples cover all 17 subtasks across perception, prediction, risk, and planning with reasoning.The visualizations are grouped across Figures 7–12.Source: §8 · p. 13

Perception

  • Perception examples count object categories by camera view and identify the closest objects in each view.The instance-count procedure returns counts by category and view; the closest-object procedure selects minimum-distance instances.Source: Figure 7 · p. 14, §8 · p. 17
  • Other perception examples report object speed and status and compare an instance’s road information with the ego car’s.The same-road procedure returns yes when their road information matches, and no otherwise.Source: Figure 7 · p. 14, §8 · p. 17

Prediction

  • Prediction examples derive ego motion from consecutive poses and rotations, and use frame-to-frame motion and velocity changes for ego and other objects.The status procedures combine motion with velocity change.Source: Figure 8 · p. 14, §8 · pp. 17–18

Risk

  • Risk examples cover oncoming objects, crossings, and braking, using motion, distance, and speed conditions to select instances.The examples visualize these three risk subtasks and provide SQL procedures for identifying relevant objects.Source: Figure 9 · p. 15, §8 · pp. 18–19
  • Additional risk examples address approaching and overtaking objects and lane changes, with procedures using motion, distance, speed, or same-road status.The lane-changing procedure checks whether an object changes from a different road to the ego car’s road.Source: Figure 10 · p. 15, Figure 11 · p. 16, §8 · pp. 18–19

Planning with Reasoning

  • The planning-with-reasoning example combines ego motion and status with risk outputs across the listed risk categories to produce a next plan.The procedure returns the risk dictionary together with motion and status information.Source: Figure 12 · p. 16, §8 · p. 19
Loading 2401.00988v1…