PaperlayerView exact revision on arXiv

Source-linked AI summary

Dolphins: Multimodal Language Model for Driving

Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, Chaowei Xiao

arXiv:2312.00438v1cs.CV

TL;DR

Autonomous driving systems need stronger understanding and responsiveness in complex real-world scenarios. Dolphins addresses this gap with a vision-language conversational driving assistant trained through Grounded Chain of Thought and driving-specific instruction tuning, demonstrating broad AV-task understanding alongside rapid adaptation and error recovery. Its real-world deployment remains constrained by computational overhead and power consumption.

  • Problem

    Autonomous driving models are limited in handling complex open-world scenarios, temporal multimodal inputs, in-context learning, and human-like adaptation.

  • Method

    Dolphins builds on OpenFlamingo, grounds reasoning with Grounded Chain of Thought, and applies driving-specific instruction tuning using specialized datasets and AV tasks.

  • Results

    Dolphins demonstrates holistic driving-scenario understanding across perception, prediction, and planning, together with in-context adaptation, reflection, and error recovery.

  • Takeaways & Limitations

    Dolphins functions as a conversational driving assistant with interpretable responses, broad AV-task applicability, rapid adaptation, and error recovery.

  • Takeaways & Limitations

    Average inference takes 1.34 seconds on an NVIDIA A100, raising feasibility concerns for high-frame-rate edge deployment and vehicle power consumption.

Abstract

from arXiv · show

The quest for fully autonomous vehicles (AVs) capable of navigating complex real-world scenarios with human-like understanding and responsiveness. In this paper, we introduce Dolphins, a novel vision-language model architected to imbibe human-like abilities as a conversational driving assistant. Dolphins is adept at processing multimodal inputs comprising video (or image) data, text instructions, and historical control signals to generate informed outputs corresponding to the provided instructions. Building upon the open-sourced pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's reasoning capabilities through an innovative Grounded Chain of Thought (GCoT) process. Then we tailored Dolphins to the driving domain by constructing driving-specific instruction data and conducting instruction tuning. Through the utilization of the BDD-X dataset, we designed and consolidated four distinct AV tasks into Dolphins to foster a holistic understanding of intricate driving scenarios. As a result, the distinctive features of Dolphins are characterized into two dimensions: (1) the ability to provide a comprehensive understanding of complex and long-tailed open-world driving scenarios and solve a spectrum of AV tasks, and (2) the emergence of human-like capabilities including gradient-free instant adaptation via in-context learning and error recovery via reflection.

1 Introduction

Dolphins addresses limitations of modular and end-to-end autonomous driving systems by combining multimodal reasoning with a conversational driving assistant. Built with Grounded Chain of Thought and driving-specific instruction tuning, it targets holistic scenario understanding, broad AV tasks, rapid adaptation, and error recovery.

  • Existing autonomous driving systems struggle to integrate perception, prediction, planning, and control across varied situations, while end-to-end designs lack interpretability.
  • Long-tail open-world scenarios, rapid adaptation, and real-time error recovery remain difficult for conventional autonomous driving systems.
  • Dolphins is a vision-language conversational driving assistant designed to reduce the gap between existing autonomous driving systems and human-like driving.
  • Grounded Chain of Thought first develops fine-grained reasoning, after which driving-specific instruction tuning aligns Dolphins with autonomous-vehicle tasks.
  • Dolphins demonstrates broad capabilities spanning scene understanding, prediction and planning, instant learning and adaptation, reflection, and error recovery.

2 Related Work

Related work spans language-model-based driving agents, large vision-language models, and multimodal in-context learning. Dolphins differs from language-only driving systems by combining language reasoning and planning with visual-feature understanding.

  • Autonomous Driving with LLMs: Recent driving agents use large language models for perception, reasoning, planning, motion planning, or object tracking.
  • Autonomous Driving with LLMs: Language-only driving approaches lack rich visual features, whereas Dolphins combines language-model reasoning and planning with diverse visual-feature understanding.
  • Large Vision-Language Models: Large vision-language model research has advanced through language backbones, visual instruction tuning, and models such as Flamingo, BLIP-2, LLaVA, and MiniGPT-4.
  • Multimodal In-context Learning: Flamingo pioneered multimodal in-context learning, while later work used text-image interleaved instruction datasets with in-context exemplars.

3 Method

Dolphins grounds fine-grained reasoning in general visual data before adapting it to autonomous-driving scenarios through specialized tasks and instruction tuning. Its dataset combines grounded reasoning, four driving tasks, detailed conversations, and retrieved in-context examples.

  • 3.1 GCoT Instruction tuning: Limited AV-specific labeled data motivates grounding fine-grained multimodal reasoning in general image datasets before transferring it to driving.The paper identifies scarce driving datasets and difficult spatial annotation as barriers to directly learning grounded chain-of-thought capabilities in AV contexts.
  • 3.1 GCoT Instruction tuning: ChatGPT-generated GCoT responses combine image descriptions, queried-object locations, conditional reasoning, and a final answer statement.This three-step pipeline grounds reasoning in visual content and spatial information before appending “So the answer is {answer}”.
  • 3.1 GCoT Instruction tuning: The method transfers grounded reasoning to autonomous driving by fine-tuning on limited AV-specific data containing driving scenarios, road conditions, and agent interactions.The transfer adapts general reasoning skills to specialized AV requirements rather than training only on generic visual data.
  • 3.2 Devised Instruction Tasks for Autonomous Driving: The driving dataset defines four tasks spanning behavior understanding, behavior reasoning, control-signal prediction, and detailed conversation.The first tasks use BDD-X annotations, while detailed conversation targets longer, human-preferred responses and instruction generalization.
  • 3.2 Devised Instruction Tasks for Autonomous Driving: The proposed dataset contains 32k video-instruction-answer triplets, including 11k detailed-conversation examples and 21k examples from the other three tasks.The dataset uses BDD-X video segments and labels, with ChatGPT generating the detailed-conversation subset.
  • 3.3 Multi-modal In-context Instruction Tuning in Autonomous Driving: Dolphins uses in-context instruction tuning with OpenFlamingo and retrieves examples by similarity to each video-text sample.The retrieval process selects behaviorally similar examples, with k nearest samples searched in text-encoded and image-encoded latent spaces.

4 Training

Dolphins extends OpenFlamingo for multimodal driving inputs by adding temporal video representations and task-level instruction-tuning conventions. Training uses image-grounded CoT data before autonomous-driving instruction tuning.

  • 4.1 Model Architecture: Dolphins combines CLIP visual encoding, a perceiver resampler, and a language model with gated cross-attention for image-text interaction.Because OpenFlamingo lacks video support, the architecture adds learned latent vectors to preserve global temporal features.
  • 4.1 Model Architecture: Table 1 uses OpenFlamingo tokens plus an <answer> token, computes loss only on green sequences or tokens, and trains with cross-entropy loss.The added <answer> token helps intercept the model’s target answer output.
  • 4.2 Instruction Tuning: The instruction-tuning format adds task-level instructions so the model can interpret the broader context shared by autonomous-driving video-instruction pairs.The data structure follows an Otter-inspired format and is summarized in Table 1.
  • 4.2 Instruction Tuning: Dolphins removes general video-text alignment and instead begins with image instruction-following data containing grounded CoT templates before visual instruction tuning.This training design targets fine-grained understanding and reasoning abilities.
  • 4.3 Training Details: Training uses DeepSpeed and AdamW with β1 = 0.9, β2 = 0.999, weight decay 0.01, four NVIDIA A100 GPUs, batch size 128, and learning rate 2 × 10^-5.The maximum sequence length is 1024 and BF16 precision is used for training and inference.

5 Demonstration

Dolphins demonstrates holistic driving-scene understanding across perception, prediction, and planning, alongside human-like adaptation, conversation, and reflection. Its demonstrations show broad scenario interpretation and interactive capabilities in zero- and few-shot settings.

  • 5.1 Holistic Understanding and Reasoning: Dolphins organizes AV understanding around attributed road agents, operational design domains, ego behavior, predicted agent behavior, and ego future plans.These subtasks extend perception, prediction, and planning with open-vocabulary semantics, environmental context, behavioral reasoning, and contingency planning.
  • 5.1.1 Perception: Dolphins captures road-agent attributes, weather and illumination, traffic conditions, road-agent behavior, and ego-agent behavior from driving-related visual content.The demonstrations report semantic attributes, operational design domains, location and traffic status, agent behaviors, and ego behavior inferred from egocentric video.
  • 5.1.2 Prediction and Planning: Dolphins predicts other vehicles’ future behaviors and generates explainable ego-car plans, including cautious actions under contingent agent behavior.Because relevant instructions were limited during training, the authors recommend in-context learning, ongoing dialogue, and control signals for prediction and planning.
  • 5.2.1 Rapid Learning and Adaptation: Dolphins rapidly adapts to unforeseen driving scenarios from sparse in-context examples and responds to unseen instruction styles such as “What if” and “What are you doing”.The demonstrations include learning common-sense knowledge from examples and formulating predictions and actions without extensive task-specific pretraining.
  • 5.2.2 Interactive Conversation: Dolphins supports multi-turn driving conversations that maintain context and provide relevant responses about hazards, ego planning, and scene reasoning.The reported interactions assess nuanced language comprehension, contextual continuity, and informative responses for real-world driving exchanges.
  • 5.2.3 Reflection and Error Recovering: Dolphins reflects on feedback to identify errors, revise reasoning, and recover from suboptimal driving responses.Examples include correcting a predicted right-turn behavior from tail-light signals and revising an unnecessary yielding decision in a narrow street.

6 Conclusion and Future Directions

Dolphins demonstrates holistic understanding and human-like reasoning across complex driving scenarios, supporting perception, prediction, planning, conversation, and rapid adaptation. Real-world deployment remains constrained by computational overhead and power consumption.

  • 6 Conclusion and Future Directions: Dolphins demonstrates holistic understanding and human-like reasoning in complex driving scenarios, addressing a wide spectrum of autonomous-driving tasks.The paper highlights multimodal inputs, Grounded Chain of Thought, interpretability, and rapid adaptation as central capabilities.
  • 6 Conclusion and Future Directions: 1.34 seconds average inference time on an NVIDIA A100 and vehicle power consumption indicate challenges for high-frame-rate edge deployment.The authors identify customized and distilled models as a promising direction for improving deployment feasibility.
  • 6 Conclusion and Future Directions: Dolphins identifies environments, traffic conditions, road-agent behavior, and ego-vehicle behavior across varied weather, lighting, and roadway scenarios.Examples include tunnels, snowy streets, rainy intersections, busy roads, and following or passing vehicles.
  • 6 Conclusion and Future Directions: Dolphins predicts other vehicles’ trajectories and behaviors, including overlapping paths, stationary vehicles, lane changes, acceleration, and deceleration.The examples cover snowy streets, U-turns, highways, and intersections.
  • 6 Conclusion and Future Directions: Dolphins uses control signals to predict the ego car’s next-second speed and turn angle and to formulate short-horizon driving plans.Planning examples include changing lanes around a stopped vehicle and stopping at a red light.
  • 6 Conclusion and Future Directions: Through in-context examples, Dolphins adapts to unseen instructions, answers What If questions, learns common sense, and supports interactive driving conversation.The demonstrations include driver-role behavior, future ego-car behavior, weather, traffic lights, and pedestrian attention.

A Data

The paper constructs instruction-following data to transfer general multimodal reasoning into autonomous driving. It combines grounded reasoning responses for images with BDD-X-based driving video instructions and retrieved in-context examples.

  • A Data: Four VQA datasets—VQAv2, OK-VQA, GQA, and TDIUC—are used with ChatGPT to generate grounded Chain of Thought responses for image instruction following.The collected data aims to support fine-grained multimodal understanding and reasoning.
  • A Data: A BDD-X-based autonomous-driving instruction dataset transfers scene understanding and reasoning from images to driving videos.The dataset targets autonomous-driving-related instruction following.
  • A Data: Retrieved in-context examples are used to create few-shot training templates that enhance in-context learning for driving instructions.The retrieval method supplies examples during dataset construction.

B Prompts

The prompts operationalize Grounded Chain of Thought generation by asking ChatGPT to connect image descriptions, object locations, questions, and answers. An example illustrates the resulting three-step reasoning format.

  • B Prompts: Table 2 presents a prompt for generating a three-step grounded Chain of Thought process for VQA tasks.The process is intended to enhance fine-grained multimodal understanding and reasoning in autonomous driving.
  • B Prompts: The prompt asks for an image description, identification of the question object with a bounding box, and reasoning when required.The system message specifies normalized bounding-box coordinates as (x1, y1, x2, y2).
  • B Prompts: The example identifies a surfboard as the queried sporting equipment and supplies its invention date as the answer.Its numbered output separates scene description, object grounding, and answer reasoning.
Loading 2312.00438v1…