Source-linked AI summary
Distilling Multi-modal Large Language Models for Autonomous Driving
Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M. Patel, Fatih Porikli
TL;DR
Autonomous-driving planners need robust performance in rare scenarios without the computational cost of LLM inference. DiMA jointly trains an MLLM and vision-based planner with structured representations and surrogate tasks, then distills the MLLM knowledge into efficient planning. Training yields lower trajectory error and collision rate, including strong gains in long-tail scenarios, while achieving state-of-the-art nuScenes performance.
Problem
LLM-based planners can improve generalization to rare scenarios but impose substantial test-time computation, motivating efficient use of their knowledge.
Method
DiMA jointly trains a vision-based planner and MLLM with a shared structured scene encoder, distillation, and surrogate tasks including masked token reconstruction, future token prediction, and scene editing.
Results
37% lower L2 trajectory error, 80% lower collision rate, and 44% lower trajectory error in long-tail scenarios are reported for DiMA training.
Takeaways & Limitations
The MLLM can be removed at inference, allowing the resulting planner to retain vision-based efficiency while using distilled language-model knowledge.
Takeaways & Limitations
Some comparisons use reported results because published code was unavailable.
Abstract
from arXiv · showhide
Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.
1. Introduction
End-to-end autonomous driving systems struggle with rare long-tail scenarios, while LLM-based planners add test-time computation. DiMA distills MLLM knowledge into an efficient vision-based planner through joint training and surrogate tasks.
- End-to-end systems struggle with long-tail navigation because they rely on task-specific, limited datasets.
- LLM-based planners can generalize using world knowledge but require substantial test-time computation.
- DiMA jointly trains a vision-based planner and MLLM using structured scene representations, planning, visual question-answering, and surrogate tasks.
- The MLLM can be discarded during planning inference, preserving the efficiency of the vision-based planner.
- 37% lower L2 trajectory error and 80% lower collision rate are reported for the vision-based planner, with a 44% trajectory-error reduction in long-tail scenarios.
2. Related Work
Prior autonomous-driving research uses modular, end-to-end, and LLM-based systems, but DiMA focuses on structured, jointly learned scene representations for planning. Its shared scene encoder serves as a trainable tokenizer for the MLLM and supports efficient vision-based inference.
- Autonomous-driving systems have progressed from separately optimized modules toward end-to-end multi-task learning.
- LLM-based approaches investigate generalization to novel scenarios and textual reasoning about driving scenes.
- DiMA uses structured scene components rather than dense, unstructured inputs to help the language model reason about vehicle dynamics.
- The shared scene encoder is jointly trained with the MLLM, unlike TOKEN's frozen scene encoder, and provides structured inputs as a tokenizer.
3. DiMA Framework
DiMA jointly trains a vision-based planner and an MLLM around a shared, structured scene encoder, using language supervision, distillation, and planning-aligned surrogate tasks. The vision planner supplies structured BEAM tokens to the MLLM while retaining the planning pathway for efficient prediction.
- 3.1. Vision-based Planner: The shared scene encoder is jointly trained as both the MLLM tokenizer and the encoder for the planning transformer.This design gives the MLLM structured inputs while aligning scene representations with planning.
- 3.1. Vision-based Planner: DiMA uses a vision-based planner whose scene encoder produces structured BEAM token embeddings for the environment, ego vehicle, and surrounding agents.The planning transformer uses these representations for waypoint prediction, while the same scene tokens are passed to the MLLM.
- 3.2. MLLM Supervision: The MLLM is supervised with visual question answering, trajectory estimation, feature distillation, and surrogate tasks designed to enrich scene representations and support planning reasoning.VQA covers scene perception, agent behavior, ego behavior, and future planning steps.
- 3.2.2. MLLM Supervision: Masked token reconstruction recovers masked BEV inputs from multimodal context, enriching visual representations through an L2 reconstruction objective.Random masking is applied after scene encoding, and latent MLLM representations are decoded into reconstructed BEV tokens.
- 3.2.2. MLLM Supervision: Scene editing adds or removes surrounding agents and creates corresponding question-answer pairs to train reasoning about their impact on the ego vehicle’s future path.New car or truck trajectories are generated subject to map constraints and inserted as agent token embeddings.
- 3.2.3. Distillation: DiMA aligns the vision planner and MLLM by minimizing KL-divergence between their penultimate-layer hidden-feature distributions.The distillation loss compares planning-transformer features with MLLM ego-token embeddings, alongside planning and other task losses.
4. Experimental Setup
Experiments evaluate DiMA on nuScenes open-loop planning using vision-based planner variants, an MLLM design, staged training, and multiple validation splits. Performance is measured with trajectory error and collision rate under standardized and VAD evaluation protocols.
- 4.1. Architecture and Models: Experiments use VAD and UniAD as vision-based end-to-end planners, with two VAD model-size variants.Both planners perform perception, motion prediction, and planning, while UniAD also performs occupancy prediction.
- 4.1. Architecture and Models: The MLLM combines adapter layers, a language model, and task-specific decoder layers, with dedicated Q-former adapters for BEAM token embeddings.The adapters project visual tokens and language tokens into a shared embedding space.
- 4.2. Training: DiMA uses two-stage training: 60 epochs of vision-only pretraining followed by 30 epochs of joint planner-MLLM training with LoRA fine-tuning.The second stage incorporates all proposed tasks and losses.
- 4.3. Datasets: The nuScenes open-loop planning data contain 28k samples split into 22k training and 6k validation samples.Scenes include 3D object annotations, vehicle attributes, and CAN-bus ego-trajectory information; DriveLM supervises VQA.
- 4.4. Evaluation details: Planning is evaluated over 3 seconds with 2 predicted waypoints per second using L2 waypoint error and ego-vehicle collision rate.The study uses standardized evaluation for fair comparison and also compares against results reported under VAD evaluation.
- 4.4. Evaluation details: Validation covers the full 6019-sample split, a 689-sample targeted split with ego turns, and long-tail events involving rare maneuvers.These splits assess both general and difficult planning scenarios.
5. Experimental Results
DiMA is evaluated across standardized and VAD settings, general and targeted scenarios, long-tail events, MLLM-assisted inference, and ablations. It improves planning accuracy and collision performance while retaining an efficient vision-based inference path.
- Evaluation protocol: The evaluation reports L2 trajectory error and collision rate, with Table 2 averaging performance across all time steps.Table 4 uses the same metrics for DiMA-VAD-Tiny ablations under VAD evaluation.
- Standardized evaluation: 35% lower L2 trajectory error and 80% lower collision rate are reported on the overall nuScenes validation set.The targeted split also shows a 35% reduction in L2 trajectory error.
- Standardized evaluation: DiMA consistently outperforms PARA-Drive and TOKEN across the full and targeted validation splits.Its collision rate improves on TOKEN by 60%.
- VAD evaluation: DiMA-VAD-Tiny outperforms VAD-Base by 47% in L2 trajectory error while running four times faster under VAD evaluation.The comparison is reported on the nuScenes validation set.
- Long-tail performance: DiMA achieves the lowest L2 trajectory error in the evaluated long-tail scenarios.The long-tail evaluation uses manually selected events.
- MLLM branch: DiMA-Dual combines vision and MLLM features, with DiMA-Dual using VAD-Tiny matching DiMA using VAD-Base.Max-pooling the penultimate-layer features encourages consistency between the two prediction branches.
6. Conclusion
DiMA is presented as an end-to-end framework for robust and efficient autonomous-driving planning. It transfers MLLM knowledge to a vision-based planner through joint training and surrogate tasks.
- DiMA targets robust and efficient end-to-end autonomous-driving planning.
- The framework distills MLLM knowledge into a vision-based planner using joint training and surrogate tasks.The tasks include masked token reconstruction, future token prediction, and scene editing.
- The appendix documents surrogate-task details, training setup, additional text annotations, and qualitative planning and VQA results.
A. Surrogate tasks overview
DiMA’s MLLM branch is trained with surrogate tasks that enrich BEAM scene representations. These tasks use latent LLM embeddings for masked reconstruction, future prediction, and scene editing.
- Surrogate tasks: The surrogate tasks module trains the MLLM alongside planning and visual question answering to enrich BEAM scene representations.It uses hidden token embeddings from the LLM’s penultimate layer.
- Masked reconstruction: Masked reconstruction predicts missing BEV token embeddings from visible BEV tokens and the remaining multi-modal input.A reconstruction head operates on penultimate-layer latent representations.
- Surrogate tasks: The surrogate-task decoder heads use BEAM token embeddings for masked reconstruction, future prediction, and scene editing.BEAM denotes bird’s-eye-view, ego, agent, and map token embeddings.
- Scene editing: Scene editing creates question-answer pairs for adding or removing a car near the ego-vehicle.The examples depict both object addition and object deletion.
B.1. Training details
DiMA training first develops the vision planner’s latent scene representations, then jointly trains the planner and MLLM with the proposed tasks and losses. The experiments use standardized nuScenes evaluation across general and challenging targeted splits.
- Training process: DiMA uses a two-stage training process consisting of vision-planner pre-training followed by joint vision-planner and MLLM training.The first stage targets informative latent scene representations; the second incorporates the proposed tasks and losses.
- Joint training: The second stage fine-tunes the MLLM language model with LoRA and uses augmented DriveLM question-answer pairs with multi-view image sequences.The question-answer data are paired with multi-view visual inputs.
- Evaluation: Table 5 evaluates L2 trajectory error and collision rate on general and targeted nuScenes validation splits using standardized evaluation.Results are summarized over t = {1, 2, 3}s and across all time steps.
- Data augmentation: The augmented DriveLM annotations are generated from nuScenes object attributes, including visibility camera, object name, and vehicle speed.Rule-based algorithms also assign brief textual descriptions of scene elements.
C. Additional Qualitative Results
The qualitative results examine DiMA’s planning and VQA behavior on nuScenes, including difficult targeted turns and comparisons with VAD and GPT-4. DiMA’s targeted-turn examples show safer and more precise trajectories, while some VQA errors remain.
- Evaluation scope: The qualitative analysis includes planning comparisons on nuScenes and visual question-answering results across multiple dataset subsets.These results are presented as additional qualitative evaluations.
- Planning results: DiMA’s targeted-turn examples show collision avoidance, fewer lane departures, and more precise turns compared with VAD.The examples cover right and left turns in difficult nuScenes samples.
- VQA results: Two qualitative scenarios contain incorrect VQA results from DiMA.The analysis compares generated answers with ground-truth annotations and also plots predicted future trajectories.
- MLLM comparison: DiMA’s MLLM branch is compared with GPT-4 on reasoning questions using text prompts and stitched multi-view images.The comparisons also display predicted planning trajectories in image and diagrammatic forms.
D. Additional Quantitative Results
DiMA-UniAD shows consistent improvements across general, targeted, and long-tail nuScenes splits, while supplementary figures illustrate generated QA pairs, predictions, and failure cases.
- DiMA-UniAD consistently improves performance across general validation, targeted challenging samples, and long-tail scenarios, with significantly lower L2 trajectory error and collision rate.It also outperforms PARA-Drive and TOKEN in almost all reported cases.
- Generated QA pairs cover perception, prediction, and planning, while behavior questions use the ego vehicle’s future motion.The pairs are generated with Llama-3.
- Supplementary visualizations show VQA responses, predicted trajectories, and failure cases from the DiMA-VAD-Tiny MLLM branch.The figures compare predicted trajectories with ground-truth trajectories on targeted nuScenes examples.