PaperlayerView exact revision on arXiv

Source-linked AI summary

Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models

Tsun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Yutong Ban, Alexander Amini, Guy Rosman, Sertac Karaman, Daniela Rus

arXiv:2310.17642v1cs.ROcs.CVcs.LG

TL;DR

Driving systems may encounter scenes beyond their training data, while the paper identifies the need for spatial detail beyond image-level foundation-model representations. It uses dense multimodal features for end-to-end control and reports improved adaptability in unpredictable open-set environments, with full-scale vehicle deployment providing an encouraging step toward practical integration.

  • Problem

    Driving scenes are too diverse to fully capture in training datasets, and systems can deteriorate on out-of-distribution data.

    Source: §I · p. 1

  • Method

    The approach replaces raw frames with dense features from multimodal foundation models for end-to-end driving, preserving spatial and semantic information.

    Source: §III · p. 3, A. Patch-wise Feature Extraction · p. 3

  • Results

    The authors report improved adaptability in unpredictable open-set environments and successful deployment on a full-scale autonomous vehicle.

    Source: §V · p. 7

  • Takeaways & Limitations

    Full-scale vehicle deployment provides encouraging first steps toward integrating multimodal foundation models with autonomous-driving perception.

    Source: §V · p. 7

Abstract

from arXiv · show

As autonomous driving technology matures, end-to-end methodologies have emerged as a leading strategy, promising seamless integration from perception to control via deep learning. However, existing systems grapple with challenges such as unexpected open set environments and the complexity of black-box models. At the same time, the evolution of deep learning introduces larger, multimodal foundational models, offering multi-modal visual and textual understanding. In this paper, we harness these multimodal foundation models to enhance the robustness and adaptability of autonomous driving systems, enabling out-of-distribution, end-to-end, multimodal, and more explainable autonomy. Specifically, we present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. To do so, we introduce a method to extract nuanced spatial (pixel/patch-aligned) features from transformers to enable the encapsulation of both spatial and semantic features. Our approach (i) demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations, and (ii) allows the incorporation of latent space simulation (via text) for improved training (data augmentation via text) and policy debugging. We encourage the reader to check our explainer video at https://www.youtube.com/watch?v=4n-DJf8vXxo&feature=youtu.be and to view the code and demos on our project webpage at https://drive-anywhere.github.io/.

I. INTRODUCTION

Autonomous driving systems face open-set scenes that training data cannot fully capture, while complex models make failures difficult to diagnose. The paper uses multimodal foundation models and spatially aligned features to support adaptable, queryable end-to-end driving and language-based simulation.

I. INTRODUCTION

  • Training datasets cannot capture the full diversity of driving scenes, and performance can deteriorate when systems encounter out-of-distribution data.Such deterioration can bring uncertainty and potential safety risks.Source: §I · p. 1
  • Complex models make it difficult to identify which learned concepts, objects, or pixels contribute to incorrect autonomous-system behavior.The paper notes that language can offer a human-comprehensible representation of system decision-making.Source: §I · p. 2
  • The method extracts per-patch features from transformer-based foundation models to preserve spatial and semantic information needed for driving decisions.The paper also proposes latent-space feature swaps using language for data augmentation and policy debugging.Source: §I · p. 2

A. Our contributions.

The paper combines open-set end-to-end driving with image- and text-queryable representations, patch-aligned features, and language-enabled latent simulation. It evaluates generalization in photorealistic simulations and validates the approach on a full-scale vehicle.

A. Our contributions.

  • The approach targets open-set end-to-end multimodal driving, with state-of-the-art results in in-distribution and OOD testing and decisions based on image- and text-queryable representations.The contribution passage does not provide benchmark values.Source: A. Our contributions · p. 2
  • The feature-extraction mechanism produces pixel- or patch-aligned features, extending foundation models that typically yield image-level vectors.Source: A. Our contributions · p. 2
  • Language-augmented latent-space simulation supports both training data augmentation and counterfactual reasoning for policy debugging.Source: A. Our contributions · p. 2
  • Photorealistic simulations test generalization to unseen environments and obstacles, and the methods are also deployed on a full-scale vehicle.Source: A. Our contributions · p. 2

II. RELATED WORK

Prior work spans end-to-end driving and foundation-model robotics, but existing pixel-aligned descriptor approaches have limitations. The method uses dense multimodal features in place of raw frames and presents patch-wise extraction and language-enabled latent simulation.

II. RELATED WORK

  • End-to-end driving networks face costly data demands and remain sensitive to scenarios that differ from training data, even when trained in simulation.Foundation models have also been applied across open-set robotics tasks and modalities.Source: §II · p. 2

III. METHOD

  • The overview maps modalities into a unified latent space, uses patch-anchored attention masks to retain spatial information, and substitutes language features for latent simulation.The caption illustrates replacing tree features with concepts such as houses, shops, or buildings.Source: Figure 2 · p. 3

II. RELATED WORK

  • Prior pixel-aligned descriptor approaches may lack multimodality, lose concepts during alignment, or depend on segmentation and crops that can be inefficient or miss regions.Source: §II · p. 3

III. METHOD

  • The method replaces raw RGB frames with dense features extracted by a multimodal foundation model as input to the end-to-end control policy.The feature representation is intended to preserve spatial and semantic information.Source: §III · p. 3

A. Patch-wise Feature Extraction

The section develops dense patch-aligned descriptors that preserve image semantics at spatial locations, then describes attention masking and finer-resolution extraction.

A. Patch-wise Feature Extraction

  • The target tensor F′ retains an input frame’s semantic information while preserving its scene locations at a desired spatial resolution.Source: A. Patch-wise Feature Extraction · p. 3
  • Attention-based extraction produces a feature for a selected patch or image area and can be applied at layers of transformer foundation models.Source: A. Patch-wise Feature Extraction · p. 3
  • Mask weights specify how much each patch contributes to a target patch’s semantic information; low-contribution patches can be suppressed, with proximity-based weighting as an option.Source: A. Patch-wise Feature Extraction · p. 4
  • Overlapping-patch inference with interpolated positional encodings yields finer-resolution multimodal features without additional training and performed consistently well in the reported experiments.Source: A. Patch-wise Feature Extraction · p. 4

B. Language-augmented Latent Space Simulation

The method simulates alternate driving concepts by replacing image-patch features with textual features, enabling counterfactual policy testing and training augmentation.

B. Language-augmented Latent Space Simulation

  • Textual features for LLM-proposed concepts replace image-patch features selected by similarity search, rather than being combined through latent-space arithmetic.The replacement may be conditioned on similarity thresholds or stochasticity, with plausible substitutes chosen using human prior or an LLM.Source: B. Language-augmented Latent Space… · p. 4
  • The method uses feature replacement because the latent space may not have a Euclidean structure suitable for standard arithmetic.The similarity measure can be a dot product, and the authors mention text inversion or prompt tuning as possible search improvements.Source: B. Language-augmented Latent Space… · p. 4
  • Language-augmented latent space simulation supports counterfactual policy debugging and data augmentation during training.The procedure replaces image features with alternative textual features to simulate different scenarios.Source: B. Language-augmented Latent Space… · p. 4
  • Table II evaluates image-trained driving policies by replacing image features with textual features when their similarity exceeds a threshold.A null threshold means driving solely on textual features; an infinite threshold means relying exclusively on image features.Source: Table II · p. 5

A. Experimental Setup

The experiments combine real and simulated driving data, evaluate lane-following and obstacle avoidance in closed-loop trials, and deploy policies on a full-scale vehicle.

A. Experimental Setup

  • The evaluation covers lane-following and obstacle avoidance, defining failures by lane-boundary departure, collision or unsafe proximity, and heading deviation over 30 degrees.Simulation uses a soft success rate: failure-free operating duration normalized by the trial horizon; real-world performance counts safety-driver interventions.Source: A. Experimental Setup · p. 5
  • Training data combine a real-world dataset with VISTA synthetic data based on about two hours of driving across varied times, weather, scenes, and seasons.Guided Policy Learning uses simulator privileged information; PID controllers supply lane-following targets and Control Barrier Functions supply obstacle-avoidance targets.Source: A. Experimental Setup · p. 5
  • The full-scale test vehicle was a retrofitted 2019 Lexus RX 450H equipped with a 130-degree camera, GPU, processor, IMUs, wheel encoders, and d-GPS.The camera operated at 30 Hz and captured 960 x 600 pixel images.Source: A. Experimental Setup · p. 5
  • Models were trained with Adam at a learning rate of 10^-3 for 10^6 iterations, using a plateau scheduler with factor 1 and patience 10.The feature-extraction backbone was BLIP2.Source: A. Experimental Setup · p. 5

B. Out-of-Distribution and Cross-modality Generalization

The experiments examine out-of-distribution performance across feature extractors and whether image-trained policies can operate with language features.

B. Out-of-Distribution and Cross-modality Generalization

  • Table III presents a debugging tool in which LLM-proposed concepts are used in language-augmented simulation to inspect policy decisions.The supplied caption does not specify table values or the outcomes of individual concept tests.Source: Table III · p. 5
  • In OOD settings, I-ViT and the proposed method surpass No-FM, while mask-based features underperform in both in-distribution and OOD settings.The authors suggest masking can remove useful context or create ambiguous crops for abstract categories; I-ViT does not incorporate language modality.Source: B. Out-of-Distribution and… · p. 5
  • Policies trained exclusively on image data perform reasonably well when image features are replaced by sufficiently similar textual features.A null threshold means text-only driving, while an infinite threshold means image-only driving; the authors describe the results as empirical evidence for language-augmented latent simulation.Source: B. Out-of-Distribution and… · p. 5

C. Debugging and Inspecting Policies With Language

Language-based feature substitution supports policy debugging, while linear classification probes how spatial feature patterns correspond to driving maneuvers.

C. Debugging and Inspecting Policies With Language

  • Language-augmented policy debugging identifies Car & Road as jointly important for recovery and stable lane maneuver, and Dark as revealing a simulation-artifact loophole.The policy uses the road end as a navigation reference; when the car strays far from lane center, the simulator can render dead or black pixels.Source: C. Debugging and Inspecting Policies… · p. 6
  • The maneuver probe clusters rollout features into four groups, projects them by distance to cluster centers, and applies linear support vector classification across patches.Cluster centers are matched to LLM-proposed textual features; the reported evaluation trains on 10 trajectories and tests on 90 distinct trajectories.Source: C. Debugging and Inspecting Policies… · p. 6
  • Table IV reports improved generalization from augmenting training with unseen, potentially relevant concepts proposed by LLMs.Its labels refer to the scenarios in Table I, including RSDDC for Rural, Spring, Dry, Day, Car.Source: Table IV · p. 6
  • Linear classifiers on projected patch features distinguish maneuvers, with coefficients in the lower image region structured around road and vehicle positions.For Car, lane-stable coefficients are negative in the road center, avoidance coefficients are positive across the road, and recovery coefficients are mainly positive at road edges; Road & Tree coefficients also vary by maneuver.Source: C. Debugging and Inspecting Policies… · p. 6, Figure 3 · p. 6

D. Data Augmentation using Language

Language-augmented latent simulation replaces selected image features with text-derived concepts for training augmentation. It improves performance in most OOD scenarios, but Dark replacement is error-prone in low-light conditions.

D. Data Augmentation using Language

  • The augmentation randomly swaps image pixel features highly similar to Tree or Dark with textual features for LLM-suggested non-drivable concepts.The selected pixel features are those with high similarity to the target concepts.Source: D. Data Augmentation using Language · p. 6
  • Performance improves in most OOD scenarios, with declines in RSDNA and RSDNC, which represent rural nighttime conditions with extremely low light.The paper attributes particular error risk to targeting Dark for replacement in these low-light environments.Source: D. Data Augmentation using Language · pp. 6–7
  • Targeting Dark for replacement makes augmentation particularly error-prone in low-light environments.Source: D. Data Augmentation using Language · pp. 6–7

E. Real Car Deployment

On a full-scale autonomous vehicle, the foundation-model approach showed near-flawless driving on a rural test track across changed road segments and objects absent from training. The conclusion presents this deployment alongside multimodal features and latent simulation as steps toward adaptable, more transparent end-to-end driving.

E. Real Car Deployment

  • The foundation-model approach yielded near-flawless driving on a rural test track, following lanes and avoiding objects absent from training.Tests took place on different road segments two years after the summer training data and included pedestrians, roadblocks, cones, chairs, and a toy house.Source: E. Real Car Deployment · p. 7
  • Inference averaged around 3 fps, versus 10 to 30 fps for nonfoundation-model policies; runtime optimization remains future work.The authors note that techniques such as quantization could improve runtime.Source: E. Real Car Deployment · p. 7

V. CONCLUSION

  • The conclusion links multimodal visual and textual understanding to adaptability in unpredictable open-set environments and insight into autonomous systems' decision-making.Source: §V · p. 7
  • Pixel/patch-aligned features and language-enriched latent simulation are presented as potential aids to training and policy debugging, while full-scale deployment is an encouraging first step.Source: §V · p. 7

Related papers

Loading 2310.17642v1…