PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving

Shuo Xing, Chengyuan Qian, Yuping Wang, Hongyuan Hua, Kexin Tian, Yang Zhou, Zhengzhong Tu

arXiv:2412.15208v2cs.CVcs.LGcs.RO

TL;DR

Modular autonomous-driving systems can face communication errors and rigid interfaces, while end-to-end AD development has relied on resource-intensive fine-tuning. OpenEMMA uses open-source models and Chain-of-Thought reasoning for end-to-end planning and perception, and reports improvements over zero-shot baselines across challenging driving scenarios.

  • Problem

    Modular AD systems can face inter-module communication errors and rigid interfaces that struggle to adapt to unforeseen conditions.

    Source: §1 · p. 1

  • Method

    OpenEMMA uses front-camera images and historical ego status with Chain-of-Thought reasoning for end-to-end planning, plus a fine-tuned YOLO model for 3D object detection.

    Source: §1 · pp. 1–2, §5 · p. 6

  • Results

    OpenEMMA reports significant improvements over zero-shot baselines and effectiveness, generalizability, and robustness across challenging driving scenarios.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The open-source framework aims to broaden access to EMMA-like trajectory-planning and perception capabilities for research and development.

    Source: §1 · pp. 1–2

    The framework uses only basic inference-time Chain-of-Thought reasoning, leaving more advanced reasoning techniques unexplored.

    Source: §6 · p. 6

Abstract

from arXiv · show

Since the advent of Multimodal Large Language Models (MLLMs), they have made a significant impact across a wide range of real-world applications, particularly in Autonomous Driving (AD). Their ability to process complex visual data and reason about intricate driving scenarios has paved the way for a new paradigm in end-to-end AD systems. However, the progress of developing end-to-end models for AD has been slow, as existing fine-tuning methods demand substantial resources, including extensive computational power, large-scale datasets, and significant funding. Drawing inspiration from recent advancements in inference computing, we propose OpenEMMA, an open-source end-to-end framework based on MLLMs. By incorporating the Chain-of-Thought reasoning process, OpenEMMA achieves significant improvements compared to the baseline when leveraging a diverse range of MLLMs. Furthermore, OpenEMMA demonstrates effectiveness, generalizability, and robustness across a variety of challenging driving scenarios, offering a more efficient and effective approach to autonomous driving. We release all the codes in https://github.com/taco-group/OpenEMMA.

1. Introduction

Autonomous driving must handle unpredictable, complex scenarios, while modular systems face scalability problems and specialized end-to-end systems struggle to generalize. OpenEMMA offers an open-source MLLM-based framework intended to broaden access to end-to-end driving research.

1. Introduction

  • Modular AD architectures face scalability issues from inter-module errors and rigid interfaces, while specialized end-to-end systems can struggle to generalize across complex scenarios.The introduction frames MLLMs as a possible approach because of their broad training and reasoning abilities.Source: §1 · p. 1
  • OpenEMMA uses front-camera images and textual ego-vehicle history with MLLM-based reasoning, and adds a fine-tuned visual specialist for 3D object detection.Its outputs include scene and behavioral descriptions and meta-driving decisions used as context for waypoint generation.Source: §1 · pp. 1–2
  • The authors evaluate a diverse selection of MLLMs for end-to-end trajectory planning on the nuScenes validation set.Source: §1 · p. 2
  • The authors release the codebase, datasets, and model weights for community use and extension.Source: §1 · p. 2

2. Methodology

OpenEMMA combines pre-trained MLLMs with historical driving status and front-camera scenes for trajectory planning, while a specialized detector handles on-road 3D object detection. Its planning process uses interpretable intermediate representations and staged reasoning.

2. Methodology

  • OpenEMMA uses pre-trained MLLMs with historical driving status and visual scenes to predict future trajectories and detect traffic participants.Source: §2 · p. 2

2.1. End-to-End Planning with Chain-of-Thought

  • The MLLM predicts speed and curvature representations, which are integrated through heading and velocity components to produce the future ego-coordinate trajectory.The decomposition is presented as a human-interpretable pathway that mirrors driving.Source: §2.1 · p. 2
  • The planning method prompts MLLMs to reason over the front-camera scene and five seconds of ego speed and curvature history, including intent, scene conditions, and major road users.Source: §2.1 · pp. 2–3

2.2. Object Detection Enhanced by Visual Sepcialist

  • For single-frame, front-camera 3D object detection, OpenEMMA selects YOLO3D, a lightweight image-only method that estimates 3D boxes from 2D boxes and 3D estimates.YOLO3D enforces consistency by assuming each 3D box is tightly enclosed within its corresponding 2D box.Source: §2.2 · p. 3

3. Experiments

The experiments compare OpenEMMA with zero-shot planning across several MLLMs, describe its YOLO11n adaptation, and illustrate behavior in varied driving scenarios.

3.1. End-to-End Trajectory Planning

  • Experiments use 150 nuScenes validation scenes, compare four MLLMs with a zero-shot baseline, and predict trajectories over five seconds.GPT-4o results cover a limited set of scenes because of budget constraints and reproducibility considerations.Source: §3.1 · p. 4
  • OpenEMMA beats zero-shot planning in both L2 error and failure rate with LLaVA-1.6-Mistral-7B, and modestly improves both with Llama3.2-11B-Vision-Instruct.With Qwen2-VL-7B-Instruct, L2 error rises because OpenEMMA predicts in cases where zero-shot fails, while failure rate falls; the authors note these trajectories can still be poor.Source: §3.1 · pp. 4–5

3.2. 3D Object Detection

  • YOLO11n[1] fine-tuned on nuImages[6], with images downsampled to 640×360, achieved its best 2D detection mAP50 of 0.60316 at epoch 290 after 300 epochs on one RTX 4060Ti.Source: §3.2 · p. 5

3.3. Visualization

  • GPT-4o visualizations show OpenEMMA following a lane through a right turn and braking when another vehicle suddenly enters its lane.The examples use the current scene and visual inputs from the preceding five seconds, or 10 frames.Source: §3.3 · p. 5
  • In low light, OpenEMMA may miss some objects but can identify key objects and plan a left-lane transition.The visualization describes this as a demonstration of robustness under reduced visibility.Source: §3.3 · p. 5

4. Related Work

Related work traces end-to-end autonomous driving from holistic task optimization to MLLM-based systems, and describes how multimodal models connect visual and textual inputs.

End-to-End AD

  • End-to-end driving systems optimize for the driving task across components, contrasting with modular architectures; approaches include imitation and reinforcement learning.The section also distinguishes reinforcement-learning work focused on decision-making from scenario-generation work intended to improve robustness testing.Source: §4 · p. 5

MLLM for AD

  • MLLMs align visual and textual information by converting image patches into visual tokens aligned with the text-token space.The section presents vision encoders such as CLIP as one means of producing those visual tokens.Source: §4 · p. 5

End-to-End AD

  • MLLM-based driving work includes language-token trajectory generation, Chain-of-Thought planning, retrieval-augmented generation, and graph-structured VQA.The section also describes systems combining MLLMs with end-to-end driving, including EMMA, which produces trajectories, perceived objects, and road-graph elements.Source: §4 · pp. 5–6

5. Conclusion

OpenEMMA is an open-source, computationally efficient end-to-end autonomous-driving framework built on MLLMs. It uses Chain-of-Thought reasoning for trajectory planning, incorporates an external visual specialist for 3D object detection, and improves over zero-shot baselines across challenging scenarios.

5. Conclusion

  • OpenEMMA is an open-source, computationally efficient end-to-end autonomous-driving framework built on MLLMs.It is presented as a framework for trajectory planning and perception.Source: §5 · p. 6
  • Using historical ego-vehicle data and front-camera images, OpenEMMA applies Chain-of-Thought reasoning to predict future speed and curvature for trajectory planning.The predicted values are integrated into the trajectory planning process.Source: §5 · p. 6
  • OpenEMMA improves over zero-shot baselines and demonstrates effectiveness, generalizability, and robustness across challenging driving scenarios.The framework also incorporates a fine-tuned external visual specialist for precise 3D on-road object detection.Source: §5 · p. 6

6. Limitation, and Future Work

OpenEMMA uses only basic inference-time Chain-of-Thought reasoning and relies on fine-tuned YOLO for object detection because current MLLMs have limited object-grounding capabilities. The authors identify stronger spatial reasoning and grounding accuracy as needed for a unified framework across autonomous-driving perception and reasoning tasks.

6. Limitation, and Future Work

  • OpenEMMA uses only basic Chain-of-Thought reasoning at inference, leaving advanced techniques such as CoT-SC and ToT as possible future additions.Source: §6 · p. 6
  • Because current MLLMs have limited object-grounding capabilities, OpenEMMA uses fine-tuned YOLO for detection rather than relying solely on the MLLM.Source: §6 · p. 6
  • The authors identify improved spatial reasoning and grounding accuracy as necessary for a unified framework covering key autonomous-driving perception and reasoning tasks.Source: §6 · p. 7

Related papers

Loading 2412.15208v2…