Source-linked AI summary
EMMA: End-to-End Multimodal Model for Autonomous Driving
Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, Mingxing Tan
TL;DR
Autonomous-driving systems often specialize in individual tasks and train on limited datasets, constraining generalization to rare or novel scenarios. EMMA makes a multimodal large language model the core of an end-to-end system, representing inputs and outputs as language across driving tasks. It achieves strong planning and perception results, while co-training supports a generalist model that matches or surpasses individually trained models.
Problem
Task-specialized autonomous-driving systems trained on limited datasets can struggle to generalize to rare or novel scenarios.
Method
EMMA recasts autonomous-driving tasks as visual question answering for a multimodal large language model, directly mapping camera data and language inputs to multiple driving outputs.
Results
EMMA achieves state-of-the-art motion-planning performance on nuScenes, competitive WOMD results, competitive WOD 3D detection, and matches or surpasses individually trained models when co-trained across tasks.
Takeaways & Limitations
Co-training planning, object detection, and road graph tasks supports EMMA as a generalist autonomous-driving model with interpretable outputs for several perception tasks.
Takeaways & Limitations
Real-world deployment is constrained by camera-only 3D spatial reasoning, expensive sensor simulation for closed-loop evaluation, and higher computational requirements than conventional models.
Abstract
from arXiv · showhide
We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built upon a multi-modal large language model foundation like Gemini, EMMA directly maps raw camera sensor data into various driving-specific outputs, including planner trajectories, perception objects, and road graph elements. EMMA maximizes the utility of world knowledge from the pre-trained large language models, by representing all non-sensor inputs (e.g. navigation instructions and ego vehicle status) and outputs (e.g. trajectories and 3D locations) as natural language text. This approach allows EMMA to jointly process various driving tasks in a unified language space, and generate the outputs for each task using task-specific prompts. Empirically, we demonstrate EMMA's effectiveness by achieving state-of-the-art performance in motion planning on nuScenes as well as competitive results on the Waymo Open Motion Dataset (WOMD). EMMA also yields competitive results for camera-primary 3D object detection on the Waymo Open Dataset (WOD). We show that co-training EMMA with planner trajectories, object detection, and road graph tasks yields improvements across all three domains, highlighting EMMA's potential as a generalist model for autonomous driving applications. We hope that our results will inspire research to further evolve the state of the art in autonomous driving model architectures.
1 Introduction
EMMA addresses the limited generalization of task-specialized autonomous-driving systems by making a multimodal large language model the core of an end-to-end, multi-task architecture. It achieves strong planning and perception results while supporting generalist, interpretable driving outputs.
- Specialized driving systems trained on limited datasets can struggle to generalize to rare or novel scenarios.
- EMMA uses a multimodal large language model without additional specialized components, recasting driving tasks as visual question answering.Camera inputs and non-vision inputs such as driving commands and historical context are represented for the model, with task-specific prompts producing driving outputs.
- EMMA directly maps camera data and language inputs to future trajectories, perception objects, road graph elements, and scene semantics.Its outputs are represented as language and jointly processed in a unified language space.
- EMMA achieves state-of-the-art motion-planning performance on nuScenes and competitive results on WOMD.Motion-planning quality can be further improved with more internal training data and chain-of-thought reasoning.
- EMMA achieves better 3D object-detection precision and recall than state-of-the-art methods on the camera-primary Waymo Open Dataset.
- Co-training motion planning, object detection, and road graph tasks allows EMMA to match or surpass individually trained models.The model also demonstrates capacity to reason and make decisions in complex, long-tail driving scenarios.
- EMMA faces deployment challenges from limited camera-only 3D spatial reasoning, expensive sensor simulation, and higher computational requirements than conventional models.
2 Method
EMMA adapts a multimodal language model for autonomous driving by expressing sensor inputs, driving context, and task outputs in a shared language space. It extends end-to-end planning with chain-of-thought rationale and supports multiple perception and road-understanding tasks in one generalist model.
- Unified multimodal formulation: EMMA uses autoregressive Gemini models that process interleaved visual and textual inputs to generate natural-language outputs for driving tasks.Camera images or videos are represented as V, while prompts and non-vision inputs are represented as T.
- Unified multimodal formulation: Driving tasks share a unified language representation by encoding prompts, ego context, trajectories, 3D locations, and road graphs as text.The approach reuses pretrained language-model knowledge, although direct textual coordinate representations may require more tokens than specialized tokenization.
- End-to-end motion planning: EMMA’s planner predicts future ego waypoints from surround-view cameras, high-level intent, and historical ego status, then converts trajectories into vehicle-specific control actions.The formulation is end-to-end and requires only future ego locations as supervision, without dedicated human labels.
- Planning with chain-of-thought reasoning: Chain-of-thought planning asks EMMA to generate four ordered driving-rationale components before predicting future waypoints.The rationale progresses from coarse scene description to critical objects, their behavior, and related driving considerations.
- Planning with chain-of-thought reasoning: Predicting rationale before trajectories does not significantly change converged quality, allowing trajectory-first prediction and early stopping for time-critical inference.This provides an inference-time option without changing the trained model’s task formulation.
- EMMA Generalist: EMMA Generalist is trained across motion planning, object detection, and road-graph tasks, with depth-sorted 3D boxes improving detection quality.The reported generalist model consistently outperforms single-task specialist models, attributed in the passage to knowledge transfer, generalization, and efficiency.
3 Experiments
EMMA is evaluated across public and internal datasets for motion planning, perception, road graph estimation, and scene understanding. Results show strong planning and detection performance, with gains from data scaling, chain-of-thought reasoning, and structured task representations.
- Datasets: EMMA uses nuScenes, WOMD, WOD, and three large-scale internal datasets spanning motion planning, 3D detection, and road graph estimation.The internal datasets contain over 24 million motion-planning scenarios, 12 million detection examples, and 8 million road graph examples.
- WOMD motion planning: EMMA outperforms MotionLM on the same dataset, while EMMA+ outperforms both MotionLM and Wayformer on the internal planning benchmark.The full EFM+ (w/ CoT) surpasses previous state-of-the-art models by 13.5% at the 5s prediction horizon.
- nuScenes motion planning: 17.1%: EMMA improves the average L2 metric over BEV-Planner under the same self-supervised setup on nuScenes.EMMA achieves state-of-the-art planning results while also improving over OmniDrive by 12.1% despite using fewer labels.
- Chain-of-thought reasoning: 6.7%: combining chain-of-thought components improves end-to-end planning quality over standard planning.Driving meta-decision and critical object identification contribute improvements of 3.0% and 1.5%, respectively.
- Data scaling: Larger training datasets produce lower evaluation perplexity before overfitting, and driving quality has not plateaued on the mega-scale dataset.Smaller datasets overfit more quickly, while the full dataset continues to show unsaturated driving quality.
- 3D object detection: 16.3%: EMMA+ increases vehicle precision at the same recall compared with BEVFormer, while recall improves by 5.5% at the same precision.EMMA+ also achieves a better F1-score than prior work, with pedestrian performance comparable to MV-FCOS3D++.
Polyline representation: ego-origin aligned sample intervals are better than naively aligned
EMMA’s road graph representation uses ego-origin alignment, geometry-aware waypoint sampling, and structured target sequences. These design choices improve lane-structure modeling and support complementary gains when road graphs are co-trained with other driving tasks.
- Polyline representation: Sampling road graph points relative to the AV position avoids arbitrary offsets introduced by transforming globally sampled points into the ego frame.The representation adapts waypoint density to lane curvature and length to capture road geometry more accurately.
- Target sequence construction: Dynamic shuffling within distance bins improves robustness and coverage when constructing ordered polyline target sequences.Lanes are grouped by endpoint distance, then shuffled within bins representing nearby and connecting lanes.
- Target sequence construction: Padding polyline targets with explicit invalid tokens prevents early termination and improves training effectiveness.Both the number of polylines and the number of points per polyline are padded, with valid or invalid markers attached to each polyline.
- Language-like representation: Up to 10%: punctuation and semantically redundant tokens improve lane-level metrics in language-like road graph representations.The authors attribute this gain to Gemini’s language-related pre-training and the similarity of the structured expressions.
- Generalist co-training: Co-training all three tasks improves the generalist model by up to 5.5% over single-task models.The reported synergies arise among end-to-end planning, 3D object detection, and road graph estimation.
- Scene understanding and visualizations: EMMA’s visualized capabilities include generalization to diverse scenarios, predictive driving, obstacle avoidance, adaptive behavior, 3D detection, and road graph estimation.Examples cover rare objects, construction zones, traffic signals, vulnerable road users, and blocked lanes.
4 Related Works
Related work spans modular autonomous-driving systems, end-to-end learning, and multimodal language models. EMMA’s visual examples connect these directions to behavior in rare objects, construction zones, intersections, and interactions with road users.
- End-to-end autonomous driving: Modular systems separate perception, mapping, prediction, and planning, but limited inter-module communication creates scalability challenges.The paper contrasts this historical design with end-to-end multimodal approaches.
- Visualized driving behavior: EMMA’s examples include trajectory adjustments for garbage bags, ladders, squirrels, construction cones, traffic controllers, and nearby cyclists.The squirrel example illustrates behavior toward an object EMMA was not explicitly trained to detect.
- Visualized driving behavior: A red-light example jointly displays waiting behavior, surrounding 3D objects, and lane-center road graph predictions.Another example shows pausing for fast-moving motorcyclists that the model identifies accurately.
- Vision-language models: Vision-language driving systems use multimodal models for explainable behavior, text-based decisions, or iterative question-answering and control prediction.Examples include DriveGPT4, LMDrive, Drive Anywhere, and related multimodal driving systems.
- Multimodal large language models: Multimodal large language models extend language models to multiple modalities while supporting generalizability, reasoning, and few-shot capabilities.The paper situates EMMA among models such as Flamingo and CoCa.
5 Conclusion
EMMA is a Gemini-powered end-to-end multimodal model that maps camera inputs and text into multiple driving outputs in a unified language space. It achieves competitive or state-of-the-art results across tasks, and a single co-trained model can match or surpass individually trained models.
- 5 Conclusion: EMMA recasts autonomous-driving tasks as visual question answering and represents inputs and outputs as plain text for joint processing with task-specific prompts.The model directly maps raw camera data to planning trajectories, perception objects, and road graph elements.
- 5 Conclusion: A single co-trained EMMA predicts multiple driving tasks while matching or surpassing individually trained models.The conclusion highlights its potential as a generalist autonomous-driving model.
A Appendix
The appendix documents supplementary analyses, reproduction materials, failure examples, and acknowledged limitations that motivate future research.
- The appendix summarizes 12 meta-decision categories used in chain-of-thought reasoning.
- It supplies details on the 3D object detection metrics.
- Three distinct failure examples provide insights for future development.
- Concrete prompts and corresponding model-predicted answers are provided to facilitate reproduction.
- The authors acknowledge limitations and propose directions for future research.
A.1 Meta Decision in Chain-of-Thought Reasoning
EMMA's chain-of-thought reasoning uses a meta decision that partitions driving decisions into 12 heuristic categories and expresses them in natural language. Speed at future 0, 1, and 3 seconds anchors these categories, with identified causes appended when available.
- The meta decision partitions driving decisions into 12 categories using heuristics and transforms them into natural language.
- Speed at future 0, 1, and 3 seconds serves as the decision points for different categories.
- When identifiable, causes of speed changes such as traffic signs or critical objects are appended to the description.
- The authors plan to explore more fine-grained meta decisions and reasoning in future work.
A.2 Distance Breakdowns of 3D Detection Metrics
Distance breakdowns show that EMMA+'s performance gap relative to baseline models diminishes for more distant objects. The authors attribute this pattern to potentially lower-resolution camera input for EMMA.
- As object distance increases, the performance gap between EMMA+ and baseline models diminishes.
- The authors attribute the diminishing gap to potentially lower-resolution camera input used by EMMA than by baseline models.
- Table 6 summarizes 12 meta-decision categories, using speed at future 0, 1, and 3 seconds as decision points.
- The distance breakdown evaluates camera-primary 3D object detection experiments on WOD using standard LET matching.
A.3 Failure Examples
The appendix presents failure scenarios spanning planning, 3D detection, and road graph estimation. Errors include insufficient motorbike clearance, delayed distant-vehicle detection, and missed upcoming lane merges.
- EMMA's trajectory provides suboptimal initial clearance from a motorbike despite later validation as the motorbike moves right.
- EMMA misses a distant oncoming vehicle for one frame before detecting it in the subsequent frame.
- The delayed detection is suboptimal because the ego vehicle plans to nudge left, whereas earlier detection would support a safer, more controlled trajectory.
- EMMA identifies immediate drivable lanes but misses an upcoming lane merge, possibly because of a degraded painted arrow.
- The failure examples cover end-to-end planning trajectories, 3D object detection, and road graph estimation.
A.4 Concrete Prompts and Answers
EMMA generates task-specific driving outputs from varying prompts, with Table 7 documenting the prompts and corresponding predicted answers.
- EMMA generates desired outputs based on varying prompts.Table 7 provides the specific prompts used and their corresponding model-predicted answers.
- Table 7 documents the prompts used and their corresponding model-predicted answers.Numerical values are color-coded blue, while predicted separators are red for visualization.
- Table 7 uses color coding to distinguish numerical values from predicted separators.Numerical values appear in blue and predicted separators in red.
A.5 Limitations, Risks, and Mitigations
EMMA faces deployment and evaluation constraints involving temporal context, multimodal sensing, output verification, simulation cost, open-loop reliability, and inference latency. The authors describe mitigations including longer-context processing, 3D sensing encoders, efficient simulation, stronger evaluation frameworks, and latency optimization.
- Memory and video capability: Processing up to 4 frames restricts EMMA’s ability to capture long-term dependencies essential for driving tasks.The authors suggest memory modules or longer video sequences as possible directions.
- Extension to LiDAR and radar input: Pre-trained MLLMs typically lack LiDAR or radar inputs, limiting EMMA’s access to these 3D sensing modalities.The authors propose large-scale 3D sensing encoders aligned with camera inputs to improve cross-modality synergy and generalization.
- Verification of the predicted driving signals: Directly predicting driving signals creates challenges for real-time and post-hoc verification because intermediate outputs are not required.EMMA can jointly predict human-readable objects, road graph elements, and chain-of-thought driving rationale, but consistency is not guaranteed.
- Sensor simulation for closed-loop evaluation: Sensor simulation is often more computationally expensive than behavior simulation, hindering thorough closed-loop testing and verification.The authors identify efficient sensor simulation as a potential way to reduce this burden.
- More targeted and accurate open-loop evaluation: Existing open-loop evaluation is sometimes unreliable because nuScenes collision rates depend on BEV grid resolution and many scenarios lack planning diversity.The authors call for more challenging and trustworthy open-loop evaluation frameworks.
- Challenges of onboard deployment: Large models pose real-time deployment challenges through increased inference latency, motivating model optimization or distillation.A latency-optimized EMMA configuration achieves 3 FPS, a 67% speedup compared to UniAD’s 1.8 FPS.