Source-linked AI summary
MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, Bolei Zhou
TL;DR
RL driving research lacks integrated environments for generalization, safety, and complex multi-agent decision-making. MetaDrive addresses this gap with a compositional simulator that combines procedural and real-data scenario generation, and its experiments show improved generalizability with larger and more diverse training sets.
Problem
RL research commonly studies driving generalization, safety, and multi-agent decision-making separately because integrated environments are limited.
Method
MetaDrive composes diverse driving scenarios from procedural generation, real traffic-data replay, configurable objects, policies, and managers.
Results
Increasing training-set diversity and size improves RL-agent generalizability across procedurally generated and real-world scenarios, while MetaDrive benchmarks safe and multi-agent RL algorithms.
Takeaways & Limitations
MetaDrive provides a compositional platform for benchmarking generalizable RL, safe exploration, and multi-agent traffic simulation in driving environments.
Takeaways & Limitations
MetaDrive retains a sim2real gap, and agents transferred to real vehicles may cause accidents because of domain gap or neural-network uncertainty.
Abstract
from arXiv · showhide
Driving safely requires multiple capabilities from human and intelligent agents, such as the generalizability to unseen environments, the safety awareness of the surrounding traffic, and the decision-making in complex multi-agent settings. Despite the great success of Reinforcement Learning (RL), most of the RL research works investigate each capability separately due to the lack of integrated environments. In this work, we develop a new driving simulation platform called MetaDrive to support the research of generalizable reinforcement learning algorithms for machine autonomy. MetaDrive is highly compositional, which can generate an infinite number of diverse driving scenarios from both the procedural generation and the real data importing. Based on MetaDrive, we construct a variety of RL tasks and baselines in both single-agent and multi-agent settings, including benchmarking generalizability across unseen scenes, safe exploration, and learning multi-agent traffic. The generalization experiments conducted on both procedurally generated scenarios and real-world scenarios show that increasing the diversity and the size of the training set leads to the improvement of the RL agent's generalizability. We further evaluate various safe reinforcement learning and multi-agent reinforcement learning algorithms in MetaDrive environments and provide the benchmarks. Source code, documentation, and demo video are available at \url{ https://metadriverse.github.io/metadrive}.
1 INTRODUCTION
MetaDrive is introduced as a compositional driving simulator for studying generalizable RL across diverse, safe, and multi-agent scenarios. It combines procedural generation, real-data importing, and scalable simulation to support varied benchmarks and tasks.
- Motivation and platform: MetaDrive addresses RL generalization by composing diverse traffic scenarios from procedural generation, real-data importing, and domain randomization.Its interactive objects have configurable parameters that can be combined, managed, and actuated into executable scenarios.
- Motivation and platform: A single 100 MB MetaDrive instance can simulate realistic physics at up to 300 FPS on a standard PC.Compatibility with RLLib and Stable-Baselines3 enables more than 100 parallel instances for batch-data generation.
- Benchmarks and tasks: MetaDrive constructs four RL tasks spanning unseen procedural scenarios, unseen real scenarios, safe exploration, and multi-agent traffic.The first three tasks use single-agent settings, while the final task uses multiple agents.
- Benchmarks and tasks: Safe exploration evaluates driving under safety constraints with randomly scattered obstacles and collision costs.The benchmark tests several safe RL algorithms in safety-critical scenarios.
- Benchmarks and tasks: Multi-agent traffic experiments use 20 to 40 continuously controlled agents in scenarios including roundabouts and intersections.The setting studies whether coordinated agent policies produce collective traffic motion.
2 RELATED WORK
The related work positions MetaDrive among RL environments and driving simulators, emphasizing the need for scalable, realistic, diverse, and multi-agent driving benchmarks.
- RL environments: Existing RL benchmarks cover single-agent, multi-agent, multi-task, safety, and generalization problems across environments such as MuJoCo, SMAC, Meta-World, Safety Gym, and ProcGen.MetaDrive is presented in relation to these distinct benchmark focuses.
- Driving simulators: Driving simulators differ in realism, vehicle modeling, control abstraction, scenario complexity, and support for single-agent or multi-agent research.Table 1 compares representative simulators across these dimensions.
- Driving simulators: Many realistic simulators preserve detailed appearance and transitions, whereas others use simple kinematic models or simplified driving tasks.The related work contrasts photorealistic systems with simulators using bicycle models or high-level discrete controls.
- Driving simulators: Macroscopic traffic simulators such as CityFlow and FLOW focus on traffic-flow simulation and are described as unsuitable for investigating individual-agent driving problems.This distinction separates traffic-flow modeling from vehicle-level RL research.
3 SYSTEM DESIGN OF METADRIVE
MetaDrive organizes simulation through objects, policies, and managers, then composes environments by combining these abstractions. This design supports flexible mixtures of agents, traffic behaviors, maps, and safety elements.
- Core design: MetaDrive abstracts low-level implementation and aggregates elementary components into traffic scenarios through compositional APIs.The abstraction is intended to make new scenarios and tasks convenient to develop and benchmark.
- Core design: Objects connect the simulation engine to the Python environment while encapsulating physical and rendering models.Vehicles, obstacles, traffic lights, and road structures expose high-level manipulation and state-retrieval APIs.
- Core design: Policies map object and environmental states to actions or new states, and can be rule-based, learned, neural, or human-controlled.Vehicle policies include cruising, lane changing, emergency stopping, IDM, PPO, external neural control, and human control.
- Core design: Managers assign roles, process states, invoke policies, actuate objects, and reset objects under termination conditions.The same object class can serve different roles, such as ego and traffic vehicles in a single-agent environment.
- Composition workflow: Scenario composition hierarchically creates managers, spawns objects, assigns policies, and exposes managed states for external RL control.Managers monitor objects throughout each episode and can create or recycle objects as needed.
- Composition workflow: Replacing or adding managers can transform environments into multi-agent, reactive-traffic, or safety-critical variants.Examples include adding Object Manager, replacing replay traffic with IDM traffic, and updating Agent Manager to Multi-agent Manager.
4 COMPOSING DIVERSE DRIVING SCENARIOS
MetaDrive composes driving scenarios from procedural or real road networks and combines map, traffic, object, and agent managers to produce varied environments.
- Scenario composition: A scenario can sequentially execute Map, Traffic, Object, and Agent Managers to sample maps, control traffic, scatter obstacles, and actuate target vehicles.Additional specialized managers can be combined with these four basic managers.
- Procedural generation: Procedural generation samples road blocks with lanes, spawn points, and connection sockets to build road networks.Road blocks are the basic ingredients for generating diverse maps.
- Map sources: MetaDrive supports both procedurally generated road networks and imported road networks from real driving datasets.These are the platform’s two stated approaches for generating maps.
- Procedural generation: Road blocks are represented as undirected graphs with additional features for composing connected road structures.The representation is given as Gω = {V,E,S,P,ω,Ω}.
- Procedural generation: The procedural map algorithm takes a maximum number of tries, a block count per map, and a required number of maps, returning a set of generated maps.The algorithm is described as producing M from these inputs.
2 Function main(T, n)
The main function repeatedly initializes road networks and invokes BIG until it collects N successful maps.
- The procedure starts with an empty map list M.
- It continues generating maps while M contains fewer than N maps.
- Each iteration initializes an empty road network Gnet.
- A generated network is retained only when BIG returns success.
- Successful networks are appended to M.
9 Return M
The helper procedure is described as appending a feasible block to the current map and returning the map with a success flag.
- BIG appends one block to the current map when feasible and returns the resulting map together with a success flag.
11 Function BIG(T, Gnet, n)
BIG incrementally constructs a road network by trying randomized blocks, checking feasibility, and reverting unsuccessful additions.
- BIG operates over a network containing n blocks.
- For each of T trials, it searches for sockets connecting the new block with existing blocks.
- A successful addition is accepted when the feasibility check returns True.
- If attempts fail, BIG removes the last block from Gnet.
- The procedure randomly creates a block before attempting its insertion.
27 Function GetNewBlock()
MetaDrive builds diverse driving scenarios from configurable road blocks, procedural generation, imported real traffic data, and customizable traffic behavior.
- Road-block construction: Road networks use connected blocks with sockets that anchor block-to-block connections, while spawn points allocate traffic participants.
- Road-block construction: BIG selects a block type, samples random parameters, docks the block to the existing network, and rejects edge crossovers.
- Configurable scenarios: Users can override map configuration to customize road networks, including fixed block sequences or randomized block counts.
- Real-data scenarios: MetaDrive imports real map and traffic data from Argoverse and Waymo, supporting either replayed trajectories or reactive IDM control.
- Traffic generation: Traffic Manager customizes or randomizes vehicle types, powertrains, behaviors, spawn points, and destinations.
- Traffic generation: PG traffic supports Respawn and Trigger modes, respectively maintaining traffic density or increasing interaction with the target vehicle.
5 BENCHMARKING REINFORCEMENT LEARNING TASKS
MetaDrive benchmarks generalization, safe exploration, and multi-agent driving through four tasks with configurable simulation and reproducible baselines. Experiments show that training diversity improves simulated and real-world generalization, while safety and policy-design trade-offs remain important.
- Task Design: MetaDrive constructs four driving tasks spanning single-agent generalization, safe exploration, and multi-agent traffic, with reproducible baseline implementations.The first two tasks use rule-based or replay traffic, the safe-driving task scatters obstacles, and the multi-agent task learns traffic flow.
- Generalization to Unseen PG Scenes: Increasing training-scene diversity reduces overfitting while raising test success and lowering rule violations and crashes for PPO and SAC.With sufficiently many environments, test performance can match training performance.
- Generalization to Unseen Real Scenarios: Increasing real training-set size improves generalization, with test performance slightly exceeding training performance once the set contains more than 100 cases.The paper attributes this pattern partly to human selection of cleaner test cases than raw-data training cases.
- Generalization to Unseen Real Scenarios: Increasing procedural-scene diversity alone does not improve Waymo test performance, whereas increasing the real-data ratio improves real-test success.These experiments indicate that simulator and real-world data distributions need to be similar for sim-to-real transfer.
- Safe Exploration and Policy Discussion: SAC-RS has superior performance but high safety violations, while Lagrangian SAC lowers cumulative costs with little reward loss; CQL outperforms BC on limited offline data.Safety generalizability also improves with training-set diversity, and state-vector observations plus displacement reward support faster learning.
6 ETHICAL STATEMENT AND LIMITATIONS
MetaDrive has limited visual realism and incomplete scene coverage, and transferring its trained agents to real vehicles remains constrained by the sim-to-real gap.
- MetaDrive’s image rendering is less realistic than photorealistic driving simulators because it prioritizes flexible scenario composition.
- Generated scenes omit traffic participants such as pedestrians and bicyclists.
- Applying agents trained in MetaDrive to real vehicles may cause accidents because of domain gap or neural-network uncertainty.
7 CONCLUSION
MetaDrive is a compositional, extensible simulator for generalizable reinforcement learning, supporting diverse scenarios and RL tasks across single-agent and multi-agent settings.
- MetaDrive composes an infinite number of diverse driving scenarios through procedural generation and real traffic data replay.
- The platform supports varied RL tasks and baselines in both single-agent and multi-agent settings.