Source-linked AI summary
SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexander Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, Jun Wang
TL;DR
Autonomous-driving research lacks realistic multi-agent simulation for competent interaction with diverse road users and scenarios. SMARTS addresses this gap with a scalable platform that bootstraps diverse behavior models and supports MARL experiments, benchmarks, and evaluation metrics for interactive driving.
Problem
Competent interaction with diverse road users in diverse autonomous-driving scenarios remains largely unsolved, while learning methods require realistic multi-agent simulation.
Method
SMARTS combines scalable multi-agent simulation with a bootstrapped Social Agent Zoo that trains, accumulates, and uses diverse road-user behavior models.
Results
SMARTS supports experiments with seven MARL algorithms across three increasingly challenging interactive driving scenarios and provides benchmark metrics for autonomous driving.
Takeaways & Limitations
The open-source platform is intended to support broader research on MARL interaction in autonomous driving through increasingly realistic and diverse simulated behavior.
Abstract
from arXiv · showhide
Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse and competent driving interactions. To meet this need, we develop a dedicated simulation platform called SMARTS (Scalable Multi-Agent RL Training School). SMARTS supports the training, accumulation, and use of diverse behavior models of road users. These are in turn used to create increasingly more realistic and diverse interactions that enable deeper and broader research on multi-agent interaction. In this paper, we describe the design goals of SMARTS, explain its basic architecture and its key features, and illustrate its use through concrete multi-agent experiments on interactive scenarios. We open-source the SMARTS platform and the associated benchmark tasks and evaluation metrics to encourage and empower research on multi-agent learning for autonomous driving. Our code is available at https://github.com/huawei-noah/SMARTS.
1 Introduction
SMARTS frames autonomous driving interaction as a multi-agent learning problem and addresses the lack of simulation environments for realistic, diverse interactions among heterogeneous road users.
- Interaction challenge: Current level-4 autonomous-driving systems often limit interaction by slowing or waiting, with conservativeness associated with rear-end crashes and sideswipes by other cars.The passage reports 57% rear endings and 29% sideswipes among autonomous-car crashes in California in 2018.
- Interaction challenge: The paper characterizes shared-road driving with diverse road users as a multi-agent problem for which multi-agent learning is a key part of the solution.It introduces multi-agent learning levels, or M-levels, to describe increasingly adaptive agent behavior.
- Interaction challenge: Complex scenarios such as double merge require sequential decisions and interaction among multiple vehicles, making handwritten rules difficult to scale.Vehicles must decide when to wait, change lanes, or rely on gaps created by other vehicles.
- Research gap: Autonomous-driving research has largely focused on M0, with highly limited attempts at M1 and M2.The paper maps the double merge scenario to MARL approaches at different M-levels in Table 1.
- Research gap: Existing simulators and interaction environments do not adequately support in-depth MARL research on autonomous-driving interaction.The paper contrasts general-purpose simulators with highway-env and BARK, which address narrower aspects of interaction scenarios or behavior models.
2 Design Goals of the SMARTS Platform
SMARTS is designed to grow realistic and diverse multi-agent interactions through a bootstrapped social-agent population while supporting heterogeneous agents, scalable computation, and benchmarking.
- Bootstrapping realistic interaction: SMARTS bootstraps realistic interaction by iteratively growing a Social Agent Zoo from self-play, population-based training, real-world driving data, or domain knowledge.The stated goal is increasing behavioral competence without sacrificing behavioral realism.
- Bootstrapping realistic interaction: As the Social Agent Zoo grows, SMARTS can simulate increasingly complex and realistic multi-agent interactions.The platform’s core covers physics, road-user behavior, road structure and regulations, and background traffic flow.
- Heterogeneous agent computing: SMARTS accommodates heterogeneous social agents that may range from full autonomous-driving stacks to simple scripts and may use different observations, actions, and computing resources.The platform dynamically associates social vehicles with social agents suited to the scenario’s fidelity requirements.
- Key features: SMARTS is natively multi-agent, allowing social agents to be as intelligent and heterogeneous as needed.This capability supports scenarios where social agents may match or differ substantially from ego agents.
- Key features: SMARTS provides scalable distributed social-agent computing alongside distributed ego-agent training and offers integrations with MARL libraries and algorithms.The platform also supports standard Gym APIs, web-streaming visualization, and broad observation and action spaces.
- Benchmarking: SMARTS includes an autonomous-driving MARL benchmarking suite with AD-specific evaluation metrics designed to pose challenging research questions while remaining grounded in AD reality.The paper states that additional SMARTS-based benchmarking suites are in development.
3 Architecture and Implementation
SMARTS uses compositional providers, scenario DSLs, and bubbles to construct and manage interactive traffic at scale. Its distributed design supports heterogeneous social agents and handoffs between traffic simulation and learned behavior.
- Runtime Architecture: Figure 3 separates learning-agent, social-agent, and traffic-provider control while allowing providers and agents to run in separate, potentially remote processes.Orange vehicles represent learning agents, dark blue vehicles social agents, and light blue vehicles the traffic provider.
- Simulation Providers: SMARTS composes the simulation from limited co-simulators, or providers, coordinated by each overall simulation step.The background traffic provider supplies traffic across maps ranging from intersections to cities, while foreground interaction can be determined by other providers and agents.
- Interaction Scenarios: A Python-based DSL specifies maps, routes, vehicle characteristics, and flows, while bubbles integrate Social Agent Zoo agents into scenarios.Bubbles are spatiotemporal, conditionally specified regions where social vehicles can transition from traffic-provider control to social-agent control.
- Interaction Scenarios: The bubble mechanism enables targeted interaction studies by handing social-vehicle control to specified agents at selected road locations.The handoff instantiates vehicle and sensor models, connects them to agent interfaces, starts agent processes, and switches control from the traffic provider.
- Distributed Computing: SMARTS elastically assigns computing resources to social agents and manages their dependencies because realistic agents may require substantial computation.This design avoids treating complex social agents as simple scripted non-player characters.
4 Support for Multi-Agent Reinforcement Learning
SMARTS supports scalable MARL research through composable scenarios, configurable agent interfaces, integrated learning libraries, and richer multi-agent evaluation metrics. Its scenario suite is designed for parallel training and evaluation across varied traffic interactions.
- Scenario and Training Support: SMARTS creates numerous scenarios varying road structure and traffic, then uses Ray and RLlib to support scalable, parallelizable training and evaluation.Figure 5 highlights examples of the scenario variety available for studying behavior and driving strategies.
- Agent Interfaces: Agents can receive configurable combinations of dynamic-object lists, occupancy grids, RGB images, vehicle states, and road structure, with adapters for tensor conversion.The observation space is selected from available sensor types according to the agent configuration.
- Algorithms and Baselines: SMARTS integrates RLlib, PyMARL, and MAlib so researchers can use multiple reinforcement-learning and multi-agent reinforcement-learning algorithms.The reported experiments use RLlib for high-concurrency training, while additional MARL implementations address paradigms beyond RLlib’s mainly single-agent focus.
- Metrics: Its evaluation framework covers model performance, population behavior distributions, and game-theoretic analysis rather than only task-specific driving outcomes.The proposed metrics address the dependence of an agent’s performance on interactions with other agents.
5 Experiments & Results
SMARTS is evaluated with seven MARL algorithms across three increasingly difficult interactive driving scenarios. Results show stronger performance for MADDPG in most tasks and increasingly complex, less uniformly high behavior as scenario difficulty rises.
- Experiments: Seven MARL algorithms are tested in three increasingly challenging scenarios requiring non-trivial interactive capabilities.The scenarios are Two-Way traffic, Double Merge, and Unprotected Intersection, with background traffic supplied by SUMO.
- Baselines: The benchmark compares independent, centralized-training, and fully centralized methods under shared observation, action, and reward functions.The baselines include DQN, PPO, MAAC, MF-AC, MADDPG, Networked Fitted-Q, and CommNet.
- Experimental Setup: The experimental observation stacks three frames containing goal position, lane-center distance, speed, steering, heading errors, neighboring-vehicle states, and a bird’s-eye image.Each frame includes driving states for at most eight neighboring vehicles.
- Experimental Setup: The discrete action vector controls longitudinal behavior through keeping lane or slowing down and lateral behavior through turning right or left.The action representation is four-dimensional.
- Experimental Setup: The reward combines components based on ego states, surrounding-vehicle interactions, and key traffic events.The reward is a weighted sum of these shaped components.
- Performance: MADDPG outperforms the other baselines in most tasks, especially Intersection, while all algorithms score higher without social vehicles.The authors associate MADDPG’s advantage in the toughest unprotected-intersection setting with access to extra information from other agents.
- Behavior Analysis: As scenario difficulty increases, behavioral differences span more metrics and some scores become harder to maximize; in Intersection, agility drops and its spread widens.Differences concentrate on Diversity in Two-Way, then include Diversity and Safety in Double-Merge and Intersection, with Agility also differing in Intersection.
6 Conclusion
SMARTS brings scalable multi-agent learning together with scalable simulation of realistic driving interaction. Its international competition support demonstrates practical use at large evaluation scale.
- SMARTS is an open-source platform combining scalable multi-agent learning with scalable simulation of realistic driving interaction.
- Thousands of agent-model submissions have been automatically evaluated across three internationally supported autonomous-driving competitions.
- The platform is intended to expose realistic driving-interaction challenges and support principled research on solving them.
A Survey of AD Related Simulators
Autonomous-driving simulators include data-replay and interactive approaches, each addressing interactive behavior differently. SMARTS aims to bootstrap behavior models that improve interactivity and realism across both approaches.
- Existing autonomous-driving-related simulators can be viewed through data-replay and interactive approaches to simulating road-user behavior.
- Data-replay simulators use recorded real-world behavior, preserving observed reactions but limiting interaction beyond the recorded data.
- Interactive simulators construct road-user behavior models instead of replaying recorded data, but their behavior is typically controlled by predefined models.
- SMARTS is designed to bootstrap realistic and diverse behavior models that make data-replay simulators more interactive and interactive simulators more realistic.
B SMARTS Simulations
SMARTS supports scalable, distributed, and multi-agent simulation, with configurable controllers, action spaces, algorithms, and visual inspection of concurrent instances.
- The platform includes visualization and benchmark reporting through screenshots, algorithm tables, and collision-rate/completion-rate results.
- SMARTS can run multiple simulation instances concurrently across processes or networked machines, with each instance sampling its own scenarios.
- Multiple ego agents can train simultaneously in one SMARTS instance, with each agent running in a separate process or remotely.
- Bubbles define regions where Social Agent Zoo vehicles interact meaningfully with ego vehicles.
- SMARTS maps controller choices to action spaces and integrates MARL algorithms across centralized, decentralized, CTDE, and networked paradigms.
E Experiment Results
SMARTS experiments are built from configurable scenarios, heterogeneous agents, and multiple MARL paradigms. Scenario and agent components expose flexible choices for traffic, behavior, interfaces, and algorithms.
- MARL Paradigms: SMARTS experiments support fully centralized, fully decentralized, CTDE, and networked agent-learning paradigms.
- Scenario Generation: A SMARTS scenario combines a map, traffic flow, agent missions, and optionally bubble specifications.
- Scenario Generation: Scenario configuration supports custom social-vehicle behaviors, traffic-flow rates, routes, actor mixtures, and one-command generation.
- Building Agents: SMARTS provides ten built-in agent types covering frequently used observation-sensor and action-controller combinations.
- Building Agents: Agent specifications combine interfaces, policies, adapters, and parameters before concrete agent instances are built.
F.3 Running Single-agent & Multi-agent Training
SMARTS supports both single-agent and multi-agent training through a configuration workflow that initializes agents, specifies algorithms, and defines scenarios. Multiple scenarios can be loaded automatically for multi-task learning.
- SMARTS training experiments configure agent initialization, algorithm specification, and scenario specification.
- Multiple training scenarios can be automatically loaded and used for multi-task learning.
- The environment configuration supports one or more scenarios, including scenario paths and execution settings such as random seed and headless mode.
- Multi-agent configurations define policies and map each agent identifier to its corresponding policy.
F.4 Running Evaluation
SMARTS evaluates algorithms with a benchmarking runner that records episode steps and computes metrics. Behavior metrics compare coverage and behavioral diversity, including interactions with and without social vehicles.
- The benchmarking runner records episode steps and evaluates algorithms through an Episode recorder and Metric class.
- Evaluation logs observations, actions, rewards, and termination information at each step before computing the configured metrics.
- Figure 11 compares behavior metrics using coverage as desirability and curve spread as behavioral diversity.
- Figure 11 distinguishes algorithms interacting with social vehicles from those evaluated without social vehicles.