PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow: A Modular Learning Framework for Mixed Autonomy Traffic

Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, Alexandre M Bayen

arXiv:1710.05465v4cs.AIcs.ROeess.SY

TL;DR

Mixed autonomy—the transition between isolated and full autonomy—is poorly understood despite its importance for vehicle controllers, transportation systems, urban planning, and policy. The paper introduces Flow, a modular deep reinforcement learning framework for complex traffic scenarios, and reports learned controllers that can practically eliminate congestion and achieve near-optimal performance.

  • Problem

    Mixed autonomy, defined as the intermediate regime between no adoption and full adoption of autonomy, is poorly understood despite its importance for transportation systems and policy.

  • Method

    The paper proposes Flow, a modular framework that uses model-free deep reinforcement learning to decouple traffic-system modeling from control-law design and compose reusable scenario components.

  • Results

    Learned control laws practically eliminate congestion in single-lane traffic across evaluated densities and exceed model-based controllers, with experiments achieving near-optimal performance.

  • Takeaways & Limitations

    Flow provides a methodological pathway for studying complex mixed-autonomy traffic through reusable traffic modules and modern reinforcement learning methods.

  • Takeaways & Limitations

    The methodology's global convergence, stability, robustness, safety, and sensitivity to simulation-model error or misspecification remain open research questions.

Abstract

from arXiv · show

The rapid development of autonomous vehicles (AVs) holds vast potential for transportation systems through improved safety, efficiency, and access to mobility. However, the progression of these impacts, as AVs are adopted, is not well understood. Numerous technical challenges arise from the goal of analyzing the partial adoption of autonomy: partial control and observation, multi-vehicle interactions, and the sheer variety of scenarios represented by real-world networks. To shed light into near-term AV impacts, this article studies the suitability of deep reinforcement learning (RL) for overcoming these challenges in a low AV-adoption regime. A modular learning framework is presented, which leverages deep RL to address complex traffic dynamics. Modules are composed to capture common traffic phenomena (stop-and-go traffic jams, lane changing, intersections). Learned control laws are found to improve upon human driving performance, in terms of system-level velocity, by up to 57% with only 4-7% adoption of AVs. Furthermore, in single-lane traffic, a small neural network control law with only local observation is found to eliminate stop-and-go traffic - surpassing all known model-based controllers to achieve near-optimal performance - and generalize to out-of-distribution traffic densities.

I. INTRODUCTION

The paper frames mixed-autonomy traffic as an important, poorly understood transition requiring computational tools for complex, large-scale systems. It introduces Flow, a modular deep-RL framework for composing and studying diverse traffic-control scenarios.

  • Motivation: The transition to autonomous vehicles could substantially alter transportation energy use, motivating tools for complex, large-scale robotic systems.The cited estimates range from a 40% decrease in U.S. fuel consumption to a possible 100% increase after autonomous fleets are deployed.
  • Motivation: Mixed autonomy is poorly understood between isolated autonomy, where AVs do not affect traffic, and full autonomy, where vehicles coordinate.The paper links this gap to vehicle-controller design, transportation efficiency, urban planning, and public policy.
  • Framework: Flow represents complex traffic-control environments as reusable modules analogous to LEGO blocks and applies model-free deep reinforcement learning.The framework decouples mathematical system modeling from control-law design and supports partially observed traffic settings.
  • Framework: The framework supports multi-lane, multi-AV, intersection, arbitrary-network, and microsimulation scenarios through composable traffic-control primitives.Flow was open-sourced to facilitate research across networks ranging from analytical composites to full traffic microsimulators.
  • Contribution: Flow’s contributions span empirical study of large-scale multi-robot control, challenging RL scenarios, learned-control intuition, and transportation methodology.The framework uses reusable traffic modules and modern RL methods to address mixed-autonomy traffic challenges.

II. MIXED AUTONOMY

Mixed autonomy asks what system performance is achievable at a given fraction of autonomous vehicles, under uncertainty from human interactions, partial observation, and partial control. The paper proposes model-free deep RL with modular scenarios to decouple system modeling from control-law design.

  • Problem definition: Mixed autonomy studies achievable traffic-system performance when a fraction p of vehicles is autonomous, between zero and full autonomy.The paper uses a system utility function U_p(·) to formalize performance under partial adoption.
  • Sources of uncertainty: Mixed autonomy combines uncertain evaluation objectives with human interactions, partial observability, and partial controllability.These challenges make direct transfer of isolated- or full-autonomy analytical approaches unsuitable.
  • Method: The paper adopts a model-free approach because strict coupling between mathematical dynamics and system evaluation is not sensible in mixed autonomy.The cited rationale is the setting’s uncertainty in dynamics and limited control over human-driven vehicles.
  • Method: Deep RL provides a control framework that accepts arbitrary system models through a (PO)MDP interface and supports expressive neural-network control laws.The decoupling permits designers to choose system dynamics and control objectives independently.
  • Related work: Existing mixed-autonomy traffic studies include formal, simulation-based, and model-based approaches, but generally rely on simplified models.Prior work spans vehicle control, intersection management, and shared-mobility abstractions.
  • Related work: Flow addresses the tight coupling between traffic mathematical frameworks and analysis by composing reusable modules for arbitrary evaluation objectives.This design aims to combine lateral, intersection, and longitudinal traffic dynamics within one learning methodology.
  • Related work: The paper positions deep RL as an early approach for designing AV controllers and evaluating their impacts on traffic flow.It contrasts learned control with prior model-based methods and reduced reward-based design supervision.

IV. PRELIMINARIES

The paper formulates traffic scenarios through finite-horizon (partially observed) Markov decision processes and uses reinforcement learning to optimize control policies. A scenario includes the complete traffic setting, including learning and nonlearning components.

  • MDP formulation: The framework targets episodic finite-horizon discounted MDPs defined by states, actions, transitions, rewards, initial-state distribution, discount factor, and horizon.The MDP formulation provides the standard interface for supported traffic scenarios.
  • Terminology: A scenario is the full traffic setting, including learning and nonlearning components, represented through a (PO)MDP interface.The term also covers settings containing only human-driver models and no learned component.
  • Horizon choice: Finite-horizon MDPs approximate traffic’s infinite-horizon behavior when the horizon captures transient or periodic phenomena.The paper motivates this choice using ephemeral or periodic traffic patterns such as jams and daily or weekly cycles.

B. Reinforcement learning

The paper formulates traffic systems as (PO)MDPs and uses policy-gradient deep RL to learn controllers within modular, composable traffic environments. This approach supports varied scenarios and accommodates dynamics that are difficult to capture in fixed model-based formulations.

  • Reinforcement learning: Policy-gradient RL optimizes stochastic policies represented by deep neural networks, which the paper calls controllers or control laws.The policy maps states or observations to actions while maximizing cumulative reward.
  • Traffic dynamics: Traffic environments use car-following ODEs to model longitudinal human-driving dynamics from headway, relative velocity, and vehicle velocity.The Intelligent Driver Model is used to model human driving in the experiments.
  • Modular framework: Flow exposes a modular framework for creating, studying, and controlling complex traffic scenarios with deep RL.The framework is designed for varied multi-agent traffic settings where different vehicles, networks, and behaviors may need to be composed.
  • Modular framework: Modules can represent networks, dynamics, metrics, initialization, and actors, allowing traffic scenarios to be assembled from configurable components.Metrics generate aggregated statistics and reward signals, while dynamics can include routes, demands, stochasticity, rules, and safety constraints.
  • Modular framework: A model-agnostic design permits disparate dynamics to be studied in one scenario through sampling-based optimization such as deep RL.This avoids some reconfiguration limits of model-based formulations when mathematical descriptions differ.

B. Architecture and implementation

Flow connects configurable traffic modules to simulation and RL interfaces, enabling experiments with mixed-autonomy scenarios. The single-lane experiment combines a randomized circular track, human-driver models, one learning-controlled AV, partial observation, and multiple controller baselines.

  • Architecture and implementation: Flow builds on SUMO for microscopic traffic simulation, Ray RLlib for RL methods, and OpenAI Gym for the MDP interface.The implementation is open source and intended to support extension with custom modules.
  • Architecture and implementation: The environment composes dynamics, advances simulation, returns observations, applies actions, computes rewards and metrics, and resets episodes.A generator creates SUMO-compatible network configuration files from network descriptions.
  • Architecture and implementation: Flow can evaluate classical model-based methods because its learning component is optional.This supports comparisons among different traffic-control approaches.
  • Experiment modules: The circular-track experiment samples track lengths uniformly from 220 m to 270 m to represent a continuous range of traffic conditions.The setup uses a single-lane circular network with 22 vehicles, each 5 m long.
  • Experiment modules: The learning agent can use local observations based on its velocity, headway, relative headway rate, speed limit, and maximum track length.The full state contains positions and velocities for all vehicles, while the restricted observation reflects directly sensed information.
  • Experiment modules: The experiment compares GRU and MLP learned controllers with FollowerStopper, PI, and IDM baselines.Twenty-one vehicles follow IDM dynamics while one autonomous vehicle uses the compared control law.
  • Experiment modules: The traffic dynamics include IDM human drivers with Gaussian acceleration noise and simulator-enforced safety failsafes.Performance is evaluated using average network velocity and an acceleration-penalizing control cost combined into the learning reward.

B. Learning setup

The AV control laws are trained with TRPO policy gradients using neural-network policies and a large simulated driving corpus.

  • Learning setup: TRPO trains the AV control laws with policy gradients, linear feature baselines, γ = 0.999, and step size 0.01.Training used 6,000,000 simulated samples, equivalent to 167 driving hours.

C. Performance bounds

The paper evaluates learned controllers against performance bounds defined by uniform-flow and stop-and-go limit cycles. These bounds are approximate because they are derived from a no-autonomy analysis but evaluated in mixed autonomy.

  • Performance bounds: Limit cycles provide steady-state performance bounds because traffic trajectories tend toward or away from these closed curves when stable or unstable.They generalize equilibrium points from fixed states to trajectories.
  • Performance bounds: Uniform flow, with constant velocity v* and headway h*, represents the desirable upper performance bound under the IDM relationship.These equilibria have high velocities but are unstable under human driving behavior.
  • Performance bounds: Stable stop-and-go limit cycles represent a practical lower bound because the traffic system tends toward traffic jams under IDM.A controller performing worse could be replaced by a human-driver model for a better outcome.
  • Limitations: The performance bounds are close approximations rather than exact bounds because they analyze no autonomy but evaluate mixed autonomy.The paper points to related work for more detailed performance bounds.
  • Results: The mixed-autonomy track demonstrates that learned controllers can surpass known model-based controllers in a challenging traffic-control scenario.The section also uses the experiment to validate Flow’s modular scenario-composition capability.

1) Performance:

Learned GRU and MLP controllers closely match optimal velocity across traffic conditions, outperforming model-based controllers while generalizing beyond training densities. They also reduce oscillations and avoid external density-specific calibration, although baseline performance can be sensitive to tuning and initial conditions.

  • Performance: GRU and MLP control laws closely match optimal velocity across 210 to 290 m circumference tracks, practically eliminating congestion.The Linear controller performs well but below MLP and GRU.
  • Velocity profiles: Both learned controllers bring the system close to the 4.82 m/s uniform-flow velocity, while GRU reaches equilibrium fastest.FollowerStopper settles at 4.15 m/s, below the 4.82 m/s equilibrium velocity.
  • Traffic structure: Learned controllers produce larger headways than PI with Saturation and FollowerStopper, which leave smaller gaps between vehicles.Smaller gaps could help prevent opportunistic lane changes in multi-lane scenarios and inform reward design.
  • Robustness: Learned control laws avoid external calibration of traffic-setting parameters, unlike model-based baselines that can be sensitive to parameters and initial conditions.PI with Saturation performs decently at its calibrated 260 m setting but degrades quickly elsewhere; FollowerStopper requires a desired velocity beforehand.
  • Generalization: Training across vehicle densities enables learned controllers to generalize to densities outside the training regime and closely track the performance upper bound during testing.They also stabilized settings with human-model noise during testing despite training without that noise.

4) Partial observability eases controller learning:

The study uses partial observations to reflect near-term AV deployments and to compare fairly with partially observed model-based controllers. Under this setting, learned controllers can achieve near-optimal traffic control with substantially shorter training and without requiring recurrent memory.

  • Partial observability eases controller learning: Partial observation is used because it better reflects near-term AV deployments and supports fair comparison with model-based controllers.The framework can compose observation components to study additional scenarios.
  • Partial observability eases controller learning: Training time decreases from around 24 hours with full observations to 6 hours with partial observations.The authors describe this as a finding warranting further investigation.
  • Partial observability eases controller learning: Partial observations still permit a near-global optimum, with the MLP closely mirroring the GRU and optimal velocity curve across vehicle densities.The result suggests memory is not necessary for near-optimal velocity with a single learned controller.
  • Open questions: The explanation for faster partial-observation training remains uncertain, and the conditions under which it yields globally optimal solutions require further study.The authors suggest fewer network weights may require fewer samples and iterations to reach a local optimum.
  • Interpreting the controllers: The MLP regulates speed and headway toward uniform flow, matching a 4.2 m/s leader at a 12 m headway.Its heatmap is a two-dimensional slice of a three-dimensional controller, with color representing acceleration.
  • Interpreting the controllers: IDM accelerates and decelerates more aggressively than learned controllers, continuing to accelerate at higher speed until headway becomes very small.This behavior results in stop-and-go traffic.

VII. REUSABLE MODULES FOR MIXED AUTONOMY

The framework composes reusable traffic modules to study mixed-autonomy scenarios with learned control laws, using human driving as the baseline. Experiments cover multiple AVs, with performance gains tied to platooning and increased roadway utilization.

  • Reusable mixed-autonomy modules: The modules can be composed to study multiple AVs, lane changes, and intersections beyond commonly studied traffic scenarios.Larger-scale scenarios remain outside the article’s scope because of current deep RL sample-efficiency limitations.
  • Experimental setup: Learned controllers are compared against human driving modeled by IDM, using average system velocity and control cost as performance criteria.The experiments use a memory-less diagonal Gaussian MLP control law.
  • Single-lane multiple-AV results: A string of consecutive AVs learns smaller headways than human-driver models, increasing roadway utilization and surpassing the prior performance bound.This behavior produces higher average velocity on the single-lane circular track.
  • Single-lane multiple-AV results: 3.70 m/s is the average velocity with three AVs, increasing to 4.44 m/s with 11 AVs on the single-lane circular track.Average velocity exceeds the uniform-flow equilibrium velocity as the number of AVs increases.

B. Multi-lane track with multiple autonomous vehicles

The framework extends to multi-lane and intersection scenarios, where learned controllers address lane-changing and right-of-way dynamics. Results show improved velocity, wave avoidance, and gains even at low autonomy levels, while broader deployment remains future work.

  • B. Multi-lane track with multiple autonomous vehicles: The multi-lane experiment uses a 230 m two-lane circular track with 44 vehicles and six AVs initially placed together in the outer lane.All vehicle positions, velocities, and lanes are observed.
  • B. Multi-lane track with multiple autonomous vehicles: 3.66 m/s average velocity improves on the 3.45 m/s uniform-flow equilibrium, while learned AVs balance across lanes and avoid stop-and-go waves.The controller uses a continuous representation despite combining acceleration and lane-change inputs.
  • C. Intersection with mixed and full autonomy: At an intersection, one autonomous vehicle makes vehicles move 1.5 times faster, while full autonomy produces an improvement of almost three times.The intersection has no traffic light; vehicles follow SUMO’s right-of-way model.
  • Scope and future work: Further study is needed to interpret learned behaviors and control laws before real-world deployment and policy analysis.The article identifies larger and more complex traffic networks and additional RL capabilities as open research directions.

APPENDIX A NETWORKS

Flow provides user-defined network primitives for modeling diverse closed and open traffic settings. The appendix describes canonical network types, intersection dynamics, and the IDM car-following model used in the experiments.

  • Network primitives: Flow supports arbitrary user-defined networks and includes closed and open networks designed to capture important traffic phenomena.Examples include circular tracks, figure-eight networks, merge networks, intersections, and highways.
  • Closed networks: Single-lane circular tracks provide a canonical 230 m-inspired benchmark, while multi-lane tracks add lane-changing behavior that complicates analytical control.Many classical techniques are therefore constrained to single-lane settings.
  • Closed networks: Figure-eight networks model perpendicular-track intersections governed by SUMO’s right-of-way model to prevent crashes.Loops with merges support merging studies and typically create congestion at merge points.
  • Open networks: Intersection modules are open networks with Poisson vehicle arrivals, where control-zone vehicles are sped up or slowed down to maximize velocity or minimize delay.These modules can build schemas for arbitrary maps.
  • Car-following model: IDM represents realistic driver behavior and traffic waves, and supplies the model used for numerical experiments and theoretical performance bounds.Its desired-headway function depends on parameters including h0, v0, T, δ, a, and b.

1) FollowerStopper:

The FollowerStopper commands an autonomous vehicle toward a desired velocity while using the lead vehicle’s speed and headway to maintain safety. A related PI-with-Saturation controller estimates equilibrium velocity from its own history.

  • 1) FollowerStopper: FollowerStopper commands the AV to maintain desired velocity U while ensuring it does not crash into the vehicle behind it.Its command velocity uses the leading vehicle’s speed and the AV’s headway within specified boundaries.
  • 1) FollowerStopper: The PI with Saturation controller estimates average equilibrium velocity U from the vehicle’s temporal speed history and drives toward that target.Its command velocity also incorporates crash-prevention constraints.
Loading 1710.05465v4…