PaperlayerView exact revision on arXiv

Source-linked AI summary

Can ChatGPT Enable ITS? The Case of Mixed Traffic Control via Reinforcement Learning

Michael Villarreal, Bibek Poudel, Weizi Li

arXiv:2306.08094v2cs.AIcs.HCcs.LGcs.RO

TL;DR

Novices face a demanding task in formulating RL components for traffic-control goals, and the paper studies ChatGPT-assisted MDP design in mixed-traffic environments. ChatGPT improves successful-policy counts in some environments but not consistently across all three.

  • Problem

    Designing MDP components that align RL agents with traffic-control goals is challenging and often requires domain expertise and experimentation.

    Source: §I · p. 1

  • Method

    The study examines non-experts designing RL state spaces and reward functions for three mixed-traffic environments, comparing ChatGPT-assisted work with novice-only work.

    Source: §VI · p. 6

  • Results

    ChatGPT increased successful policies by 150% in intersection and 135% in bottleneck, but did not increase successes in ring.

    Source: §VI · p. 6

  • Takeaways & Limitations

    ChatGPT’s contribution to novice mixed-traffic policy design varies across environments rather than yielding consistent improvements.

    Source: §VI · p. 6

    The reported gains are environment-dependent: ChatGPT did not increase successful policies in the ring environment.

    Source: §VI · p. 6

Abstract

from arXiv · show

The surge in Reinforcement Learning (RL) applications in Intelligent Transportation Systems (ITS) has contributed to its growth as well as highlighted key challenges. However, defining objectives of RL agents in traffic control and management tasks, as well as aligning policies with these goals through an effective formulation of Markov Decision Process (MDP), can be challenging and often require domain experts in both RL and ITS. Recent advancements in Large Language Models (LLMs) such as GPT-4 highlight their broad general knowledge, reasoning capabilities, and commonsense priors across various domains. In this work, we conduct a large-scale user study involving 70 participants to investigate whether novices can leverage ChatGPT to solve complex mixed traffic control problems. Three environments are tested, including ring road, bottleneck, and intersection. We find ChatGPT has mixed results. For intersection and bottleneck, ChatGPT increases number of successful policies by 150% and 136% compared to solely beginner capabilities, with some of them even outperforming experts. However, ChatGPT does not provide consistent improvements across all scenarios.

I. INTRODUCTION

Designing effective RL objectives and MDP components for traffic tasks is difficult for novices, and the study tests whether ChatGPT can help them formulate state and reward components across three mixed-traffic environments. ChatGPT’s results vary by environment, with more successful policies in some settings but no increase in ring success.

I. INTRODUCTION

  • Effective RL MDP formulation requires designing components such as state, action, reward, and discount factor, a tedious process often requiring domain expertise and experimentation.The paper notes that no standard general-purpose MDP technique works across environments and goals.Source: §I · p. 1
  • LLMs’ broad knowledge, commonsense priors, and creative capacity may help reduce the technical barrier of aligning MDP components with traffic-control strategies.The paper situates this opportunity against a nascent literature on using LLMs to assist RL problems.Source: §I · p. 1
  • The study compares 70 ITS novices who formulate state and reward components with or without unrestricted ChatGPT prompting across ring, intersection, and bottleneck scenarios.The resulting MDPs are used to train policies and evaluate performance.Source: §I · p. 1
  • ChatGPT increased successful policies in bottleneck and intersection, but did not increase ring policy success.The introduction reports increases of 150% and 136% for bottleneck and intersection, respectively.Source: §I · p. 1

II. RELATED WORK

Prior work uses LLMs in RL for robotics, reward specification, and policy replacement, but the authors identify a lack of direct evaluation of LLM assistance for novices designing MDP components. The paper introduces mixed-traffic RL as a POMDP and uses PPO to learn policies.

II. RELATED WORK

  • Prior work has used LLMs for robotics planning and navigation, exploration guidance, reward-function proxies, and policy replacement, but not directly evaluated novice MDP design assistance against expert policies.The related-work section identifies this as a gap in minimizing human effort to generate effective policies.Source: §II · p. 2

III. PRELIMINARIES

  • Mixed traffic control is modeled as a POMDP whose tuple specifies state, action, transitions, reward, initialization, discounting, horizon, observations, and observation probabilities.The robot vehicle acts according to a policy, receives a reward, and transitions to a next state.Source: A. Reinforcement Learning · p. 2

A. Reinforcement Learning

B. Mixed Traffic Control Environments

The environments pose distinct traffic-control objectives: preventing stop-and-go waves on a ring and improving outflow through a bottleneck. Their configurations and expert reward design provide context for the participant-created MDPs.

1) Ring:

  • The ring is a single-lane loop with 22 vehicles, where one robot vehicle aims to prevent stop-and-go waves caused by human-driving perturbations.The expert reward combines incentives for high average velocity and low control actions.Source: 1) Ring · p. 2

2) Bottleneck:

  • The bottleneck models capacity drop as outflow decreasing after inflow exceeds a threshold, and the robot vehicles’ objective is to improve outflow.The road narrows from 4×l to 2×l to l, and robot vehicles comprise 10% of traffic.Source: 2) Bottleneck · p. 2

XHV,V HV,XRV,V RV,o20

The section describes expert state and reward designs for bottleneck and intersection traffic control. At the intersection, the state represents vehicle positions, edge assignments, density, and speed, while the reward penalizes delay and standstills.

XHV,V HV,XRV,V RV,o20

  • The expert bottleneck state uses mean positions and velocities of both vehicle types across user-defined road segments, and its reward encourages increased bottleneck outflow.Source: XHV,V HV,XRV,V RV,o20 · p. 3

3) Intersection:

  • The expert intersection state includes each vehicle’s distance to the intersection and edge, plus edge density and average edge velocity.There are eight edges for each direction on both sides of the intersection.Source: 3) Intersection · p. 3
  • The expert intersection reward penalizes vehicle delay and traffic standstills.Its definitions identify t as the current timestep and ssn as the number of standstill vehicles.Source: 3) Intersection · p. 3

IV. USER STUDY

The user study recruited 70 graduate students from AI or machine-learning courses with minimal RL coverage and compared cohorts with and without ChatGPT. The control group’s answers commonly used intuitive collision-prevention and objective-oriented metrics; the supplied table reports validity counts by environment and group.

A. Participants and Cohorts

  • The authors found no discernible difference between the two courses’ results despite slightly greater RL knowledge among AI participants.The authors attribute this to the manuscript and ChatGPT’s assistance.Source: A. Participants and Cohorts · p. 3
  • The study recruited 70 graduate students, and the control and study cohorts included 38 and 32 participants, respectively.Participants came from AI or machine-learning courses with minimal RL coverage; the study cohort also had ChatGPT access.Source: A. Participants and Cohorts · p. 3
  • Table II organizes valid and invalid answer counts by traffic environment and control versus study group.Source: Table II · p. 3
  • The control group commonly used collision-prevention metrics and consistent vehicle position, velocity, and acceleration measures, alongside environment-specific objective metrics.Bottleneck answers often included outflow or speed metrics; intersection answers included queue length or standstill time by direction.Source: A. Participants and Cohorts · p. 3
  • Study-group ChatGPT-use selections ranged from daily use by two participants to never by three participants.The most frequent response was several times a week, selected by 13 participants.Source: A. Participants and Cohorts · p. 3

B. ChatGPT Setup

The study used a fixed GPT-4 setup and a timed manuscript-reading procedure. Participants received conduct and background-question instructions before completing the study.

B. ChatGPT Setup

  • Study-group participants used GPT-4 with an 8k context length and temperature 0.7, without fine-tuning.Source: B. ChatGPT Setup · p. 3
  • The study used the March 14, 2023 version of GPT-4 through a fresh TypingMind chat interface.Source: B. ChatGPT Setup · p. 3
  • Participants used the same ChatGPT conversation throughout the study to enable in-context learning.Source: B. ChatGPT Setup · p. 3

C. Manuscript

  • The study lasted 95 minutes, with participants required to spend the first 25 minutes reading the first three manuscript sections.They could use the remaining time to review the entire manuscript.Source: C. Manuscript · p. 3
  • The manuscript versions differed for the two cohorts and contained four sections.Source: C. Manuscript · p. 3

1) General Instructions and Background Questions:

2) Reinforcement Learning Overview:

The manuscript introduces state spaces and reward functions with transportation examples, then gives participants a metric bank and environment-specific tasks. The example figures illustrate novice prompting and contrast a conventional novice state space with a ChatGPT-assisted reward function.

2) Reinforcement Learning Overview:

  • The RL overview explains state spaces and reward functions using transportation-related examples to establish common understanding.Source: 2) Reinforcement Learning Overview · p. 3
  • In the ring examples, novices rely on ChatGPT for reward functions, while one asks whether an existing state space is sufficient and another delegates state-space design.The second novice also prompts ChatGPT to include time-to-collision based on their prior conversation.Source: Figure 2 · p. 4
  • The ring comparison shows a control novice using provided metrics in the state space, while a ChatGPT-assisted reward contains complex generated terms with reasoning.Source: Figure 3 · p. 4

3) Answer Instructions and Bank of Metrics:

4) Mixed Traffic Environment Descriptions and Questions:

  • For each ring, bottleneck, and intersection environment, participants supplied a state space, reward function, and brief rationale for the reward.The materials included a general description, problem explanation, and mixed-traffic objective for each environment.Source: 4) Mixed Traffic Environment… · p. 4

V. RESULTS

The participant-answer analysis finds that ChatGPT-assisted novices produced more valid state spaces and reward functions and used more new metrics than novices relying on prior knowledge. Examples show participants often asked ChatGPT to suggest reward functions, while its generated rewards could include intricate metrics with explanations.

A. Participant Answers Analysis

  • 82% of study-group answers were valid versus 61% for the control group, a 21% increase.Answers were considered valid when they included metrics from the metrics bank or gave well-defined explanations.Source: A. Participant Answers Analysis · pp. 4–5, Table III · p. 4
  • The study group used 363% more new metrics on average than the control group, with 35, 63, and 59 versus 8, 17, and 21 in ring, bottleneck, and intersection, respectively.Source: A. Participant Answers Analysis · p. 5
  • Participants generally asked ChatGPT for state-space suggestions after giving it environment context, then relied on it heavily for reward-function suggestions.One common prompt requested a specific metric, such as time-to-collision, in the reward function.Source: A. Participant Answers Analysis · p. 5
  • A sample ChatGPT-generated reward used intricate metrics and offered an explanation for rewarding safe time-to-collision with front and rear vehicles.The example applies the reward only when actual time-to-collision is below a specified safe threshold.Source: A. Participant Answers Analysis · p. 5

B. Experiment Setup

In the ring environment, ChatGPT assistance did not increase policy success, despite improving answer validity and introducing new metrics. Training rewards rose across groups, but higher rewards did not guarantee that the robot vehicle met the environment’s objective.

B. Experiment Setup

  • Each valid answer was trained as a policy using PPO for 200 episodes, with IDM-driven human vehicles and environment-specific two-layer network sizes.The hidden layers had size 8 for ring and intersection and size 16 for bottleneck; human-vehicle driving included stochastic noise from −0.2 to 0.2.Source: B. Experiment Setup · p. 5

1) Training:

  • Rewards improved during training for both groups in all three environments, but increasing reward did not guarantee the robot vehicle achieved its objective.The robot vehicle could pursue actions that raised reward without aligning with the environment’s goal.Source: 1) Training · p. 5

2) Ring:

  • 5 policies succeeded with ChatGPT assistance, versus 7 using only non-expert knowledge; none reached the expert’s level.The authors conjecture that new metrics may not improve prevention of stop-and-go traffic and that participants’ degree of ChatGPT use varied.Source: 2) Ring · p. 5

3) Bottleneck:

ChatGPT-assisted policies increased successes in bottleneck and intersection, but the gains varied by scenario and many policies still did not succeed. Training curves show that both novice groups developed trainable state spaces and reward functions.

3) Bottleneck:

  • Bottleneck policies were successful when outflow exceeded 1400 vehicles per hour; neither novice group produced a policy that outperformed the expert.Each trained policy was tested 10 times, with average outflow reported.Source: 3) Bottleneck · p. 5

4) Intersection:

  • Intersection success was assessed using average vehicle speed and east/westbound queue length, with nearest neighbors to the expert considered successful.Four ChatGPT-assisted policies outperformed the expert by a significant margin.Source: 4) Intersection · pp. 5–6
  • Across all networks, average rewards increased during training for both novice groups, indicating they developed trainable state spaces and reward functions.The curves show averages for Novices and Novices + ChatGPT, with shaded regions representing variance.Source: Figure 4 · p. 6

3) Bottleneck:; 4) Intersection:

  • Intersection successes rose from four to six with ChatGPT (150% increase), including four policies outperforming the expert; bottleneck successes rose from 14 to 19 (136% increase).The figure compares successful policies across all three environments.Source: 4) Intersection · pp. 5–6, 3) Bottleneck · p. 5, Figure 5 · p. 6

4) Intersection:

  • ChatGPT assistance was participant-determined, and many policies involving extensive assistance were unsuccessful.The authors note that ChatGPT’s training set was unavailable and suggest its RL or ITS training data may have been insufficient.Source: 4) Intersection · p. 6

VI. CONCLUSIONS

The study found that ChatGPT-assisted novices had more successful policies in bottleneck and intersection, but not ring. The authors note that the improvement was less than expected and outline further study directions.

VI. CONCLUSIONS

  • ChatGPT increased successful policies by 150% in intersection and 135% in bottleneck, but did not increase successes in ring.The authors suggest insufficient exposure to RL or ITS problems during ChatGPT training as a possible explanation for the lower-than-expected improvement.Source: §VI · p. 6
  • The authors propose expanding the study to more participants, traffic environments, and LLMs, and exploring fine-tuning to reduce human effort for ITS.They also identify comparative analysis of different LLMs as a possible direction.Source: §VI · pp. 6–7
  • Successful-policy examples show RVs creating gaps to prevent stop-and-go traffic, slowing to ease bottleneck congestion, and yielding so vehicles can cross safely.These are examples from ring, bottleneck, and intersection, respectively.Source: Figure 6 · p. 7
Loading 2306.08094v2…