PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning

Longchao Da, Minquan Gao, Hao Mei, Hua Wei

arXiv:2308.14284v6cs.AI

TL;DR

Simulator-trained traffic-signal policies face a performance gap in real-world settings, mainly because system dynamics differ. PromptGAT uses LLMs to profile those dynamics and ground policy actions; the authors report improved forward-model accuracy and transfer performance.

  • Problem

    Simulator-trained traffic-signal policies face a real-world performance gap mainly associated with differences between simulator and road-condition dynamics.

    Source: §1 · p. 1

  • Method

    PromptGAT uses LLM inference about weather, traffic states, and road types to model dynamics and inform grounded action transformation.

    Source: §1 · p. 2

  • Results

    The authors report that PromptGAT increases forward-model prediction accuracy and improves transfer performance.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The conclusion presents LLM-profiled real-world dynamics as support for grounded action transformation and more realistic traffic-signal policies.

    Source: §5 · p. 8

    Traditional GAT forward models require substantial real-world data covering the full state distribution, and may predict unobserved states inaccurately.

    Source: §1 · p. 1

Abstract

from arXiv · show

Numerous solutions are proposed for the Traffic Signal Control (TSC) tasks aiming to provide efficient transportation and mitigate congestion waste. In recent, promising results have been attained by Reinforcement Learning (RL) methods through trial and error in simulators, bringing confidence in solving cities' congestion headaches. However, there still exist performance gaps when simulator-trained policies are deployed to the real world. This issue is mainly introduced by the system dynamic difference between the training simulator and the real-world environments. The Large Language Models (LLMs) are trained on mass knowledge and proved to be equipped with astonishing inference abilities. In this work, we leverage LLMs to understand and profile the system dynamics by a prompt-based grounded action transformation. Accepting the cloze prompt template, and then filling in the answer based on accessible context, the pre-trained LLM's inference ability is exploited and applied to understand how weather conditions, traffic states, and road types influence traffic dynamics, being aware of this, the policies' action is taken and grounded based on realistic dynamics, thus help the agent learn a more realistic policy. We conduct experiments using DQN to show the effectiveness of the proposed PromptGAT's ability in mitigating the performance gap from simulation to reality (sim-to-real).

1 Introduction

Simulator-trained traffic-signal policies can lose performance in real-world settings because traffic dynamics differ. PromptGAT uses LLMs to profile those dynamics and inform grounded action transformation.

1 Introduction

  • Grounded Action Transformation addresses the sim-to-real gap by inducing simulator dynamics to resemble real-world dynamics while policy learning remains in simulation.The framework uses real-world data for dynamics learning.Source: §1 · p. 1
  • Traditional GAT forward models require substantial real-world data spanning the state distribution to predict real-world dynamics accurately.The forward model maps the current state and action to a possible next state.Source: §1 · p. 1
  • Conventional GAT may predict unseen real-world states inaccurately, especially in rarely represented extreme weather or rare events.Human knowledge can inform expected changes such as slower vehicle movement and smaller acceleration and deceleration under extreme weather.Source: §1 · p. 1
  • Traffic dynamics may make the same green-signal duration produce lower throughput, so policy action durations might need adjustment for similar performance.The introduction presents this as a human inference about real-world conditions.Source: §1 · pp. 1–2
  • PromptGAT incorporates LLMs into GAT to profile real-world traffic dynamics and support more realistic policy learning for sim-to-real transfer.Its prompts ask how weather, traffic states, and road types influence traffic dynamics.Source: §1 · p. 2

2 Related Work

Prior work addresses traffic-signal control and sim-to-real transfer through several approaches, while this paper focuses on changing policy output actions and applying prompt learning to transfer.

2 Related Work

  • Some traffic-signal RL studies address sim-to-real transfer by modifying the simulator, which requires simulator parameters that can match the real world.The cited examples are Müller et al. (2021) and Mei et al. (2022).Source: §2 · p. 2
  • Rather than modifying the simulator, this paper proposes modifying the output actions of policies learned in the traffic simulator.This is the paper’s stated distinction from simulator-modification approaches.Source: §2 · p. 2
  • Sim-to-real transfer includes domain randomization, domain adaptation, and grounding; in traffic-signal control, the gap is attributed mainly to dynamics rather than perception.GAT uses grounded actions to align simulator dynamics with the real world without requiring a parameterized simulator.Source: §2 · p. 2
  • Prompt learning has been widely studied in NLP, but the authors report no prior exploration of it in sim-to-real transfer or traffic-signal control.The passage describes prompts as instructions prepended to inputs to promote downstream tasks.Source: §2 · p. 2

3 Method

The method frames traffic signal control as an MDP and uses DQN within a grounded action transformation pipeline that aligns simulated transitions with real-world dynamics. PromptGAT adds LLM-inferred effects of weather, road type, and traffic state to the forward model, alongside simulation learning and real-world data collection.

3.1 Preliminaries

  • TSC is framed as an MDP in which each intersection agent selects a non-conflicting traffic phase to minimize average queue length over the next interval.The MDP comprises states, actions, transition dynamics, reward, and a discount factor.Source: RL-based Traffic Signal Control · pp. 2–3

RL-based Traffic Signal Control

Grounded Action Transformation

  • GAT transforms policy actions so simulated next states approach forward-model predictions of real-world states, aiming to reduce the transfer performance gap.The forward model predicts next states from real-world data; the inverse model maps predicted next states to actions in simulation.Source: Grounded Action Transformation · pp. 3–4

In-context Learning for Dynamics Knowledge

  • Prompt templates provide task instructions and context about weather, road conditions, and lane-level traffic; the LLM’s inferred dynamics knowledge is used by the forward model.The forward model’s direct state-and-action prediction does not itself account for domain knowledge such as weather or road conditions.Source: In-context Learning for Dynamics… · p. 4, Prompt-based Dynamics Modeling · p. 4

Prompt-based Dynamics Modeling

  • PromptGAT uses GPT-4.0 to infer acceleration, deceleration, emergency deceleration, and startup delay from traffic, weather, and road context, then fuses this knowledge with lane features.Road description and weather remain constant throughout a trajectory.Source: Prompt-based Dynamics Modeling · p. 4

Overall Algorithm

  • Algorithm 1 pretrains the policy in simulation, rolls it out in the real environment to collect data, then updates models and policy.The algorithm lists forward-model, inverse-model, and policy update steps.Source: Overall Algorithm · p. 5

4 Experiments

Experiments evaluate PromptGAT across four real-world SUMO settings against simulation-trained and baseline methods. PromptGAT reduces the sim-to-real performance gap, while further analyses associate improved forward-model accuracy with gap mitigation.

Environment Setup

  • The experiments use Cityflow as Esim and SUMO as Ereal, with settings representing ideal defaults, heavy-industry roads, and rainy or snowy conditions.V1 and V2 vary vehicle loading and startup delay; V3 and V4 represent rain and snow.Source: Environment Setup · p. 5, Table 1 · p. 5

Evaluation Metrics

  • Evaluation tracks average travel time, throughput, reward, queue length, and delay gaps, with similar Esim performance used for fair method comparisons.Smaller ATT, queue, and delay gaps and larger throughput and reward gaps are preferred.Source: Evaluation Metrics · pp. 5–6, Table 3 · p. 6

4.2 Experimental Results and Analysis

  • PromptGAT performs better than other baselines across various settings and metrics most of the time.Table 2 reports metric gaps and standard deviations over five runs.Source: Table 2 · p. 6, §4.2 · p. 7
  • Across four SUMO configurations, PromptGAT’s LLM inferences follow the same tendency as real settings across traffic-dynamics metrics.The prompt asks for acceleration, deceleration, emergency deceleration, and startup delay based on weather, road, and traffic context.Source: §4.2 · p. 6, Figure 4 · p. 7
  • PromptGAT shrinks the large direct-transfer performance gap across four real-world settings, bringing results closer to the well-trained simulation benchmark.The comparison shows direct-transfer results in orange, PromptGAT in purple, and simulation performance in blue.Source: Figure 5 · p. 7, §4.2 · p. 7

4.3 Further Study and Experiment Details

  • In the V4 case study, PromptGAT improves forward-model prediction over VanillaGAT, and accuracy improvements correlate positively with mitigation across travel-time, throughput, and queue gaps.Figure 7 reports strong correlations for accuracy improvement against each metric, except for minor outliers; the passage also reports correlations among the other metrics.Source: §4.3 · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 8

5 Conclusion

PromptGAT applies pre-trained LLM inference to grounded action transformation for traffic signal control, aiming to mitigate the sim-to-real performance gap. The paper reports improved real-world dynamics prediction and transfer performance.

5 Conclusion

  • PromptGAT uses pre-trained LLM inference to profile real-world traffic dynamics and mitigate the sim-to-real performance gap in traffic signal control.Source: §5 · p. 8
  • PromptGAT improves forward-model prediction accuracy, enabling grounded action transformation based on more accurate real-world dynamics.Source: §5 · p. 8
  • Experiments support the rationale of LLM inference and indicate PromptGAT's potential for addressing real-world policy deployment challenges.Source: §5 · p. 8
Loading 2308.14284v6…