PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim2Signal: Sim-to-Real Benchmarks for Traffic Signal Control

Ferdous Al Rafi, Susrik Mukherjee, Latika Liladhar Dekate, Jennifer Yawa Lavoe, Huaiyuan Yao, Shlok Mohanty, Longchao Da, Xuesong Zhou, Hua Wei

arXiv:2609.01676v1cs.LG

TL;DR

Sim-to-Real transfer remains difficult because traffic-signal policies trained in simulation can fail under changed real-world sensing, actions, dynamics, and objectives, while mitigation methods lack systematic benchmarking. Sim2Signal isolates these four gaps under a shared protocol and finds that mitigation effectiveness varies by network and setting, with estimation-based methods generally recovering performance more consistently than generic robustness approaches.

  • Problem

    Traffic-signal policies that perform strongly in simulation can fail after real-world deployment, and existing mitigation methods lack systematic comparative evaluation across Sim-to-Real gap sources.

  • Method

    Sim2Signal decomposes the Sim-to-Real gap into observation, action, transition, and reward gaps and induces each gap in isolation across controlled environments under a shared protocol.

  • Results

    Mitigation effectiveness depends on the gap configuration and network conditions rather than gap category or severity alone, while methods estimating what the gap changed generally recover performance more consistently.

  • Takeaways & Limitations

    Sim-to-Real mitigation should identify and correct the specific deployment gap instead of relying primarily on generic domain randomization or invariant representations.

  • Takeaways & Limitations

    The calibration assumes shared network layout, demand, and signal settings between simulation and real environments, with induced gaps applied afterward.

Abstract

from arXiv · show

Reinforcement learning achieves strong traffic signal control performance in simulation, yet policies trained in simulators often fail once deployed in the real world, a failure known as the Sim-to-Real gap. When RL is applied to traffic signal control, this gap arises from several sources: sensing, action execution, traffic dynamics, and the control objective. Their relative impact and the reliability of existing Sim-to-Real mitigation methods remain insufficiently understood, and the field lacks a standard benchmark for systematically measuring the gap and evaluating mitigation methods. We present Sim2Signal, a benchmark that decomposes the Sim-to-Real gap into observation, action, transition, and reward gaps, corresponding to mismatches in the four components of the underlying MDP, and induces each gap in isolation under a shared protocol. We evaluate 18 mitigation methods on 2 base controllers, across 33 gap settings and 10 calibrated networks built from 5 real-world locations. We find that direct transfer consistently degrades performance across all four gap sources, but the severity of the degradation does not predict the effectiveness of mitigation. Instead, mitigation effectiveness depends strongly on the network and gap setting: outside the action gap, a method that helps in one case may fail in another. The most effective methods generally estimate what the gap changes, rather than make the policy insensitive through domain randomization or invariant representations. Our code is available at https://github.com/Red-Pheonix/Sim2RealTSCBenchMark

1. Introduction

Sim2Signal addresses the poorly understood Sim-to-Real gap in traffic signal control by decomposing it into four MDP-component gaps and evaluating mitigation methods under a shared protocol. The benchmark shows that all gaps can degrade transfer, while mitigation success depends on the gap setting and network.

  • Motivation: RL-based traffic signal controllers often perform well in simulation but remain difficult to deploy reliably in real-world traffic systems.Simulation can simplify sensing, action execution, traffic dynamics, and control objectives.
  • Benchmark: The benchmark decomposes the Sim-to-Real gap into observation, action, transition, and reward gaps corresponding to the four MDP components.A second controlled environment plays the role of the real world so each discrepancy can be induced in isolation.
  • Findings: Every gap degrades deployed-controller performance, but degradation magnitude does not predict how much mitigation recovers.This finding separates the severity of transfer failure from the effectiveness of a mitigation method.
  • Findings: Effective methods generally estimate what the gap changes rather than making policies insensitive through domain randomization or invariant representations.Examples include estimating altered states, dynamics, action execution, or objectives.
  • Benchmark: Sim2Signal evaluates 18 mitigation methods with two base controllers across 33 gap settings and ten calibrated networks from five real-world locations.The shared protocol enables comparisons across gap sources, networks, settings, and controllers.
  • Findings: Action-gap methods work across the tested networks and settings, whereas methods for the other three gaps can fail when the network or setting changes.In some cases, every method performs worse than applying no mitigation.

2. Related Work

Prior traffic-signal-control research largely evaluates controllers under fixed or idealized conditions, while Sim-to-Real mitigation is rarely benchmarked systematically. Sim2Signal makes each gap source a controlled variable and evaluates mitigation methods directly.

  • RL for TSC: Most RL traffic-signal-control evaluations measure performance under the conditions used for training rather than after those conditions change.This limits what such results say about deployment.
  • Sim-to-Real Mitigation for TSC: Existing Sim-to-Real mitigation methods typically assume a known gap source and design a mechanism specifically for observation, action, or transition discrepancies.Examples include sensor randomization, admissible-action constraints, grounded action transformation, and meta-learning.
  • Deployment-Oriented TSC: Deployment-oriented work improves fidelity to specific sites using calibrated digital twins, real demand data, faulty-detector state estimation, admissible signal programs, or hardware-in-the-loop execution.These approaches address practical deployment obstacles such as imperfect detection, regulatory constraints, safety requirements, and costly online exploration.
  • TSC Benchmarks: Existing benchmarks standardize controller comparisons through fixed scenarios, multi-simulator APIs, real-intersection settings, or scaled physical endpoints rather than systematic transfer measurement.Examples include RESCO, SUMO-RL, TSLib, LibSignal, PyTSC, and deployment-oriented settings.
  • Research Gap: Across these research strands, Sim-to-Real gaps are usually identified or reduced but not benchmarked, leaving mitigation methods themselves unevaluated under controlled changes.Sim2Signal instead treats each gap source as a controlled variable and mitigation methods as the evaluation target.

3. The Sim2Signal Benchmark

Sim2Signal uses a shared sim-to-sim protocol to isolate four MDP-component gaps and evaluate mitigation across calibrated traffic networks and base controllers. Its settings cover sensing, action execution, traffic dynamics, and reward-objective mismatches.

  • Shared protocol: Sim2Signal treats Esim as the training environment and a modified Ereal as the real-world counterpart, inducing each gap in isolation.Policies train in Esim, while Ereal supplies deployment rollouts; the reward oracle is the sole exception to the no-real-reward-training rule.
  • Shared protocol: The benchmark evaluates DQN and PressLight controllers under a shared pretrain, train, and deploy pipeline.All methods start from one shared pretrained checkpoint per network-controller pair and have bounded episode budgets.
  • Benchmark coverage: Sim2Signal spans five locations and ten calibrated networks, including single- and multi-intersection settings and two new US datasets.Tempe and Bullhead networks are built from real UTDF signal-plan data with NEMA dual-ring configurations and timings.
  • Observation gaps: The benchmark induces observation gaps through noise, detector failures, detection-zone changes, and combined corruptions.Noise levels vary across four σ values, detector failure probabilities across four p values, and detection zones from 10 to 100 m.
  • Action gaps: Action gaps include countdown delays and restricted phase transitions, with delay-aware prediction and grounded action methods evaluated as mitigations.Ereal delays range from 20 to 60 s, while NEMA settings restrict phase successors using minimum green, clearance, and fixed-sequence rules.
  • Transition and reward gaps: Transition gaps vary driver, vehicle-loading, and weather dynamics, while reward gaps reweight objectives and add real-only emissions and safety components.The study evaluates domain randomization, adaptation, grounding, reward inference, multi-objective RL, and dynamic reward shaping across these settings.

4. Benchmarking Results

Sim2Signal evaluates gap severity, mitigation effectiveness, network variation, and controller robustness under a shared benchmark protocol. Performance degradation occurs across all gap sources, but mitigation success depends on the gap setting and network, with action gaps more consistently recoverable than the other gaps.

  • RQ1: Gap Severity: Direct-Transfer degrades performance under every gap source, with action delay most severe and restricted phase transitions among the least severe.Observation and transition gaps fall between these extremes, while observation severity also varies by corruption type.
  • RQ2: Mitigation Effectiveness: Mitigation effectiveness does not follow gap severity: the most severe delay gap is reliably recoverable, while less severe gaps can remain unreliable.Delay is most severe, whereas phase transitions are among the least severe, yet both action gaps shrink reliably.
  • RQ2: Mitigation Effectiveness: Outside the action gaps, the same method can improve one network and setting but worsen another, and sometimes all methods underperform no mitigation.The observation, transition, and reward gaps therefore lack improvements that hold across every tested deployment.
  • RQ2: Mitigation Effectiveness: Methods that estimate what a gap changed generally recover more consistently than methods that only make policies insensitive to the change.Examples include predicting delayed-action states and reconstructing true sensor readings; the pattern is not universal across all gap settings.
  • RQ3: Network Conditions: Congested networks tend to have the largest gaps and the most recovery, while mitigation can harm performance on the least congested network.On Ingolstadt 1, the best method does not improve on Direct-Transfer and several settings worsen after mitigation.
  • RQ4: Controller Robustness: The two controllers agree on 76% of network-setting pairs for ATT gap and 70% for reward regret, with correlations r = 0.79 and r = 0.91.The remaining gap after mitigation is mostly attributed to the network and setting rather than the controller.

5. Conclusion

Sim2Signal provides a controlled benchmark for isolating four Sim-to-Real gaps in traffic signal control and testing mitigation across deployment conditions. Its results show that network and setting determine mitigation reliability more than gap category or severity alone.

  • Conclusion: Sim2Signal decomposes traffic-signal-control Sim-to-Real transfer into observation, action, transition, and reward gaps under controlled evaluation.The benchmark is designed to test whether mitigation generalizes across networks, settings, and base controllers.
  • Conclusion: Mitigation effectiveness depends on gap configuration and network conditions, while gap severity is not a reliable indicator of recovery.This conclusion summarizes the benchmark’s cross-condition findings.
  • Conclusion: Methods that estimate altered states, dynamics, action execution, or objectives generally recover more consistently than domain randomization or invariant representations.The paper therefore motivates methods that identify and correct the specific deployment gap.
  • Conclusion: Sim2Signal offers a standardized testbed for evaluating whether mitigation methods generalize across networks, settings, and base controllers.Future extensions include additional gap sources and simultaneous multiple gaps.

A. Benchmark Details

The benchmark uses calibrated environments, shared checkpoints, explicit metrics, and fixed configurations to support controlled comparisons. Its calibration indicates that induced gaps dominate residual simulator-pair differences.

  • Scale: The benchmark reports networks, induced settings, methods per gap, and deployment-run counts in its scale tables.Method identifiers are also mapped to the full-results tables.
  • Metrics: The benchmark reports ATT, throughput, average queue length, average delay, reward, and phase-transition metrics.ATT is the main analysis metric because queue and throughput produce the same method rankings.
  • Protocol: Each episode spans 3600 steps, and pretraining uses 200 nominal episodes per network-controller pair before mitigation training starts from a shared checkpoint.This protocol reduces confounding from pretraining variance.
  • Implementation Details: Appendix tables specify observation corruptions, transition parameters and randomization distributions, hidden reward weights, and individual method implementations.These tables provide the detailed settings behind the benchmark’s controlled gap construction.
  • Reproducibility: Every reported run uses seed 0 with deterministic Python, NumPy, PyTorch, CUDA, and cuDNN settings, while configurations fix method hyperparameters.A command-line flag permits rerunning experiments with another seed.

A.1 Benchmark Formulation

Sim2Signal models traffic signal control as an MDP and decomposes Sim-to-Real mismatch across observation, action execution, transition dynamics, and reward. It varies these components under a shared protocol, using deployment metrics for the first three gaps and reward regret for the reward gap.

  • MDP formulation: At each step, policy observations produce requested actions, the environment executes them, transitions the state, and evaluates the resulting reward.Distinguishing requested from executed actions represents deployment delays and signal-transition constraints.
  • Gap decomposition: The benchmark defines observation, action, transition, and reward gaps as mismatches in state measurement, action execution, traffic dynamics, and objectives.These correspond to the four components of the underlying MDP control loop.
  • Benchmark scale: The transition-gap benchmark includes 10 networks, five settings, two base controllers, and an additional JL-GAT block restricted to five multi-intersection networks.JL-GAT contributes 40 runs beyond the 400 runs from the other five transition methods.
  • Evaluation protocol: Observation, action, and transition gaps use a shared deployment metric to measure performance changes between Esim and Ereal.The reward gap changes the metric itself, so it is evaluated using regret against a policy trained directly on the deployment objective.

A.2 Method Details

Sim2Signal evaluates gap-specific mitigation families from a shared pretrained controller across observation, action, transition, and reward settings. The protocol includes adaptation, prediction, masking, reward inference, multi-objective optimization, and reference controllers.

  • Observation: Observation methods use domain randomization, latent-space domain adaptation, or reconstruction to handle corrupted sensor inputs.Latent-space methods include LUSR, DARLA, ATC, and a plain VAE; reconstruction feeds the policy a denoised observation.
  • Action: delay: Delay methods compare forward prediction with training and action-selection ablations that isolate the effects of delay-aware training.Delayed-Q rolls a forward model through pending actions, while PRLight predicts post-countdown traffic features in one step.
  • Protocol and references: All methods follow shared pretrain, train, and deploy stages, while Direct-Transfer and the reward oracle provide reference conditions.The benchmark starts methods from shared pretrained checkpoints and evaluates deployment in Ereal.
  • Action: phase transitions: Restricted-transition methods combine training-time adaptation with optional deployment-time shielding that masks illegal actions.The benchmark evaluates domain randomization and grounded action transformation with and without shielding.
  • Transition: Transition methods randomize or infer car-following parameters and use grounded action transformation across traffic-dynamics settings.Randomization resamples headway, gap, acceleration, deceleration, and vehicle-length parameters each episode.
  • Reward: Reward methods infer real objective weights, select among simulator-trained scalarizations, or search weights with dynamic reward shaping in Ereal.The reward oracle is trained directly on the real objective and serves as an empirical upper reference.

A.3 Gap Design

The benchmark grounds each induced gap in real deployment phenomena and calibrated settings. Observation corruption, action delays and restrictions, traffic dynamics, and reward objectives are varied across networks and setting families.

  • Observation gap: Observation settings model noisy readings, stuck-at-zero sensor failures, and limited detection zones using field-motivated severity ranges.Failure probabilities range from 0.05 to 0.7, while detection distances span 10 to 100 m.
  • Action gap: delay: Delay settings apply constant real-world action delays of 20, 30, 40, or 60 seconds to decisions made every 10 seconds.These settings place two to six decisions in flight and test methods assuming a fixed 20-second countdown.
  • Action gap: phase transitions: Phase-transition settings encode real NEMA rules, including dwell-time constraints, barriers disabling roughly 43% of transitions, and cyclic successors.Only the Tempe and Bullhead locations provide real signal plans for this gap.
  • Transition gap: Transition settings perturb SUMO car-following behavior to represent loading and weather effects on startup delay, acceleration, and braking.The simulator pair itself contributes a residual gap, while additional settings use calibrated Krauss parameters.
  • Reward gap: Reward settings vary the hidden deployment objective, including an efficiency-aligned sanity check and heavier weights on terms unavailable to simulator-trained policies.Emission and TTC components are computable only in SUMO.

A.4 Reward Metrics

The benchmark scores hidden deployment objectives with normalized traffic, emissions, safety, and fairness components. Metrics are designed to capture real-world concerns while accounting for simulator-specific observability and collision handling.

  • Objective construction: The reward combines normalized cost components with weights expressing relative importance rather than unit conversions.Queue, delay, and waiting are standard components; other components are defined separately.
  • Emissions: CO2 and fuel are accumulated from SUMO’s HBEFA-based emission model and are computable only in Ereal.Simulator-trained policies can reach these quantities only through correlated proxies.
  • Safety: Safety uses surrogate rear-end conflicts when time to collision falls below 1.5 s, because SUMO resolves would-be crashes as emergency stops.Each vehicle is counted at most once per episode and conflicts are attributed to the controlling intersection.
  • Fairness: Fairness is the largest pairwise throughput gap among approaches with waiting vehicles, with equal weighting across demand-active approaches.The cost is zero when fewer than two approaches are demand-active.
  • Logged components: Fuel, emergency stops, and collisions are logged in reward-gap runs but receive zero weight in deployment objectives.Collisions carry little signal because SUMO resolves would-be crashes as emergency stops.

B. Full Results

The results report ATT gaps and absolute real-environment metrics across observation and transition settings, using top-five checkpoints and non-RL reference controllers for comparison. Reward tables instead report the hidden real objective, while reference gaps require cautious interpretation because simulator calibration differs.

  • Evaluation protocol: ATT gaps are reported as mean ± std over the top five evaluation checkpoints, with lower values better and Direct-Transfer evaluated deterministically.The tables define the gap as mean real ATT minus calibrated simulator ATT.
  • Reference controllers: Fixed-Time and MaxPressure are deployed directly in Ereal as non-RL references, but their gaps are not calibrated across simulators.A large negative reference gap can indicate simulator disagreement under congestion; bracketed real ATT is the relevant comparison.
  • Observation gap: Observation-gap tables cover sensor noise, detection-zone limits, sensor failure, and combined corruptions for DQN and PressLight.Each setting includes ATT-gap tables alongside core-metric tables.
  • Evaluation protocol: Core-metric tables report absolute Ereal ATT, average queue, average delay, and throughput for each run.These metrics are averaged over the top-five selected checkpoints.
  • Transition gap: Transition-gap tables evaluate DQN and PressLight under four car-following settings, with JL-GAT restricted to multi-intersection networks.The accompanying core-metric tables report absolute values in Ereal.
Loading 2609.01676v1…