PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim2Signal: Sim-to-Real Benchmarks for Traffic Signal Control

Ferdous Al Rafi, Susrik Mukherjee, Latika Liladhar Dekate, Jennifer Yawa Lavoe, Huaiyuan Yao, Shlok Mohanty, Longchao Da, Xuesong Zhou, Hua Wei

arXiv:2609.01676v1cs.LG

TL;DR

Evidence on how Sim-to-Real gaps affect traffic signal control and how reliably mitigation methods work remains limited. Sim2Signal isolates four gap types in a shared benchmark and finds that mitigation effectiveness depends on the gap configuration and network, while gap severity is not a reliable predictor.

  • Problem

    Sim-to-Real gaps and mitigation methods in traffic signal control have not been systematically benchmarked, leaving their relative impacts and mitigation reliability insufficiently understood.

    Source: §2 · p. 4

  • Method

    Sim2Signal isolates observation, action, transition, and reward gaps in a shared traffic-control benchmark and evaluates mitigation methods across networks, settings, and base controllers.

    Source: §1 · pp. 2–3

  • Results

    Gap severity is not a reliable indicator of mitigation effectiveness, which varies with gap configuration and network conditions; estimating what the gap changes generally yields more consistent recovery.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    The findings motivate methods that identify and correct the specific deployment gap rather than relying on generic robustness mechanisms.

    Source: §5 · p. 11

    Controlled attribution assumes calibration that makes no-gap ATT and throughput in Esim and Ereal match within reported residuals.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Reinforcement learning achieves strong traffic signal control performance in simulation, yet policies trained in simulators often fail once deployed in the real world, a failure known as the Sim-to-Real gap. When RL is applied to traffic signal control, this gap arises from several sources: sensing, action execution, traffic dynamics, and the control objective. Their relative impact and the reliability of existing Sim-to-Real mitigation methods remain insufficiently understood, and the field lacks a standard benchmark for systematically measuring the gap and evaluating mitigation methods. We present Sim2Signal, a benchmark that decomposes the Sim-to-Real gap into observation, action, transition, and reward gaps, corresponding to mismatches in the four components of the underlying MDP, and induces each gap in isolation under a shared protocol. We evaluate 18 mitigation methods on 2 base controllers, across 33 gap settings and 10 calibrated networks built from 5 real-world locations. We find that direct transfer consistently degrades performance across all four gap sources, but the severity of the degradation does not predict the effectiveness of mitigation. Instead, mitigation effectiveness depends strongly on the network and gap setting: outside the action gap, a method that helps in one case may fail in another. The most effective methods generally estimate what the gap changes, rather than make the policy insensitive through domain randomization or invariant representations. Our code is available at https://github.com/Red-Pheonix/Sim2RealTSCBenchMark

1. Introduction

RL can outperform traditional controllers in simulated traffic, but deployment is challenged by a Sim-to-Real gap involving sensing, action execution, traffic dynamics, and control objectives. Sim2Signal provides a shared benchmark for studying these mismatches and evaluating mitigation methods.

Introduction

  • Every gap degrades deployed-controller performance, but degradation magnitude does not predict mitigation recovery; estimating gap-changed quantities tends to help more than making policies insensitive.Examples include estimating the state reached after a delayed action or the true vehicle count behind noisy sensing, rather than relying on randomized training or invariant representations.Source: §1 · p. 3
  • Sim2Signal decomposes the gap into observation, action, transition, and reward sources, evaluating 18 mitigation methods with two base controllers across 33 settings and 10 calibrated networks from five locations.The benchmark includes two new datasets and introduces controlled discrepancies under a shared protocol.Source: §1 · pp. 2–3
  • Action-gap methods work across tested networks and settings, while methods for the other gaps can fail across settings or leave performance worse than applying no mitigation.This variation shows that mitigation effectiveness depends on the network and gap setting.Source: §1 · p. 3

2. Related Work

Prior TSC research reports simulated gains, proposes gap-specific mitigations, or improves fidelity for individual deployment sites, but does not systematically benchmark mitigation across Sim-to-Real gap sources. Sim2Signal makes each gap source a controlled variable and evaluates mitigation methods under a shared protocol.

RL for TSC

  • RL traffic-signal controllers have reported gains over traditional approaches in simulation, but evaluations generally test policies under training conditions rather than changed conditions.RL has been applied to traffic signal control for two decades, and simulated gains are summarized by Wei et al..Source: §2 · p. 3

Sim-to-Real Mitigation for TSC

  • TSC mitigation methods target known gap sources with tailored mechanisms, but are evaluated on separate networks, simulators, and baselines, leaving comparative effectiveness unknown.Examples include observation randomization [12], action constraints [13], transition adaptation [14] [15] [12], and broader reward objectives [13].Source: §2 · p. 3

Deployment-Oriented TSC

  • Deployment-oriented systems improve fidelity through site-specific tools, but their end-to-end, single-site evaluations make it difficult to determine which fixes reduce the gap.Examples include calibrated digital twins, real demand data, faulty-detector state estimation, admissible signal programs [18], and hardware-in-the-loop execution [19].Source: §2 · p. 3

TSC Benchmarks

  • Existing TSC benchmarks standardize controller comparisons, but none isolates individual gap sources and evaluates mitigation methods against each under a shared protocol.Benchmarks span fixed-scenario simulators such as RESCO and SUMO-RL, multi-simulator APIs such as TSLib, LibSignal, and PyTSC [24], and deployment-oriented environments.Source: §2 · p. 4

Across related work

  • Sim2Signal addresses the gap across prior work by varying each gap source under a controlled protocol and evaluating mitigation methods themselves.Prior studies typically target one gap at a time, build a single site end to end, or compare controllers under unchanged conditions, leaving deployment transfer and mitigation effectiveness unmeasured.Source: §2 · p. 4

3. The Sim2Signal Benchmark

Sim2Signal isolates observation, action, transition, and reward mismatches in a shared sim-to-sim protocol across ten calibrated networks. Mitigation recovery varies by gap and network, and severity alone does not predict effectiveness.

3.1 Shared Protocol

  • The benchmark uses CityFlow and SUMO as paired environments, induces each gap in isolation, and trains policies in the simulated environment except for the reward oracle.All methods share pretrained DQN or PressLight checkpoints and a bounded training budget; deployments for observation, action, and transition gaps report the top five evaluation checkpoints.Source: §3.1 · pp. 4–5

3.2 Networks and Calibration

  • The ten calibrated networks span five locations and single- and multi-intersection layouts; matching baseline travel time and throughput makes induced-gap effects attributable to the gap.Tempe and Bullhead networks use real UTDF signal-plan data and NEMA dual-ring configurations.Source: §3.2 · p. 5

3.3 Controlled Gap Settings

  • Observation gaps vary sensor noise, lane-detector failure, and detection range, while action gaps test delayed phase changes and restricted phase transitions.Observation mitigations include domain randomization, latent-space adaptation, and reconstruction; action methods include delay-aware prediction and grounded action transformation, sometimes with action shielding.Source: §3.3 · pp. 5–7
  • Action-delay settings have the largest Direct-Transfer gaps, gap magnitude follows congestion rather than network size, and mitigation can backfire on some settings.The figure reports one DQN run per setting; its best-method values are optimistic because the method is selected separately for each network-setting cell.Source: Figure 3 · p. 6
  • Mitigation recovers most of the two action gaps and less of the observation, transition, and reward gaps; delay is both the most severe gap and the one with greatest recovery.Methods that estimate what the gap changed tend to recover more, but can still help on some networks and settings while hurting on others.Source: Figure 4 · p. 7
  • Transition gaps perturb vehicle dynamics under load and weather settings, while reward gaps reweight shared costs and add real-only emissions and safety components.Transition mitigation includes domain randomization, simulation-based parameter adaptation, and grounded action transformation; reward methods infer weights, train multi-objective candidates, or search weights using deployed returns.Source: §3.3 · pp. 7–8

4. Benchmarking Results

Direct transfer degrades performance across all gap sources, but mitigation effectiveness depends on the gap and deployment network more than on gap severity. Most findings hold across DQN and PressLight, with residual gaps largely tied to the network and setting.

4.1 Gap Severity by Source (RQ1)

  • Every gap source degrades Direct-Transfer, with action delay most severe and restricted phase transitions among the least severe.Observation and transition gaps fall between them; within observation gaps, sensor failure has the smallest median and reduced detection zone the largest. Reward regret is positive across all networks and settings, and severity varies within as well as across gap sources.Source: §4.1 · p. 8

4.2 Mitigation Effectiveness Across Gaps (RQ2)

  • Action gaps are reliably recoverable, but observation, transition, and reward mitigation is deployment-dependent, and gap severity does not predict recoverability.Predicting the state reached by delayed actions sharply reduces delay gaps; shielding improves phase-transition methods. Reconstruction leads on observation, while Reward Inference alone improves on reward gaps; transition methods mostly recover similarly, with GAT recovering almost nothing. Methods that estimate what the gap changed often recover most, though this pattern is not universal.Source: §4.2 · p. 9

4.3 Variation Across Network Conditions (RQ3)

  • Congested networks tend to have the largest gaps and recover most through mitigation, while mitigation can worsen performance on the least congested network.Bullhead 1 and Bullhead 3 have the largest gaps; Ingolstadt 1 has the smallest, and its best method does not improve on Direct-Transfer. Tempe 1×1 is a noted exception to the rough congestion ordering.Source: §4.3 · p. 10

4.4 Robustness Across Base Controllers (RQ4)

  • DQN and PressLight agree on 76% of network-setting pairs for ATT gap (r = 0.79) and 70% for reward regret (r = 0.91).Agreement spans all four gap sources; most disagreements beyond a factor of two occur in observation gaps, with neither controller consistently more robust. The best-performing method is the same under both controllers for observation, delay, and phase-transition gaps.Source: §4.4 · p. 10, Figure 5 · p. 10

5. Conclusion

Sim2Signal benchmarks four Sim-to-Real gaps in traffic signal control and shows that mitigation effectiveness depends on gap configuration and network conditions, not gap category or severity alone. Methods that estimate and correct the altered component generally recover more consistently than generic robustness approaches.

5. Conclusion

  • Sim2Signal provides a controlled, standardized benchmark for isolating observation, action, transition, and reward gaps across networks, settings, and base controllers.The benchmark evaluates whether mitigation methods generalize across these conditions.Source: §5 · p. 11
  • Mitigation effectiveness depends on gap configuration and network conditions, while gap severity is not a reliable indicator of how well a method will work.Methods that estimate the altered state, dynamics, action execution, or objective generally achieve more consistent recovery than domain randomization or invariant representations.Source: §5 · p. 11
  • Future extensions will add gap sources such as lane closures and pedestrian interactions, and study settings where multiple gaps occur simultaneously.Source: §5 · p. 11

A. Benchmark Details

The benchmark specifies its scale, calibration, evaluation protocol, and reporting metrics. A shared pretrained checkpoint and deterministic deployment evaluation support comparisons, while training variation and network-dependent regret scale qualify reproducibility and aggregation.

Scale and methods

  • Table A1 breaks down networks, induced settings, methods per gap, and deployment runs, while Table A2 maps mitigation methods to their results-table identifiers.Table A2 expands the overview’s method families into individual evaluated methods.Source: A. Benchmark Details · pp. 16–17

Calibration

  • The no-gap residual is 0.3–19 s across all ten networks and both base controllers, versus induced gaps reaching hundreds to thousands of seconds.The passage reports median residuals of 9 s and 7 s for DQN alone and attributes transfer degradation mainly to induced gaps, not the simulator pair or interface.Source: A. Benchmark Details · p. 16

Metrics

  • Alongside ATT, the benchmark reports throughput, average queue length, average delay, and gap-specific metrics; reward regret is comparable within deployments but only ordinally across networks.regretDT measures the real objective given up by Direct-Transfer, while regretbest measures the regret of the best budgeted method in a deployment.Source: A. Benchmark Details · p. 16

Protocol

  • Each mitigation starts from the same nominal-setting checkpoint, then deploys in Ereal for up to 100 episodes, with mean ± std over the top five evaluation checkpoints.Pretraining uses 200 episodes per network and base-controller pair; reward-gap methods select on the real objective, so the top-five rule does not apply, and the reward oracle is a reference rather than a budgeted method.Source: A. Benchmark Details · p. 16

Seeding and infrastructure

  • All reported runs use seed 0 and fixed shipped hyperparameters; SUMO deployment is deterministic, while CityFlow retraining may vary slightly even at a fixed seed.The benchmark runs on CPU by default, and a single Ubuntu desktop can reproduce table runs given enough wall-clock time.Source: A. Benchmark Details · p. 17

A.1 Benchmark Formulation

Sim2Signal formulates sim-to-real traffic signal control as a benchmark over mismatches between training and deployment environments. It isolates four gap sources and distinguishes shared deployment metrics from reward-gap evaluation against the deployment objective.

A.1 Benchmark Formulation

  • The benchmark models traffic signal control as an MDP and evaluates policies trained in E_sim against deployment environment E_real, where cross-environment gaps can degrade performance.The environment executes the requested action, transitions to a new state, and evaluates the outcome with a reward; separating requested from executed actions represents deployment delays and signal-transition constraints.Source: §A.1 · p. 17
  • Sim2Signal isolates observation, action, transition, and reward gaps as mismatches in state measurement, requested-action execution, traffic dynamics, and the training-versus-deployment objective.The gaps are defined respectively by deployment observations, action execution, transition dynamics, and reward functions differing from those used in training.Source: §A.1 · p. 17
  • Observation, action, and transition gaps use a shared deployment metric, while the reward gap uses regret against a policy trained directly on the deployment objective.Because the reward gap changes the metric itself, it requires a separate reference.Source: §A.1 · p. 18

A.2 Method Details

The appendix specifies mitigation families and implementation protocols for observation, action, transition, and reward gaps, alongside reference controllers.

Observation

  • Observation-gap methods comprise joint-corruption domain randomization, latent-space adaptation, and autoencoder reconstruction that supplies denoised observations to the policy.Latent methods include LUSR [36], DARLA [37], ATC [38], and a plain VAE embedding [39]; their encoders are pretrained on randomized observations, frozen, and adapted to traffic feature vectors.Source: §A.2 · pp. 18–19

Action (delay)

  • For delay gaps, Delayed-Q [8] rolls a forward model through pending actions, while PRLight predicts post-countdown features; Ignore-Delay and Oblivious-Q [8] isolate training under delay without prediction.Ignore-Delay trains in a delay-free simulator on the same simulated budget; Oblivious-Q trains under delay but acts on the current observation.Source: §A.2 · pp. 18–19, Table A2 · p. 19

Action (phase transitions)

  • For phase-transition gaps, domain randomization varies rule tables, while GAT and UGAT transform actions; each is evaluated with and without deployment-time action shielding.Randomized tables disable 20–45% of allowed transitions while keeping every phase reachable; shielding masks illegal actions and substitutes the highest-ranked allowed action.Source: §A.2 · pp. 19–21

Transition

  • Transition-gap methods randomize five car-following parameters, infer their posterior from transition statistics and real rollouts, or ground simulated actions using GAT, UGAT [14], and JL-GAT [15].Randomization uses 300 simulated episodes without real rollouts; parameter adaptation and grounding use 100 episodes pairing one simulated and one real rollout with a policy update.Source: Table A5 · p. 20, §A.2 · p. 21

Reward

  • Reward-gap methods infer real reward weights from probe rollouts, select simulator-trained scalarizations by real evaluation, or search weights with Bayesian optimization; the reward oracle trains on the true objective in Ereal.The oracle trains for 200 real episodes and is an empirical upper reference, not a certified optimum; reward inference uses ridge regression, while dynamic shaping scores candidates in Ereal [50].Source: Table A7 · p. 21, §A.2 · p. 22

Non-RL base controllers

  • Fixed-Time and MaxPressure serve as non-RL references, deploy directly in Ereal, and receive no mitigation; Fixed-Time reads no observations, while MaxPressure selects the phase with highest queue-difference pressure.Fixed-Time cycles phases at 30-second intervals, so the observation gap cannot affect it by construction.Source: Table A2 · p. 19, §A.2 · p. 22

A.3 Gap Design

The gap design grounds observation, action, transition, and reward mismatches in field evidence and calibrated settings. It specifies isolated gap conditions, including stress settings and limits on where some action-gap tests apply.

Observation gap

  • Observation gaps combine noisy readings, persistent stuck-at-zero detector failures, and limited detection zones, with settings spanning field-plausible and stress severities.Noise follows Aslani et al. with σ ∈{3, 5, 10, 20}; failures use per-lane probabilities p ∈{0.05, 0.2, 0.5, 0.7}, drawn once per episode, and detection zones span 10–100 m. The combined settings apply all three corruption sources together.Source: §A.3 · p. 22

Action gap (delay)

  • Action-delay tests model countdowns of 20–60 seconds against 10-second decisions, while mitigation methods assume a fixed 20-second countdown.Thus, at larger delays, two to six decisions are in flight and the tests measure degradation when the real countdown exceeds the assumed one [8].Source: §A.3 · p. 22

Action gap (phase transitions)

  • Phase-transition tests derive five restrictions from real NEMA dual-ring plans, but cover only four networks because only Tempe and Bullhead have real signal plans.Flexible forbids no transition, three Barrier variants disable roughly 43% of transitions, and Cyclic fixes one successor per phase; restrictions reflect minimum green times, clearance intervals, and fixed sequences [33, 34].Source: §A.3 · p. 23

Transition gap

  • Transition gaps pair CityFlow and SUMO as the simulator and real-world environments, then perturb real-world car-following parameters for vehicle loading and weather conditions.The settings raise startup delay for loaded vehicles and reduce acceleration and braking for rain or snow; domain-randomization distributions combine Müller and Sabatelli [12] with Schrader et al. calibration values.Source: §A.3 · p. 23

Reward gap

  • Reward settings include an efficiency-aligned sanity check and three objectives emphasizing terms unavailable to the simulator-trained policy; the fairness objective was corrected to penalize starvation.Queue-and-spread fairness alone favored gridlock because equal starvation yields near-zero throughput spread; adding the waiting component, which is much larger under gridlock, prevents that exploit without changing the fairness measure.Source: §A.3 · p. 24

A.4 Reward Metrics

Reward costs sum per-component normalized quantities weighted by Table A7, so weights express relative importance rather than unit conversions. The subsection defines emissions and fuel, rear-end safety conflicts, fairness, and logged components excluded from deployment objectives.

A.4 Reward Metrics

  • Each reward cost sums normalized components weighted by Table A7, making weights express relative importance rather than unit conversion.Each raw component is divided by a fixed per-component normalizer before weighting.Source: §A.4 · p. 24
  • CO2 and fuel are episode totals from SUMO’s HBEFA-based model, so they are computable only in Ereal and simulator-trained policies can reach them only through correlated proxies.Tables report episode totals in kilograms.Source: §A.4 · p. 24
  • Safety uses rear-end conflicts when car-following time to collision falls below 1.5 s, counting each vehicle at most once per episode.Conflicts are attributed to the intersection controlling the vehicle’s lane; crossing and merging geometries are not evaluated.Source: §A.4 · p. 25
  • Fairness cost is the largest throughput gap between demand-active approaches, and is zero when fewer than two approaches have waiting vehicles.All approaches are weighted equally, and this measure is computable in both simulators.Source: §A.4 · p. 25
  • Fuel, emergency stops, and collisions are logged but unweighted in reward-gap runs; pressure and phase-switch rate also receive no deployment-objective weight.Collisions carry almost no signal because SUMO resolves would-be crashes as emergency stops.Source: §A.4 · p. 25

B. Full Results

The full-results tables report performance across observation, transition, action, and reward gaps using ATT gaps or absolute real-environment metrics. They also define how methods, reference controllers, and objective-specific outcomes are compared.

Reporting conventions

  • Gap tables report per-setting ATT gaps, while core-metric tables give absolute real-environment ATT, queue, delay, and throughput; delay is not reported as a gap.ATT gaps are mean ± standard deviation across the top five evaluation checkpoints; Direct-Transfer is a single deterministic evaluation. Reward-gap tables instead report the hidden real objective and its native-unit components, since the objective is not comparable across networks.Source: B. Full Results · p. 25

Reference controllers

  • Fixed-Time and MaxPressure appear as non-RL reference controllers, and their ATT gaps are measured against their own simulator runs rather than calibrated across simulators.Their bracketed absolute real ATT is the value to compare with the RL base controllers; a large negative reference gap indicates simulator disagreement under congestion, not a helpful induced gap.Source: B. Full Results · pp. 25–26

Observation gap

  • Observation-gap tables cover sensor noise, detection-zone limits, sensor failure, and combined corruptions for DQN and PressLight.The tables report ATT gaps and core metrics in the real environment; the supplied captions specify their scope but do not provide outcome values.Source: Table A8 · p. 27, Table A10 · p. 29, Table A12 · p. 31

Transition and action gaps

  • Transition-gap tables evaluate DQN and PressLight under four car-following settings, while action-gap tables cover actuation delay and NEMA phase-transition restrictions.For phase-transition restrictions, adaptation methods are reported with and without the shield; disallowed-transition requests are dropped silently when unshielded and are zero by construction when shielded, while force-offs cannot be avoided by shielding.Source: Table A24 · p. 43, Table A26 · p. 45, Table A28 · p. 47

Reward gap

  • Reward-gap tables report the hidden real objective for DQN and PressLight across efficiency-aligned, emission-heavy, fairness-heavy, and safety-heavy objectives.They also report native-unit metrics including total CO2, fairness, and TTC conflicts; the oracle is a reference trained directly in the real environment on the true objective, not a competing method.Source: B. Full Results · p. 25, Table A40 · p. 59, Table A41 · p. 60

Related papers

Loading 2609.01676v1…