PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffLight: A Partial Rewards Conditioned Diffusion Model for Traffic Signal Control with Missing Data

Hanyang Chen, Yang Jiang, Shengnan Guo, Xiaowei Mao, Youfang Lin, Huaiyu Wan

arXiv:2410.22938v2eess.SYcs.AIcs.LGcs.SY

TL;DR

Traffic signal control often assumes continuous sensor data from surrounding intersections, although missing data is a practical challenge. DiffLight uses conditional diffusion to combine imputation and decision-making in offline TSC, and experiments across five datasets report it as an effective controller for missing-data scenarios.

  • Problem

    Most TSC approaches assume continuous sensor data from all surrounding intersections, while sensor shortages and failures make missing-data control an insufficiently studied practical problem.

    Source: §1 · pp. 1–2

  • Method

    DiffLight uses offline conditional diffusion to integrate traffic-data imputation and decision-making, conditioning on partial rewards and using STFormer and DCM.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Experiments across five datasets and varied missing-data scenarios indicate that DiffLight is an effective, highly competitive TSC controller.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Offline training with missing traffic data is presented as a safer, more practical alternative to frequent real-world interaction under incomplete data.

    Source: §1 · p. 2

    The evaluation considers only random and kriging missing patterns, although real-world patterns are more complex; conditioning on partial rewards instead of returns could make agents short-sighted.

    Source: §6 · p. 10

Abstract

from arXiv · show

The application of reinforcement learning in traffic signal control (TSC) has been extensively researched and yielded notable achievements. However, most existing works for TSC assume that traffic data from all surrounding intersections is fully and continuously available through sensors. In real-world applications, this assumption often fails due to sensor malfunctions or data loss, making TSC with missing data a critical challenge. To meet the needs of practical applications, we introduce DiffLight, a novel conditional diffusion model for TSC under data-missing scenarios in the offline setting. Specifically, we integrate two essential sub-tasks, i.e., traffic data imputation and decision-making, by leveraging a Partial Rewards Conditioned Diffusion (PRCD) model to prevent missing rewards from interfering with the learning process. Meanwhile, to effectively capture the spatial-temporal dependencies among intersections, we design a Spatial-Temporal transFormer (STFormer) architecture. In addition, we propose a Diffusion Communication Mechanism (DCM) to promote better communication and control performance under data-missing scenarios. Extensive experiments on five datasets with various data-missing scenarios demonstrate that DiffLight is an effective controller to address TSC with missing data. The code of DiffLight is released at https://github.com/lokol5579/DiffLight-release.

1 Introduction

Traffic signal control matters for congestion, but most RL-based approaches assume continuous sensor data from every surrounding intersection, an assumption that often fails in practice. DiffLight addresses missing-data TSC offline by combining imputation and decision-making with conditional diffusion, and its experiments report strong competitiveness across five datasets and missing-data scenarios.

1 Introduction

  • Most RL-based TSC assumes continuously available sensor data from surrounding intersections, but sensor shortages, malfunctions, and errors make that assumption unrealistic.The paper says TSC with missing data better reflects actual scenarios but has not been sufficiently studied.Source: §1 · pp. 1–2
  • Offline training with missing traffic data is presented as a safer, more practical alternative to frequent real-world interaction, and it must address both imputation and decision-making.The paper focuses on offline TSC because real-world interaction can be challenging and potentially unsafe with incomplete data.Source: §1 · p. 2
  • Diffusion models offer a shared conditional-generative framing for offline decision-making and traffic-data imputation, motivating their use to unify the two tasks.The paper cites diffusion models’ ability to capture complex policy distributions in offline datasets.Source: §1 · p. 2
  • Missing rewards can be confused with padded values, while local-intersection data alone can miss network-wide spatial-temporal dependencies.The paper identifies both as challenges for imputation and decision-making under missing data.Source: §1 · p. 2
  • DiffLight combines traffic-data imputation and decision-making for offline TSC with missing data using PRCD, STFormer, and DCM.The authors report experiments across five datasets and various missing-data scenarios, with results indicating that DiffLight is highly competitive.Source: §1 · p. 2

2 Preliminaries

The preliminaries formalize traffic signal control as an offline partially observable decision problem and specify its intersection-level observations, actions, rewards, and missing-data assumptions. They also introduce diffusion-based decision-making, including observation-trajectory generation and inverse dynamics for selecting actions.

2.1 Partially Observable Markov Decision Process

  • The offline POMDP is defined by state, action, transition, reward, observation, and discount components, with the objective of learning a policy that maximizes expected return.The agent observes part of the state through an observation function.Source: §2.1 · pp. 2–3

2.2 Traffic Signal Control with Missing Data

  • At each intersection, observations contain vehicle counts and queue lengths in entrance lanes, available actions are four signal phases, and immediate reward is based on queue length.The traffic network is modeled with agents at intersections and, for a four-way intersection, 12 movements and four phases.Source: §2.2 · p. 3, Figure 1 · p. 3
  • The paper considers random and kriging missing patterns and assumes an intersection’s observations and rewards are missing simultaneously.These missing values arise when sensor data are absent or erroneous.Source: §2.2 · p. 3

2.3 Diffusion Models for Reinforcement Learning

  • Diffusion models use forward and reverse processes, and the paper adopts DDIM sampling in the reverse process to accelerate sampling.The reverse process starts from sampled noise and estimates successive denoised states.Source: §2.3 · p. 3
  • For discrete TSC actions, the approach diffuses offline observation trajectories and uses inverse dynamics to infer the action that transitions between observations.This avoids directly diffusing actions.Source: §2.3 · p. 3
  • Classifier-free guidance learns conditional and unconditional noise predictions and combines them using a guidance scale.The conditional distribution is over trajectories given a condition.Source: §2.3 · p. 4

3 Methodology

DiffLight frames traffic-data imputation and decision-making as conditional diffusion, using partial rewards, spatial-temporal modeling, and inter-intersection communication.

3 Methodology

  • DiffLight treats imputation and decision-making as conditional generation, diffuses observations, and uses inverse dynamics to produce actions.Source: §3 · p. 4

3.1 Partial Rewards Conditioned Diffusion

  • PRCD conditions on partial observable rewards rather than padded missing rewards, assigning them to corresponding masked observation sub-trajectories.Source: §3.1 · p. 5

3.2 Diffusing with Spatial-Temporal Transformer

  • STFormer embeds diffusion and trajectory timesteps, rewards, and local and neighboring observations, then uses cross-attention and separate spatial and temporal self-attention to predict noise.Source: §3.2 · p. 6

3.3 Diffusion Communication Mechanism

  • DCM disseminates noise-model-generated neighbor observations during reverse diffusion to support local decisions when neighboring observations may be absent.Source: §3.3 · p. 6

3.4 Training and Inference of DiffLight

  • Training uses self-supervised masking because missing-data ground truth is unavailable; inference imputes missing and future observations and generates next actions.Source: §3.4 · pp. 6–7

4 Experiments

Across five datasets, DiffLight performs best on most data-missing benchmarks. Ablations support the value of STFormer and partial-reward conditioning, while generalization is stronger from higher to lower missing rates than in the reverse direction.

4.1 Experimental Setup

  • Experiments use CityFlow and five real-world traffic-flow datasets, evaluating ATT across random and kriging missing patterns and comparing BC and offline-RL baselines.ATT is the average travel time of vehicles through the network; lower ATT indicates better control performance.Source: §4.1 · p. 7

4.2 Performance under Data-Missing Scenarios

  • Across five datasets, DiffLight performs best on most; it is optimal or sub-optimal on all random-missing datasets and best on most kriging-missing datasets.The experiments also report appendix analyses of no-missing performance, unobserved intersection locations, missing-rate limits, and scalability.Source: §4.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.3 Ablation Study

  • On Hangzhou1 and Jinan1, STFormer outperforms U-Net, and STFormer+PRCD outperforms STFormer under random and kriging missing.The ablation variants use DCM; missing observations are masked with Gaussian noise rather than imputed by SFM.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 Model Generalization

  • When trained at a high missing rate, DiffLight remains stable at lower test rates; training at a low rate and testing at a higher rate causes a slight decrease.The authors attribute the asymmetry to higher missing rates presenting more complex missing situations.Source: §4.4 · pp. 9–10, Figure 3 · p. 9

5 Related Work

Prior TSC research spans conventional controllers and online or offline reinforcement learning, while offline TSC with missing data remains unaddressed in the cited work. Diffusion methods and traffic-imputation models provide related approaches, but the paper applies them to this setting.

Traffic Signal Control

  • TSC research includes conventional controllers such as Fixed-time, SCOOT, and SCATS, alongside RL methods for dynamic control and inter-agent communication.The related-work review states that no existing work solves TSC with missing data in the offline setting.Source: §5 · p. 10

Diffusion-based Reinforcement Learning

  • Diffusion-based RL includes diffusion with deep Q-learning [18], latent trajectory encoding [43], and conditional trajectory planners Diffuser [16] and Decision Diffuser.The cited methods are studied without missing data, unlike this paper's TSC setting.Source: §5 · p. 10

Traffic Data Imputation

  • Traffic imputation research includes RNN methods [45] [46] and diffusion models [21]; the paper uses SFM to impute baseline observations and rewards.Source: §5 · p. 10

6 Conclusion and Limitation

DiffLight combines partial-reward-conditioned diffusion with STFormer and a diffusion communication mechanism for traffic signal control under missing data. The paper reports effective control across missing-data settings while noting limits in missing-pattern coverage and reward conditioning.

6 Conclusion and Limitation

  • DiffLight combines imputation and decision-making through PRCD, uses STFormer to capture spatial-temporal dependencies, and propagates generated observations through DCM.The authors say PRCD helps prevent missing rewards from disrupting learning.Source: §6 · p. 10
  • The study considers only random and kriging missing patterns, although real-world traffic-network missingness is more complex.The authors also note that SFM is the only imputation method used for baselines and that partial-reward conditioning instead of returns could make agents short-sighted.Source: §6 · p. 10

A Broad Impacts

  • DiffLight can perform competitively against SFM-based baselines even when an intersection has no observed neighbors, and its fast inference and stable performance indicate potential for real-time control.The paper notes that bad decisions could lead to collapse of the traffic network.Source: A Broad Impacts · p. 14

B.1 Hyperparameters of DiffLight

  • DiffLight uses batch size 64, trains for 1.5e5 steps at learning rate 2e-4, and uses 100 diffusion steps.The implementation uses history length C = 5, planning horizon H = 3, condition-removal probability 0.25, and guidance scale α = 1.2.Source: §B.1 · p. 14

B.2 Structure of STFormer

  • STFormer embeds inputs to a shared dimension and stacks spatial-temporal encoder layers built from communication cross-attention, spatial self-attention, and temporal self-attention.Communication cross-attention captures dependencies between local and neighboring intersections; the self-attention modules capture local spatial and temporal dependencies.Source: §B.2 · p. 14
  • Each encoder layer adds spatial and temporal attention outputs before an MLP residual, and the output layer converts encoder outputs into predicted noise.The implementation also adds diffusion-timestep and trajectory-timestep embeddings to attention inputs, and reward embeddings to spatial and temporal attention inputs.Source: §B.2 · p. 15

C.1 Detials of Datasets

The evaluation uses five real-world traffic-flow datasets across Hangzhou and Jinan, with offline trajectories collected from three online methods. Experiments simulate random and kriging missingness at varying rates and network locations.

C.1 Detials of Datasets

  • Five datasets cover 16 intersections in each of two Hangzhou networks and 12 intersections in Jinan; offline trajectories are collected from AttendLight, Efficient-CoLight, and Advanced-CoLight.Table 4 reports the converged performance of the three methods used to collect the offline datasets.Source: §C.1 · p. 15, Table 4 · p. 15

C.2 Missing Pattern

  • Random missing masks observations and rewards at 10%–50% probability, while kriging missing removes data at selected intersections and keeps their neighboring intersections observable.Kriging missing covers 1–4 intersections in Hangzhou1 and Hangzhou2 and 1–3 in Jinan1, Jinan2, and Jinan3.Source: §C.2 · p. 15
  • Figure 4 depicts intersections as nodes and uses masked blocks to mark traffic data with missing values.Source: Figure 4 · p. 15

D Baseline Methods

The baseline suite spans imitation learning, offline reinforcement learning, sequence modeling, and diffusion-based decision-making. Baselines are trained on complete data and tested in missing-data scenarios after SFM imputes their missing observations and rewards.

D Baseline Methods

  • BC imitates expert state-action demonstrations, CQL learns a conservative Q-function, and TD3+BC combines TD3 with behavioral cloning.Source: D Baseline Methods · p. 16
  • DT casts reinforcement learning as sequence modeling, taking past states, actions, and rewards as input and outputting future actions.Source: D Baseline Methods · p. 16
  • Diffuser iteratively refines action sequences, whereas DD diffuses over state trajectories and plans with an inverse dynamics model.Source: D Baseline Methods · p. 16
  • Baselines are trained on datasets without missing data and tested under missing-data scenarios after SFM imputes observations and rewards.Source: D Baseline Methods · p. 16
  • SFM imputes missing observations and rewards for baselines that cannot handle missing data, using a rule-based approach described as more stable than learning neural networks.Source: D Baseline Methods · p. 16

E Proof of Partial Rewards Conditioned Diffusion

The proof relates partial-reward conditioning to the target distribution by assuming independence between data collected by real and virtual sensors. It treats the unconditioned distribution as a marginal and uses classifier-free guidance to model observed and missing trajectories.

E Proof of Partial Rewards Conditioned Diffusion

F.1 Performance without Missing Data

DiffLight performs strongly across no-missing, unobserved-location, missing-rate, and larger-network evaluations. Its performance remains strong with a missing intersection whose neighbors are also missing, but drops rapidly near 90% random missingness.

F.1 Performance without Missing Data

  • DiffLight performs best on over half of the five datasets without missing data, while DD outperforms Diffuser.Source: §F.1 · p. 17, Table 5 · p. 17

F.2 Influence of Unobserved Locations

  • With a 25% kriging-missing mask that leaves one intersection missing along with all its neighbors, DiffLight still performs best.Source: §F.2 · p. 17, Table 6 · p. 17

F.3 Limit of Missing Rates

  • Under random missingness, DiffLight retains acceptable performance at 70% missing data, but drops rapidly at 90%, placing the reported limit around 70%.Source: §F.3 · p. 17, Table 7 · p. 17

F.4 Scalability of DiffLight

F.5 Additional Ablation Study on Diffusion Communication Mechanism

Ablations report that communication improves performance under kriging missingness, while inverse dynamics improves performance under both tested missing patterns. Performance remains stable as sampling steps decrease, with inference time evaluated across those settings.

F.5 Additional Ablation Study on Diffusion Communication Mechanism

  • With DCM, DiffLight performs better under kriging missingness and nearly matches the no-DCM variant under random missingness.Source: §F.5 · p. 18, Table 10 · p. 18

F.6 Additional Ablation Study on the Inverse Dynamics

  • DiffLight with inverse dynamics performs better than the no-inverse-dynamics variant under both random and kriging missingness.Source: §F.6 · p. 18, Table 11 · p. 18

F.7 Time Cost

G Discussion on MissLight

DiffLight is an offline diffusion approach contrasted with MissLight's online, two-stage imputation-and-DQN training. It achieves competitive performance against transferred and model-based CQL baselines, with the authors suggesting reward-imputation error accumulation as a possible explanation.

G Discussion on MissLight

Loading 2410.22938v2…