PaperlayerView exact revision on arXiv

Source-linked AI summary

Stability-Preserving Reinforcement Learning for Heterogeneity-Aware Composite Max-Pressure Traffic Signal Control

Jichen Zhu, Chaopeng Tan

arXiv:2610.31037v1eess.SYcs.SY

TL;DR

Directly learned signal actions do not automatically retain the analytical stability properties of max-pressure control, motivating a structured role for learning. Com-MP adapts among MP preferences while preserving decentralized MP phase selection and is reported to improve average delay over its constituent controllers and the best fixed composition.

  • Problem

    The paper asks how to adapt among heterogeneous MP preferences as local traffic and sensing change while preserving MP's network-level queue-stability structure.

    Source: §1 · p. 2

  • Method

    Com-MP uses a shared policy to adapt the relative influence of constituent MP preferences from local traffic and sensing observations while decentralized MP selects signal phases.

    Source: §1 · pp. 2–3

  • Results

    Com-MP reduces average delay by 15.6% and 8.6% relative to Q-MP and CV-MP, respectively, and by about 7.0% relative to the best fixed composition.

    Source: §6 · p. 25

  • Takeaways & Limitations

    The learned composition shifts toward Q-MP with greater queue accumulation and toward CV-MP with richer CV availability, and transfers without retraining with average delay only 2.4% higher than a target-trained policy.

    Source: §6 · p. 25

    The present implementation combines two representative constituents, and broader network scales, topologies, demand, and sensing conditions remain to be evaluated for transfer without retraining.

    Source: §6 · p. 25

Abstract

from arXiv · show

Reinforcement learning (RL) provides a flexible mechanism for adapting traffic signal control to heterogeneous traffic and sensing conditions, but directly learned signal actions do not automatically inherit the analytical properties of structured traffic-control methods. This paper proposes Com-MP, a theory-guided RL framework that learns a context-dependent composition of multiple max-pressure (MP) control preferences while retaining decentralized MP phase selection. The learned policy adapts the relative influence of constituent MP formulations according to local traffic and sensing conditions, allowing data-driven adaptation to operate within a structured traffic-control framework. We show that such intersection-specific and time-varying composition preserves strong queue stability for every demand in the strictly stabilizable region under any admissible non-anticipative composition policy, thereby separating the performance-adaptation role of learning from the stability property of the underlying controller. In a representative implementation combining detector-supported queue-based MP and connected-vehicle-supported travel-time MP, experiments on a real-world 45-intersection network show that Com-MP reduces average delay by 15.6\% and 8.6\% relative to the two constituent controllers, respectively, and by about 7.0\% relative to the best fixed composition. The learned composition responds systematically to local traffic and sensing conditions, remains effective under heterogeneous and degraded sensing, and can be transferred without retraining to an unseen network with an average delay only 2.4\% higher than that of a policy trained directly on the target network. These results show how learning-based adaptation can be introduced within a stability-preserving traffic-control structure, providing a theory-guided pathway toward trustworthy learning-enabled traffic signal control.

1. Introduction

MP offers decentralized control with queue-stability properties, but its behavior depends on pressure representation and available sensing. Com-MP uses learning to adapt among MP preferences while retaining MP phase selection, and the paper establishes a policy-independent stability guarantee alongside network-level performance gains.

1. Introduction

  • MP is attractive for local implementation, scalability, and network-level stability, but its practical behavior depends on how pressure is represented and sensed.Queue-based pressure emphasizes accumulation, whereas delay- and travel-time-based states emphasize temporal cost; their suitability can vary with local congestion and sensing availability.Source: §1 · pp. 1–2
  • Existing work often fixes one MP preference or fuses sensing to construct a chosen pressure state, leaving open how to adapt among valid MP formulations as local traffic and sensing change.The paper frames this as adapting heterogeneous MP preferences while preserving MP's network-level queue-stability structure.Source: §1 · p. 2
  • Directly learned signal actions do not automatically inherit MP's analytical stability properties unless those properties are retained in the control architecture and established analytically.The paper motivates confining learning to context-dependent adaptation within a structured controller.Source: §1 · p. 2
  • Com-MP adapts the relative influence of multiple MP preferences from local traffic and sensing observations while decentralized MP retains signal-phase selection.The representative constituents are detector-supported queue-based MP and CV-supported travel-time MP, while the framework permits a finite collection of admissible MP formulations.Source: §1 · pp. 2–3
  • Adaptive composition preserves strong queue stability for every demand in the strictly stabilizable region under any admissible non-anticipative policy, independently of the RL policy used.The guarantee applies to the representative detector-supported Q-MP and CV-supported travel-time MP states.Source: §1 · p. 3
  • Com-MP reduces average delay by 15.6% and 8.6% relative to Q-MP and CV-MP, respectively, and by about 7.0% relative to the best fixed composition.In the Qinzhou network, the learned composition also responds to local traffic and sensing conditions and transfers without retraining to an unseen network with average delay only 2.4% higher than a target-trained policy.Source: §1 · p. 3

2. Problem Setting and Constituent MP Controllers

The paper models traffic through movements and signal phases, then expresses different max-pressure controllers using a shared phase-selection structure. Its representative controllers use detector-supported queues and connected-vehicle travel times as complementary traffic states.

2.1. Network model and traffic-flow dynamics

  • The network model distinguishes finite-storage real movements from fictitious source movements, which represent demand waiting outside the physical network.Compatible signal phases define each real intersection's finite signal-state set, and admissible signal policies enforce maximum-red constraints.Source: §2.1 · pp. 3–4
  • Movement counts follow a conservation balance driven by exogenous arrivals, upstream discharges, and turning proportions held fixed during each decision interval.Signal decisions occur at epochs separated by a fixed-duration interval.Source: §2.1 · pp. 3–4

2.2. Generalized MP formulation

  • Constituent MP controllers assign movement states and downstream differential pressures, then choose feasible signal states to maximize pressure-weighted service.The network maximization decomposes across intersections, so it represents decentralized phase selection rather than centralized signal optimization; the shared phase-selection structure permits different traffic-state definitions.Source: §2.2 · p. 5

2.3. Heterogeneous sensing and representative controllers

  • Q-MP uses detector-supported, length-normalized vehicle counts to represent accumulation, while CV-MP uses connected-vehicle observations to represent accumulated travel time.Queue normalization avoids assigning greater pressure solely because an approach has more physical storage; CV pressure increases with a vehicle's time on the link and depends on observation availability.Source: §2.3 · pp. 5–6
  • Com-MP normalizes constituent phase pressures and combines them with intersection-specific composition weights while retaining the common decentralized MP phase-selection mechanism.The two representative constituents differ in both control emphasis and sensing source.Source: Figure 1 · p. 6, §2.3 · p. 6

3. Composite Max-Pressure Control and Stability Analysis

This section describes Com-MP’s adaptive composition of constituent max-pressure preferences across intersections and decision epochs while retaining decentralized phase selection. It establishes strong-stability guarantees under the stated assumptions, including for the Q–CV specialization and any admissible nonanticipative learned composition.

3.1. Adaptive composite MP

  • Com-MP adapts the relative influence of constituent controllers to local traffic and sensing conditions while retaining decentralized MP phase selection.It normalizes each controller’s phase preferences before composing them, avoiding direct combination of raw pressures with different meanings and numerical scales.Source: §3 · p. 6, §3.1 · p. 7

3.2.1. Stability setting

  • The stability analysis requires each constituent state to admit a compatible Lyapunov representation, but not a common state definition or an individually stabilizing controller.The compatible state class is closed under convex composition, and a reference policy supplies the strict service margin for strictly stabilizable demands.Source: §3.2.1 · p. 9, §3.2.2 · p. 10

3.2.2. Structural properties of adaptive composition

  • Normalized preferences are phase-equivalent to a convex combination of raw constituent pressures, including when operational constraints restrict the candidate phases.The implemented decentralized decisions differ from the network-consistent analytical MP objective by a uniformly bounded amount, preserving the queue-dependent drift margin.Source: §3.2.2 · pp. 10–11

3.2.3. Strong-stability guarantee

  • Composition changes preserve the stability argument because each update causes only a uniformly bounded Lyapunov jump while negative drift holds within each composition interval.The strict margin transfers from the reference policy to Com-MP, and bounded update effects do not remove it.Source: §3.2.3 · pp. 11–12
  • For every strictly stabilizable demand, adaptive intersection-specific Com-MP strongly stabilizes the network under any nonanticipative composition policy satisfying the stated assumptions.The guarantee is independent of how the composition coefficients are generated, including by a learned policy or heuristic.Source: §3.2.3 · p. 12

3.3. Q–CV Com-MP specialization

  • The paper applies the general result to a representative Q–CV formulation using queue-based and CV-supported travel-time states under an explicit service assumption.This part introduces the Q-MP state representation but does not provide the specialization’s full verification.Source: §3.3 · p. 12
  • Under Assumptions 1 and 3 and maximum-red, Q-MP and CV-MP states and matched Lyapunov terms are uniformly bounded and satisfy Assumption 2.For CV-MP, finite storage and FIFO bound traffic ahead of a vehicle, while recurrent positive service prevents indefinite movement occupancy; the service condition incorporates downstream restrictions and spillback.Source: §3.3 · p. 13
  • Every nonanticipative intersection-wise Q–CV composition strongly stabilizes the network for every 𝝀∈𝚲◦ under Assumptions 1 and 3 and the maximum-red requirement.Source: §3.3 · p. 13

3.4. Trustworthy Learning through Stability Preservation

  • Any nonanticipative learned policy selecting 𝜶𝑛,𝑘∈Δ𝐻 preserves strong stability for every 𝝀∈𝚲◦ under Theorem 1’s conditions.The guarantee is independent of policy parameterization or training; it concerns closed-loop stability, not RL convergence or optimality.Source: §3.4 · p. 13

4. Theory-Guided RL for Adaptive Composition

Com-MP uses reinforcement learning to adapt constituent max-pressure preferences while leaving phase selection to the stability-preserving controller. The learning policy uses local traffic and sensing context, with shared training across intersections and decentralized execution.

4. Theory-Guided RL for Adaptive Composition

  • Learning selects composition coefficients, while the lower-level Com-MP controller combines constituent pressures and chooses signal phases under the stability-preserving rule.The stability guarantee holds for the admissible composition policies described in Section 3, so learning adapts performance without replacing phase selection.Source: §4 · pp. 13–14

4.1. Shared decentralized composition policy

  • A shared stochastic actor lets each intersection choose a composition from local observations, while centralized training uses a critic and decentralized execution uses only the actor.Parameter sharing trains a common local rule across intersections; the centralized critic adds network context during training and is removed online.Source: §4.1 · p. 14

4.2. Local composition decision

  • The local decision problem adapts composition using current traffic, constituent preferences, and the sensing support available for their states.The actor selects a composition action and receives a reward for its traffic effect, while phase selection remains with the MP mechanism.Source: §4.2 · p. 15

4.2.1. Local observation

  • The actor’s observation represents constituent pressure preferences, local traffic, sensing support, current signal operation, and compatible phases.A padded movement–phase representation with validity masks accommodates intersections with different structures.Source: §4.2.1 · p. 15

4.2.2. Composition action

  • For Q–CV Com-MP, the actor selects α from {0, 0.25, 0.50, 0.75, 1}, adjusting pressure weights rather than choosing a phase directly.The endpoints recover Q-MP and CV-MP; when only one constituent state is usable, the controller uses it, and when neither is usable, it applies a predefined fallback.Source: §4.2.2 · p. 15

4.2.3. Local reward

  • The local reward favors queue reductions and penalizes persistent congestion after the lower-level MP decision is executed.It evaluates traffic effects rather than the magnitude of the composite pressure objective.Source: §4.2.3 · p. 15

4.3. Phase-equivalent policy learning

  • Phase-equivalent coefficients are grouped so the PPO likelihood ratio reflects the signal phase they induce, not distinct coefficients that produce the same phase.If every coefficient induces the same phase, the transition supplies no discriminative actor signal, though it can still inform value learning.Source: §4.3 · p. 16

5. Evaluation

The evaluation tests Com-MP on Qinzhou under changing demand and sensing, then examines its composition behavior, intersection-level performance, sensing robustness, and transfer to Lianyungang.

5.1. Experimental setup

  • The evaluation uses a 45-intersection Qinzhou network with heterogeneous link lengths and changing demand and CV penetration across three stages.The main experiment assumes full detector coverage without missed detections and tests five random seeds per method.Source: §5.1 · pp. 16–17, Figure 3 · p. 17

5.2. Experiment under time-varying demand and CV penetration rate

  • Com-MP reduces average delay by 15.6% versus Q-MP, 8.6% versus CV-MP, and about 7.0% versus the best fixed composition.It also achieves the highest average speed and throughput among Q-MP, CV-MP, and Com-MP.Source: §5.2 · pp. 17–18, Table 1 · p. 18, §5.3 · p. 20

5.3. Interpretable composition behavior

  • The learned composition responds more strongly to dynamic traffic and sensing conditions than to static intersection geometry.It shifts toward Q-MP as queue ratio rises, toward CV-MP as CV ratio rises, and toward CV-MP with greater constituent disagreement.Source: §5.3 · pp. 19–20

5.4. Intersection heterogeneity

  • Com-MP outperforms the locally better constituent at most intersections, with the spatial pattern of improvement changing across demand stages.The locally better constituent varies across intersections and stages, while Com-MP reduces delay relative to it in most cases.Source: §5.4 · pp. 20–21

5.5.1. Different detector coverage rates

  • With detector coverage rising from 10% to 90%, Com-MP average delay falls from approximately 177.2 s to 163.9 s.Even at 10% coverage, Com-MP has slightly lower average delay and higher average speed than CV-MP.Source: §5.5.1 · p. 21, Figure 8 · p. 22

5.5.2. Different missed detection rates

  • As vehicle missed detection rises from 10% to 50%, Q-MP average delay increases from approximately 193.5 s to 220.5 s, while Com-MP rises from 164.2 s to 167.7 s.Within the tested ranges, missed-detection severity on affected lanes matters more than the proportion of affected lanes.Source: §5.5.2 · pp. 22–23, Figure 9 · p. 22

5.5.3. Higher CV penetration rates

  • Under higher CV penetration, Com-MP average delay is 160.71 s, 16.7% lower than Q-MP and 2.9% lower than CV-MP.CV-MP improves markedly under richer CV observations and outperforms Q-MP across all four performance measures.Source: §5.5.3 · pp. 23–24, Figure 10 · p. 23

5.6. Cross-network transferability

  • Without retraining, the Qinzhou-trained policy transfers to Lianyungang with average delay only 2.4% higher than target-network training.The transferred policy ranks second across all four measures and outperforms both constituent controllers across those measures.Source: §5.6 · p. 24, Table 2 · p. 25

6. Conclusion and future work

Com-MP combines context-dependent performance adaptation with decentralized max-pressure phase selection and a stability guarantee independent of the learned composition policy. Experiments report lower delay than the constituent and best fixed-composition controllers, effective performance under sensing variation, and transfer to an unseen network.

6. Conclusion and future work

  • Arbitrary admissible, non-anticipative, intersection-specific, time-varying compositions preserve strong queue stability for every demand in the strictly stabilizable region under the stated finite-storage dynamics and signal constraints.The stability certificate is independent of the learned composition policy.Source: §6 · p. 25
  • Com-MP reduces average delay by 15.6% and 8.6% relative to Q-MP and CV-MP, respectively, and by about 7.0% relative to the best fixed composition.The learning policy adapts the relative influence of queue-based and CV-based preferences to local traffic and sensing context.Source: §6 · p. 25
  • Without retraining on the unseen Lianyungang network, Com-MP has average delay only 2.4% higher than a policy trained directly there and outperforms both constituent controllers.The framework also remains effective under heterogeneous and degraded sensing.Source: §6 · p. 25
  • Future work includes testing a broader family of MP formulations, richer sensing and communication uncertainty, and transfer across networks and conditions with different scales and topologies.Source: §6 · p. 25

CRediT authorship contribution statement

The contribution statement attributes writing, visualization, validation, software, methodology, and resources to Jichen Zhu, and writing, methodology, investigation, formal analysis, resources, supervision, and conceptualization to Chaopeng Tan.

CRediT authorship contribution statement

A. Proofs for the Stability of Com-MP

The appendix supports the stability result through a finite-average queue-count argument, an equivalent convex composition of constituent phase pressures, and bounded drift terms. Its proof steps rely on finite storage, bounded movements, and fixed analytical composition within decision intervals.

A. Proofs for the Stability of Com-MP

A.1. Proof of Lemma 1

  • Finite storage bounds each real-movement queue, supporting a finite long-run average total network vehicle count and hence strong stability under the stated argument.Source: §A.1 · p. 26

A.2. Proof of Proposition 1

  • The composite phase objective has the same maximizing phase as a convex combination of constituent raw phase objectives, with nonnegative coefficients summing to one.Source: §A.2 · p. 26

A.3. Proof of Lemma 2

  • The analytical composition coefficients are nonnegative, sum to one, and remain fixed within each decision interval, allowing uniform bounds over the induced composition.Source: §A.3 · p. 26

A.4. Derivation of Lemma 3

  • The drift derivation combines movement-level count drifts and bounded source, jump-correction, and signal-transition terms into a finite constant in the Lyapunov drift bound.Source: §A.4 · pp. 26–27

A.5. Proof of Proposition 2

The proof relates decentralized phase decisions to a network-consistent analytical objective using bounded discrepancies from composition, phase restrictions, and within-interval variation. It then uses strict stabilizability and bounded drift terms to support the subsequent stability argument.

A.5. Proof of Proposition 2

  • The Proposition 2 proof compares decentralized decisions with one network-consistent objective through intersection-wise composition, maximum-red restriction, and within-interval variation.The proof-equivalent objective is assembled from local objectives, with finite bounds on the resulting discrepancies and restriction losses.Source: §A.5 · pp. 27–28
  • The maximum-red restriction and within-interval variation contribute finite bounds to the objective comparison, yielding a finite total bound B.Source: §A.5 · p. 28

A.6. Proof of Lemma 4

  • For strictly stabilizable demand, a reference policy and Lyapunov functional provide a strict drift condition; bounded residual and state discrepancies support the composite drift comparison.Source: §A.6 · pp. 28–29

A.7. Proof of Lemma 5

  • At a decision epoch, updating the turning-ratio estimate and composition leaves the physical traffic state and constituent Lyapunov terms unchanged, while the real-movement quadratic and source terms do not depend on analytical coefficients.Source: §A.7 · p. 29

A.8. Completion of Theorem 1

The proof applies an extended drift identity over complete decision intervals, includes bounded update jumps, and establishes finite long-run average queues, including for arbitrary time horizons.

A.8. Completion of Theorem 1

  • The extended drift identity is applied over each complete decision interval, with update jumps bounded by (61).The resulting bounds are added across intervals.Source: §A.8 · p. 29
  • Taking the number of complete intervals to infinity proves a finite long-run average source queue.Source: §A.8 · p. 29
  • The bound extends to arbitrary T because the final partial interval is shorter than Δ, and finite storage uniformly bounds real-movement counts.Source: §A.8 · p. 29

A.9. Proof of Proposition 3

The proof establishes finite drift remainders for Q-MP and CV-MP, then derives the uniform travel-time bounds needed for the adaptive composition and verifies Assumption 2.

A.9. Proof of Proposition 3

  • Q-MP satisfies (19) with a finite remainder constant, established from its event-level Lyapunov drift relations.The proof considers unit arrivals and departures and distinguishes event-induced jumps from the decision-interval duration.Source: §A.9 · p. 30
  • CV-MP satisfies (19) with a finite upper remainder constant, using its event relations and finite-storage bounds.The proof treats departures and entries between movement-transition events.Source: §A.9 · p. 30
  • A CV’s link travel time is uniformly bounded using bounded running time, finite storage, FIFO, maximum-red service, and minimum discharge under Assumption 3.These conditions bound both the number of vehicles ahead and the service time needed for the CV to cross the stop line.Source: §A.9 · pp. 30–31
  • The resulting finite constants support the local-to-network composition and analytical-composition update bounds, and Q-MP and CV-MP satisfy Assumption 2.Source: §A.9 · p. 31

B. Training convergence of Com-MP

Com-MP’s episode reward rises rapidly early in training and then improves more gradually before approaching a stable level, indicating convergence under the adopted configuration.

B. Training convergence of Com-MP

C. Representative intersection-level performance patterns

Representative intersections show that constituent-controller preferences change across traffic stages and locations; Com-MP’s relative performance also varies, while its delay is lowest in some cases.

C. Representative intersection-level performance patterns

Related papers

Loading 2610.31037v1…