PaperlayerView exact revision on arXiv

Source-linked AI summary

VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture

Maonan Wang, Yirong Chen, Aoyu Pang, Yuxin Cai, Chung Shue Chen, Yuheng Kan, Man-On Pun

arXiv:2505.19486v2eess.SYcs.LGcs.MAcs.SY

TL;DR

Traffic signal control must handle dynamic traffic while balancing efficiency and safety, but existing approaches face limits in semantic scene understanding and safety-critical decisions. VLMLight combines multiview visual perception with an LLM meta-controller that chooses between routine RL control and structured deliberation. It reduces emergency-vehicle waiting time by up to 65% versus RL-only baselines while maintaining standard-traffic performance with less than 1% degradation.

  • Problem

    Existing traffic-control approaches have difficulty addressing dynamic, safety-critical situations, and RL state abstractions can omit semantic cues needed to prioritize emergency vehicles.

    Source: §1 · p. 2

  • Method

    VLMLight uses multiview scene descriptions and an LLM meta-controller to route routine traffic to a fast RL policy and critical cases to structured multi-agent reasoning.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    Up to 65% lower emergency-vehicle waiting time than RL-only baselines, with less than 1% degradation in standard traffic.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The full inference process completes within 11.5 seconds, fitting within the typical 13-second signal phase buffer described for urban deployments.

    Source: §6 · p. 10

    The simulator lacks diverse weather and lighting conditions, and experiments cover single intersections, leaving visual robustness and multi-intersection scalability and coordination for further study.

    Source: §6 · p. 10

Abstract

from arXiv · show

Traffic signal control (TSC) is a core challenge in urban mobility, where real-time decisions must balance efficiency and safety. Existing methods - ranging from rule-based heuristics to reinforcement learning (RL) - often struggle to generalize to complex, dynamic, and safety-critical scenarios. We introduce VLMLight, a novel TSC framework that integrates vision-language meta-control with dual-branch reasoning. At the core of VLMLight is the first image-based traffic simulator that enables multi-view visual perception at intersections, allowing policies to reason over rich cues such as vehicle type, motion, and spatial density. A large language model (LLM) serves as a safety-prioritized meta-controller, selecting between a fast RL policy for routine traffic and a structured reasoning branch for critical cases. In the latter, multiple LLM agents collaborate to assess traffic phases, prioritize emergency vehicles, and verify rule compliance. Experiments show that VLMLight reduces waiting times for emergency vehicles by up to 65% over RL-only systems, while preserving real-time performance in standard conditions with less than 1% degradation. VLMLight offers a scalable, interpretable, and safety-aware solution for next-generation traffic signal control.

1 Introduction

VLMLight addresses the difficulty of making safety-critical traffic decisions from conventional traffic states and slow, visually limited LLM approaches. It combines multiview scene understanding with an LLM meta-controller that routes routine traffic to RL and critical cases to deliberative reasoning, reducing emergency-vehicle waiting time while preserving standard-traffic performance.

1 Introduction

  • Rule-based methods struggle with dynamic, stochastic traffic, while RL approaches rely on simplified states and may miss emergency-vehicle priorities.The cited challenges include balancing emergency and regular traffic and integrating priority mechanisms with differing RL controllers.Source: §1 · p. 2
  • LLM-based traffic control faces limited scene fidelity and high inference latency, while common simulators provide statistical data rather than real-time images.These constraints limit visually grounded reasoning in complex or ambiguous situations.Source: §1 · p. 2
  • VLMLight introduces a multiview image-based intersection simulator and combines visual scene descriptions with an LLM meta-controller and dual-branch control.A fast RL policy handles routine traffic; specialized LLM agents reason about critical cases through phase analysis, planning, and rule checks.Source: §1 · pp. 2–3
  • Up to 65% lower emergency-vehicle waiting time than RL-only baselines, with less than 1% degradation in standard traffic.The framework uses a safety-prioritized meta-controller to route routine scenes to RL and critical cases to structured reasoning.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries define intersection movements and phases, then describe a vision-enabled simulator that supplies multiple camera views for observing traffic. These visual inputs include lane-level movement cues and can capture safety-critical vehicles such as fire trucks.

2.1 Traffic Signal Terminology

  • A movement connects an incoming approach to an outgoing one, and each signal phase groups nonconflicting movements that receive green simultaneously.The feasible phase set is denoted P = {p1, p2, p3, p4}.Source: §2.1 · p. 3

2.2 Vision-Based TSC Simulator

  • The simulator supports configurable bird’s-eye and directional roadside cameras, providing multiview visual inputs at intersections.The North-facing camera example captures a fire truck traversing the intersection, while lane markings help track its trajectory.Source: §2.2 · p. 3, Figure 1 · p. 3
  • VLMLight’s architecture uses a VLM to parse multiview images before an LLM meta-controller routes routine traffic to fast RL or critical cases to collaborative reasoning.The deliberative branch includes Phase Reasoning, Signal Planning, and Rule Verification agents.Source: Figure 2 · p. 4

3 Method

VLMLight combines visual scene understanding, safety-prioritized routing, fast RL control for routine traffic, and structured LLM deliberation for unusual or safety-critical cases. Its pipeline turns multiview observations into interpretable scene descriptions and then selects and verifies signal actions.

3 Method

  • VLMLight uses an LLM meta-controller to route routine scenarios to a pretrained RL policy and complex or critical events to collaborative LLM reasoning.The overall design aims to retain real-time responsiveness in typical traffic while supporting auditable decisions in safety-critical conditions.Source: §3 · p. 4

3.1 Scene Understanding via VLM

  • AgentScene converts multiview intersection images into directional text descriptions of lane semantics, congestion, and emergency or special vehicles.These structured descriptions become interpretable inputs for downstream decision modules.Source: §3.1 · p. 4

3.2 Safety-Prioritized Meta-Control

  • AgentModeSelector routes scenes consistent with the RL training objective to low-latency control and sends emergency vehicles or conflicting goals to structured reasoning.The routing decision is based on textual scene descriptions.Source: §3.2 · p. 5

3.3 Routine Control Policy

  • The routine branch encodes recent statistics across up to 12 movements and five time steps, then selects a signal phase using a Transformer-based RL policy.The policy uses PPO and a reward designed to minimize intersection congestion and delay.Source: §3.3 · pp. 5–6

3.4 Deliberative Reasoning Policy

  • For abnormal or safety-critical traffic, specialized agents map directional descriptions to phase summaries, select a phase for current objectives, and verify its legality.If the chosen action is invalid, AgentCheck selects an alternative from the feasible phase set; AgentPlan also provides a textual rationale for its decision.Source: §3.4 · p. 6

4 Experiments

VLMLight is evaluated on three real-world intersections using routine and emergency-vehicle metrics, against rule-based, RL-based, and VLM-based baselines. Results report emergency-vehicle gains, near-RL routine performance, component ablations, and inference latency.

4.1 Experimental Setup

  • Experiments use multi-view data from Songdo, Yau Ma Tei, and Massy, with Qwen models for scene descriptions and three-agent safety-critical reasoning.The simulator integrates SUMO; each 30-minute intersection dataset uses 20 minutes for RL training and 10 minutes for testing.Source: §4.1 · pp. 6–7, Figure 4 · p. 7
  • Evaluation reports ATT and AWT for overall traffic flow and AETT and AEWT for emergency-vehicle handling, comparing VLMLight with rule-based, RL-based, and Vanilla-VLM methods.No existing VLM-based method was available for direct comparison, so the study includes Vanilla-VLM, which uses VLM-generated descriptions for decision-making.Source: §4.1 · pp. 7–8

4.2 Performance under Routine and Safety-Critical Traffic Scenarios

  • Emergency-vehicle AETT and AEWT fall by over 60% versus RL-only baselines across all datasets; Songdo AEWT drops from 22.0 s to 7.48 s.VLMLight activates its deliberative reasoning branch in safety-critical scenarios.Source: Table 1 · p. 8, §4.2 · p. 9
  • Routine ATT increases by less than 1% across datasets versus the best RL methods; in Songdo, it rises from 86.80 s with UniTSA to 87.14 s with VLMLight.The Songdo AWT comparison is 39.53 s to 39.73 s; the meta-controller defers to the fast RL branch in routine settings.Source: Table 1 · p. 8, §4.2 · pp. 8–9

4.3 Ablation Study

  • Disabling AgentPhase raises AEWT from 48 s to 58 s, while removing AgentCheck has limited overall effect because invalid actions are rare.AgentPhase maps directional features to structured traffic phases; AgentCheck verifies the legality of the final decision.Source: §4.3 · p. 9, Table 2 · p. 9
  • With all reasoning modules enabled, Songdo decision latency is 11.48 seconds, within the typical 13-second green-and-yellow signal-phase buffer.Table 2 reports ablation results and corresponding inference time.Source: §4.3 · p. 9, Table 2 · p. 9

5 Related Work

Prior TSC work spans heuristic, reinforcement-learning, and language-model approaches, while existing simulators offer either structured states or visual rendering with limitations for vision-grounded intersection control.

Traffic Signal Control

  • Rule-based methods perform well under steady traffic but struggle with real-time adaptability; RL improves responsiveness but can overlook safety-critical events because of simplified states and static rewards [2].Examples include Webster’s method, SOTL, MaxPressure, and RL methods [31] [9] [33] [13].Source: §5 · pp. 9–10
  • LLM-based TSC brings high-level reasoning to rare scenarios, but relies on hand-crafted text inputs and underperforms specialized RL agents in routine traffic [23] [25].Source: §5 · p. 10

Simulators for TSC

  • SUMO, CityFlow, and LibSignal lack visual outputs, while CARLA and MetaDrive target autonomous driving and often need customization for signal scheduling [27] [41] [42].SynTraC adds image-based TSC data but is limited to single intersections and lacks user-defined or multi-view configurations [43].Source: §5 · p. 10

6 Conclusion

VLMLight combines fast RL control for routine traffic with structured semantic reasoning for safety-critical cases, reporting emergency-wait reductions and near-RL routine performance. The conclusion also identifies weather, lighting, and intersection-scale boundaries.

6 Conclusion

  • VLMLight reduces emergency-vehicle waiting time by up to 65% while maintaining comparable performance to RL-only baselines in standard conditions.The full inference process completes within 11.5 seconds, within the typical 13-second signal-phase buffer.Source: §6 · p. 10
  • The framework dynamically selects fast RL for routine traffic and a deliberative reasoning branch for safety-critical cases.The released simulator supports multi-view image inputs and dynamic scene rendering.Source: §6 · p. 10
  • The simulator lacks diverse weather and lighting, limiting evaluation of visual robustness in challenging environments.All experiments are on single intersections; multi-intersection scalability and reasoning coordination require further study.Source: §6 · p. 10

Appendix

  • The appendix provides additional method details, experimental setup, extended results, case studies, and discussion of limitations and broader impact.Source: Appendix · p. 14

A Method

The routine branch encodes movement-level traffic and signal features across spatial and temporal windows, then uses PPO to select signal phases. A meta-controller routes safety-critical scenes to sequential LLM reasoning, with legality checks and an RL fallback; prompt templates specify the agents’ roles.

A.1 Details of Routine Control Policy

  • The routine policy represents each movement with traffic-flow, occupancy, movement-type, lane-count, and signal-status features, zero-padding intersections with fewer than 12 movements.It uses the current and previous four timesteps as a five-frame observation window.Source: §A.1 · p. 14
  • A hierarchical Transformer encodes spatial and temporal patterns, then policy and value heads use PPO to select phases and estimate returns.The policy objective uses clipping to bound updates and improve training stability; the full objective combines policy and value terms.Source: §A.1 · pp. 14–15

A.2 Algorithm for VLMLight

  • A safety-prioritized meta-controller routes routine scenes to fast RL and critical scenes to sequential LLM agents for phase analysis, action planning, and legality checks.If an action remains invalid after the maximum check attempts, the system falls back to RL.Source: §A.2 · pp. 15–16

A.3 Prompt Templates

  • The prompt-template appendix covers AgentScene, AgentModeSelector, AgentPhase, AgentPlan, and AgentCheck.The listed templates correspond to scene descriptions, mode selection, phase descriptions, action selection, and rule-compliance checks.Source: Figure 6 · p. 17, Figure 7 · p. 17, Figure 8 · p. 17

B.1 Experiment Settings

Experiments use a self-developed SUMO-based simulator, locally deployed vision-language models, and a PPO-trained RL policy. The setup includes emergency-vehicle scenarios and parallel simulation processes for RL training.

B.1 Experiment Settings

  • The self-developed simulator uses SUMO 1.22, models urban roads with a 13.9 m/s speed limit, and includes police cars, ambulances, and fire trucks.Vehicle speeds follow a Gaussian distribution with a mean of 10 m/s and variance of 3.Source: §B.1 · p. 18
  • VLMs and LLMs run locally on a system with an Intel Xeon 6738P CPU, 256 GB RAM, and five A100 GPUs.Multiple VLMs run in parallel to process intersection views and generate scene descriptions.Source: §B.1 · p. 18
  • The RL components use PPO with 30 parallel simulation processes, 3e5 environment steps, and a batch size of 64.Training also uses a linearly decaying learning rate beginning at 1e −3 and a trajectory memory size of 3000.Source: §B.1 · p. 18

B.2 Dataset Details

The evaluation suite covers three real-world intersections with varied layouts, traffic intensities, and directional camera views. Comparisons include traditional, RL-based, and VLM-based traffic-signal-control methods.

B.2 Dataset Details

  • The three sites span large grid intersections in Songdo, constrained downtown roads in Yau Ma Tei, and a lighter-traffic suburban T-junction in Massy.These layouts and flow intensities provide varied settings for evaluating generalizability.Source: §B.2 · p. 18
  • Directional cameras recorded 30 minutes of footage per intersection, with all approaches covered and views supplied to the perception module.Songdo has the heaviest flows, exceeding 2 vehicles/s in some directions, while Massy has flows below 1 vehicle/s; emergency vehicles appear across all sites.Source: §B.2 · pp. 18–19, Figure 11 · p. 19

B.3 Compared Methods

  • The comparisons include three traditional methods, five RL-based methods, and Vanilla-VLM, which makes decisions from scene descriptions without RL in routine scenarios.Traditional baselines include FixTime, Webster, and MaxPressure; RL baselines include IntelliLight, UniTSA, A-CATs, CCDA, and 3DQN-TSC.Source: §B.3 · pp. 19–20

C.1 Additional Performance Analysis of RL Methods for TSC

Among the compared RL methods, IntelliLight and UniTSA achieve the highest cumulative rewards and converge rapidly, while 3DQN-TSC performs worst across scenarios. Model ablations identify scene understanding as most sensitive to model choice, while mode selection and reasoning are more resilient.

C.1 Additional Performance Analysis of RL Methods for TSC

  • IntelliLight and UniTSA achieve the highest cumulative rewards and rapid convergence across the three scenarios.Figure 12 plots cumulative reward against training episodes for the five RL-based methods.Source: §C.1 · p. 21
  • A-CATs and CCDA perform competitively but slightly below the top methods, whereas 3DQN-TSC performs worst across all scenarios.A-CATs and CCDA approach similar final performance, while 3DQN-TSC’s single-phase-per-cycle action design limits holistic phase allocation.Source: §C.1 · p. 21

C.2 Additional Ablation Study on different LLMs

  • AgentScene is most sensitive to model changes; weaker scene understanding is associated with missed emergency vehicles, mistaken prioritization, and longer travel times.Table 5 compares VLMs for scene understanding.Source: Table 5 · p. 21, §C.2 · p. 22
  • AgentModeSelector and reasoning agents are more resilient to model changes, though special vehicles may still face slight delays from inaccurate lane-to-phase mappings.Tables 6 and 7 compare LLMs for mode selection and reasoning policy.Source: Table 6 · p. 21, Table 7 · p. 21, §C.2 · p. 22

C.3 Additional Ablation Study on Inference Time

VLMLight’s inference time is reported below 13 seconds in three environments, with scene understanding taking the largest share of processing time.

C.3 Additional Ablation Study on Inference Time

  • The timing comparison uses a 10 s minimum green duration plus an additional 3 s for yellow lights as context.Source: §C.3 · p. 22
  • Scene understanding consumed most inference time, while mode selection and deliberative reasoning required considerably less.Source: §C.3 · p. 22
  • Inference times stayed below 13 s in all three environments, within the stated acceptable range for real-world deployment.Table 8 reports inference time across the three environments.Source: Table 8 · p. 22, §C.3 · p. 22

D Case Study

Three case studies trace complete traffic-signal cycles, showing VLMLight routing routine scenes to its fast RL branch and critical cases to deliberative reasoning.

D Case Study

  • The case studies cover complete TSC cycles and illustrate switching between fast RL and deliberative LLM control as circumstances demand.Source: D Case Study · pp. 22–23

D.1 Example 1: Routine Control in Yau Ma Tei

  • At Yau Ma Tei, the selector routes a scene with no anomalies or priority vehicles to RL, which activates westbound Phase-2 and clears the queue.Source: Figure 13 · p. 23, §D.1 · p. 24

D.2 Example 2: Complex Scenario in Massy

  • An ambulance detected on Massy’s west approach triggers deliberative reasoning, and the ambulance clears through the northbound turn by T + 1.AgentPhase maps candidate phases, AgentPlan recommends Phase-1, and AgentCheck verifies emergency-priority compliance.Source: Figure 14 · p. 23, §D.2 · p. 24

D.3 Example 3: Complex Scenario in Songdo

  • At Songdo, a detected fire truck prompts deliberative reasoning and moves through the intersection without interruption by T + 1.AgentPhase, AgentPlan, and AgentCheck produce a compliant and safe control action.Source: Figure 15 · p. 24, §D.3 · pp. 24–25

E Disscussion

  • VLMLight combines multi-view visual perception and structured reasoning, and its open-source vision-based simulator is compatible with RL-based TSC algorithms.Source: E Disscussion · p. 25
  • The evaluation uses single intersections and lacks diverse weather, lighting, pedestrians, bicycles, and other real-world elements.The paper also notes dependence on VLM capabilities and the need to study multi-intersection scalability and coordination.Source: E Disscussion · p. 25
Loading 2505.19486v2…