Source-linked AI summary
VLALight: A Vision-Language-Action Model for Traffic Signal Control
Pan Zhang, Siqi Lai, Kemu Dong, Hao Liu
TL;DR
Traffic signal controllers commonly depend on engineered traffic states despite roadside cameras providing rich observations, leaving end-to-end visual control underexplored. VLALight maps multi-view videos to coordinated signal actions with cooperative perception, staged training, and adaptive reasoning. Across seven real-world datasets and three urban networks, it consistently improves traffic efficiency over transportation-, RL-, and LLM/VLM-based controllers.
Problem
Existing traffic signal control largely relies on engineered traffic states or separate perception modules, while direct control from rich roadside-camera observations remains underexplored.
Source: §1 · p. 1Method
VLALight maps multi-view roadside videos to coordinated signal actions using spatiotemporal reasoning, topology-aware cooperative perception, staged supervised training, cooperative agentic reinforcement learning, and adaptive fast/slow reasoning.
Source: §1 · p. 2Results
Takeaways & Limitations
The results demonstrate the potential of VLA models for physical traffic control using roadside visual observations.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Traffic signal control (TSC) is essential for improving urban mobility and reducing congestion. Although roadside cameras are widely deployed at signalized intersections and provide rich visual observations of evolving traffic, existing TSC methods typically rely on manually engineered traffic states or separate perception modules, creating a gap between physical observations and control decisions. We present VLALight, the first vision-language-action (VLA) model for end-to-end traffic signal control from multi-view roadside videos. VLALight directly maps visual observations to coordinated signal actions through multi-target spatiotemporal traffic reasoning and topology-aware cooperative perception across intersections. To establish this capability, we develop a two-stage supervised cold-start training strategy for visual traffic understanding and signal decision-making, followed by cooperative agentic reinforcement learning that jointly optimizes local control and network-wide traffic efficiency. Furthermore, VLALight introduces adaptive fast and slow reasoning modes, enabling the policy to allocate deeper reasoning only when additional deliberation provides sufficient control benefits. Through balanced mode-aware rollouts and relative advantage optimization, VLALight learns to trade off decision quality and inference cost. Extensive experiments on seven real-world traffic-flow datasets across three urban networks demonstrate that VLALight consistently outperforms transportation-based, RL-based, and LLM/VLM-based baselines. Ablation studies validate the effectiveness of cooperative perception, network-level optimization, and adaptive reasoning. These results demonstrate the potential of VLA models for real-world physical traffic control. Our project is available at https://github.com/usail-hkust/VLALight.git.
1 INTRODUCTION
Roadside cameras capture rich traffic observations, but existing controllers largely rely on engineered traffic states, leaving direct visual control underexplored. VLALight maps multi-view observations to coordinated signal actions, and experiments report improved network-wide efficiency and cost-effectiveness across seven real-world datasets.
1 INTRODUCTION
- Existing TSC systems largely use manually engineered traffic states, while roadside cameras offer richer physical observations and make direct visual control a natural opportunity.Source: §1 · p. 1
- VLA-based TSC must reason about evolving multi-target traffic, coordinate across network-connected intersections, and balance deliberation depth against inference latency.Source: §1 · p. 2
- VLALight directly maps multi-view videos to signal actions through spatiotemporal reasoning, topology-aware cooperative perception, staged supervised training, and cooperative agentic reinforcement learning.Source: §1 · p. 2
- The paper presents VLALight as an end-to-end VLA model connecting roadside observations with coordinated signal control, followed by cooperative training for multi-intersection control and adaptive reasoning.Source: §1 · p. 2
- Across seven real-world traffic datasets, VLALight consistently improves network-wide traffic efficiency and cost-effectiveness.Source: §1 · p. 3
2 PRELIMINARIES
The preliminaries formulate the road network as connected signalized intersections and define vision-based traffic signal control as a partially observable decision process. Intersections choose signal phases from visual observations and neighboring context to optimize network-wide traffic efficiency.
2 PRELIMINARIES
- A road network is a directed graph whose nodes are signalized intersections and whose links are directed lane connections along which vehicles, congestion, and signal effects propagate.Source: §2 · p. 3
- Vision-based traffic signal control is formulated as a POMDP with latent traffic states and visual observations, including camera frames from each intersection approach.Source: §2 · p. 3
- At each signal-switching step, a shared policy selects a feasible signal phase from local visual frames, neighboring-intersection context, and the intersection’s available phases to maximize long-term network efficiency.Source: §2 · p. 3
3 METHOD
VLALight maps multi-view intersection videos to coordinated signal actions by combining traffic perception, topology-aware context, and adaptive reasoning. Its training moves from supervised visual understanding and signal decisions to cooperative reinforcement learning that balances local and network-wide efficiency with reasoning cost.
3 METHOD
- VLALight uses multi-view intersection videos to extract movement-level traffic conditions and temporal dynamics, route neighboring perceptions, and coordinate network-wide signal control.Supervised fine-tuning establishes visual traffic understanding and signal decisions before cooperative agentic RL optimizes coordination using local and network-level rewards.Source: §3 · p. 4
3.1 VISION-BASED TRAFFIC SIGNAL CONTROL
- At each step, VLALight infers vehicle movements and changing demand and queues from video frames, then routes neighboring perceptions by lane connectivity to preserve temporal dynamics and spatial context.Fast mode predicts a phase directly; Slow mode adds a reasoning sequence before phase selection, allocating extra computation when deeper reasoning is beneficial.Source: §3.1 · p. 4
3.2 COOPERATIVE POLICY TRAINING
- Cold-start supervised training first maps videos to structured movement-level traffic perceptions, then trains signal decisions using teacher-generated reasoning trajectories and mode-labeled examples.Examples are labeled Fast when the queue gap between the two largest candidate-phase queues is at least five vehicles; otherwise they are labeled Slow.Source: §3.2 · p. 5
- Cooperative RL optimizes local and network-wide traffic efficiency, output-format consistency, and reasoning cost, while balanced Fast and Slow rollouts support comparison of their relative utility.Each reward signal is normalized independently within an intersection’s rollout group so large-scale traffic rewards do not overwhelm format or efficiency signals; group-relative advantages guide mode selection.Source: §3.2 · pp. 5–6
4 EXPERIMENTS
The experiments evaluate VLALight across seven real-world traffic datasets and compare it with traffic-control baselines. They assess overall performance, cooperative-training components, and adaptive reasoning under varying traffic pressure.
4.1 EXPERIMENTAL SETUP
- Although online RL is performed only on Jinan 1 and Hangzhou 1, VLALight generalizes to all five unseen traffic traces.The evaluation uses seven datasets from three urban networks and reports ATT, AQL, and AWT.Source: §4.2 · p. 6, §4.1 · p. 6
4.2 OVERALL PERFORMANCE (RQ1)
- VLALight achieves the best result in 16 of 21 dataset–metric comparisons; on New York 1 it reduces AQL by 13.0% and AWT by 12.1%, and on New York 2 by 5.5% and 6.7%, respectively.Table 1 covers Jinan, Hangzhou, and New York datasets; lower values are better.Source: §4.2 · pp. 6–7, Table 1 · p. 7
4.3 ABLATION STUDIES (RQ2)
- Removing cooperative information causes the largest ablation degradation, while removing network-level reward worsens every metric on the two New York datasets.Removing balanced Fast/Slow rollouts also reduces performance, especially on New York 2.Source: Table 2 · p. 7, §4.3 · p. 7
4.4.1 TRAFFIC PRESSURE AND ADAPTIVE MODE SELECTION
- The Slow-mode ratio rises from 38.86% in queue-free conditions to 49.81%, 60.39%, and 63.14% across successively higher queue-pressure groups.The queue-load groups are separated at 0.5 and 1.75 vehicles per phase.Source: Figure 2 · p. 7, §4.4.1 · p. 8
4.4 ADAPTIVE REASONING AND INFERENCE EFFICIENCY (RQ3)
- Balanced rollout training assigns slow reasoning to approximately 48% of decisions on both New York traces, while the unbalanced variant exceeds 72% and still has worse ATT.SFT uses slow mode less frequently but produces longer slow-mode outputs and weaker control performance.Source: Table 3 · p. 8, §4.4 · p. 8
- Adaptive inference averages 3.936 s per decision, 12.7% below slow-mode latency, while remaining close to Pure Slow and achieving the lowest AQL on Jinan 1.Fast and Slow decisions average 2.866 s and 4.507 s, respectively; decision time accounts for the efficiency difference.Source: §4.4 · p. 8, Table 4 · p. 8, Table 5 · p. 8
4.4.1 TRAFFIC PRESSURE AND ADAPTIVE MODE SELECTION
- Fast reasoning is used when one local phase is clearly preferable; slow reasoning is activated when competing phases or topology-routed neighbour pressure make decisions more ambiguous.Source: Figure 3 · p. 9, §4.4.1 · p. 9
5 RELATED WORK
Traffic signal control has progressed from fixed schedules and pressure-based rules to RL and LLM-assisted coordination, yet prior systems largely rely on structured states or modular planners. VLA research maps visual-language inputs to actions, but end-to-end control from physical traffic scenes remains unexplored in prior TSC systems.
Traffic Signal Control
- Prior TSC systems largely depend on structured simulator states or modular planners, leaving end-to-end control from physical visual observations unexplored.The field has moved from fixed schedules and pressure rules through RL coordination to LLM-assisted control.Source: §5 · p. 9
Vision-Language-Action Models
- VLA models map visual-language inputs to executable actions, with prior work spanning robotics, embodied reasoning, efficiency, and autonomous driving.AutoVLA uses supervised Fast/Slow modes and reinforcement fine-tuning to balance planning quality and reasoning cost.Source: §5 · p. 9
6 CONCLUSION
VLALight is presented as an end-to-end VLA framework for traffic signal control from multi-view roadside videos, combining spatiotemporal reasoning, cooperative perception, staged supervision, cooperative agentic RL, and adaptive reasoning. Experiments across seven datasets in three urban networks show consistent improvements over several controller families.
6 CONCLUSION
- VLALight consistently improves over transportation-, RL-, and LLM/VLM-based controllers across seven real-world traffic-flow datasets in Jinan, Hangzhou, and New York.Ablations validate cooperative perception, network optimization, and adaptive reasoning.Source: §6 · p. 9
- VLALight is presented as the first VLA framework for end-to-end traffic signal control from multi-view roadside videos.It connects physical visual observations to executable signal actions through multi-target spatiotemporal reasoning and topology-aware cooperative perception.Source: §6 · p. 9
- Two-stage supervised initialization and cooperative agentic reinforcement learning support network-level control with adaptive Fast/Slow reasoning.The reasoning modes balance traffic efficiency and inference cost.Source: §6 · p. 9
- The results highlight the potential of VLA models for real-world physical traffic control.Source: §6 · p. 9
A.1 SETTINGS OF TRAFFIC SIGNAL CONTROL
The experimental setting uses four-arm intersections with 12 incoming lanes and four controlled phases. The phases pair non-conflicting through or left-turn movements, while right turns remain permissive.
A.1 SETTINGS OF TRAFFIC SIGNAL CONTROL
- Each experimental intersection has four arms and 12 incoming lanes, with right turns remaining permissive.Source: Figure 4 · p. 14, §A.1 · p. 14
- Four controlled phases pair non-conflicting movements: east–west through, north–south through, east–west left turns, and north–south left turns.The phases are named ETWT, NTST, ELWL, and NLSL, respectively.Source: Figure 4 · p. 14, §A.1 · p. 14
A.2 DATASETS AND TRAFFIC NETWORKS
The evaluation uses seven one-hour traffic-flow traces from networks in Jinan, Hangzhou, and New York. The appendix describes their intersection layouts and reports vehicle counts and five-minute arrival statistics for each trace.
A.2 DATASETS AND TRAFFIC NETWORKS
- Seven real-world traffic-flow datasets cover three urban networks, with each trace specifying vehicle departures and routes for a one-hour episode.Source: §A.2 · p. 14
- Jinan has three datasets on a 12-intersection 3×4 grid, while Hangzhou has two datasets on a 16-intersection 4 × 4 grid.Source: §A.2 · p. 14
- New York contributes two taxi-demand datasets across 196 intersections in a 28 × 7 grid with 300-m road segments.Source: §A.2 · p. 14
- Table 6 reports vehicle counts and five-minute arrival statistics for every trace as mean ± standard deviation and observed minimum–maximum range.Arrival bins are 300 seconds, and the standard deviation is computed across bins.Source: §A.2 · p. 14, Table 6 · p. 15
A.3 COMPARED METHODS
The section summarizes the control interfaces and design principles of the methods as configured in the experiments. The compared methods include fixed-schedule and pressure-based controllers, learning-based approaches, and LLM/VLM-based methods.
A.3 COMPARED METHODS
- The descriptions cover the configurations used in the experiments.They summarize the compared methods’ control interfaces and design principles.Source: §A.3 · p. 15
- FixedTime uses a predetermined cycle and phase schedule, while MaxPressure selects the phase with the largest pressure differential.MaxPressure computes the differential from queue imbalance between incoming and outgoing movements.Source: §A.3 · p. 15
- PressLight and MPLight use pressure-derived features, while CoLight and Advanced-CoLight use cooperative information across intersections.Efficient-MPLight and Efficient-CoLight use compact pressure observations; CityLight uses parameter-sharing MAPPO for a universal policy across heterogeneous intersections.Source: §A.3 · p. 15
- LLMLight uses an LLM for signal decisions, CoLLMLight adds cooperative traffic information, and the experiments adapt VLMLight to activate VLM control under high traffic pressure.The adapted VLMLight uses a shared Advanced-CoLight policy normally and VLM-based phase selection and validation at high pressure.Source: §A.3 · p. 15
A.4 MODEL SETTINGS
The model settings specify shared hyperparameters for RL baselines and implementation settings for VLALight. These include its Qwen3.5-4B base, LoRA fine-tuning configuration, and video input format.
A.4 MODEL SETTINGS
- All RL baselines share a learning rate of 1 × 10−3, replay-buffer capacity of 12,000, sample size of 3,000, and hidden size of 20.Source: §A.4 · p. 15
- VLALight uses Qwen3.5-4B and LoRA supervised fine-tuning with rank 16, scaling factor 32, dropout 0.05, and learning rate 1 × 10−4.Source: §A.4 · p. 16
- Each VLALight video input contains six frames at 512 × 960-pixel resolution.Source: §A.4 · p. 16
A.5 AGENTIC RL SETTINGS
This section describes VLALight’s cooperative agentic-RL training and inference pipeline, including synchronized network rollouts and topology-routed neighbor information. It also covers how Stage 2 selects a signal phase and fast or slow reasoning mode based on local and cooperative context.
A.5 AGENTIC RL SETTINGS
- Agentic RL starts from the cold-start Qwen3.5-4B policy and trains on Jinan 1 and Hangzhou 1 streams with balanced Fast/Slow rollouts and a retained perception loss.Each step uses two instances per dataset with different seeds; training runs for 50 steps at a policy learning rate of 1 × 10−6, retaining perception supervised loss with coefficient 0.1.Source: §A.5 · p. 16
A.6 COOPERATIVE VLA ROLLOUT AND POLICY UPDATE
- Each network snapshot supplies shared perceptions and routed contexts to joint candidate rollouts, whose network rewards are shared while local traffic, format, and reasoning-cost signals remain intersection-specific.The policy update combines a Fast-versus-Slow mode-selection update with a joint update across intersections that does not split by reasoning mode.Source: §A.6 · p. 16
A.7 TWO-STAGE COOPERATIVE VLA INFERENCE
- At deployment, the controller extracts structured traffic states from four approach videos, routes context through the road topology, then selects Fast or Slow reasoning before generating a signal action.Stage 2 combines local and cooperative perceptions to make the phase decision.Source: §A.7 · p. 17
A.8 CONSTRUCTION OF COOPERATIVE INFORMATION
- Cooperative information combines downstream-linked video frames estimating potential arrivals with upstream vehicle and queue counts plus travel time summarizing directional pressure.Upstream pressure is not a guarantee of arrivals because upstream signals control whether and when vehicles are released.Source: §A.8 · p. 17, Figure 5 · p. 18
A.9.1 FAST CASE
- At New York intersection 1-19, NTST’s visible and queued demand, recent increase, and unserved age make it the clear local winner, so VLALight selects it using fast reasoning.NTST has v/q = 9/9 and ∆v/∆q = +4/ + 8; upstream context offers no direct phase-level arrivals and only modest neighbor pressure.Source: Figure 6 · p. 19, §A.9.1 · p. 20
A.9.2 SLOW CASE
- At New York intersection 3-1, local demand is ambiguous, but routed context adds 19 direct ETWT arrivals and upstream pressure, prompting slow reasoning and selection of ETWT.ETWT has v/q = 20/20 and has waited two intervals, while NTST and NLSL each have v/q = 19/19 and were served more recently.Source: Figure 7 · p. 19, §A.9.2 · p. 20
A.10.1 STAGE 1: VIDEO-GROUNDED PERCEPTION
- Stage 1 uses four direction-specific videos and available upstream frames to estimate phase-level visible vehicles, stopped subsets, demand changes, persistent unmet demand, and potential arrivals.Visible vehicle count, stopped-vehicle count, and coordinated-arrival estimates are kept distinct; arrivals are future-pressure estimates, not part of local visible demand.Source: §A.10.1 · pp. 20–21
A.10.2 STAGE 2: ROUTED SIGNAL DECISION
- Stage 2 receives local and cooperative perceptions, choosing Fast when one phase is clearly preferred and Slow when local and cooperative evidence require explicit comparison.It considers visible demand and queue pressure, recent changes, unserved age, direct potential arrivals, and broad directional neighbor pressure.Source: §A.10.2 · pp. 21–22
- Choose fast mode when one candidate phase is clearly preferable; choose slow mode when phases are comparable or local trends, direct coordination, persistent demand, and neighbor context disagree.The decision output uses a mode tag and selected signal phase; slow mode also includes concise reasoning.Source: §A.10.2 · p. 22
- Stage 1’s movement-level observations are aggregated into phase-level vehicle and queue counts, while coordination entries are routed separately as potential arrivals.Stage 2 receives local and cooperative perception blocks and combines them to select the reasoning mode and signal phase.Source: §A.10.2 · pp. 22–23