PaperlayerView exact revision on arXiv

Source-linked AI summary

VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control

Kemou Jiang, Maonan Wang, Xingchen Zou, Jiayue Zhu, Yuhang Fu, Sicheng Wang, Xi Chen, Yirong Chen, Zhiyong Cui

arXiv:2609.30709v1cs.CVcs.AI

TL;DR

Existing TSC approaches represent traffic with compact hand-crafted statistics or convert images to text in cascaded VLM pipelines, limiting direct use of fine-grained visual context. VLALight maps multi-view intersection observations and signal-phase information directly to discrete actions, achieving 21.1% lower pooled emergency waiting time than VLMLight (32B).

  • Problem

    Existing TSC controllers use compact hand-crafted traffic statistics, while cascaded VLM pipelines convert images to text, leaving limited direct use of fine-grained visual context in control.

    Source: Introduction · p. 1

  • Method

    VLALight uses a compact end-to-end vision-language-action model to map unified multi-view intersection observations and signal-phase information directly to discrete signal actions.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Results

    21.1% lower pooled EWT than cloud-scale VLMLight (32B); VLALight attains the strongest emergency-vehicle service among compared controllers.

    Source: Traffic Efficiency Comparison · p. 6, Conclusion · p. 7

  • Takeaways & Limitations

    At 224 ms per decision on local hardware, VLALight stays within the 5-second decision budget for emergency-aware service without streaming surveillance feeds to the cloud.

    Source: Inference Latency and Deployment… · p. 6

    Evaluation is limited to simulation, leaving real-world validation and formal safety guarantees for future work.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Traffic signal control (TSC) is essential for mitigating urban congestion. Recent advances in vision-language models (VLMs) enable richer interpretation of intersection scenes, opening new opportunities for visual-context-aware TSC. However, the loose coupling and repeated information conversion between modules can lead to the loss of fine-grained visual details, while sequential inference introduces substantial latency. To address these limitations, we propose VLALight, a lightweight end-to-end vision-language-action framework that directly maps intersection observations and signal-phase information to discrete signal actions. To handle the multi-view nature of TSC, VLALight combines multiple directional camera views into a unified visual input and uses textual instructions to establish their correspondence with traffic movements and signal phases. This design enables direct action prediction with a compact 0.5 B-parameter model, without intermediate image-to-text descriptions or handcrafted traffic-state representations. Experiments show that VLALight delivers the best emergency-vehicle service of all compared methods, reducing pooled emergency waiting time by 21.1% over the cascaded VLMLight while running in real time on local hardware and generalizing to unseen intersection topologies and traffic-flow patterns.

Introduction

VLALight brings end-to-end vision-language-action control to traffic signals to preserve visual detail and act directly, addressing multi-view, topology, emergency-response, and real-time constraints. It combines a simulation-grounded dataset with a compact model that maps intersection observations and signal-phase information to discrete actions.

Introduction

  • Hand-crafted traffic statistics discard fine-grained scene information, while cascaded vision-language pipelines compress images into text and add latency, limiting abnormal-scenario and real-time control.These limitations matter especially during accidents, sudden congestion, and emergency-vehicle priority requests.Source: Introduction · p. 1
  • VLALight brings end-to-end vision-language-action modeling, previously focused on robot manipulation, to traffic signal control, unifying perception, language understanding, and direct action prediction.The approach is intended to preserve visual detail up to the decision layer.Source: Introduction · p. 1
  • Traffic control requires relating multiple directional camera views and lane structures, generalizing across intersection topologies, prioritizing rare emergency vehicles, and operating with practical limits on model size and adaptation cost.Emergency vehicles such as ambulances and fire engines may require immediate phase-priority shifts amid ordinary traffic.Source: Introduction · pp. 1–2
  • VLALight stitches camera views into a panorama, pairs them with instructions linking views, movements, and phases, then uses a compact 0.5B multimodal model to predict the next phase in one forward pass.The model uses LoRA adaptation and learnable phase queries, without intermediate image-to-text descriptions or handcrafted traffic-state representations.Source: Introduction · p. 2
  • VLALight’s simulation-grounded data pipeline replicates real-intersection lane topology, generates diverse flows with emergency vehicles, renders high-fidelity multi-view images, and collects expert labels under human verification.The pipeline yields a verified multi-intersection dataset for emergency-aware signal control.Source: Introduction · p. 2

Related Work

Related work traces traffic signal control from fixed-time and adaptive methods through learning-based policies, then surveys language- and vision-language systems and end-to-end vision-language-action models. It highlights VLA work in robotics, including efforts to improve efficiency, reasoning, action representations, safety, and compactness.

Traffic signal control

  • TSC has progressed from fixed-time plans and adaptive methods such as SOTL (Gershenson 2005) and Max Pressure to deep reinforcement learning.Deep RL includes value-based methods (Clifton and Laber 2020; van Hasselt, Guez, and Silver 2016; Huang, Wei, and Wang 2018), policy gradients (Schulman et al. 2017), and multi-agent coordination (Wei et al. 2019b; Chen et al. 2020; Wei et al. 2018, 2019a; Oroojlooy et al. 2020; Wang et al. 2024b).Source: Related Work · p. 2

LLMs and VLMs for traffic control

Vision-language-action models

  • VLA models cast control as generative action prediction, demonstrated at scale in robotic manipulation by RT-1 (Brohan et al. 2023), RT-2, Octo (Octo Model Team et al. 2024), and OpenVLA (Kim et al. 2024).Follow-on work addresses efficiency, reasoning, action representations, and safety; compact architectures such as RoboFlamingo (Li et al. 2024), UniVLA (Bu et al. 2025), and OpenVLA-OFT (Kim, Finn, and Liang 2025) reduce model size or latency, and Yu et al. 2025 surveys the literature.Source: Related Work · p. 2

Method

The method formulates emergency-aware traffic signal control as a vision-language-action decision problem, describes a simulation-grounded training-data pipeline, and presents the VLALight policy.

  • The method frames emergency-aware traffic signal control as a vision-language-action decision problem.Source: Method · p. 2
  • It describes a simulation-grounded pipeline for generating the training data.Source: Method · p. 2
  • It then presents the VLALight policy.Source: Method · p. 2

Problem Formulation

VLALight formulates traffic-signal control as choosing a discrete signal phase from a stitched intersection image and a topology-and-phase instruction, optimizing traffic efficiency while prioritizing emergency requests.

Problem Formulation

  • At each discretized decision step, the agent observes a stitched image of all approaching directions and an instruction describing intersection topology and signal-phase meanings.The intersection has K phases, each grouping non-conflicting movements that receive green simultaneously.Source: Problem Formulation · p. 2
  • The policy πθ(at | It, L) selects one of K discrete phases to minimize average vehicle travel and waiting times while responding to emergency-vehicle priority requests.The selected phase is represented as an integer from 0 to K−1 or an equivalent K-dimensional one-hot vector.Source: Problem Formulation · p. 3

Synthesizing Labeled Intersection Scenes from Real Topology

Because real footage and executed signal labels are scarce, VLALight’s training data is generated in simulation by reproducing real topology, collecting expert-labeled flows, and rendering realistic camera images.

Road Network Topology Extraction

Expert Label Collection

  • Routine-demand steps use maximum-pressure expert labels, while emergency-present steps use multimodal-LLM phase choices; traffic-signal experts verify all labels.Emergency vehicles are injected with predefined types, entry times, and paths, teaching the policy to balance pressure and prioritize emergency movements.Source: Expert Label Collection · pp. 3–4

Realistic Image Rendering

  • Blender renders realistic lighting, materials, and vehicle appearance to narrow the sim-to-real visual gap; each intersection uses up to four roadside cameras matching surveillance viewpoints.Source: Realistic Image Rendering · p. 4

Post-Training and Inference

VLALight adapts a frozen pretrained vision-language model with lightweight trainable modules to predict signal phases in one forward pass. It jointly processes stitched directional camera views, maps phase queries to discrete actions, and trains the policy through expert-label behavioral cloning.

Post-Training and Inference

  • Following the VLA-Adapter paradigm (Wang et al. 2026), VLALight augments a frozen pretrained VLM with lightweight trainable modules, including learnable phase queries, to predict the next phase in one forward pass.Source: Post-Training and Inference · p. 4

Mutli-view Visual

  • Directional camera views are stitched into a 2 × 2 panoramic input and jointly encoded by frozen DINOv2 and SigLIP into 256 patch tokens for cross-view phase decisions.The 2 × 2 token layout preserves each direction's identity, while textual instructions map camera views to movements and phase indices and expose intersection topology through text rather than architecture.Source: Mutli-view Visual · p. 4

TSC-Native Action Space

  • Each phase is classified as a discrete action using learnable phase queries and an action head, and deployment enforces minimum-green dwell before switching.Source: TSC-Native Action Space · p. 4

Policy Learning

  • Training uses behavioral cloning on verified expert labels, updating only the projector, LoRA, phase queries, and action head while keeping the pretrained encoders and base LLM frozen.A residual action state is refined through per-layer tri-branch attention over action, phase-query, and visual states; a zero-initialized tanh gate gradually incorporates visual context.Source: Policy Learning · pp. 4–5

Experiments · Experimental Settings

Experiments evaluate VLALight across six simulated real-world intersections under regular and varied traffic flows, using four traffic and emergency-service metrics and comparisons with rule-based, RL-based, and VLM-based controllers.

Datasets and Simulation

  • One model is evaluated in closed-loop simulation across six intersections on regular, off-peak, rush-hour, and fluctuating flows, with four seeds per city-flow pair.Figure 3 pairs each real intersection with its rendered simulation.Source: Datasets and Simulation · p. 5, Figure 3 · p. 5
  • At evaluation, VLALight selects the highest-scoring action subject to the minimum-green dwell constraint.Source: Datasets and Simulation · p. 5

Evaluation Metrics

  • Evaluation reports mean and standard deviation for ATT, AWT, ETT, and EWT in seconds, with lower values indicating better performance.ATT and AWT measure overall traffic efficiency; ETT and EWT measure emergency-vehicle service.Source: Evaluation Metrics · p. 5

Compared Methods

  • The comparison includes four rule-based controllers, four RL-based controllers, and the VLM-based VLMLight.The named methods are FixTime, Webster, SOTL, MaxPressure; IntelliLight, PressLight, AttendLight, UniTSA; and VLMLight.Source: Compared Methods · p. 5
  • VLMLight is tested with 32B and 3B backbones; its fast branch uses pretrained UniTSA/PPO, while its slow branch invokes cloud-scale VLM/LLM agents.Source: Compared Methods · p. 5
  • VLALight uses a 0.5B backbone and runs entirely locally.Source: Compared Methods · p. 5

Traffic Efficiency Comparison

VLALight provides strong emergency-vehicle service across six cities, with pooled improvements over VLMLight (32B), while its performance varies in two cities. A camera-view emergency vehicle can trigger a phase switch at the next decision step.

Traffic Efficiency Comparison

  • Pooled across six cities, VLALight improves on VLMLight (32B) by 7.0% in ETT and 21.1% in EWT, attaining the lowest values for both metrics in four cities.The exceptions are Beijing Changjianglu and Tianjin Zhijingdao, where rule-based controllers or VLMLight (32B) rank ahead.Source: Traffic Efficiency Comparison · p. 6
  • In Beijing Changjianglu and Tianjin Zhijingdao, rule-based controllers or VLMLight (32B) rank ahead of VLALight.Source: Traffic Efficiency Comparison · p. 6
  • When an emergency vehicle appears in one camera view, VLALight switches to the phase serving its movement at the next decision step, allowing it to pass without stopping.The paper reports that this 0.5B end-to-end policy can match or surpass the emergency-vehicle service of a cloud-scale cascaded system.Source: Traffic Efficiency Comparison · p. 6

Inference Latency and Deployment Analysis · Ablation Study

VLALight keeps emergency-aware decisions within the 5 s budget through a single local forward pass. In the ablation, stitching the camera views improves regular-flow ATT and ETT and reduces inference latency compared with separately encoding the views.

Inference Latency and Deployment Analysis

  • VLALight averages 224 ms per emergency-aware decision, versus 10.9 s for VLMLight’s cloud 32B backbone and 4.2 s for its local 3B backbone.The VLMLight timings are close to or beyond the 5 s decision interval; Table 2 summarizes emergency-scenario latency.Source: Inference Latency and Deployment… · p. 6, Table 2 · p. 6
  • VLALight is the only compared controller that stays within the decision budget with emergency awareness always enabled, supporting fully on-premises deployment.Its action comes from a single local forward pass without streaming surveillance feeds to the cloud.Source: Inference Latency and Deployment… · p. 6

Ablation Study

  • Stitching the four camera views reduced pooled ATT from 107.0 s to 52.1 s and ETT from 127.7 s to 44.7 s under regular flow.Figure 5 compares the stitched image with four separately encoded camera views across every city and seed.Source: Ablation Study · p. 6, Figure 5 · p. 7
  • Separately encoding the four views took 450 ms per decision, compared with 224 ms for the stitched input.The unstitched variant is twice as slow at inference.Source: Ablation Study · pp. 6–7, Figure 5 · p. 7
  • Separate encoding quadruples visual tokens from 256 to 1,024, inflates activation memory, limits training batch size, and challenges the 0.5B backbone’s cross-view self-attention.The added tokens do not provide additional spatial information.Source: Ablation Study · p. 6
  • The stitched input is superior in control quality, emergency response, memory footprint, and latency, supporting joint encoding of directional views as one image.The comparison validates the single stitched visual input design.Source: Ablation Study · p. 7
  • An ablation of the minimum-green dwell constraint is provided in supplementary Section C.The supplied passage does not report its results.Source: Ablation Study · p. 7

Generalization Analysis

VLALight generalizes to unseen traffic-flow patterns without retraining and transfers to unseen intersection topologies in few-shot and zero-shot settings.

Generalization to unseen traffic flows

  • VLALight improves on VLMLight (32B) by 5.8% in ETT and 20.8% in EWT pooled across three unseen test flows.The EWT advantage is on par under off-peak conditions, then reaches 16.5% under rush-hour and 33.1% under fluctuating flow.Source: Generalization Analysis · p. 7, Figure 6 · p. 7

Generalization to unseen topologies

  • On two unseen intersections, the few-shot variant stays within 1–3% of the fully trained model on ETT and EWT.With no target data, the zero-shot variant’s ETT and EWT rise by 4.3% and 11.4%, and ATT by 7.6%.Source: Generalization Analysis · p. 7, Figure 7 · p. 7

Conclusion

VLALight directly maps unified multi-view intersection observations to signal actions and achieves strong emergency-vehicle service with a compact model. Its current limitations include simulation-based behavior-cloning training, single-frame decisions, and evaluation without real-world validation or formal safety guarantees.

Conclusion

  • VLALight achieved the strongest emergency-vehicle service, with lowest ETT and EWT in four of six cities and 21.1% lower pooled EWT than cloud-scale VLMLight.Experiments covered six real-world intersections under four traffic-flow patterns.Source: Conclusion · p. 7
  • Stitching directional camera views into a text-grounded unified input lets a compact 0.5B model predict discrete signal actions in a single forward pass.Each decision took 224 ms on local hardware, and the framework generalized to unseen intersection topologies and traffic-flow patterns.Source: Conclusion · p. 7
  • Current limitations include behavior-cloning training on simulation-collected expert labels, single-frame decisions, simulation-only evaluation, and no formal safety guarantees.Reinforcement fine-tuning and temporal memory are identified as future directions; real-world validation and formal safety guarantees remain future work.Source: Conclusion · p. 7
Loading 2609.30709v1…