PaperlayerView exact revision on arXiv

Source-linked AI summary

RedLight-VLA: Models for traffic-rule grounding and behavioral emphasis in driving policies

Bala Murali Manoghar Sai Sudhakar, Sourab Bapu Sridhar, Sandipan Das, Rahul Ahuja, Meda Lazar, Ashish Garg, Pratik Likhar, Senthil Yogamani

arXiv:2608.28656v1cs.ROcs.CV

TL;DR

Behavior-cloned VLA driving policies underrepresent rare braking and launching behaviors and lack explicit supervision for traffic-light and stop-line state. RedLight-VLA combines behavior-derived reweighting with post-fusion auxiliary rule-token supervision using automatically generated signals. Against an otherwise identical baseline, the combined model improves rule-compliance and sliced displacement metrics, while increasing green-light false stops.

  • Problem

    Rare braking and launching examples contribute little to averaged trajectory loss, while fused VLA representations lack explicit supervision for governing traffic-light and stop-line state.

  • Method

    RedLight-VLA combines scale-preserving behavioral reweighting from expert-future longitudinal dynamics with parallel post-fusion auxiliary heads for perception-derived traffic-light and stop-line targets.

  • Results

    RedLight-VLA reduces red-light stop-line overshoot from 7.3% to 6.8% and achieves the lowest traffic-light-sliced ADE/FDE among evaluated conditions.

  • Takeaways & Limitations

    The combined model improves both traffic-light and non-traffic-light displacement measures and outperforms either mechanism alone on the four sliced displacement measures.

  • Takeaways & Limitations

    The evaluation uses open-loop point estimates on a single curated dataset, with auxiliary targets inheriting upstream perception errors and no transfer evaluation on public benchmarks.

Abstract

from arXiv · show

Behavior-cloned Vision-Language-Action (VLA) driving policies struggle with rare rule-governed maneuvers at signalized intersections. Braking and launching examples contribute little to averaged trajectory loss, while fused representations lack explicit supervision for the governing traffic-light and stop-line state. We present RedLight-VLA, a training objective that uses expert futures and automatically generated perception targets without additional manual rule annotation. First, trajectory-derived behavioral reweighting (BR) emphasizes rare deceleration and acceleration using rotation-invariant longitudinal dynamics and a scale-preserving reduction that exactly recovers the baseline when disabled. Second, parallel auxiliary (AUX) heads ground traffic-light and stop-line state in continuous post-fusion rule tokens, without autoregressive language generation or changes to the trajectory decoder. We evaluate on a curated set of 20 s sequences with a 5 s prediction horizon. Controlled variants share the same backbone, training data, decoder, and evaluation population. Against an otherwise identical VLA baseline, RedLight-VLA reduces red-light stop-line overshoot from 7.3% to6.8%, reduces stop-line velocity error by 12.7%, and improves 3 s trafficlight-sliced ADE/FDE from 0.274/0.964 m to 0.247/0.897 m. Green-light false stops increase from 3.2% to 3.9%; however, combining BR with AUX supervision mitigates the larger increase observed for AUX alone (4.0%). The combined model also improves non-traffic-light ADE/FDE from 0.268/0.956 m to 0.241/0.876 m and outperforms either mechanism alone on all four sliced displacement measures.

1 Introduction

RedLight-VLA addresses two weaknesses in behavior-cloned VLA driving policies: rare longitudinal behaviors receive little loss weight, and rule-governed events lack explicit fused-representation supervision. It combines trajectory-derived behavioral reweighting with post-fusion auxiliary rule-token supervision without manual rule annotation or trajectory-head changes.

  • Motivation: Rare braking and launching behaviors contribute little to averaged regression loss, producing underbraking and difficulty resuming from rest.These longitudinal tail behaviors are safety-critical but largely invisible to aggregate displacement error.
  • Motivation: Trajectory supervision does not explicitly identify the traffic-light and stop-line state governing correct actions at signalized intersections.Text-based grounding would couple rule supervision to tokenizer-dependent autoregressive decoding.
  • Evaluation: The framework uses two independently toggled loss-layer mechanisms and an open-loop rule-compliance suite based on existing perception signals.The suite measures red-light overshoot, green-light false stops, and stop-line velocity error.
  • Method: Behavioral reweighting emphasizes rare braking and launching using expert-future longitudinal dynamics and a scale-preserving weighted mean that exactly recovers the baseline when disabled.The method uses rotation-invariant dynamics and preserves loss calibration.
  • Method: Auxiliary rule-token supervision grounds traffic-light and stop-line state in reserved post-fusion tokens using parallel MLP heads, without language decoding or trajectory-path changes.The heads supervise fused slots against perception-derived rule outputs.

2 Related Work

Prior work addresses driving covariate shift, imbalance, auxiliary representation shaping, and traffic-rule awareness through distinct strategies. RedLight-VLA differs by reweighting continuous trajectory regression from behavior and supervising reserved post-fusion tokens with perception-derived rule targets.

  • Behavior cloning: Prior behavior-cloning work spans end-to-end control, trajectory prediction, planning, and language-conditioned policies, while DAgger and perturbation augmentation address rollout covariate shift.This paper targets demonstration-distribution imbalance and missing rule supervision instead.
  • Long-tail learning: Existing imbalance methods use class labels, optimizer-state signals, or learned parameters, whereas RedLight-VLA reweights continuous trajectory regression from behavioral dynamics.The approach is designed for rare longitudinal driving behaviors rather than discrete class imbalance.
  • Auxiliary supervision: Driving policies commonly attach auxiliary heads to shared backbone features, but RedLight-VLA reads supervised rule state from reserved tokens after language-model fusion.The reserved slots are explicitly grounded by perception-derived labels and leave the prediction path untouched.
  • Traffic-rule awareness: Traffic-rule-aware systems use perception heads, safety controllers, traffic-light tokens, or language reasoning, with evaluation often based on simulator scores or infractions.The cited approaches differ in where rule signals are learned and whether they are read from the fused sequence.

3 Method

RedLight-VLA adds two loss-level mechanisms to a standard VLA trajectory trainer: behavioral reweighting for rare longitudinal maneuvers and auxiliary post-fusion rule-token supervision. Both use existing training signals, leave the trajectory decoder unchanged, and can be enabled independently.

  • Behavioral reweighting: Behavioral reweighting emphasizes rare braking and launching samples using dynamics derived directly from expert futures.Weights are computed without extra labels and applied through the trajectory-loss reduction.
  • Behavioral reweighting: Rotation-invariant speed uses displacement-vector length, making the same behavioral thresholds apply on straights and through turns.The method derives speed and acceleration from per-step displacements rather than the forward component alone.
  • Behavioral reweighting: DCCL targets sustained deceleration, hard braking, and general slowing, while ACCL targets launches from rest, hard acceleration, and speed-up.Trigger weights are fixed from training-split frequencies, with rarer events receiving larger weights.
  • Behavioral reweighting: The weighted reduction normalizes weights over valid batch elements, exactly recovering the ordinary masked mean when all weights equal 1.Global rescaling leaves the reduction unchanged, so reweighting redistributes gradients without inflating loss scale.
  • Auxiliary rule-token supervision: Auxiliary rule-token supervision uses reserved post-fusion tokens and parallel heads to predict traffic-light and stop-line state from automatically generated perception targets.The traffic-light head predicts presence and color; the stop-line head predicts presence, distance, and orientation, without language decoding or trajectory-head changes.

4 Experiments

Experiments use a curated driving dataset with perception-derived rule targets, controlled mechanism toggles, and scenario-bucketed trajectory evaluation. The evaluation separately measures rule compliance, auxiliary-task accuracy, external-planner comparisons, and behavioral-trigger coverage.

  • Dataset and setup: The dataset supplies front-wide image history, kinematic history, maps, calibration, and goal points for fixed-horizon trajectory prediction.Auxiliary targets come from existing traffic-light tensors and road-line polylines, so no additional manual annotation is introduced.
  • Experimental conditions: The four headline conditions are Baseline, BR, AUX, and Full, formed by independently toggling behavioral reweighting and both auxiliary rule-supervision tasks.All conditions share the same VLA trajectory predictor, training setup, and evaluation population.
  • Trajectory evaluation: Trajectory evaluation buckets samples by speed, curvature, and future kinematic events separately on traffic-light and non-traffic-light slices.Per-bucket ADE at 5 s and overall ADE are reported, with future deceleration and acceleration as headline buckets.
  • Traffic-light evaluation: Traffic-light-sliced evaluation reports 3 s ADE/FDE on perception-defined traffic-light and non-traffic-light subsets.The non-traffic-light slice serves as a specificity control for whether improvements are concentrated in rule-governed scenes.
  • Rule compliance and diagnostics: Open-loop rule-compliance metrics measure red-light overshoot, green-light false stops, and stop-line velocity error.Auxiliary heads are additionally scored on traffic-light presence and color, stop-line presence, distance MAE, and orientation MAE.
  • Comparisons and diagnostics: The study also compares external VLA planners under shared data and evaluation conditions and reports joint kinematic-by-rule trigger coverage.These diagnostics quantify the imbalance targeted by the method and support direct planner comparison where adapters are available.

5 Results

RedLight-VLA improves displacement and rule-compliance outcomes over the controlled baseline, with gains across traffic-light and non-traffic-light slices. The combined model also reaches favorable data-scaling and planner-comparison results, while auxiliary-target agreement tracks perception-derived supervision rather than causal use.

  • 1.5% hard-brake and 2.8% launch-from-rest frame coverage confirms that both targeted behaviors lie in the long tail.
  • 7.3% to 6.8% red-light overshoot and 0.816 to 0.712 m/s stop-line velocity error show improved open-loop rule compliance, while green-light false stops rise to 3.9%.AUX alone reaches 4.0% green-light false stops, and Full matches AUX on overshoot while improving the other reported rule-compliance measures.
  • Full improves over both BR and AUX in every displacement cell and over AUX on all three rule-compliance metrics, although the evidence is based on point estimates.BR’s standalone rule-compliance behavior was not evaluated.
  • At 75% of the training data, four plotted metrics are lower than the full-data Baseline, while green-light false stops remain above Baseline.The Full trend crosses Baseline near 70% data for ADE/FDE, red-light overshoot, and stop-line velocity error.
  • Full attains the lowest error in all four TL/non-TL displacement cells among the compared planners, with Alpamayo-R1 the strongest comparison planner.
  • Full lowers point-estimate ADE in 19 of 20 slice–scenario cells, with the largest targeted gains in future deceleration and acceleration.The one non-improving cell is stationary TL frames, which increase by 0.0016 m.
  • Reserved-token agreement reaches 95.3% for traffic-light presence, 93.7% for color, and under 2.7 m stop-line distance MAE, but these metrics use perception-derived targets.

6 Discussion and Limitations

The results suggest a conservative tradeoff: auxiliary supervision and the combined model reduce red-light overshoot but increase green-light false stops, while displacement gains are similar across traffic-light and non-traffic-light slices. The approach remains bounded by longitudinal triggers, perception-target quality, open-loop evaluation, and a single curated dataset.

  • Tradeoffs: AUX and Full reduce red-light overshoot but increase green-light false stops, indicating a conservative bias.The text cautions that the absence of BR standalone rule metrics and uncertainty estimates prevents causal attribution.
  • Tradeoffs: Full’s displacement gains are similar on traffic-light and non-traffic-light slices, so rule-compliance metrics distinguish signalized-intersection performance.
  • Limitations: The method targets longitudinal rarity only; lateral rarity is not directly addressed.
  • Limitations: Auxiliary targets inherit upstream perception errors and are evaluated against those targets rather than independent ground truth.
  • Limitations: Evaluation is open-loop, reports point estimates without uncertainty intervals, and uses a single curated dataset without public-benchmark transfer.The paper leaves closed-loop real-scene scoring and transfer to nuPlan and NAVSIM for future work.

7 Conclusion

RedLight-VLA addresses the driving long tail with automatically generated training signals, combining behavioral reweighting and post-fusion rule-token supervision without changing the trajectory decoder. The combined model improves rule-compliance and sliced displacement metrics, while increasing green-light false stops less than AUX alone.

  • The framework uses automatically generated training signals without additional manual rule annotation.
  • Behavioral reweighting emphasizes rare braking and launching through a scale-preserving reduction that exactly recovers the baseline objective when disabled.
  • Auxiliary supervision grounds traffic-light and stop-line state from reserved post-fusion tokens without changing the trajectory head or adding autoregressive language generation.
  • 7.3% to 6.8% red-light stop-line overshoot and 12.7% lower stop-line velocity error are achieved by the combined model.
  • The combined model achieves the lowest traffic-light-sliced ADE/FDE among evaluated conditions and improves over each mechanism alone.
  • Green-light false stops increase, but the combined model mitigates the larger increase observed for AUX alone.
Loading 2608.28656v1…