PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist Systems

Yuhang Wang, Hao Zhou

arXiv:2505.11535v1cs.ROcs.CVcs.LG

TL;DR

Drivers lack a reliable way to anticipate LKA failures before performance is compromised. LKAlert combines dash-cam video, CAN data, and lane-segmentation guidance to issue predictive alerts with explanations; it reports 69.8% accuracy, 58.6% F1-score, 71.7 ROUGE-L, and approximately 2 Hz operation.

  • Problem

    Drivers lack a reliable mechanism to anticipate LKA failures before performance is compromised, amid failures that are difficult to predict from the driver's perspective.

    Source: §I · p. 1

  • Method

    LKAlert uses a VLM to combine dash-cam imagery, CAN data, and surrogate lane-segmentation masks to predict LKA failures 1–3 seconds ahead and generate explanations.

    Source: §I · pp. 1–2, 1) Inputs and Outputs · p. 4

  • Results

    LKAlert achieves 69.8% accuracy, 58.6% F1-score, and 71.7 ROUGE-L, operating at approximately 2 Hz.

    Source: §V · p. 7

  • Takeaways & Limitations

    The reported performance and approximately 2 Hz operation demonstrate suitability for real-time, resource-constrained deployment to enhance ADAS safety and usability.

    Source: §V · p. 7

Abstract

from arXiv · show

Lane Keeping Assist systems, while increasingly prevalent, often suffer from unpredictable real-world failures, largely due to their opaque, black-box nature, which limits driver anticipation and trust. To bridge the gap between automated assistance and effective human oversight, we present LKAlert, a novel supervisory alert system that leverages VLM to forecast potential LKA risk 1-3 seconds in advance. LKAlert processes dash-cam video and CAN data, integrating surrogate lane segmentation features from a parallel interpretable model as automated guiding attention. Unlike traditional binary classifiers, LKAlert issues both predictive alert and concise natural language explanation, enhancing driver situational awareness and trust. To support the development and evaluation of such systems, we introduce OpenLKA-Alert, the first benchmark dataset designed for predictive and explainable LKA failure warnings. It contains synchronized multimodal inputs and human-authored justifications across annotated temporal windows. We further contribute a generalizable methodological framework for VLM-based black-box behavior prediction, combining surrogate feature guidance with LoRA. This framework enables VLM to reason over structured visual context without altering its vision backbone, making it broadly applicable to other complex, opaque systems requiring interpretable oversight. Empirical results correctly predicts upcoming LKA failures with 69.8% accuracy and a 58.6\% F1-score. The system also generates high-quality textual explanations for drivers (71.7 ROUGE-L) and operates efficiently at approximately 2 Hz, confirming its suitability for real-time, in-vehicle use. Our findings establish LKAlert as a practical solution for enhancing the safety and usability of current ADAS and offer a scalable paradigm for applying VLMs to human-centered supervision of black-box automation.

I. INTRODUCTION

LKA failures can be difficult for drivers to anticipate, yet no reliable mechanism proactively warns them before performance is compromised. LKAlert combines VLM-based early warnings and explanations with a benchmark dataset and a surrogate-guidance framework.

I. INTRODUCTION

  • LKAlert uses VLMs to forecast potential LKA failures 1–3 seconds ahead from dash-cam video and CAN data.It provides a predictive alert and concise natural-language explanation to address system opacity.Source: §I · p. 1
  • OpenLKA-Alert is a benchmark for predictive, explainable LKA failure warnings, with synchronized multimodal data and natural-language justifications.The dataset is intended to support development and evaluation of transparent ADAS safety features.Source: §I · p. 2
  • LKAlert combines visual scenes, surrogate lane structure, and vehicle dynamics to produce timely alerts and natural-language explanations for driver oversight.The authors describe this pairing as facilitating safer and more informed driver intervention.Source: §I · p. 2
  • The proposed VLM framework guides prediction with LaneNet lane-segmentation masks and adapts the language decoder while keeping the vision encoder frozen.The authors report gains of +12.4% F1 and +68.2 ROUGE-L for the combined strategies.Source: §I · p. 2

II. RELATED WORK

Prior work documents field challenges for LKA and explores VLMs and parameter-efficient adaptation in driving applications. The paper situates its work among efforts toward interpretable multimodal driving systems and new datasets.

II. RELATED WORK

  • Prior studies report that LKA reduces crash frequency,, while poor lane-marking visibility can contribute to positioning errors or disengagement,.The section notes that co-occurring adverse factors and complex bends challenge current evaluation metrics.Source: §II · p. 2
  • VLM research spans regional visual understanding, lightweight inference, and adaptation for resource-constrained edge environments, [7],.Examples include RegionGPT, MiniVLM, and work on adapting large VLMs for edge use.Source: §II · p. 2
  • Driving studies apply LoRA to VLM alignment, closed-loop control, and scene classification, while pairing it with quantization for in-vehicle deployment, [14],, [17].The section also cites LoRA-based hazard detection and robustness benchmarks [16],.Source: §II · p. 2
  • Driving VLM work includes scene pre-annotation, planning, reasoning, and natural-language explanations, motivating new datasets and model designs –[25].The authors position Qwen2.5-VL-7B-Instruct as the backbone for their interpretable LKA alert system.Source: §II · pp. 2–3

A. Data Preparation

OpenLKA-Alert is built from failure and normal driving data, with video, CAN signals, lane masks, alert labels, and failure explanations. Its preparation uses LaneNet masks as interpretable lane features and retains temporal sequences around LKA problems.

A. Data Preparation

  • OpenLKA-Alert is built from OpenLKA-Failure and OpenLKA-Normal data, with failure windows extending 3.5 seconds forward and 2.5 seconds backward from disengagement.Video is sampled into frames every 0.5 seconds, with corresponding CAN information retrieved by time.Source: A. Data Preparation · p. 3, Figure 1 · p. 3
  • LaneNet produces binary edge-segmentation and instance-segmentation masks as interpretable intermediate lane features.The authors chose LaneNet partly to avoid overestimating the capabilities of vehicles from 2019–2022 with a state-of-the-art detector; Fig. 2 contrasts failure and normal examples.Source: A. Data Preparation · p. 3, Figure 2 · p. 3
  • The dataset retains continuous video and CAN sequences when the current image matches failure-associated features identified at the seventh frame.Other extracted failure images are removed during manual screening.Source: A. Data Preparation · p. 3
  • Failure frames are labeled for alerting and annotated with reasons, while normal frames are labeled No; an upcoming sharp curve can warrant an alert despite clear current lane markings.The example describes possible lane departure in three seconds.Source: A. Data Preparation · pp. 3–4, Figure 3 · p. 4
  • The annotated failure explanations commonly cite faded lane lines, occlusion, and low contrast between pavement and lane lines.These recurring challenges are summarized in the semantic word cloud.Source: A. Data Preparation · p. 4, Figure 4 · p. 4

B. Model Structure

LKAlert frames LKA failure prediction as conditional generation from multimodal inputs. Its output pairs a binary alert with a textual explanation.

B. Model Structure

  • LKAlert predicts imminent Lane Keeping Assist (LKA) failures and provides explanations, framing both prediction and explanation generation as a conditional generation task.Source: B. Model Structure · p. 4

1) Inputs and Outputs:

  • The input set S combines an RGB image, CAN dynamics, and LaneNet binary and instance lane-segmentation masks.The masks provide explicit geometric guidance.Source: 1) Inputs and Outputs · p. 4
  • The output O contains a Yes/No failure alert and a textual explanation.Source: 1) Inputs and Outputs · p. 4

2) Architecture Overview:

LKAlert encodes its multimodal inputs into a fused representation, then uses a LoRA-adapted decoder to generate outputs. Fine-tuning updates the decoder while using a pretrained model.

2) Architecture Overview:

C. LoRA Fine-Tuning with Mask Guidance

1) Learning Objective:

The learning objective trains LoRA parameters to generate the ground-truth alert and explanation from frozen-encoder representations. Mask-guided features embed lane geometry for the decoder to use in predicting failures and their reasons.

1) Learning Objective:

  • LoRA parameters are optimized to maximize the conditional likelihood of the ground-truth alert and explanation given fused input X.X is produced by frozen encoders from the image, masks, and CAN data.Source: 1) Learning Objective · p. 4

2) Mechanism of Mask-Guided Adaptation:

D. Validation Methods

Validation uses a held-out OpenLKA set with warning labels and reference explanations. The evaluation covers alert reliability, explanation fidelity, and runtime efficiency.

D. Validation Methods

  • The held-out validation set contains 1,000 samples, with 456 Yes and 544 No cases, each annotated with a warning label and reference explanation.Source: D. Validation Methods · pp. 4–5
  • Explanation quality is assessed with BLEU-4 and ROUGE-1/2/L, which measure token overlap and unigram, bigram, and longest-common-subsequence recall, respectively.Source: D. Validation Methods · p. 5
  • Runtime efficiency is measured in validation samples per second, alongside safety effectiveness and linguistic faithfulness.The passage identifies low false negatives as a safety-effectiveness measure and high BLEU/ROUGE as linguistic faithfulness.Source: D. Validation Methods · p. 5

A. Model Performance

The fine-tuned, mask-guided Qwen2.5-VL-7B achieves strong alert-prediction and explanation scores. Its confusion matrix also shows both detected failures and remaining missed detections.

A. Model Performance

  • 47.60% to 69.80% accuracy, 46.20% to 58.63% F1-score, and 3.57 to 71.72 ROUGE-L are reported for Qwen2.5-VL-7B after LoRA fine-tuning.The comparison is between the model before adaptation (origin) and after adaptation (final).Source: A. Model Performance · pp. 5–6
  • 69.80% accuracy, 58.63% F1-score, and 71.72 ROUGE-L are achieved by the fine-tuned, mask-guided Qwen2.5-VL-7B.The model is identified as the best-performing configuration for alert prediction and natural-language explanation generation.Source: A. Model Performance · p. 6
  • 21.4% of validation samples are true positives and 24.2% are false negatives for the final 7B model.The accompanying analysis notes that reducing false negatives remains important for safety-critical systems.Source: A. Model Performance · p. 6, Figure 6 · p. 6

B. Fine-Tuning Convergence and Efficiency

Fine-tuning improves explanation metrics early, while accuracy improves over more training steps and then plateaus. Inference speed remains largely constant after initial adaptation.

B. Fine-Tuning Convergence and Efficiency

  • BLEU-4 and ROUGE scores improve substantially within the initial 3,000 steps, particularly for the 3B and 7B models.Figure 7 plots validation metrics against training steps for different model sizes.Source: B. Fine-Tuning Convergence and… · p. 6, Figure 7 · p. 6
  • Most accuracy gains occur by 6,000–9,000 steps, after which metrics tend to plateau.Accuracy improves more steadily over training steps than the explanation metrics.Source: B. Fine-Tuning Convergence and… · p. 6
  • Inference speed remains largely constant during fine-tuning despite improvements in accuracy and ROUGE scores.The passage reports this stability after initial adaptation and describes it as maintaining the base model’s processing efficiency.Source: B. Fine-Tuning Convergence and… · p. 6

C. Ablation study: comparison of guided vs. unguided

The conclusion reports failure-prediction and explanation performance at approximately 2 Hz, while the ablation reports improved primary metrics with mask guidance and an SPS trade-off. Validation remains offline.

C. Ablation study: comparison of guided vs. unguided

  • Across model scales, guided variants improve primary performance metrics, while processing the additional masks affects SPS.Average performance rank improves from 3.67 to 2.50 for the 3B model and from 3.00 to 1.83 for the 7B model.Source: C. Ablation study: comparison of… · pp. 6–7

V. CONCLUSION

  • 69.8% accuracy, 58.6% F1-score, and 71.7 ROUGE-L are reported, with operation at approximately 2 Hz.The conclusion describes the system as suitable for real-time, resource-constrained deployment to enhance current ADAS safety and usability.Source: §V · p. 7
  • Current validation is offline; real-world deployment and user studies are needed to assess effects on driver awareness, trust, and response effectiveness.Source: §V · p. 7
Loading 2505.11535v1…