PaperlayerView exact revision on arXiv

Source-linked AI summary

Observe Before You Alert: Adaptive Driver Alerting with Vision-Language Models

Yuhang Wang, Lingyao Li, Hao Zhou

arXiv:2609.08130v1cs.CVcs.RO

TL;DR

Dashcam alerting must decide when evidence is sufficient to warn, while existing models often reduce risk to a binary prediction or scalar score. VLALERT uses an observation-aware SILENT, OBSERVE, ALERT policy with VLM-derived safety evidence. On VLALERT-Bench validation, it achieves the highest overall operating-point performance and higher DAUS than Open-BADAS.

  • Problem

    Existing evaluations provide limited evidence about nuisance alerts and normal driving, while binary or scalar-risk models lack an explicit way to defer warnings and gather evidence in ambiguous scenes.

    Source: §1 · p. 2

  • Method

    VLALERT combines a SILENT, OBSERVE, ALERT policy, where OBSERVE gathers more evidence, with VLM-generated safety evidence for danger estimation and policy prediction.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    VLALERT ranks first in overall operating-point performance on VLALERT-Bench validation and improves DAUS from 0.4752 for Open-BADAS to 0.4878.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    The results support treating driver alerting as a sequential policy decision with explicit evidence-gathering deferment rather than only binary or thresholded risk prediction.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Driver alerting from dashcam video requires sequential decision-making under partial observability: a system must decide not only whether a scene is risky, but also when the evidence is sufficient to warn. Most existing accident anticipation models output a binary risk score, leaving ambiguous scenes to be handled by thresholding. We propose VLAlert, a vision-language alerting framework that casts warning generation as a tri-action policy over SILENT, OBSERVE, and ALERT. The OBSERVE action acts as an internal evidence-gathering decision that delays uncertain warnings and changes the next observation window, creating a lightweight perception-action loop for adaptive alerting. VLAlert uses Qwen3-VL-4B as a safety-evidence generator and pools hidden states from structured belief spans to form compact representations for danger estimation and policy prediction. We evaluate VLAlert on VLAlert-Bench, a unified per-tick benchmark from four real-world dashcam alert datasets, and further test transfer to held-out naturalistic ADAS takeover clips. On VLAlert-Bench validation, VLAlert achieves the highest deployment-oriented utility among tested baselines, with DAUS 0.4878 compared with 0.4752 for Open-BADAS, and improves AUROC, AP_tick, F1_t, and balanced accuracy from 0.610, 0.176, 0.276, and 0.581 to 0.689, 0.195, 0.297, and 0.648, respectively. On 221 held-out ADAS-TO-Critic clips, VLAlert improves R@5s from 74.2% to 88.7% and F1 from 0.585 to 0.686. These results indicate that adaptive observation and safety-focused VLM representations provide measurable gains for driver-facing alert decisions.

1 Introduction

Driver alerting must balance timely, reliable warnings against unnecessary interruptions, but existing evaluations and binary or scalar-risk models offer limited evidence about nuisance alerts and no explicit way to defer uncertain warnings. VLALERT introduces a unified benchmark and an observation-aware tri-action policy with safety-evidence representations.

1 Introduction

  • Dashcam alerts must arrive early enough to support action, remain reliable enough to be trusted, and avoid unnecessary interruptions.Warnings can support braking and steering in human driving and takeover decisions when assisted driving is active.Source: §1 · p. 1
  • Most models reduce risk to a binary prediction or scalar score, leaving no explicit mechanism to defer a warning and gather evidence when a scene is ambiguous.Existing evaluations also focus largely on accident-centric datasets and ranking metrics, with limited evidence about nuisance alerts, normal driving, or human-perceived risk.Source: §1 · p. 2
  • VLALERT uses a SILENT, OBSERVE, ALERT policy in which OBSERVE defers uncertain warnings while gathering more evidence.The framework uses a VLM as a safety-evidence generator and hidden states from generated evidence spans as compact features for danger estimation and policy prediction.Source: §1 · p. 2
  • The paper builds VLALERT-Bench, a unified per-tick benchmark that combines driving-alert datasets and introduces DAUS to measure alert coverage, false-alert burden, and lead time.The benchmark also includes a real-world ADAS takeover collection to test alignment with human risk perception and reaction.Source: §1 · pp. 2–3

2 Related Work

Related work spans accident and anomaly prediction, richer perception and fusion methods, and naturalistic risk learning. These approaches inform alerting, but many remain focused on crash prediction or use hand-designed safety thresholds.

2 Related Work

  • Dashcam datasets DAD and DADA-2000 focus on crash anticipation, while DoTA adds richer traffic-anomaly categories.MM-AU provides aligned language annotations, object boxes, accident reasons, and prevention descriptions; NEXAR-Collision [5] evaluates collision and near-collision prediction across pre-event intervals.Source: §2 · p. 3
  • Accident anticipation methods use depth-enhanced scene modeling, temporal attention, LLM-assisted localization [13], and driver-attention-guided transformers.BADAS and Open-BADAS/BADAS-2.0 use video foundation models and large ego-centric dashcam corpora for end-to-end collision prediction [19] [33] [18].Source: §2 · p. 3
  • Classical surrogate safety measures such as TTC and THW are interpretable but require hand-designed thresholds that may not generalize across traffic contexts.Trajectory-based conflict models and naturalistic driving data offer alternatives, including learning unsafe deviations without crash labels.Source: §2 · p. 3

3 Method

VLALERT treats streaming dashcam alerting as a partially observed sequential decision task, with actions that shape subsequent observations. Its method combines a per-tick benchmark, VLM-derived safety beliefs, downstream danger and policy heads, and closed-loop training.

3 Method

  • VLALERT receives an 8-frame observation window at each decision tick and predicts SILENT, OBSERVE, or ALERT.The method comprises a belief-state formulation, unified alert benchmark, VLM-based alert architecture, and closed-loop training procedure.Source: §3 · p. 3

3.1 Problem Formulation

  • Driver alerting is partially observed: agent intent, occluded objects, road affordances, and driver response remain latent, so VLALERT learns an amortized belief representation without explicitly modeling transition or observation probabilities.OBSERVE defers commitment and requests more evidence; the action-conditioned sampler uses a wide sparse window after SILENT, a medium dual-resolution window after OBSERVE, and a dense short-range window after ALERT.Source: §3.1 · pp. 3–4

3.2 Benchmark Construction

  • VLALERT-Bench pools Nexar Collision, DoTA, DAD, and DADA-2000 for in-domain training and validation, while holding out ADAS-TO-Critic [37] and ACCIDENT for testing.Sources are converted to a common 1 Hz format with 8-frame windows and SILENT, OBSERVE, or ALERT labels; ticks at or after the event are discarded, and negative clips remain SILENT throughout.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.3 Model Architecture

  • The VLM generates concise safety-evidence belief spans whose hidden states form belief features for DangerHead and PolicyHead.The architecture uses Qwen3-VL-4B, extracts span-pooled features and a decision register, estimates per-frame and clip-level danger, then predicts alert actions; the decoder gates direct SILENT-to-ALERT transitions.Source: §3.3 · pp. 5–6

3.4 Training and Inference

  • Training proceeds through belief-format supervised fine-tuning, cached-feature training of DangerHead and PolicyHead, and closed-loop refinement with action-conditioned sampling.At inference, the system samples a window based on the previous action, extracts beliefs, estimates danger, predicts and decodes an action, emits sparse alerts, and feeds the action into the next tick.Source: §3.4 · p. 6

4 Experiments and Results

VLALERT leads the VLALERT-Bench validation comparison on deployment-oriented utility and several operating-point metrics. Its ablations support both BELIEF-span pooling and a distinct OBSERVE action, while the evaluation also includes held-out datasets.

4.1 Evaluation Protocol

  • Evaluation uses 11,149 ticks from 1,219 videos and reports ranking, coverage, operating-point, and lead-time metrics, including balanced accuracy for the SILENT-heavy benchmark.Each method uses one operating threshold for threshold-dependent metrics.Source: §4.1 · p. 6

4.2 Driver-Aware Utility Score

  • DAUS geometrically combines mAP@TTA, video recall, tick precision, and normalized lead-time utility, penalizing failure in any component.This makes the score sensitive to both missed hazards and disruptive false alerts.Source: §4.2 · p. 6

4.3 Comparison with Baselines

  • VLALERT raises AUROC from 0.610 to 0.689, APtick from 0.176 to 0.195, and DAUS from 0.4752 to 0.4878 versus Open-BADAS on validation.It also ranks first in Recallv, F1t, Precisiont, and balanced accuracy.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 Ablation Studies

  • BELIEF-span pooling yields AP 0.459 and AUROC 0.894, while a same-length random response span yields AP 0.204.The result supports the structured span's role in localizing safety evidence; Figure 4 reports that BELIEF-span attention focuses on driving-risk cues across the scene.Source: §4.4 · p. 8, Table 3 · p. 8, Figure 4 · p. 8
  • The three-action formulation obtains the best APt, AUROCt, and F1t across three seeds, with APt improving by 5.4% and 3.1% over the binary classifier.The ablation keeps the backbone and DangerHead features fixed while changing PolicyHead label spaces.Source: §4.4 · p. 8, Table 3 · p. 8
  • VLALERT is also evaluated on two unseen datasets: ADAS-TO takeover events and a synthetic CARLA dataset from a different roadside view.Source: §4.4 · p. 8

5 Conclusion

VLALERT reframes dashcam alerting as a sequential SILENT–OBSERVE–ALERT policy and combines it with a unified benchmark and focused VLM belief extraction. It achieves the best overall operating-point utility on VLAlert-Bench, with ablations supporting BELIEF-span pooling and a separate OBSERVE action.

5 Conclusion

  • VLALERT combines a unified per-tick benchmark and DAUS with an adaptive tri-action policy and a VLM interface that emits focused safety spans for downstream heads.The conclusion reports the best overall operating-point utility across baselines and stronger alert separability from BELIEF-span pooling than equal-length random spans.Source: §5 · p. 9

A.1 ADAS-TO-Critic: Generalization to Human Takeovers

On held-out ADAS-TO-Critic takeover clips, VLALERT improves pre-takeover alert coverage over Open-BADAS. A representative prediction stays in OBSERVE as risk develops, then switches to ALERT shortly before takeover.

A.1 ADAS-TO-Critic: Generalization to Human Takeovers

  • VLALERT ranks best in F1, R@10s, R@5s, Lead@5s, and DAUS, with R@5s of 88.7% versus 74.2% for Open-BADAS.Table 4 evaluates 221 expert-reviewed takeover clips centered at t = 10 s.Source: Table 4 · p. 12, §A.1 · p. 12
  • In the representative case, VLALERT remains in OBSERVE as risk develops, switches to ALERT one second before takeover, and peaks at takeover time.This illustrates the intended defer-then-warn behavior as intervention becomes imminent.Source: §A.1 · pp. 12–13, Figure 5 · p. 13

A.2 Out-of-Distribution Generalization on Roadside Accidents

VLALERT transfers to a fixed roadside-camera setting with changed viewpoint and scene statistics, alerting on most collision clips before impact. The reported example shows it escalating from OBSERVE to ALERT while Gemini remains silent.

A.2 Out-of-Distribution Generalization on Roadside Accidents

  • The 2,211-clip benchmark uses fixed roadside cameras, shifting from ego-view video to static third-person views where accidents often unfold across intersections.Because all clips are positive, the evaluation reports per-clip alert rate and mean time-to-accident rather than AP or AUROC.Source: §A.2 · p. 13
  • Figure 6 illustrates a collision case in which VLALERT escalates from OBSERVE to ALERT before impact and sustains the alert, while Gemini stays SILENT.The example is a roadside CARLA unprotected left-turn collision.Source: Figure 6 · p. 13
  • 88.9% of clips receive an alert over the full pre-accident window, with a 7.31 s mean lead time; 74.8% receive an alert in the last 2 s.Zero-shot Gemini-2.5-Flash-Lite rarely alerts under the same three-action protocol.Source: §A.2 · p. 13, Table 5 · p. 14

B Benchmark Construction Details

VLALERT-Bench converts source-specific annotations into a shared per-tick stream with SILENT, OBSERVE, and ALERT actions. Its intermediate OBSERVE labels depend on available risk-cue annotations, which vary across datasets.

B Benchmark Construction Details

  • Each clip becomes a 1 Hz stream of eight-frame windows labeled at the anchor frame; post-alert-window ticks are discarded for collision-style corpora, and negative clips remain SILENT.Event times come from each source’s native annotations; DAD’s clip-level labels are propagated to usable ticks, while ADAS-TO-Critic and ACCIDENT are held out for evaluation.Source: B Benchmark Construction Details · p. 14
  • Figure 7 shows train and validation tick counts by source and action after label filtering and post-event filtering.DoTA supplies nearly all OBSERVE ticks: 3,948 of 4,152 training OBSERVE ticks, compared with 106 from DADA-2000, 98 from Nexar, and none from DAD.Source: Figure 7 · p. 14, B Benchmark Construction Details · pp. 14–15
  • OBSERVE marks the interval from the first legible visual precursor to the event time, replacing the pre-event SILENT label, with source-specific onset rules.DoTA uses its first anomaly frame; DADA-2000 onset annotations were manually marked with κ = 0.79 on a 100-clip audit; Nexar estimates onset from pre-event optical flow; DAD receives no onset labels.Source: B Benchmark Construction Details · pp. 14–15
  • Manual annotation quality control used two experienced authors, third-author review of a random 10% sample per batch, and joint adjudication of disagreements.Residual disagreement rates were 4.1% for DoTA cleanup and 6.3% for DADA-2000 risk-cue onset marking.Source: B Benchmark Construction Details · p. 15

C VLALERT Training Details

VLALERT training constructs supervised belief/action ticks, fine-tunes a VLM to produce structured safety evidence, then trains danger and policy heads. Closed-loop refinement and a calibrated decoder support action-conditioned inference.

C.1 Training-Data Construction

  • Training-data construction pairs each 8-frame tick with action supervision and per-frame belief labels, using native text where available and generated, filtered safety captions otherwise.For OBSERVE samples, onset annotations bias belief text toward the developing hazard; the final in-domain split contains 80,221 training ticks and 11,149 validation ticks.Source: §C.1 · pp. 15–16

C.2 Stage 1: Belief-Format Supervised Fine-Tuning

  • Stage 1 fine-tunes belief-format responses using next-token cross-entropy, updating only LoRA weights and new-token embeddings while weighting delimiter and action tokens more heavily than belief text.The supervision reuses native annotations when available and otherwise uses filtered GPT-5.4-generated captions or label-derived fallbacks.Source: §C.2 · pp. 16–17
  • Table 6 summarizes Stage 1 belief supervision: each tick supplies eight per-frame spans, reusing native textual annotations when available and otherwise using generated, filtered safety captions.The stage uses 80,221 training ticks and yields 641,768 belief spans.Source: §C.2 · p. 16, Table 6 · p. 17

C.3 Stage 2: Belief Caching and Downstream Heads

  • Stage 2 freezes the fine-tuned VLM, caches pooled BELIEF-span features, and trains DangerHead on frame-level vectors and PolicyHead on action, risk, and decision-register inputs.DangerHead estimates per-frame and clip-level risk; PolicyHead predicts SILENT, OBSERVE, or ALERT with a transition regularizer against low-confidence SILENT-to-ALERT jumps.Source: §C.3 · p. 17

C.4 Stage 3: Closed-Loop Refinement

  • Stage 3 freezes the VLM and DangerHead while refining PolicyHead on action-conditioned observations through teacher forcing, scheduled sampling, and full rollout.The sampler selects a wide sparse window after SILENT, a medium dual-resolution window after OBSERVE, and a dense short-range window after ALERT.Source: §C.4 · p. 18

C.5 Finite-State Decoder and Event Gating

  • The rule-based decoder blocks low-confidence direct SILENT-to-ALERT transitions, and the event gate requires two consecutive ALERT ticks before emitting an alert and then enforces a 3-second refractory window.The calibrated settings are fixed across evaluations without dataset-specific retuning.Source: §C.5 · p. 18

C.6 Inference Protocol

  • At inference, the previous action selects an 8-frame window, pooled belief states feed danger and policy predictions, and event gating produces sparse driver-facing alerts.The system runs at 1 Hz with approximately 180 ms latency per tick on an RTX 5090.Source: §C.6 · p. 18

D Baseline Architectures and Training Details

The baseline suite uses shared VLALERT-Bench train and validation splits and validation-based operating-point calibration. It compares recurrent and video encoders, Open-BADAS, and a zero-shot multimodal model.

D Baseline Architectures and Training Details

  • All learned baselines use the same 80,221-tick training split and 11,149-tick validation split, with one operating threshold tuned on validation; zero-shot baselines are untrained.Training uses bf16 on a single RTX 5090.Source: D Baseline Architectures and Training… · p. 19

D.1 ResNet50-LSTM

  • ResNet50-LSTM encodes each frame independently, projects its features to 512 dimensions, and classifies from the final state of a single-layer LSTM.The ResNet-50 is frozen for a two-epoch warm-up before being unfrozen; training runs for 12 epochs with the best checkpoint selected by validation AP.Source: §D.1 · p. 19

D.2 R3D-18

  • R3D-18 stacks the eight frames into a short clip and feeds its pooled Kinetics-pretrained video feature to a binary classifier.The full backbone is trained from the first epoch for 20 epochs, with checkpoint selection by validation AP.Source: §D.2 · p. 19

D.3 MViT-V2-S

  • MViT-V2-S applies a Kinetics-pretrained video backbone to each eight-frame tick and classifies its pooled CLS feature.The full backbone is fine-tuned from epoch zero for 25 epochs, selecting the best checkpoint by validation AP.Source: §D.3 · p. 19

D.4 Open-BADAS

  • Open-BADAS uses a frozen V-JEPA2 video encoder resampled to 16 frames, followed by attention pooling and a two-layer binary collision head.Only the collision head is optimized, and its best checkpoint is selected by validation AP.Source: §D.4 · p. 19

D.5 Gemini-2.5-Flash-Lite

  • Gemini-2.5-Flash-Lite is evaluated zero-shot on eight-frame multimodal requests and returns probabilities for SILENT, OBSERVE, and ALERT.For roadside CCTV, the prompt specifies a fixed roadside camera; other prompt content is unchanged. Other learned baselines’ scalar danger scores are converted to SILENT or ALERT using a validation-tuned threshold frozen for later evaluations.Source: §D.5 · p. 20

E ADAS-TO-Critic Dataset Details

ADAS-TO-Critic is a held-out set of reviewed safety-motivated takeovers, evaluated as pre-takeover alert decisions rather than collision prediction. Its clip-level protocol uses pre-takeover ticks and fixed validation settings.

E ADAS-TO-Critic Dataset Details

  • 221 of 285 candidate clips remain after excluding takeovers not primarily motivated by safety; reviewer agreement was Cohen’s κ = 0.84.Disagreements were adjudicated by a third author, and retained clips preserve the takeover anchor at t⋆= 10 s.Source: E ADAS-TO-Critic Dataset Details · p. 20
  • The takeover at t = 10 s serves as an external signal of driver-perceived need for immediate intervention, and the test split shares no clips with in-domain training corpora.This makes the evaluation target pre-intervention alerting rather than collision occurrence itself.Source: E ADAS-TO-Critic Dataset Details · pp. 20–21
  • Each 20 s clip yields 1 Hz ticks through t = 12 s, using the eight preceding frames; later ticks and OBSERVE sub-labels are excluded from scoring.A clip counts as positive if the method fires at least one ALERT before takeover.Source: E ADAS-TO-Critic Dataset Details · p. 21
  • The evaluation records the first pre-takeover ALERT and reports R@10s, R@5s, Lead@10s, and Lead@5s.R@10s and R@5s measure alert coverage in their respective time windows; the Lead metrics average time to takeover among clips that fire in those windows.Source: E ADAS-TO-Critic Dataset Details · p. 21
  • F1 uses the binary fire/no-fire decision over the full pre-takeover window, with no threshold tuned on ADAS-TO-Critic.Learned baselines use validation-selected thresholds, while VLAlert retains its fixed decoder and event-gating parameters.Source: E ADAS-TO-Critic Dataset Details · p. 21

F CARLA Accident OOD Evaluation Details

The CARLA split tests out-of-distribution alerting on roadside-view collision clips, using positive-set measures because it contains no negative clips. Results vary by accident type, with single-vehicle loss-of-control and late-entry T-bone cases identified as key limitations.

F CARLA Accident OOD Evaluation Details

  • The 2,211-clip CARLA split contains only collisions and uses static roadside video, changing viewpoint, field of view, camera motion, and accident geometry from in-domain dashcam data.It is reserved for out-of-distribution evaluation and includes five accident categories.Source: F CARLA Accident OOD Evaluation Details · p. 22
  • Because the split has no negative clips, standard binary AP, AUROC, and multiplicative DAUS are undefined; evaluation instead reports positive-set alert rate, mTTA, and DAUS+.These measures capture alert coverage and, for mTTA, the mean time-to-accident among clips that fire.Source: F CARLA Accident OOD Evaluation Details · p. 22
  • DAUS+ combines full-window alert coverage with capped lead-time utility and is not directly comparable to validation DAUS.It is presented as a compact OOD summary of per-window alert rate and mTTA.Source: F CARLA Accident OOD Evaluation Details · p. 22
  • Evaluation uses the full pre-accident interval, the last 5 s, and the last 2 s, with decoder and event-gating parameters fixed on validation and no threshold tuned on CARLA.Decisions use argmax over the three-action output.Source: F CARLA Accident OOD Evaluation Details · p. 22
  • The zero-shot Gemini comparison covers 937 clips, whereas VLAlert is evaluated on all 2,211 clips.Gemini uses a CCTV-aware prompt and eight consecutive frames per tick.Source: F CARLA Accident OOD Evaluation Details · p. 22
  • Single-vehicle failures lack a second interacting actor, while late-entry T-bone cases may provide less than two seconds of visible pre-impact evidence.The authors identify longer context, ego-motion or vehicle-kinematic belief fields, and limited CCTV-style adaptation data as future directions.Source: F CARLA Accident OOD Evaluation Details · p. 22

VLALERT (Ours)

Loading 2609.08130v1…