PaperlayerView exact revision on arXiv

Source-linked AI summary

LANTERN: A Closed-Loop Benchmark for VLM-Based Cooperative Driving with Temporally Grounded Warnings

Yongshuo Liu, Xu Gao, Morui Zhu, Yongqi Zhu, Qi Chen, Deyuan Qu, Song Fu, Qing Yang

arXiv:2609.06368v1cs.RO

TL;DR

Cooperative-driving evaluations often measure integrated systems, leaving the effect of a warning timed to a physical event difficult to isolate. LANTERN uses matched closed-loop executions to evaluate timed warnings, and the same fine-tuned model achieves a higher CUS with warnings enabled.

  • Problem

    Integrated cooperative-driving comparisons do not isolate the contribution of a specific warning delivered at a particular stage of a physical event.

    Source: §1 · p. 1

  • Method

    LANTERN evaluates matched closed-loop executions across six safety-critical scenario families, controlling warning timing while pairing hazard routes with no-hazard controls.

    Source: §1 · p. 1

  • Results

    CUS rises from 34.6 without warnings to 75.5 with warnings for the same fine-tuned checkpoint.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Across evaluated hazards, warnings provide the greatest benefit when relevant evidence is occluded or becomes visible only after reaction distance has narrowed.

    Source: §5 · p. 7

    Hazard detection and message transmission are outside the evaluation scope; warnings are benchmark inputs constructed from the ground-truth physical event.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

We present LANTERN, a closed-loop benchmark for temporally grounded cooperative warnings. LANTERN separates warning onset, hazard onset, warning termination, and post-hazard recovery, and evaluates each physical event under matched warning and no-warning executions so that the warning's contribution is measured in isolation rather than confounded with onboard vision. The benchmark spans six safety-critical scenario families and provides 3,272 sequences with 236,309 frames for training, together with 120 matched route pairs for closed-loop evaluation. Each hazard route is evaluated under the warning and no-warning conditions, while its no-hazard control penalizes unconditional braking. We further introduce the Cooperative Unified Score (CUS), a safety-gated metric that jointly rewards route progress, anticipation, clearance, and recovery. Fine-tuning a representative VLM driving model raises CUS from 34.6 without warnings to 75.5 with them, demonstrating both the value of cooperative warnings and the discriminative power of the paired protocol. All resources will be made publicly available.

1 Introduction

LANTERN addresses the difficulty of attributing cooperative-driving gains to a warning delivered at a particular stage of a physical event. It introduces matched closed-loop evaluations and reports a higher CUS for a fine-tuned model when warnings are enabled.

1 Introduction

  • Integrated cooperative-system comparisons can show that cooperation helps without isolating the contribution of a warning timed to a particular physical event.Warnings can describe hazards before relevant evidence appears in the ego vehicle’s onboard sensors.Source: §1 · p. 1
  • LANTERN controls warning onset, physical-event onset, and warning termination in matched closed-loop executions of the same model and physical event.Its six scenario families include hazards that are occluded, become visible late, or are observable outside the ego vehicle.Source: §1 · p. 1
  • The benchmark provides 3,272 training sequences with 236,309 frames and a closed-loop suite of 120 matched hazard/control route pairs.No-hazard controls penalize models that stop unconditionally when no physical hazard is present.Source: §1 · pp. 1–2
  • CUS rises from 34.6 without warnings to 75.5 with warnings for the same fine-tuned VLA driving model, a 40.9-point gain.The benchmark pairs warning and no-warning executions of the same physical event.Source: §1 · p. 2

2 Related Work

Prior work evaluates safety-critical driving, language-based control, and cooperative systems, often at the level of an integrated system or configuration. LANTERN instead treats a temporary, event-aligned warning as a controlled variable in closed-loop evaluation.

Closed-Loop and Safety-Critical Driving Benchmarks

  • Driving benchmarks span route completion, safety-critical traffic, simulation, and controlled distribution shifts, but generally vary routes, conditions, or shifts rather than timed language warnings.Examples include CARLA, SafeBench, Waymax, ScenarioNet, NAVSIM, AdvSim, and Fail2Drive.Source: §2 · p. 2

Language-Conditioned Closed-Loop Driving

  • Language-conditioned driving studies use language as navigation or notice instructions, reasoning targets, or persistent model inputs rather than temporary event-aligned warnings.LANTERN aligns a cooperative hazard warning with physical-event evolution to measure anticipation and post-warning recovery.Source: §2 · pp. 2–3

Cooperative Driving and Language Communication

  • Cooperative driving work covers shared perception, communication-efficient feature sharing, planning, and language communication, with comparisons typically at the system or configuration level.The cited studies include OPV2V, Where2comm, UniV2X, LangCoop, and MDrive.Source: §2 · p. 3

LANTERN’s Evaluation Distinction

  • LANTERN controls warning onset, physical-event onset, and warning termination, making anticipation, event safety, and post-warning recovery observable within each rollout.Table 1 compares benchmarks by evaluation setting, cooperative interface, information timing, and reported metrics.Source: §2 · p. 3, Table 1 · p. 3

3 The LANTERN Benchmark

LANTERN benchmarks closed-loop driving with temporally controlled cooperative warnings across six safety-critical scenario families. It pairs warning and no-warning hazard runs with no-hazard controls, and scores safety, progress, anticipation, and recovery.

3.1 Task Definition and Benchmark Overview

  • LANTERN tests closed-loop driving with optional externally supplied warnings across six hazard families; hazard detection and message transmission are outside benchmark scope.Source: §3.1 · pp. 3–4

3.2 Closed-Loop Temporal Protocol

  • Warnings precede event onset and end at direct 110° front-camera observation; a 2-s anti-deadlock countdown preserves the event if the ego stops before its geometric trigger.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 Cooperative Dataset Construction

  • Training data contain 3,272 sequences and 236,309 frames in aligned six-camera and 110° front-view formats; open-loop Route A/B sequences are independently filtered, not guaranteed pairs.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4 Closed-Loop Evaluation and Scoring

  • Across 120 matched Route A/B pairs, hazard routes run with and without warnings while no-hazard controls catch false braking; CUS safety-gates weighted progress, safety, anticipation, and recovery.Source: §3.4 · p. 5

4 Experiments

Experiments compare warning and no-warning executions on matched routes, then test transfer to nuScenes and sensitivity to protocol components. Fine-tuning makes the warning benefit clear, especially when warnings provide hazard evidence before it is visible onboard.

4.1 Experimental Setup

  • The primary evaluation uses 120 fixed Route A/B pairs, with 20 valid pairs per scenario family and driving model, and compares warning conditions with ego-only and privileged baselines.Coop-SimLingo fine-tunes SimLingo without architectural changes; the comparison also includes VAD and PDM-Lite.Source: §4.1 · pp. 5–6

4.2 Closed-Loop Results

  • CUS rises from 34.6 without warnings to 75.5 with them for the same fine-tuned checkpoint, a 40.9-point gain.Original SimLingo gains 10.9 points under the same warnings, from 32.2 to 43.1.Source: §4.2 · p. 6, Table 3 · p. 6
  • Warning gains are largest for Pedestrian Emergence (+80.9) and smaller for Right-Turn Yield (+3.7), consistent with greater benefit when hazards are not yet visible onboard.Warnings provide missing evidence before onboard observation, extending the interval for deceleration or yielding; cross traffic at Right-Turn Yield is often already visible.Source: §4.2 · p. 6
  • With warnings, TTC rises from 34.8 to 70.3 and clearance from 44.4 to 97.4, while Route A collisions fall from 41.7% to zero.Anticipation rises from 53.5 to 86.7 and recovery from 62.7 to 82.4; Route B progress remains 93.7 and sustained false braking is 1.7%. Remaining failures involve interaction quality, including negotiation, excessive caution, and delayed recovery.Source: §4.2 · p. 6, Table 4 · p. 7

4.3 Real-World Transfer

  • On 600 scene-disjoint nuScenes frames, 99.0% of warning-conditioned trajectories remain in the observed drivable area and 94.8% produce a clear response.The no-warning ADE/FDE is 2.74/5.75 m; the remaining warning cases mainly involve responses below a fixed threshold rather than invalid trajectories.Source: §4.3 · pp. 6–7, Table 5 · p. 6

4.4 Ablations and Robustness

  • Late warning onset reduces anticipation to zero and raises collisions to 14.9, while held-out paraphrases reduce CUS to 57.7.A warning extended 10 s past termination raises blocking to 29.7; without the anti-deadlock rule, event exposure falls to 71.0 and blocking reaches 35.7. Two-meter position noise leaves CUS at 75.5.Source: §4.4 · p. 7, Table 6 · p. 7

5 Conclusion

LANTERN provides a temporally controlled, paired benchmark for measuring cooperative-warning responses. Results show the greatest benefit when warnings add evidence unavailable from onboard sensing, while timing, interaction planning, and recovery remain important.

5 Conclusion

  • LANTERN evaluates cooperative warnings with a paired protocol that measures whether responses are timely, safe, and recoverable.The benchmark currently uses structured warnings derived from simulator state and focuses on individual hazards; future work will study learned warning sources, communication uncertainty, and interacting hazards.Source: §5 · p. 7

A Data Collection Details

The release builds routes from recorded simulator trajectories, then filters, materializes, and validates candidates as controlled pairs. Independent quality checks and reruns support a frozen 120-pair closed-loop suite.

Route recovery and geometric diversity

  • Routes are recovered from recorded simulator trajectories, with expansion paths reconstructed between control points using CARLA GlobalRoutePlanner at 1-m sampling resolution.The suite combines routes from the original SimLingo collection with routes scanned from Bench2Drive-Base; source-clip actors and scenario states are not copied.Source: A Data Collection Details · p. 10
  • Candidates require a normal approach, conflict point, at least 29 m of recovery route, and scenario-appropriate geometry and marker ordering.The six families have distinct route-length ranges, and turning scenarios require junctions with at least three distinct road arms.Source: A Data Collection Details · p. 10
  • Near-duplicate routes are rejected using bidirectional overlap of 2-m samples, and each source clip appears at most once per scenario family.Route A and Route B share waypoints, town, weather, navigation objective, route length, and background-traffic seed; Route B retains contextual actors while disabling the target hazard and warning.Source: A Data Collection Details · p. 10

Scenario materialization and controls

  • Accepted candidates are ranked across town, source scenario, weather, route geometry, and junction structure before actors and temporal markers are added.Materialization records route-relative warning, event, conflict, and recovery locations, actor parameters, and source provenance.Source: A Data Collection Details · p. 10

Expert rollout and supervision; Open-loop acceptance

  • PDM-Lite supplies expert waypoints at 4-Hz supervision, and synchronized observations and annotations must agree across frames.Open-loop acceptance requires at least 95% route completion, no recorded collision, and complete synchronized modalities; Route A also needs an ordered event trace and recovery segment.Source: A Data Collection Details · p. 10

Closed-loop static validation; Dynamic and visual acceptance

  • Closed-loop pairs pass static, dynamic, and visual checks, followed by independent reruns; the frozen v1.1 release contains 120 pairs and validation records.Route A/B files are admitted only as jointly validated pairs, and manual review checks the event sequence and matched Route B context.Source: A Data Collection Details · pp. 10–11

B Town, Weather, and Time Distribution

The open-loop release spans ten CARLA towns, while the visually distinct closed-loop suite spans eleven. A separate environmental extension adds weather and time-of-day diversity; scenario actors follow scripted behavior while Route B retains context without the hazard.

B Town, Weather, and Time Distribution

  • The 3,272-sequence, 236,309-frame open-loop release covers ten towns and is concentrated in Town12, while 120 closed-loop pairs span eleven towns.Town05 has 35 pairs, Town04 has 21, Town12 has 19, Town13 has 14, and the remaining 31 pairs are distributed across seven towns.Source: B Town, Weather, and Time Distribution · p. 11, Figure 6 · p. 11
  • The environmental extension adds 859 sequences across cloudy-day, wet-day, sunset, clear-night, and rainy-night conditions.The profiles comprise 387 daytime, 172 dusk, and 300 nighttime sequences; Route A/B final counts differ because quality filtering is applied independently.Source: B Town, Weather, and Time Distribution · p. 11, Figure 6 · p. 11

C NPC and Scenario-Actor Behavior

  • Scenario actors use deterministic scripted motion, including pedestrian crossings at 3 m/s and wrong-way vehicles at 7–9 m/s.Actors remain until their scripted interaction completes or they clear the conflict region; a cut-in vehicle returns to Traffic Manager after merging.Source: C NPC and Scenario-Actor Behavior · p. 12
  • Route B retains visual context but disables the hazard, while interaction safeguards filter selected vehicles that could disrupt the controlled event.Safeguards preserve unrelated adjacent, opposing, and cross-lane traffic while clearing specified portions of the ego route corridor.Source: C NPC and Scenario-Actor Behavior · p. 12

D Detailed Scenario Specifications

LANTERN specifies a shared event lifecycle across six scenario families and scores matched hazard and control routes with a safety-gated measure. The Cooperative Unified Score combines safety, progress, anticipation, and recovery while penalizing unneeded stopping.

D Detailed Scenario Specifications

  • Across six families, evaluation tracks warning onset, event onset, visibility-based warning termination, hazard clearance, and downstream recovery.Warning termination follows event onset when the hazard enters the 110° front-camera view and has an unobstructed ray from the virtual camera.Source: D Detailed Scenario Specifications · p. 12, Table 7 · p. 13
  • For five families, a 2-s anti-deadlock interval triggers actor motion without overriding ego control; Right-Turn Yield needs no forced trigger.Warning and event points use 1.5-m tolerances, except Vehicle Cut-In’s 0.5-m event tolerance.Source: D Detailed Scenario Specifications · pp. 12–13

E Complete Cooperative Unified Score

  • The continuous score combines route progress, time-to-collision and clearance, anticipation, and post-clearance recovery.Clearance is normalized to 2 m for pedestrians and static obstacles and 3 m for vehicle interactions; recovery requires renewed speed and progress after hazard clearance.Source: E Complete Cooperative Unified Score · pp. 13–14
  • A Route B false-brake gate fails only for an unjustified stop at or below 0.1 m/s lasting at least 1 s during the matched approach interval.Other Route B braking does not fail this gate, and unavailable hazard fields are not imputed as successful responses.Source: E Complete Cooperative Unified Score · p. 14
  • CUS scores matched Route A/B executions, weighting Route A at 85% and Route B progress at 15%, with failed hard gates zeroing the pair.Route A contains the controlled hazard; Route B preserves the route context and objective but disables the target behavior and warning.Source: E Complete Cooperative Unified Score · pp. 13–14
  • In the worked example, the six weighted contributions sum to 0.94386; any failed gate instead makes the pair score zero.The example uses the final Coop-SimLingo warning execution on closed_occluded_pedestrian_0001.Source: E Complete Cooperative Unified Score · p. 14, Table 8 · p. 14

F Protocol Extension to Unseen Hazards

Controlled tests apply the matched protocol to hazards outside the six training families. Accurate warnings prompt earlier responses in the unseen cyclist, emergency-crossing, and reversing-vehicle cases, though all cyclist executions ultimately complete without collision or blocking.

F Protocol Extension to Unseen Hazards

  • The unseen cyclist test fixes route geometry, weather, traffic seed, checkpoint, and event logic across no-warning, free-form warning, and held-out paraphrase conditions.The 129.1-m Town15 route is absent from both the six training families and the frozen 120-pair test manifest.Source: F Protocol Extension to Unseen Hazards · p. 14
  • Both unseen cyclist warning wordings prompt sustained responses before event onset, whereas the no-warning response begins only after the event.All three executions complete without collision or blocking, making anticipatory timing—not eventual route success—the reported distinction.Source: F Protocol Extension to Unseen Hazards · p. 14, Table 9 · p. 14
  • For the emergency crossing, exact warning shifts sustained response from 0.55 s after event onset to 2.00 s before and raises route completion from 55.1 to 100 while eliminating collision.The reversing-vehicle executions all complete, but only the exact warning causes a sustained pre-event response.Source: F Protocol Extension to Unseen Hazards · p. 15, Table 10 · p. 15

G Additional Ablation Analysis

Additional tests probe tolerance to warning-location noise and wording ambiguity. Moderate localization noise leaves rounded CUS unchanged despite offsetting component changes, while removing actionable event or location content sharply degrades paraphrase performance.

G Additional Ablation Analysis

  • 75.5 CUS is reported both for the full protocol and with 2-m warning-position noise, despite more collisions and less blocking under noise.Route A progress changes from 95.4 to 98.4, collisions from 0.0 to 3.4, blocking from 5.8 to 3.3, and recovery from 82.4 to 82.1.Source: G Additional Ablation Analysis · p. 15
  • Low-ambiguity wording preserves performance without collisions, but removing event detail and location reduces matched CUS by 31.8 points and raises collisions to 50.0%.Medium ambiguity reduces matched CUS by 18.1 points; comparisons are against each disjoint subset’s matched canonical baseline.Source: G Additional Ablation Analysis · pp. 15, 17, Table 11 · p. 16
  • The position-free Pedestrian Emergence example collides after an insufficient response, whereas the position-free Obstacle Reveal route completes successfully.The figure illustrates differing outcomes for high-ambiguity warnings, not a single uniform response.Source: Figure 9 · p. 17

H Diagnostics with Non-Actionable Language Inputs

Diagnostics test whether the model can avoid reacting indiscriminately to warnings. The tested model distinguishes nearby from distant warnings and usually avoids sustained stopping for non-actionable warnings, while broader visual-language consistency remains unresolved.

H Diagnostics with Non-Actionable Language Inputs

  • In held-out control frames, the nearby warning produces a stopping trajectory, while the same warning at 100 m preserves ordinary progress.This distance diagnostic uses a separate checkpoint fine-tuned only on Pedestrian Emergence.Source: H Diagnostics with Non-Actionable… · p. 17, Figure 10 · p. 18, Table 12 · p. 18
  • Distance selectivity is demonstrated only within the trained warning family; broader contradiction reasoning remains open, including for unseen warning types and incompatible road contexts.The tested negative set does not cover qualitatively different inconsistencies such as a hazard behind the ego vehicle or an impossible road configuration.Source: H Diagnostics with Non-Actionable… · p. 18
  • Four of five non-actionable warnings meet the strict criterion; all five routes finish without collision, blocking, or sustained stopping, with mean additional speed reduction of 1.66 m/s.The parked-delivery-van warning causes a transient 2.8 m/s reduction.Source: H Diagnostics with Non-Actionable… · p. 19, Table 13 · p. 19

I Complete Results and Reproducibility

The release supports auditing and reproducing reported aggregates through a route-level results index and a provenance-rich manifest.

Complete Results and Reproducibility

  • The route-level CSV indexes all 720 final model–route pairs with Route A/B metrics, gate values, artifact provenance, and optional video paths.Its companion schema specifies fields, units, and missing-value rules.Source: I Complete Results and Reproducibility · p. 19
  • Continuous hazard terms remain empty after a hard-gate failure before valid telemetry, and audit files distinguish infrastructure failures from accepted replacements.Source: I Complete Results and Reproducibility · p. 19
  • The reproducibility manifest records software and checkpoint details, portable commands, expected outputs, aggregate-metric provenance, and the frozen route-manifest hash.Together, the files reproduce reported aggregates and preserve provenance for every included execution.Source: I Complete Results and Reproducibility · p. 19
Loading 2609.06368v1…