PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Retrieval Relevance: Scene-Grounded Risk Entailment for Vision-Language Driving

Jiaxin Liu, Ruilin Yu, Liang Peng, Jingkai Wang, Chengxiang Zhao, Zhenxin Zhu, Bing Wang, Guang Chen, Hangjun Ye, Hong Wang, Jun Li

arXiv:2609.34145v1cs.RO

TL;DR

Retrieved risk knowledge may be relevant without applying to the current driving scene, leaving risk-rule support implicit in VLM reasoning. The paper uses DRKG and SWRL to derive scene-grounded evidence before VLM decision-making; on nuReasoning, this method improves planning scores over the strongest semantic-retrieval condition.

  • Problem

    A semantically relevant risk rule does not establish a scene-specific risk, which requires its objects, entities, times, and relations to be grounded and jointly supported.

    Source: §I · p. 1

  • Method

    The framework instantiates DRKG facts from structured perception and applies SWRL reasoning to provide recognized events, directed risk relations, and activated rule descriptions to the VLM and diffusion planner.

    Source: §I · pp. 1–2

  • Results

    On nuReasoning, NC rose from 87.53 to 90.29 and NPS from 62.88 to 64.18 versus the strongest semantic-retrieval condition.

    Source: §V · p. 7

  • Takeaways & Limitations

    Scene-bound conclusions and activated rule descriptions provide complementary information for planning, with their combination yielding the highest NPS across four evidence ablations.

    Source: E. Ablation of Scene-Grounded Risk… · p. 7

    The framework depends on structured perception and covers only risk forms represented by DRKG rules, so noisy or missed observations and incomplete rule coverage constrain its evidence.

    Source: §V · p. 7

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) gives vision--language driving systems access to external safety knowledge, yet a retrieved risk rule may be relevant without applying to the current scene. A vision--language model (VLM) receiving such knowledge must ground objects, bind entities across time, and verify relations before deciding how to act, leaving the support for risk conclusions implicit. We address this relevance--applicability gap with a Driving-Risk Knowledge Graph (DRKG) and Semantic Web Rule Language (SWRL) reasoning stage before VLM decision-making. Structured perception instantiates scene facts, from which SWRL rules derive events and directed risk relations when their antecedents are jointly satisfied. Recognized events, bound risk relations, and semantic descriptions of activated rules form compact evidence that conditions the VLM and diffusion planner. In matched comparisons on nuReasoning, our method improved the nuReasoning planning score (NPS) by 1.30 points and the non-at-fault collision score (NC) by 2.76 points over the relevance retrieval-based baseline. These gains indicate that scene-applicable risk evidence improves safety-weighted planning relative to semantically retrieved risk knowledge.

I. INTRODUCTION

Semantic retrieval can identify relevant risk knowledge without establishing that its rule applies to the current scene. The paper makes applicability explicit through DRKG and SWRL reasoning, then evaluates scene-grounded risk evidence in controlled planning comparisons.

I. INTRODUCTION

  • The relevance–applicability gap arises because a retrieved rule's scene-specific conclusion requires grounded objects, bound entities and times, and jointly supported relations.The VLM otherwise performs this reasoning implicitly, without exposing support for its asserted risks as an independently checkable result.Source: §I · p. 1
  • The framework instantiates structured perception as DRKG facts and uses SWRL to derive scene events and directed risk relations paired with their activated rules.The compact evidence conditions VLM decision-making and diffusion-based trajectory planning.Source: §I · pp. 1–2
  • Controlled nuReasoning comparisons find that broader retrieval recovers more activated rules but also introduces inapplicable rules, while added matches do not reliably improve planning.Scene-grounded evidence yields better safety-weighted planning, particularly in non-at-fault collision avoidance.Source: §I · pp. 1–2
  • Scene-bound results combined with activated rule descriptions yield a higher planning score than either evidence representation alone.The paper frames this as demonstrating the downstream value of scene applicability over semantic relevance.Source: §I · p. 2

II. RELATED WORK

Prior driving systems retrieve external knowledge through varied mechanisms, while structured and formal methods represent scene relations and evaluate conditions outside a VLM. The paper formulates planning as producing an ego trajectory conditioned on scene-applicability evidence as well as visual and structured scene inputs.

A. Knowledge Access for Vision–Language Driving

  • RAG-Driver, RAD, SafeDriveRAG, KnowVal, and DriveReg access driving knowledge through demonstrations, experience, subgraphs, graph entities, or regulation text, but do not expose formally entailed risk relations between bound scene entities.DriveReg also uses an LLM reasoning agent to assess rule applicability and action compliance.Source: A. Knowledge Access for… · p. 2

B. Structured Knowledge and Formal Risk Reasoning

  • Graph-based driving methods and formal monitors represent scene dependencies or evaluate predefined safety conditions, but their resources and interfaces remain task-specific.The paper identifies a need for an executable interface that assesses heterogeneous risks and returns recognized events with object-bound, type-identified relations.Source: B. Structured Knowledge and Formal… · p. 2

A. Problem Formulation

  • The problem formulation derives risk evidence from structured perception and ego state using a fixed DRKG, then conditions VLM decision-making and diffusion planning on that evidence alongside scene inputs.The evidence contains recognized events and inferred risk relations paired with their deriving rules.Source: A. Problem Formulation · p. 2

B. Scene-applicability Reasoning

The DRKG combines an ontology that structures driving-scene entities and relations with SWRL rules that derive events and risks from scene facts. Risk entailment binds a source and target when a rule's antecedents are satisfied in the current scene.

1) Driving-Risk Knowledge Graph:

  • The DRKG consists of an OWL 2 ontology T and positive SWRL rules RG, with the ontology defining representable concepts and the rules deriving additional assertions.The ontology organizes scene entities, temporal identity, motion and events, causes, and risk phenomena, connected by spatial, kinematic, and other relations.Source: 1) Driving-Risk Knowledge Graph · p. 2
  • Structured perception and ego state instantiate DRKG scene facts; SWRL derives events and risk relations that are combined with context and rule descriptions for VLM input.The VLM decision representation then conditions the diffusion planner.Source: Figure 2 · p. 3
  • A risk relation λrisk(ei, ej) identifies the head entity as the risk source and the tail entity as its target.The deriving rule identifies the risk type.Source: 1) Driving-Risk Knowledge Graph · p. 3
  • The DRKG covers kinematic, interaction, road-structure, right-of-way, occlusion, and compositional risks, represented by rules sharing the λrisk conclusion predicate.Compositional risks combine entities, temporal states, and relations across ontology modules.Source: 1) Driving-Risk Knowledge Graph · p. 3
  • SWRL rules derive a risk relation only when all positive antecedent conditions for the entities, attributes, temporal states, and relations are satisfied in the scene.Intermediate rules can derive scene events or complex relations before a risk rule concludes λrisk(ei, ej).Source: 1) Driving-Risk Knowledge Graph · p. 3

2) Scene-Fact Instantiation:

Scene-fact instantiation maps structured perception and ego state into ontology-grounded facts, while SWRL reasoning subsequently derives complex relations and scene risks from those facts.

2) Scene-Fact Instantiation:

  • Structured perception and ego state initialize the directly obtainable scene fact set using the ontology vocabulary.The set includes entity and class assertions, measured attributes, and simple relations; SWRL rules are not applied during initialization.Source: 2) Scene-Fact Instantiation · p. 3
  • Stable participant identifiers preserve identity across frames, while time-indexed attributes record position, velocity, acceleration, and heading.The adapter also initializes the ego vehicle, traffic participants, road, and lane entities.Source: 2) Scene-Fact Instantiation · p. 3
  • Directly supported or deterministically computed spatial and kinematic relations describe relative positions, motion states, and maneuvers without assigning hazard meaning.Risk instances and other complex relations are derived in the subsequent SWRL stage.Source: 2) Scene-Fact Instantiation · p. 3

3) SWRL-Based Scene Risk Entailment:

  • SWRL reasoning repeatedly applies ontology T and rules RG to derive events and complex relations until no new assertion is produced.The resulting closure retains the initial facts and adds derived assertions; the entailment process is implemented with Pellet.Source: 3) SWRL-Based Scene Risk Entailment · pp. 3–4, Figure 3 · p. 4

4) Scene-Grounded Risk Evidence:

Scene-grounded risk evidence packages recognized events and inferred, object-bound risk relations with their deriving rules. The VLM receives this evidence with visual and ego context, and its decision representation conditions the diffusion planner.

4) Scene-Grounded Risk Evidence:

  • The extracted evidence contains recognized temporal, motion, and unexpected events together with inferred risk relations paired with the rules that derive them.Each relation identifies a risk source and target, while its paired rule identifies the risk type.Source: 4) Scene-Grounded Risk Evidence · p. 4
  • Predefined semantic descriptions render associated risk rules in human-readable text alongside events and inferred relations for VLM input.This avoids requiring the VLM to parse complex ontology-atom conjunctions or repeat low-level kinematic facts already available from perception.Source: 4) Scene-Grounded Risk Evidence · p. 4

1) Risk-Evidence-Conditioned VLM Reasoning:

  • The VLM receives the current camera frame, four historical frames sampled at 2 Hz, ego and scene context, and scene-grounded risk evidence.The textual sources are concatenated with a fixed prompt template.Source: 1) Risk-Evidence-Conditioned VLM… · p. 4
  • The VLM produces a four-level output covering scene, events, risks, and driving decision, whose decision representation is passed to the diffusion planner.The decision level ends with longitudinal and lateral meta-action tokens and a plan token.Source: 1) Risk-Evidence-Conditioned VLM… · p. 4

2) Conditioned Diffusion Planning:

The diffusion planner combines VLM-derived driving signals with geometric and dynamic scene context to generate ego trajectories through conditional denoising.

2) Conditioned Diffusion Planning:

  • The planner conditions trajectory generation on the VLM’s longitudinal, lateral, and planning signals, alongside geometric and dynamic context.The context includes BEV scene features, surrounding-agent features, and ego-motion features.Source: 2) Conditioned Diffusion Planning · p. 4
  • VLM hidden states guide trajectory refinement, while BEV features enter through trajectory cross-attention and agent and ego encodings provide additional conditions.The process performs conditional denoising at each step.Source: 2) Conditioned Diffusion Planning · p. 5

IV. EXPERIMENTS

The experiments examine whether retrieved risk rules support scene-specific conclusions and whether the distinction affects planning. Controlled nuReasoning comparisons vary supplied risk information, with separate retrieval and planning measures.

IV. EXPERIMENTS

  • The evaluation asks whether retrieved rules support scene-specific risk conclusions and whether this distinction affects planning.Controlled planning comparisons vary the supplied risk information, and an evidence ablation separates inferred results from activated rule descriptions.Source: §IV · p. 5

1) Evaluation Dataset:

  • The local nuReasoning split contains 1,943 training, 259 validation, and 387 test scenes, and differs from the unavailable official benchmark test set.Published scores on the official test set therefore provide context rather than a direct comparison.Source: 1) Evaluation Dataset · p. 5

2) Metrics:

  • Retrieval is evaluated by recall of risk-concluding and inference-chain rules and by applicability of retrieved rules to inferred risks.Planning uses NPS, which combines five normalized component scores; NC and DA are multiplicative safety gates, and ADE is reported over a five-second horizon.Source: 2) Metrics · p. 5

B. Implementation Details

The implementation trains a VLM to produce risk-focused reasoning and driving commands, then conditions a diffusion planner on its decision representation and scene information. Planning comparisons vary the supplied risk information while keeping the VLM and planner fixed.

B. Implementation Details

  • Driving-domain pretraining and supervised fine-tuning train the VLM to generate risk-focused reasoning, driving commands, and plan tokens that condition the diffusion planner.The VLM starts from Qwen3-VL-8B and is pretrained on LingoQA and CODA-LM samples before fine-tuning on aligned nuReasoning and PotentialRiskQA samples.Source: B. Implementation Details · p. 5
  • The planner uses a proximity-weighted objective combining trajectory accuracy, command consistency, comfort, and collision avoidance.Scenes with nearby traffic participants receive larger proximity weights.Source: B. Implementation Details · p. 5
  • Planner training first maps the VLM decision representation to future trajectories, then fine-tunes with bird’s-eye-view, agent, and ego-history conditions.The second stage refines the established VLM-to-trajectory mapping with geometric and dynamic scene information.Source: B. Implementation Details · p. 5
  • The KnowVal-style and DriveReg-style baselines retrieve rules from the same DRKG at top-5 and top-16 while planning comparisons hold the other inputs and models fixed.KnowVal uses keyword-based node matching and graph expansion; DriveReg uses paragraph-then-sentence text retrieval.Source: B. Implementation Details · pp. 5–6

C. Rule Retrieval and Applicability

Broader semantic retrieval raises risk-rule recall but lowers scene applicability, including for the rules needed along inference chains. The method’s perfect retrieval scores follow by construction and do not independently validate perception or rule correctness.

C. Rule Retrieval and Applicability

  • KnowVal recall rose from 18.32% at top-5 to 65.57% at top-16 while applicability fell from 11.11% to 8.05%.DriveReg showed the same pattern: recall rose from 29.30% to 33.88%, while applicability fell from 12.58% to 4.04%.Source: C. Rule Retrieval and Applicability · p. 6
  • Chain recall stayed below risk-rule recall in every baseline setting, and chain applicability was only 0.58%–1.55%.For KnowVal top-16, chain recall was 50.73% versus 65.57% for risk rules; the distinct entity, temporal, and relational bindings can leave derivation rules unmatched.Source: C. Rule Retrieval and Applicability · p. 6
  • The method’s 100% recall and applicability follow by construction against the same SWRL-derived reference, not independent validation of perception or rule correctness.It applies the full DRKG rule set to scene facts and returns rules supporting entailed risks.Source: C. Rule Retrieval and Applicability · p. 6

D. Planning with Scene-Grounded Risk Evidence

Scene-grounded risk evidence outperforms semantic retrieval on safety-weighted planning, and the ablation results favor combining inferred scene results with activated rule descriptions. The qualitative example illustrates how scene-grounded reasoning can identify an occluded pedestrian risk.

D. Planning with Scene-Grounded Risk Evidence

  • Scene-grounded risk evidence achieved the highest NPS (64.18) and NC (90.29), compared with 62.88 and 87.53 for KnowVal top-16.NC was the only component mean to rise relative to KnowVal top-16; NC and DA act as multiplicative safety gates in NPS.Source: D. Planning with Scene-Grounded Risk… · p. 6
  • The NPS gain was not accompanied by closer imitation of the reference trajectory; the method primarily changed safety-critical behavior.Source: D. Planning with Scene-Grounded Risk… · p. 6
  • Larger retrieval sets did not reliably improve planning despite higher risk-rule recall, with KnowVal NPS rising from 62.01 to 62.88 and DriveReg changing from 62.53 to 62.49.Both expanded sets lowered risk-rule applicability.Source: D. Planning with Scene-Grounded Risk… · p. 6

E. Ablation of Scene-Grounded Risk Evidence Representation

  • Inferred results alone reached 63.22 NPS, slightly above 62.68 with activated rule descriptions alone.The results were compared with activated rules held fixed.Source: E. Ablation of Scene-Grounded Risk… · p. 6
  • In the occluded-crossing example, the method identifies the applicable pedestrian risk rule and selects strong deceleration, while KnowVal selects gentle deceleration.The top view reveals a pedestrian hidden from the ego view.Source: Figure 4 · p. 7
  • Combining inferred results with activated rule descriptions yielded the highest NPS (64.18) across the four evidence ablations.Inferred events and relations provide scene-specific bindings, while rule descriptions explain the risk type and rationale.Source: E. Ablation of Scene-Grounded Risk… · p. 7
  • The single-view VLM planner reached 64.18 NPS, 92.91 DA, and 1.728 ADE, but published scores are context rather than direct comparisons.Published rows use the official test set, whereas this method uses the evaluation split defined in the dataset subsection.Source: E. Ablation of Scene-Grounded Risk… · p. 7, Table IV · p. 7

F. Qualitative Results

Scene-grounded risk reasoning improved nuReasoning scores over the strongest semantic-retrieval condition, and an occluded-crossing example shows how checking rule antecedents changes the planned response. The DRKG encodes reviewed risk rules, while its coverage and perception dependencies bound the evidence it can produce.

F. Qualitative Results

  • For an occluded pedestrian crossing, the method traced braking through hidden-cause inference to ego risk and selected strong longitudinal deceleration while maintaining its lane.KnowVal retrieved pedestrian rules from the crosswalk keyword without checking scene-specific antecedents, yielding gentle deceleration that left the immediate risk insufficiently addressed.Source: F. Qualitative Results · p. 7

V. CONCLUSIONS

  • NC rose from 87.53 to 90.29 and NPS from 62.88 to 64.18 against the strongest semantic-retrieval condition; scene-bound conclusions and rule semantics contributed complementary planning information.Source: §V · p. 7
  • Noisy or missed structured-perception observations can change which SWRL antecedents are satisfied, and incomplete DRKG rule coverage limits the hazards represented in evidence.Future work targets perception robustness, broader automatically acquired risk knowledge, and improved VLM generalization.Source: §V · p. 7

DRKG Construction and Coverage

  • The DRKG groups scenarios into risk types, maps scene entities and relations to five ontology modules, and encodes conditional event and risk derivations as six-expert-reviewed SWRL rules.It contains 177 SWRL rules, including 39 risk-specific rules covering 27 risk types; Table V reports category-level rule coverage and complexity, with atom counts excluding scene- or temporal-membership-only assertions.Source: DRKG Construction and Coverage · pp. 8–9, Table V · p. 9

Related papers

Loading 2609.34145v1…