PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Vision-Language Models in Driving Semantics: A Multi-Dataset Predicate Framework for Explainable Reasoning

Mohamed Chouai, Fazli Faruk Okumus, Stefan Kugele

arXiv:2609.31636v1cs.LGcs.CVcs.RO

TL;DR

Driving-scene relations expressed by vision-language models can be difficult to verify, and heterogeneous datasets make consistent semantics challenging. This paper derives traceable predicates from measurable scene evidence using shared rules across nuPlan and nuScenes. Predicate grounding has the highest accuracy among evaluated visual-input conditions in seven of nine NuPlanQA subtasks, under oracle grounding.

  • Problem

    Vision-language model relations can be difficult to verify against scene evidence, while dataset heterogeneity complicates consistent semantics across datasets.

    Source: §I · p. 1

  • Method

    The framework derives predicates from measurable scene evidence using fixed rules across dataset-specific groundings and materialises traceable assertions in a Predicate KG.

    Source: §I · p. 1, B. Predicate Definition and Derivation · p. 3, C. Predicate Knowledge Graph and… · p. 4

  • Results

    In seven of nine NuPlanQA subtasks, predicate grounding achieved the highest accuracy among evaluated visual-input conditions; Predicate KG only input outperformed metadata-only input in eight of nine.

    Source: §VIII · p. 7

  • Takeaways & Limitations

    Under oracle grounding, predicates provide a consistent, traceable semantic layer between driving data and vision-language reasoning.

    Source: §VIII · p. 7

    Temporally extended relations such as overtakes remain more sensitive to trajectory continuity and annotation characteristics.

    Source: §VI · p. 6

Abstract

from arXiv · show

Vision-language models are increasingly used for driving-scene understanding, yet the semantic relations expressed in their outputs are often difficult to verify against the underlying traffic situation. This paper introduces a deterministic multi-dataset predicate framework that derives driving-scene semantics from measurable geometric, kinematic, temporal, map, and traffic-control evidence. Dataset-specific interfaces are used only to recover the required scene information, while predicate definitions remain unchanged across nuPlan and nuScenes and are materialised in a common Predicate Knowledge Graph. Quantitative semantic validation against manually annotated predicate relations on 200 scenarios from each dataset yields macro F1 scores of 0.94 on nuPlan and 0.93 on nuScenes, with an average cross-dataset difference of 0.02 across the shared predicates. The Predicate KG is further evaluated using a frozen LLaVA-OneVision-7B model on the nine NuPlanQA subtasks. Predicate grounding achieves the highest accuracy among the evaluated visual-input conditions in seven of nine NuPlanQA subtasks, including Traffic Light (53.2% to 71.5%), Situation Assessment (76.2% to 86.1%), and Action Recommendation (82.9% to 89.0%). Weather/Lighting remains essentially unchanged (89.4% vs. 88.8%), consistent with the absence of corresponding predicates, while Predicate KG only input outperforms metadata-only input in eight of nine subtasks. The results show that deterministic predicates provide a consistent and traceable semantic representation and, under oracle grounding, can reduce visual dependence for reasoning tasks covered by the predicate vocabulary.

I. INTRODUCTION

Driving-scene relations matter for reasoning, but natural-language outputs can be difficult to verify against scene evidence, and dataset heterogeneity complicates consistent semantics. The paper proposes fixed, measurable predicate rules and a traceable representation to connect heterogeneous driving data with vision-language reasoning.

I. INTRODUCTION

  • VLM relation statements can sound plausible yet conflict with traffic geometry or dynamics, while dataset differences complicate consistent cross-dataset reasoning.The relevant differences include object taxonomies, maps, trajectories, coordinate systems, and annotation formats.Source: §I · p. 1

II. RELATED WORK

  • Existing structured representations make traffic relations explicit, but often lack operational definitions or target particular datasets and downstream tasks.The paper distinguishes semantic grounding from dataset integration: each relation must be computed by a fixed, reproducible definition.Source: §I · p. 1, §II · pp. 2–3

I. INTRODUCTION

  • The framework derives relations from measurable scene evidence using explicit rules, then materialises them in a Predicate KG for traceable downstream reasoning.The rules cover geometric, kinematic, temporal, map, traffic-control, and risk-related evidence.Source: §I · pp. 1–2
  • The same predicate definitions are intended to remain unchanged across dataset-specific groundings, supporting a shared semantic interface across heterogeneous data.The framework is designed to facilitate extension to additional datasets and downstream applications.Source: §I · p. 2, §II · p. 3

III. DETERMINISTIC PREDICATE FRAMEWORK

The framework separates dataset-specific recovery of scene evidence from shared, deterministic predicate rules. It uses geometric, temporal, map, interaction, traffic-control, and safety conditions to derive driving relations.

A. Dataset Grounding

B. Predicate Definition and Derivation

  • Each predicate is a Boolean decision over participating entities and evidence within a predicate-specific time window, using fixed rule parameters and thresholds.The window may contain a single frame or multiple observations; safety-related thresholds draw on UN R157.Source: B. Predicate Definition and Derivation · p. 3
  • Safety predicates use UN R157 criteria for following distance, time gap, collision risk, and lane-change gaps.For the stated imminent-collision condition, avoidance requires braking demand of at least 5 m/s2; regular lane changes also have a 3.0 m/s2 deceleration limit.Source: B. Predicate Definition and Derivation · p. 3
  • Spatial relations use the reference participant’s local orientation, while map predicates combine agent geometry with the HD map.Using the participant footprint matters near lane boundaries and intersections, where its centre may remain outside a map element it occupies.Source: B. Predicate Definition and Derivation · p. 4
  • The crossesInFrontOf predicate checks for a shared conflict point along both participants’ forward paths, bounded by a 10 m forward range.The finite range prevents distant trajectory-line intersections from being treated as local crossing events.Source: B. Predicate Definition and Derivation · p. 4
  • Traffic-control predicates distinguish a signal’s state, the movement it controls, and the approaching agent affected before entering the controlled lane connector.The agent-specific relation is no longer emitted once the agent begins to occupy that connector.Source: B. Predicate Definition and Derivation · p. 4

C. Predicate Knowledge Graph and Implementation

The Predicate KG stores derived relations with their entities, predicate, time, and provenance, then materialises them for querying independently of the source dataset organisation.

C. Predicate Knowledge Graph and Implementation

  • A derived relation records its participating entities, predicate, time, and provenance, including source evidence and the rule used to derive it.Unary predicates use the same representation without a relational object.Source: C. Predicate Knowledge Graph and… · p. 4
  • A representative 41-frame nuPlan scenario contains 741,311 predicate assertions, or approximately 18,081 per frame.One inspected frame contains 89 dynamic entity nodes and 4,752 semantic edges representing 21,357 predicate assertions; multiple predicates can describe the same entity pair, and graphs can be filtered for downstream tasks.Source: C. Predicate Knowledge Graph and… · p. 4
  • The implementation separates dataset grounding, predicate computation, and graph materialisation, with definitions, evidence requirements, applicability conditions, and thresholds specified in the Appendix.The implementation and predicate specification are available in the repository.Source: C. Predicate Knowledge Graph and… · p. 4

IV. EXPERIMENTAL EVALUATION

Evaluation of derived predicates combines programmatic checks with semantic inspection of traffic scenes, using temporal context when a predicate depends on interaction evolution.

IV. EXPERIMENTAL EVALUATION

  • The evaluation focuses on derived predicates defined in Sec. III-B.Source: §IV · p. 4
  • Validation combines programmatic checks with semantic inspection of the underlying traffic scenes.Source: §IV · p. 4
  • Temporal context is considered when a predicate’s interpretation depends on the evolution of an interaction.Source: §IV · p. 4

A. Automatic Validation

Validation finds strong agreement between rule-derived predicates and manual labels across nuPlan and nuScenes, while predicate grounding also improves results on many NuPlanQA subtasks. The evaluations use candidate-level scene review and frozen-model reasoning with temporally available, oracle-grounded inputs.

A. Automatic Validation

  • Automated checks test entity and argument validity, numerical values, temporal continuity, predicate dependencies, mutually exclusive relations, and rule-specific invariants.For nuPlan, map-dependent predicates are also checked against the native HD map; rules and thresholds are fixed before semantic evaluation, including crossesInFrontOf at L = 10 m.Source: A. Automatic Validation · pp. 4–5

B. Semantic and Temporal Validation

  • Semantic validation reviews higher-level predicate candidates from 200 scenarios per dataset, with 1,607 instances for nuPlan and 1,266 for nuScenes.Both asserted and nonasserted candidates are labelled present, absent, or ambiguous; ambiguous cases are excluded from precision, recall, and F1, and account for 2.92% in nuPlan and 2.45% in nuScenes.Source: B. Semantic and Temporal Validation · p. 5
  • Temporal review assesses manoeuvre relations over their supporting evolution and persistent relations over consecutive observations; future trajectories check crossesInFrontOf only during validation.Future observations and observed-future predicates are excluded from vision-language experiment inputs.Source: B. Semantic and Temporal Validation · p. 5
  • Macro F1 is 0.94 on nuPlan and 0.93 on nuScenes; shared predicates have mean absolute cross-dataset difference ∆F1 = 0.02, with seven differing by no more than 0.03.Overtakes has F1 = 0.93 on nuPlan and 0.87 on nuScenes; isRelevantSignal reaches F1 = 0.95 on nuPlan but is excluded from the cross-dataset comparison.Source: Table II · p. 5, B. Semantic and Temporal Validation · pp. 5–6

V. PREDICATE-GROUNDED VISION–LANGUAGE REASONING

  • The NuPlanQA evaluation uses a frozen LLaVA-OneVision-7B model with fixed questions and decoding, comparing visual, metadata, and Predicate KG inputs.Predicate selection is fixed and question-independent, uses information only through the visual history’s final timestamp, and evaluates oracle grounding from annotated nuPlan states and map information.Source: §V · p. 6
  • With visual input, predicate grounding leads in seven of nine subtasks, including Traffic Light (53.2% to 71.5%) and Situation Assessment (76.2% to 86.1%).Weather/Lighting remains essentially unchanged (89.4% vs. 88.8%), consistent with the absence of weather predicates.Source: §V · p. 6
  • Without visual input, Predicate KG outperforms raw metadata in eight of nine subtasks, including Traffic Light (64.2% vs. 47.6%) and Situation Assessment (79.3% vs. 69.3%).Source: §V · p. 6

VI. DISCUSSION AND LIMITATIONS

The paper reports cross-dataset validation and NuPlanQA grounding results, while noting evidence and vocabulary limits. It also identifies structured querying and sensor-based predicate extraction as directions for future work.

VI. DISCUSSION AND LIMITATIONS

  • Temporally extended relations such as overtakes remain more sensitive to trajectory continuity and annotation characteristics.Source: §VI · p. 6
  • Predicate reliability depends on dataset evidence, track quality, map geometry, temporal coverage, and predicate-specific threshold assumptions; rare interactions may need broader validation.Some predicates cannot be instantiated across datasets when equivalent evidence is unavailable.Source: §VI · p. 7
  • Beyond VLM grounding, the Predicate KG can support structured querying and consistency checking against measurable scene evidence.Future work will investigate extracting predicates directly from sensor observations.Source: §VI · p. 7

VIII. CONCLUSION

  • Macro F1 was 0.94 on nuPlan and 0.93 on nuScenes, with a 0.02 mean F1 difference across shared predicates.Seven of eight shared predicates differed by no more than 0.03.Source: §VIII · p. 7
  • Predicate grounding achieved the highest accuracy among visual-input conditions in seven of nine NuPlanQA subtasks, chiefly benefiting interaction- and reasoning-oriented tasks.The conclusion also reports that Predicate KG only input outperformed metadata-only input in eight of nine subtasks.Source: §VI · p. 6, §VIII · p. 7

APPENDIX

The appendix specifies operational definitions for seven predicate families and establishes common notation for their geometric and temporal evidence. Spatial predicates use heading-relative sectors and mutually exclusive distance or contact states.

APPENDIX

  • The implementation defines 83 predicates across seven families: spatial, motion, temporal, map, interaction, traffic-control, and risk.The families contain 12, 17, 17, 25, 8, 3, and 1 predicates, respectively.Source: APPENDIX · p. 7
  • A predicate holds exactly when its defining condition is satisfied; ordinary equalities store numerical quantities.Source: APPENDIX · p. 7

A. Common Notation

  • Common notation defines subject and object positions, heading-relative displacement, and oriented footprints; positive longitudinal and lateral displacements mean ahead and left.Source: A. Common Notation · pp. 7–8
  • Timestamps are normalised to seconds, and continuity is preserved only when the temporal gap is at most 0.75 s under the default settings.The defaults are a 0.5 s sample interval and a maximum temporal-gap factor of 1.5.Source: A. Common Notation · p. 8

B. Spatial Predicates

  • Spatial predicates use the subject body-heading frame, with longitudinal, lateral, and pure-side deadbands of 1.0 m, 0.75 m, and 2.0 m; valid pairs receive one primary sector.Source: B. Spatial Predicates · pp. 8–9
  • The four distance or contact states, including veryNear and near, are mutually exclusive.Source: B. Spatial Predicates · p. 9

C. Motion Predicates

Motion predicates represent entity kinematics and select conservative travel directions from map and motion evidence; temporal predicates track changes and distinguish continuous observation streaks from total history.

C. Motion Predicates

  • The motion predicates store planar velocity and speed, acceleration components, relative-speed measures, closing speed, and velocity toward a target.Tracked-object velocities use a common global frame, while ego velocity is converted to the global geometric-center velocity.Source: C. Motion Predicates · p. 9
  • Effective travel heading is selected conservatively from map, velocity, and displacement directions, subject to evidence thresholds and agreement checks.Displacement must reach 0.40 m and velocity 0.75 m/s to be informative; available motion cues must agree within 0.45 rad, and motion-derived direction must agree with an unambiguous map heading within 0.60 rad.Source: C. Motion Predicates · pp. 9–10

D. Temporal Predicates

  • Temporal predicates record elapsed time, displacement, heading and speed changes, and center- and free-space-distance changes between observations.Source: D. Temporal Predicates · p. 10
  • Continuous observed counts and durations restart after a gap exceeds 0.75 seconds, while total observation counts include observations separated by gaps.The continuous count increments only when successive observations satisfy the 0.75-second condition; temporal derivatives and continuous-streak quantities are not bridged across larger gaps.Source: D. Temporal Predicates · p. 10

E. Map Predicates

Map predicates represent entity membership, local map attributes, and relationships between map primitives. Primary-map matching is conservative when candidate evidence is ambiguous.

E. Map Predicates

  • Map predicates encode center and footprint membership, local heading, curvature and speed limit, and relationships to parent roadblocks and intersections.Source: E. Map Predicates · pp. 10–11
  • Primary lane or connector assignment is made only when one candidate is conservatively selected; sufficiently similar candidates do not receive a forced assignment.The matching uses exact unbuffered map geometry, with candidates assessed by center containment or sufficient footprint overlap.Source: E. Map Predicates · p. 10
  • Spatial map relations classify primitives as identical, adjacent, successor, predecessor, or unrelated according to directed map topology.Signed path distance uses accepted direct or connected successor paths and does not force a route through an ambiguous branch.Source: E. Map Predicates · p. 11

F. Safety-Related Thresholds

Safety-related thresholds draw on UN Regulation No. 157, while interaction predicates combine geometric, kinematic, map, and history evidence to represent driving relations and maneuvers.

F. Safety-Related Thresholds

  • Imminent collision risk uses a braking-demand threshold of 5 m/s2, while regular lane changes constrain target-lane deceleration to 3.0 m/s2 and require at least a 1.0-second-distance gap.The safety thresholds for following, collision risk, and lane-change gaps follow UN Regulation No. 157 criteria.Source: F. Safety-Related Thresholds · p. 11

G. Interaction Predicates

  • Following requires compatible motor vehicles on a reliable directed path, with a unique nearest forward leader; queuesBehind adds a stop-and-go condition.The shared following requirements include a positive gap, nonoverlapping footprints, and persistence; queue following is also represented by follows.Source: 1) follows · p. 12, 2) queuesBehind · p. 12
  • Lane-change decoding uses stable source and target corridors, physical onset and completion evidence, and validated map adjacency or geometric fallback criteria.The described implementation uses overlap thresholds and temporal support frames to assess onset and completion.Source: 3) changesLane · p. 12
  • MergesInFrontOf and mergesBehind require a target-stream relation supported by pre-entry presence, target-path overlap, compatible travel direction, and stable order.Stable order is checked for at least two frames within a 2.0-second confirmation window.Source: 4) mergesInFrontOf and mergesBehind · p. 12
  • CrossesInFrontOf evaluates intersecting finite forward rays and filters for dynamic road users, observation history, and local context.The described filters include at least three observations, 1.0 second of simultaneous observation, and 0.30 m of track displacement.Source: 5) crossesInFrontOf · p. 13
  • YieldsTo requires a verified crossing in front of the subject and checks a pre-event response, conflict clearance, and post-event verification.A relevant red signal invalidates the yield interpretation.Source: 6) yieldsTo · p. 13
  • Overtaking is defined as a complete, fully observed same-direction Case-1 maneuver, with at least 0.5 seconds required for each phase.The subject must return stably to the original corridor for two frames after at least 1.0 m of full clearance.Source: 7) overtakes · p. 13

H. Traffic-Control Predicates

Traffic-control predicates represent which lane-connector movements a signal controls, the latest valid signal state, and whether the signal is relevant to an agent.

H. Traffic-Control Predicates

  • The controls relation links a traffic signal to its controlled lane-connector movements independently of the signal’s current color.Source: H. Traffic-Control Predicates · p. 13
  • A signal is relevant to an agent only before the agent enters its controlled movement, and the predicate stops once the agent occupies the connector.The signal-state predicate stores the latest valid state at time t.Source: H. Traffic-Control Predicates · pp. 13–14

I. Risk Predicates

The conflict-risk predicate combines motion, footprint, and interaction conditions with predicted-safety criteria. It assumes constant velocity and fixed heading, and does not use observed future ground truth.

I. Risk Predicates

  • Risk requires two dynamic road users, valid oriented footprints without existing overlap, radial closing speed of at least 0.50 m/s, and a qualifying interaction condition.Interaction conditions include lane-path topology, proximity for a VRU pair, a tight same-flow corridor, or local cross/opposite motion.Source: §I · p. 14
  • The broad-phase condition uses minimum center distance under the constant-velocity model, and the prediction evaluates time and oriented-footprint clearance at the predicted minimum.H denotes the configured prediction horizon.Source: §I · p. 14
  • An imminent collision requires collision-avoidance braking demand of at least 5 m/s^2; lane-change conflicts also consider target-lane deceleration above 3.0 m/s^2 or a gap below one second of ALKS travel.The predicted conflict must satisfy the applicable safety criteria.Source: §I · p. 14
  • Prediction assumes constant velocity and fixed heading, uses a 2.0 s base horizon (3.0 s in the retained standard launcher), and does not use observed future ground truth.The risk computation also uses a 0.50 m predicted-clearance threshold, minimum radial closing speed of 0.50 m/s, minimum clearance reduction of 1.0 m, and 24 bounded optimization iterations; it emits once per unordered pair and frame.Source: §I · p. 14

Related papers

Loading 2609.31636v1…