PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene Graph-based Driving Scenario Extraction for Automotive Egocentric Datasets

Stefan Ramdhan, Kyanna Dagenais, Vera Pantelic, Victor Bandur, Mark Lawford

arXiv:2609.00333v1cs.RO

TL;DR

Large unlabelled sensor datasets are difficult to search for safety-critical ADS scenarios, limiting practical scenario-level coverage analysis. The paper combines scene-graph sequences from egocentric data and HD maps with LTL specifications and model checking to extract localized scenario instances. Evaluation on Argoverse 2 logs and dashcam videos reports high accuracy, while identifying tracking quality and dataset scope as important boundaries.

  • Problem

    Large ADS datasets are costly to inspect manually, creating a need to locate safety-critical scenarios for scenario-based coverage analysis.

  • Method

    The method generates scene-graph sequences from egocentric data and HD maps, specifies scenarios with LTL, and uses model checking to extract instances.

  • Results

    The method achieves high accuracy on 850 15-second Argoverse 2 driving logs and dashcam videos, with strong Level II and III interval-matching and temporal-localization F1 scores.

  • Takeaways & Limitations

    Temporal-logic queries over scene graphs provide a viable option for identifying real-world scenario instances and enabling scenario-level ADS dataset coverage analysis.

  • Takeaways & Limitations

    The evaluation is midsize and North America-only, and the absence of ground-truth scenario labels requires a manually verified rule-based benchmark.

Abstract

from arXiv · show

Extracting scenarios from unlabelled real-world sensor data streams is a critical but challenging task in the development process of automated driving systems (ADS). Automatically sifting through large datasets to spatially and temporally locate critical scenarios can enable scenario-based coverage analysis of ADS datasets. In this paper, we present a method for extracting scenarios from egocentric datasets using scene graphs and Linear Temporal Logic (LTL). We first process egocentric sensor data and HD maps to generate a sequence of scene graphs representing a driving scenario. Next, we use LTL to formally specify driving scenarios of interest, then extract all instances of the scenarios from the dataset using an off-the-shelf model checker, which evaluates the LTL formula against the sequence of scene graphs. Our approach can be used on both simulated and real world datasets. We evaluate the method on the training and validation datasets from Argoverse 2 consisting of 850 15-second real-world driving logs, and several videos of dashcam footage. We demonstrate the effectiveness of our approach for extracting and querying scenarios by evaluating against a rule-based benchmark based on track annotations and HD maps.

I. INTRODUCTION

Large ADS datasets are essential for development and V&V but too costly to inspect manually for safety-critical scenarios. The paper proposes semantic, temporally localized extraction from real-world egocentric data using scene graphs and LTL.

  • Large scenario datasets support ADS training, deterministic safety-model parameterization, and V&V across common and edge-case ODD scenarios.
  • Manual interpretation is prohibitively expensive because datasets are too large for identifying safety-critical scenarios such as stopped-school-bus encounters.
  • The method represents datasets with scene-graph sequences, specifies target scenarios in LTL, and focuses on extraction rather than runtime monitoring.
  • It can determine whether scenarios occur and spatially and temporally localize all instances for dataset coverage analysis.
  • The evaluation uses 850 15-second Argoverse 2 driving logs plus dashcam videos, with high accuracy when vision semantics are complemented by state and HD-map data.
  • The toolchain improves ROADSCENE2VEC through track-based labeling and refined lane identification, while the paper introduces no new algorithms.

B. Scenario Extraction

Existing extraction methods use annotations or feature clustering, but annotation rules miss camera-derived semantics and clustering methods may require costly retraining. The paper positions scene graphs as a sensor-agnostic representation for deterministic, vision-conditioned extraction.

  • Rule-based methods are interpretable but limited by annotation content, especially for camera-derived semantics such as weather and road conditions.
  • Feature-clustering methods learn scenario representations, but some require retraining and additional data for scenario classes absent from training.
  • Video provides valuable semantic context, yet its high dimensionality makes direct extraction difficult and off-the-shelf VLM repurposing performs poorly.
  • The authors identify a gap for deterministic, interpretable extraction conditioned on vision-derived semantics.
  • A scene graph represents objects as nodes and spatial relationships as edges, providing a structured semantic basis for scenario reasoning.
  • The paper applies scene graphs to egocentric real-world, vision-based scenario extraction, a use the authors state has not previously been addressed.
  • For real-world egocentric video, the method uses ROADSCENE2VEC, an open-source scene-graph generator originally developed for scene-level embeddings, risk assessment, and collision prediction.

D. Linear Temporal Logic

The method turns forward-facing video into enriched scene-graph sequences, then uses semantic and state information to support temporal scenario extraction. Scene-graph enrichment addresses tracking, lane representation, relations, and vision-derived attributes, while retaining deterministic scenario evaluation after enrichment.

  • Pipeline: The pipeline represents each camera frame as a scene graph, enriches graphs with relevant semantics, and evaluates scenario specifications over the resulting sequence.The approach uses scene graphs as the semantic representation of driving data and includes HD-map and vision-derived information when available.
  • Map enrichment: HD maps correct RS2V’s three-lane simplification by adding lane and intersection nodes and graph edges representing road geometry.Without HD maps, lane assignment may instead be estimated with a vision-based lane detector.
  • Scene-graph limitations and enrichment: RS2V’s coarse semantics result from grouped distance ranges, heading-independent spatial relations, and a limited ontology, although richer scene graphs are better suited to extraction.The approach extends the toolchain with tracking, state-space and relation enrichment, map correction, and MLLM-based semantic enrichment.
  • Object tracking: Temporally consistent object labels are generated with Kalman filtering and the Hungarian algorithm, enabling entities to be tracked across scene graphs.The resulting perception tracks provide the temporal consistency required for scenario extraction.
  • State-space enrichment: Track annotations can enrich scene-graph nodes with position, velocity, and heading, while their existence is not required because tracking and state-estimation models may substitute for them.The method also revises spatial relations using lower-noise state estimates.
  • Vision-derived attributes: After MLLM answers enrich the graphs, the graphs remain static, so the extraction process retains determinism and interpretability despite MLLM non-determinism.Weather and road-surface conditions can be queried once per short clip, while school-bus stop-sign status is queried per frame for interval extraction.

C. Scenario Specification

Scenarios are specified as LTL formulas over predicates that describe scene-graph relations, state-space conditions, and HD-map data. Temporal operators compose ordered events, while additional predicates can constrain attributes such as weather or deceleration.

  • Scenario specification: A scenario specification is an LTL formula composed of predicates over scene-graph relations, state spaces, and HD-map data.The predicates use atomic conditions such as ego motion or lane occupancy and can refer to symbolic entities consistently across graph sequences.
  • Scenario specification: The aggressive cut-in example defines three ordered events: adjacent right-lane driving, entering the ego lane, and a near collision in the ego path.The events are represented by predicates evaluated over scene-graph nodes and relations.
  • Temporal composition: The cut-in formula requires φ1 until φ2 occurs, φ2 until φ3 occurs, and φ3 eventually ends.This encodes both event ordering and persistence over the scene-graph sequence.
  • Attribute constraints: Weather can be represented by a predicate evaluated for every scene graph and conjoined with the existing scenario specification.For constant weather over a short scenario, the predicate checks whether each graph records snowy conditions.
  • Attribute constraints: Logical scenarios can constrain state-space variables, such as requiring longitudinal deceleration to exceed a threshold during the scenario.The threshold condition is added as a conjunction to the scenario formula.

D. LTL Checking

LTL checking is treated as model checking over a sequence of scene graphs. The implementation adapts SceneFlow to identify satisfied scenario specifications rather than specification violations.

  • LTL checking: The method uses SceneFlow, a deterministic finite-automaton implementation, to evaluate LTL specifications over scene-graph sequences.SceneFlow was modified to flag specification satisfactions instead of violations.

IV. EVALUATION & RESULTS

The evaluation uses Argoverse 2 logs and supplemental dashcam videos to cover scenarios not present in the annotated driving logs. The dashcam clips provide school-bus examples using video-only scene graphs.

  • Datasets: The evaluation uses Argoverse 2 training and validation splits containing 15-second logs with track annotations and HD maps.These resources enrich the scene graphs used by the pipeline.
  • Datasets: Several dashcam videos were added because the Argoverse 2 logs lacked school buses stopping with their stop signs extended.The supplemental footage targets a scenario absent from the available logs.
  • Datasets: Seven dashcam videos met the filtering criteria, and their scene graphs were produced only from video because sensor and map data were unavailable.The footage had to be steady and clearly show a stopped school bus with its stop sign extended.

A. Ablation

The ablation evaluates how progressively enriching scene graphs affects scenario extraction, then applies LTL-defined scenarios across driving, weather, road, braking, and school-bus conditions.

  • Ablation: The ablation starts with temporally consistent coarse scene graphs, enriches them with track-annotation kinematics, and continues to Level III.Level I uses coarse RS2V ontology semantics with temporal consistency; Level II adds kinematic data from track annotations.
  • Scenario Definitions: Longitudinal car following is extracted as an abstract scenario specified by a conjunction of predicates.
  • Scenario Definitions: The method also extracts an abstract cut-in-from-right-side scenario defined using an LTL specification.
  • Conditional Scenarios: Weather and road conditions can be added to the extracted longitudinal-following and cut-in scenarios through additional logical conditions.The paper describes this extension as trivial to apply to weather or road conditions.
  • Conditional Scenarios: A harsh-braking logical scenario is defined by the point of view vehicle decelerating at ≥3 m/s2, using β = 3 in its LTL specification.
  • School-Bus Scenarios: School-bus-present and school-bus-stopped scenarios are specified over single events using the same general convention as longitudinal following.The passages introduce formal definitions for both school-bus scenarios but do not reproduce their formulas.

C. Scenario Extraction Benchmark

The benchmark compares extraction against rules over track annotations and HD maps, with manual verification and metrics that account for track identity, temporal coverage, fragmentation, and rare scenarios.

  • Benchmark Construction: The benchmark defines longitudinal-following and cut-in rules over Argoverse 2 track annotations and HD maps, while school-bus cases use filtering or human annotation.School-bus-present filters school-bus tracks within the forward-facing camera FOV; school-bus-stopped uses intervals where the stop sign is extended.
  • Verification: 30 longitudinal-following and cut-in benchmark scenarios were manually verified, all positive school-bus-present logs were verified, and school-bus-stopped labels required no further verification.
  • Evaluation Purpose: The benchmark evaluates agreement with a simple rule-based approach while supporting scenarios such as weather, road conditions, and stopped school buses that rules cannot extract.
  • Results: Level I is expected to disagree more because it relies on noisy video-derived state estimates, whereas Levels II and III include less noisy data.Track annotations provide higher-confidence state estimates derived from multiple sensors.
  • Tracking Metrics: Track fragmentation can create false negatives because switching a track ID breaks the symbolic-entity continuity required for scenario detection.
  • Tracking Metrics: Critical scene-graph track associations are manually checked by comparing their temporal evolution with visualizations based on corresponding track annotations.
  • Scenario Extraction Metrics: Instance-level extraction identifies the scenario interval and track of interest, using many-to-one matching to accommodate fragmented extracted intervals.
  • Scenario Extraction Metrics: An instance is detected when overlapping extracted intervals cover more than 50% of the benchmark interval and identify the correct track.The union of overlapping predicted intervals is used before applying the coverage threshold.

E. Results

Results show that richer scene graphs substantially improve scenario extraction, while tracking and perception errors remain important sources of imperfect performance. The method supports temporal localization and semantic scenario analysis, including vision-derived conditions.

  • Scenario extraction improves as state and HD map information enrich scene graphs with position, velocity, heading, and lane assignment.
  • Level II and III scene graphs achieve very strong F1 scores for interval matching and temporal localization.
  • 0.552 to 0.827: long-following F1 rises when the track-aware constraint is ignored, indicating correct intervals but frequent track misidentification.
  • 0.742 F1 for school-bus presence contrasts with 0.400 interval-matching F1 because detection fragmentation and stop-sign identification compound.
  • Log-balanced accuracy reaches 0.822, 0.819, and 0.901 for long-following at 30 meters, long-following at 60 meters, and cut-in, respectively.
  • Track fragmentation affects extraction: AV2 tracks average 1.587 perception tracks, 34.1% have multiple tracks, and longest-track coverage averages 63.6%.

A. Benefits of the Method

The method enables semantic, deterministic, interpretable, and spatio-temporal scenario extraction from egocentric data, supporting dataset coverage analysis while retaining white-box benefits.

  • Vision-derived semantics: Single-camera extraction captures vision-derived semantics, including weather, road conditions, and object features that existing approaches cannot extract.The approach does not require HD maps or track annotations for the stopped-school-bus demonstration.
  • Semantic scenario specification: Scene graphs provide a semantic abstraction for defining scenarios over entities, relationships, and their temporal evolution.Standardized relational semantics may support reuse of scenario definitions across datasets.
  • Determinism: Given correct scene graphs, specifications, and predicates, the model checker deterministically returns all satisfying intervals and tracks.Non-deterministic MLLM querying is separated from the extraction process.
  • Interpretability: Detailed model-checker traces make the white-box extraction process interpretable and allow misclassifications to be traced to graphs, predicates, or specifications.The method supports spatio-temporal localization and coverage analysis of scenarios in unlabelled egocentric datasets.
  • Temporal Scenario Specification: LTL specifies precise temporal evolution, including ordered events that Boolean logic alone cannot represent.The approach uses atomic predicates over scene-graph semantics.

C. Considerations of SGG and LTL

Extraction quality depends strongly on the scene-graph ontology, semantic perception, evaluation scope, and the benchmark used without ground-truth scenario labels.

  • SGG considerations: The diversity of extractable scenarios is constrained by the SGG ontology, detector coverage, and available relationships between entities.Missing semantics such as an activated school-bus stop sign prevent extraction of corresponding scenarios.
  • Evaluation scope: The evaluation uses a midsize North American dataset, despite diversity across seasons, weather, times of day, and urban settings.The authors plan to expand the scenario set and build a library of LTL-specified scenarios.
  • SGG considerations: Extraction is highly sensitive to SGG quality, so improvements in SGG tooling are expected to improve results.
  • Evaluation scope: Without ground-truth scenario labels, evaluation relies on a paper-specific rule-based benchmark, although manual verification increased confidence in its validity.
  • Conclusion: The demonstrated approach is viable for identifying scenario instances and enabling scenario-level coverage analysis, including weather and road conditions.Future work considers LLM assistance for predicates and LTL specifications, more expressive formal languages, and SPIN for higher assurance.
Loading 2609.00333v1…