PaperlayerView exact revision on arXiv

Source-linked AI summary

CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving

Jenny Schmalfuss, Despoina Paschalidou, Simon Gerstenecker, German Ros, Jose M. Alvarez

arXiv:2609.07094v1cs.CV

TL;DR

Driving reasoning lacks a representation that makes spatial, temporal, and causal dependencies explicit and verifiable, alongside a context-complete human reference. CASCADE introduces a structured scene graph and a human-annotated dataset; across 2,066 clips, it includes over 110,000 populated semantic values and thousands of causal links. Its causal links are expert attributions from observational video, not interventionally verified causation.

  • Problem

    Driving reasoning lacks a representation that grounds scene elements in space and time while recording causal dependencies, and lacks a complete human-annotated reference for evaluation.

    Source: §1 · p. 1

  • Method

    CASCADE combines a typed spatio-temporal-causal scene graph with human annotations that capture bidirectional causal chains around the ego and full action and property traces.

    Source: §1 · p. 2

  • Results

    Across 2,066 clips, CASCADE annotates over 110,000 populated semantic values, including 3,749 because-of and 2,979 influenced-by connections.

    Source: §3.2.2 · p. 8

  • Takeaways & Limitations

    CASCADE provides a human reference for benchmarking model reasoning and scoring automatically generated reasoning labels without a judge.

    Source: §5 · p. 12

    Because annotations use observational video without interventions or controlled counterfactuals, causal links are expert attributions rather than interventionally verified causation.

    Source: §4 · p. 11

Abstract

from arXiv · show

Reasoning is a promising route to the generalization that autonomous driving requires in the long tail, as it can infer how the elements of a scene depend on one another and traverse those dependencies to conclusions beyond what is observed. Yet it is hard to tell whether a model's conclusions follow the scene's dependencies, because no driving representation makes them explicit enough to test against. Text-based reasoning traces lack spatio-temporal grounding, spatio-temporal scene graphs lack causal links, and reasoning annotations at scale are increasingly model-generated and hard to verify. To this end, we introduce CASCADE (Causal Spatio-Temporal Analysis of Driving Environments), which encompasses two components: (1) a structured scene representation for reasoning in driving scenes and (2) a human-annotated dataset built on it. For every actor that interacts with the ego vehicle, the CASCADE representation records frame-by-frame, for as long as the actor is visible, what action is taken, where it occurs, and how it depends on the actions and states of others. The resulting structure makes reasoning predictions machine-verifiable: they can be scored against it element by element, without relying on (M)LLM judges. The CASCADE dataset provides comprehensive human annotations for 2,066 driving clips of the PhysicalAI dataset, with over 34K elements that establish the spatio-temporal and causal context of each scene, including 8.6K time-stamped ego and agent actions, 3.7K causal links and 2.9K potential influences, and 6.1K annotations for agents, objects, traffic lights, and environments. Being entirely human-annotated, CASCADE provides the reference for this comparison: benchmarking the reasoning abilities of Physical AI models, and verifying the quality of automatically generated reasoning labels. The CASCADE dataset is available at https://huggingface.co/datasets/nvidia/cascade.

1. Introduction

Driving reasoning lacks a representation that grounds actions in space and time while explaining their causes, and lacks complete human-annotated context for verification. CASCADE pairs a frame-by-frame spatio-temporal-causal scene graph with a human-annotated dataset covering ego-related causal chains and their action traces.

1. Introduction

  • Prior driving reasoning lacks both a grounded, verifiable representation of scene dependencies and a context-complete human reference for evaluating model reasoning.Text traces lack spatial and temporal grounding, scene graphs lack causal links, and reasoning annotations at scale are often model-generated.Source: §1 · pp. 1–2
  • CASCADE represents scenes as typed nodes for five entity types, with edges for causes and spatial relations and time segments grounding actions, properties, and states.Predicted graphs can be verified against ground truth element by element.Source: §1 · p. 2
  • The dataset records bidirectional causal chains around the ego and the full action and property traces of linked actors and objects, with all annotations created and reviewed by human experts.Annotating each 20-second clip takes 0.5–2 hours of expert effort.Source: §1 · p. 2
  • CASCADE’s two contributions are a verifiable frame-by-frame reasoning graph and 2,066 human annotations that capture the ego’s complete causal context over time.The graph and dataset together provide a basis for evaluating reasoning predictions without judges.Source: §1 · p. 2
  • CASCADE combines a spatio-temporal-causal scene graph with a human-annotated dataset that captures complete causal context for actors and objects on the ego’s causal chain.Its 2,066 annotations are human-created and reviewed.Source: §1 · p. 2

2. Related Work

Prior driving reasoning representations trade off causal explanation, spatial and temporal grounding, verifiability, complete causal context, and human annotation. CASCADE presents explanatory graphs for driving videos and a dataset with multi-hop, bidirectional causal context and full action traces.

Structure in reasoning representations

  • Driving reasoning spans causal attributions, text reasoning traces, and scene graphs, but these forms variously lack spatio-temporal grounding, causal links, or verifiability.CASCADE introduces explanatory graphs for driving videos.Source: §2 · pp. 3–4

Representation grounding

  • Earlier representations ground reasoning in space, time, or causes, but CASCADE combines causes anchored in space with frame-by-frame temporal grounding.Text traces and causal attributions typically use coarse temporal grounding, while scene graphs lack causal grounding.Source: §2 · p. 3

Representation verifiability

  • Structured graph ground truth can be scored deterministically, whereas driving reasoning traces are scored by (M)LLM judges whose judgments exhibit biases and instability.CASCADE’s graph is machine-verifiable against ground truth element by element.Source: §2 · p. 3

Dataset complete causal context

  • CASCADE records causal chains of any depth in both directions around the ego, together with full action traces for actors along those chains.The related-work comparison notes that prior datasets often capture only first-degree causes or omit causes from action-traced data.Source: §2 · pp. 3–4

Dataset human annotation

  • CASCADE is identified as the first driving dataset with fully human-annotated causal context.The comparison table assesses datasets by grounding, verifiability, complete causal context, and human annotation.Source: §2 · p. 4, Table 1 · p. 4

3. CASCADE

CASCADE pairs a hierarchical scene representation that makes driving scenes’ spatio-temporal and causal dependencies explicit with a human-annotated dataset linking entities, states, actions, and relations. Its annotations characterize explicit causal dependencies, including triggers and influences, bidirectional connections, and multi-hop reasoning.

3.1.2. Representation Annotation Tracks

  • Temporal tracks describe actions, actor properties, signal and object states, and environmental conditions; dense tracks cover the relevant visible interval, while sparse tracks mark attributes only when present.Actions are dense for ego and agents and can carry illegal flags; properties include speed, status, driving style, and signaling, while states and conditions capture changing lights, objects, and local environments.Source: §3.1 · p. 4, §3.1.2 · pp. 5–6

3.1.1. Representation Entities

  • Five entity classes—ego, agents, traffic lights, traffic objects, and environments—combine persistent descriptors with temporal tracks and relational links, and are annotated while visible with visual-grounding keypoints.Environments provide spatial anchors; traffic-light signal heads separately track states and controlled lanes, while agents and objects carry types and relevant attributes.Source: §3.1.1 · p. 5

3.1.3. Representation Linkage Mechanisms

  • CASCADE links parallel entity timelines into a spatio-temporal-causal graph, representing direct causes, broader influences, action targets, and spatial grounding.Because-of links can connect actions to other actions, properties, signal states, objects, environments, or defined causes, allowing causal chains; influenced-by links capture factors affecting decisions without necessarily causing a specific action.Source: §3.1.3 · p. 6

3.2.1. Data Collection and Annotation

  • From 300,000 PhysicalAI-AV clips, researchers selected 2,800 complex scenes and used two independently annotated-and-reviewed stages to build the dataset.Stage 1 identifies ego actions, relevant entities, and causal links; Stage 2 adds environments, containment, and ego-relative poses, with failed reviews returned for rework.Source: §3.2.1 · pp. 7–8

3.2.2. Dataset Overview

  • 2,066 train and validation clips contain more than 110,000 populated semantic values, averaging 54 per clip across five entity classes.Static scene context contributes about 50% of annotation weight. The dataset includes 5,324 ego and 3,387 agent action segments, 2,057 object and 1,631 signal-head states, and 3,749 because-of, 2,979 influenced-by, 1,235 action, and 2,082 containment links.Source: §3.2.2 · p. 8

3.2.3. Dataset Detailed Statistics

  • Across 8,711 action segments and more than 30 action types, agents add pedestrian-specific behaviors, while vulnerable road users make up about one fourth of annotated agents.Roughly one salient agent is annotated per clip based on relevance to the ego, not exhaustively; the action vocabulary ranges from frequent lane-keeping to rare reversing or running.Source: §3.2.3 · p. 8
  • The 1,165 environment-condition annotations are mostly construction zones and wet or snowy surfaces; traffic-light labels include 213 yellow states, and traffic objects are chiefly signs, cones, and barriers.About 500 signs fall into the free-text “other” category, spanning regulatory, lane/direction, construction, and warning types beyond fixed schema categories.Source: §3.2.3 · pp. 8–9
  • The 2,979 influenced-by entries contain 3,451 influence references: 2,646 entries name one influence and 333 name two to seven.Most clips have single-hop causal chains, while some reach depth five.Source: §3.2.3 · pp. 9–10, Table 3 · p. 10, Figure 12 · p. 10
  • Influenced-by links mostly point to standing infrastructure, whereas because-of links identify dynamic triggers such as agent actions and traffic-light states.The distinction records not just the relevant entity but, for causes, the specific action or signal state.Source: §3.2.3 · p. 10
  • Nearly 96% of CASCADE clips contain explicit causal dependencies; 28% have causal chains of depth 2 or more, while 84 clips have none.Most chains are single hop, but multi-cause chains are sufficiently frequent to warrant inclusion in reasoning evaluations.Source: §3.2.3 · p. 11
  • Causal connections predominantly link entities to ego reactions, but annotations also capture cases where ego affects another entity.This bidirectionality indicates that reasoning graphs represent effects in both directions, not only causes of ego behavior.Source: Table 5 · p. 10, §3.2.3 · p. 11
  • Yielding and stopping are the most frequently caused actions, and 335 cases mark the trigger as unknown when it cannot be inferred from video.Source: §3.2.3 · p. 11
  • Regulatory infrastructure, including traffic lights, cones, and signs, dominates influences on both ego and other agents; ego has more influence annotations than other agents.The greater number for ego is expected because the annotations are ego-anchored.Source: §3.2.3 · p. 11, Figure 14 · p. 11
  • CASCADE combines diverse, densely annotated scene elements with explicit reasoning and dense multi-hop annotations absent from prior human-labeled driving datasets.Source: §3.2.3 · p. 11

4. Limitations

CASCADE’s causal links are human attributions from observational video, not interventionally verified causation, and its causal chains are rooted in the ego. These boundaries reflect what the annotations capture and the deliberate scope of annotation.

4. Limitations

  • CASCADE’s causal links are expert human attributions of positive dependencies, not interventionally verified causation.The annotations record why an action occurred, not why an alternative did not.Source: §4 · p. 11
  • Because the annotations use observational video without interventions or controlled counterfactuals, they do not verify causal links through interventions.Source: §4 · p. 11
  • CASCADE’s ego-rooted chains omit interactions among other actors that never involve the ego, a deliberate scope choice to focus annotation effort.The authors report roughly 1,500 annotation and review hours, plus 300 training hours.Source: §4 · p. 11

5. Conclusion

CASCADE pairs a spatio-temporal-causal scene graph with an entirely human-annotated dataset to support driving-reasoning evaluation. It instantiates the representation on 2,066 real-world clips, with over 34K reasoning-relevant elements and over 110K annotated fields.

5. Conclusion

  • 2,066 real-world clips contain over 34K reasoning-relevant elements described by over 110K annotated fields.Source: §5 · p. 12
  • CASCADE’s typed reasoning graph grounds where, when, and why actions occur and makes predictions verifiable against ground truth element by element, without (M)LLM judges.Source: §5 · p. 12
  • The human-reviewed dataset captures ego-rooted causal chains of arbitrary depth in both directions and full action and property traces for actors along them.Source: §5 · p. 12
  • CASCADE provides a human reference for benchmarking model reasoning and checking pseudo-label quality.The paper describes this as a prerequisite for reliable reasoning supervision at scale.Source: §5 · p. 12

A. CASCADE Annotation Schema

The CASCADE annotation schema is a reasoning scene graph whose entity nodes and linkage-mechanism edges represent scene entities and their causal and spatial connections.

A. CASCADE Annotation Schema

  • The schema represents annotations as a graph with nodes for five driving-scene entity types and edges for causal and spatial linkage mechanisms.The appendix covers entity types, linkage mechanisms, cross-cutting conventions, and clip-level metadata.Source: A. CASCADE Annotation Schema · p. 13

A.1. Entities

Each entity’s annotation schema uses type properties, time-dependent tracks, and links to other entities. The appendix’s entity descriptions enumerate the available options for those building blocks.

A.1. Entities

  • Each entity is described using type properties, annotation tracks, and linkage mechanisms.Source: §A.1 · p. 13
  • Type properties are clip-level attributes; tracks annotate actions, properties, state, and conditions over time; linkage mechanisms encode causal or spatial relations to other entities.Source: §A.1 · p. 13
  • Table S1 summarizes which building blocks each entity uses, and options marked Other require a free-text description.Source: §A.1 · p. 13

A.1.1. Ego Vehicle

The ego vehicle is represented with one action at a time across its full clip, using a detailed vocabulary for movement and transitions. Links and sparse properties capture what its actions relate to and other relevant states.

A.1.1. Ego Vehicle

  • Ego actions are dense, with exactly one action at a time, spanning driving, speed changes, yielding, lane maneuvers, turns, transitions, and fallback actions.Follow-road-user and overtake actions link to the relevant agent; nudge, enter, and exit link to the related agent or traffic object, or Unknown.Source: §A.1.1 · p. 13
  • Sparse properties may overlap actions and use the agent property vocabulary, including stopped, slow, fast, aggressive, erratic, emergency, on duty, and double parked.The listed property options also include Outside camera view, Signal, and Other.Source: §A.1.1 · p. 13

Linkage mechanisms.

Ego linkage mechanisms record causes, broader influences, and spatial containment or location in the scene.

Linkage mechanisms.

  • Because-of links attach to each action and are mandatory for Stop, Decelerate, and Yield, identifying causes such as actions, properties, signals, traffic lights, objects, or environments.Unknown and Other are available when a cause is not visible or identifiable, or needs a free-text description.Source: Linkage mechanisms · p. 13
  • Influenced-by links are sparse and attached to the ego as a whole, with targets including agents, agent properties, traffic-light states, traffic lights, traffic objects, environments, Unknown, and Other.Source: Linkage mechanisms · p. 13
  • Containment is dense, keeping the ego in at least one environment at every moment; its spatial target is an environment with a lane.Ego-relative Pose is not applicable.Source: Linkage mechanisms · p. 13

A.1.2. Agents

CASCADE annotates road users that interact with or influence the ego, including parked vehicles, throughout each agent’s visibility window. Agent tracks classify road-user type, actions, properties, and links to other scene elements.

A.1.2. Agents

  • Agents include road users that interact with or influence the ego, including parked vehicles, and each is annotated for its visibility window.Source: §A.1.2 · p. 13
  • Each agent receives one road-user class, with unspecified Vehicle or Pedestrian labels used when subtype is indeterminable; amount is recorded as single or row/group.Classes include vehicle and pedestrian subtypes, animals with descriptions, and Other.Source: §A.1.2 · p. 13, Table S1 · p. 14
  • Actions are dense, one at a time, and vocabulary depends on agent type, with shared actions plus vehicle maneuvers and pedestrian or animal movement actions.Vehicle-only actions include parking, lane changes, nudges into the ego lane, and overtakes; pedestrian and animal actions include standing, walking, running, and jaywalking.Source: §A.1.2 · pp. 13–14
  • Sparse agent properties may overlap actions; Signal properties record source, intent, and addressed road users, while Because-of links are mandatory for Stop, Decelerate, and Yield.Signal sources include flashing lights, hand gestures, or signs; intents include indicators, stopping, slowing, proceeding, following, caution, and danger.Source: §A.1.2 · p. 14

Linkage mechanisms.

The representation specifies annotation and linkage patterns for traffic objects, signal heads, environments, and scene-graph relations. Shared conventions make tracks and links temporally grounded and provide clip-level context.

Linkage mechanisms.

  • Entity-specific linkage patterns tailor containment and pose to what is being annotated, including lane-aware environment targets and ego-relative position and direction for agents.Objects and environments target environments through containment, while traffic lights distinguish structure location from the lanes controlled by each signal head.Source: Linkage mechanisms · p. 14, §A.2 · p. 16

A.1.3. Traffic Objects

  • Traffic objects are static or regulatory items that can influence behavior, classified by object type and annotated over visibility; they have dense state and containment but no actions or Influenced-by links.Object classes include signs, infrastructure, portable indicators, fallen objects, and debris; edge placements can record a near flag and side.Source: §A.1.3 · p. 14

A.1.4. Traffic Lights

  • Traffic lights are annotated over the physical structure’s visibility window, with tracks and links recorded per signal head; yellow states add three yes/no questions about the ego’s path, intersection position, and safe clearance.A physical structure can hold one or more signal heads; states record color, shape, and mode.Source: §A.1.4 · pp. 14–15

A.1.5. Environments

  • Environments represent visible road infrastructure, and several may be active at once, such as a road and its approaching intersection.Each environment records its class, lane count, and one-way status, with optional overlapping conditions such as construction, wetness, snow, or overgrowth.Source: §A.1.5 · p. 15

A.2. Linkage Mechanisms

  • Linkage mechanisms ground causal relations with Because-of and Influenced-by, and spatial relations with Containment and Ego-relative Pose; temporal grounding comes from time segments on links and connected elements.Because-of attaches to an action and can chain multiple causes; Influenced-by attaches to an entity as a whole for influences that do not trigger a specific action.Source: §A.2 · pp. 15–16

A.3. Shared Conventions and Clip Metadata

  • Shared conventions time-stamp every track and link; Actions, State, Containment, and Ego-relative Pose are dense over visibility windows, while Properties, Conditions, Because-of, and Influenced-by are sparse and may overlap.The conventions also define illegal flags, sparse image-space keypoints, free-text fields, and clip-level relevancy, brief description, and ego driving judgment.Source: §A.3 · p. 16

Related papers

Loading 2609.07094v1…