PaperlayerView exact revision on arXiv

Source-linked AI summary

TrafficImag: A Benchmark for Counterfactual Roadside Traffic Video Generation

Xiangyu Li, Tianyi Wang, Zhihao Dou, Christian Claudel, Zhaomiao Guo

arXiv:2609.30722v1cs.CVcs.AI

TL;DR

Existing roadside datasets do not evaluate whether models can generate counterfactual futures that respect road topology and preserve unrelated traffic. TrafficImag combines a roadside corpus with a factorized evaluation protocol; its studies find that improving program and anchor inputs raises downstream success, while rollout failures remain.

  • Problem

    Existing roadside datasets support perception, forecasting, and visual question answering but do not evaluate generated counterfactual futures that preserve road topology and unrelated traffic.

    Source: Introduction · pp. 1–2

  • Method

    TrafficImag combines a roadside image and video corpus with a factorized benchmark linking behavior reasoning, image editing, conditional generation, and four traffic-validity checks.

    Source: Introduction · p. 2, Data Collection · p. 4, Task Definition · p. 5

  • Results

    Oracle audits show that correcting the anchor yields a larger gain than correcting the program alone, and auditing both raises success further without closing the remaining gap.

    Source: Discussion · p. 9

  • Takeaways & Limitations

    TrafficImag diagnoses failures across an execution chain and tests topology-aligned conditions beyond aggregate video-quality metrics.

    Source: Conclusion · p. 10

    The benchmark covers two fixed-camera sites, and its image-coordinate topology cannot certify metric speed, clearance, or collision risk.

    Source: Actor Programs and Topology-Aligned… · p. 5, Conclusion · p. 10

Abstract

from arXiv · show

Existing roadside traffic datasets support perception, forecasting, and visual question answering, but they do not evaluate counterfactual video generation, in which a selected actor is modified and the generated future should remain consistent with road topology and unrelated traffic. We introduce TrafficImag, the first benchmark for counterfactual roadside traffic video generation. TrafficImag combines a large-scale roadside dataset (9,022 annotated images, 7,043 deduplicated video clips, and 31,145 actor-centered history-future samples) with an executable protocol that supports behavior reasoning, intervention-aware image editing, and conditional video generation. Each intervention is represented as an actor-level program describing the target actor, intended behavior, legal route, interaction order, and temporal constraints, enabling a unified evaluation interface across heterogeneous foundation models. TrafficImag evaluates four complementary validity dimensions: initial-state correctness, route and behavior validity, interaction consistency, and non-target preservation, and considers an end-to-end counterfactual successful only when all four are satisfied. Across state-of-the-art foundation models, the strongest reasoner reaches 80.4% macro F1, the complete condition interface raises end-to-end success from 23.3% to 55.0% for the best generator. Oracle studies further show that conditional video execution is the primary remaining bottleneck. TrafficImag provides a reproducible benchmark for evaluating and diagnosing counterfactual traffic video generation beyond perceptual video quality.

Introduction

Fixed roadside views expose interactions across lanes and conflict zones, but existing datasets and benchmarks do not test whether models execute actor-level counterfactuals while preserving unrelated traffic. TrafficImag combines a roadside dataset with an evaluation protocol for reasoning, image intervention, and conditional video generation.

Introduction

  • A valid counterfactual must edit the initial state consistently and generate a legal, identity-consistent rollout while preserving traffic outside the intervention’s influence.Errors in the anchor’s location, scale, or heading can persist through the generated horizon, while a correct initial frame does not guarantee a correct future.Source: Introduction · pp. 1–2
  • 9,022 annotated images, 7,043 deduplicated clips, and 31,145 actor-centered history–future samples form TrafficImag’s roadside resource.The resource spans two fixed-camera sites and includes trajectory-derived behavior labels and topology associations.Source: Introduction · p. 2
  • TrafficImag evaluates behavior reasoning, image intervention, and conditional video generation using four validity dimensions and requires their conjunction for a successful counterfactual.The dimensions distinguish initial-state correctness, route and behavior validity, interaction consistency, and non-target preservation.Source: Introduction · p. 2
  • TrafficImag’s cross-stage study evaluates reasoners, image editors, and video generators under shared scenarios, using ablations, oracle substitutions, paired tests, and failure analysis.These analyses localize errors along the execution chain.Source: Introduction · p. 3

Related Work

Prior roadside resources support perception and reasoning, while traffic generators and video benchmarks address other tasks; the supplied passages describe no prior protocol for verifying actor-level counterfactual execution in fixed-view roadside video.

Related Work

  • Roadside datasets and traffic VQA benchmarks support perception, forecasting, and semantic reasoning, but do not require generating a modified future from a fixed roadside observation.Rope3D, DAIR-V2X, V2X-Seq, and SUTD-TrafficQA cover detection, cooperative perception, forecasting, or event reasoning.Source: Related Work · p. 3
  • Driving world models and structured traffic generators offer future prediction or controllable multi-agent synthesis, but do not define a protocol for verifying actor-level intervention execution in roadside video.The described methods primarily use ego-view observations or trajectory-space agent states.Source: Related Work · p. 3
  • Existing video benchmarks cover perceptual, compositional, physical, and driving-oriented quality, but do not jointly test legal actor routes, non-target preservation, and matched factual–counterfactual branches.TrafficImag fixes the camera viewpoint, grounds interventions in observed history, and checks identity, route connectivity, interaction order, and non-target preservation.Source: Related Work · p. 3

Dataset

TrafficImag combines broad roadside image and video coverage with a densely annotated temporal subset. Identity-linked tracks support actor-centered history–future samples and trajectory-derived behavior supervision.

Data Collection

  • 9,022 images, 7,043 clips, and a dense subset of 4,348 frames, 63,543 track-linked boxes, and 1,724 identity-consistent tracks make up three annotation layers.The clips come from two fixed-camera sites; the dense subset includes daytime and nighttime frames annotated at 6 fps.Source: Data Collection · p. 4
  • 31,145 actor-centered samples pair 16 observed frames with 9 future frames, about 2.7 seconds of history and 1.5 seconds of future at 6 fps.Overlapping windows retain distinct contexts, and multiple actors from one scene interval yield separate samples.Source: Data Collection · p. 4

Data Annotation

  • Nine trajectory-derived behavior labels cover stationary motion, movement, route trends, slowing or stopping, turns, lane shifts, and starting.Labels may be multi-valued and are benchmark supervision rather than unqualified physical ground truth; automated checks and manual review audit tracks and ambiguous cases.Source: Data Annotation · p. 5

Benchmark

TrafficImag defines counterfactual roadside video generation as producing a future from an observed history, road topology, and actor intervention. Its benchmark factors execution into replaceable stages and tests actor-level programs against topology and four validity dimensions.

Task Definition

  • The task generates an H-step future from observed roadside history, image-plane road topology, and an actor intervention, then evaluates the intervened rollout.The execution chain factors behavior reasoning, image editing, and image-to-video generation so stages can be evaluated independently across models with different internal representations.Source: Task Definition · p. 5

Actor Programs and Topology-Aligned Conditions

  • For each affected actor, the benchmark records behavior, route or goal, longitudinal mode, interaction order, timing, and uncertainty, then validates the program against road topology.Illegal lane-exit pairs are rejected; removed actors cannot remain in the program, and inserted actors need a valid initial lane and heading.Source: Actor Programs and Topology-Aligned… · p. 5

Validity Dimensions

  • A video succeeds only if initial-state, behavioral, relational, and preservation predicates all hold; the rubric tests intervention completion, legal motion, interaction, and non-target stability.Table 3 reports behavior-label F1, editing success, and video validity metrics; it uses 1,500 behavior samples, 60 editing cases with two repeats, and 20 video scenarios with three repeats.Source: Validity Dimensions · p. 6, Table 3 · p. 6

Experiments

The experiments assess behavior reasoning, image editing, and conditional video generation across a shared counterfactual execution chain. Full conditions improve end-to-end success, while oracle studies indicate that rollout execution remains a major source of failure.

Experimental Setup

  • Models are evaluated without task-specific fine-tuning across three research questions covering actor-program reasoning, the execution chain, and sources of end-to-end failure.The study includes open and hosted reasoners, hosted image editors, and three hosted video generators; stage-specific metrics supplement the four-predicate conjunction.Source: Experimental Setup · pp. 6–7

RQ1: Behavior Reasoning

  • 80.4% macro label F1 is the highest reported, achieved by Gemini 2.5 Pro; state-transition labels remain less reliable than turn and stationary-motion labels.Slowing-or-stop reaches 67.8% F1 and start-moving 70.5%, while left- and right-turn F1 reach 85.6% and 87.1%, respectively.Source: RQ1: Behavior Reasoning · p. 7

RQ2: Cross-Stage Counterfactual Execution

  • 55.0% execution success makes Seedance 2.0 the strongest generator among the three evaluated, ahead of Sora 2 Pro at 43.3% and Veo 3.1 at 35.0%.Seedance retains target identity in 83.3% of outputs and reaches 78.3% route success; Sora more often follows an incomplete path, while Veo more often changes unprogrammed actors or loses the route.Source: RQ2: Cross-Stage Counterfactual… · pp. 7–8, Figure 3 · p. 9
  • Removal is easier than insertion: GPT Image 2 records 90.0% removal success and 76.7% insertion success, with other editors showing gaps of 13.3–16.7 points.Insertion additionally couples lane location, heading, scale, illumination, ground contact, and occlusion; outside-mask LPIPS is 0.052–0.081.Source: RQ2: Cross-Stage Counterfactual… · p. 7, Figure 2 · p. 8

RQ3: Bottlenecks, Failure Modes, and Repair

  • 55.0% end-to-end success under Full conditions, up from 23.3% with direct prompting; auditing both program and anchor raises success to 76.7%.One repair raises Full-interface success from 55.0% (33/60) to 61.7% (37/60), while the residual oracle gap is consistent with unresolved condition following, identity maintenance, and long-horizon execution.Source: Table 4 · p. 8, RQ3: Bottlenecks, Failure Modes, and… · pp. 8–9

Discussion

  • The three research questions show why one video-quality score cannot distinguish reasoning, initialization, and rollout errors that affect counterfactual validity.Oracle tests show that auditing the anchor gains more than auditing the program alone, yet auditing both inputs still leaves failures during execution; paired factual-counterfactual tests assess response to the intervention across matched outputs.Source: Discussion · pp. 9–10

Conclusion

TrafficImag benchmarks counterfactual roadside video generation through a staged execution protocol and a densely tracked roadside corpus. Its reported results show improved execution with topology-aligned conditions, while oracle and paired tests identify persistent rollout limitations.

Conclusion

  • TrafficImag decomposes counterfactual generation into behavior reasoning, intervention initialization, conditional video generation, and traffic-aware verification, using four validity dimensions beyond perceptual fluency.The corpus combines a large roadside image and video collection with a densely tracked benchmark subset.Source: Conclusion · p. 10
  • Topology-aligned conditions improve execution, but oracle and paired tests identify persistent limitations in anchor geometry, long-horizon condition following, and coupled multi-actor behavior.The benchmark releases scenario definitions, adapters, predicate-level scores, and failure records for reproducible evaluation and diagnosis.Source: Conclusion · p. 10
  • The benchmark covers two fixed-camera sites and uses normalized image coordinates, so it cannot certify metric speed, clearance, or collision risk.It is intended for evaluation, visualization, and failure analysis, not signal control, safety certification, or replacement of calibrated traffic simulation.Source: Conclusion · p. 10

A Dataset Statistics and Evaluation Subsets

The appendix describes separate image, temporal, and evaluation subsets, with actor-centered samples and multi-valued behavior labels. Its accounting distinguishes dataset quantities and benchmark cases rather than treating them as directly comparable.

Dataset Layers and Class Composition

  • Image-bank boxes and dense-subset tracks are separate statistics intended to characterize their subsets independently, not to be compared directly.Table 5 summarizes class distributions across the image-annotation bank and dense temporal subset.Source: Dataset Layers and Class Composition · p. 13, Table 5 · p. 13

Behavior-Label Distribution

  • Behavior labels are multi-valued across 31,145 samples, including 18,871 stationary, 8,345 left-turn, and 417 start-moving positives.Longitudinal states are usually exclusive, while lateral shifts may co-occur; 25 transition windows retain adjacent states.Source: Behavior-Label Distribution · p. 13, Figure 4 · p. 14

Evaluation Subsets and Accounting

  • The reasoning, editing, and video suites use distinct evaluation subsets; the video suite has 20 scenarios with three repeats, and failed calls remain in the denominator.The editing suite contains 30 removal and 30 insertion cases; Table 6 separates benchmark cases, repeats, and repair calls.Source: Evaluation Subsets and Accounting · p. 14, Table 6 · p. 14

B Benchmark Condition Construction

The benchmark converts actor-level interventions into shared conditions for video generation and evaluates outputs against four traffic-validity predicates. Its protocol specifies compilation, rating, model records, serialization, and retry accounting.

Actor-Program Schema

  • Each affected actor is represented by a program record whose fields and consistency checks are listed in Table 7 and used to generate model-specific prompts.The passage identifies the record as Equation (2), without enumerating its fields in the supplied text.Source: Actor-Program Schema · p. 14, Table 7 · p. 15

Topology-Aware Compilation

  • The compiler maps topology, the actor program, and the edited anchor into visual, semantic, target-binding, route-and-anchor, and timing-and-invariant conditions.It rejects invalid lane-to-exit pairs and contradictory or out-of-range constraints; normalized image coordinates support legal-route and relative-motion checks, not metric checks.Source: Topology-Aware Compilation · p. 15, Figure 5 · p. 15

Four Binary Validity Predicates

  • End-to-end success requires all four binary predicates for the intervention, target behavior, interaction, and preservation; texture flicker is recorded separately.Three blind raters inspect the edited anchor, bindings, route overlay, video, and preserve-set tracks.Source: Four Binary Validity Predicates · p. 16, Table 8 · p. 16

Rater and Statistical Protocol

  • Video confidence intervals use scenario-level cluster bootstrap, keeping repeats and paired branches from the same scenario together; failed calls remain in the denominator.Direct and Full conditions are paired by scenario, while behavior reasoning uses a clip-level bootstrap.Source: Rater and Statistical Protocol · p. 16

Model Identifiers

  • Execution records preserve model and request metadata, settings, available seeds, latency, returned dimensions, status, and error codes; benchmark models are not fine-tuned.Tables 9 and 10 list model identifiers and generation settings, with hosted access dates retained.Source: Model Identifiers · pp. 16–17, Table 9 · p. 17, Table 10 · p. 17

Common Semantic Serialization

  • Adapters serialize common semantic fields, retaining image-coordinate references and lane, entry, conflict-zone, and exit descriptions while adapting to service interfaces.Scenario constraints may prohibit overtaking, lane changes, identity swaps, or preserve-set motion.Source: Common Semantic Serialization · p. 17, Table 11 · p. 17

Retry and Failure Policy

  • Automatic retries cover only requests returning no media; incorrect outputs are not resubmitted, while the repair study permits one semantic retry after scoring.Both attempts are retained.Source: Retry and Failure Policy · pp. 17–18

E Metric Definitions and Detailed Behavior-Reasoning Audit

The section defines behavior-reasoning and stage-specific metrics, then describes a 1,500-sample audit and its scoring scope. The audit excludes ambiguity-margin samples from longitudinal scoring while retaining their route-scale labels.

Metric Definitions

  • Macro label F1 is the unweighted mean of nine label-wise F1 scores, while parse success measures schema-conforming responses without manual correction.An unparsable response contributes zero to the fixed-schema metrics.Source: Metric Definitions · p. 18
  • Continuous image-editing and video-generation diagnostics accompany binary decisions, but do not alter them or override a failed traffic predicate.Per-output edit-preservation and perceptual diagnostics remain available for diagnosis even when they are not aggregated in the main tables.Source: Metric Definitions · p. 18, Table 12 · p. 18

Aggregate and Class-Conditional Reasoning Audit

F Component, Oracle, and Paired Results

The section specifies image-editing evaluation, matched oracle substitutions, paired factual-counterfactual measures, a one-step repair policy, and traceability records.

Image-Editing Components

  • Image-editing results cover 60 cases—30 removals and 30 insertions—repeated twice per editor, with geometry assessed on insertion outputs.Outside-mask LPIPS is reported separately from completion, geometry, and non-target preservation to diagnose unintended changes beyond the edited region.Source: Table 15 · p. 19, Image-Editing Components · p. 19

Oracle Substitutions

  • Oracle substitutions compare a fixed generator across 60 matched scenario and repeat slots per input setting, and their gains need not be additive.Only the named upstream component changes in each setting; the scenario, generator, and repeat slot remain fixed.Source: Oracle Substitutions · p. 20, Table 16 · p. 20

Paired Factual and Counterfactual Response

  • Paired factual-counterfactual response measures use all matched pairs for pair validity, but compute IRC, order change, and median time differences only for valid pairs.Time differences are counterfactual minus factual; removal is expected to advance entry or reduce waiting, while insertion is expected to have the opposite effect.Source: Table 17 · p. 20, Paired Factual and Counterfactual… · p. 20

G Failure Taxonomy and One-Step Repair

  • One localizable failure may receive one corrective condition and retry, while coupled failures are not repaired.The Full + One Repair result comprises 60 final decisions and 27 additional calls; Table 18 defines categories and permitted responses rather than reporting frequencies.Source: G Failure Taxonomy and One-Step Repair · pp. 20–21, Table 18 · p. 21

H Scenario Manifest and Release Records

  • Scenario, execution, score, and relation records are stored separately, and retries preserve first-pass records rather than overwriting them.Annotations, topology, scenarios, and adapters have separate version fields; paths are relative, identifying metadata are removed, and available media checksums are stored.Source: H Scenario Manifest and Release Records · p. 21, Table 19 · p. 21

I Additional Counterfactual Example

An additional insertion example specifies two vehicles sharing an image-plane lane and contrasts their generated rollouts. This case is excluded from aggregate results.

I Additional Counterfactual Example

  • The example uses a GPT Image 2 insertion anchor, which is evaluated for lane-consistent heading, scale, illumination, occlusion, and road contact.The corresponding removal anchor is evaluated for road reconstruction and preservation.Source: I Additional Counterfactual Example · p. 22, Figure 6 · p. 22
  • The eight-second rollout instructs the inserted sedan T1 to follow the leading SUV T2 toward the upper-left exit, maintaining a visible non-overlapping gap.T2 moves first, and the figure marks T1 and T2 with red and cyan boxes, respectively.Source: Figure 7 · p. 23
  • The rollout uses the supplied image as time zero, starts T1 0.3–0.5 seconds after T2, and specifies anchors every two seconds.The camera, road, school bus, and unboxed vehicles are required to remain fixed; T1 must not overtake or overlap T2.Source: I Additional Counterfactual Example · p. 23
  • In the insertion example, Seedance follows the requested vehicle order and route; Sora leaves both vehicles nearly stationary, while Veo adds vehicles and fails preservation.The case is excluded from all aggregate results.Source: I Additional Counterfactual Example · pp. 22–23
Loading 2609.30722v1…