PaperlayerView exact revision on arXiv

Source-linked AI summary

StateBridge: Phase- and Track-Grounded Reasoning for Synthetic-to-Real Traffic Video Captioning and VQA

Blessing Agyei Kyem, Richard Dzinyela, Joshua Kofi Asamoah, Eugene Denteh, Andrews Danyo, Solomon Owusu-Ansah, Dickson Gyimah Asiedu, Armstrong Aboah

arXiv:2610.31898v1cs.CV

TL;DR

Traffic safety captioning and VQA must transfer from synthetic training videos to real traffic despite differences in visual appearance and human-behavior cues. StateBridge grounds model outputs in phase, role, track, and checklist evidence and combines them through conservative caption fusion and VQA consensus. On AI City Challenge 2026 Track 2, it achieves an official S2 score of 55.4679 using synthetic training data.

  • Problem

    Synthetic-only adaptation must support traffic captioning and VQA on real footage, where appearance cues may transfer less reliably than scene structure and independent outputs can conflict.

    Source: §1 · pp. 2–3

  • Method

    StateBridge aligns clips by phase and actor role, summarizes tracks, fine-tunes vision-language adapters on synthetic examples, and combines outputs with fact-locked caption fusion and evidence-diverse VQA consensus.

    Source: §1 · p. 3, §4.1 · p. 6, §4.4 · p. 8

  • Results

    55.4679 official S2 score on AI City Challenge 2026 Track 2 under the synthetic-to-real protocol.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The results support transferring stable traffic structure—phase, role, geometry, and checklist semantics—rather than relying only on appearance-level adaptation.

    Source: §6 · p. 14

    Training uses synthetic source examples only; real target videos are used at inference, without target caption or answer supervision.

    Source: §3.2 · p. 5, §4.4 · p. 8

Abstract

from arXiv · show

Traffic safety videos provide rich evidence for understanding interactions between pedestrians and vehicles, but synthetic-to-real evaluation makes this setting challenging because models must learn from simulated scenes and generalize to real-world traffic. In this work, we present StateBridge, a traffic video-language framework designed to operate under this synthetic-only training protocol. Our approach uses phase-aware clip construction, pedestrian and vehicle track summaries, and structured prompts to ground vision-language models in stable traffic evidence rather than appearance alone. We fine-tune public vision-language models on synthetic captioning and visual question answering examples, then combine their outputs through fact-locked caption fusion and evidence-diverse VQA consensus to improve consistency across actor roles, behavioral phases, and question types. StateBridge achieves an official S2 score of 55.4679 on AI City Challenge 2026 Track 2 using only synthetic training data. These results demonstrate the effectiveness of structured evidence transfer for traffic safety captioning and visual question answering.

1 Introduction

The task requires explaining pedestrian–vehicle interactions across time, but synthetic-only training must transfer to real traffic despite an uneven gap in what simulation reproduces reliably. StateBridge grounds captioning and VQA in phase, role, track, and checklist evidence, then conservatively combines model outputs.

1 Introduction

  • The challenge requires explaining safety-critical pedestrian–vehicle interactions over time, including actor locations, attention, motion, and the surrounding road environment.Track 2 trains on synthetic Digital Twin videos and evaluates on real-world WTS footage.Source: §1 · p. 2
  • Digital Twin data preserve road layout, viewpoint, event timing, tracks, and annotation schema, while fine-grained behavior and appearance cues are harder to simulate faithfully.Synthetic supervision can teach output style while leaving models reliant on visual evidence that is unreliable at test time.Source: §1 · p. 2
  • Phase-level, actor-specific captions and multiple-choice questions can contradict one another or alter correct distance, speed, and phase-specific action details when generated independently.The task evaluates multiple projections of the same traffic state.Source: §1 · p. 2
  • StateBridge combines phase- and track-grounded evidence with fact-locked caption fusion and evidence-diverse VQA consensus for synthetic-only traffic captioning and VQA.The framework aligns clips by behavioral phase and actor role, summarizes pedestrian and vehicle tracks, and connects captioning and VQA through shared structured evidence.Source: §1 · p. 3
  • The paper contributes an asymmetric structured-state formulation, normalized temporal track summaries, fact-preserving caption selection, and evidence-diverse VQA consensus in a synthetic-only pipeline.Caption edits are accepted only when they preserve supported attributes, relations, context, and phase-specific behavior.Source: §1 · p. 3

2 Related Work

Prior work develops traffic video-language tasks, grounded multimodal reasoning, and structured caption generation, with many traffic systems studied in real-data settings. StateBridge applies these lines to synthetic-to-real transfer using phase, role, geometry, tracks, and checklist semantics as its evidence interface.

2.1 Traffic Video-Language Understanding

  • Traffic video-language work progresses from generic captioning and VideoQA to driving-language datasets and pedestrian-centered safety tasks, but primarily studies real-data WTS settings.StateBridge instead targets synthetic-to-real transfer through phase, role, geometry, and checklist semantics.Source: §2.1 · pp. 3–4

2.2 Grounded Multimodal Reasoning

  • Grounded multimodal methods link language to objects, regions, relations, and temporal moments; StateBridge uses challenge-provided boxes and phases to summarize tracks instead of a target-domain detector.Prior grounding work includes detection and tracking, scene-graph reasoning, phrase grounding, coordinate-aware dialogue, and temporal localization.Source: §2.2 · p. 4

2.3 Structured and Consistent Caption Generation

  • Structured captioning seeks fluent language that preserves semantic fields, and StateBridge accepts phase-role caption fields only when they remain consistent with shared traffic state.Prior approaches include metric-aligned training, explicit detection, scene-graph guidance, controllable generation, and AI City caption refinement.Source: §2.3 · pp. 4–5

3 Task and Problem Setup

Track 2 evaluates phase-level pedestrian and vehicle captions and multiple-choice VQA on real WTS videos after synthetic training. The paper models the transferable task state as structured semantics whose meaning remains stable across domains despite changes in video rendering.

3.1 Task Definition

  • Track 2 trains on labeled SynWTS Digital Twin videos and tests on real WTS videos without real-world supervision, requiring phase-level actor captions and multiple-choice VQA.The official S2 score combines caption metrics with VQA accuracy; test inputs include phase metadata, questions, answer options, and challenge annotations when available, but no target captions or answers.Source: §3.1 · p. 5

3.2 Synthetic-to-Real Structured-State Transfer

  • The task state comprises phase- and role-specific location, attention, and behavior plus shared context, which captions verbalize and VQA items query.The domain shift changes video distributions while the annotation schemas for the same state and outputs remain aligned; StateBridge estimates state support from official target inputs without target labels or real WTS training videos.Source: §3.2 · pp. 5–6

4 Method

StateBridge grounds traffic captions and VQA in phase-, role-, and track-based evidence, then uses task-specific controllers to select outputs. Its pipeline combines phase-aware clips, normalized track summaries, synthetic-trained adapters, fact-locked caption selection, and evidence-diverse VQA voting.

4.1 Framework Overview

  • StateBridge constructs phase- and role-grounded evidence from aligned video views and normalized actor tracks, then controllers select outputs using different rules for captions and VQA.Caption fields pass fact-preserving gates, while multiple-choice answers use evidence-diverse consensus.Source: §4.1 · p. 6

4.2 Phase-Aware Evidence Construction

  • Phase-aware construction clips each view to the event phase, while environment questions can retain broader scene context when they are not tied to a short behavior interval.Matching training instances to evaluation timing reduces leakage from neighboring behavioral phases.Source: §4.2 · pp. 6–7

4.3 Track-Grounded Prompting

  • Track prompts encode each actor’s first, middle, last, and mean boxes on a normalized grid, exposing coarse motion and relative geometry independent of video resolution.Optional overlays and interaction crops provide alternative evidence streams rather than replacing raw phase clips.Source: §4.3 · p. 7

4.4 Synthetic VLM Adaptation

  • Public multimodal models are adapted with LoRA on caption, VQA, or joint synthetic examples, with variants using serialized boxes, overlays, crops, or combined grounding views.All supervised examples are synthetic; target videos are used only at inference.Source: §4.4 · pp. 7–8

4.5 Fact-Locked Caption Fusion

  • Caption fusion scores alternative role-phase fields against trusted fields and accepts an edit only when it improves the proxy score and passes overlap and fact-consistency gates.Scoring rewards overlap, four-gram preservation, supported-fact agreement, and compatible length while penalizing repetition, contradictions, artifacts, and incomplete endings.Source: §4.5 · pp. 8–9

4.6 Evidence-Diverse VQA Consensus

  • VQA combines predictions from complementary synthetic-trained adapters by weighted option voting; the final submission uses equal weights and the joint adapter as a tie fallback.This prevents a weaker grounding view from changing a high-confidence answer by itself.Source: §4.6 · p. 9

5 Experiments

The experiments evaluate synthetic-only training and real-world inference, compare output-fusion variants, and illustrate phase- and track-grounded predictions. The final system ranks 8th in Track 2, while the discussion highlights gains in some caption metrics alongside a possible CIDEr trade-off.

5.1 Dataset and Protocol

  • Training uses only SynWTS synthetic data, producing 14,198 supervised rows; real WTS videos are used only for inference.The training set contains 2,465 phase-level caption examples and 11,733 VQA examples; test submission assembly produces 798 caption rows and 19,624 VQA rows.Source: §5.1 · p. 9

5.2 Implementation Details

  • Caption and VQA adapters use distinct task formats, while evidence variants alter visual grounding and fusion rules constrain which candidate outputs are accepted.The final VQA answer uses equal-vote consensus with the strongest joint Qwen3-VL adapter breaking ties; final caption fusion has no fixed edit budget.Source: §5.2 · p. 10

5.3 Official Challenge Results

  • StateBridge achieves the best S2 among the verified synthetic-only variants and ranks 8th in AI City Challenge Track 2.The compared challenge submissions use the same evidence-diverse VQA consensus, isolating caption-fusion behavior.Source: §5.3 · p. 10, Table 1 · p. 11

5.4 Ablation Study

  • Table 3 comparisons indicate that small, fact-locked caption edits are more reliable than broad rewriting or aggressive field replacement.The ablation rows vary VQA consensus and caption-fusion strategies against a joint Qwen3-VL baseline.Source: §5.4 · p. 11, Table 3 · p. 12

5.5 Qualitative Results

  • Real public-test predictions show role-specific captions and VQA answers across five behavioral phases, while examples illustrate caption repair and evidence-diverse consensus.The figure examples cover overhead and vehicle views, a dangling caption ending, and a synthetic-validation VQA example.Source: §5.5 · p. 11, Figure 3 · p. 12, Figure 4 · p. 13

5.6 Discussion

  • Fact-locked fusion improves METEOR and ROUGE-L over the direct caption source, but accepted edits can reduce CIDEr when they change frequent reference-style phrases.The discussion motivates field-level replacement rather than full-caption rewriting and says VQA answers change only when complementary streams provide enough support.Source: §5.6 · p. 14

6 Conclusion

StateBridge addresses synthetic-to-real traffic safety captioning and VQA by combining phase-aware evidence, track-grounded adaptation, and conservative output fusion. On AI City Challenge 2026 Track 2, it achieved an official S2 score of 55.4679 using the synthetic-to-real protocol.

  • StateBridge combines phase-aware evidence construction, track-grounded VLM adaptation, fact-locked caption selection, and evidence-diverse VQA consensus.Source: §6 · p. 14
  • 55.4679 was StateBridge’s official S2 score on AI City Challenge 2026 Track 2 under the synthetic-to-real protocol.Source: §6 · p. 14
  • The authors conclude that transferring stable traffic structure offers value beyond relying only on appearance-level adaptation.They describe StateBridge as a practical foundation when real-world supervision is unavailable.Source: §6 · p. 14

Related papers

Loading 2610.31898v1…