Source-linked AI summary
SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events
Li Xu, He Huang, Jun Liu
TL;DR
Traffic-event reasoning matters for intelligent-transportation applications, while existing video-QA approaches often emphasize accuracy and use fixed computation pipelines. This paper introduces SUTD-TrafficQA’s six causal reasoning tasks and Eclipse’s adaptive video reasoning, reporting superior performance with significant computation efficiency.
Problem
Video reasoning for traffic applications must balance accuracy and efficiency, but existing video-QA methods often use fixed computation pipelines rather than adapting resource allocation to reasoning structure.
Source: §1 · p. 1Method
The paper constructs SUTD-TrafficQA with six traffic-reasoning tasks and diverse videos, and proposes Eclipse to adaptively select glimpse frames and computation granularity.
Source: §1 · p. 1Results
Eclipse achieves superior performance with significant computation efficiency, and is reported to reach state-of-the-art reasoning accuracy with significantly improved computation efficiency.
Source: §6 · p. 8, §5.1 · p. 8Takeaways & Limitations
SUTD-TrafficQA provides a benchmark for video reasoning over complex traffic events, including varied causal-reasoning tasks and real-world videos.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Traffic event cognition and reasoning in videos is an important task that has a wide range of applications in intelligent transportation, assisted driving, and autonomous vehicles. In this paper, we create a novel dataset, SUTD-TrafficQA (Traffic Question Answering), which takes the form of video QA based on the collected 10,080 in-the-wild videos and annotated 62,535 QA pairs, for benchmarking the cognitive capability of causal inference and event understanding models in complex traffic scenarios. Specifically, we propose 6 challenging reasoning tasks corresponding to various traffic scenarios, so as to evaluate the reasoning capability over different kinds of complex yet practical traffic events. Moreover, we propose Eclipse, a novel Efficient glimpse network via dynamic inference, in order to achieve computation-efficient and reliable video reasoning. The experiments show that our method achieves superior performance while reducing the computation cost significantly. The project page: https://github.com/SUTDCV/SUTD-TrafficQA.
1. Introduction
SUTD-TrafficQA addresses the need for challenging benchmarks for causal reasoning in complex traffic events. The introduction presents six reasoning tasks and Eclipse, which adapts video computation through dynamic glimpsing.
1. Introduction
- The paper introduces SUTD-TrafficQA as a benchmark for causal reasoning in complex traffic scenarios, motivated by intelligent-transportation applications and the importance of comprehensive benchmarks.The introduction identifies assisted driving, violation detection, and congestion forecasting as example applications.Source: §1 · p. 1
- Six tasks require models to forecast future events, infer past situations, explain accident causes, and provide preventive advice across varied traffic scenarios.The tasks are designed to explore complex causal structures in traffic-event inference.Source: §1 · p. 1
- Video question answering is used to construct the dataset because answering requires logical reasoning and spatio-temporal cognition of events.Source: §1 · p. 1
- Online collection and offline capture provide videos spanning varied traffic events, road agents, surroundings, and recording perspectives.Source: §1 · p. 1
- Existing video-QA methods emphasize accuracy and often use fixed computation pipelines rather than allocating resources according to event-reasoning structure.The introduction notes that computation and energy budgets can be constrained in applications such as assisted driving.Source: §1 · p. 1
- Eclipse adaptively selects frames to glimpse and computation granularity, skipping irrelevant segments; joint Gumbel-Softmax optimizes these discrete decisions end to end.The paper presents simultaneous adaptive frame localization and feature-granularity determination as its novel dynamic reasoning process.Source: §1 · p. 2
2. Related Works
The related work distinguishes SUTD-TrafficQA from prior traffic datasets and video-QA benchmarks by its focus on causal reasoning in complex real-world traffic. It presents Eclipse as adaptive video reasoning designed for computation efficiency.
Intelligent Transportation
- Unlike existing traffic datasets and methods, this work investigates causal reasoning with video QA over varied traffic scenarios and proposes SUTD-TrafficQA with Eclipse.Source: §2 · p. 2
Video QA Datasets
- MovieQA and TVQA use movie and television videos, while CLEVRER studies collisions among simple objects in controlled synthetic videos; SUTD-TrafficQA targets complex traffic in the wild.It introduces six tasks based on diverse real-world traffic videos.Source: §2 · p. 2
Video QA Methods
- Eclipse differs from existing video-QA methods by learning an adaptive glimpse policy for reasoning with computation efficiency.Source: §2 · p. 2
Computation-Efficient Models
- Eclipse uses text and visual context to guide frame selection and computation granularity in dynamic video-QA inference.The passage situates this approach among computation-efficiency strategies such as pruning, sparsification, quantization, and dynamic routing.Source: §2 · p. 3
3. SUTD-TrafficQA Dataset
SUTD-TrafficQA contains 10,080 traffic-scene videos and 62,535 question-answer pairs spanning six reasoning tasks, with videos collected across varied real-world conditions.
3. SUTD-TrafficQA Dataset
- The six tasks cover basic scene and event understanding, forecasting future events, reasoning backward from a clip, and inferring hypothetical outcomes.Counterfactual questions ask what would follow under a condition that does not occur in the video.Source: §3 · p. 3
- Introspection asks for preventive actions, while attribution asks models to explain traffic-event causes; questions use two to twelve candidate answers.Candidate answers are sampled to balance the dataset and limit repeated correct answers within each task.Source: §3 · p. 3
3.1. QA Collection
- Online harvesting and offline capture yielded 10,080 diverse clips; annotators wrote at least three questions per video, with weekly annotation checks.The clips vary in weather, time, road conditions, traffic events, camera perspective, and length.Source: §3.1 · pp. 3–4
3.2. Dataset Statistics
- Question length averages 8.6 words, and the dataset spans varied question beginnings and reasoning-task categories.Figure 2 presents question-length, question-type, and reasoning-task distributions.Source: §3.2 · p. 4, Figure 2 · p. 4
4. Eclipse Network
Eclipse replaces fixed full-video processing with dynamic reasoning that selects frames and feature granularity using the QA pair, current frame, and historical cues. Its modules predict answers and decide when to stop, with training objectives and joint Gumbel-Softmax addressing the discrete frame and granularity choices.
4. Eclipse Network
- Eclipse dynamically selects video frames and feature granularity instead of applying a fixed computation architecture to the whole video, while producing answer predictions.The architecture uses QA information, the currently selected frame, and historical cues to guide recurring reasoning.Source: §4 · p. 4
- At each inference step, the Interaction Module fuses QA information, the selected frame, and historical cues into a representation for downstream reasoning.It combines a context-query sub-module with an interaction LSTM.Source: §4 · p. 5
- The Glimpse-Determination Module chooses the next frame and whether to compute coarse or fine features; it can skip both forward and backward through the video.Coarse features use a lightweight CNN, while fine-grained features use a more representative, computation-heavier CNN.Source: §4 · p. 5
- The glimpse objective encourages selected frames to increase the correct-answer margin across steps and penalizes computation-heavy fine-feature extraction.The feature penalty applies when the policy chooses fine features.Source: §4 · p. 6
- The Exit-Policy estimates remaining information gain from answer-margin differences and can stop reasoning early when that gain is small.Its training labels use the gap between the current margin and the final-step margin.Source: §4 · p. 6
- Joint Gumbel-Softmax makes the discrete frame-selection and granularity decisions jointly differentiable; exit decisions are used at testing, not training.Training uses a fixed number of reasoning steps, while testing can stop at the Exit-Policy's chosen step.Source: §4 · pp. 6–7
5. Experiments
The experiments evaluate Eclipse against text-only and video-input baselines in binary and four-choice setups. Eclipse achieves state-of-the-art reasoning accuracy with significantly improved computation efficiency, while ablations examine its adaptive policies.
5. Experiments
- Experiments use binary answer-correctness prediction and four-choice selection, comparing text-only baselines with video-and-text methods.The paper also reports 16 ms per testing video on an Nvidia RTX 2080Ti GPU.Source: §5 · p. 7
5.1. Results and Analysis
- Eclipse achieves state-of-the-art reasoning accuracy with significantly improved computation efficiency; video-input models also outperform text-only baselines that perform nearly like random choice.The reported discrepancy between neural-network and human accuracy indicates the dataset remains challenging.Source: §5.1 · pp. 7–8, Table 2 · p. 8
- Adaptive granularity achieves nearly the same accuracy as fine features alone at much lower computation cost, while outperforming coarse features alone in reasoning accuracy.The comparison evaluates the granularity-policy ablation.Source: Table 3 · p. 8, §5.1 · p. 8
- Removing the skip-policy for uniform frame sampling loses to Eclipse, which achieves the best result at small computation cost.The reported ablation attributes reduced computation to selecting useful frames for dynamic reasoning.Source: Table 4 · p. 8, §5.1 · p. 8
- The exit-policy yields Eclipse's best accuracy-to-computation ratio compared with inference through the final step.A qualitative example shows dynamic frame selection and larger computation budgets for relevant frames; frame-location distributions are shown for the first three steps.Source: Table 5 · p. 8, §5.1 · p. 8, Figure 4 · p. 8
6. Conclusion
The conclusion presents SUTD-TrafficQA as a six-task video-QA dataset for traffic-event reasoning and Eclipse as a network that uses dynamic glimpse and adaptive exit policies. The authors report superior performance with significant computation efficiency.
6. Conclusion
- SUTD-TrafficQA is a video-QA dataset focused on reasoning over traffic events, with six tasks requiring varied levels of causal reasoning.Source: §6 · p. 8
- Eclipse is proposed for video QA, using a dynamic glimpse policy and an adaptive exit policy.Source: §6 · p. 8
- The authors report that Eclipse achieves superior performance with significant computation efficiency.Source: §6 · p. 8