Source-linked AI summary
TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding
Yuqiang Lin, Yan Shi, Sam Lockyer, Harish Tayyar Madabushi, Adrian Evans, Wenbin Li, Yinhai Wang, Nic Zhang
TL;DR
Traffic anomaly understanding requires detecting, reasoning about, and explaining anomalous transportation events despite query-dependent and sparse evidence. TAU-Agent uses an agentic retrieval-augmented pipeline with captioning and open-vocabulary tracking tools, followed by a fine-tuned VLM. It ranks second on Track 3, twelfth on Track 7, and fifth on Track 8.
Problem
Traffic anomaly understanding requires systems to detect, reason about, and explain anomalous events across varied question types, but relevant evidence is query-dependent and sparse in space and time.
Method
TAU-Agent uses a main retrieval agent to orchestrate Video Captioning and Open-Vocabulary Tracking Tools, then integrates selected evidence and sampled frames with a supervised fine-tuned VLM.
Results
TAU-Agent ranks second on in-domain Track 3, twelfth on out-of-domain Track 7, and fifth on out-of-domain Track 8.
Takeaways & Limitations
The reported rankings provide evidence that TAU-Agent performs in-domain and can generalize across different traffic-video domains and task formulations.
Takeaways & Limitations
Performance on FETV remains below the highest-ranked methods, particularly for structured attribute prediction, with limited adaptation to fisheye imagery and structured JSON output.
Abstract
from arXiv · showhide
Traffic Anomaly Understanding (TAU) requires models and systems to detect, reason about, and explain anomalous events in transportation videos. To address this challenge, we propose TAU-Agent, an agentic retrieval-augmented framework for traffic anomaly understanding. Given a task query, a central retrieval agent orchestrates two visual perception tools, namely a Video Captioning Tool and an Open-Vocabulary Tracking Tool, to retrieve and select query-relevant evidence, including captions, temporal intervals, and object trajectories. The selected evidence, together with sampled video frames and the input query, is provided to a supervised fine-tuned vision-language model for final reasoning and answer generation. We evaluate TAU-Agent on both the in-domain and the out-of-domain benchmarks from the AI City Challenge 2026. TAU-Agent achieves scores of 0.6779 on Track 3, 0.3998 on Track 7, and 67.9275 on Track 8, ranking second, twelfth, and fifth, respectively. Code is available at: https://github.com/siri-rouser/TAU-Agent.
1 Introduction
Traffic anomaly understanding requires systems to detect, reason about, and explain anomalous transportation events across diverse question types. TAU-Agent addresses query-dependent, sparse evidence needs with a multi-stage retrieval-augmented pipeline.
- Traffic anomaly understanding requires detecting, reasoning about, and explaining anomalous events from multiple perspectives.The task includes binary, multiple-choice, temporal-grounding, and free-text questions.
- Query-dependent questions may require different temporal segments, objects, and evidence within the same video.A video can contain multiple anomalies and normal events, while each query may target one specific event, object, interaction, or context.
- Sparse spatial and temporal evidence makes uniform sampling liable to miss critical information or introduce redundant content.Relevant objects may occupy small regions, and queried events may occur briefly in long videos.
- TAU-Agent decomposes the task through a main agent that adaptively retrieves query-relevant evidence using video captioning and open-vocabulary tracking tools.The retrieved evidence is integrated with the question and video source by a supervised fine-tuned VLM.
2 Related Works
Prior work has expanded video anomaly detection toward language-assisted detection, multitask understanding, reasoning-centric methods, and agent-based retrieval. These approaches increasingly distribute perception and reasoning across specialized processes or agents.
- Video anomaly understanding extends anomaly scoring and temporal localization to describing events and reasoning about temporal, spatial, and causal context.This broader capability is especially relevant to transportation videos involving evolving interactions among road users.
- Language-assisted methods use captions, scene-specific rules, event-level modeling, or prompt refinement to improve anomaly analysis.Examples include LAVAD, AnomalyRuler, EventVAD, and PrismVAU.
- Multitask VAU methods adapt multimodal language models with anomaly-oriented instruction data for localization, description, and question answering.The passage lists VAD-R1, VAD-LLaMA, Holmes-VAD, HAWK, CUVA, Holmes-VAU, and TAU-R1.
- TAU-R1 is identified as the only reviewed method specifically evaluated with promising results in the transportation domain.
- Agent-based systems iteratively plan, retrieve video segments, invoke perception tools, and refine predictions for reasoning-intensive video understanding.Single-agent systems use shot retrieval or coarse-to-fine temporal search, while multi-agent systems distribute perception and reasoning roles.
3 Methodology
TAU-Agent uses a retrieval-augmented main agent to interpret queries, retrieve captions and object trajectories, and select relevant frames before downstream VLM reasoning. Optional cross-question context extends retrieval for related Track 3 questions.
- Framework Overview: The main agent interprets the question, invokes captioning, conditionally invokes tracking, and selects relevant evidence and frame ranges.The selected captions, trajectories, and frame range are passed with the question and sampled video to the question-answering VLM.
- Video Captioning Tool: The Video Captioning Tool provides temporally localized event captions, a chronological video summary, and global scene context.It uses uniformly sampled frames to generate scene-level description and arranges captions chronologically.
- Open-Vocabulary Tracking Tool: The Open-Vocabulary Tracking Tool supplies object-centric spatiotemporal information relevant to the query.The framework uses a hybrid detection and tracking pipeline for traffic-related and non-COCO open-vocabulary queries.
- Main Agent Workflow: The main workflow jointly reasons over the question and retrieved evidence to refine frame ranges, select captions and tracks, and score relevance.Additional tool calls are possible when evidence remains insufficient or ambiguous.
- Cross-Question Context Agent: An optional Cross-Question Context Agent retrieves complementary factual, potential, and related-question evidence for interrelated Track 3 questions.The paper limits this capability to benchmarks where multiple related questions share a video.
7. Evidence Output
The evidence-output workflow converts a query into selected temporal, textual, and object-level evidence for downstream VLM reasoning. The VLM combines retrieved evidence with sampled frames and task-specific training signals to produce formatted answers.
- The agent analyzes the question to identify referenced events, objects, interactions, anomalies, and temporal context.
- Video captioning provides high-level video understanding and candidate caption segments for query-focused retrieval.
- Object-level evidence is retrieved through open-vocabulary tracking when the query requires it.
- The agent jointly reasons over captions and object tracks to refine the frame range, select evidence, and assign relevance scores.
- Additional tool calls are performed when retrieved evidence remains insufficient or ambiguous.
- The downstream VLM receives the query-relevant frame range, caption segments, and object tracks.
- The question-answering VLM uses slow-fast frame sampling and retrieved textual evidence to generate the final answer.The full video preserves temporal context, while the relevant range is sampled more densely for fine-grained visual information.
- Parameter-efficient LoRA fine-tuning jointly trains on questions, sampled frames, retrieved evidence, task prompts, reasoning traces, and final answers.This aligns responses with diverse reasoning patterns and target answer formats.
4 Experiments
TAU-Agent is evaluated on in-domain TAR and out-of-domain FETV and PSI-VQA benchmarks using benchmark-specific preprocessing and post-processing. It ranks second on TAR, twelfth on FETV, and fifth on PSI-VQA.
- Benchmarks: TAU-Agent evaluates on TAR, FETV, and PSI-VQA, covering in-domain and out-of-domain traffic-video benchmarks.TAR is in-domain, while FETV and PSI-VQA introduce different visual domains and task formulations.
- TAR Test: TAR Test contains 80 videos and ten tasks, evaluated with accuracy, BERTScore F1, and mIoU, with temporal localization excluded from the overall score.The tasks include verification, question answering, scene description, summarization, temporal localization, causal linkage, and event description.
- TAR Test: 0.6779 mean score ranks TAU-Agent second on TAR, only 0.0009 below the top-ranked entry.It also achieves the highest listed scores on causal linkage, temporal description, and video summarization, while matching the best BCQ and MCQ results.
- FETV: FETV preprocessing consolidates 12 attributes into one JSON-formatted question and retrieves captions and object tracks before VLM prediction.The constructed question, original video, and retrieved evidence are jointly passed to the fine-tuned question-answering VLM.
- FETV: 0.3998 overall score ranks TAU-Agent twelfth on FETV, comprising description 0.3513 and categorical mean 0.4484.FETV uses fisheye-camera videos and requires prediction of 12 structured attributes plus free-text output.
- PSI-VQA: 67.9275 overall score ranks TAU-Agent fifth on PSI-VQA, with Open QA Cue-F1 of 0.7791, highest among listed submissions.The Open QA score is 0.1117 higher than the second-best result; cross-question context is retained for Open QA but treated selectively elsewhere.
5 Conclusion
TAU-Agent coordinates visual perception tools to retrieve query-relevant evidence for traffic anomaly understanding. It ranks second on Track 3, twelfth on Track 7, and fifth on Track 8, with future work targeting streaming and real-time understanding.
- Conclusion: TAU-Agent coordinates visual perception tools to retrieve query-relevant evidence for traffic anomaly understanding.The framework is described as agentic and retrieval-augmented.
- Conclusion: TAU-Agent ranks second on Track 3, twelfth on Track 7, and fifth on Track 8.These results provide evidence of in-domain performance and generalization across traffic-video domains and task formulations.
- Conclusion: Future work will extend TAU-Agent to streaming and real-time video understanding for more efficient deployment in transportation scenarios.