PaperlayerView exact revision on arXiv

Source-linked AI summary

When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis

Ruixuan Zhang, Beichen Wang, Juexiao Zhang, Zilin Bian, Chen Feng, Kaan Ozbay

arXiv:2501.10604v1cs.CVcs.AIcs.CL

TL;DR

Traffic-camera footage is abundant, but finding rare safety-critical events and turning video into actionable insights remains difficult and labor-intensive. SeeUnsafe combines MLLM agents, severity-based video aggregation, and structured multimodal prompts for accident classification and visual grounding. On the Toyota Woven Traffic Safety dataset, it reports 76.31% accuracy and a 51.47% visual-grounding success rate, with best performance against vanilla GPT-4o and other state-of-the-art models.

  • Problem

    Finding rare collisions and near-misses in large volumes of traffic footage and deriving comprehensive scene-level safety insights require substantial manual effort and traffic expertise.

    Source: §1 · pp. 1–2

  • Method

    SeeUnsafe uses MLLM agents, clip-level severity aggregation, and task-specific multimodal prompts to produce structured responses for accident classification and visual grounding.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    76.31% accuracy and a 51.47% visual-grounding success rate; SeeUnsafe reports the best performance compared with vanilla GPT-4o and other state-of-the-art models.

    Source: §6 · p. 18

  • Takeaways & Limitations

    SeeUnsafe is presented as a user-friendly, zero-shot, scalable solution for processing large amounts of traffic footage without extensive technical adaptation.

    Source: §6 · p. 18

    Near-misses can be confused with collisions under occlusion or limited visibility from a single camera view.

    Source: §5.2 · p. 16

Abstract

from arXiv · show

The increasing availability of traffic videos functioning on a 24/7/365 time scale has the great potential of increasing the spatio-temporal coverage of traffic accidents, which will help improve traffic safety. However, analyzing footage from hundreds, if not thousands, of traffic cameras in a 24/7/365 working protocol remains an extremely challenging task, as current vision-based approaches primarily focus on extracting raw information, such as vehicle trajectories or individual object detection, but require laborious post-processing to derive actionable insights. We propose SeeUnsafe, a new framework that integrates Multimodal Large Language Model (MLLM) agents to transform video-based traffic accident analysis from a traditional extraction-then-explanation workflow to a more interactive, conversational approach. This shift significantly enhances processing throughput by automating complex tasks like video classification and visual grounding, while improving adaptability by enabling seamless adjustments to diverse traffic scenarios and user-defined queries. Our framework employs a severity-based aggregation strategy to handle videos of various lengths and a novel multimodal prompt to generate structured responses for review and evaluation and enable fine-grained visual grounding. We introduce IMS (Information Matching Score), a new MLLM-based metric for aligning structured responses with ground truth. We conduct extensive experiments on the Toyota Woven Traffic Safety dataset, demonstrating that SeeUnsafe effectively performs accident-aware video classification and visual grounding by leveraging off-the-shelf MLLMs. Source code will be available at \url{https://github.com/ai4ce/SeeUnsafe}.

1 Introduction

Traffic cameras generate valuable safety footage, but identifying rare critical events and deriving scene-level insights across large networks require substantial expertise and manual effort. SeeUnsafe uses MLLM agents, severity-based aggregation, and task-specific multimodal prompts to support scalable accident classification and visual grounding.

1 Introduction

  • Rare collisions and near-misses are buried in vast routine footage, while comprehensive scene-level analysis across cameras demands substantial manual effort and traffic safety expertise.Existing object detection and tracking do not by themselves integrate camera-level perception into comprehensive safety insights.Source: §1 · pp. 1–2
  • Prior MLLM transportation work largely targets vehicle-centric awareness rather than helping traffic managers process large-scale footage for accident analysis.The paper instead focuses on accident-aware video classification and visual grounding for road users involved in critical events.Source: §1 · pp. 2–3
  • SeeUnsafe addresses long-video scalability, unstructured MLLM responses, and weak fine-grained grounding with clip-based severity aggregation and task-specific multimodal prompts.Users can choose clip granularity; visual prompts add task-specified annotations to support object-level grounding.Source: §1 · p. 3
  • The paper contributes an MLLM-integrated framework, structured-response prompts, and an MLLM-based metric for matching key information in safety-critical responses.Experiments on the Toyota Woven Safety dataset cover accident-aware classification, object grounding, and infrastructure-vehicle co-reasoning.Source: §1 · p. 3

2 Related works

Related work spans MLLM transportation applications, conventional vision-based traffic analysis, and datasets for safety research. The paper positions SeeUnsafe as addressing city-management accident analysis and identifies the Toyota Woven Traffic Safety dataset as particularly suited to pedestrian-vehicle interaction analysis.

2.1 Multimodal Large Language Models in Intelligent Transportation Systems

  • Existing MLLM transportation studies largely address vehicle-centric surrounding awareness, while few explore traffic accident analysis from a city-management perspective.SeeUnsafe applies MLLM agents to accident-aware video classification and identifying road users involved in critical events.Source: §2.1 · p. 3

2.2 Vision-based traffic event analysis

  • Conventional detection, tracking, and segmentation support trajectory or scene analysis, while multimodal vision models produce natural-language outputs but lack interoperability for safety-critical analysis.Prior traffic incident analysis commonly requires task-specific algorithms and post-processing to derive safety measures.Source: §2.2 · p. 4

2.3 Datasets for traffic safety research

  • The Toyota Woven Traffic Safety dataset records pedestrian-centric events from multiple views and provides fine-grained interaction details suited to vision-based accident analysis.Critical traffic events are rare, making real-world datasets scarce and difficult to collect.Source: §2.3 · p. 4

3 Methodology

SeeUnsafe combines MLLM-based video analysis with severity-based aggregation, structured textual responses, and object-level visual grounding. It also introduces IMS to evaluate how well generated responses match ground truth.

3.1 Problem statement

  • The task classifies each video as normal activity, near-miss, or collision, and grounds the two road users involved in critical events.Source: §3.1 · p. 4

3.2 Severity-based aggregation

  • The framework splits videos into clips and selects the highest-severity event class, aggregating descriptions from clips at that severity.The method supports user-defined splitting and fusion functions; the experiments use uniform splitting and heuristic fusion.Source: §3.2 · pp. 5–6

3.3 Task-specifc multimodal prompt

  • Textual prompts request structured outputs for event class, scene context, object description, and justification.The paper states that structured responses facilitate attribute-based retrieval and can slightly improve classification accuracy.Source: §3.3 · p. 6
  • Visual prompts use object detection and segmentation to track road users; object boundaries are overlaid to support fine-grained grounding.The system uses GroundingDINO to detect persons, cyclists, and cars, then Segment Anything to track objects and generate masks.Source: §3.3 · p. 7

3.4 Information matching score

  • IMS uses an MLLM to score matching between generated responses and ground truth across scene context, object description, and justification.The motivating example shows BLEU dropping from 0.89 to 0.70 after a change in error-term position, while ROUGE remains at least 0.90 despite confusing a cyclist with a pedestrian.Source: §3.4 · pp. 7–8

4 Experiments

Experiments use the Toyota Woven Traffic Safety dataset to examine accident-aware classification and visual grounding. The setup specifies the dataset annotations, video sampling, and model comparisons.

4 Experiments

  • All 249 Toyota Woven Traffic Safety scenarios are used, with ground-truth captions summarized into four attributes and manually verified.The dataset contains 167 training and 82 validation scenarios; each scenario includes overhead-camera video and an in-vehicle dash-camera video.Source: §4.1 · p. 9
  • Ground-truth captions are organized into event class, scene context, object description, and justification.The captions are summarized using GPT-4o and manually verified.Source: §4.1 · p. 9
  • The comparison includes LLaVA-NeXT, GPT-4o mini, GPT-4o, and VideoCLIP, using nine uniformly sampled frames split into three clips.Visual grounding counts as successful only when both road users are correctly indexed.Source: §4.1 · p. 9
  • The word cloud displays the 100 most frequent words in processed WTS captions, with larger words indicating greater frequency.It provides a visual comparison of caption-word frequencies.Source: Figure 5 · p. 9

4.2 Quantitative analysis

The analysis compares SeeUnsafe with other MLLMs, examines IMS-based response alignment and visual grounding, and tests how agent choice and lighting relate to performance. Results vary by task and setting, with visual grounding available only in SeeUnsafe among the compared models.

4.2 Quantitative analysis

  • SeeUnsafe with GPT-4o is compared with LLaVA, GPT-4o, and GPT-4o mini; GPT-4o vanilla ranks second-best across most metrics, while VideoCLIP scores lowest and cannot generate responses.LLaVA-NeXTvideo was excluded from accident-aware classification because it consistently predicted the same class; SeeUnsafe is the only compared model reported to perform visual grounding.Source: §4.2 · p. 9, Table 2 · p. 10

4.2.1 Alignment evaluation of IMS metric

  • IMS evaluations show a positive relationship among scene-context, object-description, and justification alignment, while BLEU and ROUGE-L are less informative and sometimes misleading.The reported patterns persist across several SeeUnsafe agents; stronger MLLM scorers give stricter evaluations, whereas weaker scorers can produce overconfident values.Source: §4.2.1 · p. 10

4.2.2 Performance using different MLLM agents

  • Overall performance improves with more capable MLLM agents, but visual prompts can reduce classification performance, especially for GPT-4o mini and LLaVA-NeXTinterleave.GPT-4o handles added object boundaries more robustly; GPT-4o mini is more sensitive, while LLaVA-NeXTinterleave classifications skew toward near-miss with prompts.Source: §4.2.2 · pp. 11–12, Table 4 · p. 12

4.2.3 Performance of visual grounding

  • SeeUnsafe with GPT-4o identifies the correct road users in 70 of 136 critical-event videos (51.47%), rising to 87.5% among the 88 videos with valid masks.With GPT-4o mini, success falls to 25.74% of all 136 videos and 39.78% of the 88 with valid masks; 48 videos had at least one involved object tracked incorrectly.Source: §4.2.3 · p. 12

4.2.4 Correlation between classification accuracy and response alignment

  • Correctly classified videos generally have better-aligned responses, but incorrect classifications can sharply reduce justification alignment while leaving scene-context alignment largely unchanged.For GPT-4o mini, some misclassified normal videos still have higher scene-context and object-description alignment when clear objects are visible.Source: §4.2.4 · p. 13

4.2.5 Performance under different lighting conditions

  • Classification accuracy and response alignment are generally better in daytime; visual prompts help daytime collision and near-miss performance but degrade nighttime performance.The authors attribute nighttime degradation possibly to added prompt noise obscuring already noisy images; limited nighttime samples introduce bias, so overall accuracy is the fairer comparison.Source: §4.2.5 · pp. 13–14, Table 5 · p. 14

4.3 Qualitative results

The examples illustrate how SeeUnsafe classifies near-miss, collision, and normal traffic clips using visible road-user interactions and severity-based aggregation.

4.3 Qualitative results

  • In the near-miss example, clips progress from normal classifications when the vehicle and pedestrian are absent or far apart to a near-miss when the vehicle stops safely near the pedestrian.Severity-based aggregation classifies the video overall as near-miss, and the involved objects are highlighted with segmentation masks.Source: §4.3 · p. 14
  • In the collision example, early clips are classified as normal when only pedestrians are visible, while clips showing vehicle contact are classified as collisions.The sedan approaches from behind and hits a pedestrian, with the final clips showing them remaining in close contact.Source: §4.3 · p. 14
  • For a normal event, the prompt selects the two most visible and camera-near objects, a pedestrian on the crosswalk and a sedan turning right.The example includes several pedestrians and vehicles, illustrating object description in a routine traffic scene.Source: §4.3 · p. 14

5 Discussion

The discussion considers synchronized overhead and dash-camera views, outlines limitations in event classification and visual grounding, and describes broader uses of SeeUnsafe in traffic management and safety research.

5.2 Limitations

  • Near-misses can be confused with collisions under occlusion or limited single-camera visibility, and visual-grounding failures often stem from inaccurate detection and tracking.The authors also identify intuitive splitting and aggregation functions, and suggest keyframe selection, uncertainty-aware aggregation, fine-tuning, and human feedback as improvements.Source: §5.2 · p. 16

5.3 Broader impacts

  • SeeUnsafe’s structured outputs can support content-based video retrieval, while dash-camera use is proposed for real-time surrogate safety measurements and driver-behavior insights.The authors also identify possible applications to driver training, traffic policies, and safer road-system design.Source: §5.3 · pp. 16, 18

5.1 From traffic camera to dash camera

  • The synchronized overhead and dash-camera views classify the same events consistently, while the narrower in-vehicle view produces more detailed responses.The case study reports correct classification of the entire event and suggests multi-view understanding can help when surveillance footage occludes interactions.Source: §5.1 · pp. 14, 16, Table 9 · p. 17

6 Conclusions

SeeUnsafe uses MLLMs to coordinate accident-aware video classification and visual grounding. On the reported evaluation, it achieves 76.31% classification accuracy and a 51.47% visual-grounding success rate, outperforming vanilla GPT-4o and other state-of-the-art models.

6 Conclusions

  • 76.31% classification accuracy and a 51.47% visual-grounding success rate give SeeUnsafe the best performance compared with vanilla GPT-4o and other state-of-the-art models.The conclusion characterizes the framework as zero-shot and scalable, while noting that it needs no extensive technical adaptation.Source: §6 · p. 18
  • SeeUnsafe uses MLLMs as the central agent to orchestrate accident-aware video classification and visual grounding.The authors describe it as a user-friendly solution for processing large amounts of traffic footage.Source: §6 · p. 18
  • The authors identify uniform video splitting and severity-based aggregation, single-view occlusion, and MLLM computational demands as areas for improvement.They point to keyframe sampling, uncertainty-aware aggregation, multi-view footage, and lightweight models as potential directions.Source: §6 · p. 18

A.1 Complete textual prompts

The textual prompts specify classification tasks, output fields, visual-grounding instructions, and a rubric for comparing predicted descriptions with ground truth.

A.1 Complete textual prompts

  • Classification responses are structured to include a video-class code, object description, scene context, and justification.The prompts ask for road-user appearance, environmental context, and behavioral explanations for anomalies or their absence.Source: §A.1 · p. 22
  • The classification prompt assigns each video to normal, near-miss, or collision, with definitions based on traffic behavior and whether road users make impact.For vehicle-view classification, the prompt uses three chronologically ordered images at 0.1-second intervals and asks the model to consider the ego vehicle’s interactions with other road users.Source: §A.1 · p. 22
  • For visual grounding, the prompt asks the model to identify the labeled pedestrian and car involved in an incident by their IDs.The instructions specify one pedestrian and one car, with pedestrians labeled green and cars red.Source: §A.1 · p. 23
  • The matching prompt evaluates scene context, object description, and justification separately against ground truth on a 0–100 scale.It emphasizes justification, including object dynamics, movement, and relative positions, and treats missing predicted content as incorrect.Source: §A.1 · p. 23

A.2 Dataset statistics

The dataset-statistics table covers classification videos by class and duration.

A.2 Dataset statistics

  • The dataset-statistics table reports video counts per class.The supplied table description does not give the class-specific values.Source: §A.2 · p. 24, Table 10 · p. 24
  • The table also reports the duration of videos used in the work.The supplied table description does not state the durations.Source: §A.2 · p. 24
  • The table is scoped to the dataset used for video classification.Its caption identifies it as the classification dataset statistics table.Source: Table 10 · p. 24

A.3 Evaluation of IMS

The evaluation visualizes IMS alongside BLEU and ROUGE-L across several MLLM agents. IMS shows a consistent positive correlation with the two NLP metrics, while GPT-4-based IMS evaluation indicates better response quality as judged by the MLLM evaluator.

A.3 Evaluation of IMS

  • GPT-4-based IMS evaluation places points more often in the upper-right with brighter colors, indicating better response quality perceived by the MLLM evaluator.The passage reports this visual pattern when comparing IMS calculated with GPT-4 versus GPT-4o.Source: §A.3 · p. 24, Figure 13 · p. 26
  • IMS, BLEU, and ROUGE-L show a consistent positive correlation in the qualitative evaluation across tested MLLM agents.The evaluation visualizations cover GPT-4o mini, LLaVA-NeXTinterleave, and LLaVA-NeXTvideo agents.Source: §A.3 · p. 24, Figure 10 · p. 24, Figure 11 · p. 25
  • The evaluation compares IMS computed by GPT-4 and GPT-4o for SeeUnsafe responses generated with a GPT-4o agent.The comparison is presented qualitatively.Source: Figure 13 · p. 26
Loading 2501.10604v1…