Source-linked AI summary
Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4
Maroa Mumtarin, Md Samiullah Chowdhury, Jonathan Wood
TL;DR
Crash-narrative analysis is a common traffic-safety task, and the study compares ChatGPT, BARD, and GPT-4 on information extraction from narratives. Similarity was low for the complex collision-manner query, and the authors caution that answers were not validated and may be inconsistent.
Problem
The study addresses how popular public LLM interfaces perform when classifying or extracting information from crash narratives, an area where complex queries can challenge conventional machine-learning models.
Source: §1 · p. 2Method
Results
35% similarity was reported for manner-of-collision answers; GPT-4 was judged more reliable for pedestrian detection, while event-network patterns differed across interfaces.
Source: §5.3 · p. 8, §5.4 · p. 9, §6 · p. 13Takeaways & Limitations
The authors advise using multiple models to extract information and exercising caution when using LLM interfaces for safety-related information.
Source: §6 · p. 12, §5.4 · p. 12The answers were not validated against an actual crash database, and the interfaces were inconsistent in their results.
Source: §5.4 · p. 12
Abstract
from arXiv · showhide
In traffic safety research, extracting information from crash narratives using text analysis is a common practice. With recent advancements of large language models (LLM), it would be useful to know how the popular LLM interfaces perform in classifying or extracting information from crash narratives. To explore this, our study has used the three most popular publicly available LLM interfaces- ChatGPT, BARD and GPT4. This study investigated their usefulness and boundaries in extracting information and answering queries related to accidents from 100 crash narratives from Iowa and Kansas. During the investigation, their capabilities and limitations were assessed and their responses to the queries were compared. Five questions were asked related to the narratives: 1) Who is at-fault? 2) What is the manner of collision? 3) Has the crash occurred in a work-zone? 4) Did the crash involve pedestrians? and 5) What are the sequence of harmful events in the crash? For questions 1 through 4, the overall similarity among the LLMs were 70%, 35%, 96% and 89%, respectively. The similarities were higher while answering direct questions requiring binary responses and significantly lower for complex questions. To compare the responses to question 5, network diagram and centrality measures were analyzed. The network diagram from the three LLMs were not always similar although they sometimes have the same influencing events with high in-degree, out-degree and betweenness centrality. This study suggests using multiple models to extract viable information from narratives. Also, caution must be practiced while using these interfaces to obtain crucial safety related information.
1 Introduction
The introduction frames crash narratives as valuable but resource-intensive to analyze with conventional machine-learning models, and proposes comparing public LLM interfaces for extracting safety information.
1 Introduction
- Crash narratives record circumstances and contributing factors that can support accident analysis, including weather, location, driver actions, and road hazards.Source: §1 · p. 2
- Training conventional machine-learning models requires substantial data and computational resources, while performance depends on the data’s amount and quality.Source: §1 · p. 2
- The study examines ChatGPT, BARD, and GPT-4 to extract crash information, assess their capabilities and limits, and compare their responses.Source: §1 · p. 2
- The introduction identifies complex tasks such as reconstructing event sequences as an area where current machine-learning models struggle to retrieve information directly.Source: §1 · p. 2
2 Literature Overview
Prior work has used crash narratives with machine-learning and NLP methods, but this study shifts attention to public LLM interfaces and their limitations in crash analysis.
2 Literature Overview
- Crash narratives contain information on crash circumstances and contributing factors, including weather, location, vehicles, injuries, driver actions, and road hazards [4].Source: §2 · p. 2
- Prior studies used logistic regression and other ML/NLP methods to classify speeding, pedestrian, and specialty crashes, including work-zone crashes [5, 6, 7, 8, 9, 10].Source: §2 · p. 2
- The cited BERT study classified traffic injury types with 84.2% accuracy using over 750,000 crash narratives.Source: §2 · p. 3
- The paper describes ChatGPT, BARD, and GPT-4 as pre-trained generative transformers, with different training histories, token limits, and knowledge cutoffs.Source: §2 · p. 3
- The authors note risks of incorrect information and improper assumptions, and say explicit query guidance minimized the possibility of irrelevant or misleading responses.Source: §2 · p. 3
3 Data Description
The study analyzed recent crash narratives collected from Iowa and Kansas highway patrol websites, using their publicly available narrative fields.
3 Data Description
- Iowa State Patrol and Kansas Highway Patrol websites provided publicly available crash reports with narrative fields, with Kansas records excluding the turnpike county.Source: §3 · p. 3
- The dataset contained 100 narratives dated April 20 to May 14, 2023: 15 from Iowa and 85 from Kansas.Source: §3 · p. 3
- LLM responses were collected through web-based chat interfaces in May and June 2023, without using an API.Source: §3 · p. 3
4 Method
The method applies three LLM interfaces to crash narratives using guided prompts, then compares direct answers and analyzes event-sequence responses with network measures.
4 Method
- The researchers analyzed 100 narratives with five queries, restricting answers to categories from Iowa DOT’s crash reporting guide to reduce redundant responses.Source: §4 · p. 3
- The at-fault query tests whether LLM responses can assign responsibility, a field the authors link to law enforcement, insurance disputes, and identifying crash contributors.Source: §4 · pp. 3–4
- The collision-manner query uses Iowa accident-reporting categories and asks the LLMs to classify batches of roughly 20–25 narratives in a table [19].Source: §4 · p. 4, Figure 1 · p. 4
- The work-zone query asks whether a crash occurred in a work zone, addressing the authors’ concern that such crashes may be under-represented in databases.Source: §4 · p. 4
- The pedestrian query asks for a tabular yes/no-style identification, which the authors connect to analysis of pedestrian crash patterns and reducing report misclassification.Source: §4 · pp. 4–5
- The fifth query asks the LLMs to derive harmful-event sequences from narratives using the Model Minimum Uniform Crash Criteria guide and Iowa DOT’s event list [19].Source: §4 · p. 5, Figure 2 · p. 5
- Because question 5 produced 4 to 6 sequences of events per crash, researchers established event networks and applied degree, in-degree, out-degree, and betweenness centrality to identify common and influential crash events.Source: §4 · pp. 5–6
5 Results and Discussion
The interfaces disagreed on several classification tasks, while their event-sequence networks shared some prominent patterns but differed in event emphasis. The authors recommend combining models cautiously because outputs were not validated and were inconsistent.
5.1 Query 1
- 79% of BARD–GPT-4 answers, 78% of ChatGPT–BARD answers, and 70% of GPT-4–ChatGPT answers were similar on fault assignment.These are pairwise similarities among the three interfaces.Source: §5.1 · p. 6, Table 1 · p. 6
- ChatGPT alone chose vehicle 3 in one complex case; BARD alone chose vehicle 2 in another, while the other interfaces chose vehicle 1.Whenever GPT-4 assigned vehicle 1, both other interfaces agreed.Source: §5.1 · p. 6
5.2 Query 2
- ChatGPT and GPT-4 were most similar on collision manner; BARD most often chose “Sideswipe - same direction,” while the other two most often chose “Non-collision (single vehicle).”Source: §5.2 · p. 6, Table 2 · p. 7
5.3 Query 3 & Query 4
- GPT-4 identified only 3 work-zone crashes, and all three interfaces recognized the construction-zone narratives; ChatGPT and BARD also flagged other roadside events.ChatGPT flagged a barrier wall and an emergency-light patrol vehicle; BARD flagged a crash barrel and debris removal.Source: §5.3 · p. 7
- BARD counted deer collisions and ChatGPT counted ejection narratives as pedestrian crashes; the authors judged GPT-4 more reliable at pedestrian detection.Source: §5.3 · p. 8
5.4 Query 5
- All three networks shared the prominent crossed-centerline-to-traffic-collision edge, but differed in prominent nodes and event weighting.ChatGPT and GPT-4 included a railway-vehicle node unsupported by the narrative; BARD prominently used loss of traction, which the narratives did not clearly establish.Source: §5.4 · pp. 8–9, Figure 4 · p. 9, Figure 5 · p. 10
- Traffic collisions, ditch impacts, and rollovers rank high in degree across models; top-node similarity is higher for ChatGPT–GPT-4 on out-degree and betweenness, and ChatGPT–BARD on in-degree.Source: §5.4 · p. 11, Table 4 · p. 12
- Because answers were not validated against actual crash data and interfaces were inconsistent, the authors advise caution; multi-model agreement may reduce human quality-control effort.Source: §5.4 · p. 12
6 CONCLUSION
Across four questions, the three LLM interfaces were more similar on binary work-zone and pedestrian judgments than on manner-of-collision judgments. Their harmful-event networks shared some influential events but were not always alike, leading the study to recommend multiple models and caution for safety-related use.
6 CONCLUSION
- Similarity across the three interfaces was 70% for at-fault, 35% for manner of collision, 96% for work-zone occurrence, and 89% for pedestrian involvement.The study reports higher similarity for direct binary questions and lower similarity for more complex questions.Source: §6 · pp. 12–13
- The event networks were not always similar, though all three identified vehicle-in-traffic collisions, ditch collisions, and overturns or rollovers among higher-degree events.The networks also differed in which events they emphasized.Source: §6 · p. 13
- The study suggests combining multiple models to extract information from crash narratives and exercising caution when using their outputs for safety-related information.Source: §6 · p. 13