Source-linked AI summary
AISA: AI Safety Assistant Framework for Continuous Improvement of Highway Construction
Mason Smetana, Trevor Neece, Lev Khazanovich
TL;DR
Highway safety planning is hindered by unstructured incident narratives that are difficult to use when preparing JSAs. This paper develops a local, deterministic AI framework for classifying, scoring, and retrieving safety information, finding above-chance accident retrieval across most activities while document QA was led by an open-weight embedding model.
Problem
Historical highway construction incident records are often unstructured narratives that are difficult to consult during JSA and pre-task planning.
Method
AISA combines lightweight embedding-based classification and quality scoring with retrieval of historical accidents, imagery, and industry documents for daily safety planning.
Results
Accident retrieval recovered relevant incidents above chance for most activities, while industry document QA was led by an open-weight decoder-based embedding model.
Takeaways & Limitations
The results support low-cost, auditable, offline tooling for surfacing historical safety lessons during highway construction activity planning.
Takeaways & Limitations
The fused pseudo-quality score is preliminary because it primarily measures softmax saturation and requires calibration and labeled validation to establish its relationship with correctness.
Abstract
from arXiv · showhide
Job Safety Analysis (JSA) and pre-task planning can benefit from prior incident records, yet historical accident data is often stored as unstructured narratives that are difficult to consult at the point of planning. A novel framework centered on large language models (LLMs) for highway construction safety reporting and planning is proposed as a foundation for future agentic applications, prioritizing deterministic, local inferencing. The first aim is to enable classification and quality scoring of incident narratives for existing and future reporting purposes. The second is to evaluate retrieval of relevant historical accidents, related imagery, and trusted industry documents for incorporation into daily safety plans. Neural probes were trained to classify incidents along four multiclass and two binary Occupational Injury and Illness Classification System (OIICS) fields and to derive an overall quality score, evaluated on a test set of over 15,000 narratives and a held-out set of 100 author-labeled records, benchmarked against a majority-vote LLM ensemble. The retrieval of historical accidents, reference imagery, and industry documents was benchmarked across embedding models using standard information retrieval metrics. OIICS classification reached 75% held-out accuracy, though the two binary flags were degenerate. The quality score, while meaningful on one database, was distorted on out-of-distribution fatalities in the held-out dataset. Accident retrieval recovered relevant incidents far above chance, performing best on lexically distinct construction activities. On document question answering, an open-weight decoder embedding model surpassed proprietary models. Overall, this work provides a new framework rooted in local inferencing and text embedding models for future agentic applications, with emphasis on bridging external data to JSA reports.
1 Introduction
The introduction frames highway construction as a high-hazard domain where JSA and pre-task planning remain primary defenses, while historical accident narratives are difficult to operationalize. It proposes AISA, an AI Safety Assistant framework combining accident repositories, imagery, trusted industry documents, and text-embedding retrieval to support continuous JSA improvement and daily planning.
- Motivation: Highway construction exposes workers to severe hazards involving heavy equipment, formwork, maintenance, and live traffic, making JSA and pre-task planning central safety practices.JSA breaks an upcoming task into steps and identifies hazards at each step.
- Related work: Recent computational safety research has progressed from predictive analytics toward using NLP and LLMs to read and use accident reports.Predictive analytics combines historical accident reports with project variables such as weather, time of day, and site activity to forecast hazards and allocate safety resources.
- Related work: Existing conversational safety systems include TrafficSafetyGPT, which extracts accident-report information, and AutoRepo, which is primarily image-driven and does not use historical accident data.TrafficSafetyGPT is an open-weight model fine-tuned on a specialized transportation safety dataset and outperforms general-purpose models.
- Framework: AISA proposes continuous JSA improvement by combining publicly available OSHA accident repositories, related imagery, and trusted industry documents for more meaningful and specific daily safety planning reports.The framework is intended to assist daily planning efforts.
- Evaluation: The daily-planning task evaluates modern text-embedding models in a simple RAG pipeline for retrieving incident narratives and relevant industry documents into JSA reports.The evaluation is intended to inform future researchers and practitioners about the capabilities of out-of-the-box embeddings.
2 Methods
The AISA framework centers on an LLM safety assistant that integrates historical accidents, trusted industry documents, and related imagery for two complementary construction-safety tasks. It classifies and scores unstructured accident narratives while using retrieval-augmented generation to assemble JSAs from planned activities.
- Framework architecture: A central LLM safety assistant draws on historical accident records, trusted industry documents, and related imagery as shared construction-safety data sources.The documents include toolbox talks, standards, and specifications; imagery depicts equipment and site conditions.
- Task 1: Narrative classification: Task 1 uses multilayer perceptron probes to classify unstructured accident narratives into standardized codes and fuse outputs into a single quality score.The score quantifies free-text narrative quality for reporting purposes.
- Task 2: JSA planning: Task 2 uses retrieval-augmented generation to assemble a Job Safety Analysis report from a description of the day’s planned activity.This task operates in the forward direction, applying retrieved safety information to planned work.
2.1 Task 1: Incident Reporting and Quality Scoring System
Task 1 develops a deterministic incident-reporting system that classifies free-text narratives and scores how much usable safety information they contain. It uses frozen-embedding probes for OIICS prediction, combines model outputs into a preliminary quality score, and adds exploratory word-level interpretability.
- Task objectives: The task aims to classify incidents deterministically and improve narrative quality, including inconsistent near-miss reporting.Both aims depend on automatically judging how much usable safety information a free-text narrative contains.
- OIICS framework: OIICS provides the standardized framework for classifying injury, illness, and fatality characteristics across Nature, Part of Body, Source, and Event.The framework was developed by the U.S. Bureau of Labor Statistics.
- Classification method: Lightweight MLP probes trained on frozen embeddings examine and operationalize linguistic information for incident classification.The probes hold the encoder fixed while training a small classifier for the target task.
- Classification method: The pipeline trains on human-coded SIR records, truncates OIICS codes to two-digit major groups, and evaluates held-out narratives from a separate IMIS database.Targets are converted to one-hot vectors, and each network learns a probability distribution over its category classes.
- Quality scoring: The preliminary pseudo-quality score averages prediction determinacy across MLP outputs but can mistake confident commitment for correctness.Entropy measures concentration and uncertainty rather than correctness, so confidently wrong predictions may still appear high quality.
- Interpretability: Leave-one-out word saliency identifies tokens that support or oppose the top-1 OIICS prediction by measuring confidence changes after token removal.The analysis is exploratory and intended to improve interpretability of otherwise black-box neural predictions.
2.2 Task 2: Daily Safety Planning and Retrieval
Task 2 grounds daily safety planning in structured highway-construction knowledge and retrieves relevant incident narratives, visual hazard references, and vetted external documents. It uses transparent cosine-similarity retrieval and synthetic QA data generation to support factually grounded safety planning.
- Knowledge grounding: The framework grounds planning in PennDOT’s JSA Manual, whose ontology maps accident narratives to structured construction jobs, steps, and hazards.Prior work mapped unstructured OSHA narratives onto job/step/hazard categories for asphalt pavement construction.
- Retrieval method: Cosine similarity was selected for transparent, low-cost retrieval, using embedding models that included open-weight and OpenAI proprietary encoders.Naïve retrieval embeds queries and candidate documents with the same encoder and scores relevance by cosine similarity.
- Incident retrieval: 965 narratives were mapped to 46 of the 47 construction activities, providing the incident corpus for retrieving accidents relevant to planned work.One activity had zero labeled accident narratives and was excluded.
- Visual references: Synthetic images extend retrieval with visual depictions of activity- or equipment-associated hazards for inclusion in JSA documents.Narratives are transformed into structured descriptions that support generative visualization of highway construction hazards.
- External documents: Over 200 vetted toolbox talks, agency standards, and equipment documents are searched to surface supplementary controls and procedures beyond typical JSA content.An LLM-generated QA-pair pipeline converts the unstructured document corpus into supervised retrieval data without manual annotation.
3 Results · 3.1 Incident Classification and Scoring
The framework classifies incident narratives across OIICS fields, scores prediction determinacy as a quality signal, and tests generalization against independently sourced fatality narratives. On held-out author-labeled records, neural networks were competitive with frontier LLMs, while quality-score interpretation varied across databases.
- 3.1.1 OIICS Classification Models: The 102,922-incident SIR database was split 70/15/15 for training, validation, and testing, with narratives embedded before OIICS classification.The test set contained 15,439 narratives.
- 3.1.1 OIICS Classification Models: The OIICS test metrics tracked task difficulty, with binary flags near-perfect, multiclass structures strong, and Source the weakest predictive capability.Accuracy measures correctness against original OIICS labels; Acc@1, Acc@3, NDCG@10, and MRR@10 evaluate increasingly broad or ranked predictions.
- 3.1.1 OIICS Classification Models: Predicted-probability distributions were skewed toward high confidence, with mean confidence and width σ indicating commitment and consistency.Curves near 1.0 indicate repeated commitment to one code, whereas broader lower curves indicate probability spread across several codes.
- 3.1.2 Held-Out IMIS Database: The IMIS evaluation used 1,197 OSHA fatality narratives absent from training, but lacked gold OIICS labels, preventing direct accuracy measurement.Its fatalities and differing language created systematic distribution differences from SIR.
- 3.1.2 Held-Out IMIS Database: 57.2% mean agreement across six OIICS fields made the neural networks second to gemini-3.5-flash-lite at 62.0% on 100 author-labeled IMIS records.The four multiclass MLPs achieved roughly 60–70% agreement, while the two binary fields were barely informative.
- 3.1.3 Quality Scoring Mechanism: The composite quality score averages determinacy across four multiclass columns, where high determinacy indicates commitment and low determinacy indicates uncertainty.Each prediction receives a top-ranked OIICS code and a determinacy value.
- 3.1.3 Quality Scoring Mechanism: SIR determinacy ranged from 0.78-0.85 across columns, with 20% of records above 0.9; quality means were nearly identical for IMIS and SIR at 0.811 and 0.812.The passages describe SIR as balanced and IMIS as more saturated, limiting the interpretability of comparable mean scores.
- 3.1.4 Word Saliency Example: Leave-one-out word saliency identifies words whose removal most reduces confidence in a target category and highlights overlapping influence across OIICS structures.Words are removed, the narrative is re-embedded, and the category probability is recomputed.
3.2 Information Retrieval Tasks
The study applies naïve retrieval across historical accidents, hazard imagery, and parsed industry documents using shared-space embeddings and cosine similarity. Retrieval performance varies by task, with construction-activity distinctiveness benefiting accident retrieval, indirect image retrieval limiting interpretation, and Qwen-Embedding-0.6b leading document question answering.
- Task design: Naïve RAG embeds a corpus and query in the same vector space, then ranks documents by cosine similarity across three retrieval tasks.The tasks retrieve historical accidents, reference imagery, and passages from parsed industry documents.
- Historical accident retrieval: 965 incident narratives were mapped to 46 construction activities, with retrieval metrics macro-averaged so each activity contributed equally.The narratives came from SIR and IMIS databases and were labeled according to construction activities described in PennDOT Pub 517.
- Historical accident retrieval: Accident retrieval was uneven across activities, with Wood-Chipper – Chipping Brush achieving AP = 1.00 and Garage – Tire Removal/Install achieving AP = 0.604 for text-embedding-3-large.Other listed activities included Broom/Towable Broom (AP = 0.833), Skid Steer – General (AP = 0.670), and Scaffold Erecting/Dismantling (AP = 0.619).
- Reference imagery retrieval: Image retrieval linked planned activities to hazard imagery, but images were retrieved indirectly through their paired accident narratives rather than directly from image content.This indirect procedure is identified as a limitation of the current study.
- Industry-document question answering: 6,214 synthetic query-passage pairs evaluated document retrieval, with the strongest model placing the target passage first about 45% of the time on Acc@1.The questions were generated with gpt-4o-mini and were not manually vetted, so the results indicate retrieval performance without certifying question quality.
- Industry-document question answering: Qwen-Embedding-0.6b surpassed the proprietary OpenAI models on all document-retrieval metrics, reversing the historical-accident retrieval pattern.The open-weight decoder-based model therefore matched or exceeded the proprietary models on this document question-answering task, but required substantially higher inference and GPU resources.
4 Discussion
The discussion presents classification confidence and fused quality scoring as promising aids for safety reporting, while identifying calibration, taxonomy complexity, and optimization limits. Retrieval is feasible for historical incidents and imagery, and document QA notably favored an open-weight embedding model over proprietary alternatives.
- Limitations: Classification results reflect task difficulty and a nonexhaustively optimized configuration, while larger, more convoluted label spaces offer limited practical benefit.The discussion suggests leaving lower-level granular coding to future work because such networks can become trapped in local minima.
- Classification: Classification confidence can indicate when labels are trustworthy or should be presented as a top-3 shortlist for human review.Part of Body achieved Acc@1 = 0.843 and Nature achieved Acc@1 = 0.815, while Source was weakest across statistics.
- Classification: The four multiclass models transfer incidents between schemas with roughly 60-70% agreement with author labels, consistent with ≈75% held-out testing accuracy.This supports converting records from systems such as IMIS into SIR/OIICS.
- Quality scoring: The fused quality score is interpretable and discriminating on SIR, but currently measures softmax saturation more than true input quality across databases.Further calibration and labeled validation are needed before determinacy can be treated as a proxy for correctness.
- Retrieval: Accident retrieval is feasible, although single-label relevance and ambiguous activities make reported retrieval figures lower bounds rather than absolute ceilings.Retrieval depends on query wording, and additional embedding fine-tuning may improve performance.
- Retrieval: Qwen-Embedding-0.6b led every tested proprietary and general-purpose model on industry document QA, surpassing OpenAI embeddings across all metrics.Image retrieval could pair JSAs with realistic hazard visuals, but sparse datasets, site access limits, and ethical concerns constrain deployment.
5 Conclusions
The AISA framework supports deterministic, local safety reporting and planning for highway construction through OIICS classification, narrative quality scoring, and retrieval of historical accidents and industry documents. Its proof-of-concept results are promising but require human review, further validation, and improved multimodal capability.
- Framework contribution: AISA integrates deterministic local inference into continuous improvement of highway-construction Job Safety Analysis reports and has a prototype for near-future field exposure.Industry partners and safety professionals can assess the proposed features directly.
- Classification: Lightweight multilayer perceptrons provide useful deterministic coding of accident narratives into OIICS, performing strongest where coding categories are clearly separated.Performance was weakest on the largest, most overlapping label space.
- Quality scoring: The fused MLP pseudo-quality score is promising and trustworthy on in-distribution data but can distort confidence for out-of-distribution fatalities.Leave-one-out saliency analysis identifies words influencing each prediction.
- Retrieval: Dense retrieval recovered relevant historical accidents above chance for most activities, but human reviewers remain necessary to select the most relevant results.Accident retrieval favored proprietary embeddings, whereas industry-document question answering was led by an open-weight decoder-based model.
- Limitations: The framework remains a proof of concept because its out-of-the-box methods lack fine-tuning and multimodal capability, while image retrieval currently uses coupled narratives rather than image content.The prototype’s AI-generated imagery should be replaced with proprietary images, and human feedback and iteration remain necessary.
- Future applications: Local models support privacy and security for sensitive data, while incorporating new accidents and near-misses into the retrieval corpus enables iterative safety planning.The study positions these deterministic components as a foundation for potential agentic applications.
Disclaimer: On the use of Generative AI
The authors used OpenAI’s ChatGPT for manuscript proofreading and revision while acknowledging LLM limitations, including potential biases, errors, and knowledge gaps.
- OpenAI’s ChatGPT assisted with proofreading and revising the manuscript.
- The authors acknowledge that LLMs may involve biases, errors, and gaps in knowledge.