PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements

Jianwei Wang, Mengqi Wang, Yinsi Zhou, Zhenchang Xing, Qing Liu, Xiwei Xu, Wenjie Zhang, Liming Zhu

arXiv:2505.22959v1cs.CL

TL;DR

HSE assessment is high-stakes, while benchmark coverage of diverse sources, complex contexts, and legal reasoning remains limited. HSE-Bench evaluates LLMs using curated, IRAC-structured HSE questions and RoE’s expert-role prompting. Results reveal reasoning gaps in current models, while RoE improves DeepSeek-R1 accuracy by up to 25.01% compared with CoT.

  • Problem

    HSE benchmarks remain limited in diverse data sources, complex contexts, and sophisticated legal reasoning, despite the high stakes of HSE compliance assessment.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    HSE-Bench uses curated questions from varied HSE sources and IRAC reasoning stages, while RoE prompts expert roles to reason independently and synthesize a decision.

    Source: §1 · p. 2, §3.5 · p. 6

  • Results

    Current LLMs rely largely on semantic matching and lack systematic legal reasoning; RoE improves DeepSeek-R1 accuracy by up to 25.01% compared with CoT.

    Source: §1 · p. 2, §3.5 · p. 6

  • Takeaways & Limitations

    HSE-Bench highlights reasoning gaps in current models and provides a foundation for future research in HSE.

    Source: §5 · p. 9

    HSE-Bench focuses on compliance assessment, leaving other HSE stages such as incident response underexplored; human oversight remains essential in high-stakes scenarios.

    Source: §5 · p. 9

Abstract

from arXiv · show

Health, Safety, and Environment (HSE) compliance assessment demands dynamic real-time decision-making under complicated regulations and complex human-machine-environment interactions. While large language models (LLMs) hold significant potential for decision intelligence and contextual dialogue, their capacity for domain-specific knowledge in HSE and structured legal reasoning remains underexplored. We introduce HSE-Bench, the first benchmark dataset designed to evaluate the HSE compliance assessment capabilities of LLM. HSE-Bench comprises over 1,000 manually curated questions drawn from regulations, court cases, safety exams, and fieldwork videos, and integrates a reasoning flow based on Issue spotting, rule Recall, rule Application, and rule Conclusion (IRAC) to assess the holistic reasoning pipeline. We conduct extensive evaluations on different prompting strategies and more than 10 LLMs, including foundation models, reasoning models and multimodal vision models. The results show that, although current LLMs achieve good performance, their capabilities largely rely on semantic matching rather than principled reasoning grounded in the underlying HSE compliance context. Moreover, their native reasoning trace lacks the systematic legal reasoning required for rigorous HSE compliance assessment. To alleviate these, we propose a new prompting technique, Reasoning of Expert (RoE), which guides LLMs to simulate the reasoning process of different experts for compliance assessment and reach a more accurate unified decision. We hope our study highlights reasoning gaps in LLMs for HSE compliance and inspires further research on related tasks.

1 Introduction

HSE compliance assessment is high-stakes, yet LLMs’ capacity for HSE-specific knowledge and structured legal reasoning remains underexplored. The paper introduces HSE-Bench to evaluate that capacity and RoE to guide expert-role reasoning.

1 Introduction

  • HSE compliance assessment supports risk management, while failures may threaten employee well-being and compliance with legal obligations.The paper identifies HSE assessment as important in industries including construction, manufacturing, and healthcare.Source: §1 · pp. 1–2
  • HSE-Bench uses curated, scenario-based questions from diverse sources and four IRAC reasoning dimensions to evaluate LLMs.The sources include regulations, court cases, safety exams, and fieldwork videos; questions are adversarially generated and manually checked by two experts.Source: §1 · p. 2
  • Evaluations find that current LLMs rely largely on semantic matching and lack the systematic legal reasoning needed for rigorous HSE compliance assessment.The paper evaluates foundation, reasoning, and vision models with multiple prompting strategies.Source: §1 · p. 2
  • RoE prompts models to reason from different domain-expert roles and synthesize their independent judgments into a unified decision.The approach is proposed to improve LLM capacity for HSE compliance assessment.Source: §1 · p. 2

2 Related Works

Prior work evaluates LLMs in legal, regulatory, and high-risk settings, but existing benchmarks remain limited in diverse data, complex contexts, and sophisticated legal reasoning. The paper proposes HSE-Bench to address these gaps.

2 Related Works

  • Legal benchmarks decompose tasks into fine-grained capabilities, while regulation-centric work also examines rule-grounded reasoning across real-world policies [19].A study also explores fine-tuning LLMs for HSE applications.Source: §2 · pp. 2–3
  • Other high-risk benchmarks and applications span laboratory hazards, regulatory compliance, traffic incidents, accident analysis, equipment failure, construction checks, and PPE verification [52] [8].Related efforts also explore VR-augmented safety environments and LLM-assisted safety learning.Source: §2 · p. 3
  • Prior work identifies limitations in handling diverse data sources, complex contexts, and tasks requiring sophisticated legal reasoning; HSE-Bench is proposed as a systematic evaluation framework.The paper positions the benchmark for HSE scenarios.Source: §2 · p. 3

3 HSE-BENCH

HSE-Bench combines diverse HSE source materials, an IRAC-based reasoning taxonomy, and a two-phase construction process. Its evaluation motivates RoE, which simulates expert roles and synthesizes their judgments for compliance assessment.

3 HSE-BENCH

  • The framework constructs benchmark instances from four sources, evaluates 12 LLMs, explores prompts and settings, and introduces expert-role reasoning for complex HSE tasks.The sources are regulations, court cases, safety exams, and fieldwork videos.Source: §3 · p. 3

3.1 Dataset Instance Collection

  • The dataset draws on regulations, court cases, safety exams, and fieldwork videos, converting videos into text descriptions for text-only model evaluation.Regulations are collected across the US, Canada, China, and Australia; court cases primarily come from Australian states.Source: §3.1 · p. 4

3.2 Dataset Taxonomy

  • IRAC assesses issue spotting, rule recall, rule application, and rule conclusion as distinct stages of HSE compliance reasoning.The stages cover identifying concerns, finding governing rules, applying rules to scenario facts, and reaching a decision.Source: §3.2 · p. 4

3.3 Benchmark Construction

  • Benchmark construction uses initial IRAC-style question generation followed by adversarial contextual enhancement.The enhancement adds distractors and logical complexity while removing confidential information.Source: §3.3 · pp. 4–5

3.3.1 Initial Prompt-Based Generation

  • Initial prompts guide an LLM to generate IRAC-style questions, answer choices, labels, explanations, and referenced rules, with outputs manually reviewed and filtered.Prompts include a task description and relevant source material.Source: §3.3.1 · p. 5

3.3.2 Adversarial Contextual Augmentation

  • Adversarial augmentation adds plausible distractors and complicating factors while preserving key facts and the original answer; two experts check each question.Complications can include unclear responsibilities, conflicting information, or partial compliance.Source: §3.3.2 · p. 5

3.4 Dataset Characteristics

  • 1,020 IRAC-formatted questions comprise 448 regulation, 152 court-case, 320 exam, and 100 video questions.Court-case questions average 111.76 words and video questions 106.65, longer than exams at 92.49 and regulations at 60.24; the table presents dataset statistics.Source: §3.4 · p. 5, Table 1 · p. 5

3.5 Reasoning of Experts

  • RoE improves DeepSeek-R1 accuracy by up to 25.01% compared with CoT.The method prompts multiple domain-specific experts to reason independently using IRAC, then synthesizes their viewpoints; the evaluation also finds reliance on surface-level semantic matching and inadequate systematic legal reasoning.Source: §3.5 · pp. 5–6

4 Evaluation

Across HSE-Bench evaluations, LLM performance depends strongly on answer options and prompting strategy, while integrated IRAC reasoning remains challenging. RoE improves results for both tested models, but reasoning models do not outperform foundation models overall.

4.1 Setup

  • The evaluation covers 12 LLMs, six prompt strategies, and Accuracy and AUC-ROC metrics.Accuracy uses scenario, question, and answer options; AUC-ROC uses free-form responses to the scenario and question, with BERT similarity scores as confidence estimates.Source: §4.1 · p. 6

4.2.1 Overall Results

  • AUC-ROC decreases by an average of 24.73% versus accuracy across models, phases, and data sources.With options, models can compare the context with candidate choices; without them, they rely on internal legal knowledge and show less discriminative behavior.Source: §4.2.1 · pp. 6–7, Figure 2 · p. 7

4.2.2 Prompt Evaluation Results

  • RoE significantly outperforms the other tested prompting strategies on both DeepSeek-V3 and DeepSeek-R1.Its average accuracy gains over the baselines range from 2.91% to 4.54% for DeepSeek-V3 and from 3.37% to 5.18% for DeepSeek-R1.Source: Figure 3 · p. 8, §4.2.2 · p. 9

4.2.1 Overall Results

  • Reasoning models do not outperform foundation models, and their AUC-ROC drop versus accuracy is 29.72%, compared with 25.52% for foundation models.The authors suggest general-purpose reasoning patterns may not align with the legal reasoning required by complex HSE scenarios.Source: §4.2.1 · p. 8
  • Average AUC-ROC is 70.17% for issue spotting, 60.75% for rule recall, 66.96% for rule application, and 64.27% for rule conclusion.Rule recall is particularly challenging, and model-performance variance widens in later phases.Source: §4.2.1 · p. 8

4.2.3 IRAC Pipeline Evaluation

  • In the integrated four-phase pipeline, Claude-3.7-Sonnet averages 68.06% accuracy, versus 91.30%, 90.58%, 96.58%, and 96.99% on the four single phases.DeepSeek-R1 averages 68.42% on integrated questions, while QwQ achieves 61.60%.Source: Figure 4 · p. 9, §4.2.3 · p. 9

5 Conclusion

HSE-Bench evaluates LLMs on HSE compliance assessment and reveals gaps in regulatory knowledge and structured reasoning; RoE uses expert-role simulation to improve legal decision-making. The benchmark focuses on compliance assessment, while incident response and other HSE lifecycle stages remain underexplored.

5 Conclusion

  • HSE-Bench is a comprehensive benchmark of over 1,000 questions for evaluating LLMs in HSE compliance assessment, and RoE uses expert-role simulation to enhance legal decision-making.The paper reports that models perform reasonably well with semantic cues but fall short on deep regulatory knowledge and structured reasoning.Source: §5 · p. 9
  • HSE-Bench primarily covers HSE compliance assessment; incident response and other lifecycle stages remain underexplored, and high-stakes scenarios require human oversight.The authors call for evaluation and prompting research across the full HSE lifecycle.Source: §5 · p. 9

A Evaluated LLMs

  • The dataset, benchmark-generation and evaluation scripts, and metric-computation code are publicly released.The dataset is released under the Creative Commons Attribution License (CC BY 4.0).Source: A Evaluated LLMs · p. 14
  • The experiments evaluate 12 LLMs spanning foundation, reasoning, and vision-language models.Model names, sizes, and API providers are listed in Table 2.Source: A Evaluated LLMs · p. 14, Table 2 · p. 14

B Experimental Setup

The experimental setup compares six prompting strategies, including conventional prompting baselines and RoE, which simulates domain experts independently reasoning before synthesizing a decision.

B.1 Evaluation prompts

  • The six evaluated prompts are zero-shot, few-shot, CoT, 0-CoT, CoT+SC, and RoE.The first five are literature-based prompting strategies; RoE is the newly proposed prompt.Source: §B.1 · pp. 14–15
  • Zero-shot prompting tests model responses without examples, while few-shot prompting supplies five examples.Few-shot prompting is described as particularly aiding rule application tasks.Source: §B.1 · p. 14
  • CoT uses example-guided intermediate reasoning, while 0-CoT explicitly instructs the model to reason systematically.The evaluation also includes CoT with Self-Consistency, which selects the majority answer across five CoT runs.Source: §B.1 · pp. 14–15
  • RoE simulates domain-specific experts giving independent judgments that are synthesized into a final decision.The paper frames this as reflecting compliance assessment procedures used in HSE cases.Source: §B.1 · p. 15

B.2 Metric computation

The section defines Accuracy and AUC-ROC as complementary classification metrics. Accuracy checks the selected label against ground truth, while AUC-ROC evaluates how highly the ground-truth option is ranked against negative options.

B.2 Metric computation

  • Accuracy is the proportion of correctly predicted labels among all samples.The computation uses the predicted and ground-truth labels and an indicator function.Source: §B.2 · p. 15
  • AUC-ROC measures whether the ground-truth option is ranked above negative options using predicted scores.Here the score is the cosine similarity between the LLM response and an option.Source: §B.2 · p. 15

C.1 Data sources

HSE-Bench turns four kinds of HSE material into scenario-based questions organized around the IRAC reasoning stages. Its construction combines source-specific generation with adversarial refinement and expert checks.

C.1 Data sources

  • The benchmark draws on regulations, court cases, safety exams, and fieldwork videos.Regulations were collected from the US, Canada, China, and Australia; the video set contains 25 videos derived from nine selected videos, including segmented clips.Source: §C.1 · pp. 15–17
  • Regulation-based scenarios are generated from randomly sampled regulation segments, while court cases are generalized into scenarios specifying who did what and where.Court-case questions test the four IRAC components and are generated as single-choice questions.Source: §C.1 · pp. 15–16
  • Each source is used to construct scenario-based questions covering issue spotting, rule recall, rule application, and rule conclusion.The questions have one correct answer; prompts also request explanations and plausible distractors with subtle errors.Source: §C.1 · pp. 16–17
  • The safety-exam prompts transform existing labeled HSE questions, while video prompts use textual descriptions of scenes and actions.The video descriptions are based on frames extracted at fixed intervals.Source: §C.1 · pp. 16–18

C.2 Implementation details of benchmark constructions

  • Initial prompt-based question generation is followed by adversarial contextual enhancement, and domain experts double-check the generated questions.The refinement prompt can make scenarios more realistic and non-obvious while preserving the correct answer and answer choices.Source: §C.2 · p. 18

C.3 Example of the benchmark taxonomy constructed using IRAC

The example presents HSE-Bench through the four stages of IRAC, moving from identifying an issue to concluding from rule application. Table 3 illustrates questions and options for the stages using a shared scenario.

C.3 Example of the benchmark taxonomy constructed using IRAC

  • Issue spotting identifies the HSE issue, rule recall identifies relevant regulations, rule application connects rules to case facts, and rule conclusion answers the legal question.The conclusion reflects the reasoning developed in the preceding stages.Source: §C.3 · pp. 18–19
  • Each IRAC pipeline contains four questions sharing a scenario, with specific prompts and answer options.Table 3 gives examples across IRAC phases and highlights the correct options in bold.Source: §C.3 · p. 19, Table 3 · p. 19

D.1 Additional results analysis

Across the tested prompting strategies, chain-of-thought variants do not consistently outperform simpler prompts on AUC-ROC. RoE further boosts CoT performance on both tested models.

D.1 Additional results analysis

  • CoT-based strategies show no significant AUC-ROC improvement over zero-shot and few-shot prompting and sometimes perform worse.The comparison covers DeepSeek-V3 and DeepSeek-R1.Source: §D.1 · p. 19, Figure 5 · p. 20
  • For DeepSeek-V3, average AUC-ROC is 65.60% for zero-shot, 65.32% for few-shot, 64.97% for 0-CoT, 66.24% for CoT, and 67.36% for CoT+SC.These values illustrate the reported variation across prompt strategies.Source: §D.1 · p. 19
  • RoE boosts CoT performance by up to 6.07% on DeepSeek-V3 and up to 8.03% on DeepSeek-R1.The paper reports larger prompting-strategy gains for reasoning-centric models than for foundation models.Source: Figure 5 · p. 20, §D.1 · p. 20

D.2 Case study

Four case studies examine LLM reasoning traces in HSE compliance assessment. They contrast option-sensitive reasoning and gaps in legal reasoning with changes under RoE and adversarial contextual augmentation.

D.2 Case study

  • With options, DeepSeek-R1 evaluates and compares choices, while without them it recalls rules and analyzes the scenario, illustrating reliance on semantic matching over principled legal reasoning.The case study reports that the model identifies applicable legal principles and the primary legal issue when options are omitted.Source: §D.2 · p. 20, Table 4 · p. 21
  • DeepSeek-R1 and QwQ-32B recall high-level rules and analyze the incident from perspectives such as the employer, but lack rigorous legal reasoning and reach inaccurate conclusions.The case study identifies difficulty applying the specific legal reasoning required for HSE compliance assessment.Source: §D.2 · p. 20, Table 5 · p. 22
  • With RoE, the LLM produces a more structured reasoning process and a more accurate outcome than its original version.The accompanying table is described as highlighting structured legal reasoning aligned with HSE compliance requirements.Source: §D.2 · p. 20, Table 6 · p. 23
  • The fourth case study presents an adversarially augmented question to demonstrate the augmentation process’s effectiveness.The provided description does not specify the augmentation’s content or its measured effect.Source: §D.2 · p. 20, Table 7 · p. 24

Related papers

Loading 2505.22959v1…