PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance

Alexander Thomas, Hubert P. H. Shum, Darren Nellis, Manli Zhu, Phatpicha Yochum, William Bartle, Daniel Wrightson

arXiv:2608.21036v1cs.AI

TL;DR

Dangerous-goods shipping requires reliable interpretation of complex IMDG rules, but evidence on LLM performance in this safety-critical domain was missing. The paper introduces DGEval, a 1,678-question benchmark evaluating 13 models across complementary compliance tasks and web-search conditions. Models perform relatively well on some knowledge tasks but remain unreliable in operationally critical areas, supporting governed use with practitioner oversight and authoritative verification.

  • Problem

    No systematic evaluation had established whether LLMs can reliably interpret IMDG requirements for dangerous-goods compliance.

  • Method

    DGEval evaluates 13 models on 1,678 questions built from expert-written e-learning content and structured DGL lookups, including multiple thinking configurations and web search.

  • Results

    Models are weakest in stowage, segregation, and regulatory recall; web search improves DGL lookup accuracy by +25.0 pp on average, while the best model exceeds the 83.8% human multiple-choice baseline.

  • Takeaways & Limitations

    LLMs may assist with structured compliance lookups, but aggregate performance does not establish reliable dangerous-goods decision-making.

  • Takeaways & Limitations

    The benchmark covers only the IMDG Code and uses isolated, well-formed questions rather than whole multi-step operational scenarios.

Abstract

from arXiv · show

The transport of dangerous goods by sea is a high-consequence activity governed by the International Maritime Dangerous Goods (IMDG) Code, a complex regulatory framework where errors in classification, packaging, stowage, or segregation can result in fire, explosion, toxic release, or loss of life or vessel. Correct compliance requires accurately interpreting hundreds of pages of interacting provisions, updated on a two-year amendment cycle. Practitioners increasingly use Large Language Models (LLMs) as decision-support tools, yet no systematic evaluation exists of whether they can reliably interpret IMDG requirements for safety-critical use. This paper introduces DGEval, the first benchmark for evaluating LLM knowledge of IMDG Amendment 42-24. Built from expert-written questions on the NCB Hazcheck e-learning platform and structured lookups from the Dangerous Goods List (DGL), it comprises 1,678 questions across multiple-choice, open-ended, DGL lookup, and regulatory identification tasks. We evaluate 13 models from six providers across multiple thinking configurations, including one maritime domain-specific fine-tuned model, and test the effect of web search. Although the best-performing model exceeds the human practitioner baseline on multiple-choice questions, all models are weakest in the operationally safety-critical areas of stowage, segregation, and regulatory recall. These results indicate that LLMs may support compliance tasks, particularly structured DGL lookups with web search, but unreliability in operational areas and regulatory-text recall means human oversight and authoritative source verification remain necessary before deployment in any safety-critical context. DGEval is designed as a safety assurance instrument to be applied continuously as models evolve, not as a settled characterisation of current capability.

1. Introduction

Dangerous-goods shipping is governed by a complex, high-consequence regulatory code, yet no systematic benchmark had tested whether LLMs could interpret it reliably. DGEval addresses this gap with a multi-model, repeatedly applicable evaluation focused on IMDG Amendment 42-24.

  • Motivation: IMDG compliance spans hundreds of pages and interacting requirements for classification, documentation, packaging, stowage, and segregation.Stowage and segregation require multi-step reasoning across interacting DGL fields and Code sections.
  • Motivation: LLM adoption in professional and maritime settings has grown despite factual errors in AI-generated dangerous-goods guidance.The paper notes that such errors are already circulating in professional forums.
  • Research gap: No systematic evaluation previously measured LLM performance on dangerous-goods compliance tasks.Existing general and domain-specific benchmarks did not address dangerous-goods transport, while LLaMarine was not a compliance benchmark.
  • Contribution: DGEval is the first benchmark specifically designed to evaluate LLM knowledge of IMDG Amendment 42-24.It uses expert-written e-learning content and structured DGL-derived questions.
  • Contribution: The study evaluates 13 models from six providers across multiple thinking configurations and tests web-search effects.The benchmark is intended for repeated re-application as models change.
  • Contribution: The paper provides actionable findings including model rankings, systematic weak areas, and cost-performance analysis.These findings are intended for industry practitioners and benchmark designers.

2. Background

The IMDG Code provides the international framework for dangerous-goods transport by sea, while the DGL supplies substance-level regulatory properties. Compliance decisions are interdependent across practitioner roles and across the transport chain.

  • IMDG framework: The IMDG Code implements SOLAS and MARPOL provisions for classifying, documenting, packaging, marking, segregating, stowing, and handling dangerous goods.It is maintained by the International Maritime Organization.
  • IMDG framework: IMDG amendments follow a two-year cycle, with Amendment 42-24 taking effect on 1 January 2025 and becoming mandatory from 1 January 2026.The framework permits a one-year overlap during which two versions may be valid.
  • Dangerous Goods List: The Dangerous Goods List maps each dangerous substance or article to regulatory properties using its UN number and Proper Shipping Name.Its columns include hazard, packing, segregation, stowage, and emergency information.
  • Dangerous Goods List: DGL properties are interdependent, so segregation, marking, placarding, packaging, and stowage often require multi-step interpretation.Exceptions and additional requirements can modify these decisions.
  • Practitioner roles: The benchmark considers consignors or freight forwarders, packers or cargo handlers, ship loaders, and ship operators.Each role has different day-to-day IMDG information needs.
  • Practitioner roles: Compliance is a property of the whole transport chain because early decisions constrain later handling, loading, stowage, and discharge.Packing incompatible goods together can propagate risk to every later participant.

3. Related Work

Dangerous-goods compliance combines established rule-based automation with emerging LLM capabilities, but its safety-critical nature demands specialised evaluation. DGEval fills the missing benchmark for LLM performance on IMDG compliance.

  • Existing compliance systems: Traditional dangerous-goods compliance relies on trained personnel, manual IMDG consultation, and digital tools, while overlooked provisions can leave shipments non-compliant.Packaged dangerous-goods incidents are associated with substantial maritime casualty risk.
  • Existing compliance systems: Commercial compliance systems primarily encode IMDG provisions as deterministic rule-based engines for classification, segregation, and documentation validation.Academic work has also formalised Code knowledge for automated reasoning.
  • LLM capabilities: LLMs answer open-ended questions from broad text training, while web search and additional reasoning computation can alter their responses.The evaluation tests both capabilities.
  • LLM limitations: LLMs can generate fluent but factually incorrect answers and lack a reliable internal signal indicating uncertainty.These limitations are consequential in regulated, safety-critical applications.
  • Benchmarking literature: General-purpose and domain-specific benchmarks measure broad or specialised capabilities, including legal, financial, scientific, and laboratory safety reasoning.Safety-critical benchmarks report substantial limitations among current systems.
  • Benchmarking literature: Dangerous-goods compliance requires hazard identification and reasoning across interacting packaging, segregation, and stowage constraints.Errors during preparation, packing, and loading remain associated with dangerous-goods incidents at sea.
  • Research gap: DGEval addresses the absence of a benchmark evaluating dangerous-goods compliance against the IMDG Code.It fills both the specific dangerous-goods gap and the broader regulatory-compliance evaluation gap.

4. Data Description and Methodology

DGEval combines expert-authored e-learning questions with structured DGL lookups and regulatory-recall tasks. Its evaluation compares models under multiple answer formats and scoring procedures while accounting for important human-baseline caveats.

  • Data sources: DGEval draws on NCB Hazcheck e-learning content and a curated DGL aligned with IMDG Amendment 42-24.The two sources support complementary practitioner-knowledge tasks.
  • Data sources: 520 usable expert-authored e-learning questions cover 20 subsections spanning classification, packing, consignment, stowage, segregation, and emergency response.Image-referencing questions were excluded from the original 609 aligned questions.
  • Data sources: The DGL source yielded 9,443 eligible structured lookup questions across UN-number, PSN, and specific-property queries.Variant UN/PSN combinations were excluded to avoid ambiguity.
  • Benchmark design: DGEval keeps questions and answer keys private to reduce test-set memorisation and support controlled evaluation through an evaluation server.The paper distinguishes this design choice from a limitation.
  • Benchmark sections: The benchmark includes multiple-choice, open-ended, DGL lookup, and regulatory-recall sections targeting distinct knowledge capabilities.Regulatory recall identifies the IMDG section containing an answer and provides a route to authoritative sources.
  • Benchmark sections: Multiple-choice questions reproduce the e-learning format for direct comparison with the human baseline, whereas open-ended questions remove answer-option scaffolding.Open-ended items require enumeration of all correct answers when multiple answers apply.
  • Benchmark sections: The DGL section randomly sampled 485 questions from 9,443 eligible lookups, excluding overlap with the e-learning sections.UN-number and PSN queries each comprised approximately 25% of the sample.
  • Scoring: Single-answer tasks use binary scoring, while multi-answer tasks use partial credit based on the fraction of correct answers identified.Providing excess answers incurs a symmetric penalty.

5. Results

Across the benchmark, model performance varies sharply by task: models perform best on multiple-choice and DGL lookup, but struggle with operational stowage and segregation and with regulatory-section recall. Gemini 3.1 Pro leads overall, while web search substantially improves DGL lookup but not multiple-choice performance.

  • 5.1 Overall Performance: Gemini 3.1 Pro leads all four sections, scoring 95.8%, 86.0%, 89.6%, and 36.8% at its highest thinking level.No other model comes within 5 percentage points on Section 1.
  • 5.1 Overall Performance: Only Gemini 3.1 Pro and GPT-5.5 at Low or above exceed the corrected human practitioner baseline of 83.8% on Section 1.Ten model configurations exceed the 75% practitioner pass mark.
  • 5.2 Subsection Analysis: Section 1 classification subsections average 82–85%, whereas stowage and segregation subsections average approximately 49–50%, a gap exceeding 35 percentage points.The same hierarchy appears in the strongest model, indicating task-difficulty effects rather than only model-specific weakness.
  • 5.2 Subsection Analysis: Stowage and segregation are consistently weak: Stowage Code averages 39.8%, while Segregation Group averages 18.4%.The five hard DGL fields are operational and safety-critical; even Gemini 3.1 Pro High reaches only 51.5% on Stowage Code.
  • 5.5 Web Search: Web search raises Section 3 performance by an average of 25.0 percentage points but changes Section 1 performance by only −0.3 percentage points.For GPT-5.4 Mini, DGL lookup improves from 32.2% to 81.9%, a gain of 49.6 percentage points.

6. Discussion

DGEval shows that leading models can perform strongly on some compliance tasks, but none is uniformly reliable in the operationally riskiest areas. The discussion therefore recommends task-specific evaluation, authoritative verification, practitioner approval, and continuous re-evaluation.

  • Model Performance for Practitioners: Gemini 3.1 Pro leads all four sections and exceeds the human practitioner baseline on Section 1, but remains weak on stowage, segregation, special provisions, and regulatory recall.Its Section 1 score is 95.8%, compared with the 83.8% human baseline.
  • Thinking Mode Effects: Increased thinking does not reliably improve performance, with several models showing penalties at higher thinking levels.For GPT-5.4 Mini, Section 3 falls from 47.5% to 46.4% and Section 4 from 15.6% to 12.6% when moving from Medium to High thinking.
  • Google Models and Training Data: Google models’ cross-section dominance may reflect broader or licensed regulatory-content coverage, but undisclosed training data prevents determining the source of the advantage.The persistent weakness on stowage codes, segregation groups, and transport procedures is consistent with incomplete enumeration of operational fields in training sources.
  • Safety Guardrails and Refusals: Safety refusals disproportionately affect some critical dangerous-goods topics, limiting usefulness when models cannot distinguish legitimate compliance questions from adversarial queries.Refused topics include pathogens, inhalation toxicity, dibromomethane, and nitroglycerin.
  • Limitations: DGEval is limited to IMDG and textual, well-formed, one-question-at-a-time evaluation, whereas real compliance is multimodal, multi-party, interdependent, and often ambiguous.The study identifies multimodal inspection and more realistic practitioner-query settings as future extensions.
  • Safe Deployment and Continuous Assurance: DGEval should be reapplied whenever models are updated, retired, or replaced to provide current evidence about reliability on dangerous-goods compliance tasks.The benchmark is intended as a continuing safety-assurance instrument rather than a one-time characterization.
  • Safe Deployment and Continuous Assurance: No model is reliable enough to act without oversight, so deployment should use governed workflows with trained practitioners retaining approval authority and verifying outputs against authoritative systems.Composite scores should not be the sole deployment criterion because they can mask failures in high-danger-criticality sub-domains.

7. Conclusion

DGEval finds that leading models can outperform the practitioner baseline on multiple-choice questions, but all models remain unreliable in safety-critical operational areas. The results support governed decision-support use and ongoing evaluation rather than unaided compliance decisions.

  • Across 1,678 questions, 13 models, and 35 configurations, Gemini 3.1 Pro achieves the best results across all benchmark sections.
  • Section 4 regulatory recall is uniformly difficult, with a mean score of 14.2%, while stowage and segregation remain weak points across models and sections.
  • Web search improves Section 3 DGL lookup accuracy by 25.0 percentage points on average, although cost-efficiency varies substantially by model.
  • LLaMarine performs no better than base Llama 3.3 (70B) on DG questions, while Gemini Flash-Lite with web search offers a lower-cost lookup option.
  • None of the evaluated models is validated or certified for operational DG compliance decisions, so deployment should use trained-practitioner oversight and governed workflows.
  • DGEval is intended as an ongoing safety-assurance instrument as model capabilities change, not as a point-in-time verdict.

Appendix A: Evaluation Metrics Detail

The benchmark combines question-level partial-credit scoring with weighted aggregation across subsections, DGL columns, and the four equally weighted sections.

  • Evaluation Metrics Detail: Partial credit scores questions with multiple correct answers using n, m, and k, penalising over-prediction symmetrically with under-prediction.
  • Evaluation Metrics Detail: Sections 1, 2, and 4 use normalised subsection weights to calculate weighted averages of subsection mean scores.
  • Evaluation Metrics Detail: Section 3 applies the same weighted-aggregation formula with DGL columns replacing subsections.
  • Evaluation Metrics Detail: The final benchmark score averages the results of the four sections with equal section weights.

Appendix B: E-learning Subsection Details

Appendix B documents the e-learning subsections, their danger-criticality weights, and their relevance to practitioner courses.

  • E-learning Subsection Details: The table also records subsection relevance to the Consignor/Freight Forwarder, Packer/Cargo Handler, Ship Loader, and Ship Operator courses.
  • E-learning Subsection Details: Table A1 lists the 20 e-learning subsections evaluated in Sections 1, 2, and 4, alongside expert-assigned scores and normalised benchmark weights.
  • E-learning Subsection Details: Two in-house NCB Hazcheck DG experts assigned danger-criticality scores through independent ratings followed by consensus, with weights proportional to those scores.
  • E-learning Subsection Details: Subsections E54, E55, E56, and E69 appear only in non-specific Standard and Advanced courses.

Appendix C: DGL Column Descriptions and Weights

Appendix C lists the DGL fields evaluated in Section 3 and specifies how their importance weights are assigned.

  • DGL Column Descriptions and Weights: Table A2 lists 16 evaluated DGL properties with their descriptions and importance weights.
  • DGL Column Descriptions and Weights: UN Number and PSN each receive 25% weight, while the remaining 14 columns share the other 50% proportionally to expert scores.

Appendix D: LLM Judge Validation

The appendix validates an LLM judge for Section 2 by comparing its ratings and explanations with expert review. Results support judge reliability, especially for score validity, while explanation-quality agreement between experts is weaker.

  • Validation design: 90 response pairs from three low-thinking models were judged across five quality dimensions, producing 450 evaluation records.The records covered Factual Accuracy, Completeness, Relevance and Conciseness, Clarity and Readability, and Overall Coherence.
  • Human review: 100 of the 450 records were randomly sampled for independent review by two in-house dangerous-goods experts.Experts assessed agreement with the judge’s score and the quality of its textual justification.
  • Judge alignment: 74.00% and 71.00% JAR values indicated broad expert agreement with the judge for Experts 1 and 2, respectively.The corresponding 95% confidence intervals were [64.63, 81.60] and [61.46, 78.99].
  • Judge alignment: 73.00% and 78.00% EUI values indicated that experts rated the judge’s explanations as precise in those respective evaluations.The confidence intervals overlapped, indicating consistent evaluations.
  • Inter-expert agreement: κ = 0.7932 and Pa = 96.19% showed strong inter-expert agreement on score validity, versus κ = 0.1228 and Pa = 66.00% for explanation quality.The lower explanation-quality kappa was associated with high chance agreement and a skewed label distribution; overall results supported judge reliability.

Appendix E: Benchmark Access

DGEval uses controlled release to preserve the benchmark as an independent evaluation tool. Access is governed by licensing conditions, and submissions return scores without exposing the questions or answers.

  • Controlled release: DGEval withholds benchmark questions and answer keys to prevent direct training on the test set and preserve long-term evaluation value.The approach follows established submission-based benchmarks that return official scores without exposing hidden test labels.
  • Submission access: The planned evaluation server requires a licence agreement and returns benchmark scores without exposing underlying questions or answers.The authors state that further access details are available on request.
  • Licensing boundary: Access to DGEval does not authorize reproduction or redistribution of IMO publications, and users remain responsible for required IMDG Code licences.The access terms separate benchmark use from rights to access or use IMO materials.
Loading 2608.21036v1…