PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152

Vahid Zolfaghari, Nenad Petrovic, AndrÉ Schamschurko, Alois Knoll

arXiv:2608.16394v1cs.AIcs.IR

TL;DR

LLMs can struggle to apply regulatory numerical values accurately when generating test scenarios. RegulaRAG combines reference-aware chunking and retrieval with reranking. It achieves an average Meta-Score of 82.99, 43% above the next-best baseline, NoRAG at 57.94.

  • Problem

    LLMs often fail to extract and apply numerical values accurately when generating regulation-based scenarios.

    Source: B. Evaluation system · p. 6

  • Method

    RegulaRAG uses SmartChunking, reference-aware enrichment, and semantic retrieval with reranking to generate test scenarios from regulatory standards.

    Source: §II · p. 2, §VI · p. 14

  • Results

    82.99 average Meta-Score, 43% above next-best NoRAG (57.94), was RegulaRAG’s highest average across the evaluated language models.

    Source: 2) RAG system comparison · p. 10, §VI · p. 14

  • Takeaways & Limitations

    RegulaRAG offers a favorable balance between accuracy, runtime, and token usage for generating scenarios from complex technical standards.

    Source: §VI · p. 14

    Evaluation was exclusive to UN Regulation No. 152, and differing document structures may require retrieval-hyperparameter retuning before deployment.

    Source: Limitations · p. 14

Abstract

from arXiv · show

Generating regulation-compliant test scenarios is essential for validating safety-critical automotive systems, yet Large Language Models (LLMs) struggle to ground outputs in long, hierarchical standards. We present RegulaRAG, a Retrieval-Augmented Generation (RAG) pipeline that couples SmartChunking, reference-aware enrichment of paragraphs and tables via graph traversal, with Smart Retrieve & Rerank over these enriched units. To test our system, we evaluate on a manually curated dataset covering all scenarios in UN Regulation No. 152 (AEBS). Our study comprises: (i) a three-step progressive search that identifies near-optimal retrieval parameters without exhaustive grid search; (ii) head-to-head comparisons against five baseline RAG systems; and (iii) a robustness stress test that scales the source corpus with distractor content. Outputs are evaluated using a customized penalized scoring metric. Across all experiments, RegulaRAG achieves the highest average Meta-Score (82.99), outperforming the next-best system by 43% (NoRAG: 57.94), while operating at 14k-25k tokens per query versus up to 500k for graphcentric baselines. It maintains strong performance, remaining stable even as the number of regulatory sources grows, whereas competing RAG systems degrade sharply in both quality and robustness.

I. INTRODUCTION · II. LITERATURE REVIEW

RegulaRAG targets the difficulty of generating traceable, regulation-compliant automotive test scenarios from long, interconnected standards. It combines reference-aware document enrichment with selective retrieval to produce simulation-ready scenarios rather than compliance verdicts or invented stress cases.

I. INTRODUCTION

  • LLMs can hallucinate, miss rare details and numerical conditions, and struggle with long regulations; supplying full documents also raises token-cost, confidentiality, and context-dilution concerns.These limitations make precise, regulation-compliant scenario generation difficult to automate reliably.Source: §I · p. 1
  • RegulaRAG combines hierarchical PDF chunking, graph-based enrichment of linked paragraphs and tables, and query-aware retrieval and reranking to surface dispersed regulatory evidence compactly.Its core novelty is BFS traversal over explicit cross-references; bounded depth, deduplication, and metadata compression are described as scalability optimizations.Source: §I · p. 1

II. LITERATURE REVIEW

  • Automotive standards challenge retrieval through fragmented tables, technical terminology, varied clause wording, and cross-references that standard chunk retrieval can miss.Tables may span pages, while requirements may be distributed across clauses and annexes.Source: §II · p. 2
  • Prior compliance approaches produce verdicts, assessment text, explanations, code, or classifications, [10], [11], [12], [13],, whereas RegulaRAG generates traceable, simulation-ready scenarios from regulatory clauses and tables.The cited systems address process-rule alignment, safety assessments, document-level inconsistency analysis, financial rule translation, knowledge-graph QA, or standards checks.Source: §II · pp. 2–3
  • RDR2, TableRAG, and Oliveira et al. address document structure, table integrity, or legal relations, but not the combined task of reconstructing regulatory evidence and generating scenarios.RegulaRAG uses a rule-based heuristic to reconstruct fragmented PDF tables, BFS reference-closure enrichment, and compliance-grounded scenario text.Source: §II · p. 2
  • Rather than perform general entity and relation extraction, RegulaRAG uses numbered headings and embedded table pointers as a lightweight proxy for knowledge linkage, avoiding LLM-heavy extraction passes.This design is motivated by nested legal clauses and domain-specific cross-references that make reliable open-domain extraction challenging.Source: §II · pp. 2–3
  • Although evaluated on UN Regulation No. 152, the workflow is intended for other similarly structured UN regulations, with prompts updated while its core algorithms remain unchanged.The paper also says it could potentially be adapted to other languages with little modification.Source: §II · p. 3
  • Unlike ADS scenario systems that create diverse or challenging cases,,,,, RegulaRAG extracts regulation-faithful scenarios prioritizing completeness, traceability, and compliance.Its natural-language scenarios can feed downstream code-generation systems, as illustrated by Lebioda et al..Source: §II · p. 3

III. SYSTEM DESCRIPTION

RegulaRAG structures lengthy regulations into retrieval-ready evidence so LLMs can generate regulation-compliant test scenarios. Its three phases extract and normalize the document, enrich chunks with referenced content, and retrieve and assemble context for generation.

System overview

  • RegulaRAG’s three phases extract and normalize regulations, create enriched chunks, then retrieve and assemble evidence for LLM scenario generation.The pipeline’s goal is to extract and structure relevant parts of lengthy regulation documents for regulation-compliant test scenarios.Source: §III · p. 3

Phase 1: Extraction

  • MinerU converts regulation PDFs to normalized Markdown, preserving stable IDs while aligning chunk boundaries with paragraph hierarchy and exposing cross-references.Normalization converts paragraph numbers into canonical headers, repairs extraction artifacts, and standardizes tables.Source: §III · pp. 3–4

Phase 2: Chunking (SmartChunking)

  • SmartChunking segments structured text into coherent base chunks, then uses BFS over a reference graph to add recursively referenced headings and tables.It deduplicates closure content and caps expansion at 50 nodes per chunk; reference metadata is stored as integer IDs or pointers until context assembly.Source: §III · p. 4

Phase 3: Retrieval and Generation

  • Smart Retrieve and Rerank scores queries against enriched chunk representations, enabling matches through referenced tables or sub-paragraphs even when base text scores poorly.Selected base chunks and their closure items are assembled into the final LLM context.Source: §III · p. 4
  • Context assembly expands selected chunks’ references, removes duplicates by stable ID, and orders the evidence by regulation hierarchy before concatenation.The paper states that canonical ordering restores logical flow and, with referenced tables merged into their defining clauses, reduces hallucinations and improves numerical consistency in generated scenarios.Source: §III · pp. 4–5

A. Dataset description

The dataset contains manually extracted and categorized AEBS test scenarios from UN Regulation No. 152, built by combining general test conditions with scenario-specific requirements. It covers three scenario categories and is publicly released on Hugging Face.

  • The manually extracted and categorized dataset is publicly available on Hugging Face at vahidzolf/un_152.Source: A. Dataset description · p. 5
  • The dataset covers Car-to-Stationary-Car, Car-to-Moving-Car, and Car-to-Pedestrian scenarios, representing stationary targets, moving vehicles, and crossing pedestrians, respectively.Car-to-Stationary-Car tests vary vehicle load and approach speed; the other categories consider relative speeds and specified conditions, or pedestrian speeds and trajectories.Source: A. Dataset description · p. 5
  • Each scenario entry has a unique identifier, test title, and detailed description of its specific test conditions.Source: A. Dataset description · p. 5

Dataset construction and verification

  • The ground-truth dataset was constructed from UN Regulation No. 152 by combining general conditions distributed across regulation sections with scenario-specific numerical requirements.The researchers instantiated category templates by varying parameters according to regulatory values, then reviewed scenarios for consistency and crosschecked them against the source sections and tables.Source: A. Dataset description · p. 5

B. Evaluation system

The section evaluates generated scenarios against manually curated UN Regulation No. 152 ground truth, augmenting semantic similarity with explicit checks for compliance-critical numerical values and conditions.

B. Evaluation system

  • Generated scenarios are compared with manually curated ground truth using 384-dimensional sentence-transformer embeddings and cosine similarity.The scoring model is sentence-transformers/all-MiniLM-L6-v24, distinct from the retrieval model and selected for large-scale pairwise scoring.Source: B. Evaluation system · p. 6
  • LLMs often reproduce scenario wording but assign incorrect numerical values or confuse laden and unladen conditions, producing incorrect post-condition values.The examples include test speeds and the AEBS-required final speed for M1 vehicles.Source: B. Evaluation system · p. 6
  • The penalized score compares normalized numbers through tolerance-based matching and loading-condition terms through Jaccard distance, applying λ = 0.2 per discrepancy.The method extracts numbers and categorical terms with regex families and combines their discrepancy with cosine similarity; incorrect critical values can reduce scores below θ = 0.9.Source: B. Evaluation system · pp. 6–7
  • The scoring framework is largely domain-agnostic, requiring adaptation of categorical keywords and tuning of λ and θ for a new domain.Its numeric matching, cosine-similarity backbone, and Meta-Score transfer without modification.Source: B. Evaluation system · p. 7
  • Because regulations lack a universal compliance-accuracy metric, F1 serves as a scenario-level compliance proxy, while requirement, parameter, and traceability coverage remain future metrics [27]–.The proxy relies on ground-truth scenarios encoding regulation-specific pass/fail criteria, tolerances, and traceability.Source: B. Evaluation system · p. 7

IV. EXPERIMENTAL SETUP · A. Language Models

RegulaRAG separates retrieval from generation, evaluating the pipeline with three LLMs chosen to span deployment settings rather than to identify a best generator. Shared prompts and decoding settings keep retrieval comparisons controlled, while model snapshots and fixed settings support reproducibility.

A. Language Models

  • The study uses gpt-4o-2024-11-20, Llama 3.3 70B, and DeepSeek-chat to represent proprietary API, open, and low-cost API deployment settings.Pinned model snapshots were preferred for reproducibility.Source: A. Language Models · p. 8
  • RegulaRAG selects regulatory content upstream of generation, making the language model an interchangeable pipeline component rather than part of the contribution.A different or newer generator can be substituted without changing the pipeline.Source: A. Language Models · p. 8
  • Using the same generator, prompts, scenario definitions, and decoding settings across RAG systems isolates retrieval strategy in comparisons.Retrieval hyperparameters are shared across LLMs and determined via grid search.Source: A. Language Models · p. 8
  • DeepSeek-chat is selected over DeepSeek-reasoner because it offers competitive accuracy at over 12× lower input-token cost ($0.14 vs. $1.74 per million tokens on a cache miss).The chat variant is the practical choice for large-scale pipeline execution.Source: A. Language Models · p. 8
  • The deepseek-chat endpoint has no pinned checkpoint version, so its underlying model may change without a change to the model string.Source: A. Language Models · p. 8
  • All models use low temperature (τ = 0.1) to reduce output variance, with decoding settings held fixed across methods.Source: A. Language Models · p. 8

B. Evaluation Protocol … 1) Hyperparameter selection via grid search:

The evaluation uses a staged design to select retrieval settings, compare RAG systems under matched conditions, and test corpus-scaling robustness. The grid search identifies top k=30 and chunk size=2000 as a near-optimal RegulaRAG configuration for GPT-4o on CtoStC.

B. Evaluation Protocol

  • Scenario-level Precision, Recall, and F1 are reported, with F1 primary; matching requires penalized cosine similarity of at least θ=0.9.Numeric scenario values use absolute and relative tolerances of 0.05; failing runs are retried up to two times, and each configuration has a single run without trial-level dispersion statistics.Source: B. Evaluation Protocol · p. 8

C. Compute and Reproducibility

  • The experiments fix random seeds where applicable, prompt templates, and decoding parameters, and make the dataset, code, templates, and evaluation CSVs publicly available.Runs use a Linux workstation with a GTX 1080 Ti, Intel Core i7-3770 CPU, 32 GB RAM, and Python 3.10.12; commercial models require API keys.Source: C. Compute and Reproducibility · p. 8
  • Retrieval uses all-mpnet-base-v2 to embed enriched chunks and queries, while all-MiniLM-L6-v2 computes scenario-pair cosine similarity for evaluation.The two sentence-transformer models are from sentence-transformers v3.4.1.Source: C. Compute and Reproducibility · p. 8

D. Prompting System

  • All LLMs receive identical prompts without advanced prompting techniques, using a shared system prompt and three scenario-specific user prompts.Each user prompt includes one structural example; prompts avoid instructions for deriving scenarios to support task focus and automated ground-truth evaluation.Source: D. Prompting System · p. 8

A. Experiment Design

  • The experiment design varies six RAG systems, three scenario families, top k values, chunk sizes, corpus scale, and language models.Corpus scaling adds eight automotive-software UN regulations, while GPT-4o, DeepSeek-Chat, and Llama 3.3 probe model effects; a progressive design avoids an exhaustive, confounded grid.Source: A. Experiment Design · p. 9

1) Hyperparameter selection via grid search:

  • At top k=30 and chunk size=2000, RegulaRAG reaches its peak F1 in the grid search, and top k=30 is used for retrieval-based baselines.The sweep uses GPT-4o on CtoStC with UN-152 as the sole source; the response surface has broad plateaus and some sharp maxima.Source: 1) Hyperparameter selection via grid… · p. 9

2) RAG system comparison.:

RegulaRAG combines reference-aware chunk enrichment with smart retrieval and reranking, and achieves the strongest overall Meta-Score in the comparison. The section also explains how its robustness-adjusted metric and handling of cross-referenced tables relate to performance.

RAG system comparison

  • RegulaRAG compares queries against enriched chunks and referenced content, then retrieves deduplicated original and reference chunks for context assembly.For tables split across PDF pages, it attaches consecutive HTML table blocks indicated by cues such as “the following table,” preserving their order to reduce omissions and numerical hallucinations.Source: 2) RAG system comparison · pp. 9, 11
  • The study-specific Meta-Score aggregates F1 across three LLMs while penalizing variability, so it reflects both average performance and robustness.Its formulation uses a coefficient-of-variation dispersion penalty; highly inconsistent results can produce negative Meta-Scores.Source: 2) RAG system comparison · p. 10
  • 82.99 average Meta-Score makes RegulaRAG the top system, ahead of second-best NoRAG at 57.94 by approximately 43%.RegulaRAG’s individual-model Meta-Scores are 80.99 for GPT-4o, 87.90 for DeepSeek-chat, and 80.08 for LLaMA-3.3.Source: 2) RAG system comparison · p. 10
  • 82.99 versus 8.88 average Meta-Score for RegulaRAG and R&R+RCS, respectively, indicates the joint contribution of reference-aware enrichment and smart reranking.R&R+RCS shares RegulaRAG’s embedding model and RecursiveCharacterSplitter chunking but omits those two components; the paper notes that a fully factorial component ablation remains future work.Source: 2) RAG system comparison · pp. 10–11
  • CtoP is the easiest category, while CtoStC and CtoMoC are more retrieval-sensitive because their requirements span paragraphs and shared tables; weaker retrieval also destabilizes model results.DeepSeek-chat and LLaMA-3.3 show instability or baseline difficulties, particularly for table interpretation and paragraph aggregation, while GPT-4o is more stable under RegulaRAG than under baseline RAG systems.Source: 2) RAG system comparison · p. 11

B. input document scalability

As the regulatory corpus grows, RegulaRAG remains computationally compact but its retrieval quality declines, while HippoRAG sustains high accuracy at much greater runtime and token cost.

Evaluation setup

  • The scalability test expanded the retrieval corpus from one regulation to bundles of eight, holding k = 30 and chunk size∗= 3000 constant to isolate corpus-size effects.The eight-regulation pool varied in document length and structural complexity.Source: B. input document scalability · pp. 11–12

Runtime and token scaling

  • RegulaRAG’s stable runtime is attributed to optimized enrichment, including bounded BFS expansion, while HippoRAG’s runtime growth is driven by LLM-heavy OpenIE and graph construction.RegulaRAG also removes deep copies, avoids storing enriched text, and compresses metadata into integer references; HippoRAG’s token-intensive stages include NER, triple extraction, graph expansion, reranking, and auxiliary calls.Source: B. input document scalability · p. 13

Retrieval quality, runtime, and token usage

  • As the corpus grows, REGULARAG’s F1 declines to 82.35% while runtime stays 47–59 s and tokens 14k–25k; HIPPORAG sustains 97–100% F1 but exceeds 200 s and approaches 500k tokens.HIPPORAG token usage rises from 34k on one document to nearly 500k on the full set.Source: Figure 7 · p. 13, Figure 8 · p. 13, Figure 9 · p. 13

C. Failure Mode Analysis

Manual inspection finds three recurring errors across evaluated RAG systems: wrong numerical assignments, confusion between laden and unladen conditions, and misleadingly high cosine similarity.

C. Failure Mode Analysis

  • LLMs often preserve scenario structure but misassign regulated numbers, including CtoStC speeds (20 km/h, +0/−2 km/h) or values from the Maximum Relative Impact Speed table.The required CtoStC speed is read from regulation text and must be applied before braking.Source: C. Failure Mode Analysis · p. 13
  • Confusing laden and unladen states produces wrong post-condition values; for example, the AEBS may need to reduce speed to 0 km/h, as specified by the regulation table.Generated scenarios may sound plausible while remaining numerically incorrect.Source: C. Failure Mode Analysis · p. 13
  • Faithful scenario templates can yield cosine similarity near 1.0 despite incorrect numerical content, so Algorithm 2’s penalized metric penalizes numeric and categorical discrepancies.The penalization is intended to surface errors hidden by raw similarity scores.Source: C. Failure Mode Analysis · pp. 13–14
  • Reference-aware enrichment supplies relevant table chunks to address the first two errors, but exact numeric misreading despite supplied tables remains out of scope and is planned for future prompting or constrained decoding.The model may misread or ignore explicit table values even when the correct table is present in context.Source: C. Failure Mode Analysis · p. 14

VI. CONCLUSION AND FUTURE WORK · Limitations

RegulaRAG achieves the paper’s strongest reported results at substantially lower context cost. The authors propose extending and integrating the pipeline, while noting limits in evaluation scope, tuning, and component-level evidence.

VI. CONCLUSION AND FUTURE WORK

  • 82.99 average Meta-Score, 43% above NoRAG (57.94), is RegulaRAG’s top result, with 14k–25k tokens per query versus HippoRAG’s up to 500k on an eight-document corpus.The pipeline combines SmartChunking, reference-aware enrichment, and semantic retrieval.Source: §VI · p. 14
  • Future work targets chunk-enrichment, scoring, and ranking refinements, broader regulatory domains, and integration with automotive simulators and real testbench infrastructure.Examples named are CARLA and Car-Maker.Source: §VI · p. 14

Limitations

  • Evaluation covers only UN Regulation No. 152, though the core pipeline could transfer to similarly structured standards after replacing its UN-152 load-condition regex vocabulary.The passage identifies hierarchical numbered paragraphs, explicit cross-references, and tabular specifications as shared structural properties; it says the cosine-similarity backbone, numeric matching, penalty weight λ, and Meta-Score formula transfer unchanged.Source: Limitations · p. 14
  • top k=30 and chunk size=2000 were tuned on UN-152 CtoStC, so different document density, cross-reference depth, or chunk-length distributions may require retuning.Prior work [30] suggests GPT-4o and GPT-3.5-turbo may be less affected by an untuned top k; that evidence does not cover chunk size, and retuning both remains advisable.Source: Limitations · p. 14
  • The same subject-matter experts built the dataset and conducted qualitative evaluation, creating potential consistency bias that role separation or independent validation could reduce.Source: Limitations · p. 14
  • RegulaRAG enriches and reorders references but does not reason over a full knowledge graph; explicit graphs may capture deeper cross-reference semantics at higher complexity and token/runtime cost.The passage cites the difficulty of extracting entities and relations reliably from regulatory legal language and the growth of KG pipeline costs with corpus size.Source: Limitations · p. 14
  • Each configuration was evaluated once, and no ablation isolated BFS reference-closure depth, deduplication, or canonical chunk reordering.Baseline comparisons provide partial ablation evidence: R&R+RCS isolates BFS enrichment and smart reranking, while NoRAG isolates the retrieval pipeline.Source: Limitations · p. 14

APPENDIX A · USER PROMPTS FOR SCENARIO GENERATION

Appendix A provides scenario-specific prompts using UN Regulation No. 152 context to generate structured M1 AEBS test scenarios for CtoStC, CtoMoC, and CtoP. The prompts require complete speed coverage and a tightly specified output format.

APPENDIX A

  • The prompts target all M1 test scenarios in three families: Car-to-Stationary-Car (CtoStC), Car-to-Moving-Car (CtoMoC), and Car-to-Pedestrian (CtoP).They use regulation context and follow the structured format of examples defined in UN Regulation No. 152.Source: USER PROMPTS FOR SCENARIO GENERATION · p. 15
  • For CtoStC, generate every test speed in the regulation tables and keep the system active across the M1 Maximum Relative Impact Speed range.The prompt specifies that the technical service is required to test any speed.Source: USER PROMPTS FOR SCENARIO GENERATION · p. 15
  • Each test case must begin with a scenario title, followed immediately by its full description and exactly one separator line.For CtoP, the description must use the example's exact wording, adapting only relevant parameters.Source: USER PROMPTS FOR SCENARIO GENERATION · p. 15
  • The prompts require strict adherence to the example structure and prohibit explanations, notes, extra text, index numbers, or other identifiers.Only the requested scenarios and separators should be output.Source: USER PROMPTS FOR SCENARIO GENERATION · p. 15
  • For CtoMoC, include all speeds defined in the regulation tables and keep the system active across the M1 Maximum Relative Impact Speed range.The prompt states that the technical service is required to test any speed.Source: USER PROMPTS FOR SCENARIO GENERATION · p. 15
  • For CtoP, disregard speeds in the supplied context, generate the complete regulation-table speed set, and keep the system active across the M1 Maximum Impact Speed range.The prompt requires using the example wording while adapting only relevant parameters.Source: USER PROMPTS FOR SCENARIO GENERATION · p. 15

Related papers

Loading 2608.16394v1…