PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152

Vahid Zolfaghari, Nenad Petrovic, AndrÉ Schamschurko, Alois Knoll

arXiv:2608.16394v1cs.AIcs.IR

TL;DR

LLMs struggle to generate regulation-compliant automotive test scenarios from long, hierarchical standards, especially when extracting precise numerical conditions. RegulaRAG combines reference-aware document enrichment with retrieval and reranking, achieving the strongest reported performance while using substantially fewer tokens than graph-centric alternatives.

  • Problem

    LLMs struggle to extract precise numerical values and conditions from long, hierarchical regulations when generating compliant automotive test scenarios.

  • Method

    RegulaRAG combines SmartChunking, reference-aware enrichment of paragraphs and tables, and semantic retrieval with reranking for regulation-based scenario generation.

  • Results

    82.99 average Meta-Score, 43% above NoRAG at 57.94, made RegulaRAG the strongest and most stable system across tested language models.

  • Takeaways & Limitations

    RegulaRAG maintained strong performance as regulatory sources grew while using 14k–25k tokens per query, compared with up to 500k for graph-centric alternatives.

  • Takeaways & Limitations

    The system was developed and evaluated exclusively on UN Regulation No. 152, and its retrieval hyperparameters may require retuning for regulations with different structures.

Abstract

from arXiv · show

Generating regulation-compliant test scenarios is essential for validating safety-critical automotive systems, yet Large Language Models (LLMs) struggle to ground outputs in long, hierarchical standards. We present RegulaRAG, a Retrieval-Augmented Generation (RAG) pipeline that couples SmartChunking, reference-aware enrichment of paragraphs and tables via graph traversal, with Smart Retrieve & Rerank over these enriched units. To test our system, we evaluate on a manually curated dataset covering all scenarios in UN Regulation No. 152 (AEBS). Our study comprises: (i) a three-step progressive search that identifies near-optimal retrieval parameters without exhaustive grid search; (ii) head-to-head comparisons against five baseline RAG systems; and (iii) a robustness stress test that scales the source corpus with distractor content. Outputs are evaluated using a customized penalized scoring metric. Across all experiments, RegulaRAG achieves the highest average Meta-Score (82.99), outperforming the next-best system by 43% (NoRAG: 57.94), while operating at 14k-25k tokens per query versus up to 500k for graphcentric baselines. It maintains strong performance, remaining stable even as the number of regulatory sources grows, whereas competing RAG systems degrade sharply in both quality and robustness.

I. INTRODUCTION · II. LITERATURE REVIEW

LLMs can support regulation-compliant scenario generation but struggle with hallucinations, precise parameters, conditional test distinctions, and long automotive standards. Prior work addresses document processing, compliance verification, and scenario generation, whereas RegulaRAG reconstructs traceable, simulation-ready scenarios from regulatory clauses, tables, and cross-references.

  • I. INTRODUCTION: LLMs face hallucinations, outdated knowledge, rare-information gaps, and difficulty grounding outputs in long regulatory documents.RAG integrates external documents during inference to improve accuracy and traceability.
  • I. INTRODUCTION: Without RAG, LLMs may miss numerical parameters, confuse laden and unladen conditions, and incur high token usage when processing full regulations.These challenges persist even with context windows of up to 100k tokens.
  • II. LITERATURE REVIEW: Automotive PDF processing is difficult because multi-page tables may be split during conversion and technical standards contain specialized legal and domain terminology.Similarity-based retrieval can therefore misinterpret fragmented tables and technical language.
  • II. LITERATURE REVIEW: Existing compliance frameworks combine RAG with structured knowledge, but they primarily produce verdicts, explanations, classifications, or code rather than simulation-ready test scenarios.These approaches improve compliance reasoning, transparency, or computational enforcement without directly generating domain-grounded scenario artifacts.
  • II. LITERATURE REVIEW: RegulaRAG combines SmartChunking, reference-aware enrichment, and selective retrieval to reconstruct traceable scenario definitions from clauses, tables, and cross-references.It operationalizes regulatory text into simulation-ready test scenarios rather than static compliance outputs.
  • II. LITERATURE REVIEW: Unlike general knowledge-graph pipelines, RegulaRAG avoids open-domain entity and relation extraction, using document-embedded pointers as a lightweight, structure-preserving linkage mechanism.This design reflects nested clauses, conditional syntax, and domain-specific cross-references in regulatory standards while avoiding graph-expansion token costs.
  • II. LITERATURE REVIEW: Although evaluated on UN Regulation No. 152 for AEBS, the workflow is intended for other regulations with hierarchical paragraphs, annexes, and interlinked references.The passage also indicates potential adaptation to other languages with limited changes.
  • II. LITERATURE REVIEW: Driving-scenario studies generate diverse, challenging, or formalized ADS test cases, whereas RegulaRAG extracts regulation-compliant scenarios directly from standards for traceable safety verification.Its contribution bridges scenario generation and formal safety requirements through structured extraction rather than invention of novel corner cases.

III. SYSTEM DESCRIPTION

RegulaRAG structures lengthy regulations for compliant scenario generation through extraction, reference-aware SmartChunking, and enriched retrieval with canonical context assembly. Its design resolves internal dependencies, removes redundancy, and restores regulatory order before supplying evidence to the LLM.

  • Pipeline overview: The pipeline has three phases: extraction and normalization, reference-aware SmartChunking, and Smart Retrieve and Rerank over enriched chunks.Its goal is to extract and structure relevant regulation content so LLMs can generate regulation-compliant test scenarios.
  • Phase 1: Extraction: Phase 1 converts PDFs to normalized Markdown, canonicalizes paragraph numbers as headers, repairs extraction artifacts, and standardizes tables while preserving stable IDs.The resulting structure preserves paragraph hierarchy and supports detectable cross-references for later enrichment, retrieval, and reranking.
  • Phase 2: Chunking (SmartChunking): SmartChunking segments structured text into semantic chunks and uses breadth-first search to resolve referenced headings and tables into each chunk’s reference closure.This makes chunks self-contained with respect to internal regulatory dependencies and records defined headings, referenced headings, and referenced tables.
  • Phase 3: Retrieval and Generation: Retrieval computes query similarity against enriched representations combining each base chunk with its reference closure, allowing matches through referenced tables or subparagraphs.The method then ranks candidates and selects the top-k chunks for context construction.
  • Context assembly: The final context expands references, deduplicates repeated regulatory elements by stable IDs, and restores canonical regulation order before presenting the evidence block to the LLM.This ordering and table-clause merging produces logically structured context intended to reduce hallucinations and improve numerical consistency.

A. Dataset description

The authors construct and publicly release a manually curated dataset of AEBS test scenarios derived from UN Regulation No. 152. It covers three scenario categories and records each case with its identifier, title, and detailed test conditions.

  • Dataset description: The dataset contains manually extracted and categorized test scenarios derived from UN Regulation No. 152 for evaluating LLM-based scenario generation.The dataset is publicly released on Hugging Face at vahidzolf/un_152.
  • Dataset description: The scenarios are organized into three categories: Car-to-Stationary-Car (CtoStC), Car-to-Moving-Car (CtoMoC), and Car-to-Pedestrian (CtoP).CtoStC varies vehicle load and approach speeds, CtoMoC considers relative speeds, and CtoP concerns pedestrians crossing at predefined speeds and trajectories.
  • Dataset construction and verification: Each scenario entry includes a unique identifier, test title, and detailed textual description of the specific test conditions.The ground-truth dataset was constructed by systematically reviewing the regulation and consolidating conditions distributed across multiple sections.

B. Evaluation system

The evaluation compares generated scenarios with manually curated UN Regulation No. 152 ground truth using semantic similarity and a penalization mechanism for critical discrepancies. Structured extraction of numerical and loading-condition terms makes scoring sensitive to compliance-relevant details while supporting broader domain transfer.

  • Evaluation procedure: Generated scenarios are evaluated against manually curated ground-truth scenarios for accuracy and completeness.The RAG system retrieves regulatory context, generates scenarios, and compares them directly with the reference dataset.
  • Semantic similarity: Cosine similarity compares 384-dimensional sentence embeddings produced by sentence-transformers/all-MiniLM-L6-v24.The metric is selected for efficient, scale-invariant, and robust semantic comparison across text-length variations.
  • Compliance-sensitive extraction: Numeric and load/condition key sets expose critical compliance attributes that cosine similarity may miss.Text is normalized before extracting numerical values and categorical regulation-relevant terms, including loading and motion conditions.
  • Penalized scoring: λ = 0.2 penalizes combined numeric and categorical discrepancies while tolerating minor lexical variation.The discrepancy combines tolerance-based one-to-one numeric F1 matching with Jaccard distance over categorical term sets.
  • Metric generalizability: The penalized F1 metric is regulation-agnostic, while Meta-Score ( ¯F1 −σ) summarizes mean performance and cross-condition stability.Transferring the framework requires replacing load terms and retuning λ and θ on representative data; the remaining framework is unchanged.
  • Compliance-oriented metrics: Reported F1-score serves as a proxy for scenario-level compliance rather than a universal automotive compliance-accuracy metric.The manually constructed UN Regulation No. 152 scenarios encode regulation-specific pass/fail criteria, tolerances, and traceability, with penalties checking critical numerical values.

IV. EXPERIMENTAL SETUP · A. Language Models · B. Evaluation Protocol

The experimental setup evaluates RegulaRAG with three interchangeable language models and a scenario-level protocol centered on F1, cosine-threshold matching, retries, and runtime tracking. Prompts require scenarios to follow UN Regulation No. 152 with detailed vehicle, speed, TTC, loading, and post-condition constraints.

  • A. Language Models: gpt-4o-2024-11-20 is accessed via the OpenAI API as one of three representative language models.
  • A. Language Models: llama-3.3-70b-versatile, identified as Llama 3.3 70B, is served through the Groq API.
  • A. Language Models: DeepSeek-chat is invoked through the vendor’s direct API as the third representative language model.
  • A. Language Models: RegulaRAG operates upstream of generation by selecting which regulation content reaches the prompt, while the language model remains interchangeable.
  • B. Evaluation Protocol: Precision, Recall, and F1 are reported at scenario level, with F1 designated as the primary metric.
  • B. Evaluation Protocol: Failing runs are automatically retried up to two additional times and marked ok after the first successful completion; corpus-scaling tests additionally record end-to-end runtime.
  • B. Evaluation Protocol: Scenario matching uses θ = 0.9, counting a generated scenario as a true positive only when penalized similarity to its best-matching ground-truth scenario meets the threshold.

C. Compute and Reproducibility … 1) Hyperparameter selection via grid search:

The study fixes its computational and prompting conditions for reproducibility, then systematically evaluates RAG design choices and selects a robust configuration before cross-family pipeline comparison.

  • C. Compute and Reproducibility: Experiments used a Linux workstation with a GTX 1080 Ti, Intel Core i7-3770 CPU, 32 GB RAM, and Python 3.10.12.Random seeds, prompt templates, and decoding parameters were fixed; the dataset and reproducibility materials were publicly available.
  • C. Compute and Reproducibility: Retrieval used all-mpnet-base-v2 with 768-dimensional embeddings, while evaluation used all-MiniLM-L6-v2 with 384-dimensional embeddings.The models served separate pipeline stages: similarity-based top-k selection and pairwise cosine similarity scoring, respectively.
  • D. Prompting System: Identical prompts were used across LLMs, with one system prompt and three user prompts targeting Car-to-Stationary-Car, Car-to-Moving-Car, and Car-to-Pedestrian scenarios.Each user prompt included one example as a structural template and omitted scenario-derivation instructions.
  • A. Experiment Design: The experiment decomposed interacting pipeline components into controllable factors, exploring performance, robustness, and scalability systematically rather than exhaustively.The design varied RAG system, scenario family, retrieval breadth, and chunking granularity.
  • A. Experiment Design: Six RAG systems were evaluated across three scenario families, with retrieval breadth varied over top k ∈ {10, 15, 20, 25, 30}.The systems were RegulaRAG, R&R+RCS, NoRAG, OpenAI-RAG, HippoRAG, and Hybrid; the families were CtoStC, CtoMoC, and CtoP.
  • 1) Hyperparameter selection via grid search:: The full grid sweep tested top k ∈{10, 15, 20, 25, 30} and chunk size ∈{800, 1000, 1400, 1600, 2000, 3000, 4000} for RegulaRAG with GPT-4o on CtoStC using UN-152 alone.The sweep was designed to estimate robust retrieval hyperparameters.
  • V. RESULTS AND EXPERIMENTS: top k=30 and chunk size=2000 formed the stabilized configuration, despite broad response-surface plateaus and a few sharp maxima.The resulting configuration was fixed before evaluating six RAG pipelines across all three scenario families on UN–152.

2) RAG system comparison.:

RegulaRAG combines reference-aware SmartChunking with Smart Retrieve and Rerank, and achieves the strongest overall Meta-Score and stability across the evaluated language models. Its gains are especially pronounced against retrieval baselines that omit reference enrichment or smart reranking.

  • RegulaRAG and baselines: RegulaRAG combines reference-aware enrichment with Smart Retrieve and Rerank, comparing enriched chunks with queries before assembling deduplicated original and referenced content.The pipeline retrieves over enriched units, then gathers corresponding original chunks and appends reference chunks.
  • Overall comparison: 82.99 was RegulaRAG’s average Meta-Score across GPT-4o, DeepSeek-chat, and LLaMA-3.3, ahead of NoRAG at 57.94, Hybrid at 56.71, and HippoRAG at 55.75.The Meta-Score rewards average F1 while penalizing variability across the three language models.
  • Overall comparison: 43% was the approximate improvement over the second-best system, while RegulaRAG scored 80.99 for GPT-4o, 87.90 for DeepSeek-chat, and 80.08 for LLaMA-3.3.Competing methods showed greater variability, including a negative Meta-Score for DeepSeek-chat under R&R+RCS.
  • Ablation evidence: 82.99 versus 8.88 was the averaged Meta-Score gap between RegulaRAG and R&R+RCS, which shared the embedding model and RecursiveCharacterSplitter but omitted BFS enrichment and smart reranking.This comparison provides partial ablation evidence for the joint contribution of reference-aware BFS enrichment and smart reranking.
  • Scenario and model differences: CtoP was easiest, whereas CtoStC and CtoMoC were more retrieval-sensitive because they required integrating information dispersed across paragraphs and shared tables.CtoP is anchored in a self-contained regulatory section with a dedicated table.
  • Scenario and model differences: DeepSeek-chat and LLaMA-3.3 became unstable or struggled substantially under weaker baselines, while RegulaRAG improved or remained competitive, particularly for precise table interpretation and paragraph aggregation.DeepSeek-chat showed several negative Meta-Scores for R&R+RCS and OpenAI-RAG; LLaMA-3.3 struggled especially on table and paragraph integration.

B. input document scalability

As the retrieval corpus expands from one regulation to eight, RegulaRAG preserves stable runtime and compact token usage, while its retrieval accuracy declines on increasingly heterogeneous inputs. HippoRAG maintains higher accuracy but incurs steep computational and token-growth costs.

  • Experimental setup: The corpus scales from a single regulation to bundles containing eight documents, with k = 30 and chunk size∗ = 3000 held constant.The setup isolates semantic clutter and cross-document interactions by varying only corpus size.
  • Runtime stability: 47–59 s: REGULARAG maintains stable runtime across all corpus sizes, whereas HIPPORAG runtime grows steeply.RegulaRAG’s scaling is attributed to optimized enrichment using bounded BFS expansion, compressed metadata, and removal of deep copies and stored enriched text.
  • Token usage scaling: 14k–25k tokens: REGULARAG remains compact, while HIPPORAG rises from 34k tokens on one document to nearly 500k on the full set.HippoRAG’s growth reflects multiple LLM-dependent stages, including NER, triple extraction, graph expansion, reranking, and auxiliary calls.
  • Retrieval quality: 97–100% F1: HIPPORAG delivers consistently strong accuracy, while REGULARAG declines to 82.35% as the corpus becomes more heterogeneous.The comparison reflects a trade-off between HippoRAG’s graph-structured reasoning and RegulaRAG’s computational efficiency and targeted paragraph-level retrieval.

C. Failure Mode Analysis

Manual inspection identifies three recurring failure patterns across evaluated RAG systems: incorrect numerical assignments, loading-condition confusion, and cosine-similarity inflation. RegulaRAG improves grounding for the first two through reference-aware table enrichment, while penalized scoring exposes discrepancies that similarity metrics can hide.

  • Failure modes: Incorrect numerical value assignment occurs when LLMs reproduce scenario structure but extract or apply regulation values inaccurately.For example, CtoStC requires 20 km/h with a +0/−2 km/h tolerance before braking, but models may substitute values absent from the regulation.
  • Failure modes: Loading condition confusion causes incorrect post-condition values when LLMs fail to distinguish laden from unladen conditions.The correct post-condition depends on loading state and must be read from the regulation table.
  • Failure modes: Cosine similarity can approach 1.0 despite incorrect numerical content because generated scenarios preserve the ground-truth template structure.This inflates raw similarity scores and can conceal substantive errors.
  • Mitigation: Reference-aware enrichment addresses numerical and loading failures by ensuring the relevant table chunk appears in retrieved context.Exact numeric hallucination may persist even when the correct table is supplied, because the LLM can misread or ignore explicit values.

VI. CONCLUSION AND FUTURE WORK · Limitations

RegulaRAG is presented as a regulation-agnostic pipeline that balances accuracy, runtime, and token usage for regulation-compliant scenario generation. The authors identify transferability, parameter tuning, evaluation design, and deployment integration as priorities for future work.

  • VI. CONCLUSION AND FUTURE WORK: RegulaRAG combines SmartChunking, reference-aware enrichment, and semantic retrieval in a two-stage pipeline for generating regulation-compliant test scenarios.It was designed for complex technical standards, including UN Regulation No. 152.
  • VI. CONCLUSION AND FUTURE WORK: Future work will refine chunk enrichment, scoring, and retrieval ranking, then generalize RegulaRAG to additional regulatory documents and domains.The authors also envision integrating generated scenarios with CARLA, CarMaker, and real testbench infrastructure.
  • Limitations: RegulaRAG was developed and evaluated exclusively on UN Regulation No. 152, although its core pipeline and scoring components are described as regulation-agnostic.Transferability to other regulations therefore remains a limitation requiring validation.
  • Limitations: top k=30 and chunk size=2000 were tuned by grid search on the CtoStC scenario family using UN-152 alone.Regulations with different document density, cross-reference depth, or chunk-length distributions may require parameter re-tuning before deployment.
  • Limitations: Each configuration was evaluated in a single run without repeated-trial statistics because of resource constraints.The baseline comparison offers partial ablation evidence, but component-level isolation of BFS closure depth, deduplication, and canonical chunk reordering was not conducted.

APPENDIX A · USER PROMPTS FOR SCENARIO GENERATION

Appendix A presents three scenario-specific user prompts for generating structured AEBS test scenarios from UN Regulation No. 152. The prompts cover CtoStC, CtoMoC, and CtoP cases for M1 vehicles and impose exhaustive speed coverage and strict output formatting.

  • APPENDIX A: Appendix A contains the user prompts used for regulation-compliant AEBS scenario generation.The appendix identifies the scenario-generation prompts as part of the paper’s supplementary material.
  • USER PROMPTS FOR SCENARIO GENERATION: Three prompts target Car-to-Stationary-Car, Car-to-Moving-Car, and Car-to-Pedestrian scenario families.Each prompt receives RAG context from UN Regulation No. 152 and requests M1-vehicle test scenarios.
  • USER PROMPTS FOR SCENARIO GENERATION: CtoStC prompts require all test cases across the regulation-table speed range for M1 vehicles.They also require system activity within the Maximum Relative Impact Speed range.
  • USER PROMPTS FOR SCENARIO GENERATION: Every scenario must begin with a title, continue with its full description, and end with exactly one separator line.This structure is explicitly repeated for CtoStC, CtoMoC, and CtoP prompts.
  • USER PROMPTS FOR SCENARIO GENERATION: The prompts prohibit explanations, notes, extra text, index numbers, and other identifiers beyond the requested scenarios and separators.They require strict adherence to the provided example structure.
  • USER PROMPTS FOR SCENARIO GENERATION: Each prompt supplies an example test case and instructs the model to start generation immediately.The examples are represented by the placeholder {example_test_case}.
  • USER PROMPTS FOR SCENARIO GENERATION: CtoMoC prompts require inclusion of all speeds defined in the regulation tables for M1 vehicles.The system must remain active within the M1 Maximum Relative Impact Speed range.
  • USER PROMPTS FOR SCENARIO GENERATION: CtoP prompts disregard contextual speeds and generate the complete speed set defined in the regulation tables.They specify the M1 vehicle range using Maximum Impact Speed and require exact example wording with only relevant parameters adapted.
Loading 2608.16394v1…