PaperlayerView exact revision on arXiv

Source-linked AI summary

From Regulation to Implementation: A Critical Evaluation of LLM-Assisted Regulatory Compliance in Industry

Adriana Watson, Marco Bücheler, Grant Richards

arXiv:2608.21317v1cs.AI

TL;DR

The paper addresses limited evidence about how prompting and regulatory vagueness affect LLM-generated compliance artifacts, an issue spanning heterogeneous DPP data and underspecified DPIA documentation. It benchmarks models across DBP and DPIA tasks using repeated runs and gold-standard schema comparisons. Results indicate that DPIA outputs generally improve with more context, while DBP outputs are comparatively stable across context levels; the study remains limited by theoretical case studies.

  • Problem

    Research has not adequately examined how prompting differences affect LLM-generated compliance artifacts despite challenges from heterogeneous DPP data and vague DPIA requirements.

  • Method

    The study benchmarks five models across DBP and DPIA tasks, four vagueness levels, and repeated runs, comparing consolidated outputs with manually created gold-standard schemas.

  • Results

    DPIA outputs generally become more consistent and complete with higher-context prompts, whereas DBP outputs remain comparatively stable across context levels.

  • Takeaways & Limitations

    Prompt design requires greater care for less-defined artifacts such as DPIAs, while established DBP frameworks can reduce dependence on detailed prompts.

  • Takeaways & Limitations

    The study analyzes a small set of theoretical case studies rather than real-world DBP and DPIA documentation.

Abstract

from arXiv · show

The European Union (EU) has emerged as a leading regulatory body in the development of sustainability and privacy regulations. While new regulation requirements vary, many include a documentation artifact to ensure compliance. Notably, the Ecodesign for Sustainable Products Regulation (ESPR) introduces Digital Product Passports (DPPs) for life cycle transparency, while the General Data Protection Regulation (GDPR) mandates Data Protection Impact Assessments (DPIAs) to mitigate privacy risks. Creating these compliance artifacts, however, is challenging. Industrial data, which often exists in heterogeneous formats and is scattered across company and supplier systems, is required for DPPs and can be difficult to extract into compliant DPP formatting. Furthermore, DPIA documents require interdisciplinary expertise and follow no standardized format, making development difficult for novel systems. To address the particular complexity of compliance artifact creation for both regulations, researchers have proposed the use of LLMs in the generation process; however, the impact of the aforementioned problems on the output of these systems is largely unaddressed. This work investigates the existing research gap by exploring how data extraction instructions and regulatory vagueness impact the quality and consistency of LLM-produced compliance artifacts. The resulting artifacts are evaluated by benchmarking different models against manually created ground-truth schemas. The results reveal that less strict guidelines, such as DPIA formatting, require higher context prompts to maintain consistency and completeness. Stricter guidelines, such as formatting for Digital Battery Passports (DBP), result in consistent results regardless of prompt context, but may lead to more hallucinations in the output

I. INTRODUCTION

The paper examines LLM-assisted generation of EU compliance artifacts, focusing on how regulatory structure and prompting affect DPP and DPIA production. It addresses challenges from heterogeneous industrial data, vague DPIA requirements, and limited research on prompting effects.

  • Research gap: Prior work applies LLMs to AAS creation, regulation interpretation, compliance verification, and initial DPIA generation, but prompting effects remain underexplored.The paper therefore studies how LLM-generated artifacts vary with prompt specificity across distinct compliance artifacts.
  • Regulatory context: ESPR DPPs document product life cycles, while GDPR DPIAs document high-risk data-processing assessments.DPPs require product, supply-chain, material, and environmental information; DPIAs address processing purposes, risks, and mitigation.
  • DPP generation: Industrial DPP generation requires combining data scattered across company and supplier systems and stored in heterogeneous IT and OT formats.The AAS provides interoperable submodels for structuring product information.
  • DPIA generation: DPIA creation is difficult because GDPR specifications are vague, interpretations vary, and no universal development method is required.The literature also identifies a tension between organizational risk assessments and DPIAs’ focus on risks to individuals.

C. Key Problems with LLM-Generated Artifacts

LLM-generated compliance artifacts face risks from hallucination, specialized-domain interpretation, and data leakage. The study evaluates prompting across DPP and DPIA tasks using structured inputs and multiple models.

  • Risks: LLMs may hallucinate invalid structures, information, or scenarios when generating compliance artifacts.Hallucination means inventing material rather than retrieving it from real sources.
  • Risks: Specialized terminology and logic can create interpretation problems in DPP and DPIA generation.These artifacts require domain knowledge that broad training data does not guarantee models can apply correctly.
  • Risks: Data leakage is a concern because DPP and DPIA workflows may process protected internal or private external data.The paper identifies direct attacks and unintentional verbatim reproduction as leakage routes.
  • Experimental setup: The case study generates ESPR/EU Battery Regulation-compliant DBPs and GDPR-compliant DPIAs with five models across four vagueness levels.Each configuration used three runs, yielding 120 total runs.
  • DBP prompting: DBP prompts progressively add AAS structure, regulatory requirements, domain context, extraction constraints, and anti-hallucination instructions.The baseline and low-context prompts leave more field-selection responsibility to the model, whereas higher-context prompts prescribe mandatory data and structure.

2) DPIA Prompting Strategy:

The DPIA prompting strategy holds the role assignment and industrial scenario constant while varying the amount and specificity of regulatory and structural guidance.

  • Shared scenario: All DPIA prompts use the same compliance-expert role and high-risk parcel-logistics scenario.The scenario covers 18 EU centers, approximately 12,000 workers, sensitive data streams, automated scheduling, rankings, and HR-related flags.
  • Baseline prompting: The baseline prompt supplies quoted GDPR Article 35(7) text and Recital 84 clarification before requesting a structured JSON DPIA.The output must contain four top-level sections mapped to Article 35(7)(a)–(d).
  • Low context: The low-context prompt simplifies the output instruction and asks for sections covering components the model considers necessary.It retains JSON-only formatting but provides less explicit structure than the other prompts.
  • Medium context: The medium-context prompt adds a plain-language summary covering processing description, necessity and proportionality, risks, and mitigation.Its requirements address data flows, retention, lawful basis, data-subject rights, vulnerable groups, and residual risk.
  • High context: The high-context prompt expands requirements into individual sections and mandates their inclusion in the output.This adds explicit structural guidance beyond the baseline, low-context, and medium-context variants.

C. Evaluation

The evaluation compares model outputs with manually created gold-standard schemas to measure regulatory completeness and consistency across tasks, vagueness levels, and models.

  • Gold standards: Gold-standard schemas for DBP and DPIA scenarios were based on legal, industrial, and academic structures and examples.The schemas were designed to reflect current best practices.

1) Schemas:

The study uses distinct gold-standard schemas for DBP and DPIA artifacts, reflecting their different regulatory structures and source templates.

  • The DBP schema uses the Digital Battery Passport Part 6: Material Composition submodel and an exemplary industrial battery as its reference file.
  • The DPIA schema combines narrative and linear published templates into sections with Boolean and written responses.
  • Each DPIA field is marked as required when mandated by GDPR or recommended by existing templates and documentation.

2) Methodology for Analysis:

The analysis evaluates generated artifacts for output stability and regulatory completeness by comparing repeated runs and consolidated outputs against gold-standard schemas.

  • Consistency measures whether the same model, task, and vagueness level produce stable structures and content across repeated runs.
  • The gold-standard schema used for evaluation is available in the linked codebase but omitted from the paper because of length restrictions.
  • A field is stable when it appears in every repeated run, yielding a stability score of 1.0.
  • Regulatory compliance compares consolidated model outputs with gold-standard schema requirements using required-component inclusion and completeness.

IV. RESULTS

Results differ between DBP and DPIA artifacts: DBP outputs are generally highly consistent and complete, whereas DPIA performance varies more with context and model. DBP compliance deficits are concentrated in hazardous-substance fields.

  • 1) Consistency:: 1.00 cross-run consistency was maintained by Qwen-2.5 and GPT-4o across every DBP context level.
  • 1) Consistency:: 0.77 was Claude’s minimum DBP consistency at medium context, while high context restored it to 1.00.
  • 2) Compliance:: 1.00 DBP completeness was achieved by all models at baseline, while Claude fell to 0.97 ± 0.06 at low context and 0.87 ± 0.08 at medium context.
  • 2) Compliance:: Seven of ten DBP required fields reached 100% inclusion for every model, with differences confined to hazardous-substance fields.
  • 2) Compliance:: 75% inclusion of Hazardoussubstanceconcentration was observed for Claude, compared with 92% for Mistral-7B; GPT-4o, Llama 3.1, and Qwen-2.5 reached 100% across all fields.
  • 1) Consistency:: 0.80 was Mistral’s highest reported DPIA consistency, whereas GPT-4o reached only 0.49 when given regulation text alone.
  • 1) Consistency:: DPIA models generally improved consistency and completeness with more context, although Claude remained comparatively stable but average across context levels.

2) Compliance:

LLM-generated GDPR artifacts were more complete and consistent with greater prompt context, whereas structured DBP artifacts remained stable across vagueness levels but showed model-specific compliance risks.

  • GDPR compliance: 93% complete was achieved by GPT-4o and Mistral at the highest context levels, while Llama-3.1 reached 49% at the lowest context level.Most models improved in completeness as context increased, whereas Claude remained stable but average across context levels.
  • GDPR compliance: Required-field inclusion varied substantially across GDPR artifacts, with compliance mechanisms, residual risks, and data subject rights frequently excluded.Identified risks, data-subject categories, data categories, and technical measures were always included, while several other fields varied by model.
  • DBP compliance: Three of four models maintained near-perfect DBP compliance and consistency across vagueness levels, unlike the open-ended GDPR requirements.The authors suggest that the concrete AAS submodel field structure compensates for prompt ambiguity.
  • DBP compliance: Claude’s lower- and medium-context DBP performance may risk omitting legally required hazardous-substance fields, while fully specified prompts cost more tokens than minimal prompts.Claude achieved full compliance at baseline and high context; the authors therefore favor fully specified or minimal prompts over partial context.
  • Regulatory implications: More context generally improved GDPR consistency and completeness, indicating that regulation alone may not provide enough specificity for reliable artifact generation.The authors connect frequently missed fields to requirements or wording located elsewhere in the GDPR.
  • Conclusion: Further research is needed to define prompting best practices before industrial implementation, and the EU could formalize output specifications in delegated acts and guidance.The paper reports distinct problems for clearly defined and vaguely defined compliance artifacts.

VII. APPENDIX

The appendix provides a high-context DBP prompt that directs an LLM to generate a structured AAS JSON representation from regulatory, product, and template inputs.

  • DBP prompt: The prompt requests a Digital Battery Passport as valid JSON mapped explicitly to required AAS submodel fields.It prohibits prose outside the JSON output.
  • DBP prompt: The Material Composition submodel covers manufacturer information, electrical characteristics, efficiency, resistance, temperature boundaries, lifetime indicators, and power capability.These are presented as structured battery properties.
  • Extraction constraints: The prompt instructs extraction only from explicitly supported material-composition documentation, preserving field names and JSON structure while leaving unavailable values empty.It also requires metric units and prohibits hallucinating missing values.
  • DBP prompt: The supplied inputs comprise regulatory context, battery material-composition documentation, and an AAS Material Composition submodel template.The prompt frames the model as a compliance, AAS, and battery-material-composition expert.
  • Product selection: For the specified BoilerCell Battery, material masses, hazardous-substance concentrations, and CRM flags must come from Sections 4–6, excluding the shared-components register.The product is identified as NMC with a total pack mass of 450.0 kg.

B. GDPR High Context Prompt

The GDPR high-context prompt structures a DPIA around processing description, necessity and proportionality, risk assessment, and mitigation measures.

  • Processing description: The prompt requires descriptions of personal-data categories, data subjects, purposes, lawful bases, data flows, retention periods, and recipients.These elements form the processing-description portion of the DPIA.
  • Necessity and proportionality: The necessity-and-proportionality section asks whether each data category is strictly necessary and whether less privacy-intrusive alternatives could achieve the same outcomes.It also includes data minimization and purpose limitation measures.
  • Rights and retention: The prompt also addresses data-subject rights and storage-limitation justification.The listed rights include access, rectification, erasure, restriction, objection, and portability.
  • Risk assessment: The risk-assessment section requires specific risks to data subjects, likelihood and severity ratings, an overall processing risk level, and heightened risks for vulnerable subgroups.Examples include unlawful automated employment decisions, surveillance chilling effects, discriminatory profiling, data breaches, and function creep.

C. Code and Datasets

The appendix identifies locations for the study’s code, ground-truth files, material used for DPP generation, and the remaining GDPR prompt content.

  • Resources: The study’s code, gold-standard ground-truth data, and files are available through a linked repository.The appendix also identifies a separate link for material used in DPP generation.
  • GDPR prompt: The remaining GDPR prompt content specifies technical and organizational risk-mitigation measures, automated-decision contest mechanisms, residual risk, and supervisory-authority consultation.These requirements complement the earlier processing, proportionality, and risk-assessment sections.
Loading 2608.21317v1…