ARCCS: An Automated Regulatory Compliance Checking System

• Manual compliance review is high-stakes, slow, expensive, and hard to scale because it demands legal interpretation, applicability judgments, and explicit evidence.Source: §1 · p. 1
• Existing legal tools target individual artefacts or regulations, leaving no end-to-end route from an unseen regulation to an auditable requirement-level decision.Source: §1 · p. 1
• ARCCS converts unseen regulations into atomic, traceable requirements and evaluates target documents using grounded evidence, abstention, and human deferral.Source: §1 · p. 2
• The evaluation spans GDPR policy-document assessment with multiple LLM judges and an EU procurement benchmark of more than 1,200 rule-level checks.Source: §1 · p. 2
• ARCCS is presented as a free, fully open-source compliance workflow that generates structured, auditable reports through an application requiring no programming or pipeline configuration.Source: §1 · p. 2

2.1 Regulatory Processing and Extraction Module

• RPEM preserves legal structure, converts regulatory chunks into traceable atomic requirements, and removes non-enforceable or duplicate constraints while retaining source pointers.Each requirement records its legal function, regulated actor, scope, mandatory conditions, and originating provision.Source: §2.1 · p. 2

2.2 Compliance Classification Module

• CCM excludes out-of-scope requirements, retrieves relevant target-document passages, and assigns evidence-based labels, confidence scores, and justifications citing evidence and source provisions.Retrieved evidence, rather than the model’s parametric memory, conditions the compliance decision.Source: §2.2 · pp. 2–3

2.3 Compliance Labels

• Four labels distinguish supported compliance or non-compliance, insufficient documentary evidence, and cases deferred to experts for ambiguity, conflict, or external context.Insufficient Information reflects missing documentary evidence; Human Required reflects evidence needing expert interpretation.Source: §2.3 · p. 3
• For each applicable requirement, the report pairs its decision with confidence, supporting evidence, and the governing provision to support end-to-end audit and human review.The report entry is (yi, κi, Ji, srci).Source: §2.3 · p. 4

3.1 Setting 1: GDPR Policy-Document Compliance

• For GDPR, three applications’ terms of use are assessed against 100 derived requirements, with 30 sampled decisions per document judged by three evaluator models.Each judge receives the regulation, extracted requirement, relevant document evidence, and full ARCCS output; the sample totals 90 judged instances.Source: §3.1 · p. 4

3.2 Setting 2: Public-Procurement Event-Log Compliance

• The procurement setting covers 12 rules from Articles 4, 48, 56, and 73 of Directive 2014/24/EU, not the full Directive.The rules are tested on 100 TED-schema traces: 60 with at least one targeted violation and 40 compliant controls, with labels derived independently. The authors describe this as a controlled sanity check, not evidence of full real-world coverage.Source: §3.2 · p. 4
• The procurement benchmark yields 1,200 rule-level decisions and reports accuracy, precision, recall, F1, case-level performance, and exact match.Exact match requires all 12 rule decisions for a trace to be correct; evaluation uses gpt-5.4, gpt-5-mini, and gpt-5.2.Source: §3.2 · p. 4

3.3 Models

• Experiments use GPT-5 API models without model-specific fine-tuning or local GPU requirements; gpt-5.2 is the default, with medium reasoning and a 0.60 Human Required threshold.The included tables report sampled evaluator accuracy and agreement on binary verdicts.Source: §3.3 · p. 4, Table 1 · p. 5, Table 2 · p. 5

4.1 Policy-Document Compliance

• 90.00–96.67% of sampled GDPR cases were judged consistent with the regulation and evidence, with substantial overall evaluator agreement (Fleiss’ κ = 0.69, p < 0.001).The two evaluators gpt-5.2 and gpt-5.2-pro agreed perfectly (κ = 1.00, p < 0.001).Source: §4.1 · p. 5

4.2 Public-Procurement Logs

• 98.5–98.8% accuracy and 98.8–100.0% recall across models show strong rule-level procurement performance, with gpt-5.2 and gpt-5-mini strongest on precision, recall, and F1.The rule-level benchmark covers 100 traces and 1,200 decisions.Source: Table 3 · p. 5, §4.2 · p. 5

4.1 Policy-Document Compliance

• ARCCS flagged potential conflicts in WhatsApp and ChatGPT terms, including one later confirmed by Ireland’s regulator, while leaving final legal interpretation to experts.The ChatGPT flag concerned Article 79 and mandatory arbitration and an exclusive San Francisco forum clause.Source: §4.1 · p. 5

4.2 Public-Procurement Logs

• 100.0% on all case-level metrics for gpt-5.2 and gpt-5-mini contrasts with gpt-5.4’s 98.0% accuracy and 98.4% F1.All models correctly flagged every violating trace.Source: §4.2 · p. 5, Table 4 · p. 6
• 87.0% exact match for gpt-5-mini was the best score when all twelve decisions had to be correct, with errors attributed to isolated extra false alarms.Its Insufficient Information rate was 2.3%, versus 7.3% for gpt-5.2 and 8.8% for gpt-5.4.Source: Table 5 · p. 6, §4.2 · p. 6

5 Demonstration

• Users select or upload a regulation, upload a target document, then launch analysis against the extracted requirements through a three-step wizard.Preloaded regulations such as the GDPR reuse cached requirements.Source: §5 · p. 6
• Each report row links a requirement’s verdict and confidence to its governing provision, quoted document evidence, and natural-language justification.Human Required decisions below the 0.60 confidence threshold and Insufficient Information cases are identified for manual review.Source: §5 · p. 6
• The application runs locally, retains run history, and exports structured reports; only the user-selected language-model provider is contacted externally.A self-hosted open-weight model can support fully air-gapped deployments.Source: §5 · p. 6

6 Conclusion

• The openly available, on-premise demo positions ARCCS as a transparent tool for human-in-the-loop regulatory review.The system produces an auditable compliance report from a regulation and target document.Source: §6 · pp. 6–7
• ARCCS decomposes regulations into atomic, traceable requirements and assesses documents requirement by requirement using retrieved evidence, confidence scores, and explanations.The system does not rely on a fixed checklist or hand-crafted rules.Source: §6 · p. 6
• The evaluation reports strong performance in both policy-document compliance assessment and deterministic procurement checking, including accurate rule-level violation detection.The conclusion presents the same architecture as supporting both assessment settings.Source: §6 · p. 6

Limitations

• The GDPR evaluation lacks deterministic ground truth and relies on GPT-5-family judges, including GPT-5.2 as the default backbone; expert legal evaluation remains future work.Some evaluator pairs showed only moderate, statistically non-significant agreement.Source: Limitations · p. 7
• Procurement results come from balanced, controlled TED-schema traces, so performance on naturally occurring, noisier logs may differ.The benchmark ground truth reflects an operational formalisation of the Directive, not adjudicated legal outcomes.Source: Limitations · p. 7
• The study does not compare simpler prompting or vanilla RAG, systematically ablate pipeline stages, or independently evaluate requirement extraction, whose errors may propagate.Source: Limitations · p. 7
• Evaluation covers 90 judged GDPR instances, 100 procurement traces under one Directive, two regimes, and English; generalisation elsewhere remains un demonstrated.Reported experiments also use only the GPT-5 family through a commercial API, and thresholds were not systematically calibrated.Source: Limitations · p. 7
• The controlled benchmark does not establish false-Compliant failure prevalence in natural settings, and consequential determinations should remain under human oversight.Source: Limitations · p. 7

Ethical Considerations

• ARCCS must not make final, consequential compliance decisions without expert oversight.The paper also says deployment in a new jurisdiction or legal domain requires validation by qualified legal experts.Source: Ethical Considerations · p. 7
• False negatives could create a false sense of safety, so ARCCS is tuned conservatively and surfaces confidence and evidence for review prioritisation.The system also presents Non-Compliant flags as candidate conflicts for human verification and makes abstention explicit through Insufficient Information and Human Required labels.Source: Ethical Considerations · p. 7

A Demonstration Walkthrough

• Users upload a regulation and target document in sequence, then inspect a report with label counts and per-requirement evidence, confidence, and justification.The interface also streams execution logs and preserves prior runs for auditing and cross-document comparison.Source: A Demonstration Walkthrough · p. 10

Compliant example

• For GDPR Article 2, the WhatsApp terms are Compliant because they reference a separate Privacy Policy without directly conflicting with the material-scope provision.The example reports confidence 0.78.Source: B Qualitative Example · p. 10

Non-Compliant example

• The GDPR Article 5(1)(d) example is Non-Compliant because WhatsApp disclaims accuracy and the explanation finds no commitment to maintain or rectify it.The example assigns confidence 0.62.Source: B Qualitative Example · p. 10

Insufficient Information example

• For GDPR Article 1, ARCCS returns Insufficient Information because the Netflix excerpt contains no relevant statements about personal-data processing or protection.The system abstains rather than guessing when the document does not speak to the obligation.Source: B Qualitative Example · p. 10

Human Required example

• The Netflix example is deferred as Human Required because the cited clauses neither clearly violate nor satisfy the obligation and confidence is below 0.70.The example concerns promotional eligibility information and a consent-gated remote-access support tool.Source: B Qualitative Example · pp. 10–11

Across examples

• Across the examples, ARCCS gives definite verdicts when evidence supports them and abstains or defers when the document is uninformative or uncertain.The outputs are intended for expert review rather than automatic legal determination.Source: B Qualitative Example · pp. 10–11

C Implementation Details

• RPEM extracts typed regulatory objects and filters incomplete or non-actionable items; CCM retrieves document evidence and returns a label, confidence, quotes, and justification.The fixed priority is contradiction → Non-Compliant; no relevant information → Insufficient Information; confidence below 0.60 → Human Required; otherwise Compliant.Source: C Implementation Details · p. 11

C.1 Error Analysis

• All 14 residual procurement errors for gpt-5-mini are false positives, with zero false negatives; nine of twelve rules are perfect and errors cluster on three temporal rules.The reported pattern is conservative over-flagging on temporal reasoning, with temporal normalisation identified as a promising improvement target.Source: Table 6 · p. 12, §C.1 · p. 12

D Model Cards

• The experiments use specified model snapshots, with gpt-5.2 as the default ARCCS backbone, medium reasoning effort, and a fixed 0.60 Human Required threshold.Models are accessed through the OpenAI API without model-specific fine-tuning.Source: D Model Cards · p. 12, Table 7 · p. 12

E Full List of Procurement Rules

• The twelve procurement rules derive from Articles 4, 48, 56, and 73 and span monetary limits, duplicate publication, award-procedure timing, and contract lifecycle consistency.Examples include a 70-day publication-to-award limit and constraints on contract start, end, and termination.Source: E Full List of Procurement Rules · pp. 12–13

F Terms-of-Use Documents

• The GDPR terms-of-use evaluation compares WhatsApp, Netflix, and ChatGPT documents of differing length and specificity; the longer WhatsApp document yields more determinate outcomes.The authors attribute this pattern to the longer document providing more evidence for grounding decisions.Source: F Terms-of-Use Documents · p. 13, Table 8 · p. 13

G Public-Procurement Benchmark Details

• The procurement benchmark derives twelve rules from Directive 2014/24/EU and evaluates temporally ordered TED traces using deterministic checks independent of ARCCS outputs.The rules cover monetary constraints, duplicate publication, award-procedure timing, and contract lifecycle consistency.Source: G Public-Procurement Benchmark Details · p. 13

H Compliance Label Distribution

• Insufficient Information is the dominant terms-of-use outcome because public documents often lack the internal governance evidence needed to assess GDPR obligations.The longer WhatsApp document provides more retrievable evidence and yields more determinate outcomes and more cases routed to Human Required; conflict flags remain intended for expert review.Source: H Compliance Label Distribution · pp. 13–14, Table 9 · p. 13

I Intended Users and Deployment

• ARCCS is intended for legal, compliance, data-protection, and procurement practitioners who may not have machine-learning engineering expertise.The paper also identifies smaller organisations without dedicated ML infrastructure as intended users.Source: I Intended Users and Deployment · p. 14
• The free, fully open-source application exposes the workflow through a browser interface, while allowing technical users to inspect, adapt, or extend its codebase.Reports and run history are stored locally for review and repeated analyses.Source: I Intended Users and Deployment · p. 14