ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detection

1 Introduction

• Over 60,000 regulatory events were tracked across 190 countries in 2022, and global banks have paid over $300 billion in fines and settlements since the 2008 financial crisis.The passage says this volume far exceeds human compliance teams’ capacity.Source: §1 · p. 1
• Automated regulatory compliance remains underserved: no end-to-end system bridges NLP advances to production-ready monitoring, while commercial GRC platforms rely heavily on rules and manual curation.Prior systems also target single frameworks, whereas ComplianceNLP jointly handles three.Source: §1 · p. 1
• ComplianceNLP monitors regulatory changes, extracts obligations, and identifies policy gaps across SEC, MiFID II, and Basel III, covering approximately half of annual update volume.Its KG-augmented RAG achieves 94.2% grounding accuracy (r=0.83 vs. human judgments).Source: §1 · pp. 1–2
• 87.7 gap detection F1 at δ=0.6 and 85.3 macro F1 at δ=0.45, where Full Gap recall reaches 91.2%.The deployment threshold is recall-optimized.Source: §1 · p. 2
• The Regulatory Knowledge Graph links 12,847 provisions, with 94.7% edge precision and 87.3% estimated recall.The system’s contribution list also reports four months of parallel-run deployment over 9,847 updates, with estimated 96.0% recall.Source: §1 · p. 2
• Domain-specific distillation from 70B to 8B models plus Medusa decoding yields 2.8× speedup, with 91.3% token acceptance attributed to regulatory text’s low entropy.The system also reports sub-second p50 inference.Source: §1 · p. 2

2 Related Work

• DERECHA reports 89.1% precision for GDPR compliance checking but assumes pre-structured policy clauses and targets one framework; ComplianceNLP reports 90.7% production precision across three frameworks.The cited passage points to Table 4 for the production-scale comparison.Source: §2 · p. 2
• Sun et al. (2025) propose a RAG-based compliance checker, while ComplianceNLP differs in multi-framework coverage, typed-graph grounding, learned obligation extraction, and four months of deployment evidence.The passage says none of the cited related systems handles multi-framework obligation extraction with production deployment.Source: §2 · p. 2
• Related foundations span legal benchmarks and models, KG-enhanced RAG, hallucination checking, and efficient inference; Zagyva et al. (2025) demonstrated Medusa plus knowledge distillation at Booking.com.ComplianceNLP adapts that setup for compliance and reports domain-specific findings on acceptance rates.Source: §2 · p. 2

3 The ComplianceNLP System

• The pipeline ingests regulatory documents into a vector store and knowledge graph, extracts obligations, then maps them against internal policies for gap reports.The extraction module uses LEGAL-BERT and feeds a separate LLaMA-3 generator; the models share no parameters.Source: §3 · pp. 2–3, Figure 1 · p. 3

3.1 Regulatory RAG Pipeline

• Hybrid retrieval combines dense and BM25 scores, then re-ranks the top five passages using graph proximity to linked provisions.The dense encoder is fine-tuned on 50K regulatory passage pairs; α=0.7 and β=0.3 weight the retrieval and KG scores, respectively.Source: §3.1 · p. 3
• The Regulatory Knowledge Graph contains 12,847 provision nodes and 34,219 edges, with 94.7% edge precision and estimated recall of 87.3% ± 3.8%.It is updated nightly; new provisions are inserted incrementally, with affected embeddings recomputed within a two-hop neighborhood.Source: §3.1 · p. 3
• During the maximum 18-hour synchronization blind spot, retrieval falls back to embeddings only; newly added provisions are excluded from KG re-ranking until validation.For time-critical announcements, the passage describes this embedding-only fallback as the expected operating mode.Source: §3.1 · p. 3

3.2 Multi-task Obligation Extraction

• Obligation extraction jointly trains NER, deontic-modality classification, and cross-reference resolution over a shared LEGAL-BERT encoder.The evaluation annotations are partitioned by document to prevent leakage; cross-reference resolution has the lowest reported inter-annotator agreement (κ=0.78).Source: §3.2 · pp. 3–4

3.3 Compliance Gap Analysis

• Each obligation is aligned to policy clauses using dense similarity multiplied by learned fuzzy type matching; low alignment is flagged as a potential gap.The generator classifies cases as COMPLIANT, PARTIAL GAP, or FULL GAP using modality, entity types, and RKG enforcement history, then produces cited reports with severity and remediation recommendations.Source: §3.3 · p. 4

4 Production Optimization

• In isolation, KD has numerically higher NER retention than SFT (96.8% vs. 95.1%), though confidence intervals overlap; domain Medusa heads retain ≥97.4% across tasks.The table describes latency-accuracy trade-offs for the distilled student model without system components.Source: Table 1 · p. 4
• End-to-end document processing takes 4.2 min/doc, while “real-time” refers specifically to the generator LLM’s inference-latency constraint.The full document process includes extraction, retrieval, and gap analysis.Source: §4 · p. 4
• Distilling LLaMA-3-70B into LLaMA-3-8B and adding three Medusa heads yields 2.8× combined speedup (659 ms p50).Regulatory text has entropy H=2.31 bits versus 3.87 on C4, with 91.3% token acceptance versus 82.7% for general-text heads.Source: §4 · p. 4

Datasets

• REGOBLIGATION contains 1,847 annotated regulatory sentences from SEC, MiFID II, and Basel III, while GAPBENCH contains 423 obligation–policy pairs from one institution.REGOBLIGATION annotations cover NER entity types, deontic modalities, and cross-references; GAPBENCH labels compliant, partial-gap, and full-gap cases.Source: §5 · p. 4
• The evaluation also uses ObliQA’s 5,574 questions and the COLING 2025 Challenge’s 312 questions.GAPBENCH reports κ=0.81 and a ±4.2 95% confidence interval for Full Gap F1.Source: §5 · p. 4

Baselines

• Baselines span non-retrieval models, retrieval-augmented models, and a regulatory QA system; results are averaged over three seeds.Paired-bootstrap significance testing uses 10,000 resamples.Source: §5 · p. 5

6 Results and Analysis

• 87.7 gap detection F1 and +3.5 gap detection F1 over GPT-4o+RAG are reported at δ=0.6; gains across four tasks are statistically significant.The reported gains are +2.7 NER F1, +2.2 Deontic F1, and +5.1 QA F1; GPT-4o+RAG has 85.1% grounding accuracy.Source: §6 · p. 5
• At δ=0.45, the deployment threshold prioritizes Full Gap recall (91.2%) at the cost of lower precision, while Full Gap F1 remains similar across thresholds.The reported 95% confidence interval for Full Gap F1 is ±4.2; analysis of 17 missed Full Gaps identifies implicit obligations (35%), multi-hop cross-references (29%), and jurisdiction-specific nuance (21%).Source: Table 3 · p. 5
• Removing KG re-ranking causes the largest ablation drop (−4.6 gap F1), while removing MiniCheck lowers grounding accuracy from 94.2% to 86.7% with minimal F1 impact.Removing multi-task training reduces NER by 2.2 F1; additive analyses find KG re-ranking contributes +2.2 or +2.5 gap F1, depending on the starting model.Source: §6 · p. 5
• On 150 held-out documents, extraction errors reduce F1 from 87.7 to 83.4, with NER boundary errors accounting for 68% of propagated errors.At δ=0.45, false alerts rise to 3.7 per day while missed obligations fall to 0.7 per document; analysts find this trade-off acceptable.Source: §6 · pp. 5–6
• Basel III has the lowest cross-reference F1 (81.2), compared with 85.3 for MiFID II and 89.7 for SEC; its documents receive mandatory analyst review.The passage attributes this difficulty to nested multi-level references spanning accord revisions.Source: §6 · p. 6

6.1 Grounding Metric Validation

• 84.6% grounding accuracy on obligations with 3+ cross-references contrasts with 97.1% on obligations with none; nested conditional obligations reach an estimated ∼79%.Human validation reports MiniCheck grounding accuracy of 91.8% agreement with majority labels and Pearson r=0.83; the deployment policy requires analyst review for High and Critical findings with at least three cross-references.Source: §6.1 · p. 6

7 Deployment and Lessons Learned

• Over four months and 9,847 updates, production recall was estimated at 96.0% with 90.7% precision, and sustained analyst efficiency improved 3.1×.The current Phase 2 cost is $48.2K/month versus a $45K manual baseline while all outputs still receive human review; projected later-phase costs assume reduced review burden.Source: §7 · p. 6
• System-assisted review reduced processing time from 47 to 12 minutes in a counterbalanced study of 12 analysts and 96 updates; sustained processing time later stabilized at 15 minutes.The respective reported gains were 3.9× during the study and 3.1× after novelty attenuation.Source: §7 · p. 6
• The system achieved gap detection F1 of 84.1 without institution-specific fine-tuning, a 3.6-point drop from primary evaluation, and 86.3 after fine-tuning ftype on just 50 examples.A controlled perturbation experiment supports an estimate that ∼70% of difficulty is regulation-specific and ∼30% institution-specific. Full cross-institutional evaluation of ∼1,200 examples is expected by Q3 2026.Source: §7 · p. 7

7.1 Key Lessons

• Embedding-only cross-reference resolution reached 72.3 F1, compared with 89.1 F1 after adding KG-based structural relationships.The authors identify this as the single most impactful design decision.Source: §7.1 · p. 7
• Medusa acceptance was 91.3% on regulatory text versus 82.7% on general text, alongside entropy values of 2.31 and 3.87 bits, respectively.The authors suggest speculative decoding may also be effective in other low-entropy domains, including medical records and patent filings.Source: §7.1 · p. 7
• Analysts were more loss-averse to missed compliance gaps than to F1 trade-offs, motivating recall-optimized thresholds and prominent confidence scores.The authors contrast the institutional risk of compliance false negatives with individual risk in clinical NLP.Source: §7.1 · p. 7
• Mapping the institution’s 847-category, 15-year-old GRC taxonomy took roughly three months, comparable to the model development cycle.The authors recommend budgeting at least 40% of project time for GRC integration and beginning taxonomy mapping alongside model development.Source: §7.1 · p. 7
• Analyst re-verification fell from 78% of outputs in Month 1 to 23% in Month 4, while processing time dropped from 22 to 12 minutes per update.The authors report weekly digests that disclosed both system successes and failures as the most effective trust-building intervention.Source: §7.1 · p. 7

8 Conclusion

• The end-to-end system combines KG-augmented RAG, multi-task obligation extraction, and compliance gap analysis, with domain-specific distillation for production.The conclusion reports 87.7 F1 at δ=0.6, 83.4 F1 under realistic error propagation, and 94.2% grounding accuracy.Source: §8 · p. 7
• Across two additive analyses, KG re-ranking provides the largest marginal gain on gap detection (+2.2/+2.5 F1), which the authors connect to cross-reference-heavy tasks.At δ=0.45, deployment achieved estimated 96.0% recall and 90.7% precision over four months and 9,847 updates, with 3.1× sustained analyst efficiency gain.Source: §8 · p. 7
• Future work includes extending beyond the three frameworks toward full 60K+ annual coverage and completing the multi-institutional GapBench expansion.The conclusion also describes later deployment phases toward full production.Source: §8 · p. 7

• Three frameworks cover approximately 48% of annual regulatory updates; expanding coverage requires additional parser engineering.Source: Limitations · pp. 7–8
• GAPBENCH contains 423 examples from one institution, with Full Gap F1 of 81.5 ± 4.2.Source: Limitations · p. 8
• The system processes English-language texts only, and the 12-analyst user study was modest and unblinded.Source: Limitations · p. 8
• Production metrics come from a parallel run with human review of every output; performance under reduced oversight remains unverified.Source: Limitations · p. 8

Ethical Considerations

• High-severity findings require human review and audit trails, and the system should not be the sole basis for compliance decisions.Training and evaluation use public regulatory texts; internal policies were anonymized, and customer personal data are not handled. Initially, 34% of analysts accepted outputs without independent verification, falling to 12% after calibration exercises and confidence-score visualization.Source: Ethical Considerations · p. 8

A GRC Platform Comparison

• Table 5 compares commercial GRC platforms using vendor-reported capabilities, not independently verified performance.The caption also notes that the market is changing quickly, with vendors announcing LLM-based capabilities.Source: Table 5 · p. 11

B Algorithm and Notation

• Table 6 summarizes the key hyperparameters.Source: Table 6 · p. 11

C Dataset Details

• REGOBLIGATION contains 1,847 sentences across SEC, MiFID II, and Basel III, while GAPBENCH contains 423 institution-derived obligation–policy pairs.The appendix also gives a pipeline from obligation extraction and KG lookup through retrieval, gap analysis, verification, and reporting.Source: C Dataset Details · p. 11

D RKG Schema

• The RKG schema defines five node types and five typed edge types; it contains 12,847 provision nodes and 34,219 edges.Updates include nightly synchronization and validation controls; pending provisions do not contribute to KG re-ranking until a full rebuild validates their edges.Source: D RKG Schema · pp. 11–12

E MiniCheck Validation Details

• Table 7 reports MiniCheck validation against human judgments on 200 regulatory-text samples.Source: Table 7 · p. 12

F False Negative Analysis

• Missed gaps most often involved implicit obligations (35%), multi-hop cross-references (29%), or jurisdiction-specific nuance (21%); removing KG re-ranking reduced Full Gap F1 from 81.5 to 76.1.For Full Gap cases, the reported drop was 5.4 F1 (CI: [−9.1, −1.8], p=0.007); the power analysis estimated approximately 200 examples are needed to detect a 3-point F1 difference at 80% power.Source: F False Negative Analysis · p. 12

G Additive Component Analyses

• Across two base models, KG re-ranking provided the largest gap-detection gain; from the GPT-4o+RAG base, that gain was +2.2.Multi-task extraction contributed most to QA (+2.7 F1), while MiniCheck improved grounding accuracy from 86.7% to 94.2%, an impact not captured by F1.Source: Table 8 · p. 12, G Additive Component Analyses · p. 12

H Additional Experimental Results

• 83.4 end-to-end F1 corresponds to approximately 2.1 missed obligations per 100-page document and 1.3 false gap alerts per day.Grounding accuracy declines from 96.1% for outputs under 50 tokens to 89.4% for outputs over 150 tokens; common extraction errors include boundary errors (34%) and multi-document cross-reference failures (28%).Source: H Additional Experimental Results · p. 13, Table 12 · p. 13

I Hyperparameter Sensitivity

• Across reasonable ranges, each of four key hyperparameters varied gap-detection F1 by less than ±2 points.Source: I Hyperparameter Sensitivity · p. 13, Table 13 · p. 14

J Annotation Guidelines and Entity Distribution

• Three compliance experts annotated REGOBLIGATION using detailed guidelines and adjudication; inter-annotator agreement was κ=0.84.The guidelines covered entity types, span boundaries, deontic modality, and cross-reference resolution.Source: J Annotation Guidelines and Entity… · p. 13

K Deployment Details

• During the four-month parallel run, sustained analyst efficiency gain was 3.1×, and 24 gaps missed by the manual process were confirmed.Monthly precision rose from 88.2% to 92.4% and recall from 94.1% to 97.3%; analysts processed 96 updates in the user study, with system-assisted time stabilizing at 15 minutes per update. Phase 2 costs exceeded the manual baseline because all outputs required review.Source: K Deployment Details · p. 14, Table 15 · p. 15

L Production Recall Estimation Caveats

• The 96.0% production recall is estimated because the true number of gaps is unknowable; sensitivity analysis gives 95.6–95.8% if manual recall is 90–95%.The estimate uses gaps caught by the system plus system misses caught in manual review, and the two processes may share blind spots.Source: L Production Recall Estimation Caveats · p. 14

N Cross-Institutional Generalization

• On 87 European-partner pairs, adaptation improved Partial/Full Gap combined by +3.1 F1.Synthetic policy perturbations reduced gap-detection F1 by 3.2 under moderate and 6.8 under aggressive perturbation; the authors estimate about 70% of difficulty is regulation-specific and 30% institution-specific.Source: N Cross-Institutional Generalization · p. 15

O Additional Lessons Learned

• Full fine-tuning degraded gap F1 from 86.3 to 81.7, while distillation to an 8B model was reported as more practical and less costly.The authors report that GRC integration took about three months; transparent weekly accuracy digests supported trust calibration, and monitoring detected three distribution-shift events requiring retraining cycles and a parser update.Source: O Additional Lessons Learned · pp. 15–16

P Additional Limitations

• The 12-analyst, 96-update user study is modest and unblinded, limiting subgroup estimates and leaving efficiency gains without stronger causal evidence.Detecting a 20% processing-time difference between junior and senior analysts would require about 40 participants per group; post-study efficiency stabilized at 3.1× versus 3.9× during the controlled period.Source: P Additional Limitations · p. 16
• Grounding accuracy falls from 95.4% on simple obligations to 84.6% for obligations with 3+ cross-references and an estimated ∼79% on nested conditional obligations.The validation set is smaller for the nested-conditional estimate; sentence-level verification may miss clause-level distinctions, so High and Critical findings require analyst review.Source: P Additional Limitations · p. 16
• The 1,082 ms p99 inference latency exceeds the sub-second target, although tail latency has minimal operational impact for batch processing.The cited factors include deep KG traversal, Medusa rejection cascades, and FAISS contention; traversal limits risk missing deep cross-references.Source: P Additional Limitations · pp. 16–17
• The nightly KG update cycle creates an 18-hour maximum blind spot, and uncovered frameworks such as Dodd-Frank Title VII and EMIR may leave cross-framework references unresolved.During the blind spot, retrieval uses embeddings without KG reranking, which ablations associate with a 4.6-point reduction in gap detection F1.Source: P Additional Limitations · p. 17

Q Example System Output

• The examples trace regulatory input through structured obligation extraction and KG retrieval to gap classification with grounding verification.The displayed confidence scores use deployment threshold δ=0.45.Source: Q Example System Output · p. 17

Q.1 Example 1: MiFID II Client Suitability (Major Gap)

• The MiFID II case is a major partial gap: policy §4.3 omits reassessment at each advisory interaction and detailed suitability procedures in Delegated Regulation Arts. 54–56.All three gap-description sentences were verified against source provisions, and the analyst confirmed escalation to the policy review committee.Source: §Q.1 · p. 17

Q.2 Example 2: SEC Regulation S-K Disclosure (Compliant)

• The SEC example is compliant because policies §7.1 and §7.4 address the pay-versus-performance disclosure requirements and both cited cross-references.The retrieval trace says KG reranking had minimal effect on these standardized SEC texts.Source: §Q.2 · p. 18

Q.3 Example 3: Basel III Liquidity Coverage Ratio (Critical Gap)

• The Basel III case is a critical full gap: KG traversal added CRR Art. 412(1), absent from initial retrieval, while grounding flagged a borderline sentence for mandatory analyst review.The policy lacked revised BCBS d424 criteria, including reserve requirements and supervisory approval; the analyst confirmed the gap and escalated it within 24 hours.Source: §Q.3 · pp. 18–19