Source-linked AI summary
RegDivergence-101: An LLM Benchmark for Cross-Jurisdiction Regulatory Contradiction Detection in Life Sciences
Chuchu Wu, Zhiyin Zhou, Jingzhuo Hu, Liang You
TL;DR
Sponsors developing medicines in the US and EU must reconcile separately published regulatory guidance, a currently manual expert task. This paper defines cross-jurisdiction divergence detection, releases a 101-pair benchmark, and reports pilot-scale findings across four baseline methods.
Problem
US–EU medicine development requires reconciling separately published regulatory guidance, and missing a divergence during protocol design can cost a year or more.
Method
The paper defines divergence detection by classifying FDA and EMA requirements on the same topic as AGREE, DIVERGE, or SILENT, and releases RegDivergence-101.
Results
The four-method baseline hierarchy and pilot observations show that SILENT requires explicit absence-aware formulation, while pair-level graphs improve over lexical methods but trail flat prompting.
Takeaways & Limitations
At pilot scale, the findings motivate corpus-level graph construction as the architectural target for large-scale silent-detection.
Takeaways & Limitations
The 101 pairs from three source studies limit statistical power to directional observations, while co-topicality judgments are single-annotator.
Abstract
from arXiv · showhide
Pharmaceutical sponsors developing a drug for both the United States and the European Union must reconcile guidance issued independently by the FDA and the EMA. Where the two agencies require substantively the same thing, a sponsor can file once; where they diverge, a single trial design risks rejection in one region; where one agency is silent on a point the other regulates, the sponsor must infer obligations. Today this reconciliation is performed manually by regulatory-affairs experts. We introduce cross-jurisdiction regulatory divergence detection: given an FDA requirement and an EMA requirement on the same topic, classify their relationship as AGREE, DIVERGE, or SILENT. SILENT is inherently directional (SILENT_FDA vs. SILENT_EMA); we record direction per pair and report per-direction F1 alongside the collapsed label. We release RegDivergence-101, a 101-pair expert-grounded pilot evaluation benchmark (labels grounded in three peer-reviewed FDA/EMA comparison studies and primary FDA/EMA/ICH guidance text; dual-annotation inter-annotator kappa = 0.85), and systematically characterise a four-method baseline hierarchy: lexical heuristic (0.511 macro-F1, 95% CI [0.411-0.605]), NLI cross-encoder (0.233), obligation-level Graph-RAG (0.663 [0.570-0.747]), and flat LLM judge / Claude Haiku (0.830 [0.747-0.908]). Three directional observations emerge at pilot scale (n = 101): SILENT is semantically detectable but invisible to entailment-only formulations; pair-level obligation graphs improve over lexical methods but trail flat-LLM context (CIs partially overlapping); and corpus-level graph construction is the indicated architectural target for large-scale silent-detection. RegDivergence-101 is a pilot release establishing the task formulation and baseline hierarchy; four unrepresented regulatory domains and an expansion roadmap are described in Section 7.
1 Introduction
The paper frames FDA/EMA guidance reconciliation as a manual cross-jurisdiction task and introduces a three-way divergence-detection benchmark with directional SILENT labels.
- FDA and EMA publish guidance separately, requiring sponsors developing medicines in both regions to reconcile two regulatory sources.
- A gene-therapy example contrasts FDA’s 15-year long-term follow-up recommendation with EMA’s case-by-case risk assessment.The same topic can therefore receive materially different levels of regulatory specificity.
- Missing a divergence during protocol design can cost a sponsor a year or more.
- Reconciling the two regulatory corpora is currently a manual expert task, despite available NLP components for document NLI, contradiction detection, and Graph-RAG.
- The paper introduces AGREE, DIVERGE, and directional SILENT classification over pre-aligned FDA/EMA requirement pairs.SILENT distinguishes SILENT_FDA from SILENT_EMA and reports per-direction F1.
- RegDivergence-101 contains 101 pairs across 13 topic areas and compares four methods at pilot scale with bootstrap confidence intervals.The benchmark uses a documented two-track construction protocol and reports three directional observations about task structure.
2 Related Work
Prior work addresses contradiction detection, compliance, and regulatory comparison within documents or jurisdictions, while RegDivergence-101 targets automated FDA/EMA comparison with an explicit SILENT class.
- Document-level NLI and legal contradiction benchmarks primarily address entailment or contradiction within individual documents or contracts.
- Graph-RAG systems support regulatory knowledge retrieval, policy-context alignment, or compliance question answering rather than parallel-jurisdiction divergence detection.
- Sino-US-DrugQA shares the concept-alignment premise but cannot express the SILENT class.It contains 11,871 US–CN multiple-choice items.
- FDA/EMA divergence is documented in human-authored regulatory-science comparisons across dosing, oncology labelling, and multi-country SmPCs.
- RegDivergence-101 is presented as the first FDA/EMA benchmark with an explicit SILENT class, extending policy-aligned evaluation to cross-jurisdiction regulation.
- The identified gap is automated comparison of parallel FDA and EMA regulatory corpora, which existing contradiction and Graph-RAG work does not cover.
3 Task and Dataset
The paper defines directional relationship labels for pre-aligned FDA/EMA requirements, constructs a two-track 101-pair dataset, and reports annotation reliability alongside scope limitations.
- 3.1 Task Definition: For each pre-aligned topic, the task assigns AGREE, DIVERGE, or directional SILENT to corresponding FDA and EMA requirements.AGREE preserves equivalent obligations, DIVERGE captures substantive conflicts, and SILENT records which jurisdiction is silent.
- 3.1 Task Definition: SILENT means one jurisdiction does not address what the other regulates, with direction recorded as SILENT_FDA or SILENT_EMA.A unified SILENT label is retained for backward compatibility in Table 1.
- 3.3 Annotation Protocol: SILENT labels remain epistemically soft because they assert that exhaustive corpus search found no counterpart requirement.All 21 SILENT pairs are MODERATE confidence, and downstream work should treat them as soft labels.
- 3.1 Task Definition: The formulation assumes requirement pairs are pre-aligned by shared regulatory topic; topic alignment is treated as a separate upstream problem.
- 3.2 Dataset Construction: Candidate pairs at different regulatory hierarchy levels were excluded to avoid confusing specificity differences with substantive divergence.The pilot does not release a machine-readable list of excluded pairs.
- 3.3 Annotation Protocol: Human cross-framing agreement was κ=0.85 across 101 pairs, with SILENT agreement at κ=0.98 and more variable AGREE/DIVERGE agreement near 0.79–0.81.
- 3.3 Annotation Protocol: A secondary LLM consistency check on a 36-pair sample yielded κ=0.542 as test-retest agreement rather than human inter-annotator agreement.
4 Baselines
The pilot evaluates four zero-shot or few-shot baselines for classifying regulatory relationships, using lexical, NLI, obligation-graph, and flat-LLM approaches.
- Lexical Heuristic: The lexical heuristic combines TF-IDF similarity with five conflict signals, including negation, divergence cues, modal strength, numeric thresholds, and specificity.A shared ICH citation is treated as a strong AGREE signal.
- NLI Cross-Encoder: The NLI cross-encoder maps contradiction, neutral, and entailment probabilities to labels, but models without SILENT supervision map absence of regulation to neutral.This structurally collapses SILENT recall, while stronger encoders improve AGREE/DIVERGE F1 without resolving the gap.
- Graph-RAG Pair-Level Classifier: Graph-RAG extracts policy triplets and classifies aligned obligation-node pairs, with a hard-coded DIVERGE rule for prohibited obligations.A second LLM call reasons over obligation structure and does not automatically treat modal-register differences as divergence.
- Flat LLM Judge: The flat LLM judge prompts Claude Haiku once per pair with both texts and explicit AGREE, DIVERGE, and SILENT definitions, without retrieval or graphs.The Graph-RAG and flat judge comparison is a within-model ablation, not a cross-family architectural claim.
5 Results
Results compare macro-F1, accuracy, per-class F1, and directional SILENT F1 across four methods on 101 regulatory pairs, with partially overlapping bootstrap intervals.
- Overall Results: Table 1 reports macro-F1, accuracy, and per-class F1 with bootstrap 95% confidence intervals for all four methods.The evaluation contains 101 pairs.
- Directional SILENT Results: Per-direction SILENT F1 distinguishes SILENT_FDA, with 13 pairs, from SILENT_EMA, with 8 pairs.SILENT_FDA means FDA is silent while EMA regulates; SILENT_EMA reverses that direction.
- Overall Results: Point estimates increase monotonically from lexical to Graph-RAG to the LLM judge, but overlapping confidence intervals support a ranking trend rather than significance claim.The lexical ceiling reflects errors with high lexical overlap but opposed regulatory stances, such as differing Mayo-score instruments.
6 Analysis
Analysis identifies a formulation constraint for SILENT detection, a pilot-scale graph-versus-flat-prompt trade-off, and distinct pair-level and corpus-level architectural targets.
- Observation 1: SILENT F1 is 0.025 under NLI versus 0.778 under the LLM judge, indicating that explicit SILENT formulation matters for absence detection.The authors characterize this as a structural constraint of entailment framing rather than a model-capacity effect.
- Observation 2: Graph-RAG improves over lexical performance by +0.15 F1 but trails flat prompting by −0.17 F1 for Claude Haiku on this dataset.Confidence intervals partially overlap at pilot scale.
- Observation 2: Obligation graphs expose structure, but flat prompting retains full-text context for distinguishing modal-register agreement from threshold divergence.The paper illustrates this distinction using same-topic requirements with different thresholds.
- Observation 3: Pair-level classification is largely addressed by the flat LLM judge at 0.830, whereas corpus-level silent detection requires finding aligned pairs across agency corpora.The proposed target is a corpus-level policy graph.
- Robustness Analysis: The LLM judge scores 0.769/0.745 on Track A AGREE/DIVERGE and 0.966/0.923 on Track B, with source-study exposure not inflating performance.Track-level macro-F1 is omitted because Track B contains no SILENT pairs.
- Robustness Analysis: Performance is 0.804 on the JCC stratum and 0.844 on all other pairs, making performance marginally higher outside the concentrated source.Forty of 101 pairs come from Vieujean et al.
7 Future Work
Future work targets corpus-level policy-graph construction and expands the benchmark beyond the pilot’s represented domains through larger, independently validated annotation.
- Corpus-Level Policy Graph: The proposed corpus-level architecture builds FDA and EMA policy graphs, aligns requirement nodes, classifies aligned pairs, and generates explanations.Alignment becomes a graph-construction step rather than a pre-processing input.
- RegDivergence-500: RegDivergence-500 will expand coverage to CMC, adaptive trial design and estimands, statistical analysis plans, and labelling/SmPC.The planned benchmark contains approximately 500 pairs and uses a community-annotation pipeline seeded from about 45 comparison sources.
- Validation and Expansion: Planned validation includes multi-model replication, an independent third-annotator pass, and a cross-pair generalisation probe.A future FDA×PMDA subset of approximately 40 pairs is also planned.
8 Limitations
The pilot's evidence is constrained by limited scale, coverage, annotation scope, and model diversity. Its SILENT labels and pair-level formulation also require cautious interpretation.
- 101 pairs from three source studies limit statistical power to directional observations.
- Co-topicality judgements are single-annotator, while Track A inherits source-study alignment methodology.
- Four domains, including oncology, rare diseases, medical devices, and vaccines, are absent from the pilot.
- All 21 SILENT pairs are uniformly treated as MODERATE confidence and should be treated as soft labels.
- The pair-level SILENT task is a proxy for the correct corpus-level formulation.
- All LLM-dependent components use Claude Haiku, so comparisons are within-model rather than cross-family architectural claims.
- RegDivergence-101 is not sufficient for deployment or AI-procurement decisions and is specific to the FDA/EMA pair until cross-pair probes are run.
9 Conclusion
The paper introduces an expert-grounded pilot benchmark and four-method baseline hierarchy for cross-jurisdiction regulatory divergence detection. Pilot observations favor explicit absence-aware formulations and identify corpus-level graph construction as the next architectural target.
- RegDivergence-101 is a 101-pair expert-grounded pilot benchmark with per-direction SILENT annotation and a four-method baseline hierarchy.
- SILENT requires explicit absence-aware formulation, with NLI F1 0.03 → LLM F1 0.78.
- For Claude Haiku, pair-level graph structure improves over lexical methods by +0.15 F1 but trails flat prompting by −0.17 F1.
- The pair-level versus corpus-level scope distinction motivates corpus-level graph architecture as the next contribution.
- RegDivergence-500 is planned to establish the task at production scale.
A Prompts and Code
The paper provides prompts, implementation scripts, evaluation scripts, and the dataset schema. The methods include a flat LLM judge and a two-stage Graph-RAG pipeline with hard classification rules.
- Full prompts, evaluation scripts, and the dataset schema are available in the project repository.
- The LLM judge defines AGREE, DIVERGE, and SILENT with explicit absence framing and pairs FDA and EMA text.
- The LLM judge uses Claude Haiku with temperature 1.0 and a maximum of 100 tokens.
- Graph-RAG Stage 1 extracts subject, obligation level, requirement, and conditions before Stage 2 classifies the aligned node pair.
- Hard rules map MANDATORY ↔ PROHIBITED to DIVERGE and two SILENT nodes to SILENT.