From Document Silos to Process Intelligence: A Multi-Layer Knowledge Graph for CMC Process Development

1 Introduction

• CMC lacks a modular architecture for both precise document-level questions and corpus-wide questions requiring cross-document, ontological retrieval.The paper also identifies entity resolution and rigorous process-development benchmarking as challenges.Source: §1 · p. 3
• A modular pipeline combines lossless lexical ingestion of heterogeneous CMC documents with an ontology-guided domain graph and question-selected LLM agents.The domain graph is anchored to lexical evidence, with canonical entities and relations resolving to source passages.Source: §1 · pp. 3–4
• The three-tier protocol measures deployment fidelity across increasing reasoning complexity and pairs reliability assessment with a failure taxonomy.The paper distinguishes this from competence benchmarks that assess whether a system knows a domain’s context.Source: §1 · p. 4
• A hybrid system using both graph layers addresses a wider spectrum of questions than either layer alone, while routing improves accuracy and traceability.The domain graph is provenance-anchored so canonical entities and relations resolve to the source passages that justify them.Source: §1 · p. 4

2 Platform Architecture and Methods

• Lossless ingestion preserves document structure in a lexical layer, while an ontology-driven intelligence layer connects domain concepts across documents and projects.The domain graph's entities remain anchored to source chunks, combining cross-document structure with passage-level evidence.Source: §2 · pp. 4–5, Figure 2 · p. 5

2.1 Corpus and document heterogeneity

• The 38-document English testbed includes 31 dated report snapshots spanning 1–2 years; the platform is designed to handle multilingual handwritten scans too.The snapshot lineage deliberately tests changing conditions and conclusions across revisions, alongside independent documents.Source: §2.1 · pp. 5–6

2.2 Knowledge layer: lexical graph

• Docling preserves document hierarchy and reading order; element-aligned chunks combine text retrieval with BM25 and vector search, while document trees remain disconnected.Each Docling element becomes a chunk, with oversized tables split into row windows; sub-threshold fragments and figure payloads are retained but not embedded.Source: §2.2 · p. 6

2.3 Intelligence layer: domain graph

• LLMs extract typed entities and relations using a controlled domain vocabulary, then global resolution merges surface forms into canonical entities linked to mentioning chunks.The ontology serves as a vocabulary and consistency constraint, not an inference engine.Source: §2.3 · pp. 6–7
• 98.9% effective provenance coverage means entities are directly anchored or one hop from an anchored entity, supporting verification of corpus-wide answers against source text.Some labels lack direct mention links, but remain reachable through a directly anchored neighbor.Source: §2.3 · p. 7

2.4 Reasoning: agentic workflow

• The lexical agent handles precise single- or few-document queries, while the domain agent uses parameterized graph templates for corpus-wide aggregation, overlap, and connectivity questions.Template selection constrains graph traversal rather than leaving the agent to author Cypher freely.Source: §2.4 · pp. 7–8
• A two-stage router sends unambiguous quantity or lot-number questions to lexical retrieval and cross-document aggregation questions to the domain agent, dispatching both when needed.Deterministic rules handle clear cases first; otherwise, one tool-free LLM call classifies the question.Source: §2.4 · p. 8

2.5 Benchmarking

• The proprietary-corpus benchmark derives expert questions from extraction data and evaluates lexical QA through four-choice accuracy, option-free LLM judging, and stratified SME review.Questions span anchored, unanchored, hard, and comparative cohorts; 505 were selected for the final evaluation run.Source: §2.5 · pp. 8–9

3.1 Graph composition

• The graph contains 12,353 nodes and 45,471 edges; its lexical layer preserves document structure while 537 canonical entities form the domain layer.The lexical graph includes 38 documents, 1,452 sections, and 10,326 chunks; domain entities connect to the lexical layer through HAS_MENTION edges.Source: §3.1 · pp. 9–10
• Global resolution collapses 3,236 document-local mentions into 537 canonical entities, with 88.8% of merges from exact names, aliases, or curated synonyms.Embedding similarity accounts for only 2.0% of merges; no LLM adjudication is used at this stage.Source: §3.1 · p. 10

3.2 Graph reproducibility

• Entity extraction varies across repeated runs, and fixing temperature alone does not make extraction reproducible; downstream stages are reproducible given fixed extraction output.The authors identify seeded inference or source-text verification as requirements for regulated use.Source: §3.2 · pp. 10–11

3.3 Lexical agent: reliability and limits

• Lexical_ReAct scores 95.0% (480/505) T1 accuracy and 84.7% (428/505) T2 pass rate, versus vector-RAG at 88.5% T1 and 65.5% T2.The reported improvements over vector-RAG are 6.5% on T1 and 19.2% on T2.Source: §3.3 · pp. 11–12, Figure 4 · p. 12
• Tier-2 scores fall more on unanchored and comparative cohorts, where answers require synthesizing evidence across multiple documents.For baseline vector-RAG, the comparative-cohort score drops from 94% on T1 to 19% on T2.Source: §3.3 · p. 12, Figure 4 · p. 12
• Among reviewed failures, question-quality issues are largest at 43% (n=32), followed by extraction errors at 36% (n=27).Extraction errors concentrate on hard-unanchored questions, which can become significant at much larger corpus sizes.Source: §3.3 · p. 13

3.4 Domain agent as the complement

• For water and 2-MeTHF, the domain agent returns five shared processes while lexical retrieval returns an incomplete list of two.The domain traversal returns the entities with source chunks, so answers remain checkable against documents.Source: §3.4 · p. 14
• The router selects lexical, domain, or combined retrieval by question type; lexical suits precise local queries, while domain suits corpus-wide process-chemistry questions.The domain graph supports aggregation, overlap, entity evolution, synthesis-path connectivity, and process fan-in through shared inputs.Source: §3.4 · pp. 14, 16

Summary

• The platform combines lossless lexical ingestion with an ontology-guided, provenance-anchored domain graph, while routing questions to agents suited to local detail or corpus-wide properties.The lexical layer supports keyword and meaning-based retrieval; the domain graph resolves entities across documents.Source: §4 · pp. 16, 18, Figure 6 · p. 17
• The Lexical layer outperformed vector-RAG on Tier-1 MCQ accuracy (95% vs 89%) and Tier-2 pass rate (85% vs 65%), but lagged on corpus-wide and unanchored questions.Source: §4 · p. 18

Limitations

• A comparable-scale domain-agent benchmark across more documents and projects remains a priority; the current three-tier protocol was applied only to the lexical agent.The domain graph was assessed on a small question set, head-to-head with the lexical agent.Source: §4 · p. 18
• The lexical agent’s advantage over vector-RAG cannot be attributed to graph structure alone because tool access, iteration budget, and prompt effort also differed.An experiment holding those factors constant while toggling graph navigation is needed to isolate the graph’s contribution.Source: §4 · p. 18
• Zero-temperature extraction was not reproducible: repeated runs differed in entity counts, descriptions, and chunk anchoring, although downstream phases reproduced given fixed extraction output.The reported counts were 83 versus 86; proposed remedies include seeded inference or source-span verification.Source: §4 · p. 18
• The ontology was only a prompt-level schema with five instantiated classes, and extraction required corrections for under-populated solvent roles and overtagged strong bases.Loading the OPC/PROCO ontologies with a reasoner and expanding coverage could support inferred relationships and broader anomaly detection.Source: §4 · p. 19
• The pipeline does not read reaction-scheme figures, and the authors’ proposed long-context alternative for small-to-moderate corpora was not tested.The authors locate the graph’s motivating regime in programs with 300+ internal documents.Source: §4 · p. 19

Outlook

• The authors anticipate that provenance-anchored graphs, shared benchmarking, and process-chemistry ontologies could support more trustworthy and comparable institutional memory across biopharma.The architecture is modular: its lexical layer is content-independent, while its domain graph can adopt tailored ontologies.Source: §4 · pp. 18–19

Data Availability and Ethics

• The testbed concerns a discontinued Sanofi small-molecule program; its proprietary documents cannot be released, and no human-subjects or animal data were used.The compound’s process patent is available separately.Source: Data Availability and Ethics · p. 20

Conflicts of Interest

• The authors are Sanofi US employees and declare no other competing interests.Source: Conflicts of Interest · p. 20

A.1 Question-designer agent

• Each generated item includes a stem, four options, correct letter, explanation, evidence quote and location, source document, question-type label, and confidence score.The prompt combines a common preamble with switchable anchoring-mode and difficulty rule blocks.Source: §A.1 · p. 22
• A diversity hint uses the previous five stems and covered facets to encourage questions on uncovered material.Source: §A.1 · p. 22
• A distractor is rejected only when word overlap is at least 0.78, numeric tokens match, and character similarity is at least 0.75.Requiring identical numeric tokens preserves discriminating pairs such as different times and temperatures.Source: §A.1 · p. 22
• An unanchored pH question requires locating the continuous-quench run from its technical description, while a comparative distractor reverses the direction of a research-focus shift.Source: §A.1 · p. 23
• The bank includes comparative questions about research-focus shifts and analytical-method questions about a reported melting point.The examples also include hard-anchored procedure questions and hard-unanchored process-condition questions.Source: §A.1 · p. 23

• The table reports judge-score and embedding-similarity contingency categories for each cohort and the pooled data.Source: Table 3 · p. 24
• “On-topic” and “paraphrased” cases have high judge scores, respectively with high similarity and low similarity.Source: Table 3 · p. 24
• “Trap” cases pair low judge scores with high similarity, while “off-topic” cases are low on both measures.Source: Table 3 · p. 24

• The example asks which process steps have water and 2-MeTHF in common.Source: C Worked head-to-head example: shared… · p. 24
• Appendix C.1 gives the answers behind the example in subsection 3.4.Source: C Worked head-to-head example: shared… · p. 24
• The reproduced answers are truncated for length.Source: C Worked head-to-head example: shared… · p. 24

• Two returned steps share water and MeTHF: S2, oxime elimination to nitrile, and S3, SNAr workup.Source: §C.1 · p. 24
• S2 was studied in MeTHF across water contents from anhydrous (<100 ppm) to saturated (∼4%) to assess whether it could be telescoped without water removal.Source: §C.1 · p. 24
• S3 uses DI-water extraction of the crude MeTHF reaction mixture, with Table 10 reporting a 1:1 water-to-crude-solution volume ratio.Source: §C.1 · p. 24
• The answer is incomplete because water and 2-MeTHF co-occur in 2,233 chunks, while the agent can inspect only a few at a time.Source: §C.1 · p. 24

C.2 Domain agent

• The domain agent lists five operations using water and 2-MeTHF, including purification, azeotropic drying, phase separation, aqueous quench, and aqueous backwashing.Source: §C.2 · pp. 24–25
• These processes are primarily workup and purification operations, with MeTHF serving as the organic solvent.Source: §C.2 · p. 25

C.3 Expert assessment

• Expert assessment contrasts canonical unit-operation names with lexical S2/S3 report terms and conditions; machine-generated entity names need better primary-name selection for readability.Source: §C.3 · p. 25