ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering

• Routine enterprise records are work by-products, so many questions require recovering implicit relations across heterogeneous sources rather than retrieving an explicitly stated fact.Source: §1 · p. 2
• Realistic enterprise questions may lack complete, unique answers in released sources, a problem avoided—but not resolved—by tailoring questions to visible documents.Source: §1 · p. 2
• ENTLORE reconstructs an audited enterprise world and compiles question families into graph programs that filter ambiguous candidates, certify derived relations, compute complete answers, and link premises to released evidence.Source: §1 · p. 2
• The aligned anonymized release contains 2,341 documents from three source types and 907 questions, preserving routine-record structure while evaluating lookup, composition, latent reasoning, evidence acquisition, answer generation, and release fidelity.Source: §1 · pp. 2–3
• Across 56 model and access configurations, structure-materializing access leads, lexical access remains a strong baseline, and flat dense retrieval and agentic pipelines lose the most attainable performance on latent questions.Source: §1 · p. 2
• Its truth graph draws on enterprise documents, authoritative organizational tables, and operational records, separating native, metadata, heuristic, and certified derived relations.Source: §1 · p. 2

Enterprise and document-grounded QA

• Enterprise and finance QA benchmarks broaden corpus realism and evidence types, but generally answer from facts explicit in documents, knowledge bases, or table and database programs.Examples include TechQA, doc2dial, MultiDoc2Dial, EKRAG, WixQA, EnterpriseRAG-Bench, FinQA, TAT-QA, MultiHiertt, ARQA, and MuDABench (Castelli et al., 2020; Feng et al., 2020, 2021; Yu et al., 2025; Cohen et al., 2025; Sun et al., 2026; Chen et al., 2021; Zhu et al., 2021; Zhao et al., 2022; Wang and Balloccu, 2026; Li et al., 2026).Source: §2 · p. 3

Multi-hop and executable benchmark construction

• Multi-hop datasets distribute explicit supporting facts, while executable benchmarks use materialized knowledge, rules, or typed records; ENTLORE reconstructs and certifies withheld organizational relations.HotpotQA, 2WikiMultiHopQA, IIRC, and MuSiQue annotate explicit facts or paths (Yang et al., 2018; Ho et al., 2020; Ferguson et al., 2020; Trivedi et al., 2022); KILT, KQA Pro, ProofWriter, and QO-Bench use shared corpora, executable programs, proofs, or typed event tuples (Petroni et al., 2021; Tafjord et al., 2021; Zhang et al., 2026).Source: §2 · p. 3

Retrieval-augmented and graph-based access

• ENTLORE privately certifies organizational relations in an audited truth graph while withholding target relations from evaluated systems.Retrieval and graph methods vary how they access released evidence, but do not themselves certify which unstated organizational relations are valid, complete, and grounded in released premises. DPR, RAG, and FiD establish retrieve-and-read or generate approaches; IRCoT and ReAct interleave retrieval with reasoning or actions (Karpukhin et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021; Trivedi et al., 2023; Yao et al., 2022). RAPTOR, Self-RAG, HippoRAG, G-Retriever, GraphRAG, and CRAG explore hierarchical, adaptive, or graph-based access (Sarthi et al., 2024; Asai et al., 2024; Gutiérrez et al., 2024; He et al., 2024; Edge et al., 2025; Yang et al., 2024).Source: §2 · p. 3

3 The ENTLORE Framework

• ENTLORE reconstructs a provenance-bearing enterprise graph, certifies organizational inferences, anonymizes the evidence corpus, and compiles questions into executable programs with golden answers and proofs.The private graph certifies answers, while evaluated systems access only the released evidence world.Source: §3 · pp. 3–4

3.1 Audited Truth Graph and Released Evidence World

• The raw graph combines routine documents and traces with authoritative records, preserving edge provenance; heuristic associations aid discovery and decoys but cannot establish gold truth.Source: §3.1 · p. 4

3.2 Organizational Conventions

• Versioned organizational conventions certify inference edges only with human authorization, retaining each rule version and dependency chain while remaining private to evaluated systems.The truth graph includes certified rule applications rather than an unrestricted closure.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.3 Privacy-Preserving Corpus Projection

• Each released document transcribes exactly one raw document, preserving natural omissions and cross-source identity while adding no facts from tables, other documents, or certified inferences.The shared anonymization map also shifts dates while preserving their relative organization.Source: §3.3 · p. 5

3.4 Question Construction and Capability Decomposition

• Typed graph operators generate questions from audited answers and evidence, while language models phrase them without seeing answers; L1/L2 targets are stated, but L3 targets are absent and convention-certified.L1 tests single-source acquisition, L2 cross-source composition, and L3 additionally requires reasoning over implicit organizational structure.Source: §3.4 · p. 5

3.5 Verification and Golden Answers

• Verification checks valid derivations, answer completeness, unique denotation, and— for L3—certified proofs plus corpus-wide target absence; human review checks question meaning.Direct values are anchored to source spans, and candidates requiring an indispensable private-only premise are routed to verified abstention.Source: §3.5 · p. 5

4.1 Experimental Setup

• ENTLORE evaluates 907 questions across 2,341 released documents, with systems denied private organizational records, certification rules, the truth graph, and instance-level gold relations.Deployable systems share the released corpus but vary in indexing, offline construction, interaction, and compute; the L3 gold-document residual is an upper bound on relation-recovery failures, not a pure derivation measure.Source: §4.1 · pp. 6–7

4.2 RQ1: Stated-Evidence Performance

• For stated evidence, GraphRAG, BM25, and the LLM wiki average 63.2%, 62.4%, and 61.7% on L1; the LLM wiki and BM25 lead L2 at 52.5% and 51.4%.Across levels, BM25, GraphRAG, and the LLM wiki score 52.6%, 52.2%, and 50.9% question-weighted accuracy, versus 36.5% for agentic retrieval and 36.0% for Flat RAG.Source: §4.2 · p. 8

4.3 RQ2: Access Organization for Latent-Relation Questions

• On L3, GraphRAG averages 36.2% accuracy across eight models, ahead of BM25 at 34.0%, the LLM wiki at 27.7%, Flat RAG at 18.9%, and agentic retrieval at 13.5%.GraphRAG leads for five models and BM25 for three, so the average advantage is model-dependent rather than uniform.Source: §4.3 · p. 8
• Department attribution shows the largest L3 separation: GraphRAG reaches 53% versus BM25 at 2% on 42 questions, while containment and hierarchy rollup remain difficult for every deployable condition.Gold-document references reach 84% for department attribution, 66% for containment attribution, and 59% for hierarchy rollup; no deployable method exceeds 34% on the latter two families.Source: Figure 3 · p. 7, §4.3 · p. 8
• When flat retrievers assemble the complete annotated gold set, BM25 leads GraphRAG 54.9% to 48.5%; otherwise, GraphRAG leads BM25 31.0% to 25.1%.The latter group is harder overall: gold-document reference accuracy falls from 82% to 64%. The grouping is descriptive, not a general measure of retrievability.Source: Figure 3 · p. 7, §4.3 · p. 8
• On L3, agentic retrieval trails Flat RAG 13.5% to 18.9%, while BM25’s advantage over dense-backed systems is associated mainly with evidence assembly rather than answering after gold-set delivery.With gold document paths, the same agentic loop reaches 69.6% on L3, showing substantially better use of supplied evidence than acquisition under the evaluated dense backend.Source: §4.3 · p. 8

4.4 RQ3: What Remains with Gold Documents

• With gold document paths, 30.4% of L3 questions remain unanswered, versus 12.6% of L1 and 6.2% of L2; this residual is an upper bound on L3 relation-recovery failures.The residual also includes navigation, interaction, answer-extraction, and prompt effects, so it is not a pure derivation estimate.Source: Figure 3 · p. 7, §4.4 · p. 9

4.5 RQ4: Release Fidelity

• On 176 aligned items, the anonymized release preserves coarse comparative behavior, but fine deployable-condition ordering is unproven: ρ = 0.71 over eight configurations, with exact permutation p = 0.06.The three oracle configurations rank in the top three in both worlds, while deviations concentrate in BM25 (−0.28) and GraphRAG (−0.19).Source: §4.5 · p. 9

5 Conclusion

• ENTLORE reconstructs an audited truth graph, certifies organization-specific derivations, and releases aligned anonymized documents while keeping executable answers and proofs private.It distinguishes explicit lookup and cross-source composition from latent organizational reasoning without exposing target relations to evaluated systems.Source: §5 · p. 9
• 30.4% of latent organizational questions remain unanswered with gold document paths; induced entity graphs and navigable knowledge bases give the strongest deployable results.This residual is not a pure measure of derivation failure, and release-fidelity analysis supports broad group-level comparisons while leaving fine-grained rankings tentative.Source: §5 · p. 9
• Enterprise QA systems should recover typed relations across sources, preserve provenance through multi-step reasoning, and abstain when organizational inference is unwarranted.ENTLORE keeps benchmark truth private while making its premises auditable, providing a testbed that measures progress beyond document recall alone.Source: §5 · p. 9

A Extended Release-Fidelity Analysis

• Across 176 paired items, the three oracle configurations stay at the top in both worlds, while deployable configurations reshuffle in rank.Figure 4 shows crossings as rank changes and overall displacement as absolute shift.Source: Figure 4 · p. 13
• The BM25 discrepancy is hypothesized to reflect sensitivity to tokenization, identifier segmentation, and anonymized aliases, not added semantic information.The audit is interpreted as preserving comparative evaluation behavior rather than reproducing private-world scores exactly.Source: A Extended Release-Fidelity Analysis · p. 13
• The audit compares fixed private indices with per-model release indices and Chinese private text with anonymized, translated release text, so the BM25 deviation cannot be attributed to anonymization alone.It covers 176 of 258 paired items; 82 are excluded because compound pseudonyms cannot be cleanly inverted.Source: A Extended Release-Fidelity Analysis · p. 13

• Aggregation, footprint, handover, and time-window questions are secondary when base sets and time attributes are certified; private-only premises yield abstentions, and weak co-occurrence or negative-space assumptions never define positive gold.Source: B Certified Relation Families · p. 14

C Release Gate Pipeline

• Candidates are bound to pinned revisions, screened for released-source legality, checked for rule certification, and required to ground each premise in a released paragraph.Private metadata and heuristic associations cannot independently establish answerability, and endpoint co-occurrence does not suffice as premise grounding.Source: C Release Gate Pipeline · p. 14
• L3 candidates must have a target relation absent from every released source, exact released-evidence coverage of the proof, and certified uniqueness or set closure.Source: C Release Gate Pipeline · p. 14
• Privacy and fidelity checks trace each released document atom to its private source, screen for identifiers and rare details, and replay typed outputs before and after anonymization.Items with a required private-only premise are routed to verified abstention.Source: C Release Gate Pipeline · p. 14
• Question hygiene removes answer values, proof-path hints, and ambiguous wording; bank controls remove near duplicates and balance families, sources, and answer types.Source: C Release Gate Pipeline · p. 14
• Valid items enter L1/L2/L3 only when all proof premises are grounded in released documents; private-only premises require verified abstention, while invalid targets, proofs, or question semantics trigger rejection.The pipeline records the terminal gate and reason for every rejected candidate.Source: C Release Gate Pipeline · p. 14

D Privacy and Identity Ledgers

• Shared mappings rename people, projects, customers, and systems while preserving alias-resolution difficulty and reporting cadence.Alias families map isomorphically, dates shift by a fixed wholeweek offset, and released documents use stable anonymous identifiers.Source: D Privacy and Identity Ledgers · p. 15
• A private source ledger records provenance, identity mappings, and links from source records to questions.It includes raw paths, hashes, per-atom support, and metadata origins.Source: D Privacy and Identity Ledgers · p. 15
• Release checks include lexical leakage scanning, rare-event review, and structural sensitivity checks, but do not guarantee against all re-identification.Source: D Privacy and Identity Ledgers · p. 15

E Private Convention Library and Proof Certificates

• Each private convention entry specifies a versioned rule, typed premise and conclusion schemas, applicability conditions and exclusions, permitted premise source classes, and generic-entity examples.Source: E Private Convention Library and Proof… · p. 15
• The initial library covers containment transfer, hierarchy rollup, selected genre anchoring, and ticket role semantics; default-scope rules require explicitly encoded and audited exceptions.The listed exceptions are cross-team support, reviews, meetings, research, and reported team-level progress.Source: E Private Convention Library and Proof… · p. 15
• Each certified derived relation carries a proof certificate (r, rule_id, rule_version, Epremise, Erelease, v), where v is the human authorization verdict.A necessity certificate records whether every valid derivation route requires at least one organizational convention.Source: E Private Convention Library and Proof… · p. 15

F Post-hoc Matched Triples

• The analysis found 23 triples: 21 hybrid-source and only 2 text-grounded, leaving the text-versus-hybrid contrast underpowered.Triples used anchors with questions at all three levels and at least one shared released evidence document between L1 and L3, after filtering abbreviation noise; the analysis is a consistency check, not a controlled cross-level comparison.Source: F Post-hoc Matched Triples · p. 15, Figure 5 · p. 16
• Because triples match anchor entities, not base premises, and n=23 is small, a premise-matched escalation across three levels is the proposed next construction step.An L1 and L3 item in a triple need not rest on the same underlying fact.Source: F Post-hoc Matched Triples · p. 15
• On matched triples, hybrid LLM-wiki accuracy drops by ∆2→3 = +0.19 per triple, Ω drops 0.80 →0.65 with gold paths, while GraphRAG rises 0.25 to 0.34.The same triples were run with gold document paths through the same agentic loop; GraphRAG materializes the organizational relation.Source: F Post-hoc Matched Triples · p. 15, Figure 5 · p. 16

Benchmark statistics

• The 907-item bank contains 469 L1, 204 L2, and 234 L3 questions, anchored by 62 typed graph operators to 1,150 distinct gold documents.Items require 2.53 gold documents on average; L1 items require 1.26. Covered families include lookup, cross-document composition, latent project attribution, organizational aggregation, and incident roles.Source: G Experimental Details · p. 16

Access implementations and oracle variants

• The evaluation compares BM25, Flat RAG, agentic retrieval, an LLM wiki, GraphRAG, and four gold-evidence oracle variants, without exposing target relations.The oracle variants present gold evidence as canonical paragraphs, full documents, page-by-page paths, or through the compiled knowledge base; they diagnose localization, recall, navigation, and index-chain quality. GraphRAG and the wiki are compiled automatically from the released corpus without benchmark questions, truth-graph edges, or gold relations.Source: G Experimental Details · p. 16

Models and protocol

• The eight-model evaluation fixes settings before testing, repeats agentic systems on a stratified subset, and uses question-level bootstrap intervals.For five open-weight models, closed-book, BM25, and Flat RAG results use reasoning-disabled reruns because reasoning-enabled runs produced empty completions; all other cells use reasoning-enabled results. Exact versions and budgets are in the manifest, and infrastructure failures are reported separately.Source: G Experimental Details · pp. 16–17

Code, data, and configuration availability

• The supplement provides the corpus, question bank, construction and evaluation code, prompts, scoring and analysis code, manifest, and machine-readable outputs for reproducibility.The artifacts are slated for release under the MIT License upon publication.Source: G Experimental Details · p. 16

Score distribution at L3

• Under GraphRAG, 49% of L3 items score zero and 23% score one; among oracle-zero L3 answers, 19.5% explicitly say the fact is absent.The oracle instruction to use only supplied pages creates tension at L3, where answers are not stated in the evidence; this means the gold-evidence residual is an upper bound on relation-recovery failure.Source: G Experimental Details · p. 17

The tension is total for one answer type

• All models score zero on eight L3 PROJECT_HEADCOUNT items, which measure instruction-task interaction rather than capability and should be reworded or excluded.Models enumerate contributing people but decline to give a total because no document states one; one item also has ambiguous wording.Source: G Experimental Details · p. 17

Answer judging and type-specific scoring

• Scoring combines programmatic exact-match or sequence metrics for deterministic answer types with fixed-judge entailment scoring for free-form claims.Entity answers use alias-normalized exact match; sets use precision, recall, F1, and exact-set accuracy; counts require exact match; ordered answers use exact sequence accuracy and Kendall’s τ. The judge is Claude Opus 4.8 and is blind to system identity.Source: G Experimental Details · p. 17