Source-linked AI summary
ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering
Akrin Zheng, Alexander Wu, Alaia Liu
TL;DR
Enterprise QA often requires recovering organizational relations that are implicit across routine, heterogeneous records rather than merely retrieving stated facts. ENTLORE reconstructs an audited truth graph to build and evaluate such questions, finding that structured access performs best while gold documents still leave 30.4% of latent questions unanswered.
Problem
Existing enterprise QA benchmarks mainly test composing explicitly materialized facts, leaving recovery of target organizational relations absent from documents insufficiently evaluated.
Method
ENTLORE reconstructs an audited enterprise world and compiles realistic questions through graph programs that certify derived relations, answerability, golden answers, and proof dependencies.
Results
Structured access via an induced entity graph or navigable knowledge base gives the strongest deployable results, yet gold document paths leave 30.4% of latent questions unanswered.
Takeaways & Limitations
Enterprise QA systems should treat the organization behind documents as a first-class inference object and recover typed relations across sources with provenance.
Takeaways & Limitations
The residual under gold-document conditions is an upper bound on organizational-relation recovery failures, not a pure derivation measure.
Abstract
from arXiv · showhide
Enterprise question answering is framed as retrieving internal documents and generating grounded answers. Routine enterprise records, however, are work by-products in which required organizational relations remain implicit across heterogeneous sources. Existing benchmarks provide realistic multi-source evidence, but often materialize a predefined answer path and therefore test the composition of stated facts rather than recovery of a target relation absent from the corpus. We call the latter capability latent organizational reasoning. We introduce ENTLORE, a graph-grounded benchmark construction framework that reconstructs an audited enterprise world from routine documents, authoritative organizational tables, and operational records. Versioned organizational conventions certify derived relations in a truth graph, enabling complete golden answers and proof certificates. The aligned anonymized release exposes only the document corpus while withholding private structure and target relations. ENTLORE contains 2,341 documents from three source types and 907 questions spanning explicit lookup, cross-source composition, and latent organizational reasoning, evaluated across 56 model and access configurations. Structuring the released world as an induced entity graph or navigable knowledge base gives the strongest deployable results. Yet supplying gold documents still leaves 30.4% of latent questions unanswered, versus 12.6% and 6.2% for explicit and compositional questions. Enterprise QA therefore depends not only on document recall, but also on whether implicit organizational relations become usable. The benchmark, data, and code are publicly available at https://github.com/scitix/entlore .
1 Introduction
ENTLORE treats enterprise QA as recovery of implicit, organization-specific relations from fragmented routine records rather than retrieval of explicitly stated facts. Its graph-grounded framework reconstructs an audited enterprise world, produces an aligned anonymized corpus, and evaluates latent organizational reasoning without exposing gold relations.
- Routine enterprise documents often do not state the facts needed to answer questions, because they are by-products of day-to-day work rather than self-contained QA descriptions.
- Existing benchmarks improve multi-source evaluation but commonly organize evidence around predefined answer paths whose required steps are explicitly materialized.
- ENTLORE reconstructs an audited enterprise world from heterogeneous private sources, then compiles question families into executable graph programs that certify organization-specific derived relations.
- The released benchmark is an aligned, anonymized projection preserving routine records’ forms, source-derived content, and cross-document structure while withholding instance-level gold relations.
- 2,341 documents and 907 questions define ENTLORE, evaluated through a 56-configuration matrix spanning models and access paradigms.
- Latent organizational reasoning requires combining heterogeneous grounded facts to recover organization-specific relations before answering routine enterprise questions.
2 Related Work
Related work spans enterprise document-grounded QA, multi-hop and executable benchmarks, and retrieval-augmented or graph-based access. These directions provide heterogeneous evidence, explicit reasoning supervision, provenance, executable programs, and structured retrieval, while ENTLORE targets latent organizational relations.
- Enterprise and document-grounded QA: Enterprise QA benchmarks increasingly use authentic, heterogeneous document collections rather than curated passages.Examples include TechQA, doc2dial, MultiDoc2Dial, EKRAG, and WixQA.
- Multi-hop and executable benchmark construction: Multi-hop benchmarks distribute explicit supporting facts and supervise paths, answerability, connected reasoning, provenance, executable programs, or explicit proofs.HotpotQA, 2WikiMultiHopQA, IIRC, MuSiQue, KILT, KQA Pro, and ProofWriter represent these approaches.
- Retrieval-augmented and graph-based access: Retrieval-augmented and graph-based methods combine retrieval with reading, generation, reasoning, actions, hierarchical organization, critique, or graph structure.Representative systems include DPR, RAG, FiD, IRCoT, ReAct, RAPTOR, Self-RAG, HippoRAG, G-Retriever, GraphRAG, and CRAG.
3 The ENTLORE Framework
ENTLORE builds a provenance-bearing enterprise world, certifies organization-specific inferences in a private truth graph, anonymizes its document projection, and compiles verified graph programs into benchmark questions. Its level taxonomy distinguishes stated lookup, stated composition, and latent relations absent from released sources.
- World reconstruction: ENTLORE reconstructs a raw graph from routine documents, authoritative records, and operational traces, then certifies authorized organization-specific inferences to form a truth graph.Raw edges retain provenance; heuristic associations support candidate discovery and decoys but never independently establish gold truth.
- World reconstruction: A shared anonymization map projects the private world into released documents while withholding organizational records, certification rules, target edges, and proof certificates.The projection consistently renames entities and aliases and shifts dates while preserving relative organization; released documents introduce no new facts.
- Question construction: Typed graph operators generate questions from eligible truth-graph regions, compute typed answers and proof dependencies, and leave language models responsible only for surface wording.Each question is paired with associated released evidence units without exposing its golden answer.
- Question construction: L1 is stated lookup, L2 is stated composition across released sources, and L3 targets relations absent from every released source but certified in the truth graph.L1 and L2 provide diagnostic evidence-use baselines, whereas L3 additionally requires recovery from distributed observations and organizational conventions.
- Verification: Every question undergoes automated and human verification for valid derivations, completeness, unique denotation, intended semantics, provenance, replay, privacy, and— for L3—target absence.L3 verification additionally requires a certified organizational proof and a release-wide alias-aware absence scan.
4 Experiments
ENTLORE experiments separate stated-evidence handling from recovery of relations absent from released documents, using 907 questions over 2,341 documents and comparing deployable access paradigms. GraphRAG is strongest on average for latent L3 questions, yet gold documents leave 30.4% unanswered, showing substantial residual difficulty beyond evidence retrieval.
- Benchmark and task: The benchmark contains 907 questions over 2,341 released documents, while L3 relations must be recovered from documents without private records, truth graphs, or gold relations.The organizational graph contains 1,153 entities and 3,784 typed relations.
- Systems: All deployable systems use the same released corpus but differ in indexing, offline compilation, query-time interaction, and computational cost.Flat RAG and agentic retrieval share the same dense index, isolating iterative interaction; agentic retrieval and the LLM wiki expose different knowledge substrates.
- Interpretation: The oracle gap and residual are diagnostic rather than causal; the L3 residual upper-bounds failures of organizational relation recovery rather than measuring pure derivation.The residual includes every error remaining under the gold-document agentic condition.
- L3 results: 36.2%: GraphRAG achieves the highest average L3 accuracy, followed by BM25 at 34.0%, the LLM wiki at 27.7%, Flat RAG at 18.9%, and agentic retrieval at 13.5%.GraphRAG leads five of eight models, while BM25 leads three, so structured access does not uniformly dominate lexical retrieval.
- Relation-family results: 53% versus 2%: GraphRAG most separates from BM25 on department attribution, while no deployable method exceeds 34% for containment attribution or hierarchy rollup.The department result covers 42 questions, where the gold-document reference reaches 84%; gold-document references reach 66% and 59% for containment and hierarchy rollup.
- Residual error: 30.4% of L3 questions remain unanswered with gold document paths, compared with 12.6% for L1 and 6.2% for L2.The L3 residual is 4.9 times the L2 residual and accounts for 47.7% of the remaining error of the best deployable condition.
5 Conclusion
ENTLORE makes latent organizational reasoning evaluable by reconstructing an audited truth graph and releasing an aligned anonymized document corpus while retaining executable answers and proofs privately. Results show that graph-based access is strongest, yet gold document paths leave a substantial latent-relation challenge.
- Benchmark construction: ENTLORE reconstructs an audited truth graph and certifies organization-specific derivations for enterprise questions whose organizational relations remain implicit across heterogeneous records.The aligned anonymized release exposes the document corpus while executable answers and proofs remain private.
- Benchmark construction: ENTLORE separates explicit lookup and cross-source composition from latent organizational reasoning within the benchmark setting.This separation makes the distinct reasoning demands evaluable across enterprise records.
- Results: Induced entity graphs and navigable knowledge bases give the strongest deployable results across the same corpus.Plain lexical retrieval outperforms flat dense and agentic retrieval.
- Results: 30.4% of latent organizational questions remain unanswered when supplying gold document paths.Removing document-search errors does not remove the latent-relation challenge, although the residual is not a pure measure of derivation failure.
A Extended Release-Fidelity Analysis
The private-to-release audit finds that oracle configurations retain their top ranking, while deployable configurations reshuffle. BM25 discrepancies are attributed to release-text sensitivity and asymmetric indexing and corpus conditions rather than added semantic information.
- Private-to-release fidelity: Oracle configurations stay together at the top in both private and released worlds, while deployable configurations change ordering.Figure 4 defines crossings as rank changes and overall displacement as absolute shift across 176 paired items.
- Retrieval sensitivity: BM25 discrepancy reflects lexical sensitivity to release projection, including Chinese tokenization, identifier segmentation, and anonymized-alias standardization.These changes can alter lexical term overlap without adding or removing an underlying organizational fact.
- Audit asymmetries: The audit compares fixed private-corpus indices with per-model released-corpus builds, while also contrasting original Chinese text against anonymized, translated text.Retrieval-dependent conditions inherit the index difference, and BM25 deviation combines anonymization with language differences.
B Certified Relation Families
The initial release certifies relation families with explicit typed premises and auditable business semantics. Secondary families require certified base sets and time attributes, while private-only premises abstain and weak assumptions never define positive gold.
- Certified Relation Families: Certified relation families include containment attribution, hierarchy rollup, personnel bridges, and distinct typed ticket roles.Containment combines activity on a module with module–project context; hierarchy aggregates fully grounded child relations; personnel bridges connect projects through separately grounded activities.
- Certified Relation Families: Department attribution is included only when the project–department premise has predicate-level document support.
- Certified Relation Families: Aggregation, footprint, handover, and time-window questions remain secondary families when their base sets and time attributes are certified.
- Certified Relation Families: Private-only structural premises produce abstentions, while weak co-occurrence and negative-space assumptions never define positive gold.
C Release Gate Pipeline
The release gate pipeline audits candidates in sequence for versioning, source legality, proof validity, privacy, question quality, and dataset balance. Candidates are released as answerable L1/L2/L3, routed to verified abstention, or rejected based on these checks.
- Pipeline control: Gates run in order, and every rejected candidate is logged with its terminal gate and reason.This provides an auditable rejection record for the release process.
- Pipeline control: Version lock binds each item to pinned source, graph, corpus, schema, operator, rule, and realizer revisions.Rule certification additionally checks the rule identifier, version, applicability conditions, typed conclusion, and authorization verdict.
- Proof and release validity: Every indispensable proof premise must resolve to a released paragraph, while private-only premises trigger verified abstention and endpoint co-occurrence cannot establish grounding.The pipeline also confirms that an L3 target relation is absent from every released source.
- Proof and release validity: Additional gates certify exact evidence coverage, unique denotation or complete closure, raw-source faithfulness, privacy, question hygiene, and raw-to-release replay.Replay requires identical typed program outputs before and after anonymization.
- Routing and bank controls: Candidates route to answerable L1/L2/L3 when all proof premises are release-grounded, verified abstention when a required premise is private-only, and rejection when target, proof, or question semantics is invalid.Bank-level controls remove near duplicates and enforce family, source, and answer-type balance.
D Privacy and Identity Ledgers
ENTLORE anonymizes enterprise entities and dates while preserving alias-resolution difficulty and reporting cadence. A private source ledger maintains provenance, identity mappings, and links between source records and questions.
- Anonymization and provenance: Shared mappings rename persons, projects, customers, and internal systems, while isomorphic alias families preserve source alias-resolution difficulty.Dates shift by a fixed wholeweek offset to retain reporting cadence, and released documents use stable anonymous identifiers.
- Anonymization and provenance: The private source ledger records raw paths, hashes, per-atom support, metadata origins, identity mappings, and reverse links from source records to questions.
E Private Convention Library and Proof Certificates
ENTLORE’s private convention library formalizes versioned organizational rules with typed schemas, applicability constraints, source permissions, and generic examples. Certified derived relations include proof certificates with human authorization verdicts, while necessity certificates identify relations requiring organizational conventions.
- E Private Convention Library and Proof Certificates: Each private convention entry specifies a rule name and version, typed premise and conclusion schemas, applicability conditions, exclusions, permitted premise source classes, and generic-entity examples.
- E Private Convention Library and Proof Certificates: The initial library covers containment transfer, hierarchy rollup, selected genre anchoring, and ticket role semantics.
- E Private Convention Library and Proof Certificates: Default-scope rules are included only after exceptions for cross-team support, reviews, meetings, research, and reported team-level progress are explicitly encoded and audited.
- E Private Convention Library and Proof Certificates: Each certified derived relation carries a proof certificate containing the relation, rule identifier and version, premise and release evidence, and human authorization verdict.
- E Private Convention Library and Proof Certificates: A necessity certificate records whether every valid derivation route requires at least one organizational convention.
F Post-hoc Matched Triples
A post-hoc check of 23 anchor-entity triples supports the aggregate finding that accuracy declines when stated composition becomes latent derivation, including with gold document paths. Because triples were not premise-matched and the sample is small, the analysis is a consistency check rather than a controlled comparison.
- Triple construction: 23 anchor-entity triples, including 21 hybrid-source triples, were identified for the post-hoc consistency check.Triples required questions at all three levels and shared released evidence between their L1 and L3 items.
- Consistency results: 0.80 → 0.65 accuracy under the gold-document agentic reference Ω reproduces the drop from stated composition to latent derivation.Under the LLM wiki hybrid arm, the corresponding per-triple decline is ∆2→3 = +0.19.
- Consistency results: 0.25 to 0.34 accuracy under GraphRAG reverses the direction, rising when the latent relation is materialized.The matched-triple check therefore agrees with the main paper’s oracle analysis and indicates the effect is not explained by document-search recall alone.
- Limitations: The analysis is limited because triples are anchor-entity-matched rather than premise-matched, and n=23 is small.A premise-matched construction is proposed as the natural next step for a controlled comparison.
G Experimental Details
The benchmark comprises 907 questions anchored by typed graph operators to 1,150 gold documents, with latent L3 reasoning remaining especially difficult. L3 scores are often all-or-nothing, and models universally fail integer counts over latently derived sets because they avoid stating totals not explicitly documented.
- Benchmark statistics: 907 questions are distributed across 469 L1, 204 L2, and 234 L3 items, anchored by 62 typed graph operators to 1,150 distinct gold documents.Each item requires 2.53 gold documents on average, while an L1 item requires 1.26.
- Evaluation conditions: Reasoning-enabled results are reported per condition and model, so closed-book, BM25, and Flat RAG columns are not reasoning-enabled measurements for open-weight models.The reasoning-enabled pass supplied the remaining cells, including agentic, LLM Wiki, GraphRAG, and Ω columns, without empty completions.
- L3 score distribution: 49% of L3 items score exactly zero and 23% score exactly one under GraphRAG, indicating nearly all-or-nothing rather than graded performance.Partial credit is almost entirely confined to set-valued answers.
- Latent count questions: Every model scores zero on eight PROJECT_HEADCOUNT L3 items under every condition, while the oracle recovers one item out of 64 model-item pairs.Models correctly enumerate contributing people but decline to state a total when no document explicitly states one, so the failure is not arithmetic.