PaperlayerView exact revision on arXiv

Source-linked AI summary

CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

Sil Hamilton, Albert Yu Sun, Oscar J. Romero, Carl-Leander Henneking, David Mimno, Bishan Yang, Igor Labutov

arXiv:2608.27391v1cs.AIcs.CLcs.IRcs.LG

TL;DR

Enterprise-scale evaluations lack realistic, shareable corpora that preserve cross-document consistency. CorporateBench addresses this gap with procedurally generated, human-validated corporate networks and finds that model performance worsens as corpus scale grows. Its results establish a benchmark for enterprise retrieval and reasoning, while its email-only and LLM-generated data limit direct generalization to broader human-written corporate communications.

  • Problem

    Enterprise-scale evaluations lack realistic, interconnected corporate corpora because internal communications are scarce and existing synthetic benchmarks are oversimplified.

  • Method

    CorporateBench generates four procedurally simulated companies and deterministic enterprise tasks from temporally evolving knowledge bases with arbitrary scale and logical consistency.

  • Results

    Performance decreases as corpus scale grows, while the KB–RAG difference on QA widens from 0.24 (S) to 0.37 (XL).

  • Takeaways & Limitations

    CorporateBench provides researchers and practitioners a benchmark for assessing LLM retrieval and reasoning on enterprise communication networks.

  • Takeaways & Limitations

    The benchmark focuses on email and uses LLM-generated documents, so results may differ for broader human-written corporate communications.

Abstract

from arXiv · show

LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.

1 Introduction

CorporateBench targets the gap between increasingly capable LLMs and evaluations that capture complex, interconnected enterprise reasoning at realistic scale. It introduces internally consistent corporate communication networks and tasks designed to require evidence synthesis across many documents.

  • State-of-the-art LLMs still struggle with complex reasoning over many documents in enterprise scenarios.
  • 87.6 documents per question gives CorporateBench a higher evidence-to-question ratio than the closest benchmark, EKRAG at 1.5.The higher ratio requires synthesizing information across many documents.
  • Current evaluation metrics fail to capture this enterprise-scale failure mode, hindering wide-scale industry adoption.
  • Synthetic benchmarks are often simplified, while realistic enterprise benchmarks remain scarce because internal data is constrained by nondisclosure agreements and corpus-generation costs.
  • CorporateBench addresses limited realism and scale by generating large, internally consistent corporate communication networks from procedural knowledge bases.
  • Its task suite preserves temporal sequencing, role-respecting communication directions, deeply interconnected evidence, and deterministic labels computed from the knowledge base.

2 Related Work

Related work spans synthetic long-context tests, document-based reasoning benchmarks, and enterprise-domain datasets. These approaches trade realism, scale, or data quality against the constraints of corporate data access.

  • Needle-in-a-haystack descendants identify long-context failure modes but trade realism for generation speed.
  • Book- and movie-script benchmarks test document reasoning, but their limited number of documents does not qualify as enterprise scale.
  • Enterprise benchmarks cover financial, legal, knowledge-work, and corporate tasks, yet many transform non-corporate data to avoid nondisclosure restrictions.
  • Synthetic knowledge bases provide a way to represent structured enterprise environments and produce grounded synthetic documents.

3 Dataset Construction

CorporateBench constructs scalable corporate corpora by simulating a temporally evolving knowledge base, generating consistent documents from it, and validating the resulting ontology and communications.

  • The construction pipeline supports arbitrary scale while maintaining consistent quality across generated documents.For any desired corpus size n, it can generate at least n documents.
  • Logical consistency requires that no document contradicts any fact established in the ground-truth knowledge base.
  • The pipeline first generates a company knowledge base, then creates emails that substantiate its organizational relationships, tasks, and project progress.
  • Templated documents provide arbitrary scale, while the knowledge graph can in principle be reconstructed from all sampled documents.
  • The ontology models companies, departments, teams, employees, projects, meetings, and tasks through seven relationship predicates across organizational, work, and communication axes.
  • Company generation simulates hierarchy, quarterly work, role-constrained meetings, and approximately 10% hiring that forces team splits and creates cross-document employment evidence.
  • Emails contain KB-derived senders, recipients, topics, timestamps, and evidence strings, with Claude Haiku 4.5 filling in remaining content for lexical diversity.Evidence strings cover all seven relationship types and vary in explicitness and temporal framing.

4 Dataset Validation

Validation shows that CorporateBench scales from a small company to a corpus exceeding 230,000 documents while preserving network structure, topic growth, and human-checkable relationship assertions.

  • CorporateBench contains four companies ranging from 12 to 10,210 individuals, with approximately 38 relationships per employee in the smallest company and approximately 37 in the largest.
  • 263,466 documents form four corpora containing 354, 3,926, 26,493, and 232,693 documents, respectively.
  • Topic counts scale from 6 in Zenith to 6,000 in Pound across the four company corpora.
  • Human validation uses balanced positive and negative email examples judged by non-expert reviewers on whether each document asserts a relationship.The validation sample contains 100 random emails and produces 1,000 reviewer judgements.

5 Tasks

CorporateBench separates extraction from question answering: extraction builds structured knowledge from documents, while QA reasons over an existing representation across temporal, topical, and integrated tasks.

  • Extraction: Extraction tests whether models recover relations and topics from company corpora.
  • Extraction: KB Evaluation extracts and deduplicates entities and relationships, producing triples evaluated against ground-truth triples with F1.
  • Extraction: Topic Classification assigns each document to 31 Biocure classes or 17 Pound classes on stratified 1,000-document test sets.
  • Question answering: QA uses factual questions generated from 250 templates, with complexity determined by the number of knowledge-graph nodes and documents requiring integration.
  • Question answering: RAG retrieves from all documents, whereas KB uses SQL over imported ground-truth data as an upper-bound comparison.
  • Question answering: KB QA handles entities, relationships, and temporal reasoning; Topic QA examines email topics; Integrated QA combines both capabilities.

6 Results

Across extraction and QA, performance exposes scale-related weaknesses: relational and retrieval-based reasoning deteriorate as corpora grow, while topic classification depends strongly on dataset structure and supervision.

  • KB evaluation: Entity extraction remains stronger and more stable than relationship extraction across models and dataset sizes.Entity F1 stays at 0.715–0.824, while relationship F1 declines from smaller to larger datasets.
  • KB evaluation: Temporal relationship extraction is hardest, declining from F1 0.142–0.470 on S to 0.062–0.173 on XL.
  • Topic classification: LLMs match TF-IDF+LR on Biocure with 200× less data but underperform the data-rich baseline on Pound.Biocure LLM F1 reaches 0.877–0.982 with 50 examples versus TF-IDF+LR’s 0.993 with 10K; Pound reaches 0.598–0.751 versus 0.983 with 100K.
  • QA results: Direct KB access consistently outperforms RAG, and the KB–RAG gap on KB QA grows from 0.24 at S to 0.37 at XL.The evaluation holds questions constant while larger companies provide proportionally more documents per question.
  • QA results: KB QA and Topic QA degrade with corpus size, whereas Integrated QA slightly improves for both methods.The authors associate the integrated pattern with richer cross-source context for synthesis.
  • Model differences: GPT-5 models perform best with RAG, while Claude Sonnet 4.5 excels at KB querying; GPT-5.1 also shows more early stopping in KB runs.Early stopping occurs 191 times in KB versus 56 times with RAG.

7 Conclusion

CorporateBench provides a large-scale benchmark for enterprise-specific retrieval and reasoning, with synthetic documents generated from consistent corporate knowledge bases. Its evaluations indicate that contemporary LLM performance declines as corpus scale approaches realistic enterprise conditions.

  • CorporateBench targets enterprise-specific retrieval and reasoning tasks for researchers and industry practitioners.
  • 263,466 synthetic documents span four procedurally generated corporate knowledge bases at varying scales.The benchmark supports corpora of arbitrary scale while maintaining inter-document logical consistency.
  • Five recent LLMs are marginally competent on enterprise tasks, with performance decreasing inversely with corpus scale up to and beyond 10^5 documents.
  • Future work should close the gap between synthetic and realistic benchmarks and test procedural knowledge-base generation in other domains.

Limitations

The benchmark has explicit scope and realism boundaries. It focuses on email-based communication over a 90-day simulation using LLM-generated documents, leaving broader channels, longer organizational evolution, and human-written corpora for future work.

  • CorporateBench focuses on email, excluding channels such as Slack and Microsoft Teams and document types such as reports and presentations.These alternatives have distinct structural characteristics that may pose different challenges for LLMs.
  • The simulation covers one quarter and does not model longer-term phenomena such as strategic pivots, mergers, cultural shifts, or institutional knowledge accumulation.The authors identify expansion beyond the 90-day window as future work.
  • LLM-generated documents may contain systematic biases or artifacts, and evaluated models may perform differently on human-written corporate documents.The authors characterize the tasks as a lower bound in difficulty relative to likely messier human-written corpora.
  • The QA datasets include assumptions and design decisions documented in Appendix J.

Ethical Considerations

CorporateBench constructs synthetic corporate worlds by combining procedurally generated knowledge bases, diverse evidence strings, hierarchical topics, and role-conditioned meetings. Its generation choices aim to preserve structured relationships and temporal or progress cues across documents.

  • Synthetic materials are generated through OpenAI and Anthropic API endpoints with content filtering enabled.The authors also note that benchmark use during training can create benchmark-specific output idiosyncrasies.
  • Evidence strings vary by temporal stage and explicitness, while meeting strings encode task and project progress through four progress bins.
  • Topic generation uses hierarchical structures whose depth scales with corpus size, with non-meeting topics sampled using a sparse Dirichlet concentration of 0.75.
  • Meeting topics are selected from a separate pool conditioned on meeting type, and minimal-content calendar emails are excluded from several topic and QA tasks.
  • Meeting simulation uses organizational roles and relationships to determine participants, schedules, generated artifacts, and substantiated knowledge-graph relationships.

F QA Benchmarks

The QA benchmark presents diverse document-grounded questions whose answers may require integrating many documents and applying constraints. Examples illustrate topic, date, organizational, and relationship reasoning across generated corporate emails and meeting artifacts.

  • Questions are generated from a small subset of examples designed to show diversity in questions, answers, and required documents.Many questions require analyzing many documents, but broad examples are omitted for conciseness.
  • Constraint count measures the number of document or knowledge-base filters an agent must apply, but does not necessarily indicate question difficulty.Low-constraint questions can be difficult because their answer scope is expansive.
  • Meeting-invitation reasoning can require identifying attendees and treating the answer as an unordered set.The example lists James Vera and James Moriera as attendees.
  • A meeting-minutes example uses progress language to support an estimated 30–40% completion state for target-company due diligence.
  • Topic QA can distinguish a granular topic from its broader hierarchy, with Label Negotiation Process nested under Labeling Requirements and Drug Approval.
  • A technical-debt example identifies Emilia Anglada as Jared Chartier’s only direct report, while the email falls outside the question’s date range.
  • Minimally constrained questions over the Pound knowledge base can return several thousand names, exceeding typical LLM context-window limits.

M KB Evaluation Results

Relationship extraction remains substantially harder than entity extraction, especially for temporal links and event-based relationships, while hierarchical organizational relationships are comparatively easier.

  • Relationship Error Patterns: 69% over-prediction for non-temporal worksOn affects Gemini 2.5 Flash Lite, while attend is under-predicted by 63–71% across models.Temporal extraction also shows over-prediction for worksOn, attend, and organize, with attend under-predicted at 50–54% across models.
  • Relationship Error Patterns: Core relationships such as reportsTo and worksAt have mostly < 10% error rates, indicating better performance on organizational hierarchies.
  • Topic Classification: F1 reaches 0.993 for TF-IDF+LR on Biocure with 10K examples, compared with LLM performance of 0.872–0.950 zero-shot and 0.877–0.982 with 50 examples.On Pound, LLM F1 rises from 0.357–0.531 zero-shot to 0.598–0.751 with 50 examples, while the baseline reaches 0.983 with 100K examples.

O QA Error Analysis

Programmatic failures increase with query complexity and are more frequent for knowledge-base queries than RAG queries, with long-context errors concentrated at larger scales.

  • Overall Error Rates: 697 of 30,000 model runs failed, a 2.3% overall failure rate; GPT-5.1 had the highest rate and Sonnet 4.5 the lowest reported count.GPT-5.1 recorded 247 errors, or 4.1% of its runs, while Sonnet 4.5 recorded 91 errors, or 3.0%.
  • Method Comparison: KB-based queries produced 547 errors versus 150 for RAG-based queries, or 4.7× more errors.The passage attributes this disparity to tool calling, output validation, and the multi-step demands of structured database queries.
  • Scale Effects: XL-scale KB questions generated 175 errors compared with 107 for S-scale questions, while 72 “Prompt too long” errors occurred almost exclusively at L and XL scales.The reported pattern links increasing query complexity with difficulties in context management and multi-step reasoning.
Loading 2608.27391v1…