Source-linked AI summary
The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents
Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
TL;DR
Enterprise LLM agents lack safe production data and exact ground truth for evaluation. The Era by Eon Benchmark generates a complete seeded fictional enterprise across product simulators and internal databases, with answers computed from final records. Across 23 companies, realism improved from 61.8 to 97.0 with zero records flagged synthetic, while a nine-model simulator comparison produced accuracy estimates from 42.4% to 76.8%.
Problem
Enterprise LLM agents cannot be evaluated directly on confidential customer production data, while existing substitutes lack adequate history or ground truth.
Method
The benchmark generates a seeded enterprise estate whose simulators and internal databases share one entity graph, with exact answers computed from completed records.
Results
61.8 to 97.0: mean realism across 23 generated companies, with zero records flagged as synthetic; nine models’ simulator accuracy estimates ranged from 42.4% to 76.8%.
Takeaways & Limitations
The benchmark measures factual enterprise tool use over a coherent fictional company with exact ground truth and reports statistical evidence alongside model accuracy estimates.
Takeaways & Limitations
The reported comparison covers one generated company and 33 Gong, Salesforce, and Zendesk simulator questions, not the full 83-scenario corpus.
Abstract
from arXiv · showhide
LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We present the Era by Eon Benchmark for evaluating LLM agents that use enterprise tools. The benchmark is built around a complete fictional company. It includes product simulators, company-specific internal databases, benchmark questions, and computed answer keys. Industry, company size, business model, application portfolio, and a seed define each company. One seeded entity graph supplies shared company data to simulators of Salesforce, Zendesk, Slack, Gong, and other products. A questionconditioned generator creates the schemas and records for internal databases. It takes shared entities, keys, and values from the same graph before generating database-specific facts. Both mechanisms therefore describe one consistent enterprise estate. Every expected answer is computed from the final records, so grading is exact. Design and answer-key checks validate the internal databases. A realism scorecard and adversarial detector validate the entity graph. Across 23 generated companies, the mean realism score rose from 61.8 to 97.0, with zero records flagged as synthetic. In the reported simulator-track comparison, nine models answered the same 33 questions three times each. Accuracy estimates ranged from 42.4% to 76.8%, and three of 36 pairwise differences remained supported after correction.
1 Introduction
The benchmark addresses the lack of confidential production data and exact ground truth for evaluating enterprise LLM agents by generating a coherent, repeatable fictional estate. Its simulators, internal databases, validation checks, and computed answers support exact evaluation.
- Motivation: Enterprise LLM agents need production-like data, but customer confidentiality prevents direct evaluation and existing substitutes lack sufficient history or ground truth.Vendor sandboxes are sparse, while manually built environments are narrow and costly to extend.
- Benchmark design: The benchmark deterministically generates a complete fictional enterprise from industry, company size, business model, application portfolio, and a seed.The shared entity graph includes employees, customers, sales, support, calls, messages, and documents.
- Benchmark design: Product simulators project one underlying entity graph, so cross-system records refer to the same entities through shared join keys.The data plane is read-only during evaluation.
- Benchmark design: A question-conditioned generator creates schemas and records for company-specific databases while copying shared entities and attributes from the graph.Consistency checks reject databases whose shared keys or values disagree with the graph.
- Answer generation: Every expected answer is computed from completed accessible records, including absent-information cases, enabling exact grading.Product-simulator questions use code templates, while internal-database questions guide schema and record generation.
- Validation: 61.8 to 97.0: mean realism across 23 generated companies, while records flagged as synthetic fell from 55.2% to zero.The scorecard and adversarial detector guided revisions of the entity-graph generator.
2 Related work
Prior enterprise-agent benchmarks rely largely on hand-built environments, whereas this benchmark combines multi-system generation, exact computed answers, measured realism, and generated internal databases.
- Hand-built benchmarks: Prior benchmarks cover limited domains, organizations, collaboration tools, or browser tasks using manually authored environments and tasks.Examples include τ-bench, CRMArena, TheAgentCompany, WorkArena, and WorkArena++.
- This benchmark: The benchmark differs by spanning many systems linked through one entity graph, computing answers from generated records, and measuring realism instead of assuming it.It also generates schemas and records for company-specific databases from business questions.
- Text-to-SQL connections: Controlled dirty values and unanswerable questions extend lessons from text-to-SQL benchmarks into enterprise-agent evaluation.These features test robustness to data quality and whether agents abstain when information is absent.
- Synthetic-data connections: Synthetic-data fidelity work motivates evaluating generated enterprises against reference statistics and detecting synthetic artifacts.The benchmark adopts realism scorecards and an adversarial detector for its entity graph.
3 One company, many systems
The benchmark constructs one seeded company graph and projects it into many faithful, partial product simulators and internal systems. Company activity is generated with dependencies, correlations, realistic temporal structure, and portfolio-specific integrations.
- Entity graph: One seeded generator creates the company identity, workforce, customers, and connected business activity shared by product simulators and internal databases.The graph provides the common factual substrate for the estate.
- Entity graph: The workforce models organizational structure, job ladders, managers, tenure, geography, employment type, and departed employees whose records remain.Offboarding and access-review scenarios motivate retaining records created by former employees.
- Entity graph: Customer activity is generated in dependency order, with customers assigned to sales representatives and downstream contacts, deals, documents, and communications built around them.Business-to-consumer configurations represent customers as individuals rather than companies.
- Entity graph: Larger accounts produce more tickets, meetings follow participants’ business days and time zones, and timestamps respect each record’s lifecycle.These correlations and constraints add operational structure beyond independent random draws.
- Product simulators: Simulators implement real-product interfaces and project slices of the same graph, including faithful cross-product representations and deliberately product-specific records.The fleet includes 66 products, with configured subsets provisioned per tenant.
- Application portfolio: The application portfolio controls which simulators and integration artifacts exist, while the underlying business remains unchanged across portfolio configurations.Unselected products provide empty workspaces, and the data plane is read-only with tenant-bound access.
4 Questions with computed answers
Questions and exact answers are generated from each completed scenario rather than maintained manually, with consistency checks supporting cross-system reasoning and internal-database validity. The benchmark also tests abstention, multi-step composition, and reproducibility.
- Simulator questions: Prior manual answer maintenance is replaced by questions and answers derived from each generated company.A scenario contains a generated company, its questions, and their computed answers.
- Simulator questions: Code templates instantiate scenario-specific questions and compute expected answers directly from generated records.Because answers derive from the data, they cannot disagree with it.
- Simulator questions: Adding a scenario requires only a company specification and seed, while adding a question requires only a new template; no answer is written by hand.This design makes benchmark extension inexpensive.
- Cross-system questions: Consistency checks verify that named entities appear across systems and refer to the same entity, rather than assuming cross-system agreement.These checks support the validity of composed questions.
- Cross-system questions: Twelve composed questions require multi-system, multi-step reasoning, such as grouping by attributes held elsewhere or ranking accounts by support activity.Their computations use keys that agents can follow through product interfaces.
- Question semantics: Two questions request absent information, and deterministic grading distinguishes exact values, sets, sequences, mappings, and abstention.Question text states definitions and tie-breaking rules to avoid unstated conventions.
- Internal databases: Internal databases are generated from business questions and graph entities, with schemas, planted rows, near misses, consistency gates, and answer keys computed from finished tables.Shared identifiers and values are copied from the graph before database-specific facts are added.
- Reproducibility: The same seed reproduces the identical database byte for byte, while a different seed changes values without changing schema and structure.Answers are recomputed for each generated company.
5 Validating the generated estate
The benchmark validates its generated estate with mechanism-specific checks: internal databases are compared with declared designs, while the shared entity graph is assessed for realism and synthetic artifacts. Iterative validation improved measured realism substantially, with safeguards against misleading small-sample scores.
- Internal databases: Internal-database validation checks adherence, question specificity, answer-key fairness, and declared dependencies without requiring external data.Checks compare completed databases with stored designs or transformed copies of their rows.
- Entity graph: The entity-graph realism scorecard evaluates marginal, joint, temporal, structural, and content properties against operational and published reference targets.Each axis targets a distinct way generated enterprise data can appear artificial.
- Measured improvement: 61.8 to 97.0: mean realism score across 23 companies after five generator-improvement passes, while the worst-company score rose from 48.6 to 92.8.Later passes raised the mean to 98.6; the adversarial detector flagged 55.2% of records initially and none after the fifth pass.
- Measured improvement: The scorecard was corrected for small-company sample effects, and the 61.8 baseline was recomputed under the same final scoring rules.This makes the reported improvement a comparison of two generators under one identical scorecard.
6 Evaluating agents on the estate
The evaluation harness runs agents through read-only MCP tools on isolated, frozen estates and grades answers deterministically against stored keys. In the simulator track, performance varied widely, but most pairwise differences were inconclusive and difficult multi-step operations dominated errors.
- Evaluation harness: Agents answer questions through roughly 290 MCP tools across simulators, with isolated tenants, read-only access, equal budgets, and verified serving.The harness prevents cross-run data leakage, record changes, budget differences, and simulator defects from being counted as model failures.
- Evaluation design: Nine models answered 33 questions three times each on one frozen estate, producing 99 recorded answers per model across single-system and composed questions.The reported track used Gong, Salesforce, and Zendesk records after excluding unsupported or unexposed source questions.
- Capability results: 3.7% multi-hop accuracy versus 92.6% for simple filtering: exhaustive aggregation, joins, temporal summaries, and rankings were harder than basic filtering.No model answered the median ticket-resolution-time question correctly, and several Gong or Salesforce–Gong tasks appeared consistently among the hardest.
- Capability results: 561 correct answers, 119 wrong answers, 99 false abstentions, six malformed outputs, and 106 runs ending before an answer were recorded across 891 runs.List-price inference for the 891 recorded runs totaled $343.27.
7 Limitations
The benchmark’s reported evaluation has limited coverage, its realism targets may be inaccurate, and its templated text cannot fully reproduce human-writing diversity.
- The model comparison covers one generated company and 33 simulator question shapes, not the full corpus of 83 scenarios.It evaluates only simulator questions using Gong, Salesforce, and Zendesk, excluding other existing questions and documents.
- Realism scores measure agreement with operational or published targets, some of which are estimates rather than independently verified truth.An incorrect target can pull the generator toward an incorrect distribution.
- Template-generated text passes current variety and coherence checks but cannot reach the full diversity, idiosyncrasy, and error patterns of human writing.A stronger detector might distinguish the generated text from human writing.
8 Conclusion
The Era by Eon Benchmark generates a coherent enterprise estate with exact answer keys for evaluating factual tool use. Its validation improved realism across generated companies, while simulator results showed broad model accuracy differences with limited statistical support.
- The benchmark generates vendor applications, internal databases, questions, and exact answers for one fictional company.A shared entity graph and question-conditioned database generator ensure that simulators and internal databases describe the same company.
- Mean realism rose from 61.8 to 97.0 across 23 companies, while the flagged share fell from 55.2% to zero.Design-based checks validate internal databases, questions, and labels; a scorecard and adversarial detector validate the entity graph.
- Accuracy estimates ranged from 42.4% to 76.8% across nine models, but broad intervals and corrected paired tests left most apparent differences inconclusive.The comparison contained 891 recorded answers and reports both point estimates and statistical evidence.