Source-linked AI summary
Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
TL;DR
Computable questions state the rules agents need, and code-enabled agents answer most of them, leaving limited separation among strong models. The paper adds eight questions that require inferring hidden facts from company data and evaluates agents on them; agents struggle especially to identify the right record among similar ones.
Problem
Computable questions barely separate strong agents, while business questions can depend on facts that no question or document states and that other data only implies.
Source: §1 · p. 1Method
The paper adds eight hidden-knowledge question templates, computes each generated company’s exact answer without a language model, and evaluates agents using two programs, one of which can run code.
Source: §3 · p. 3, §11 · p. 9Results
Agents sometimes found rules stated on calls, but almost none selected the right record from dates or small data marks; two such questions were answered correctly once in 72 runs.
Source: §11 · p. 9Takeaways & Limitations
The results distinguish agents’ ability to compute over data from their ability to identify which records and rules the company relies on.
Source: §9 · p. 8Results come from one company and one seed, with only 24 runs per agent and confidence intervals up to 37 percentage points wide.
Source: §10 · p. 8
Abstract
from arXiv · showhide
In the Era by Eon benchmark, each question states the rules for its answer, and code computes the answer from a generated company's data. When agents can run code, the four strongest models each answer 22 to 25 of 27 such questions, so the benchmark barely separates them. We add eight question templates that depend on hidden facts. No question or document states a hidden fact, and the records that seem to hold it show something else. Other data implies it. For example, the sales system says a customer dropped a purchase because of timing. On a recorded call, the customer blames an outage. For each generated company, code fills each template and computes an exact answer without a language model. We evaluate 12 agents. Each pairs a model with an agent program, which connects it to the company's systems. The best agent answers 18 of its 24 attempts, three per question, correctly. Four of the six models answer at most 6 of 24 with any program. The hardest questions require picking one of several similar records, such as which of three renewal offers a customer signed. All agents together answered two such questions correctly in only 1 of 84 attempts.
1 Introduction
The paper extends Era by Eon because its computable questions barely separate strong agents, adding questions that require hidden knowledge rather than only stated rules. It defines this challenge, describes how the questions are generated, and evaluates agents on them.
1 Introduction
- With code, each of the four strongest models answered 22 to 25 of 27 computable questions, leaving little separation among them.These results motivate adding questions that depend on facts not stated in a question or company document.Source: §1 · p. 1
- Hidden knowledge is a fact unstated in questions and documents but implied by other data, as when a call contradicts a CRM loss reason.The agent must find the relevant remark among many calls and connect it to the pertinent support ticket.Source: §1 · p. 1
- The best agent answered 18 of 24 runs correctly; two record-selection questions were answered correctly in only 1 of 84 runs across all agents.The hardest questions require choosing the right record among similar alternatives, such as which renewal offer was valid when signed.Source: §1 · p. 1
- The paper defines hidden knowledge, distinguishes three kinds, and states four conditions for a question that depends on it to have one exact answer.These contributions are described in Sections 4 through 6.Source: §1 · p. 1
- The paper adds eight question templates, explains how each question and answer are generated without a language model, and evaluates 12 agents on one company’s questions.The templates select an entity from generated company data, fill its name into the question, and compute the answer; changing the random seed changes the selected entities and answers.Source: §1 · pp. 1–2
2 Background: the Era by Eon benchmark
Era by Eon generates a fictional company and serves its data through simulators of business applications. The simulators share a consistent entity graph, and code computes an exact answer for each question.
2 Background: the Era by Eon benchmark
- The benchmark generates a fictional company from a specification that includes its industry, size class, business model, applications, and random seed.Generation is deterministic: the same specification produces the same company and entity graph.Source: §2 · p. 2
- Simulators serve company records through REST APIs and MCP servers, including CRM, help-desk, engineering, call-recording, chat, and file-store systems.Agents have read-only access, and a deal, ticket, or recorded call with its transcript counts as a record.Source: §2 · p. 2
- Because all simulators read from the same entity graph, company details remain consistent across systems, although some systems are made to disagree deliberately for benchmark questions.The authors describe the simulators together as a digital twin of the fictional company.Source: §2 · p. 2
- Code computes the correct answer from served data, and a grader compares an agent’s answer with that key.The benchmark calls the generated-company answer the key.Source: §2 · p. 2
- The company’s business data includes accounts, contacts, deals, contracts, proposals, support tickets, Jira issues, and a sales pipeline whose committed amount is computed by each company’s own method.The company is generated as a large B2B fintech firm, with records spread across its simulated systems.Source: §2 · p. 2
3 Limits of computable questions
The paper asks whether agents that can run code can answer the benchmark’s computable questions, where every needed rule is stated. It compares two agent programs and finds that code-enabled agents answer most of these questions, motivating tests of hidden knowledge.
3 Limits of computable questions
- The earlier benchmark tested nine models on 33 computable questions, with average accuracy varying sharply by capability, from 3.7% on chained lookups to 92.6% on condition-based record selection.The earlier setup used a small fintech company and three runs per question.Source: §3 · pp. 2–3
- The LangGraph program adds sandboxed Python execution to the same simulator interfaces available through the Era agent program.The Era program cannot run code; LangGraph scripts send requests through the agent program to simulator REST APIs.Source: §3 · p. 3
- The new Era program can use both simulator REST APIs and MCP servers, whereas the earlier version used MCP servers only and could not run code.Its limits are 165 turns, 165 tool calls, and 32,000 characters per tool result.Source: §3 · p. 3
- The current computable-question set has 27 questions, and most ask for several values about one subject, with the whole answer required to be correct.One example requires joining 250 Salesforce accounts with 3,000 Zendesk tickets and returning percentages for three account tiers.Source: §3 · p. 3
- With LangGraph, the four strongest models answered 22 to 25 of the 27 computable questions correctly, placing them within three questions of one another.For GPT-6 Astra and GPT-5.6 Sol, code execution added 16 and 21 correct answers, respectively; the authors caution that one run per question gives little evidence for ranking the four strongest models.Source: §3 · p. 3, Table 1 · p. 3
- Computable questions state every rule needed to derive the answer, while questions in this paper depend on facts unstated in the question or company documents.A script can directly follow lookups, read records, and sort results, but hidden-knowledge questions require interpreting other data, such as a call remark.Source: §3 · pp. 2–3
4 Hidden knowledge
The paper distinguishes hidden facts from computable rules: answers may depend on facts unstated in the question, documents, and records being queried, while other served data provides evidence. It groups the questions into three kinds and requires four conditions to make each answer uniquely gradable.
4 Hidden knowledge
- The section introduces hidden facts as an extension beyond computable questions, whose answer rules are stated in the question or a document.It also previews the three kinds of hidden knowledge and the four conditions for an exact answer.Source: §4 · p. 3
4.1 Definition
- A hidden fact determines the key, is stated neither in the question nor any document or queried record, and is evidenced by other served data.In the service-credit example, the contract shows zero while a recorded-call remark supplies evidence for the promised rule.Source: §4.1 · p. 4
4.2 Three kinds of hidden knowledge
- The eight questions span three kinds: call remarks provide evidence in four, competing records must be resolved in three, and one requires recovering a computation method from an old report.The service-credit question is one of the four based on a call remark.Source: §4.2 · p. 4
4.3 Conditions for an exact answer
- Every hidden-fact question must use served evidence, yield a unique answer, defeat the obvious record-based answer, and specify an exact, fully graded response.The answer consists of two or three values, and every value must match the key.Source: §4.3 · p. 4
5 From templates to questions
Eight templates turn generated company data into questions whose answers depend on hidden facts. Code creates the companies, selects cases, writes clues, and computes answers deterministically; checks ensure a question has a supported answer before agents receive it.
5 From templates to questions
- Figure 1 divides the service-credit template into four generator steps that shape the company and three assembly steps that produce the question and key.The section follows this template from company generation through the agent’s question.Source: §5 · p. 4, Figure 1 · p. 5
- The service-credit template selects a customer with urgent tickets in at least two contract months, computes a capped credit, and stores the answer outside the company systems.For Dovewood Analytics, four qualifying months produce a 15% credit starting in December 2023, while its contract record remains at zero.Source: §5 · p. 5
- A call remark states the credit rule, while tickets and the contract provide the values; the benchmark recomputes the answer from served data and drops questions that fail its checks.The checks require exactly one contract, a zero credit field, and agreement between the recomputed answer and the stored key.Source: §5 · p. 5
- For fintech companies, the four sizes mid, large, enterprise, and hyperscale had enough eligible customers for all eight templates, so the benchmark adds the questions to those companies.Source: §5 · p. 6
- Code generates the company, chooses cases, writes clues, and computes answers without a language model; identical specifications and seeds produce identical results.The eight templates require enough eligible customers, which companies of size mid or larger have.Source: §5 · p. 6
- With seed 43, the same template selects Swinburn Holdings and produces a different question and answer, including 13 covered months rather than Dovewood Analytics’ 10.Both customers reach the 15% cap, but Swinburn Holdings’ contract ends three months later.Source: §5 · p. 6
6 The seven other templates
The seven other templates fall into three patterns: clues on recorded calls, evidence that distinguishes among competing records, and rules recovered from an old report. Each produces an exact question for a generated company.
6.1 A remark on a call
- Four templates use a call remark as a clue; the agent must combine its rule with system records because the answer itself is not stated on the call or stored in a system.The service-credit template is one example of this pattern.Source: §6.1 · p. 6
- The lost-deal and departed-contact questions ask agents to identify a roadmap blocker or a former call participant, including the associated amount or departure details.The examples ask for a deal amount and issue key, or a contact name, departure month, and CRM activity status.Source: §6.1 · p. 6
- The dropped-expansion question asks for the actual cause, its related support ticket, and the difference between the lost deal amount and renewal value.Source: §6.1 · p. 6
6.2 Several records, one is right
- Three templates require identifying which record the company relies on, using evidence such as a system-of-record property, a project’s last-change date, or a document’s price-validity line.Source: §6.2 · p. 6
- The conflicting-amount and roadmap questions ask for the forecast-committed deal amount and quarter, or the engineer with the most open roadmap stories.The examples involve different CRM amounts and dates, and old versus live Jira roadmap projects.Source: §6.2 · pp. 6–7
- The signed-proposal question distinguishes among renewal rounds by asking which proposal was executed and what uplift it carried.Source: §6.2 · p. 7
6.3 A method in an old report
- One template recovers the commit model from a pipeline-review report and reapplies it to current Salesforce records to calculate the salesperson’s committed pipeline.The example asks for the total committed dollars and the number of opportunities counted under the review’s rules.Source: §6.3 · p. 7
7 Experimental setup
The experiment tests 12 combinations of six language models and two agent programs on eight hidden-knowledge questions. Each agent answers each question three times, and correctness requires an exact match on every requested field.
Agents
- The 12 agents pair each of six models with both programs: Era, which cannot run code, and LangGraph, which can.Source: §7 · p. 7
Limits and access
- Agents have read-only access to company systems, and the benchmark’s generated data, answer keys, and answer file are hidden from the sandbox.Era runs stop at 150,000 output tokens; LangGraph runs stop after one hour, with scripts limited to five minutes.Source: §7 · p. 7
Runs and grading
- Each agent makes three attempts per question for 24 runs, and a run is correct only if every requested JSON field matches the key.Abstentions and runs that reach a limit without answering count as incorrect.Source: §7 · p. 7
What we report
- The report gives correct runs out of 24, Wilson 95% confidence intervals, and abstention counts, alongside each agent’s score on 27 computable questions.The computable-question comparison indicates whether strength on those questions also corresponds to strength on hidden knowledge.Source: §7 · p. 7
8 Results
Across the 12 agents, Claude Fable 5.1 performed best, while questions requiring record selection were especially difficult. Code helped GPT-6 Astra on computable questions but not on the hidden-knowledge questions.
Questions with a remark on a call
- Across four questions about a remark on a call, agents answered 54 of 144 runs correctly; the departed-contact question had 21 of 36 correct, versus 10, 13, and 10 for lost-deal, dropped-expansion, and service-credit questions.Source: §8 · p. 7
Overall results
- Claude Fable 5.1 answered 18 of 24 runs in Era and 15 in LangGraph; GPT-6 Astra answered 13 and 12, while the other four models scored at most 6 in either program.Table 2 reports results for 12 agents across eight questions, with three runs per question.Source: §8 · p. 7, Table 2 · p. 8
Several records, one is right
- The roadmap question was correct in none of 36 runs, and the signed-proposal question once; together, agents answered these two questions correctly once in 72 runs.Ten of 12 agents got both values wrong in every run.Source: §8 · p. 8
A method in an old report
- 12 of 36 runs were correct on the pipeline question, all from the four Claude Fable 5.1 and GPT-6 Astra agents, each of which answered all three runs correctly.Source: §8 · p. 8
How much the agents searched
- The four Claude Fable 5.1 and GPT-6 Astra agents averaged 7–17 tool calls on roadmap and signed-proposal questions, versus 17–109 on the other six, and none abstained.They answered the two hardest questions quickly and incorrectly.Source: §8 · p. 8
Does code help?
- GPT-6 Astra answered 24 computable questions in LangGraph versus 8 in Era, but scored 12 versus 13 on the eight hidden-knowledge questions; Claude Fable 5.1 scored 15 with code versus 18 without.DeepSeek-V3.2 answered none in either program.Source: §8 · p. 8
9 Discussion
The results distinguish computing over company data from identifying which records and rules the company relies on. The hardest record-selection questions expose this distinction, while a written store of employee-held facts is proposed as a way to measure whether it helps.
Two abilities
- Strong models nearly solve computable questions with code, but only partly pass questions about which records the company relies on and which rules apply.The paper frames these as two distinct abilities: computing over data and knowing which records and rules matter.Source: §9 · p. 8
Several records
- All agents answered the roadmap and signed-proposal questions correctly once in 72 runs, despite the four strongest agents making fewer calls on them and never abstaining.The proposed explanation is that agents read the named record, find a plausible value, and stop.Source: §9 · p. 8
Does code help?
- Code did not help on the hidden-knowledge questions, although it helped substantially on computable ones; reading many records does not identify which ones the company relies on.Source: §9 · p. 8
A written store of facts
- The missing knowledge is described as a handful of employee-learned facts, and the benchmark could measure the effect of a written store by comparing agents with and without it.Source: §9 · p. 8
10 Limitations
The evaluation covers one generated company and one seed, with 24 runs per agent and no human baseline. The paper also identifies limitations in the generated call data, the logged results, and the range of models evaluated.
Evaluation scope
- All results come from one company and one seed; although the seed-43 company’s eight questions pass Section 5 checks, agents were not run on them, and 95% intervals are up to 37 percentage points wide.Source: §10 · p. 8
Human comparison and repeated remarks
- Human performance on the eight questions from the systems alone was not measured, and repeated call remarks could be learned because they use the same sentence across companies.Source: §10 · p. 9
Generated-data defects
- In four templates, the remark is the final call turn although earlier turns do not refer to it; in two templates, remark-based record selection uses metadata rather than content.Source: §10 · p. 9
Models and result files
- The six tested models come from three providers, and results may differ for other models; result files omit answers and tool calls and describe models available in September 2026.Source: §10 · p. 9
11 Conclusion
The paper extends Era by Eon with eight templates whose answers depend on hidden knowledge and can be generated exactly from company data without a language model. On the evaluated company, agents answered some hidden-knowledge questions, but rarely identified the right record when only subtle data clues pointed to it.
Benchmark extension
- The benchmark adds eight hidden-knowledge question templates, with each generated company's questions and exact answers computed from its data without a language model.The evidence is in company systems but not in the record named by the question.Source: §11 · p. 9
Results and next step
- The best agent answered 18 of 24 runs, while four of six models answered at most 6; two record-selection questions were correct once in 72 runs.The authors propose measuring whether a written store of employee-known facts closes some of the gap.Source: §11 · p. 9