PaperlayerView exact revision on arXiv

Source-linked AI summary

EPSBench: Benchmarking LLM Agents for Earnings Forecasting

Yanbin Yin, Chuyu Xu, Liqin Ye, Jing Peng, Chao Zhang

arXiv:2610.38293v1cs.CLcs.AI

TL;DR

Quarterly earnings forecasts require agents to turn incomplete, sometimes conflicting evidence into reported earnings magnitudes. EPSBench records cited estimates of company-specific income-statement drivers and evaluates the resulting EPS forecasts and intervals. Three models show no detectable paired difference in point forecast error from consensus, but all six models’ 80% intervals fall short of nominal coverage.

  • Problem

    Forecasting a reported earnings magnitude requires weighing incomplete and sometimes conflicting evidence, since higher sales need not mean higher profit if costs rise faster.

    Source: §1 · p. 1

  • Method

    EPSBench has agents estimate company-specific income-statement drivers from dated filings, earnings-call transcripts, and news, then uses a shared calculator to produce GAAP diluted EPS.

    Source: §1 · pp. 1–2

  • Results

    Three models show no detectable paired difference in point forecast error from consensus and three have higher error; all six achieve only 37–70% coverage for nominal 80% intervals among runs returning an interval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The driver-level records locate most of the error gap between model groups in operating-income estimates.

    Source: §6 · p. 9

    The benchmark does not establish whether its forecasts are suitable for investment decisions.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Quarterly earnings forecasting requires an agent to connect financial evidence to the income-statement line items that produce a company's reported earnings. We introduce EPSBench, a benchmark for studying this ability across 149 firm-quarters from 86 US companies. Agents read a fixed collection of filings, earnings-call transcripts, and news, then estimate company-specific income-statement drivers with cited evidence. A shared accounting calculator converts these estimates into GAAP diluted earnings per share (EPS), providing a common basis for comparing forecast assumptions. We evaluate six models under a common harness, assessing point forecast error relative to date-matched analyst consensus and checking whether their prediction intervals achieve the stated coverage. Three models show no detectable paired difference in point forecast error from consensus, and three have higher error. Across all six models, nominal 80% intervals contain reported EPS in only 37--70% of runs that return an interval, and most misses fall above the upper bound because the forecasts run low. The driver records trace most of the error gap between models to operating-income estimates.

1 INTRODUCTION

EPSBench structures quarterly earnings forecasting around the income-statement drivers that determine GAAP diluted EPS. Across six models, three show no detectable paired difference in point forecast error from consensus, while all six miss the stated coverage of their 80% intervals.

1 INTRODUCTION

  • A magnitude forecast must weigh incomplete and sometimes conflicting evidence because higher sales need not mean higher profit when costs rise faster.The forecast should put a size on these effects and state what remains uncertain.Source: §1 · p. 1
  • Unlike benchmarks that ask for an EPS figure and leave the route to it in the model’s trajectory, EPSBench records income-statement estimates and their evidence.Its contributions include company-specific schemas, an evaluation protocol linking dated evidence and driver estimates to scores, and released materials for reproducible comparisons.Source: §1 · pp. 1–2
  • EPSBench organizes quarterly earnings forecasting around company-specific income statements and contains 149 firm-quarters from 86 US companies.The income-statement line items combine into GAAP diluted EPS.Source: §1 · p. 1
  • Agents estimate company-specific drivers from pre-release filings, earnings-call transcripts, and news, support estimates with cited evidence, and submit them to a shared calculator.The calculator returns implied EPS, while agents report prediction intervals; fixed arithmetic keeps scores attached to the driver estimates and evidence that produced them.Source: §1 · p. 2, Figure 1 · p. 2
  • Three models show no detectable paired difference in point forecast error from date-matched analyst consensus, while three have higher error.All six models’ 80% intervals cover only 37–70% of outcomes that return an interval, and the driver records locate most of the error gap in operating-income estimates.Source: §1 · p. 2

2 RELATED WORK

EPSBench connects financial evidence to structured income-statement forecasts of future GAAP diluted EPS. It also combines point and interval outcomes with execution metrics and controls intended to address retrospective-evaluation concerns.

2 RELATED WORK

  • EPSBench evaluates GAAP diluted EPS, so adjusted earnings figures must be distinguished from the target when used as evidence.Companies and analysts also report adjusted or non-GAAP earnings that exclude selected items.Source: §2 · p. 2
  • EPSBench compares quantitative assumptions by having agents estimate company-specific drivers whose accounting relationships determine EPS.Outcome-based metrics assess both the point forecast and interval.Source: §2 · p. 3
  • Unlike financial benchmarks centered on disclosed-report reasoning or document retrieval, EPSBench forecasts future earnings from point-in-time evidence through structured income-statement forecasts.The driver estimates determine GAAP diluted EPS.Source: §2 · p. 3
  • EPSBench combines point and interval evaluation with execution metrics, tool-enforced point-in-time access, and a memorization test to address evaluation concerns.The cited concerns include uncertainty calibration, execution cost and reliability, and retrospective-forecast validity risks from training exposure or revised or misdated information.Source: §2 · p. 3

3 EPSBENCH

EPSBench asks agents to forecast reported GAAP diluted EPS using dated documents and company-specific accounting schemas. Its workflow preserves driver evidence and calculator outputs, while the evaluation measures point error, interval behavior, and execution.

3 EPSBENCH

  • Each EPSBench instance asks for GAAP diluted EPS using documents available by the day before results, with driver estimates and evidence passed through accounting identities to produce an 80% interval forecast.The forecast and interval are scored against reported EPS.Source: §3 · p. 3

3.1 TASK INSTANCES AND INFORMATION CUTOFF

  • The benchmark contains 149 firm-quarters from 86 large, widely followed US companies across eight sectors, with the target quarter already ended but results not yet published at the cutoff.The announcement window is 2 December 2025 to 24 June 2026, and the sample is shared across model, information-source, and harness comparisons.Source: §3.1 · p. 3

3.2 COMPANY-SPECIFIC ACCOUNTING SCHEMAS

  • Each firm-quarter schema combines a year-ago baseline, forecast drivers, and a directed acyclic graph of accounting identities that maps line items to diluted EPS.Schemas are built from each company’s historical statements; 146 of 149 reproduced reported EPS within one cent when validated on extracted year-ago values, and the other three within four cents.Source: §3.2 · p. 4

3.3 AGENT FORECASTING

  • Agents use filings, the prior earnings call, and news through a point-in-time tool server, submit evidence lines for each driver, and use a calculator to generate EPS.They report p10 and p90 bounds without moving the calculator’s EPS point forecast to the interval midpoint; each run also records the trajectory and intermediate accounting nodes.Source: §3.3 · p. 4

3.4 EVALUATION MEASURES

  • Evaluation compares price-deflated point error with dated analyst consensus and assesses intervals by coverage, tail rates, width, and interval score.Forecasts are paired by instance for point-error comparisons; interval outcomes are pooled across runs that return an interval, and resource use and format compliance are also recorded.Source: §3.4 · pp. 4–5

4 EXPERIMENTAL SETUP

The main experiment compares six models on the same 149 tasks and varies information sources, harnesses, and reasoning effort in additional experiments. The setup also checks temporal separation from model training data and tests for memorization.

4 EXPERIMENTAL SETUP

  • Six models from four vendors are evaluated on the same 149 tasks, schemas, information environment, and protocol, with three runs per configuration.Additional experiments vary GPT-5.5’s information and vary harness or reasoning effort for GPT-5.5 and Opus 4.6.Source: §4 · p. 5
  • The primary harness disables shell, file, and web tools, while additional tests use vanilla OpenAI and Claude Code harnesses and remove selected information sources.Harnesses receive the same prompts apart from changes required when sources are removed.Source: §4 · p. 5
  • All benchmark announcements follow the published knowledge cutoffs of four models; a memorization test found no detectable recall of benchmark outcomes for Qwen 3.5 under the tested protocol.GLM 4.5 Air predates the evaluation window, while Qwen 3.5 was released within it; two models do not disclose a cutoff.Source: §4 · p. 5

5 RESULTS

Three models have point forecast error statistically indistinguishable from analyst consensus, while three have higher error; none of the models’ 80% intervals reaches nominal coverage. Model differences are largest on quarters with large year-over-year earnings changes, and the error gap is concentrated in operating income; source removals and harness changes reveal further differences in information dependence and resource use.

5.1 POINT FORECASTS

  • Three leading models showed no detectable paired PDE difference from consensus, while the other three had higher error; median PDE was 0.113–0.131% for leaders versus 0.190–0.215% for trailers.Each leading model had lower point forecast error than each trailing model, while no two leading models differed significantly.Source: Table 1 · p. 6, Figure 2 · p. 6, §5.1 · p. 6

5.2 INTERVAL FORECASTS

  • Only 37% of Qwen 3.5’s and 70% of Opus 4.6’s reported-EPS outcomes fell inside their 80% intervals; every model had 20–38% above its upper bound.Low point forecasts pull nearly symmetric intervals downward; coverage alone does not rank interval quality, as GPT-5-mini’s mean interval score was 19.61 versus 2.87–6.48 for every other model.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 ANALYSIS OF FORECAST DIFFERENCES

  • On 50 far instances, leading models’ median PDE was 0.392–0.398%, versus 0.599–0.716% for trailing models; the groups’ error gap lies almost entirely in operating income.Leading models halved year-ago EPS error on 29–30 of 50 instances versus 12–19 for trailing models, and cited a number for the most-adjusted driver in 95–96% versus 35–72% of runs.Source: §5.3 · pp. 7–8, Table 2 · p. 8

5.4 CONTRIBUTIONS OF FILINGS, TRANSCRIPTS, AND NEWS

  • For GPT-5.5, removing both news and filings/transcripts raised paired PDE by +0.033 pp [+0.019, +0.047] and lowered the consensus win rate from 50% to 31%.Either source alone barely moved median error, yet 50 instances depended on a particular source that could not be replaced.Source: §5.4 · pp. 8–9, Table 3 · p. 9

5.5 SENSITIVITY TO THE AGENT HARNESS

  • Changing the harness or reasoning effort barely moved point error (−0.004 to +0.001 pp; all five intervals included zero), while resource use changed considerably.For Opus 4.6 under OpenCode, maximum effort raised median output tokens per run from 9.7K to 40.5K.Source: Table 4 · p. 9, §5.5 · p. 9

6 CONCLUSION

EPSBench studies how agents turn financial evidence into earnings forecasts and compares their point and interval forecasts with analyst consensus. The conclusion reports that leading agents match consensus on point-error comparisons, while all six models miss nominal interval coverage; accompanying statements describe study boundaries, release materials, and related work.

6 CONCLUSION

  • Leading agents show no detectable point-error difference from date-matched analyst consensus, but all six models’ intervals fall below 80% coverage, with more outcomes above than below the bounds.Model differences are largest when earnings move sharply from the year-ago level, with most of the gap concentrated at the operating-income stage.Source: §6 · p. 9

AI USE STATEMENT

  • The authors report using generative AI for schema extraction, benchmark implementation, and manuscript drafting and revision, while retaining responsibility for validating the implementation and manuscript accuracy.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses public filings, licensed earnings-call transcripts, CC-News, academic IBES/WRDS consensus, and Yahoo Finance share prices; it does not establish suitability for investment decisions.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The release includes schemas, tools, prompts, scores, and trajectories, but trajectories replace transcript text with document identifiers and IBES consensus values are not released.Extending the benchmark to new quarters will require another extractor because the schema-generation model used in 2026 is no longer served.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL RELATED WORK

  • EPSBench situates its task in earnings forecasting by having agents estimate company-specific income-statement drivers from retrieved documents and evaluate quarterly GAAP EPS forecasts and intervals.Source: A ADDITIONAL RELATED WORK · p. 15
  • EPSBench applies controlled model and harness comparisons to numerical forecasting, holding the main task, tools, calculator, and harness fixed while varying information sources or harnesses in additional comparisons.Source: A ADDITIONAL RELATED WORK · pp. 15–16

B ENVIRONMENT AND PROTOCOL DETAILS

B.1 SCHEMA CONSTRUCTION AND VALIDATION

The appendix describes how EPSBench constructs and verifies company-specific schemas, prepares date-restricted documents, and runs agents through a shared tool protocol. Schema checks closely reproduce reported EPS, while news effective dates remain an approximation of historical availability.

B.1 SCHEMA CONSTRUCTION AND VALIDATION

  • Historical filings supply income-statement data for schema generation, including the four preceding quarterly statements; a missing standalone fourth-quarter statement is derived from annual and nine-month figures.o4-mini extracts statement tables, DeepSeek’s deepseek-reasoner generates baseline leaves, formula DAG, and drivers, and GPT-5.4 extracts tax and share-count references.Source: §B.1 · p. 16
  • Schemas use four driver forms—year-over-year growth, margin change, absolute value, or tax-rate override—and historical reference values provide defaults for tax and share-count drivers.The median of last-quarter, trailing-twelve-month, and year-ago references is used for most tax and share-count defaults; adjusted tax-rate guidance is excluded because it omits items in evaluated GAAP earnings.Source: §B.1 · pp. 16–17
  • 146 of 149 schemas reproduce historical EPS within one cent, and the remaining three within four cents; released verification results support reproduction without regenerating schemas.The authors note that rounding individual line items can accumulate in the final EPS calculation.Source: §B.1 · p. 17

B.2 INFORMATION ENVIRONMENT

  • News articles receive a conservative effective date based on the latest structured-markup, body, or URL date, with crawl date substituted when the body date is missing.Because timestamps may be incomplete and pages may be updated, the effective date is treated as an approximation rather than a perfect reconstruction of historical availability.Source: §B.2 · p. 17
  • Agents are offered prior-quarter and year-ago filings and a prior-quarter transcript, while the server blocks target-quarter documents and material published after the cutoff.Source: §B.2 · p. 17

B.3 TOOLS AND RUN PROTOCOL

  • The tool suite supports selective filing reads, transcript and news retrieval, and financial calculation, which applies submitted drivers to the schema’s accounting DAG.Source: §B.3 · p. 18
  • Each run is a single attempt with no retry or turn limit beyond context; missing or post-cutoff document requests return errors, and unparsable final JSON is recorded as a format failure.Source: §B.3 · p. 18

B.4 EVALUATION AND AGGREGATION

The evaluation appendix defines an interval score for reported quantiles and summarizes resource-use measures and a company-level resampling check. The score balances interval width against penalties for missed outcomes, and company-level resampling leaves the consensus comparisons unchanged.

B.4 EVALUATION AND AGGREGATION

  • The interval score penalizes misses by ten times their distance from the nearest bound and charges for interval width, with lower scores indicating better performance.The authors report mean scores over runs returning intervals because propriety holds for the expected score; a median would fall among covered runs and reward narrow intervals.Source: §B.4 · p. 18
  • Efficiency measures include model requests, tool calls, output and reasoning tokens, total input tokens, and final-message format compliance, summarized by medians except format compliance.Source: §B.4 · p. 18
  • Resampling companies rather than instances moves consensus-comparison interval endpoints by at most 0.002 pp and changes no conclusion.Source: §B.4 · p. 18

C COMPANIES IN THE BENCHMARK

The benchmark sample covers large, widely followed US companies across eight sectors. Its 86 companies are predominantly above $10 billion in market capitalization.

C COMPANIES IN THE BENCHMARK

D MODEL DETAILS

The appendix describes the benchmark companies and evaluated model specifications, then explains how harness and reasoning-effort comparisons are controlled. Harnesses mediate model interaction with tools, while the experiments vary harness or effort against otherwise fixed inputs.

D MODEL DETAILS

  • The benchmark spans 86 companies across eight sectors, and Table 6 records model specifications and vendor-disclosed release dates and knowledge cutoffs.Qwen 3.5 and GLM 4.5 Air have undisclosed knowledge cutoffs.Source: D MODEL DETAILS · p. 18, Table 5 · p. 19, Table 6 · p. 19

E HARNESS SETTINGS

  • Harness experiments hold the 149 instances, schemas, MCP server and tools, and prompts fixed while varying harness and reasoning effort.All harnesses use the same benchmark prompts and MCP server; selected models also run at maximum reasoning effort.Source: E HARNESS SETTINGS · p. 19
  • OpenCode is the primary harness; Claude Code runs Opus 4.6, while a vanilla OpenAI SDK loop tests GPT-5.5 without added context management or orchestration.OpenCode and Claude Code disable built-in shell, file, and web tools, leaving the six MCP tools as the interface.Source: E HARNESS SETTINGS · p. 19

F MEMORIZATION TEST FOR QWEN 3.5

The memorization test probes whether Qwen 3.5 can recall reported EPS without tools or documents, including during the benchmark period. Under the tested hit threshold, it produces no hits on benchmark instances and none for announcements from January 2025 onward.

Probe

  • The test asks for named-company GAAP diluted EPS without tools or documents, using direct and guided prompts and three attempts per prompt.A hit requires PDE at most 0.02%; copying analyst consensus, the strongest tested non-memorizing strategy at that threshold, has a 9% hit rate.Source: F MEMORIZATION TEST FOR QWEN 3.5 · p. 20, Table 8 · p. 20

Firm-quarters probed

  • The test spans announcements from June 2024 to June 2026, with 247 sampled firm-quarters across nine quarter buckets plus all 149 benchmark instances.Qwen 3.5 was released during the benchmark window without a published knowledge cutoff.Source: F MEMORIZATION TEST FOR QWEN 3.5 · pp. 19–20

Results

  • Across all 149 benchmark instances, Qwen 3.5 produced no hit at PDE ≤0.02% across six responses per instance, indicating no detectable recall under this protocol.There were also no hits for announcements from January 2025 onward; hits occurred only for announcements in 2024.Source: F MEMORIZATION TEST FOR QWEN 3.5 · p. 20, Table 9 · p. 21

G ADDITIONAL RESULTS

The additional analyses describe interval calibration, resource use, information-source effects, and how forecast error is decomposed across accounting stages. They also specify the measures and comparison procedures used to interpret these results.

Detailed evaluation results

  • Interval coverage, tail rates, width, and score are computed only over runs that return an interval, with per-model denominators reported.Table 10 defines signed error as (y−ŷ)/P; positive values indicate underprediction.Source: G ADDITIONAL RESULTS · p. 21, Table 10 · p. 21

Resource use and compliance

  • All models return parseable forecasts in at least 98.2% of runs, but format compliance does not track point forecast error.GPT-5-mini has the most input at 1.35M tokens per run, while Opus 4.6 and GPT-5.5 achieve lower error with fewer input tokens.Source: G ADDITIONAL RESULTS · p. 21, Table 11 · p. 22

Source-condition interaction

  • For GPT-5.5, the median instance-level change in PDE from removing both source types beyond their separate effects is +0.037 pp, with a 90% interval of [+0.020, +0.066].The estimate is not the difference between the displayed medians in Table 3.Source: G ADDITIONAL RESULTS · pp. 21–22

Source-removal groups

  • Source-removal effects are differences in per-instance median PDE; a removal is harmful above the 0.057 pp median run-to-run spread, and the single-removal effects have Spearman correlation −0.01.The table groups instances by whether no removal, joint removal only, or at least one single removal is harmful.Source: G ADDITIONAL RESULTS · p. 22, Table 12 · p. 22

Accounting-stage decomposition; Evidence coding for Table 2

  • The accounting-stage decomposition expresses operating-income error and its pretax-income and EPS increments on the PDE scale, then counts which component is largest in absolute magnitude.It covers 125 instances with an operating-income subtotal; negative increments indicate error shrinks at that stage.Source: G ADDITIONAL RESULTS · pp. 22–23

Evidence coding for Table 2

  • For far instances, evidence coding resets each driver to its year-ago value and identifies the reset that changes recomputed EPS most.A run counts as citing a specific number when the evidence line for that driver includes a dollar amount, percentage, thousands-separated figure, or number with a unit.Source: G ADDITIONAL RESULTS · p. 23

H CASE STUDY: BOEING’S DIVESTITURE

The Boeing case study traces forecast differences to a one-time divestiture gain and other drivers. The appendix also gives a generated schema example and the prompts and operating rules used to elicit driver estimates and intervals.

H CASE STUDY: BOEING’S DIVESTITURE

  • Boeing’s $10.55B sale price was not itself earnings; the forecast depended on the gain recognized in the quarter, which entered operating income.The leading models estimated an $8.0–9.6B gain, while Qwen 3.5 and GLM 4.5 Air entered gains close to zero; the gain accounts for $10–12 of their roughly $17 EPS gap from the leading models.Source: H CASE STUDY: BOEING’S DIVESTITURE · p. 23
  • Boeing reported GAAP diluted EPS of $10.23 against consensus of −$0.31, while median model forecasts ranged from −$7.57 to $10.48.Opus 4.6, GPT-5.5, and GPT-5.4 forecast medians were $8.78, $9.16, and $10.48, respectively.Source: H CASE STUDY: BOEING’S DIVESTITURE · p. 23

I COMPLETE EXAMPLE OF A GENERATED SCHEMA

  • The example uses year-ago values as its baseline and keeps reported validation EPS inaccessible to the agent.The schema uses absolute income taxes with the year-ago expense as default and includes last-quarter, trailing-twelve-month, and Q4 share-count references.Source: I COMPLETE EXAMPLE OF A GENERATED SCHEMA · p. 24
  • The Microsoft schema example represents the income statement as linked formulas from operating income through EPS and gives the agent a prediction schema of driver fields.The example includes revenue growth, cost-of-revenue margin change, expense growth, other expense, and tax fields.Source: I COMPLETE EXAMPLE OF A GENERATED SCHEMA · pp. 24–25

J PROMPTS

  • The prompts define a fixed-information workflow in which reduced-source conditions change tool availability and prompt text while releasing prompts for every condition.The full environment provides filings, transcripts, and news tools; source-removal conditions disable the corresponding tools.Source: J PROMPTS · pp. 26–27
  • Agents must support each required driver estimate with field-specific evidence, call the calculator with those estimates, and return its computed results in structured JSON.The prompt prohibits back-solving driver values from aggregate revenue or EPS and requires explicit fallback reasoning when segment-specific evidence is absent.Source: J PROMPTS · pp. 27–28
  • The Microsoft prompt requires forecasting the whole income statement, including below-pretax tax and diluted share count, using recent quarters to judge trends and seasonality.YoY values use the same fiscal quarter one year earlier as the Q-4 baseline.Source: J PROMPTS · p. 28
  • The required 80% interval is reported as p10 and p90 around the calculator’s unchanged point forecast, and scoring accounts for both coverage and width.The prompt specifies a 10% chance of outcomes below p10 and above p90.Source: J PROMPTS · p. 29
Loading 2610.38293v1…