PaperlayerView exact revision on arXiv

Source-linked AI summary

Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?

Huangfei Jiang, Anning Hu, Kai Sheng, Yuchen Cai, Rui Ge, Xiangyu Kong, Jianghao Lin, Shuzi Chen, Jiyi Weng, Junchi Yan

arXiv:2610.38342v1cs.AI

TL;DR

Existing benchmarks cover parts of financial research, but full-process evaluation needs decision-time evidence and professional criteria. ILE combines a versioned database, analyst-deliverable tasks, and expert-linked financial dependencies. Astra leads overall at 68.1 yet scores 52.5 on Forecast & Model, illustrating that strong research performance does not consistently translate into accurate forecasts.

  • Problem

    Existing benchmarks assess isolated financial-research capabilities, while full-process evaluation requires connecting evidence, assumptions, and conclusions under a strict information cutoff.

    Source: §1 · p. 2

  • Method

    ILE combines a versioned financial database, a professional task taxonomy, and expert-led criteria linked to sources, calculations, and assumptions through financial dependency graphs.

    Source: §1 · pp. 2–3, §2.1 · p. 4

  • Results

    68.1 overall leads for GPT-6 Astra, but its 52.5 on Forecast & Model shows that strong overall research performance does not consistently translate into accurate forecasts.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    Evaluating the evidence–assumption–judgment chain helps locate where analytical strengths fail to translate into forecasts and investment judgments.

    Source: §1 · p. 3

    Historical outcomes may be memorized during pretraining, and market noise can make sound intermediate reasoning diverge from realized outcomes.

    Source: §5 · p. 11

Abstract

from arXiv · show

Financial agents must form forecasts and investment judgments from decision-time evidence, with predictions evaluated against subsequent outcomes. Existing benchmarks assess financial knowledge, research workflows, or forecasting, but evaluating the full research process requires a strictly frozen information set and professional evaluation criteria. We introduce Investor's Last Exam (ILE), an equity-research benchmark led by experienced buy-side investors and fund managers. Its three components are a versioned financial database spanning 3.43 million document versions and nine source categories, supplemented by structured market data, with validated timestamps governing retrieval and computation; a task taxonomy organized around professional analyst deliverables; and an expert-led framework linking scoring criteria to sources, calculations, and assumptions through financial dependency graphs. Evaluation spans 86 assignments and jointly assesses research quality, forecast accuracy, and simulated investment performance. GPT-6 Astra and Claude Opus 5.5 lead overall at 68.1 and 66.9 out of 100, yet even Astra scores only 52.5 on Forecast & Model. Our results show that even state-of-the-art models fail to translate overall research capability into predictive performance. ILE shifts the evaluation unit from isolated capabilities to the evidence--assumption--judgment chain, allowing researchers to locate where analytical advantages fail to translate into forecasts and investment judgments, and providing a traceable, diagnostic foundation for training and evaluating financial agents.

1 Introduction

Professional equity research requires a coherent chain from evidence through forecasts to investment judgments, which isolated capability tests do not capture. ILE benchmarks that chain under a decision-time information boundary and finds that strong overall research scores can coexist with weaker forecasting.

1 Introduction

  • Existing benchmarks cover isolated workflow stages, while publication-date filtering can admit post-cutoff content and does not control provenance across retrieval, computation, and scoring.The missing evaluation unit connects evidence, assumptions, and conclusions under a strict cutoff.Source: §1 · p. 2
  • ILE combines a versioned database of 3.43 million document versions, a professional task taxonomy, and expert-linked scoring criteria grounded in sources, calculations, and assumptions.Its criteria assess both analysis and designated outcomes, making reasoning paths reviewable and traceable.Source: §1 · pp. 2–3
  • 68.1 overall leads for GPT-6 Astra, followed by Claude Opus 5.5 at 66.9, but Astra scores 52.5 on Forecast & Model.Astra scores 78.9 on Research & Evidence and 77.6 on Evaluate & Decide, while forecast-outcome and simulated-return rankings differ from overall research rankings.Source: §1 · p. 3
  • The benchmark evaluates professional judgments across the analyst workflow rather than testing isolated capabilities.Its time-frozen environment restricts source-version availability across retrieval and computation, while its multidimensional framework links judgments to supporting analyses and realized outcomes.Source: §1 · p. 3

2 Benchmark Design

ILE constrains the evidence available at decision time and organizes evaluation around professional deliverables and their financial dependencies. Experts construct verifiable tasks and criteria, with AI assisting investigation and review.

2 Benchmark Design

  • ILE’s design constrains both the evidence available at decision time and the financial requirements an answer must satisfy.The benchmark uses professional research deliverables as task carriers and versioned evidence as the information boundary.Source: §2 · p. 4

2.1 Financial Research Environment

  • Version-level availability addresses a weakness of date-filtered search: a dated query can still retrieve later content, as in the Tesla O74 example.The corpus retains historical versions and their availability times, while shared tool checks enforce each task’s cutoff across searches, reads, downloads, and structured data.Source: §2.1 · pp. 4–5

2.2 Professional Task Taxonomy

  • The taxonomy classifies work by analyst deliverable, with five categories and twenty subtypes, plus Integrated Research and Market Trading.Tasks test interdependent measurements, assumptions, and decisions; for example, higher operating cash flow alone does not establish lower financing needs.Source: §2.2 · p. 5

2.3 Expert-Led Task Construction

  • Experts define task elements and build dependency graphs linking source-grounded facts, assumptions, calculations, and investment judgments; AI assists investigation, but experts decide which suggestions to adopt.The refinancing example makes drawable credit under a covenant a decisive dependency.Source: §2.3 · pp. 5–6
  • Figure 4 pairs a dated-query example that retrieves post-cutoff Tesla deliveries with an AEP case where higher operating inflow accompanies a wider pre-financing shortfall.Its AEP badges count systems earning full credit on every criterion in the indicated rubric group.Source: Figure 4 · p. 6
  • Every scoring criterion links to evidence, reproducible calculations, or explicit judgments, and expert review revises ambiguous requirements exposed by trial answers.In the AWS O15 example, review changes uncertainty credit to depend on both informativeness and predictive accuracy.Source: §2.3 · pp. 6–7

3 Evaluation and Results

ILE evaluates financial agents under a common scoring protocol across research, forecasting, investment outcomes, and information-boundary compliance. The results show that overall research scores, predictive performance, and simulated returns can rank systems differently.

3.1 Protocol and scoring

  • The protocol scores weighted criteria on a common 0–100 scale, using deterministic checks for numerical criteria and GPT-6 Astra for semantic criteria.Strict evaluation requires successful submission; the main results use answer-based evaluation, which scores the final available answer and gives zero credit when none is available.Source: §3.1 · pp. 7–8

3.2 Performance across financial tasks

  • 68.1 overall for Astra and 66.9 for Opus contrast with DeepSeek V4.1 Flash’s predictive-performance lead at 40.3, ahead of Astra at 37.7 and Opus at 37.6.Astra scores 52.5 on Forecast & Model, while Opus leads that category at 52.6; research quality and predictive performance capture different strengths.Source: §3.2 · p. 8
  • DeepSeek V4.1 Flash averages 113.8 tool calls per task, while Opus gains 3.9 points over Flash at USD 7.96 per task versus USD 0.36.Effective performance requires coordinating evidence retrieval, computation, and financial reasoning rather than extensive tool use alone.Source: §3.2 · p. 8, Figure 5 · p. 9

3.3 From fundamental forecasts to stock returns

  • Hunyuan leads simulated returns at 8.50%, followed by Flash at 7.98% and Astra at 7.18%, versus 1.18% for always long.Kimi returns 7.15% versus Opus’s 6.39%, despite Kimi’s lower research score; the return sample covers twenty assignments, eleven companies, and thirteen disclosures.Source: §3.3 · p. 9, Table 4 · p. 9

3.4 Public-web research and database coverage

  • 57% of Astra’s public-web trajectories contain post-cutoff sources, so comparable scores do not establish compliance with the information boundary.Astra scores 68.1 on the database and 68.3 on public-web Codex, but only the evidence source itself guarantees that sources precede the cutoff.Source: §3.4 · p. 9

4 Where Financial Research Breaks Down

Case analyses locate breakdowns in financial dependencies, forecast assumptions, model consistency, task completion, and rubric specificity. These failures show why recovering inputs or producing calculations alone does not establish a sound financial judgment.

4.1 A complete case: from cash flows to a funding judgment

  • All eleven systems recover eight reported inputs and reconcile both cash-flow identities, but only five compute both funding shortfalls and five reach the supported funding conclusion.Operating cash flow rises from USD 2,671 million to 3,421 million while investing cash use rises from 5,347 million to 6,602 million, widening the pre-financing shortfall by 505 million.Source: §4.1 · p. 10

4.2 Calculation, forecasting, and joint consistency

  • Three systems earn every supporting-analysis point on Tesla O80 but no forecast-outcome credit because deliveries, revenue, and margin miss realized targets.Arithmetic propagates selected assumptions but does not establish their predictive validity.Source: §4.2 · p. 10
  • Home BancShares forecasts must describe a single accounting state, while GE Aerospace’s position limit must follow from a supported equity-downside assumption.Qwen Max earns full formula credit on the latter task but does not explain its protection boundary.Source: §4.2 · p. 10

4.3 Submission Reliability and Task Completion

  • Hunyuan stops without submitting on 17 trajectories, while Opus has three rejected submissions on each of five, preventing useful analysis from guaranteeing task completion.Table 5 reports score drops under strict submission requirements; the passage identifies the failures as a limitation of agent performance.Source: §4.3 · p. 10, Table 5 · p. 11

4.4 What expert refinement improves

  • Expert refinement makes errors attributable by breaking general requirements into distinguishable criteria linked to evidence, calculations, or realized outcomes.Examples include separating additive from overlapping Microsoft revenue segments, fixing a dated JPMorgan prior, and jointly scoring AWS interval accuracy and uncertainty.Source: §4.4 · p. 11

5 Limitations

ILE limits retrieval look-ahead through versioned evidence, but pretrained models may already encode historical outcomes. The paper also notes that realized outcomes can diverge from sound intermediate reasoning because of exogenous market noise.

Limitations

  • Versioned retrieval cannot rule out frontier models’ implicit memorization of historical outcomes during pre-training, which requires counterfactual or cold-start experiments to disentangle.Outcome-only evaluation may also underestimate research quality when exogenous market noise moves realized values away from otherwise sound analyses.Source: §5 · p. 11
  • Future work proposes synthetic counterfactual scenarios, composable micro-task graphs, difficulty-weighted scoring, and improved separation of reasoning failures from stochastic forecast fluctuations.The proposed scenario engine would perturb real statements and create virtual peers to isolate parametric memory.Source: §5 · p. 11

6 Conclusion

ILE combines versioned financial evidence, professional equity-research tasks, and expert-linked scoring to evaluate the evidence–assumption–judgment chain. Its conclusion is that strong research scores do not consistently translate into accurate forecasts or well-supported financial conclusions.

6 Conclusion

  • ILE’s 3.43 million document versions and financial dependency graphs support decision-time evidence boundaries and trace links among sources, calculations, and assumptions.The conclusion frames the evidence–assumption–judgment chain as the unit of evaluation and a diagnostic foundation for measuring and improving agents.Source: §6 · p. 11

A Related Work

  • ILE differs from related financial QA and workflow benchmarks by organizing tasks around financial deliverables and coupling research with task-specific information boundaries.The related work also contrasts ILE with forecasting and trading benchmarks by its combination of versioned data, professional deliverables, and dependency-level construction.Source: A Related Work · p. 15
  • ILE evaluates designated forecasts against realized outcomes alongside professional deliverables, while related work includes expert rubrics for occupations and investment-banking workflows.The cited comparison distinguishes ILE’s buy-side equity focus and version-level cutoffs from those broader professional-work benchmarks.Source: A Related Work · p. 15
  • ILE’s financial dependency graph links graded propositions to calculations, assumptions, and source versions, which experts can inspect and edit.This differs from provenance approaches that distinguish source locations from the input dependencies explaining an output.Source: A Related Work · p. 15

B Database Inventory

The database combines a large, versioned financial-document collection with structured market data. Each task’s cutoff and authorization limit govern which source versions are available across retrieval and computation.

B Database Inventory

  • 3,430,336 document versions across nine source categories occupy 1.52 TB, with structured market data provided separately.Revisions are counted separately; the collection spans US, mainland Chinese, Hong Kong, and other markets and multiple file formats.Source: B Database Inventory · p. 15, Table 6 · p. 16

B.1 Versioned data architecture

  • The shared catalogue links issuers, securities, financial records, and immutable source files or derived representations to reporting periods, publication times, and supporting time evidence.Source: §B.1 · p. 15

B.2 Cutoff enforcement

  • Each task releases only authorized versions whose availability time precedes the earlier of its specified cutoff and authorized access limit.Date-only records become eligible only after that date ends; default discovery selects the latest collected eligible version, while history mode exposes other eligible versions.Source: §B.2 · p. 16
  • Search, reads, downloads, and cached results enforce the same authorization and availability rules, and structured files spanning publication dates are filtered before release.Source: §B.2 · p. 16
  • Downloads and local computation share an isolated task workspace, while reference analyses, rubrics, and later outcomes remain in a separate evaluation environment.Forecasts are matched to realized values by metric, period, and unit; consensus estimates serve as research inputs.Source: §B.2 · p. 16

C Professional Task Taxonomy

The taxonomy organizes assignments around professional deliverables and judgments, while financial dependencies specify how evidence, measurements, assumptions, and decisions must cohere. Rubric dimensions assess answer requirements, and scores aggregate task-normalized credit across tasks.

C Professional Task Taxonomy

  • The taxonomy maps professional deliverables to judgments, with integrated research combining analytical functions and market trading adding outcome-based subtypes.It defines 20 professional subtypes across five categories, plus integrated research and three market-forecasting subtypes; each task is grouped by its principal objective even when techniques overlap.Source: Table 7 · p. 16, C Professional Task Taxonomy · pp. 16–17, Table 8 · p. 17

C.1 Financial dependencies across assignments

  • Tasks test whether measurements are comparable, forecast updates are justified by new evidence, and investment stances remain consistent with assumptions and reversal conditions.Examples include reconciling Tesla reporting definitions, enforcing a Home BancShares accounting identity, and supporting Apple, JPMorgan, or GE Aerospace decisions with explicit risks or downside assumptions.Source: §C.1 · pp. 17–18

D A Worked Financial Dependency

  • JPMorgan O26 models a peer disclosure’s effect on prior investment-banking fees and Markets revenue using disclosed growth rates and a chosen read-through coefficient.Updated totals add the affected-line increments while holding other lines fixed; verification separately checks the source and period, dated prior, coefficient choice, and reconciliation.Source: D A Worked Financial Dependency · p. 18

E.1 Dimension definitions

  • Rubric dimensions label individual criteria by the financial requirement assessed, whereas professional task categories identify an assignment’s principal deliverable.Evidence checks reporting basis and period, Calculation derives a value, Reconciliation checks linked identities, Financial logic tests interpretation, and Decision consistency tests whether the conclusion follows from the analysis.Source: §E.1 · p. 18, Table 9 · p. 19

E.2 Score aggregation

  • Overall scores normalize criterion credit within each task and weight tasks equally; task-category scores average complete task scores within each category.The benchmark includes 78 company-research assignments and eight market-forecasting assignments; a market task’s deterministic outcome loss maps to one 100-point Result criterion.Source: §E.2 · p. 18
  • Dimension means average task-normalized credit only across tasks containing that dimension, and are computed separately from the overall score.Result covers all 86 tasks; Evidence, Financial logic, Calculation, Reconciliation, Decision consistency, and Uncertainty cover 66, 61, 59, 47, 62, and 43 tasks, respectively; an applicable task without a final answer contributes zero.Source: §E.2 · pp. 18–19
  • Strict evaluation requires an accepted submit_answer call, with omitted or rejected submissions receiving zero.The strict results use the same tasks, criteria, and aggregation as the main score table.Source: §E.2 · p. 19, Table 10 · p. 19

E.3 Full results with strict submission requirements

Strict evaluation makes a valid, accepted submission a condition of receiving credit. Omitted and rejected submissions count as zero in the strict scores.

E.3 Full results with strict submission requirements

  • Omitted or rejected submissions receive zero in each applicable strict mean, and Table 11 reports scores by dimension and task category.The strict evaluation requires an accepted submit_answer call and uses the same tasks, criteria, and aggregation as Table 3.Source: §E.3 · p. 20, Table 11 · p. 20

F.1 Matched-task comparison

The section describes matched-task revisions, interval scoring, rubric profiles, and realized forecast outcomes. The cases show that coherent supporting analysis can coexist with missed realized targets, while reporting scope can affect revised forecast scores.

F.1 Matched-task comparison

  • The matched-task comparison follows 40 structured assignments across four development stages using GPT-6 Astra and Qwen 3.8 27B as solvers and Astra as judge.The stages include validity repair, forecast-scoring revision, and question revision with new answers where public instructions change; unchanged assignments retain prior answers and scores.Source: §F.1 · pp. 20–21, Figure 6 · p. 20, Table 12 · p. 21

F.2 Interval scoring

  • Central 80% intervals are scored with an interval score using α = 0.2, then mapped to task-specific credit using a scale and interval budget.Microsoft and AWS examples allocate 70% to point accuracy, 15% to interval quality, and 15% to supporting analysis; other forecasts use their declared allocations.Source: §F.2 · p. 21

G Rubric Profiles and Financial Failure Cases

  • Figure 8 presents absolute rubric-dimension scores for the remaining three systems on a radial scale from 40 to 95.Its dimension scores use Table 3(a)’s aggregation; Figure 1(b) instead compares relative answer-quality strengths.Source: G Rubric Profiles and Financial… · p. 21, Figure 8 · p. 22

G.1 Realized forecast outcomes

  • Forecast-outcome credit is normalized by available outcome points and averaged over applicable tasks, separating predictive requirements from supporting analysis in full task scores.The combined outcome column pools point, interval, probability, and market criteria within each task before averaging.Source: §G.1 · p. 21, Table 13 · p. 22

G.2 Forecast accuracy and local consistency: Tesla

  • Every one of nine Tesla forecast errors exceeded its zero-credit limit, and Astra, Qwen Max, and Opus each earned zero of 50 outcome points despite receiving all 50 remaining points.The errors covered Q2 2026 deliveries, GAAP revenue, and automotive gross margin; answers included calculation bridges and coherent recommendations but missed realized targets.Source: §G.2 · p. 22, Figure 9 · p. 23

H.1 Reporting scope and forecast accuracy: Microsoft

  • Astra’s Microsoft O58 score changed from 73.4 to 71.0 and Qwen 27B’s from 78.1 to 75.2 after the question revision.The revision clarified that Cloud overlaps the three segments and that interval endpoints represent the 10th and 90th percentiles, requiring additive segment reconciliation while keeping Cloud separate.Source: §H.1 · p. 23

H.2 Joint interval quality: AWS

These cases show how forecast scoring, dated updates, cash-flow interpretation, and position sizing depend on explicit assumptions and traceable evidence.

H.2 Joint interval quality: AWS

  • 0 interval credit for Qwen’s revised central 80% interval contrasts with Astra’s 3.16 of 15 points for a covering interval.The joint rule rewards coverage while penalizing interval width.Source: §H.2 · p. 23

H.3 A dated prior and an attributable update: JPMorgan

  • About 10.26 points are earned by Astra’s updates from a March 31, 2026, 16:00 ET prior using corresponding peer-growth signals.The dated prior makes each subsequent change attributable to the peer signal.Source: §H.3 · p. 23

H.4 Cash-flow interpretation: American Electric Power

  • USD 3,181 million pre-financing shortfall in 2026 exceeded USD 2,676 million in 2025 even as operating cash rose; only five systems reached the supported funding conclusion.The deferred-fuel adjustment was a net USD 72 million and did not identify gross fuel collections.Source: §H.4 · p. 24

H.5 From contingent obligations to position sizing: GE Aerospace

  • GE’s disclosed credit-support amounts cover different populations and measurements, so they establish neither an aggregate loss ceiling nor a stock-return downside.The task separately assesses formula, disclosure scope, downside rationale, and the protection boundary.Source: §H.5 · p. 24

I.1 Interaction length and cost

  • 73.8 interaction rounds and 113.8 tool calls per task are DeepSeek V4.1 Flash’s average workload.Its tool use averages 22.2 searches, 41.6 document reads, 27.4 table operations, and 19.9 computation calls per task.Source: §I.1 · p. 24

I.2 Forecast-driven stock-return simulation

  • The simulation converts forecast-versus-reference signals into equal-weight long, short, or cash positions over eligible disclosure windows, with specified trading costs.Its retrospective sample contains twenty opportunities covering eleven companies and thirteen disclosures.Source: §I.2 · p. 25

I.3 Judge model comparison

  • OSS grading moves Flash from third to second, Opus to third, and Hunyuan above GLM, while Astra remains first.The fixed answers, rubrics, and eight objective market scores remain unchanged.Source: §I.3 · p. 25

I.4 Public-web research and database coverage

The database comparison contrasts cutoff-enforced retrieval with public-web research, while the accompanying sections specify shared tools, submission, and grading conditions.

I.4 Public-web research and database coverage

  • 62.1 versus 58.5 for Qwen Max and 60.4 versus 58.8 for Hunyuan under ILE versus public web; Astra scores 68.1 versus 68.3.The comparison uses the same 86 questions, rubrics, and Astra semantic judge.Source: Table 18 · p. 26, §I.4 · p. 26
  • 49 Astra trajectories (57%) receive sources dated after their cutoff, including all eight Market Trading assignments.Version-level availability checks exclude such sources from the research context.Source: §I.4 · p. 26

I.5 Shared tools on Finance Agent Benchmark v2

  • Shared tools are tested on public Finance Agent Benchmark v2 questions with DeepSeek V4 Flash as solver and rubric judge, retaining the benchmark’s loop, prompts, and criteria.Agents use document search, reading, downloading, and local computation tools; FAB v2’s price-history tool is retained.Source: §I.5 · p. 27

J.1 Shared system prompt

  • The shared prompt separates information cutoffs from reporting, valuation, and forecast dates, and distinguishes supported facts from assumptions and forecasts.It also requires material factual claims to be supported with source identifiers and precise locators where explanatory text is requested.Source: §J.1 · p. 28

J.2 Document retrieval and local computation

  • Six tool families cover search, reading, downloading, structured-table analysis, local computation, and answer submission, with access bounded by task authorization and cutoff.Table results retain the availability boundary, and a table_ref identifies a frozen result.Source: §J.2 · p. 28, Table 20 · p. 28, Table 21 · p. 29

J.3 Answer submission

  • Submission validates task-specific schemas for up to three attempts; accepted answers receive private-criteria grading, while strict evaluation requires acceptance.A third rejection ends the task.Source: §J.3 · p. 29, Table 22 · p. 29

Related papers

Loading 2610.38342v1…