Source-linked AI summary
FinSkillBench: Evaluating AI Agents and Domain Skills for Investment Management
Jermyn Zhen Yong Bek, Zhuang Qiang Bok, Zhongtian Sun
TL;DR
Existing finance evaluations often test financial knowledge or question answering rather than whether agents can execute auditable, tool-using investment workflows. FinSkillBench evaluates this process across investment-management tasks and finds that curated procedural skills substantially improve performance, whereas self-generated skills provide little benefit.
Problem
Existing evaluations often test financial knowledge or finance questions, leaving limited evidence about agents’ complete, auditable investment workflows using tools, documentation, and reusable skills.
Method
FinSkillBench evaluates agents across 12 subtasks in portfolio construction, risk management, and fundamental analysis using point-in-time data, factorial skill conditions, and task-specific verifiers.
Results
+0.162 aggregate improvement over no-skill baselines for curated skills, while self-generated skills show no aggregate accuracy improvement in the one-episode setting.
Takeaways & Limitations
Reliable, validated procedural skills improve investment-management agent performance, while writing procedures within an episode is insufficient for comparable accuracy.
Takeaways & Limitations
Single-sample evaluation cells leave agent-loop variance unmeasured, particularly in episodes that exhaust the turn budget.
Abstract
from arXiv · showhide
Investment management is a high-stakes domain in which agentic AI systems must do more than generate plausible text. They must retrieve point-in-time data, assemble correct computational inputs, invoke specialized methods, and produce auditable structured outputs. We introduce FinSkillBench, an evaluation suite designed to measure whether language model agents can effectively use financial domain skills to solve investment management tasks. The benchmark spans three domains, portfolio construction, risk management, and fundamental analysis, and includes 12 subtasks with 2,603 task episodes. Each episode provides point-in-time inputs, hidden ground truth, and a task-specific verifier.We compare three conditions: no skill, curated skill packages consisting of procedural documents and executable components, and self-generated skills in which the agent writes and reuses its own procedures within an episode. Across 9 models and a large-scale evaluation, curated skills consistently improve performance, raising mean scores from 0.366 to 0.528, with the largest gains in portfolio construction and risk management. In contrast, self-generated skills provide little benefit despite higher computational cost. An independent evaluation using a separate agent framework (Hermes Agent, 8 models, 5,280 episodes total) reproduces the directional pattern across all three domains, with the magnitude of skill effects varying by subtask and harness. These results showthat in investment management agents, access to reliable procedural skills can be as important as model choice, while naive self-generation of skills is often ineffective. We release the benchmark, evaluation tools, curated skill packages, and full trajectories to support further research.
1 Introduction
FinSkillBench evaluates investment-management agents as end-to-end, auditable workflows rather than systems that merely answer finance questions. It tests whether reusable domain skills improve performance across three domains using controlled conditions, point-in-time tasks, and automated verification.
- Motivation: Investment-management agents must retrieve and process quantitative inputs, use reliable tools, produce structured outputs, and support verifiable results.Tasks include portfolio construction, mandate monitoring, stress-loss estimation, earnings-quality metrics, and XBRL normalization.
- Motivation: Existing finance evaluations are insufficient because agentic workflows also require procedural loading, numeric-input integrity, formula selection, and auditable outputs.The evaluation target is the full process, not only the underlying model.
- Benchmark: FinSkillBench covers portfolio construction, risk management, and fundamental analysis through twelve investment-management subtasks.The domains include optimization and allocation, mandate and stress workflows, and financial-statement metric analysis.
- Experimental design: 2,603 task episodes form a point-in-time benchmark with fixed tasks, tools, turn budgets, and scoring across no-skill, curated, and self-generated conditions.The factorial protocol isolates the effect of reusable skill access.
- Evaluation: Task-specific automated verifiers assess outputs including numeric accuracy, constraint satisfaction, ranked risk recovery, attribution accuracy, trade matching, and financial-statement metric extraction.These verifiers support auditable evaluation of the agentic workflow.
2 Related Work
Prior work establishes that curated procedural skills improve agent performance, while financial and tool-use benchmarks mainly evaluate narrower components. FinSkillBench extends these directions to agentic investment-management workflows combining retrieval, procedures, executable tools, and structured outputs.
- Agent skills: Curated skills improved SkillsBench pass rates by approximately +16 pp on average, while self-generated skills produced no average benefit.FinSkillBench extends this analysis to investment management.
- Financial NLP benchmarks: FinQA, ConvFinQA, and TAT-QA primarily assess numerical reasoning over given financial-report evidence, whereas FinSkillBench evaluates broader agentic workflows.The workflows combine point-in-time data retrieval, domain procedures, executable tools, and structured submission.
- Tool-use and agent benchmarks: AgentBench and ToolBench study multi-step tool interaction across broad domains, while FinSkillBench narrows the domain and deepens task contracts around investment-management methods.Ground-truth computations include mean-variance optimization, Black–Litterman allocation, factor exposure analysis, Piotroski F-Score, and Beneish M-Score.
3 Benchmark Design
FinSkillBench’s benchmark design uses point-in-time, tool-accessible financial data and hidden offline ground truth to evaluate 12 subtasks across portfolio construction, risk management, and fundamental analysis. Episodes pair structured inputs with task-specific verification under exact, semi-structured, or accounting-derived grading regimes.
- Episode structure: 12 subtasks are organized into portfolio construction, risk management, and fundamental analysis episodes with unique task metadata, visible inputs, hidden expected outputs, and verification objects.Portfolio-construction and risk-management subtasks expose 40 tasks each in the full run, while fundamental-analysis manifests are larger and filtered.
- Data substrates: Three data substrates support the benchmark: a market stack for portfolio and risk tasks, and SEC EDGAR submissions plus XBRL CompanyFacts for fundamental analysis.Derived inputs include shrinkage covariance, rolling factor exposures, expected-return views, and a market-cap benchmark proxy.
- Point-in-time evaluation: Explicit as_of_date controls every episode, with prompts and tools restricted to point-in-time slices to prevent use of post-date information.This design tests whether agents retrieve and pipe accurate temporal inputs rather than relying on unavailable future data.
- Point-in-time evaluation: Lossy prompt compaction hides full numeric precision from visible text while tool calls provide complete structures, testing accurate retrieval and numerical data flow.Covariance matrices are summarized in prompts but remain available at full precision through tools.
- Ground truth and grading: Offline hidden ground truth spans exact numerical, semi-structured ranked, and accounting-derived labels, with each subtask paired to a matching verifier.Examples include solver outputs, Black–Litterman posteriors, stress P&L, risk identification, canonical accounting items, and segment-level bridges.
4 Skills as a Scientific Manipulation
The study manipulates agents’ access to procedural resources while holding tasks, tools, turn budget, temperature, and scoring fixed. Its curated packages combine procedural documentation with validated executable components, within a fixed function-calling evaluation loop.
- Resource-access manipulation: The central manipulation varies procedural-resource access while keeping tasks, tools, turn budget, temperature, and scoring fixed.The design is factorial over resource content rather than interface availability, with invariant tool signatures across conditions.
- Resource-access manipulation: The no-skill condition withholds both the skill corpus and finance-domain tools, serving as the main-results reference baseline.The cross-harness validation uses the same variant, withholding both the SKILL.md package and domain tools.
- Curated skill packages: Each domain’s curated package combines Markdown procedures, formulas, failure modes, API patterns, and validated Python entry points with JSON I/O.The declarative–imperative split addresses models’ difficulty passing large numeric structures through tool calls at full precision.
- Evaluation harness: The function-calling ReAct loop provides skill discovery, loading, authoring, task-field retrieval, scoped script execution, domain queries, and final JSON submission.The turn budget is capped at 12, with up to 3 retries for transient provider failures.
5 Experimental Setup
The experiment evaluates nine models across three conditions and 12 subtasks, totaling 17,820 episodes under fixed execution settings. Validity was high overall, but Phi-4 frequently exhausted the turn budget and was generally excluded from condition aggregates.
- Models: Nine models spanning frontier and open-weight families were evaluated.The models were gpt-5.4, claude-sonnet-4.6, gpt-4.1, gemini-2.5-pro, grok-4, DeepSeek-V3.2, Phi-4, glm-5.1, and gemini-3.1-flash-lite.
- Run: 17,820 episodes covered 9 models × 3 conditions × 12 subtasks.Tasks per cell were 40 for portfolio construction and risk management subtasks and 100 for fundamental analysis subtasks, with one sample per cell.
- Validity: 96.4% of 17,820 evaluations returned valid JSON.There were 644 invalid submissions, 3,736 turn-budget exhaustions, and 75 incomplete submissions; Phi-4 contributed 1,980 rows with mean score 0.000, dominated by turn-budget exhaustion, so most condition analyses excluded it.
6 Results
Curated skills substantially improve investment-management agent performance, especially on computationally demanding subtasks, while self-generated skills offer little accuracy benefit despite their cost. Skill effects vary across subtasks and models rather than increasing monotonically with base-model strength.
- Aggregate results: The pooled mean score is 0.375 across 17,820 evaluations, rising to 0.422 after excluding Phi-4; curated skills improve results substantially, while self-generated skills barely help.The per-condition means show a large, stable curated-skill improvement over no-skill, but little movement from self-generated skills.
- Where skills help: Curated-skill gains are largest when skills provide executable computational primitives.This pattern is reported in Table 3.
- Subtask variation: Subtasks with near-zero or modest no-skill baselines show the largest improvements, whereas subtasks with high baselines show small or negligible gains.Normalization receives no curated benefit because it relies mainly on direct XBRL extraction and simple arithmetic already supported by the global XBRL query tool.
- Model variation: Skill effects vary across base models: gpt-4.1 gains +0.405, gpt-5.4 gains +0.091, and claude-sonnet-4.6 starts at 0.534 and gains +0.125.The results indicate an interaction among model capability, tool-calling behavior, and skill-following reliability rather than a monotone relationship with base-model strength.
- Self-generated skills: Self-generated skills load in 97.7% of non-Phi-4 episodes but rarely improve accuracy, making them expensive in context and turns without delivering gains.Writing a procedure within an episode is not equivalent to using a validated domain skill with tested scripts and precise data-access guidance.
7 Cross-Harness Validation
An independent Hermes Agent replication reproduces the positive curated-skill effect across aggregate, domain, and model groups. Skill gains are largest in portfolio construction and risk management, while cross-harness magnitudes vary by subtask and baseline distribution.
- Replication design: The Hermes replication used the same domain skills, tools, prompt template, submission protocol, and turn budget while disabling other built-in capabilities.Eight models were evaluated with only Hermes skills_list and skill_view activated.
- Domain results: +0.4 non-overlapping CIs: portfolio construction and risk management showed substantial, statistically significant gains, while fundamental analysis improved less but significantly.The same positive ordering held across all three domains, with fundamental analysis showing the lowest improvement.
- Model results: +0.49: claude-sonnet-4.6 showed the largest Hermes improvement with skills, while GLM-5.1 scored 0.779 with curated skills.Open-weight models also improved by 38.3 pp, 31.5 pp, and 26.7 pp for GLM-5.1, DeepSeek-V3.2, and gemma-4-31b-it.
- Cross-harness interpretation: Cross-harness skill deltas varied by task: risk identification improved +0.309 versus +0.414, whereas constrained optimization improved +0.358 versus +0.087.These results show that reported skill uplift depends on each harness’s no-skill baseline distribution, tools, prompts, and turn budget.
8 Discussion
The discussion emphasizes that curated skills combine procedural, tool-use, executable, and failure-warning support rather than pure knowledge. It also cautions that strict self-generation is usually ineffective in one-shot settings and that evaluation design can materially shape conclusions about financial-agent capability.
- Curated skill gains: Curated skills provide procedural reminders, API and tool-use guidance, executable financial algorithms, and warnings about common failure modes.Their gains therefore reflect more than knowledge acquisition, including implementation and reliability support.
- Self-generated skills: Self-generated skills require inference, writing, reloading, and task completion within 12 turns, creating overhead and unvalidated instructions.The empirical result is that they are usually insufficient for one-shot improvement, though persistent multi-episode learning could allow testing and correction.
- Evaluation design: Evaluation design strongly affects conclusions about financial-agent capability, including score inflation, confounded skill effects, and potentially misleading constraint-status metrics.Dropping invalid or max-turn submissions inflates reliability estimates, shared deterministic solvers confound measured skill effects, and label-only constrained-optimization scoring may reward portfolios far from optimizer solutions.
9 Limitations
The benchmark’s conclusions are limited by single-sample evaluation, incomplete coverage of markets and data, dependence on engine-derived labels and executable skills, provider instability, and non-equivalence across harnesses. These constraints leave agent-loop variance and broader reproducibility only partially assessed.
- Evaluation design: Single-sample cells and temperature-0 evaluation reduce within-cell variance but leave agent-loop variance unmeasured, especially in 21% of turn-budget-exhausted calls.Paired comparisons identify the population-level treatment effect under deterministic verifiers, but repeated runs would tighten within-cell estimates.
- Scope: The 30-ticker S&P 500 universe does not represent international markets, other asset classes, derivatives, private assets, or illiquid securities.The restricted universe supports controlled evaluation but limits generalization across investment settings.
- Ground truth: Engine-derived labels measure agreement with deterministic solvers and routines, while reasonable modeling alternatives could produce different labels.Examples include convex solvers, Black–Litterman routines, and alternative shrinkage intensities.
- Data coverage: Incomplete segment-level data in the processed XBRL panel caps attainable driver-decomposition performance for agents that avoid unsupported domain guesses.This limitation affects expected driver labels rather than necessarily reflecting agent capability.
- Skill assumptions: Because curated skills include executable Python scripts, the evaluation measures discovering and using domain skills rather than unaided financial reasoning.This supports studying practical agents but complicates comparisons with text-only benchmarks.
- Reproducibility: Changing providers, model versions, and API routing constrain external reproducibility, while the Hermes replication demonstrates directional transfer without establishing harness equivalence.Phi-4 and grok-4 showed tool-calling failures on this run that may not generalize.
10 Responsible Use and Ethics
FinSkillBench evaluates analytical correctness in controlled investment-management tasks, not investment advice or autonomous financial decision-making. Its safeguards address harms from erroneous calculations, hallucinated evidence, stale data, hidden assumptions, and inappropriate objectives.
- Scope and limitations: The benchmark is designed for analytical correctness in controlled investment-management tasks, rather than investment advice or autonomous trading.It should not justify suitability determinations or client-specific recommendations.
- Risks: Financial agents may cause harm through erroneous calculations, hallucinated evidence, stale data, hidden assumptions, or inappropriate optimization objectives.
- Safeguards: Point-in-time discipline, structured outputs, and explicit scoring are intended to support safer evaluation of financial agents.
11 Conclusion · A Data Processing and Ground Truth Derivation · A.1 Universe and Shared Infrastructure
FinSkillBench evaluates investment-management agents as systems combining language models, tools, task data, and reusable procedural knowledge. Across full-run episodes, curated skills improve performance substantially over no-skill baselines, whereas self-generated skills add overhead without improving aggregate accuracy.
- 11 Conclusion: FinSkillBench evaluates agents as systems combining language models, tools, task data, and reusable procedural knowledge.
- 11 Conclusion: +0.162 improvement over no-skill baselines was statistically stable across 17,820 full-run episodes and 9 models.The confidence interval was [+0.152, +0.171].
- 11 Conclusion: +0.005 aggregate accuracy change from self-generated skills was not an improvement in the one-episode setting.The confidence interval was [−0.002, +0.011], while token and turn overhead increased.
- A.1 Universe and Shared Infrastructure: All three domains use a common 30-ticker universe drawn from the S&P 500 and stratified across all 11 GICS sectors.
- A.1 Universe and Shared Infrastructure: The universe is frozen at a single snapshot date to avoid survivorship bias.
- A.1 Universe and Shared Infrastructure: The shared security records include symbol, CIK, ISIN, CUSIP, sector, subsector, and exchange.
- A Data Processing and Ground Truth Derivation: Each data pipeline follows the architecture Download → Process → Construct → Ground Truth → Tasks → Validate.
- A.1 Universe and Shared Infrastructure: A single run_pipeline.py loads the security master, creates shared context, resolves phases, and executes them in dependency order.
A.2 Portfolio Construction Data … C.3 Fundamental Analysis
FinSkillBench builds deterministic, point-in-time ground truth and structured tasks across portfolio construction, risk management, and fundamental analysis, with examples spanning optimization, monitoring, stress testing, remediation, and financial-statement analysis. Its prompts and conditions control data access, tool use, skill availability, and structured JSON submission.
- A.2 Portfolio Construction Data: Portfolio construction combines daily prices, dividend-adjusted prices, market capitalization, six-factor returns, covariance estimates, expected-return views, and tiered constraint sets.The data cover 30 symbols across 2024-01 through 2025-12; constraints progress from easy long-only limits to hard factor-neutral portfolios.
- C.1 Portfolio Construction: 144 portfolio-optimization ground-truth files cover 12 dates, 3 difficulties, and 4 objectives, with 96 optimal and 48 infeasible cases.Ground truth uses convex quadratic programming; all infeasible cases occur at hard difficulty because factor neutrality and tight constraints cannot be satisfied with 30 symbols.
- A.3 Risk Management Data: Risk-management data construct portfolios, mandates, and historical or hypothetical stress scenarios with deliberately planted risk characteristics.The reference risk engine deterministically derives risk identification, constraint monitoring, stress-testing, and minimum-turnover remediation ground truth.
- C.3 Fundamental Analysis: Fundamental-analysis data integrate filing metadata, XBRL CompanyFacts, and pre-parsed statements into normalized financial metrics, earnings-quality scores, and quarter-to-quarter driver decompositions.Normalization extracts 10 canonical metrics, while earnings quality uses Piotroski and Beneish measures and driver decomposition attributes revenue and margin changes.
- C.3 Fundamental Analysis: 2,243 fundamental-analysis tasks map one-to-one to ground-truth entries, with raw-source derivation and 75.4% exact cross-source agreement.The task families are normalization, earnings quality, and driver decomposition.
- A.5 Abbreviated Input/Output Examples by Subtask: Abbreviated examples specify structured inputs and outputs for portfolio optimization, rebalancing, Black–Litterman, risk analysis, stress testing, remediation, normalization, earnings quality, and driver decomposition.Outputs include weights, constraint maps, optimizer-call specifications, trade lists, posterior returns, ranked risks, P&L attribution, compliance flags, and canonical financial metrics.
- B System Prompts and Task Prompts: Prompts enforce point-in-time data use, tool-mediated computation, and final JSON matching the expected schema, while conditions vary skill access and executable domain tools.The no-skill condition lacks domain-specific tools, curated skills expose validated scripts, and self-generated skills require saving and reloading procedures before solving.
C.3.1 Normalization (100 tasks evaluated; 244 total). … C.4 Summary of Cognitive Demands
The financial-analysis subtasks require agents to retrieve point-in-time XBRL data, compute standardized metrics or analytical indicators, and produce scorer-compatible structured outputs. Difficulty varies from relatively straightforward normalization to challenging driver decomposition, while the summary formalizes cognitive demands and scorer specifications.
- C.3.1 Normalization (100 tasks evaluated; 244 total).: Normalization takes a ticker, reporting period end date, form type, and query_xbrl access, returning 10 canonical financial metrics.The metrics include revenue, operating_income, net_income, eps_diluted, total_assets, total_liabilities, stockholders_equity, operating_cash_flow, operating_margin, and ebitda.
- C.3.1 Normalization (100 tasks evaluated; 244 total).: 0.681 was the no-skill baseline for normalization, while the curated skill changed performance by Δ = −0.003.The task requires XBRL field mapping and derived-metric computation, including operating_margin = operating_income / revenue and ebitda = operating_income + depreciation_amortization.
- C.4 Summary of Cognitive Demands: The subtasks test financial statement literacy, XBRL navigation, multi-period comparison, ratio computation, segment analysis, revenue attribution, and domain knowledge of business structures.These capabilities are summarized as primary and secondary cognitive and procedural demands across subtasks.
- C.3.2 Earnings Quality (100 tasks evaluated; 1,002 total).: Earnings Quality compares current and prior comparable periods to compute nine Piotroski components, eight Beneish ratios, accruals, income quality, and concern flags.Its output includes a Piotroski F-Score, Beneish M-Score, beneish_flag, and flags array; scoring uses Piotroski accuracy, Beneish flag match, and flag F1.
- C.3.3 Driver Decomposition (100 tasks evaluated; 997 total).: Driver Decomposition outputs revenue_delta, up to five named revenue drivers, and margin drivers with directions, magnitudes, contribution percentages, or percentage-point deltas.The scorer averages driver recall and precision, direction and numeric accuracy when enabled, margin-driver performance, and revenue-delta accuracy.
- C.3.3 Driver Decomposition (100 tasks evaluated; 997 total).: Driver Decomposition is the most challenging fundamental-analysis subtask because XBRL provides aggregate rather than segment-level data for attributing revenue changes.Agents must use company business-segment knowledge, while the curated driver_decomposition.py script can process segment data when available.
- C.4 Summary of Cognitive Demands: Table 10 specifies scorers mathematically, with most component scores clipped or averaged into [0, 1].The exp05 ranked-list scorer does not final-clip NDCG or the composite and can slightly exceed 1 when duplicate predicted risk types receive repeated relevance credit.