Source-linked AI summary
StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?
Yanxu Chen, Zijun Yao, Yantao Liu, Amy Xin, Jin Ye, Jianing Yu, Lei Hou, Juanzi Li
TL;DR
Static financial question-answering benchmarks do not adequately evaluate LLM agents’ behavior in dynamic trading environments. STOCKBENCH addresses this gap with a contamination-free, multi-month benchmark using daily market signals and sequential decisions, finding that most agents struggle to outperform simple baselines.
Problem
Existing financial benchmarks largely use static question-answering and do not capture realistic trading dynamics.
Method
STOCKBENCH evaluates LLM agents through contamination-free, multi-month trading with daily prices, fundamentals, news, sequential actions, and financial performance metrics.
Results
Most LLM agents fail to outperform an equal-weight buy-and-hold baseline in cumulative and risk-adjusted return, despite strong financial QA performance.
Takeaways & Limitations
Static financial QA success does not necessarily translate into effective trading behavior, leaving substantial room for improved LLM trading agents.
Takeaways & Limitations
The four-month, daily-trading setup covers only large U.S. stocks and omits high-frequency trading, long-term cycles, transaction costs, slippage, and liquidity limits.
Abstract
from arXiv · showhide
Large language models (LLMs) demonstrate strong potential as autonomous agents, with promising capabilities in reasoning, tool use, and sequential decision-making. While prior benchmarks have evaluated LLM agents in various domains, the financial domain remains underexplored, despite its significant economic value and complex reasoning requirements. Most existing financial benchmarks focus on static question-answering, failing to capture the dynamics of real-market trading. To address this gap, we introduce STOCKBENCH, a contamination-free benchmark designed to evaluate LLM agents in realistic, multi-month stock trading environments. Agents receive daily market signals -- including prices, fundamentals, and news -- and make sequential buy, sell, or hold decisions. Performance is measured using financial metrics such as cumulative return, maximum drawdown, and the Sortino ratio, capturing both profitability and risk management. We evaluate a wide range of state-of-the-art proprietary and open-source LLMs. Surprisingly, most models struggle to outperform the simple buy-and-hold baseline, while some models demonstrate the potential to achieve higher returns and stronger risk management. These findings highlight both the challenges and opportunities of LLM-based trading agents, showing that strong performance on static financial question-answering do not necessarily translate into effective trading behavior. We release STOCKBENCH as an open-source benchmark to enable future research on LLM-driven financial agents.
1 Introduction
Existing financial benchmarks largely test static question-answering rather than realistic, sequential trading, motivating STOCKBENCH as a contamination-free benchmark for profitability and risk management. Its evaluation shows that strong static financial QA performance does not necessarily translate into effective trading behavior.
- Research gap: Most existing financial benchmarks focus on static question-answering and fail to capture realistic trading dynamics.This limitation motivates evaluation in environments with market interaction and continuous decisions.
- Motivation and contribution: STOCKBENCH evaluates LLM agents in realistic stock-trading environments using profitability and risk-management metrics.The benchmark is designed around realistic market conditions, sequential decisions, and contamination-free data.
- Empirical finding: Most LLM agents fail to outperform an equal-weight buy-and-hold baseline in cumulative and risk-adjusted return despite strong financial QA performance.The benchmark compares proprietary and open-weight models against a simple passive strategy.
- Evaluation design: The benchmark incorporates realistic market dynamics, diverse inputs, and multiple financial metrics for holistic agent evaluation.It uses daily prices, fundamentals, news, and metrics including cumulative return, maximum drawdown, and the Sortino ratio.
- Contribution: The implementation is released openly to support reproducibility and further research on LLM-based financial agents.The release is intended to facilitate community contributions.
2 STOCKBENCH
STOCKBENCH combines a back-trading environment with a minimal agent workflow that exposes LLMs to realistic, continuous stock-trading decisions. The design uses selected DJIA stocks, market and news data, staged analysis, and execution validation to prioritize realism, fairness, and reproducibility.
- Framework overview: STOCKBENCH consists of a back-trading environment and an associated workflow that evaluates backbone LLMs as trading agents.The framework is illustrated in Figure 1.
- Back-Trading Environment: Agents receive investment targets, historical prices and fundamentals, and recent news as inputs for trading decisions.Fundamentals include market capitalization, P/E ratio, dividend yield, and trading range; news covers the previous 48 hours.
- Back-Trading Environment: The benchmark selects 20 high-weight DJIA stocks across technology, finance, and manufacturing to stabilize and diversify evaluation.The fixed investment targets keep the action space tractable and reduce fluctuations from stock selection.
- Features of STOCKBENCH: The evaluation uses recent data and daily decisions over a multi-month horizon to support contamination-free, continuous trading assessment.The design prioritizes temporal separation, realism, fairness, and reproducibility.
- Agent Workflow: The workflow proceeds through portfolio overview, in-depth stock analysis, decision generation, and execution validation.Agents choose increase, decrease, or hold actions, which are converted into share quantities and checked against available liquidity.
3 Main Experiments
STOCKBENCH evaluates diverse LLM agents in a four-month, contamination-free trading environment using profitability and downside-risk metrics. Most tested models outperform the passive baseline, but performance does not consistently track reasoning specialization.
- 3.1 Experiment Setup: The benchmark evaluates models over 82 trading days with daily decisions across the top 20 DJIA stocks, using market information and recent trading context.Each model starts with $100,000 in cash and zero holdings; inputs include recent actions, news, and opening prices.
- 3.1 Experiment Setup: The study benchmarks open-weight and closed-source LLMs against an equal-weight buy-and-hold strategy that holds selected stocks unchanged through the evaluation period.The passive strategy serves as a reference point for comparing active trading performance.
- 3.1 Experiment Setup: Performance is assessed with final return, maximum drawdown, Sortino ratio, and a composite z-score ranking that balances profitability with downside exposure.Maximum drawdown measures the largest peak-to-trough decline, while the Sortino ratio focuses on downside volatility.
- 3.2 Experiment Results: Most tested LLM agents outperform the passive buy-and-hold baseline, which returns 0.4% with −15.2% maximum drawdown and a 0.0155 Sortino ratio.The results are reported across percentage return, maximum drawdown, Sortino ratio, and a composite ranking.
- 3.2 Experiment Results: Qwen3-235B-Ins achieves a lower maximum drawdown than Qwen3-235B-Think, −11.2% versus −14.9%, showing reasoning tuning does not guarantee better trading performance.The comparison separates strong general reasoning performance from effective decision-making in dynamic, noisy financial environments.
4 Analysis
STOCKBENCH analyses how trading performance changes with portfolio size, input information, model type, and market regime. Results show scalability and bearish-market adaptation remain difficult, while news and fundamentals generally support returns.
- Investment Target Size: All evaluated models exhibit performance degradation as the investment target expands, showing that scalability is inherently challenging.The study compares portfolios of 5, 10, 20, and 30 DJIA stocks and measures return variability across repeated runs.
- Influence of Error in the Trading Workflow: Thinking models make fewer arithmetic errors but more schema errors than instruct models during trading.The reported pattern is attributed to stronger reasoning ability for arithmetic and more complex outputs for schema violations.
- Ablation Study on Data Sources: Cumulative returns drop as news and then fundamental data are removed, confirming that both information sources guide trading decisions.Kimi-K2 is relatively robust without news alone, whereas GPT-OSS-120B declines more sharply when explicit signals are removed.
- Impact of Evaluation Window: All LLM agents underperform the passive baseline during the downturn but outperform it in the upturn, highlighting weaker performance in bearish markets.The evaluation windows are January–April 2025 and May–August 2025, respectively.
- Impact of Evaluation Window: GPT-OSS-120B moves from bottom-ranked in the downturn to top-ranked in the upturn, while Kimi-K2 remains relatively stable across periods.The ranking shifts indicate that model performance depends on market conditions, with Kimi-K2 showing greater robustness to fluctuations.
5 Related Work
Prior LLM-agent benchmarks span software, science, and commercial workflows, but few evaluate decisions with direct economic consequences. Financial benchmarks mainly test static question-answering, whereas STOCKBENCH evaluates continuously updated, multi-stock trading environments.
- Agent Benchmarks: Existing agent benchmarks cover software engineering, scientific discovery, and commercial workflows, while financial trading remains comparatively underexamined.The motivation for extending agent evaluation to finance is its connection to profitability, risk management, and high-stakes decisions.
- Financial Agents and Benchmarks: Most financial benchmarks evaluate static question-answering, which does not reflect the iterative and dynamic nature of real-world trading.These benchmarks assess financial reasoning and domain knowledge but not sequential trading behavior.
- Financial Agents and Benchmarks: INVESTORBENCH explores agent-based trading but primarily uses single-stock settings and historical data up to 2021, limiting scope and raising contamination concerns.STOCKBENCH instead embeds agents in realistic multi-stock environments with continuously updated market data.
6 Conclusion
The paper introduces STOCKBENCH for evaluating LLM agents in realistic stock-trading environments with dynamic markets and long-horizon decisions. Experiments show that agents can be profitable but rarely outperform simple baselines, motivating further research.
- Conclusion: STOCKBENCH evaluates LLM agents in realistic stock-trading environments with dynamic markets and long-horizon decision making.The benchmark is released to support future research on more capable trading agents under complex market dynamics.
- Conclusion: Current agents can be profitable but rarely outperform simple baselines, revealing substantial room for improvement.The conclusion positions this result as evidence that stronger autonomous financial agents remain an open research challenge.
7 Limitations
STOCKBENCH evaluates daily trading over several months on a fixed set of large DJIA stocks, so its scope excludes several market settings and trading frictions.
- Limitations: The benchmark does not cover high-frequency trading, long-term market cycles, or a wide range of assets.Its four-month window and fixed set of large U.S. stocks limit exposure to extended markets, rare events, and diverse asset classes.
- Limitations: Because agents trade only once per day and trading costs, slippage, and liquidity limits are omitted, the setup is easier than real markets.Intraday and event-driven strategies are therefore not tested.
8 Ethical Statement
STOCKBENCH follows applicable financial, data-protection, and academic ethical standards. Its data come from authorized sources, and the benchmark is limited to academic and noncommercial research.
- STOCKBENCH complies with applicable financial regulations, data-protection laws, and academic ethical standards.
- The benchmark uses market data collected through licensed vendors or public APIs that allow research use.
- STOCKBENCH is provided for academic and noncommercial research purposes only.
A Prevent Data Leakage
The study addresses potential financial data leakage by restricting evaluation to a recent, narrow time window. This design aims to reduce exposure to future information and better reflect real-world evaluation conditions.
- GPT-5 accurately predicted AAPL’s 2021 stock trend without search, illustrating how learned financial knowledge can exaggerate performance.
- The evaluation uses a more recent and narrow time frame to minimize potential data leakage from future information.
- This approach is intended to ensure that results reflect LLM agents’ predictive and decision-making capabilities without unintended access to future data.
B Model Return Variance
The section examines return variance as an indicator of model performance stability in financial decision-making. DeepSeek-V3 shows the smallest fluctuation, while GPT-OSS-120B shows the highest return variance.
- Higher return variance may indicate less predictable behavior, which is undesirable in high-risk financial decision-making.
- DeepSeek-V3 exhibited the smallest performance fluctuation among evaluated models, indicating high stability.
- GPT-OSS-120B exhibited the highest return variance, indicating volatility in its performance.
C The Use of Large Language Models
The authors use LLMs for code implementation and proofreading, while human researchers conduct architecture design. LLMs are not used for creative research tasks such as ideation or experiment design.
- LLMs assist with data gathering and experiment implementation by completing code snippets, while human researchers design the architecture.
- LLMs are used as writing tools to correct grammar and refine the paper draft.
- LLMs are not responsible for creative tasks including ideation, experiment design, or paper organization.