PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Trader: Benchmarking Autonomous Agents in Real-Time Financial Markets

Tianyu Fan, Yuhao Yang, Yangqin Jiang, Yifei Zhang, Yuxuan Chen, Chao Huang

arXiv:2512.10971v1q-fin.CPcs.CE

TL;DR

AI-Trader addresses the lack of systematic benchmarking for autonomous agents in live financial environments. It introduces a fully autonomous, data-uncontaminated benchmark across three markets using minimal information and evaluates six LLMs. The results show that general intelligence does not ensure effective trading, with most agents exhibiting poor returns and weak risk management.

  • Problem

    Existing evaluations do not adequately measure autonomous decision-making under real-time financial conditions requiring continuous adaptation, uncertainty management, and consequential actions.

    Source: §1 · pp. 1–2

  • Method

    AI-Trader evaluates six LLMs across U.S. stocks, A-shares, and cryptocurrencies using live conditions, multiple trading frequencies, and a minimal-information paradigm requiring autonomous search, verification, and synthesis.

    Source: §1 · p. 2

  • Results

    General intelligence does not automatically translate into effective trading: most agents show poor returns and weak risk management, while excess returns are more readily achieved in highly liquid markets.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    The findings expose limitations in current autonomous agents and indicate that risk control is important for robustness across financial markets.

    Source: §1 · p. 2, §5 · p. 12

    Agents may make irrational judgments when they adopt unverified news without cross-verification.

    Source: §4.4 · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable potential as autonomous agents, approaching human-expert performance through advanced reasoning and tool orchestration. However, decision-making in fully dynamic and live environments remains highly challenging, requiring real-time information integration and adaptive responses. While existing efforts have explored live evaluation mechanisms in structured tasks, a critical gap remains in systematic benchmarking for real-world applications, particularly in finance where stringent requirements exist for live strategic responsiveness. To address this gap, we introduce AI-Trader, the first fully-automated, live, and data-uncontaminated evaluation benchmark for LLM agents in financial decision-making. AI-Trader spans three major financial markets: U.S. stocks, A-shares, and cryptocurrencies, with multiple trading granularities to simulate live financial environments. Our benchmark implements a revolutionary fully autonomous minimal information paradigm where agents receive only essential context and must independently search, verify, and synthesize live market information without human intervention. We evaluate six mainstream LLMs across three markets and multiple trading frequencies. Our analysis reveals striking findings: general intelligence does not automatically translate to effective trading capability, with most agents exhibiting poor returns and weak risk management. We demonstrate that risk control capability determines cross-market robustness, and that AI trading strategies achieve excess returns more readily in highly liquid markets than policy-driven environments. These findings expose critical limitations in current autonomous agents and provide clear directions for future improvements. The code and evaluation data are open-sourced to foster community research: https://github.com/HKUDS/AI-Trader.

1 INTRODUCTION

AI-Trader addresses the gap between static agent benchmarks and the demands of real-time financial decision-making. It evaluates autonomous agents in live markets using minimal information, showing substantial limitations in trading performance and generalization.

  • Financial markets test planning, information retrieval, numerical reasoning, and time-critical decisions under volatility, uncertainty, and economic consequences.Source: §1 · pp. 1–2
  • Existing evaluations lack continuous autonomy, genuine time constraints, market volatility, and economic consequences needed for realistic financial assessment.Source: §1 · pp. 1–2
  • AI-Trader is a fully autonomous, live, data-uncontaminated benchmark spanning U.S. stocks, A-shares, and cryptocurrencies.It operates under real-time market conditions with strict temporal filtering and information isolation.Source: §1 · p. 2
  • Agents receive only holdings, real-time prices, and tools, then independently search, verify, synthesize, and act without human intervention.The minimal-information design exposes information acquisition, synthesis, and strategic decision-making under time-critical constraints.Source: §1 · p. 2
  • Across six mainstream LLMs, general intelligence does not automatically yield effective trading: most agents show poor returns, weak risk management, and limited cross-market generalization.Excess returns are more readily achieved in highly liquid U.S. markets than in policy-driven A-share environments.Source: §1 · p. 2

2 RELATED WORKS

Related work has progressed from static evaluation toward live environments and tool-using agents, but financial benchmarks still fall short of genuine autonomy and real-time constraints.

  • Tool use sustains the perception–decision–action loop by enabling information retrieval, computation, and external actions through standardized interfaces such as MCP.Source: §2 · p. 3
  • AI-Trader extends prior work by requiring agents to search, verify, and synthesize live financial information with only essential tools and zero human supervision.Source: §2 · p. 3
  • Live agent benchmarks range from static task banks with dynamic execution to fully dynamic tasks and environments that continuously update.Examples include SWE-Bench, GAIA, BrowseComp, XBench, Tau2Bench, LiveCodeBench, and FutureX.Source: §2 · p. 3
  • Financial benchmarks cover static factual knowledge and simulated trading, but existing approaches still lack genuine autonomy and real-time constraints.This limitation separates current financial evaluations from actual financial environments.Source: §2 · p. 3

3 THE FRAMEWORK OF AI-TRADER

AI-Trader combines a live, autonomous trading environment with modular agents that observe markets, reason through tools, and execute constrained actions across three markets and two frequencies.

3 THE FRAMEWORK OF AI-TRADER

  • AI-Trader separates a live trading environment from trading agents to evaluate autonomous strategies under real market pressures, volatility, uncertainty, and economic consequences.Source: §3 · pp. 3–4

3.1 LIVE TRADING ENVIRONMENT

  • The benchmark covers U.S. stocks, A-shares, and cryptocurrencies at hourly and daily frequencies to test generalization across market structures and time horizons.Source: §3.1 · p. 4

3.1.1 U.S. STOCK MARKET

  • The U.S. portfolio uses Nasdaq-100 constituents and a zero-return cash asset, providing liquid growth-oriented equities and risk-free allocation options.Source: §3.1.1 · pp. 4–5

3.1.2 A-SHARE MARKET

  • The A-share portfolio uses 50 SSE-50 constituents, representing liquid Chinese blue-chip sectors under distinct regulatory and investor conditions.Source: §3.1.2 · p. 5

3.1.3 CRYPTOCURRENCY MARKET

  • The cryptocurrency portfolio contains 10 major Bitwise Index pairs traded continuously in a highly volatile, rapidly changing environment.Its 24/7 operation tests responsiveness to news, technical developments, and sentiment shifts.Source: §3.1.3 · pp. 5–6

3.2.1 DESIGN OF BASE TRADING AGENTS

  • Agents follow a closed-loop Observe-Reason-Act process in which tool-derived observations and autonomous reasoning produce executable trading decisions.Natural-language reasoning traces are recorded for observability, auditability, and reproducibility.Source: §3.2.1 · p. 6
  • Each asset supports buy, sell, or hold, while liquidity or market-rule violations trigger rejection and self-correction through renewed tool use and reasoning.Source: §3.2.1 · pp. 6–7

3.2.2 TOOL CHAIN FOR AGENTS

  • The MCP-based toolchain provides price data, time-filtered search, financial news, computation, and compliant trade execution within a modular, extensible architecture.Source: §3.2.2 · p. 7

4 EXPERIMENTAL EVALUATION

AI-Trader evaluates six identically structured LLM agents across three markets and trading frequencies using live performance and risk metrics. Results show weak generalization, market-dependent profitability, and the importance of adaptive risk control and information verification.

4 EXPERIMENTAL EVALUATION

  • The evaluation used six mainstream models under identical objectives and tools, daily trading for A-shares and cryptocurrencies, and hourly monitoring for U.S. equities.Performance was assessed with cumulative return, Sortino ratio, volatility, and maximum drawdown.Source: §4 · p. 7, §4.1 · pp. 7–8, §4.2 · p. 8

4.3 PERFORMANCE

  • MiniMax-M2 achieved 9.56% cumulative return, a 4.42 Sortino ratio, and -4.92% maximum drawdown in U.S. stocks, while no agent beat the 1.65% SSE-50 benchmark in A-shares.In A-shares, MiniMax-M2 came closest with 1.31% return, 6.72% volatility, and -2.15% maximum drawdown.Source: §4.3 · pp. 8–9
  • AI strategies generated excess returns more readily in U.S. stocks, where MiniMax-M2, DeepSeek-v3.1, and Claude-3.7-Sonnet outperformed QQQ, than in A-shares, where no agent beat SSE-50.The results associate stronger performance with the more liquid, continuously traded U.S. market and weaker performance with the policy-driven A-share environment.Source: §4.3 · p. 9
  • Cross-market generalization remained limited: DeepSeek-v3.1 performed strongly in U.S. stocks, turned negative in A-shares, and regained an advantage in cryptocurrency markets.Its U.S. strategy favored high-liquidity or clear-momentum environments but struggled in the policy-driven A-share market.Source: §4.3 · p. 9
  • Agents changed position sizes and trading behaviors across markets, with volatile A-shares inducing more speculative activity and relatively stable U.S. stocks encouraging steadier trend-following strategies.Gemini-2.5-Flash’s A-share positions surged to 30 before liquidation, whereas U.S. and A-share holdings differed substantially across agents.Source: §4.3 · pp. 9–10, Figure 2 · p. 9

4.4 CASE STUDY

  • Case studies showed both adaptive and flawed information use: DeepSeek-v3.1 reduced technology exposure and held cash after analyzing tariff news, while unverified news led to an A-share misjudgment.The defensive adjustment reduced technology allocation from about 99% to 70% and maintained a 17.3% cash ratio; the contrasting case lacked cross-verification.Source: §4.4 · pp. 10–11

5 CONCLUSION

AI-Trader benchmarks LLM agents in dynamic, real-time financial markets and finds that current general capabilities do not reliably translate into effective autonomous trading.

  • AI-Trader evaluates LLM agents in U.S. stocks, A-shares, and cryptocurrency markets at hourly and daily trading granularities.Source: §5 · p. 12
  • Most agents show poor returns and weak risk management, with performance highly sensitive to market conditions.Source: §5 · p. 12
  • The benchmark exposes limitations in current LLM agents and identifies directions for future system improvements.Source: §5 · p. 12

A PROMPTS FOR BASIC AGENTS IN AI-TRADER

Basic AI-Trader agents receive portfolio and price context, use tools to gather information, and directly execute portfolio-management operations toward maximizing returns.

B OTHER METRICS FOR EVALUATION

The supplementary evaluation reports risk and performance metrics across U.S. equities, A-shares, and cryptocurrencies, showing market-dependent advantages for model-driven trading.

  • MiniMax-M2 shows robust performance across all three markets, while DeepSeek-v3.1 adapts uniquely to high-volatility environments.Source: B OTHER METRICS FOR EVALUATION · p. 17
  • Model-driven strategies significantly outperform the baseline in U.S. stocks but do not necessarily maintain that advantage in A-shares and cryptocurrencies.Source: B OTHER METRICS FOR EVALUATION · p. 17
  • Figure A1 tracks Cumulative Return, Sortino Ratio, Volatility, and Maximum Drawdown over time for agents in both markets.Source: Figure A1 · p. 17
Loading 2512.10971v1…