LiveTradeBench: Seeking Real-World Alpha with Large Language Models

1. Introduction

• Static benchmarks use fixed inputs and single-turn reasoning, so strong scores need not reflect the ability to perceive, act, and adapt through feedback.The cited examples include knowledge quizzes, math reasoning, and instruction-following tasks.Source: §1 · p. 1
• Prior trading evaluations often rely on offline backtesting vulnerable to information leakage or model single-asset buy, sell, and hold actions.The paper also identifies limited treatment of uncertainty, volatility, feedback, and cross-asset planning in these approaches.Source: §1 · p. 2
• LiveTradeBench streams market data and signals while asking agents to allocate portfolios across multiple assets, integrating risk management and decisions over time.Agents observe market conditions, contextual signals, and their past decisions before updating allocations to balance risk and return.Source: §1 · p. 2
• High LMArena scores do not imply superior trading outcomes; models instead show distinct portfolio styles and can use live signals to adapt decisions.The evaluations cover U.S. stock trading and Polymarket betting.Source: §1 · pp. 2–3

2. Related Work

• Prior work spans historical backtests, synthetic simulators, and emerging live evaluation, which remains largely unexplored for trading.Backtests face leakage and generalization concerns, while simulators mainly support behavioral analysis rather than realistic market-aligned actions.Source: §2 · p. 3
• Trading formulations include discrete single-asset actions, alpha predictions, and betting probabilities; portfolio allocation unifies them as executable multi-asset decisions.The portfolio formulation emphasizes risk–return trade-offs and inter-asset correlations.Source: §2 · p. 3
• The benchmark uses a ReAct-style agent configuration with tool use and memory within a broader landscape of fine-tuned and multi-agent trading systems.Related work also identifies self-reflection as a component used in trading-agent frameworks.Source: §2 · p. 4

3.1. Definition of Portfolio Management

• The task is a partially observable Markov decision process with latent market conditions and portfolio actions, and trades map allocation decisions to executable buy, sell, or hold operations.The environment combines exogenous market evolution with portfolio adjustment; the modeling assumption is that trades in selected liquid assets negligibly affect real-world prices.Source: §3.1 · pp. 4–5

3.2. Market Selection

• Stocks are described as smoother and more correlated, whereas Polymarket contracts react more sharply and asynchronously to information, motivating different portfolio-management demands.The paper characterizes stock investing as requiring long-horizon analysis and diversification, and prediction-market betting as requiring rapid, event-driven adaptation.Source: §3.2 · pp. 5–6, Figure 1 · p. 5
• The U.S. stock environment uses 15 sector-spanning equities and ETFs plus cash, while Polymarket tracks ten active binary markets across politics, crypto, technology, and finance.The two settings are intended to test generalization across markets with different structure, information flow, and reaction speed.Source: §3.2 · pp. 5–6

3.3. Observation Space

• Each observation combines current holdings, market prices, and contextual signals such as real-time news, giving agents quantitative, portfolio, and qualitative inputs.Prices include stock closing prices or real-time Polymarket prices; news summaries are included in prompts as contextual information.Source: §3.3 · p. 7
• Holdings are continuous and non-negative, so agents represent fine-grained capital allocation but cannot take short positions.The position vector includes assets and cash.Source: §3.3 · p. 7

3.4. Action Space

• Actions are continuous portfolio allocations that sum to the total portfolio value and use a default long-only setting, abstracting from low-level execution.The action space focuses on portfolio-level return, risk, and stability objectives.Source: §3.4 · p. 7
• In stocks, each action component specifies the share of portfolio value allocated to a stock and determines the post-trade position through proportional rebalancing.This expresses capital distribution across sectors.Source: §3.4 · p. 7
• For each binary prediction contract, the action assigns portfolio allocations to complementary YES and NO outcomes, with positive net exposure indicating greater confidence in YES.The action vector has two components per active market.Source: §3.4 · p. 8

4. Designing LLM-based Agents for Portfolio Management

• The agent conditions portfolio allocations on current observations and internal memory, combining tool use, memory, and reasoning in a closed decision loop.Its policy maps observations and memory to allocations.Source: §4 · p. 8
• Tool use acquires and filters live market and contextual signals, including price-derived dynamics and market-relevant news, into structured features for allocation decisions.The module serves as the agent’s interface with the external environment.Source: §4 · pp. 8–9
• Memory stores a fixed-length sequence of past observations and action outcomes, providing temporal context for volatility, allocation changes, and drawdown trends.The memory horizon is denoted by Δ, and decisions condition on both current observation and memory.Source: §4 · p. 9
• Before acting, the reasoning component uses ReAct to interpret signals, recall experiences, and hypothesize outcomes through explicit reasoning traces.The traces connect perception and decisions and can help researchers analyze model behavior.Source: §4 · p. 9

5.1. Backbone LLMs for Evaluation

• The live evaluation spans 50 trading days and 21 LLM backbones, selected from six representative families and run through a shared agent framework.The framework converts market observations into natural-language prompts and parses responses into portfolio-allocation vectors.Source: §5 · p. 9, §5.1 · pp. 9–10

5.2. Trading Metrics for Evaluation

• Trading performance is assessed with cumulative return, volatility, Sharpe ratio, maximum drawdown, and win rate, covering returns, risk, risk-adjusted efficiency, downside protection, and profitable-step frequency.Sharpe ratio uses a short-term Treasury yield as the stock-market risk-free rate and zero for Polymarket.Source: §5.2 · p. 10

5.3. Evaluation Results

• Sharpe performance across U.S. stocks and Polymarket is weakly related, and LMArena scores do not reliably predict trading results.The cross-market Sharpe correlation is close to zero; LMArena-score correlations with cumulative returns are 0.054 for stocks and −0.38 for Polymarket.Source: §5.3 · pp. 11–12
• Models adopt distinct risk and portfolio styles across markets, from conservative allocations to risk-seeking strategies, concentrated holdings, diversification, and higher cash balances.Claude-Opus-4.1 and Grok-4 are conservative, while Kimi-K2-Instruct and GPT-5 accept higher risk; GPT-4o concentrates in a few stocks, whereas GPT-5 diversifies more broadly.Source: §5.3 · p. 12
• Reasoning-focused models do not outperform other models in trading and show Polymarket volatility above 140.The paper reports that excessive deliberation can introduce instability and degrade trading consistency.Source: §5.3 · p. 12

5.4. Analysis and Discussion

• 0.03% stock returns at k=2 contrast with 2% Polymarket degradation as k increases; slower updates generally worsen performance.The rolling-k delta compares cumulative returns using positions delayed by k days against current decisions.Source: Figure 5 · p. 13, Figure 6 · p. 13, §5.4 · p. 13
• Reasoning traces cite news most often, followed by price history, while Polymarket agents rely more on news and stock agents emphasize price trends.References can overlap because a trace may use multiple information sources.Source: Figure 7 · p. 13, §5.4 · p. 14

6.1. Cash Asset Dynamics in the U.S. Stock Market

• 17% to 7.5%: average cash holdings across 21 models fell over four trading days during the August 28 technology-stock rally.GPT-4.1, identified as having the highest cumulative return rate, cited earnings momentum and positive analyst sentiment while increasing exposure to technology stocks.Source: §6.1 · p. 14
• On October 10, most agents increased cash after stock prices fell, and Gemini-2.5-Pro, which had held relatively high cash and converted more assets before the downturn, had the smallest daily loss.Tesla fell over 5%, while Amazon and Nvidia fell more than 4%; all agents had negative returns that day.Source: §6.1 · p. 14, Figure 8 · p. 15

6.2. Russia–Ukraine Ceasefire Market in Polymarket Prediction Market

• On October 13, agents shifted from No to Yes after optimistic headlines, but the price barely moved and the reactive trades produced no profit.The case highlights the challenge of distinguishing attention-grabbing news from events that materially affect ceasefire likelihood.Source: §6.2 · p. 15, Figure 9 · p. 16
• On October 17, agents strengthened and held Yes positions after Zelenskyy’s White House visit; prices rose through October 18, producing tangible profits.Agents cited diplomatic developments and a price jump to 0.18 as confirming evidence, unlike the earlier reactive shift.Source: §6.2 · pp. 15–16, Figure 9 · p. 16

7. Conclusion

• LiveTradeBench evaluates LLM agents’ portfolio management through live, multi-market interaction in stock and prediction markets.The conclusion frames it as a setting for studying how agents perceive, reason, and act under uncertainty.Source: §7 · p. 16
• Strong performance on LMArena does not necessarily translate into superior trading, and strong performance in one market does not generalize to others.The conclusion describes a gap between text intelligence and dynamic decision-making, alongside heterogeneity across market types.Source: §7 · p. 16
• Agents rely jointly on historical price trends, market news, and allocation history, while displaying distinct behavioral patterns under extreme conditions.Source: §7 · p. 16

Limitation and Future Work

• The evaluation omits transaction fees, bid–ask spreads, liquidity constraints, and other trading frictions, which may overestimate returns, especially for frequent rebalancing.The authors propose adding cost models and slippage simulations.Source: Limitation and Future Work · p. 17
• Limited context length restricts accessible price, position, and news histories, supported markets, and news inputs to titles and abstracts rather than full articles.The authors suggest longer horizons, richer text, and dynamic market expansion.Source: Limitation and Future Work · p. 17
• The ReAct-based agent design uses basic tools and memory and remains limited in reasoning depth and temporal abstraction.Proposed directions include specialized tools, richer memory, multi-agent setups, and reinforcement learning.Source: Limitation and Future Work · p. 17

Open-source Application

• The open-source live-trade-bench Python package provides APIs for data collection, environment setup, and agent construction, alongside a web app for real-time deployment and performance monitoring.Source: Open-source Application · p. 17

S1. Data Collection

• Data collection combines live prices and news from public finance and Polymarket endpoints, Google News, and Reddit, binding retrieved items to tickers or market IDs.Items are presented alongside account history.Source: S1. Data Collection · p. 25

S1.1. Market News Data

• News queries use a short [t−3, t−1] window; articles are timestamp-normalized, invalid timestamps are dropped, and results are tagged and sorted by ticker or question.Stock queries combine ticker and company name; Polymarket queries use the market question.Source: S1.1. Market News Data · p. 25

S1.2. Stock Price Data

• U.S. stock prices use a 10-day lookback ending at t−1 and a curated default universe of 15 tickers, with current prices and compact daily histories exposed.Missing dated closes fall back to the best available price that day.Source: S1.2. Stock Price Data · p. 25

S1.3. Polymarket Price Data

• Polymarket token histories use a 10-day lookback ending at t−1; markets are filtered for deduplication, observable history, and non-flat price ranges, with quotes normalized to probabilities in [0, 1].The system exposes current token prices and per-day histories.Source: S1.3. Polymarket Price Data · pp. 25–26

• The full model input pairs a dynamic context prompt summarizing market status, recent news, and account information with a fixed instruction header specifying objectives, portfolio principles, and output requirements.Separate complete prompts are provided for stocks and Polymarket.Source: S2. Prompting Details · p. 26
• Stock prompts present prices and short histories by ticker, ticker-grouped news, and account allocations and performance, then request JSON allocations summing to 1.0 including CASH.The instructions foreground risk-adjusted return, diversification, and turnover awareness.Source: S2. Prompting Details · p. 26
• Polymarket prompts show YES/NO implied probabilities and histories, question-grouped news, and account performance; agents may choose at most one side per question and allocate across outcomes and CASH.The decision instructions compare the agent’s belief p with market probability pmkt while considering transaction costs.Source: S2. Prompting Details · p. 26

• The same pool of mainstream chat LLMs is evaluated in both stock and Polymarket settings, with model families and variants summarized in Table S1.Source: S3. Model Details · p. 26, Table S1 · p. 27
• Models are routed through LiteLLM by vendor-prefixed names or default to Together AI; standard chat models use temperature 0.3 and 16,000 maximum tokens, while structured-reasoning styles use provider defaults.Source: S3. Model Details · pp. 26–27
• Every model must return JSON reasoning and allocations that sum to 1.0 across available assets, optionally including CASH; responses are parsed and validated before application.Source: S3. Model Details · p. 27

• The leaderboard ranks models by profit return rate, with stock accounts starting at 1000 USD and Polymarket accounts at 500 USD.Source: S4. Frontend UI Details · p. 27, Figure S1 · p. 32
• The Stock and Polymarket pages each show all 21 models; their model cards provide detailed views of current and historical allocations, profits, and LLM inputs and outputs.Source: S4. Frontend UI Details · p. 27, Figure S3 · p. 34, Figure S4 · p. 35
• The news page presents recent stock-market and Polymarket news; the example feed includes Apple, Amazon, and other stock headlines with dates and snippets.Source: S4. Frontend UI Details · pp. 27–28, Figure S6 · p. 37
• Supplementary tables provide example stock and Polymarket context and decision prompts.Source: Table S2 · p. 28, Table S3 · p. 29, Table S4 · p. 30