AlphaQuanter: An End-to-End Tool-Augmented Agentic Reinforcement Learning Framework for Stock Trading

1 Introduction

• Existing LLM trading attempts lack autonomous tool orchestration and consistently evidence-grounded decision traces, while prompt-based and multi-agent methods face robustness or efficiency challenges.The introduction also identifies partially observable reasoning, heterogeneous signals, and calibrated-confidence actions as core challenges.Source: §1 · p. 1
• The framework unifies trading-oriented planning, reasoning, tool use, and on-demand information seeking in a single agent.It uses reinforcement learning with verifiable rewards to optimize selective tool use and evidence gathering.Source: §1 · p. 1
• Its end-to-end RL approach actively acquires useful information and selects supporting evidence while directly optimizing the decision-making process for long-term profitability.Outcome- and process-based rewards guide training across diverse actions.Source: §1 · p. 2
• AlphaQuanter’s evaluations report state-of-the-art performance on key financial metrics and improved decision-trace faithfulness, with coherent, evidence-grounded patterns for human analysis.The introduction identifies backtesting protocols and human evaluation as evaluation settings.Source: §1 · p. 2

Traditional Machine Learning

• Traditional machine-learning approaches frame trading as price-direction classification, which the discussion describes as oversimplified and poorly generalizing in trading environments.The passage gives SVM and random forest as examples.Source: §2 · p. 2

Deep Reinforcement Learning

• Deep reinforcement learning work targets end-to-end trading returns, imitation-based stabilization, risk-sensitive position adjustment, and multi-dataset returns, but remains black-box and lacks on-demand external signals.The related-work discussion describes these limitations for the methods it reviews.Source: §2 · p. 2

LLMs-Based Trading Agents

• LLM trading systems use memory retrieval, multi-agent debate, multimodal tools, or human-in-the-loop factor mining, but face coordination, prompt-sensitivity, or automation limits.The reviewed debate-style processes can yield conflicting signals, while human-in-the-loop factor mining is difficult to scale autonomously in high-frequency settings.Source: §2 · p. 2

LLM-Based Reinforcement Learning Optimization

• LLM-based RL methods optimize trading rewards with policy gradients or staged SFT and RL, yet lack end-to-end trading simulation, autonomous exploration, and proactive tool orchestration.The reviewed methods also lack spontaneous perception of information gaps.Source: §2 · p. 2

3 Problem Definition

• The agent’s state accumulates initial context, invoked tools, and their corresponding query results.The initial context includes stock symbol and date metadata.Source: §3 · p. 3
• Available actions comprise queries across market, fundamental, sentiment, and macroeconomic sources, or a terminal BUY, SELL, or HOLD decision.A query updates the state; a decision ends the process.Source: §3 · p. 3
• Query transitions append the query and result to the history, while a decision action terminates the episode.The passage describes state transitions as deterministic.Source: §3 · p. 3
• The policy maximizes cumulative reward over a trajectory of query actions ending with the first decision action.The reward is designed to favor BUY in positive conditions, SELL in negative conditions, HOLD in neutral conditions, and informative queries.Source: §3 · p. 3

4.1 Cognitive Workflow

• The workflow begins with a plan and iterates through acquiring evidence, reasoning over it, and either gathering more information or committing to a trading decision.This couples stepwise hypothesis testing with evidence collection.Source: §4.1 · pp. 3–4

4.2 Reward Formulation

• The outcome score smooths future returns across multiple horizons to filter short-lived fluctuations and emphasize the medium-term trajectory.The smoothed return is thresholded to identify market regimes and assign discrete rewards by action.Source: §4.2 · p. 4
• Taking the opposite side of a strong trend receives a −1.0 reward, compared with −0.75 for failing to act on an opportunity.The asymmetric penalties are intended to provide a more informative learning signal and nudge the policy toward risk-aware behavior.Source: §4.2 · p. 4
• The process score combines format and tool-use scores, penalizing unsuitable reasoning length, tool-call counts, degenerate collect-then-conclude behavior, and malformed calls.The format score targets a token interval; the tool score targets a tool-call interval.Source: §4.2 · p. 4
• Total reward is α times the outcome score plus the format and tool scores, with α emphasizing the outcome score.The paper identifies outcome as the primary importance in this weighting.Source: §4.2 · p. 4

5.1 Backtesting Protocol

• Backtests apply the daily-trained policy sequentially across a historical period and measure the resulting portfolio performance.The protocol evaluates the strategy over extended horizons.Source: §5.1 · p. 4

5.2 Portfolio State and Transition Dynamics

• Table 2 lists backtesting simulation parameters, while Table 3 identifies action-dependent portfolio-state transition rules.The transition from the current to the new portfolio state is determined by the action.Source: §5.2 · p. 4, Table 2 · p. 4, Table 3 · p. 4

5.3 Evaluation Metrics

• ARR measures annualized profitability, SR measures risk-adjusted performance, and MDD measures the largest peak-to-trough decline, with lower MDD indicating better downside-risk control.The metrics follow prior work cited by the authors.Source: §5.3 · pp. 4–5

6.1 Dataset and Simulation Period

• The dataset covers GOOGL, MSFT, META, NVDA, and TSLA, split chronologically into non-overlapping train, validation, and test sets with an approximately 30-trading-day gap.The gap is intended to prevent leakage and look-ahead bias; Table 4 summarizes splits and trading-day counts.Source: §6.1 · p. 5, Table 4 · p. 5

6.2 Information Sources

• The tool combines daily prices and technical indicators, corporate financial data, news and social-media sentiment, and macroeconomic indicators.These sources support technical analysis, intrinsic-value assessment, sentiment tracking, and macroeconomic context, respectively.Source: §6.2 · p. 5

6.3 Implementation

• Seven baseline categories span passive, rule-based, RL, LM, LLM, multi-agent, and untrained AlphaQuanter agents; trained agents use Qwen2.5 backbones, GRPO, deterministic decoding, and three-run means.The reported training configurations use Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct backbones.Source: §6.3 · p. 5

7.1 Overall Performance Comparison

• 6.54% and 18.45% absolute ARR gains for AlphaQuanter-3B and -7B, respectively, exceed the strongest baseline; 7B beats all baselines on three of five stocks.The backtesting table compares methods over six months.Source: Table 5 · p. 6, RQ1: Single or multi-agent, which is… · p. 6

RQ1: Single or multi-agent, which is better?

• Except for GPT-4o, single-agent frameworks outperform multi-agent ones on key metrics, while prompt-only baselines (except GPT-4o) fail to beat buy-and-hold on average.The authors attribute prompt-only underperformance to difficulty calibrating when signals should trigger BUY rather than HOLD.Source: RQ1: Single or multi-agent, which is… · p. 6

7.2 Training Dynamics and Validation Performance

• 3B shifts to fewer tool calls and shorter responses, while 7B refines after step 200; validation ARR and SR rise for both, but MDD trends diverge.The 7B model's MDD trends downward, whereas the 3B model's MDD oscillates with an upward bias.Source: Figure 2 · p. 7, §7.2 · p. 7, Figure 3 · p. 7
• AlphaQuanter delivers the best ARR at token usage of only 0.15× the multi-agent baseline, despite prompt-only SA being most token-efficient.Inference cost is measured as average LLM-generated tokens per trading decision across five datasets.Source: Table 6 · p. 7, §7.2 · p. 7

7.3 Faithfulness Analysis

• SA+RL leads Alignment, Evidence Grounding, and Conciseness; MA scores lowest on alignment and conciseness, with α=0.78 inter-rater reliability.Three Ph.D. raters independently scored 50 sampled inputs.Source: Table 7 · p. 7, §7.3 · p. 7

7.4 Tool Usage Patterns

• Tool use evolves during training; 3B remains diffuse, whereas 7B develops concentrated, selective information prioritization.Figure 4 heatmap intensity represents percentile-based reliance on information sources across training steps.Source: Figure 4 · p. 8, §7.4 · p. 8

7.5 Ablation Studies

• Removing format and tool rewards cuts average ARR 53.2% and 43.0%; θ shifts of ±0.5% cut ARR 42.1% and 39.2%, while larger θ lowers trading frequency and MDD.Figure 5 shows the effect of θ values on action distributions during training.Source: Table 8 · p. 8, §7.5 · p. 8, Figure 5 · p. 8

7.4 Tool Usage Patterns

• The 7B model prioritizes trend, momentum, and volume indicators, uses sentiment and macroeconomic context secondarily, and largely downweights low-frequency fundamentals.The paper suggests low-frequency fundamentals add limited value for the task's rapid decisions.Source: §7.4 · p. 8

• AlphaQuanter is a single-agent framework that uses RL to optimize the decision-making process.Source: §8 · p. 8
• Across five large-cap U.S. stocks, AlphaQuanter improves ARR while keeping token cost per decision low, supporting practical deployment.Source: §8 · p. 9
• It learns when and what to retrieve, treating tool use as a policy action, and produces more consistent traces with higher alignment and conciseness.Source: §8 · p. 9

Limitations

• The agent can orchestrate existing functions but cannot generate novel analytical methods.The authors identify broadening the framework to portfolio-level decisions as an important extension.Source: Limitations · p. 9
• The framework focuses on single-asset decision making, leaving portfolio-level decisions outside its current scope.Source: Limitations · p. 9
• Accumulating observations from iterative tool calls can make the input context excessively long under computational constraints.The authors identify agent memory for summarizing and managing reasoning history as a needed direction.Source: Limitations · p. 9

Ethics Statement

• The research is strictly confined to a simulated backtesting environment using historical data.Source: Ethics Statement · p. 9
• The authors state that, as a research prototype, the work does not directly introduce new risks to live markets or participants.Source: Ethics Statement · p. 9
• The authors describe transparency and use of publicly available financial data under applicable terms and licenses as ethical principles of the work.Source: Ethics Statement · p. 9

A.1 Market Data; A.1.1 Price and Volume Data; A.1.2 Technical Indicators

• Market inputs include daily OHLC prices, adjusted close, and volume, alongside queryable technical indicators grouped by analytical function.Examples include volatility, volume, trend, and momentum indicators such as ATR, OBV, SMA, and RSI.Source: §A.1 · p. 12, §A.1.1 · p. 12, §A.1.2 · p. 12

A.2 Fundamental Data; A.3 Sentiment Data; A.4 Macroeconomic Indicators

• Additional information sources include company financial data, news and Reddit sentiment, and macroeconomic indicators such as Treasury yields, the federal funds rate, CPI, oil, and copper prices.Reddit posts are summarized to manage input context length.Source: §A.2 · p. 12, §A.3 · p. 13, §A.4 · p. 13

B.1 Prompt Design

• The prompt guides the agent to form and test hypotheses, use one tool at a time, and show its reasoning in a structured format, with a maximum tool-call constraint.The complete prompt is shown in Figure 6.Source: §B.1 · p. 13, Figure 6 · p. 14

B.2 Process Score Computation

• The process score combines a FormatScore for response structure and a ToolScore for tool-use quality, and both are used during RL training.The assessed behaviors include valid action labels and reasoning tags, valid tool arguments, duplicate calls, and reasonable call counts.Source: §B.2 · p. 13

B.3 Hyperparameters for RL Training; B.4 Detail of Baseline

• Baselines include buy-and-hold, technical rules, RL agents, price-forecasting, prompt-based LLM traders, and multi- and single-agent configurations.The single-agent baseline uses the custom prompt and four data categories but lacks AlphaQuanter’s RL-optimized workflow; Table 9 lists training hyperparameters.Source: §B.4 · pp. 14–15, Table 9 · p. 16

C.1 Full Results of Main Table

• Across the full backtests, AlphaQuanter-7B achieves the highest ARR on MSFT and NVDA, with strong SR and relatively low MDD on MSFT.For NVDA, its SR and MDD are comparable to the market baseline; the tables provide stock-specific results.Source: §C.1 · pp. 15–16, Table 10 · p. 17, Table 11 · p. 17

C.2 Rolling-Window Robustness Analysis

• 52.26% Avg. ARR and 0.26 Avg. SR are the best overall rolling-window results for AlphaQuanter-7B, with 17.48% Avg. MDD.The evaluation uses a three-month rolling horizon with a seven-day step; the threshold and other hyperparameters remain unchanged across windows.Source: §C.2 · pp. 16, 18, Table 12 · p. 18

C.3 Reward Decomposition Analysis

• Figure 7 shows validation reward components: both models’ result scores rise, while the 7B model outperforms the 3B model across scoring metrics.After about 250 steps, the 7B reasoning-length-related format score continues upward while the 3B score declines; further training rectifies early 3B tool-score problems.Source: Figure 7 · p. 18, §C.3 · p. 18

C.4 Detailed Faithfulness Analysis

• In Section 7.3, three Ph.D. raters independently rate each output on an ordinal 0–2 scale for three human-rated faithfulness metrics, with higher scores indicating better performance.Method-level results are reported by averaging scores.Source: §C.4 · pp. 18–19

C.5 Full Results of Ablation Study

• On MSFT, NVDA, and TSLA, and on average, ablated or threshold-adjusted models perform worse than complete AlphaQuanter-7B.Increasing the decision threshold consistently lowers MDD across stocks, while decreasing it raises MDD; the authors relate this pattern to trading frequency.Source: Table 13 · p. 19, Table 14 · p. 19, §C.5 · p. 20

C.4 Detailed Faithfulness Analysis

• SA+RL achieves the best alignment and conciseness, while grounding remains competitive across settings with a small, consistent advantage.Its traces more consistently preserve a reason-tool-response chain and avoid unnecessary back-and-forth; remaining low scores involve minor inconsistencies with retrieved evidence.Source: §C.4 · pp. 19–20
• After RL training, the example explicitly states an information plan and calls the corresponding tools, unlike the pre-training mismatch between reasoning and actions.Source: Figure 8 · p. 20, §C.4 · p. 20

D A Working Example

• In the MSFT example, overbought technical indicators, mixed news, and insider share disposals inform a final HOLD recommendation.Source: D A Working Example · p. 22