PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist

Wentao Zhang, Lingxuan Zhao, Haochong Xia, Shuo Sun, Jiaze Sun, Molei Qin, Xinyi Li, Yuqing Zhao, Yilei Zhao, Xinyu Cai, Longtao Zheng, Xinrun Wang, Bo An

arXiv:2402.18485v3q-fin.TRcs.AI

TL;DR

Financial trading systems must handle multimodal information and diverse, changing tasks, while existing methods have limited multimodal processing and generalizability. FinAgent combines multimodal market intelligence, reflection, diversified retrieval, memory, and expert tools; across six datasets it outperforms state-of-the-art baselines, with over 36% average profit improvement.

  • Problem

    Existing financial trading methods face inadequate multimodal data handling and limited generalizability across diverse market conditions and tasks.

  • Method

    FinAgent is a multimodal foundation agent that uses market intelligence, dual-level reflection, diversified memory retrieval, reasoning, and expert trading tools.

  • Results

    Over 36% average improvement on profit was achieved across 6 financial datasets and 6 financial metrics against 12 state-of-the-art baselines.

  • Takeaways & Limitations

    FinAgent provides a multimodal, tool-augmented foundation agent for financial trading that integrates market information and historical trading patterns into decisions.

  • Takeaways & Limitations

    Its auxiliary agents are specialized for stocks, yielding lower returns than FinMem on cryptocurrency ETH; a generalized auxiliary agent could raise returns from 44% to 54%.

Abstract

from arXiv · show

Financial trading is a crucial component of the markets, informed by a multimodal information landscape encompassing news, prices, and Kline charts, and encompasses diverse tasks such as quantitative trading and high-frequency trading with various assets. While advanced AI techniques like deep learning and reinforcement learning are extensively utilized in finance, their application in financial trading tasks often faces challenges due to inadequate handling of multimodal data and limited generalizability across various tasks. To address these challenges, we present FinAgent, a multimodal foundational agent with tool augmentation for financial trading. FinAgent's market intelligence module processes a diverse range of data-numerical, textual, and visual-to accurately analyze the financial market. Its unique dual-level reflection module not only enables rapid adaptation to market dynamics but also incorporates a diversified memory retrieval system, enhancing the agent's ability to learn from historical data and improve decision-making processes. The agent's emphasis on reasoning for actions fosters trust in its financial decisions. Moreover, FinAgent integrates established trading strategies and expert insights, ensuring that its trading approaches are both data-driven and rooted in sound financial principles. With comprehensive experiments on 6 financial datasets, including stocks and Crypto, FinAgent significantly outperforms 9 state-of-the-art baselines in terms of 6 financial metrics with over 36% average improvement on profit. Specifically, a 92.27% return (a 84.39% relative improvement) is achieved on one dataset. Notably, FinAgent is the first advanced multimodal foundation agent designed for financial trading tasks.

1 INTRODUCTION

FinAgent addresses the difficulty of applying trading systems to multimodal, rapidly changing markets by combining multimodal analysis, reflection, diversified retrieval, and expert tools. Across six financial datasets, it outperforms state-of-the-art baselines on six financial metrics.

  • Rule-based and reinforcement-learning trading systems struggle with volatility, data demands, explainability, generalization, noise, and multimodal market intelligence.
  • FinAgent integrates numerical, textual, and visual market information to analyze market dynamics and historical trading patterns.
  • Its dual-level reflection and diversified memory retrieval modules support rapid adaptation and learning from historical data while separating retrieval from trading tasks.
  • FinAgent uses expert guidance and technical-indicator trading strategies to provide domain-informed decision support and reasoning for actions.
  • Over 36% average improvement on profit was achieved across 6 financial datasets and 6 financial metrics against 12 state-of-the-art baselines.

2 RELATED WORK

Prior work spans financial prediction, reinforcement learning, LLM agents, and tool augmentation, but text dependence and limited multimodal trading capabilities leave a gap that FinAgent targets.

  • LLM research expanded from language processing toward agents with reasoning, planning, memory, collaboration, and complex task capabilities.
  • Tool-augmented language models retrieve current information and address reasoning tasks requiring external tools.
  • Text dependence limits many LLM agents’ perception and interaction with their environments, motivating multimodal agents.
  • Most finance-focused LLMs emphasize NLP, while FinAgent targets multimodal, tool-augmented financial trading.
  • Financial AI research includes recurrent, graph-based, news-informed, ensemble, and reinforcement-learning methods for trading and prediction.

3 PROBLEM FORMULATION

The paper formulates financial trading as an MDP and extends the reinforcement-learning pipeline with multimodal LLM reasoning, memory, tools, and specialized modules.

  • 3.1 Financial Trading as MDP: Financial trading is modeled as an MDP in which an investor interacts with a market through discrete investment actions to earn rewards.
  • 3.1 Financial Trading as MDP: The state can include prices, order-book information, technical indicators, trend predictions, news, expert behavior, and overall market status.
  • 3.1 Financial Trading as MDP: The single-asset action space contains buy, sell, and hold choices, with rewards based on changes in market capital.
  • 3.2 Problem Formulation: FinAgent extends the reinforcement-learning framework by integrating multimodal LLMs and independently defining, learning, and applying reasoning processes.
  • 3.2 Problem Formulation: At each step, FinAgent uses a task-relevant prompt generator, passes the prompt to a multimodal LLM, and parses the response into environment-compatible actions.
  • 3.2 Problem Formulation: FinAgent comprises market intelligence, memory, low-level reflection, high-level reflection, and decision-making modules.
  • 3.2 Problem Formulation: The formulation distinguishes memory for the modules and uses Kline charts, trading charts, and task-specific prompt generators.
  • 3.2 Problem Formulation: The overall objective is to find policies that optimize total discounted returns using memory, augmented tools, and designed modules.

4 FINAGENT FRAMEWORK

FinAgent combines multimodal market intelligence, diversified memory retrieval, dual-level reflection, and tool-augmented decision-making for financial trading. Its modules organize current and historical information into analyses, reflections, and final investment decisions.

  • Framework overview: The framework comprises market intelligence, memory, low-level reflection, high-level reflection, and decision-making modules.The memory module supports the other modules through storage and vector retrieval, while decision-making integrates their information with augmented tools and trader preferences.
  • Market intelligence: Market intelligence collects, summarizes, and analyzes stock news, prices, financial reports, and other market information.The module uses both latest and historical information to extract insights for trading decisions.
  • Memory and retrieval: Diversified retrieval assigns multiple retrieval types to historical information, enabling targeted searches across distinct market perspectives.Retrieving the top K items separately for M retrieval types forms an M × K combination of historical market intelligence.
  • Memory and retrieval: The memory mechanism uses vector storage for market intelligence, low-level reflection, and high-level reflection memories.Summarization creates query text fields that support later memory storage and retrieval.
  • Reflection: Low-level reflection connects observations such as news, reports, Kline charts, and indicators with price movements, while high-level reflection evaluates past decisions and recommends future actions.The framework separates immediate market-price reasoning from higher-level assessment of trading outcomes.

5 EXPERIMENT SETUP

The experiments evaluate FinAgent on six real-world datasets using multimodal asset data and compare it with conventional, machine-learning, reinforcement-learning, and LLM-based baselines. Performance is assessed with six financial metrics under a common trading environment.

  • 5.1 Datasets: The evaluation covers six real-world datasets: five US stock datasets and one cryptocurrency dataset.Each dataset includes daily asset prices, visual Kline and trading charts, and daily news coverage from multiple sources.
  • 5.1 Datasets: The assets comprise AAPL, AMZN, GOOGL, MSFT, TSLA, and ETHUSD, spanning stocks and cryptocurrency.The selection represents multiple market sectors and is used to assess versatility and consistency across financial assets.
  • 5.2 Evaluation Metrics: Performance is measured with ARR, SR, CR, SoR, MDD, and VOL, covering profit, risk-adjusted profit, and risk.ARR is annualized average return; SR, CR, and SoR are risk-adjusted measures, while MDD and VOL quantify risk-related properties.
  • 5.3 Baselines: FinAgent is compared with four rule-based strategies, three ML/DL models, three reinforcement-learning algorithms, and two LLM-based systems.The baselines include B&H, MACD, KDJ&RSI, ZMR, LGBM, LSTM, Transformer, SAC, PPO, DQN, FinGPT, and FinMem.
  • Experimental protocol: All benchmarks are trained and evaluated in the same reinforcement-learning environment, while FinAgent can operate without a GPU.The reported benchmark experiments used a single NVIDIA RTX A6000 GPU for benchmark methods.

6 EXPERIMENTAL RESULTS

FinAgent outperforms baseline methods across financial metrics, with particularly strong profitability and cumulative-return gains. Its performance is generally robust across assets, but stock-specialized auxiliary agents limit cryptocurrency performance.

  • FinAgent significantly outperforms 9 baseline methods across 6 financial metrics, with especially large improvements in profitability.The comparison covers five stocks and cryptocurrency datasets.
  • At least 10% ARR% and 19% SR enhancements are reported across five stocks versus the best-performing baseline.On TSLA, the improvements reach 84% for ARR% and 118% for SR.
  • FinAgent consistently outperforms the broader market in profitability across datasets, while rule-based methods control risk more effectively than they capture returns.The paper attributes FinAgent’s stronger returns partly to accepting slightly higher risk and balancing risk and reward.
  • FinAgent surpasses other methods in cumulative returns, particularly on TSLA, where a short position hedges losses after anticipating a price drop.The reported trading example links market intelligence and reflection to the short-position decision.
  • FinAgent yields slightly lower returns than FinMem on ETH because its auxiliary agents are specialized for stocks rather than higher-frequency cryptocurrency trading.The paper reports that a generalized cryptocurrency auxiliary agent could potentially raise returns from 44% to 54%.

7 ABLATION STUDIES

Ablation studies show that reflection and diversified retrieval improve trading performance, while stock-specific rule-based auxiliary agents can harm cryptocurrency results. The component effects therefore depend on the asset domain.

  • Effectiveness of Each Component (RQ2): Adding low-level reflection increases ARR% by 45% to 101% for TSLA and ETHUSD while reducing risk by 14% to 44%.The comparison is against market intelligence alone and the M-plus-L configuration.
  • Effectiveness of Each Component (RQ2): Adding high-level reflection to market intelligence and low-level reflection significantly improves ARR% and SR while reducing risk.For TSLA, this improvement includes a minor 7% rise in MDD%.
  • Effectiveness of Augmented Tools (RQ3): Adding stock-specialized rule-based auxiliary agents causes cryptocurrency performance to drop by over 20%.The ablation identifies specialization mismatch as the reason these agents help stocks but hurt ETH.
  • Effectiveness of Augmented Tools (RQ3): In ETHUSD, the tools-only method achieves 16% ARR% versus 29% for the B&H strategy, showing the cost of stock-specific auxiliary methods.The paper advises selecting auxiliary agents according to market characteristics rather than adding them indiscriminately.
  • Effectiveness of Diversified Retrieval (RQ4): Diversified retrieval improves ARR and SR on AAPL, while t-SNE separates embeddings associated with different retrieval types.The visualization compares FinAgent with and without diversified retrieval and examines retrieved market intelligence on the validation set.

8 CONCLUSION AND FUTURE WORK

The paper presents FinAgent as a multimodal, tool-augmented trading agent designed for reasoning, generalizability, and analysis across market dynamics and historical trading behavior. Its architecture combines multimodal understanding, reflection, retrieval, and auxiliary tools.

  • FinAgent integrates textual and visual data to understand market dynamics and historical trading behaviors.The conclusion characterizes it as an LLM-powered multimodal agent.
  • FinAgent independently leverages auxiliary tools for detailed market-data analysis across different time scales.The conclusion also emphasizes diverse retrieval and multiple perspectives.
  • Its multi-perspective and diverse retrieval approach contributes to the agent’s generalist trading design.The supplied conclusion passage introduces this approach but does not provide a quantitative result here.

B DETAILS OF DATASETS AND PROCESSING

The evaluation uses multimodal financial datasets spanning stocks, foreign exchange, and cryptocurrency, with numerical, textual, and visual information. Data are split into training and testing periods and compared against baseline methods using financial metrics.

  • The evaluation covers 6 real-world datasets, including five US stock datasets and one cryptocurrency dataset.The datasets combine multiple forms of information from various sources.
  • The datasets contain day-level asset prices, historical Kline and trading charts, news, and expert analysis.Price fields include open, high, low, close, and adjusted close values.
  • The asset portfolio includes AAPL, AMZN, GOOGL, MSFT, TSLA, a foreign-exchange pair, and Ethereum.The selection is intended to assess versatility across different financial assets and market sectors.
  • Training uses data from 2022-06-01 to 2023-06-01, while testing uses data from 2023-06-01 to 2024-01-01.The latter half of the year is allocated for testing and the penultimate year for training.
  • FinAgent is compared with 9 baseline methods using 6 financial metrics, with results organized in Table 7 and Figure 4.The comparison reports significant gains, especially in profitability.

D DETAILS OF BENCHMARK METHODS

The paper details conventional and advanced trading benchmarks alongside FinAgent’s modular workflow for multimodal analysis, reflection, retrieval, and decision-making. Case studies illustrate how the agent combines market signals, historical reflections, strategies, and financial constraints when selecting actions.

  • Benchmark methods: The benchmark set includes rule-based strategies, reinforcement-learning methods, and LLM-based systems for trading-performance comparison.The listed methods include B&H, MACD, KDJ&RSI, ZMR, SAC, PPO, DQN, FinGPT, and FinMem.
  • Workflow and implementation: The workflow retrieves and stores market intelligence and reflections at multiple levels before producing a trading action and reasoning.The implementation includes latest and past market-intelligence retrieval, low-level and high-level reflection retrieval, memory updates, and decision output extraction.
  • Workflow and implementation: FinAgent’s prompts are modularized for latest and past market intelligence, low- and high-level reflection, and decision-making.The templates use parameter placeholders and XML parsing because the authors report frequent formatting errors with strict JSON.
  • Market analysis inputs: The agent’s visual inputs include Kline charts with technical indicators, historical trading curves, and cumulative-return curves.Auxiliary strategies provide explanations that support BUY, HOLD, and SELL decisions.
  • Decision-making examples: FinAgent’s decisions synthesize market intelligence, price trends, expert investment advice, historical trading patterns, and risk management.The case studies describe both anticipatory position adjustment and liquidity-aware selling under immediate uncertainty.
  • Decision-making examples: A BUY decision is supported by positive medium-term sentiment, gains across short-, medium-, and long-term horizons, and available cash without an existing position.The reasoning also considers negative news, expert guidance, trading strategies, historical performance, and innovation.

G.2 Case Study: II

FinAgent’s case studies show trading decisions grounded in multimodal market analysis, historical reflections, technical indicators, financial constraints, and short- versus long-term outlooks. The examples emphasize strategic foresight, adaptability, liquidity management, and capital protection.

  • Decision rationale: AAPL SELL decisions prioritize immediate negative factors, limited cash reserves, liquidity, and risk mitigation despite technical strength and longer-term bullish signals.The absence of a clear trading-strategy signal further supports the reported SELL decision.
  • Decision rationale: Alphabet BUY decisions reflect positive long-term cloud and AI prospects, investor confidence, bullish momentum, and lessons from past trades.Short-term concerns are considered but do not outweigh the reported long-term opportunities and upward signals.
  • Risk management: SELL decisions can seek to secure profits and improve liquidity before potential downturns, while incorporating reflections to avoid overly conservative timing.The examples describe a forward-looking approach to market uncertainty and future buying opportunities.
  • Strategic foresight: BUY decisions may favor long-term opportunities over temporary weakness when financial flexibility and positive market signals support expected growth.One example links AI advances and regulatory relaxation with a long-term opportunity despite short-term challenges.
  • Overall assessment: The case studies characterize FinAgent as balancing strategic foresight, adaptability, proactive risk management, and disciplined use of past trading reflections.The reported decisions aim to capitalize on opportunities while navigating potential risks.
  • Decision rationale: FinAgent integrates market intelligence, historical trends, technical indicators, risk management, and financial constraints when selecting BUY or SELL actions.The examples combine sentiment, price movements, MA5, Bollinger Bands, liquidity, and prior trading reflections.

H.2 Analysis of Using FinAgent as a router for technical indicators-driven strategies for trading.

This experiment evaluates FinAgent’s decision module as a router among technical-indicator trading strategies, without market intelligence, memory, or reflection. The router generally produces positive, less-extreme outcomes but remains constrained by strategy quality, conflicting signals, and prompt-related randomness.

  • Experimental setting: The router experiment uses only FinAgent’s decision module, excluding market intelligence, memory, and reflection, so no FinAgent training phase is required.The underlying strategies were fine-tuned with Optuna on designated training data, and their effectiveness was provided to the LLM.
  • Router performance: FinAgent’s router consistently delivers robust and positive trading results, although it does not always match the optimal individual strategy.Across assets, its behavior is described as less extreme than relying on a single strategy.
  • Limitations: The router’s variable strategy preferences can create unprofitable sequential decisions when different strategies issue conflicting signals in the same period.In the cited ETHUSD period, buy decisions were mainly influenced by KDJ&RSI and SO&BB, while ZMR primarily guided sell decisions.
  • Limitations: Router performance is constrained when all underlying strategies are weak, and excluding reflection limits decision-making capability in those periods.The cited AAPL period reports no profitable strategy among the evaluated alternatives.
  • Limitations: Trading decisions vary with prompt changes and repeated trials because OpenAI API response randomness cannot be controlled, reducing robustness of the decision-only setup.The paper identifies prompt engineering and response randomness as an additional risk factor.

H.3.1 Analysis of PPO for trading.

PPO’s conservative behavior can produce prolonged inactivity and poorly timed trades, while DQN also struggles to balance risk and reward in changing markets.

  • PPO limitations: PPO may remain idle during declining cumulative returns, missing profitable opportunities in rapidly changing markets.Its cautious learning process prioritizes risk avoidance over potential gains.
  • PPO limitations: PPO can buy at relatively high prices because it struggles to predict short-term price movements and identify optimal entry timing.Figure 22 describes a missed low-value point on September 27, 2023.
  • PPO improvements: PPO requires parameter tuning, training optimization, and potentially complementary features such as sentiment analysis and macroeconomic indicators.Regular monitoring and adjustment are also proposed for changing market conditions.
  • DQN limitations: DQN may adopt a conservative stance that misses bullish-market gains and holds depreciating assets during bearish conditions.This reflects difficulty balancing risk and reward effectively.
  • DQN improvements: DQN refinement can incorporate advanced indicators, real-time data, diverse market data, and risk-management strategies.Regular updates with current market data and trends are described as essential.

H.3.3 Analysis of SAC Algorithm for Stock Trading.

SAC benefits from entropy-based exploration but remains vulnerable to conservative and poorly timed decisions in volatile cryptocurrency and stock markets.

  • SAC strengths: SAC’s entropy regularization helps it explore unknown states and formulate strategies through environmental interaction.The objective is to maximize expected cumulative reward.
  • Cryptocurrency trading: In ETHUSD trading, SAC made only two buys during a bullish phase, missing significant market movements because of long-term observation.The cautious approach avoided potential losses but limited participation in profitable trends.
  • Stock trading: SAC’s stock-trading performance varies by asset, with suboptimal entry and exit timing reported for AMZN and MSFT.The examples involve selling before later peaks or while bullish trends continued.
  • Overall assessment: SAC requires careful tuning and enhancement to better leverage its framework in dynamic and volatile markets.The paper identifies this need despite SAC’s robust reinforcement-learning foundation.

H.4 Analysis of FinGPT for Trading

FinGPT exhibits several trading limitations, including missed price peaks, incorrect forecasts of sharp movements, and a mismatch between predictions and timely actions.

  • Decision limitations: FinGPT may continue holding after a stock reaches a peak, missing opportunities to maximize returns.One example describes selling at $336.37 after a $358 peak, following continued holding.
  • Prediction errors: FinGPT can incorrectly predict price increases before sharp declines, undermining its reliability for investment guidance.The TSLA example describes a recommendation at $245.34 followed by a decline to $215.49.
  • Suggested improvement: The reported TSLA failure motivates continued refinement of FinGPT’s predictive algorithms for changing financial conditions.The paper links refinement to improving predictive accuracy and investment-guidance reliability.
  • Prediction errors: FinGPT can incorrectly predict decreases and recommend selling even when stock prices continue rising.This inconsistency raises concerns about actionable alignment with market behavior.
  • Decision limitations: FinGPT’s reasoning can mismatch its actions by recommending prolonged holding despite successful price predictions.The model may therefore fail to provide timely and proactive investment strategies.
  • Reasoning limitations: After three consecutive wrong predictions, FinGPT may generate generic explanations with little value to investors.The responses can state that exact price movements are difficult to predict while offering vague positive and negative factors.

H.5 Analysis of FinMem for Trading

FinMem retrieves historical news and reports through layered memory and uses reflection, but its trading decisions can misread mixed signals, ignore cash constraints, and become unstable.

  • Decision process: FinMem uses layered memory to retrieve historical news and financial reports, then reflects on their possible links to future price movements.The analysis examines both its decisions and the reasoning generated during decision-making.
  • AAPL analysis: FinMem bought AAPL despite a broader downward trend after overweighting two positive news items among mostly negative reports.The decision shifted returns from 6% to -3%.
  • Portfolio constraints: FinMem may issue further buy decisions despite insufficient cash to purchase more shares when buying means going all in.The example indicates weak awareness of both the environment and its current financial state.
  • Decision stability: FinMem’s decisions can become unstable, including repeated buy-then-sell and sell-then-buy operations.Such instability is described as potentially costly in highly volatile markets.
Loading 2402.18485v3…