Source-linked AI summary
Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States
Yunan Ye, Hengzhi Pei, Boxin Wang, Pin-Yu Chen, Yada Zhu, Jun Xiao, Bo Li
TL;DR
Portfolio management must make sequential allocation decisions despite heterogeneous information, noisy markets, and distribution shifts. SARL augments reinforcement-learning states with asset-movement predictions derived from prices or alternative data such as news. Across Bitcoin and HighTech datasets, the paper reports better portfolio value and Sharpe ratio, with simulations supporting the usefulness of state augmentation.
Problem
Portfolio management must integrate heterogeneous, noisy information while coping with versatile, non-stationary financial environments and difficult price prediction.
Method
SARL encodes asset-movement predictions from internal prices or external news and embeds them as augmented states in a standard reinforcement-learning portfolio-management framework.
Results
SARL achieves significantly better portfolio value and Sharpe ratio on Bitcoin and HighTech datasets, with simulations validating state augmentation.
Takeaways & Limitations
State augmentation supports more generalizable and robust reinforcement-learning portfolio management under data-distribution shift.
Takeaways & Limitations
The framework assumes market data and financial news are available up to the current time, while precise price-change prediction remains difficult under uncertainty and distribution shifts.
Abstract
from arXiv · showhide
Portfolio management (PM) is a fundamental financial planning task that aims to achieve investment goals such as maximal profits or minimal risks. Its decision process involves continuous derivation of valuable information from various data sources and sequential decision optimization, which is a prospective research direction for reinforcement learning (RL). In this paper, we propose SARL, a novel State-Augmented RL framework for PM. Our framework aims to address two unique challenges in financial PM: (1) data heterogeneity -- the collected information for each asset is usually diverse, noisy and imbalanced (e.g., news articles); and (2) environment uncertainty -- the financial market is versatile and non-stationary. To incorporate heterogeneous data and enhance robustness against environment uncertainty, our SARL augments the asset information with their price movement prediction as additional states, where the prediction can be solely based on financial data (e.g., asset prices) or derived from alternative sources such as news. Experiments on two real-world datasets, (i) Bitcoin market and (ii) HighTech stock market with 7-year Reuters news articles, validate the effectiveness of SARL over existing PM approaches, both in terms of accumulated profits and risk-adjusted profits. Moreover, extensive simulations are conducted to demonstrate the importance of our proposed state augmentation, providing new insights and boosting performance significantly over standard RL-based PM method and other baselines.
Introduction
Portfolio management remains difficult because market prices are noisy and traditional methods rely on restrictive assumptions, while RL must handle heterogeneous data and uncertain environments. SARL addresses these challenges by augmenting RL states with asset movement predictions and demonstrates improved performance across datasets and information settings.
- Motivation: Market prices are highly noisy, stochastic, and chaotic, making accurate forecasting difficult for traditional portfolio-management approaches.Markowitz-based methods also assume that past asset-return distributions fully represent the future.
- Motivation: RL is a natural fit for portfolio management because asset allocation requires sequential reallocation decisions based on continually updated information.However, existing RL approaches face heterogeneous inputs and non-stationary market environments.
- Empirical findings: 140.9% improvement over the state-of-the-art method is reported for accumulative return on Bitcoin, while SARL also outperforms portfolio-management baselines on portfolio value and Sharpe ratio.The reported baselines include CRP, OLMAR, and WMAMR.
- Empirical findings: 11.73% and 40.45% improvement in Sharpe Ratio are reported on the Bitcoin and HighTech datasets, respectively.The paper also reports greater generalizability than standard RL-based portfolio-management approaches under data-distribution shift.
- SARL framework: SARL embeds predictions from diverse information sources into augmented RL states for portfolio management.The framework can use structured asset prices or alternative data such as financial news.
- Empirical findings: Simulations indicate that diverse information, high-density noisy signals, and low-density high-accuracy signals can improve portfolio-management performance.These experiments examine how external-information characteristics affect state augmentation.
Related Work
Deep-learning models have been used for market-behavior prediction and asset allocation, but their lack of market interaction limits their suitability for portfolio decision making. Reinforcement learning addresses this interaction-oriented decision process.
- Related Work: Prior neural-network and deep-learning studies have shown effectiveness for asset-price prediction and asset allocation.The cited applications include financial text and market-behavior prediction.
- Related Work: Deep-learning models that do not interact with the market have a natural disadvantage for decision-making problems such as portfolio management.This motivates reinforcement-learning approaches for portfolio decisions.
Background and Problem Formulation
The paper models portfolio management as sequential interaction between an agent and a market environment containing prices and, when available, financial news. It represents allocations, relative prices, and rewards within a Markovian decision process, then augments price states with encoded movement information.
- Problem formulation: The market environment provides asset prices and, when available, financial news, while an agent observes information and rebalances the portfolio through trial and error.The setup assumes sufficient liquidity for immediate execution with minimal market impact.
- Problem formulation: Portfolio management uses historical prices and news up to the current step to predict asset changes or high-level movements for allocation decisions.High-level asset-movement prediction may be more reliable than precise price-change prediction under uncertainty and distribution shifts.
- State representation: The price vector contains closing prices for n assets, with highest and lowest prices also defined for each time step.The assets are not necessarily fully invested in portfolio-management problems.
- State representation: A cash dimension is added to the asset-price vector and remains constant because temporal variations are normalized relative to cash value.The cash component is indexed as the first dimension, v0,t.
- Decision process: The portfolio-management problem is formulated as a Markovian decision process with state set S, action set A, and reward function r : S × A → R.The Markov formulation states that the next state depends only on the current state and action.
- Decision process: Relative prices are formed from consecutive price vectors, and the reallocation weight vector assigns fractions to cash and each asset.The RL agent buys or sells according to changes between successive allocation vectors.
- State augmentation: SARL encodes internal prices or external news into asset-movement information and integrates that representation with asset prices before policy generation.Figure 1 depicts the original price state, encoded external information, augmented state, and portfolio-management strategy.
SARL: A Framework of Deep RL with Augmented Asset Movement Prediction States
SARL combines supervised asset-movement prediction with reinforcement learning by augmenting observable asset-price states with representations from internal prices or external news. It also adapts the portfolio objective and reward to sequential reallocation with transaction costs.
- State augmentation: SARL extracts asset-movement information from historical prices or news, integrates it with asset prices, and trains an RL agent on the augmented state.The framework uses an end-to-end network and can process heterogeneous internal or external information sources.
- State augmentation: The encoder δ summarizes high-level observations, such as movement predictions from past prices or news, while s* represents current asset prices.This separates observable low-level market information from encoded high-level predictions.
- State augmentation: Internal augmentation uses an LSTM to predict binary price movements, whose outputs guide strategy selection alongside raw prices.The framework also supports external financial-news features for movement prediction.
- Objective optimization: Because portfolio assets are repeatedly reallocated, SARL replaces the standard discounted-reward summation with accumulated portfolio value, equivalently maximizing summed logarithmic value.The policy objective is evaluated over the interval from time 1 to T and normalized by T for differing time-period lengths.
- Action and portfolio dynamics: The agent represents portfolio allocations with fraction vectors and adjusts the desired reallocating weights at each time step under a sum-to-one constraint.Price movements influence the subsequent allocation vector through element-wise portfolio updating.
- Action and reward: The immediate reward combines portfolio profit, at · yt, with a transaction-cost penalty based on changes between desired and current allocation weights.The formulation applies a fixed 0.25% commission rate for buying and selling non-cash assets.
Experimental Results
Experiments compare SARL with standard RL and financial portfolio-management baselines across Bitcoin and HighTech data, while testing data imbalance, noise, uncertainty, prediction accuracy, sparsity, and news representations. SARL improves portfolio value and risk-adjusted performance, with state augmentation remaining effective under imperfect or sparse predictions.
- Data Challenges: HighTech news is highly unbalanced: Apple has 33 times more articles than Baidu, and some time steps have no corresponding external feature.The experiments address imbalance alongside noisy external data and market-environment uncertainty.
- Data Challenges: News classifiers reach no more than 66% training and 61% testing accuracy, indicating substantial noise in collected financial news.The study compares multiple word-embedding methods and uses these noisy predictions for state augmentation.
- Portfolio Value: SARL improves portfolio value by 140.9% on Bitcoin and 15.7% on HighTech versus DPM, the state-of-the-art standard RL method.The evaluation uses two real-world datasets from different markets; Bitcoin contains cryptocurrency prices, while HighTech combines stock prices with financial news.
- Sharpe Ratio: SARL attains the highest Sharpe Ratio in most reported testing periods, despite its training objective omitting an explicit risk term.The paper interprets this pattern as evidence that SARL learns risk-adverse strategies and that external features enhance robustness.
- Prediction Accuracy and Sparsity: A 70%-accurate label signal at 50% density improves portfolio value by 90.3% over DPM on Bitcoin.Simulations also show that higher prediction accuracy and higher label density improve SARL’s portfolio-management performance.
- News Representation: Financial-news embeddings outperform randomly simulated labels with similar testing accuracy, revealing exploitable information beyond accuracy alone.This comparison supports SARL’s ability to learn from external information for portfolio management.
Conclusion
SARL is presented as a generic state-augmented RL framework that integrates heterogeneous data sources into standard RL pipelines for portfolio-management strategies. On Bitcoin and HighTech datasets, it achieves better portfolio value and Sharpe ratio, supported by comparative experiments and simulations.
- SARL integrates heterogeneous data sources into standard RL training pipelines for learning portfolio-management strategies.
- SARL achieves significantly better portfolio value and Sharpe ratio on the Bitcoin and HighTech datasets.
- Comparative experiments and extensive simulations validate SARL's superior performance in portfolio management.
Compatibility of SARL
The paper compares different RL algorithms using SARL on Bitcoin and HighTech datasets, and reports Sharpe ratios across multiple time periods for each dataset.
- Figures compare different RL algorithms with the SARL framework on the Bitcoin dataset.
- Figures compare different RL algorithms with the SARL framework on the HighTech dataset.
- Table 4 reports Sharpe ratios for different time periods on the Bitcoin dataset, including weekly and monthly periods.
- Table 5 reports Sharpe ratios for different time periods on the HighTech dataset, including weekly and monthly periods.
Sparsity of HighTech
The HighTech sparsity test evaluates SARL with financial-news embeddings under different label densities. Label density varies across 20%, 50%, 80%, and 100%.
- The sparsity test evaluates SARL on the HighTech dataset using financial-news embeddings with different sparsity levels.