Source-linked AI summary
A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
Zhengyao Jiang, Dixing Xu, Jinjun Liang
TL;DR
Portfolio management requires repeated allocation decisions across multiple assets, while discrete or price-prediction-based approaches have limited applicability. The paper develops a model-free deep RL framework combining EIIE, portfolio-vector memory, online stochastic batch learning, and an explicit reward. Across three cryptocurrency back-tests, its CNN, basic RNN, and LSTM versions outperform compared strategies, with reported scope limitations in market assumptions and evaluation coverage.
Problem
Discrete single-asset RL signals and price-prediction approaches do not directly solve continuous multi-asset portfolio management.
Method
The framework uses EIIE to output portfolio weights, PVM to incorporate prior weights and transaction costs, OSBL for training, and an explicit logarithmic-return reward.
Results
The three EIIE networks occupied the top three positions in risk-adjusted score across all three tests and outperformed compared algorithms in final accumulated portfolio value.
Takeaways & Limitations
The framework provides an extensible deep RL approach for multi-asset portfolio management, realized with CNN, basic RNN, and LSTM networks.
Takeaways & Limitations
The work assumes zero market impact and zero slippage and tests adaptability in only one market.
Abstract
from arXiv · showhide
Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along with a number of recently reviewed or published portfolio-selection strategies, examined in three back-test experiments with a trading period of 30 minutes in a cryptocurrency market. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. All three instances of the framework monopolize the top three positions in all experiments, outdistancing other compared trading algorithms. Although with a high commission rate of 0.25% in the backtests, the framework is able to achieve at least 4-fold returns in 50 days.
1. Introduction
The paper frames portfolio management as continuous fund reallocation under uncertain markets and motivates a model-free RL approach that directly manages multiple assets. Its EIIE framework incorporates portfolio memory, online learning, and explicit rewards, then outperforms compared strategies in cryptocurrency back-tests.
- Traditional portfolio methods rely on financial models, whose performance depends on their validity across markets.
- Price-prediction trading methods depend on difficult-to-achieve future-price accuracy and require an additional logic layer to convert predictions into market actions.
- Existing model-free deep RL trading algorithms generally output discrete single-asset signals, limiting their applicability to multi-asset portfolio management.
- The EIIE framework evaluates each asset’s potential growth, adjusts scores by intentional weight changes, and uses softmax outputs as portfolio weights and market actions.
- The framework combines Portfolio-Vector Memory, Online Stochastic Batch Learning, and an explicit average logarithmic-return reward function.
- Three EIIE variants were tested at 30-minute intervals in a cryptocurrency exchange market, and they significantly beat compared portfolio strategies in all three experiments.
2. Problem Definition
The paper formalizes portfolio management as periodic reallocation among assets, with returns determined by price relatives and portfolio weights. Transaction costs shrink portfolio value through a transaction remainder factor, while back-tests assume zero slippage and zero market impact.
- Trading Period: The agent reallocates capital among assets at equal-length trading periods, using T = 30 minutes in all experiments.
- Mathematical Formalism: The portfolio contains asset prices and weights, with the first asset serving as cash and Bitcoin used as cash in the experiments.
- Mathematical Formalism: Price relatives are element-wise ratios of consecutive price vectors and determine period-wise portfolio value changes through portfolio weights.
- Objective: The manager maximizes final portfolio value, equivalently maximizing average logarithmic cumulated return when initial investment and process length are fixed.
- Transaction Cost: Reallocation from post-market weights to target weights incurs commissions represented by a transaction remainder factor µt ∈ (0, 1].
- Transaction Cost: The transaction remainder factor is solved iteratively because it appears inside a rectified linear function, with approximations controlled by convergence tolerance or iteration count.
- Transaction Cost: The experiments use a constant 0.25% commission rate for both selling and purchasing non-cash assets.
- Back-Test Assumptions: Back-tests assume trades execute at the last price without slippage and that the agent’s capital has no market impact.
3. Data Treatments
The experiments preselect liquid cryptocurrency assets, normalize historical price inputs, and construct a tensor for neural-network policies. Missing pre-training histories are explicitly handled, while selection based on recent volume is designed to avoid survival bias.
- Asset Pre-Selection: The experiments use Poloniex data and preselect the 11 highest-volume non-cash assets plus Bitcoin cash, producing a 12-asset portfolio.
- Asset Pre-Selection: Higher trading volume is used as a proxy for liquidity and for reducing the agent’s potential market influence under the framework’s assumptions.
- Asset Pre-Selection: Using future volume rankings could create survival bias and unreliable positive results, so rankings immediately before each back-test are used instead.
- Price Tensor: Historical prices are fed into the network as a rank-3 tensor containing three normalized feature matrices over n = 50 prior periods and m non-cash assets.
- Price Tensor: The policy maps the price tensor and previous portfolio vector to current weights, whose next-period logarithmic return becomes the immediate RL reward.
- Missing Data: Some selected coins lack earlier history, creating NANs in training data that must be replaced before neural-network processing.
4. Reinforcement Learning
The paper casts portfolio management as an RL problem whose state combines market-price history with the previous portfolio vector. A deterministic policy is optimized by gradient ascent using an explicit, batch-compatible logarithmic-return reward.
- RL Formulation: The RL agent is a software portfolio manager acting in a financial-market environment whose complete state information is unavailable.
- State: The state combines the external price tensor Xt with the previous portfolio vector wt−1, which carries transaction-cost-relevant internal information.
- Assumptions: The framework assumes portfolio actions do not influence future price states, while actions affect subsequent rewards through portfolio reallocation.
- Objective: The agent’s objective of maximizing final wealth is equivalent to maximizing average logarithmic cumulated return.
- Reward: The reward uses an explicit logarithmic return normalized by episode length, enabling fair mini-batch training across runs of different lengths.
- Reward: Exact episodic and cumulated rewards allow the same market-history segment to evaluate different action sequences, reducing the need for new trials.
- Optimization: The deterministic policy maps states to continuous portfolio actions and is optimized through gradient ascent without exploration.
- Optimization: Mini-batch gradient updates improve training efficiency and support online learning as new market history becomes available.
5. Policy Networks
The paper builds portfolio-policy networks from shared-parameter asset-specific evaluators, augmented with portfolio-vector memory and online stochastic batch learning. CNN, basic RNN, and LSTM realizations support portfolio outputs while addressing transaction costs, sequential data, and ongoing market updates.
- Network architectures: CNN, basic RNN, and LSTM networks map price tensors to portfolio vectors for the policy functions.The CNN and recurrent implementations use the same portfolio-management task but differ in their asset-processing subnets.
- Network architectures: EIIE processes the m assets through independent streams with shared parameters, connecting them only at the softmax layer.Each stream evaluates an individual non-cash asset, while softmax produces non-negative weights summing to unity.
- Network architectures: EIIE improves performance by allowing evaluators to judge assets from recent events without exposing their identities.This contrasts with an integrated network that may remain reluctant to invest in assets with historically unfavorable performance.
- Network architectures: EIIE training scales roughly linearly with m, reuses an interval of price history across assets, and supports real-time asset-collection changes without retraining from scratch.Shared parameters provide scalability, data-usage efficiency, and plasticity to the portfolio’s asset collection.
- Portfolio-Vector Memory: Portfolio-Vector Memory stores portfolio vectors chronologically, reading the previous period’s vector and overwriting the current period with the network output.The memory is initialized with uniform weights and supplies previous portfolio weights for training and transaction-cost-aware decisions.
- Online training: The memory enables simultaneous mini-batch training, while batches remain ordered in time because the learning framework uses sequential inputs.This makes mini-batch training plausible without treating time-series observations as unordered supervised-learning samples.
6. Experiments
The paper evaluates three EIIE policy networks against neural, benchmark, and traditional portfolio-selection strategies across three cryptocurrency back-tests, using normalized portfolio value and risk-sensitive metrics. The EIIE networks lead on profitability and Sharpe ratio, while results also show sustained market outperformance despite high trading costs.
- Experimental setup: Three back-tests compare the EIIE policy networks with an integrated CNN, traditional portfolio-selection strategies, and benchmarks on Poloniex cryptocurrency data.The experiments use 30-minute price periods and report final accumulated portfolio value, Sharpe ratio, and maximum drawdown.
- Performance measures: Normalized accumulated portfolio value sets the initial portfolio value to p0 = 1, enabling comparisons across management runs with different starting values.The final APV is measured at the end of each back-test and is closely related to accumulated return.
- Performance measures: Sharpe ratio adjusts mean return for deviation, while maximum drawdown measures the biggest loss from a peak to a trough.Maximum drawdown emphasizes downside movement, which Sharpe ratio treats symmetrically with upward volatility.
- Results: The CNN EIIE is the best algorithm by fAPV or SR in Back-Tests 1 and 2, with final wealth more than twice the runner-up in the first experiment.Across all back-tests, the three EIIEs occupy the top three positions for fAPV and SR, losing only on MDD.
- Results: All three EIIEs outperform the best asset by fAPV in all three back-tests and achieve at least 4-fold returns in 20 days under different market conditions.The experiments use a 0.25% commission rate and relatively high half-hourly trading frequency, conditions under which many traditional strategies perform poorly.
- Results: CNN and basic-RNN EIIEs beat the market throughout all three back-tests, whereas traditional strategies do so only briefly or during the second half of Back-Test 3.The comparison uses APV trajectories over time against Best Stock and UCRP market benchmarks.
7. Conclusion
The framework combines EIIE, PVM, and OSBL components for extensible portfolio management, and its CNN, basic RNN, and LSTM versions outperform comparison algorithms across cryptocurrency back-tests. The experiments also expose performance variation across network types and scope limitations for future work.
- Conclusion: The framework combines the EIIE meta topology, PVM, and OSBL scheme to solve general portfolio management and support extensible, scalable neural-network implementations.The EIIE topology handles multi-channel inputs and outputs portfolio weights, while the PVM incorporates transaction costs and OSBL supports online learning.
- Back-test findings: The EIIEs led throughout Back-Test 1, steadily climbed in the worst experiment, and achieved their best final wealth in Back-Test 3 despite major drawdowns.Back-Test 3’s dips contributed to high Maximum Drawdown, while Back-Test 1 showed only a few drawdown incidents.
- Conclusion: The CNN, basic RNN, and LSTM EIIE versions achieved higher final accumulated portfolio values than the other compared trading algorithms in all three cryptocurrency back-tests.The EIIE networks also occupied the top three positions in risk-adjusted score across all three tests.
- Network comparison: The LSTM had much lower scores than the CNN and basic RNN, possibly because the same structural hyper-parameters were used for both recurrent networks.The authors also suggest that the vanilla RNN may better exploit repetitive price patterns because it is not designed to forget input history.
- Limitations: The back-tests assume zero market impact and zero slippage and evaluate the framework in only one market.The authors identify broader-market testing, real-world trading data, and reward-function changes for longer-term reactions as future needs.
Appendix A. Proof of Theorem 1
The proof of Theorem 1 is prepared through five lemmas, including monotonicity of the function f(µ). This monotonicity follows from the monotonicity of the linear rectifier.
- Appendix A. Proof of Theorem 1: The proof of Theorem 1 is organized around five preliminary lemmas.The appendix introduces these lemmas before presenting the general convergence theorem.
- Appendix A. Proof of Theorem 1: f(µ) is monotonically increasing: if µ2 > µ1, then f(µ2) ⩾ f(µ1).The result is obtained from the monotonicity of the linear rectifier (x)+.
Lemma A.2
Lemma A.2 establishes a positivity property under commission rates below the impractically high threshold of 38%.
- Lemma A.2: For commission rates below 38%, f(0) > 0 always holds.The paper characterizes 38% as an impractically high commission rate.
Lemma A.3
The appendix proves convergence by analyzing cases for the initial value µ⊙ and using monotonicity, boundedness, induction, and the Squeeze Theorem. For every µ⊙∈[0, 1], the sequence ˜µ(k) converges to µt.
- Case analysis: For intermediate initial values, monotonicity and bounds construct sequences that bracket ˜µ(k).The proof uses the monotonicity of f and cases involving inequalities between bµ(j), µ⊙, and bµ(j+1).
- Proof tools: The proof uses mathematical induction and the Monotone Convergence Theorem to establish monotonicity and boundedness in the remaining cases.These arguments provide the bounds needed for convergence.
- Case analysis: The proof handles the cases µ⊙ = µt, 0, or 1 directly, with convergence to µt guaranteed for the latter two values.The appendix identifies the case µ⊙ = µt as the solution of µ = f(µ).
- Case analysis: The bracketing sequences converge through the Squeeze Theorem when no equality terminates the construction earlier.If an equality holds, the sequences coincide from a later index onward and converge to µt.
- Conclusion: For any initial value µ⊙∈[0, 1], the sequence ˜µ(k) converges to µt.The appendix states this as the concluding result of the case analysis.
Appendix B. Hyper-Parameters
Appendix B specifies the framework’s adjustable hyper-parameters and explains how experimental values were selected. Cross-validation was used for tuning while avoiding overlap with back-test periods, and deeper IIEs did not improve scores.
- The geometric-distribution parameter β controls selection of online training sample batches.β is identified as the parameter in Equation 26 of Section 5.3.
- The listed hyper-parameters were chosen using network scores on the cross-validation set, but remain adjustable within the framework.Table B.1 records the values used in the paper’s experiments.
- The cross-validation time range and back-tests do not overlap, a design choice intended to avoid over-fitting.Hyper-parameter values were selected to maximize network scores in the cross-validation range.
- Deeper IIE network structures than those shown in Figures 2 and 3 did not improve cross-validation scores.