Source-linked AI summary
Cryptocurrency Portfolio Management with Deep Reinforcement Learning
Zhengyao Jiang, Jinjun Liang
TL;DR
The paper addresses portfolio management with model-dependent traditional methods and cryptocurrency-specific market conditions. It proposes a deterministic deep reinforcement learning CNN that maps historical market data to portfolio weights, outperforming most compared strategies on accumulative return while achieving lower risk than PAMR.
Problem
Traditional portfolio methods depend on financial models whose validity may vary across markets, motivating a more extensible portfolio-management approach for cryptocurrency markets.
Method
A deterministic deep reinforcement learning CNN directly produces portfolio weights from historical price data, treating portfolio actions, market states, and capital changes as reinforcement-learning components.
Results
The CNN outperforms most benchmarks and compared algorithms in accumulative return, while achieving lower risk and a higher Sharpe ratio than PAMR.
Takeaways & Limitations
The approach is presented as highly extensible because it does not rely on financial theory and can incorporate risk terms and volumes.
Takeaways & Limitations
The paper’s deterministic formulation directly optimizes the reward without probability techniques, constraining the method to a consistent action model rather than standard probabilistic reinforcement learning.
Abstract
from arXiv · showhide
Portfolio management is the decision-making process of allocating an amount of fund into different financial investment products. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. This paper presents a model-less convolutional neural network with historic prices of a set of financial assets as its input, outputting portfolio weights of the set. The network is trained with 0.7 years' price data from a cryptocurrency exchange. The training is done in a reinforcement manner, maximizing the accumulative return, which is regarded as the reward function of the network. Backtest trading experiments with trading period of 30 minutes is conducted in the same market, achieving 10-fold returns in 1.8 months' periods. Some recently published portfolio selection strategies are also used to perform the same back-tests, whose results are compared with the neural network. The network is not limited to cryptocurrency, but can be applied to any other financial markets.
I. INTRODUCTION
The paper develops a model-free portfolio-management approach that directly outputs portfolio weights from historical market data, avoiding price prediction and hand-designed financial models. It applies deterministic deep reinforcement learning to continuous portfolio actions and evaluates the resulting strategy against existing methods in cryptocurrency back-tests.
- Motivation: Traditional portfolio methods depend on prior financial models, historical-pattern assumptions, or combinations of methods.The paper groups existing approaches into Follow-the-Winner, Follow-the-Loser, Pattern-Matching, and Meta-Learning.
- Motivation: Price-prediction systems require human-designed rules to convert predicted movements into trading actions, whereas the proposed system outputs portfolio vectors directly.The design targets a model-less trading algorithm and focuses on portfolio profit rather than prediction accuracy.
- Problem: Continuous portfolio actions limit standard deep reinforcement-learning methods designed for discrete actions, because discretization can create concentrated and unknown risks.The paper identifies stochastic policy gradients and deep Q-learning as examples of methods limited to discrete actions.
- Approach: The proposed deterministic policy-gradient approach directly optimizes the portfolio-management reward while avoiding Q-function estimation.This approach is presented as the core innovation for handling continuous actions.
- Evaluation: The trading algorithm is back-tested on Poloniex using 30-minute periods and compared with three recent portfolio-selection algorithms.The portfolio contains coins selected by previous trading-volume rankings.
- Evaluation: Cryptocurrency markets provide an experimental setting with decentralized participation, open exchanges, and many small-volume currencies.The paper argues these characteristics make cryptocurrency markets suitable for testing the proposed portfolio-management method.
II. PROBLEM DEFINITION
The problem is formulated as sequential portfolio allocation over asset-price histories. At each period, the agent chooses portfolio proportions whose returns determine portfolio-value changes, with transaction fees included in the setting.
- Market representation: The global price matrix G contains price sequences for m selected assets across n trading periods.Each row represents one asset’s price sequence, and each column represents a period’s price vector.
- Market representation: The price-change vector y_t is obtained by element-wise dividing the next-period price vector by the current-period price vector.This vector represents the relative price changes used to evaluate each period’s allocation.
- Portfolio action: The portfolio vector ω_t specifies the proportion of total capital invested in each asset, with proportions summing to one.The initial portfolio places all capital in the riskless asset or fiat currency, represented by Bitcoin in the experiment.
- Reward and value: Ignoring transaction fees, the capital change rate r_t equals the dot product of the current portfolio vector and the next period’s price-change vector.This rate is the next period’s total capital divided by the current period’s total capital.
- Reward and value: With commission rate C, transaction fees are added to the trading-period calculation; the experiment uses C = 0.0025.The stated rate is the maximum commission rate at Poloniex.
- Reward and value: The agent generates a sequence of portfolio vectors to maximize accumulative capital, and portfolio value after n periods equals initial value plus total return.The value is normalized so that α_0 = 1.
B. Two Hypothesises
The evaluation uses historical back-testing under assumptions that trades execute immediately at the last quoted price and do not materially affect the market. The CNN receives fixed windows of historical prices and next-period price changes for training and evaluation.
- Back-test assumptions: Back-testing simulates trading from historical points without access to future market information.The agent pretends to operate at a past market time and performs paper trading from that point onward.
- Back-test assumptions: Each trade is assumed to execute immediately at the last price when orders are placed.This is the market-liquidity assumption used in the experiment.
- Back-test assumptions: The invested capital is assumed to be too small to influence the market.This capital-impact assumption supports treating the historical prices as unaffected by the strategy.
- Data and inputs: The data span one year, with a trading period of 30 minutes, and are organized into a global price matrix.The same historical data structure supports the back-test and model inputs.
- Data and inputs: The CNN input is an m × w matrix of coin-price sequences from the previous w trading periods; the experiment uses m = 12 and w = 50.Each row contains one coin’s history within the trading window.
- Data and inputs: Next-period price-change vectors y_t are also required for training and performance evaluation.They define the period-level price changes used by the reward formulation.
A. Coin Selection
The strategy selects assets using trading volume and normalizes prices before feeding them to the network. The selection procedure addresses liquidity and market-impact assumptions while recognizing potential information leakage and market instability.
- Coin selection: The strategy selects the 12 highest-volume assets from roughly 220 cryptocurrencies available on Poloniex.Higher volume is used as a proxy for better liquidity and lower strategy-induced market impact.
- Coin selection: Longer volume windows may be preferable because cryptocurrency trading volumes can change sharply over short periods.The paper specifically contrasts several-day windows with a single 30-minute period.
- Coin selection: Selecting assets by current volume can create a volume-prediction problem by exposing the agent to future test-set information.The paper states that this leakage can influence final trading performance.
- Coin selection: Volume rankings are based on the average volume during the 30 days before each back-test time slot.This procedure is used instead of ranking assets by current-time volume.
- Price normalization: Input prices are normalized because the agent uses price changes rather than absolute price levels for trading decisions.The normalized local price matrix is constructed over the input window before entering the neural network.
2) Filling Empty History Data:
Missing cryptocurrency history is represented numerically so the CNN can process it, while the preprocessing reflects that nonexistent assets should not receive investment during training.
- Recently launched coins have missing pre-launch history, which is marked as NAN in the training data.
- NAN values are replaced with 1 because the CNN requires real-number inputs.
- The preprocessing treats pre-launch periods as unsuitable for investment, while Bitcoin provides a riskless asset in the asset set.
C. Dividing Data into Three Sets
The study separates price data for training, testing, and hyperparameter tuning, then frames portfolio management as reinforcement learning over market states, portfolio actions, and returns.
- C. Dividing Data into Three Sets: The global price matrix is split into training, test, and cross-validation sets in a 0.7 : 0.15 : 0.15 ratio.
- C. Dividing Data into Three Sets: Training tunes network weights, testing evaluates final performance against other algorithms, and cross-validation tunes hyperparameters.
- The history price matrix represents the market state, the portfolio vector is the action, and total capital change is the reward.
- The network directly outputs portfolio weights rather than predicted prices or action probabilities for discrete cases.
- The algorithm seeks to maximize portfolio value through its reward function.
- Transaction costs are omitted from the reward because the input excludes the previous period’s portfolio vector, and returns are averaged logarithmically.
C. Advantages and Limitations
The approach directly optimizes rewards for continuous portfolio actions and leverages action-independent state transitions, but training is sensitive to initialization and uses regularization against overfitting.
- C. Advantages: Direct reward optimization avoids probability techniques, reduces the high-variance problem, and supports a consistent action model.
- C. Advantages: The method can accommodate added risk terms and trading volumes, providing extensibility compared with prediction-based methods.
- C. Advantages: Assuming the agent’s investment does not affect asset prices, actions do not influence the next market state, enabling supervised-learning-style training techniques.
- C. Advantages: Network weights are tuned with gradient-based methods to maximize the reward on the training set.
- C. Advantages: Action-independent inputs permit shuffled mini-batch training to accelerate computation.
- C. Limitations: Performance varies substantially across initializations, indicating that training can end in different local minima.
VI. NETWORK TOPOLOGY
Model selection favored a compact CNN that maps 12-coin price histories to portfolio weights, although its training results were initialization-sensitive and deeper networks did not improve performance.
- VI. NETWORK TOPOLOGY: The selected model uses multiple initializations and cross-validation reward to choose hyperparameters and topology.
- VI. NETWORK TOPOLOGY: The best CNN contains one convolution layer and one fully-connected layer.
- VI. NETWORK TOPOLOGY: CNN training trials show high standard deviation and a large maximum–minimum gap, making final performance sensitive to initial weights.
- VI. NETWORK TOPOLOGY: Fully-connected networks have smaller performance variation but lower best scores than CNNs.
- VI. NETWORK TOPOLOGY: The CNN uses a 12 × 4 filter with 12 filters, no pooling, and preserves location information in the price matrix.
- VI. NETWORK TOPOLOGY: Deeper architectures did not outperform the selected topology, possibly because noisy price information and only 12,000 training points promote overfitting.
- VI. NETWORK TOPOLOGY: The network takes 12×50 matrices representing 12 coins over 50 trading periods and outputs a 12-neuron portfolio vector.
C. Fully Connected Network
The fully connected network produced more stable performance than the CNN, but its best performance was worse. Experiments used rolling time spans, risk-adjusted metrics, and comparisons showing the CNN generally outperformed alternatives.
- The fully connected network was more stable than the CNN, but its best performance was worse.
- Three rolling global price-matrix time spans were used to train, select, and backtest networks against benchmarks and portfolio algorithms.The backtests covered 2016/03/14–2016/05/03, 2016/04/14–2016/06/03, and 2016/05/14–2016/07/03.
- Sharpe ratio and maximum drawdown complemented final portfolio value and return standard deviation as evaluation measures.Sharpe ratio measures risk-adjusted return, while maximum drawdown measures the maximum loss from peak to trough before a new peak.
- The CNN agent outperformed most benchmarks and compared algorithms in accumulative return, losing only to Passive Aggressive Mean Reversion.Despite lower accumulative return than PAMR, the CNN achieved significantly lower risk and a higher Sharpe ratio.
- CNN performance was better when the backtest was closer to the training set.The authors suggest placing training closer to current time or training online for real trading.
C. Dilemma Between Performance Evaluation and Hyperparameters tuning
The paper presents deterministic deep reinforcement learning that directly outputs portfolio weights from historic prices, while evaluation and tuning depend strongly on the temporal placement of training data. The authors identify model-selection and data limitations that constrain real-market application.
- C. Dilemma Between Performance Evaluation and Hyperparameters tuning: The network’s performance strongly depended on the time location of the training dataset.
- C. Dilemma Between Performance Evaluation and Hyperparameters tuning: Short-term market patterns can become overfitting factors when the target set is far from the training set.The passage contrasts short-term patterns with longer-lasting patterns across different temporal separations.
- C. Dilemma Between Performance Evaluation and Hyperparameters tuning: Selecting hyperparameters on a more distant set may suppress overfitting while overlooking short-term patterns useful in testing.
- VIII. CONCLUSION: The proposed method directly produces portfolio vector ω from historic prices using deterministic deep reinforcement learning.The approach does not rely on financial theory and is described as highly extensible.
- VIII. CONCLUSION: The cryptocurrency backtest compared the CNN strategy with three benchmarks and three portfolio-management algorithms, reporting positive results but lower cumulative return than PAMR.
- VIII. CONCLUSION: The study’s major limitation is that historical training and testing cannot completely simulate real online trading, and its cross-validation placement complicates real-market model selection.
- VIII. CONCLUSION: The training set was small and the market sample limited, making it difficult to build a deeper network structure.
APPENDIX
The appendix identifies an additional back-test time slot and lists the CNN agent's hyperparameter table, but provides no further hyperparameter values here.
- APPENDIX: 2016/03/14-2016/05/03 is identified as an additional back-test time slot.
- APPENDIX: Two additional simulated-trade periods are specified: 2016/03/14-2016/05/03 and 2016/04/14-2016/06/03.
- APPENDIX: Table IV is titled “HYPERPARAMETERS OF THE CNN AGENT.”