Source-linked AI summary
EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading
Molei Qin, Shuo Sun, Wentao Zhang, Haochong Xia, Xinrun Wang, Bo An
TL;DR
RL methods developed mainly for low-frequency trading face long training horizons and changing Crypto market trends in HFT. EarnHFT combines optimal-value supervision, trend-specific agents, and a minute-level router. Experiments report that it outperforms six baselines across six financial criteria, exceeding the runner-up by at least 30% in profitability.
Problem
RL methods focused on low-frequency trading face extremely long trajectories and market-trend changes that challenge robust performance in HFT.
Source: §1 · p. 1Method
EarnHFT uses a dynamic-programming Q-teacher for second-level agents, builds a profitable pool of trend-preference agents, and trains a minute-level router to select among them.
Source: §4.1 · p. 4, §4.2 · p. 5Results
EarnHFT outperforms 6 state-of-art baselines across 6 popular financial criteria and exceeds the runner-up by at least 30% in profitability.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
High-frequency trading (HFT) uses computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performance. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second-level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.
1 Introduction
EarnHFT addresses the difficulty of training robust RL for high-frequency crypto trading, where long horizons and changing market trends pose challenges. Its three-stage framework combines a Q-teacher, trend-specific agents, and a minute-level router; experiments report gains over six baselines.
1 Introduction
- Crypto’s nonstop trading and dramatic price fluctuations are cited as reasons HFT is widely used in the market.The passage notes that 24/7 trading prevents overnight risk for Crypto holders and that price fluctuations offer HFT opportunities.Source: §1 · p. 1
- RL methods successful in low-frequency trading rarely maintain robust performance in HFT, where trajectories are extremely long and changing trends can make historical-data agents lose heavily.The HFT horizon is around 1 million steps; a dramatic market trend change can cause tremendous losses.Source: §1 · p. 1
- EarnHFT uses dynamic programming to compute a Q-teacher, trains diverse second-level agents for different return-rate preferences, and uses a minute-level router to select among a profitable subset.Buy-and-hold return rates indicate preferences; agent selection is based on profitability across market categories.Source: §1 · p. 1
- EarnHFT outperforms 6 state-of-art baselines across 6 popular financial criteria and exceeds the runner-up by at least 30% in profitability.Experiments cover various Crypto market trends in a high-fidelity simulation trading environment.Source: §1 · p. 2
2 Related Works
The related-work discussion surveys HFT in Crypto and RL for quantitative trading, highlighting limits of technical indicators and single-agent approaches, and an unexplored HFT setting for existing hierarchical frameworks.
2.1 High-Frequency Trading in Crypto
- Crypto technical indicators, including imbalance volume and MACD, can produce false signals in volatile markets, and results can depend on take-profit and stop-loss settings.The cited indicators aim to capture buying and selling pressures across timescales.Source: §2.1 · p. 2
2.2 RL for Quantitative Trading
- DeepScalper, DRA, and CDQNRP improve short-term RL trading in different ways, but these approaches focus on one agent and neglect performance over longer periods.DeepScalper uses hindsight bonus and an auxiliary task; DRA uses LSTM and PPO; CDQNRP uses random perturbation with convolutional DQN.Source: §2.2 · p. 2
- Prior hierarchical trading frameworks address portfolio management, while their application to HFT trading a single asset remains unexplored.The section names HRPM and MetaTrader as examples of hierarchical approaches in portfolio management.Source: §2.2 · p. 2
3 Problem Formulation
The paper formulates single-asset HFT as a hierarchical MDP: a second-level trader acts on micro-market information, while a minute-level router selects among traders as market trends change.
3.1 Financial Foundations for HFT
- A limit order book records unfilled orders, OHLC summarizes executed orders, and technical indicators are derived from OHLC and LOB data.The formulation uses these financial concepts to represent market information.Source: §3.1 · p. 2
- Execution price is calculated from order-book levels and commission fees, while net value tracks cash plus the value of the current position.The text defines the execution-price variables and gives the net-value expression.Source: §3.1 · p. 2
- The trading objective is to maximize net value by placing market orders on a single asset at a second-level timescale, with only long positions permitted.The position is nonnegative, and net value combines cash with the value of the held position.Source: §3.1 · p. 2
3.2 Hierarchical MDP Framework
- The MDP formulation defines states, actions, transitions, rewards, discounting, and horizon, with the objective of learning a policy that maximizes discounted reward.The agent observes a state, acts, then receives the next state and a reward.Source: §3.2 · p. 2
- Market drift and conflicting effective strategies make it difficult for one agent to remain profitable across trends; macro-level information provides insight into micro-level dynamics.The framework separates second-level execution from minute-level trend adjustment.Source: §3.2 · pp. 2–3
- The low-level trader observes 54 micro-market features and its position, chooses a target position from a finite pool, and receives second-level net-value changes as reward.Its market features use a rolling window of second-level OHLC and limit-order-book snapshots.Source: §3.2 · p. 3
- The high-level router observes 19 macro-market features and the current position, selects a pretrained low-level agent each minute, and receives that agent’s minute-level return.The framework seeks a set of traders and a router that maximize total profit.Source: §3.2 · p. 3, Figure 1 · p. 3
4 EarnHFT
EarnHFT trains second-level traders with optimal-value supervision, constructs a pool of agents suited to different market trends, then trains a minute-level router to choose among them.
4 EarnHFT
- The framework has three stages: Q-teacher training for low-level agents, trend-based construction of a diverse pool, and router training for non-stationary markets.The router selects an agent from the pool to maintain profitability across market conditions.Source: §4 · p. 3
4.1 Stage I: Efficient RL with Q-Teacher
- Dynamic programming uses future prices to compute optimal action values for low-level agents, while KL-divergence supervision and optimal-policy transitions support learning.The supervisor supplies action-value information without requiring exploration of every action; optimal transitions add experience and help prevent local traps.Source: §4.1 · pp. 3–4
4.2 Stage II: Construction of Agent Pool
- Agents are trained on data chunks representing continuous market trends, with return-rate preferences controlling chunk-sampling priorities.The method uses different preference parameters and kernel-density estimates in its sampling procedure.Source: §4.2 · p. 5
- The agent pool retains a small profitable subset: agents are evaluated across labeled market segments and initial positions, then selected by average return for each combination.Market segments are labeled using denoising, extrema-based segmentation, and merging based on DTW and slope differences.Source: §4.2 · pp. 4–5
4.3 Stage III: Dynamic Routing Optimization
- Because the minute-level router would otherwise explore too many low-level agents, the method restricts candidates to agents whose initial position matches the current position.This reduces the router’s possible choices to the number of market trends; the authors use DDQN without a Q-teacher at this level.Source: §4.3 · p. 5
5 Experiment Setup
The experiments test EarnHFT across four Crypto datasets, define trading-performance criteria, and compare it with established RL and indicator-based methods. The setup also specifies how agents are trained and how baseline implementations are handled.
5.1 Datasets
- Testing uses four Crypto datasets, with the last 9 days held out for testing and the penultimate 9 days for validation.The test period exceeds a week and covers bull and bear conditions; the remaining data are used for training, with router selection on validation data.Source: §5.1 · p. 5
5.2 Evaluation Metrics
- Total Return, Annual Volatility, Drawdown, Annual Sharpe, Annual Calmar, and Annual Sortino characterize returns, risk, and risk-adjusted performance.Sortino uses downside deviation, defined as the standard deviation of negative return rates.Source: §5.2 · p. 6
5.3 Training Setup
- Training generated 200 agents using β values −90, −10, 30, and 100, with 50 epochs per value, and used Adam for DDQN.The commission fee is 0 for BTCT and 0.02% for other datasets; official baseline hyperparameters were used when implementations were available.Source: §5.3 · p. 6
5.4 Baselines
- The comparison includes PPO, DRA, DQN, CDQNRP, MACD, and IV.The RL methods use importance sampling, LSTM representations, experience replay, or perturbed target frequencies; MACD and IV are technical indicators.Source: §5.4 · p. 6
6 Results and Analysis
EarnHFT achieves the highest profit across four datasets and the highest risk-adjusted profit in three, while showing a risk trade-off. Its router combines trend-specialized agents, and ablations report training and return benefits from optimal-value supervision.
6.1 Comparison with Baselines
- Highest profit on all 4 datasets and highest risk-adjusted profit on 3; EarnHFT is also described as a radical trader that performs moderately in risk on some datasets.The authors attribute the risk trade-off to optimal-value supervision and the optimal actor emphasizing profit-related rather than risk-related information.Source: §6.1 · p. 6
- In ETH, EarnHFT opens and closes a position within 30 seconds, profiting from a trend viewed as a minute-level pullback.Figure 2 depicts the trading process in ETH.Source: §6.1 · p. 7, Figure 2 · p. 7
6.2 The Effectiveness of Hierarchical Framework
- Bull and rally agents tend to buy and hold, sideways agents trade less, and pullback and bear agents tend to close positions; the router achieves the best profit on all 4 datasets.The router’s selections are more balanced across five trends on high-volatility ETH and GALA, but focus on two dynamics on lower-volatility BTCT and BTCU.Source: §6.2 · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
6.3 The Effectiveness of Optimal Action Value
- On GALA, optimal supervision takes 15% of DDQN’s steps to converge and yields higher return; the optimal actor further improves return at the cost of more convergence steps.On bull-market ETH, supervision does not reduce convergence steps but substantially increases return; the authors say its instructions vary with the agent policy, unlike actor demonstrations.Source: §6.3 · p. 7, Table 3 · p. 7
7 Conclusion
EarnHFT combines a diverse pool of second-level agents with a router to address training efficiency and changing market dynamics. The supplied sections also introduce HFT concepts, prior RL approaches, and the financial and market-segmentation foundations used by the method.
7 Conclusion
- EarnHFT uses a three-stage hierarchical RL framework to improve training efficiency and address data shifting; ablations show its components are effective.The stages compute optimal action values, train agents for different market trends, and train a router to select an agent.Source: §7 · p. 7
A.1 High-Frequency Trading in Crypto
- HFT uses computer algorithms to place, cancel, or execute orders at high speed, with the paper describing execution within seconds.The background also introduces imbalance volume and MACD as Crypto-market indicators.Source: §A.1 · p. 9
A.2 Reinforcement Learning for HFT
- Prior HFT reinforcement-learning work includes value-based methods such as DQN variants and policy-based methods such as DDPG and PPO.The cited overview also mentions replay-based, perturbation, and risk-adjusted reward approaches.Source: §A.2 · p. 9
A.3 Hierarchical RL for Quantitative Trading
- Prior hierarchical trading methods divide work across levels such as stock selection, portfolio weighting, and order execution.The section frames hierarchical RL as a way to decompose long-horizon tasks.Source: §A.3 · p. 9
B.1 Basic Financial Concepts
- A limit order book records unfilled buy and sell orders, while OHLC summarizes the first, highest, lowest, and last executed prices in a period.The book displays available prices and liquidity on both sides.Source: §B.1 · p. 9
- Market orders prioritize immediate execution, but orders larger than the best-level liquidity consume less favorable prices at deeper levels.The paper notes that repeated trading also incurs costs from the ask-bid spread.Source: §B.1 · p. 10
B.2 Motivation for Hierarchical Formulation
- Incompatible strategies across market regimes make one agent unreliable; a fixed pool of diverse agents stabilizes router training.The paper reports that goal-conditioned training did not improve efficiency because bull- and bear-market traders share few skills.Source: §B.2 · pp. 10–11
C.1 Algorithm Description
- Market segmentation uses slope and DTW criteria to decide whether adjacent time-series segments should receive the same label.The method section introduces market segmentation and labeling as part of EarnHFT.Source: §C.1 · p. 11, C EarnHFT · p. 11
C.2 Proof of Convergence
The convergence argument targets the optimal action value in finite MDPs by proving the method's operator is a contraction in the sup-norm.
C.2 Proof of Convergence
- For a finite MDP, the convergence proof reduces to showing that operator H is a contraction in the sup-norm.The stated target is convergence to the optimal action value Q*.Source: §C.2 · p. 11
- The proof first establishes that the optimal Q-function q* is a fixed point of H.Source: §C.2 · p. 11
- Algorithm 1 takes a dataset, risk threshold, and label count, and returns market segments with labels.Its steps describe filtering, segmenting, merging, and assigning labels.Source: §C.2 · p. 11
D.1 Dataset
The dataset section describes four Binance trading pairs with different histories, market characteristics, and test-period trends.
D.1 Dataset
- BTC/TUSD is a recently launched, trend-unstable pair, while BTC/USDT has the longest history and a relatively stable trend.BTC/TUSD began trading on March 23, 2023; BTC/USDT data extend back to 2016.Source: §D.1 · p. 12
- ETH/USDT has significant liquidity and volatility, and its test period coincides with a roughly 20% decline in the first half of May.Source: §D.1 · p. 12
- GALA/USDT is described as highly volatile, with its test period showing an increase of around 10% in the first half of August.Source: §D.1 · p. 12
D.2 Data Difference
Validation and test data differ across trading pairs, while the paper distinguishes dataset-specific trading settings from algorithm-specific RL settings.
D.2 Data Difference
- BTC-USDT shows no large validation-test difference, while GALA test data are more stable than validation data and both trends rise.Source: §D.2 · p. 12
- Around 30%, ETHUSDT falls in the test set despite an upward validation trend; BTC-TUSD tests show a stable bear market.Source: Figure 3 · p. 12, §D.2 · p. 12
D.3 Parameter Setting
- Trading parameters vary by dataset, whereas RL-training parameters vary by algorithm and are shared across datasets.The text directs readers to Tables 1 and 2 for the respective settings.Source: §D.3 · p. 12, Table 1 · p. 12
D.4 Experiments Results
This section presents net-value curves and trading examples for the datasets, alongside a target-network update setting for value-based methods.
D.4 Experiments Results
- Trading examples are shown for BTCT, BTCU, ETH, and GALA to illustrate EarnHFT's trading behavior.The text describes these examples as demonstrating the rationality of EarnHFT’s trading behavior.Source: §D.4 · p. 12, Figure 8 · p. 13, Figure 9 · p. 14
- Value-based methods use soft target-network updates, except CDQNRP, whose hard-update frequency perturbation is a contribution.Source: Table 2 · p. 13
- Net-value curves are shown for BTCT, BTCU, ETH, and GALA.The figures present curves for each dataset without stating comparative outcomes in the supplied text.Source: §D.4 · p. 12, Figure 4 · p. 13, Figure 5 · p. 13