PaperlayerView exact revision on arXiv

Source-linked AI summary

FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance

Xiao-Yang Liu, Hongyang Yang, Jiechao Gao, Christina Dan Wang

arXiv:2111.09395v1q-fin.TRcs.LG

TL;DR

Quantitative traders face a steep learning curve when implementing DRL strategies because the workflow is error-prone and debugging-intensive. FinRL provides an open-source, configurable three-layer pipeline with market environments, fine-tuned algorithms, reward functions, and live-trading support. The framework organizes reusable tools and benchmark tasks across markets to support strategy development and iteration.

  • Problem

    Building DRL trading strategies requires error-prone programming and time-consuming work across data preprocessing, environment construction, state management, and backtesting.

  • Method

    FinRL combines a three-layer architecture, configurable DRL algorithms and reward functions, standardized historical-data environments, and live trading APIs.

  • Results

    FinRL provides an open-source framework spanning market environments, DRL algorithms, automated backtesting, live trading support, and benchmark trading tasks.

  • Takeaways & Limitations

    FinRL offers practitioners reusable, customizable stepping stones for applying DRL across portfolio allocation, cryptocurrency trading, high-frequency trading, and other finance tasks.

Abstract

from arXiv · show

Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely \textit{to decide where to trade, at what price} and \textit{what quantity}, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework \textit{FinRL} as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and extensibility under the key principles, \textit{full-stack framework, customization, reproducibility} and \textit{hands-on tutoring}. Embodied as a three-layer architecture with modular structures, FinRL implements fine-tuned state-of-the-art DRL algorithms and common reward functions, while alleviating the debugging workloads. Thus, we help users pipeline the strategy design at a high turnover rate. At multiple levels of time granularity, FinRL simulates various markets as training environments using historical data and live trading APIs. Being highly extensible, FinRL reserves a set of user-import interfaces and incorporates trading constraints such as market friction, market liquidity and investor's risk-aversion. Moreover, serving as practitioners' stepping stones, typical trading tasks are provided as step-by-step tutorials, e.g., stock trading, portfolio allocation, cryptocurrency trading, etc.

1 INTRODUCTION

DRL is promising for automating complex, multi-factor trading decisions, but building DRL trading strategies requires error-prone programming and time-consuming debugging. FinRL addresses this need with an open-source, three-layer framework for configurable strategy development and common trading applications.

  • DRL supports dynamic trading decisions by learning from market interactions while offering portfolio scalability and market-model independence.
  • Existing DRL finance applications span stock trading, futures, alternative data, and high-frequency trading.
  • DRL strategy development requires preprocessing data, building environments, managing states, and backtesting, creating a steep learning curve.
  • FinRL streamlines strategy development through configurable data APIs and DRL algorithms within a three-layer framework.
  • The framework emphasizes full-stack coverage, customization, reproducibility, and hands-on tutoring, with modular components and benchmark strategies.
  • The paper proceeds from related work to the framework, benchmark trading tasks, and conclusions.

2 RELATED WORKS

Related work establishes DRL’s broad algorithmic, library, and financial-task foundations. It covers value-based, policy-based, and actor-critic methods, open-source tooling, and applications across trading and mathematical finance.

  • DRL algorithms are grouped into value-based, policy-based, and actor-critic approaches.
  • Value-based methods estimate state-action values, while policy-based methods directly update policy parameters from states to action distributions.
  • Actor-critic methods combine policy updates by an actor with state-action value estimation by a critic.
  • Open-source libraries provide standardized environments, common algorithms, fast prototyping, scalability, and customizable neural-network tooling.
  • DRL finance applications include volatile stock trading, futures strategies with volatility scaling, alternative data, and high-frequency trading.
  • Deep Hedging applies DRL to liquid-derivative risk management, highlighting scalable and model-free mathematical-finance applications.
  • Cryptocurrency research explores DRL for automated trading, portfolio allocation, and market making in a more volatile asset class.

3 THE PROPOSED FINRL FRAMEWORK

The proposed framework combines a FinRL overview with a training-testing-trading pipeline for evaluating trading performance.

  • FinRL is introduced through its framework overview and a training-testing-trading pipeline for standard performance evaluation.

3.1 Overview of FinRL Framework

FinRL uses a three-layer architecture that connects trading tasks, DRL agents, and market environments through modular interfaces. The layers are designed for transparency, independent updates, and extensibility.

  • FinRL consists of application, agent, and environment layers.
  • The application layer offers demonstrative trading tasks, the agent layer provides plug-and-play fine-tuned DRL algorithms, and the environment layer wraps market data and APIs into Gym environments.
  • Upper-layer trading tasks call DRL algorithms and market environments from the lower layers.
  • The layered architecture exposes lower-layer APIs upward while allowing independent layer updates when interfaces remain unchanged.
  • Modular, self-contained functions let users select components for trading tasks and develop new modules through reserved interfaces.

3.2 Application Layer

The application layer maps trading strategies into DRL by specifying state, action, and reward components, with configurable market observations, actions, and objectives.

  • Users define a trading strategy through its state space, action space, and reward function.These components translate an algorithmic trading strategy into the DRL framework.
  • The state space can use multiple time granularities and include balances, holdings, OHLC prices, volume, and technical or fundamental indicators.Supported time steps include daily, hourly, and minute intervals.
  • The action space represents selling, holding, or buying one or multiple shares, with k defining the maximum trade size.For one share, actions are −1, 0, or 1; for multiple shares, actions range from −k to k.
  • FinRL supports user-defined rewards for risk aversion or market friction alongside portfolio-value change and log-return rewards.The portfolio-value-change reward is v′ − v, while the log-return reward is log(v′/v).
  • The Sharpe ratio is defined as (E(R_t) − R_f)/std(R_t), using portfolio return and the risk-free rate.Here, R_t = v_t − v_t−1 and R_f is the risk-free rate.

3.3 Agent Layer

The agent layer provides unified APIs and plug-and-play support for multiple DRL libraries, allowing users to select, customize, and train trading agents through a common workflow.

  • FinRL fine-tunes Stable Baselines 3, RLlib, and ElegantRL as representative open-source DRL libraries.Users can also design new algorithms by adapting existing implementations.
  • FinRL uses unified Python APIs so users can plug in DRL algorithms and train agents with selected environments.Environments may be built from historical data or live trading APIs.
  • The agent workflow initializes an environment and algorithm, applies default environment and model parameters, trains the agent, and returns a trained model.The main APIs hide environment, algorithm-import, and agent-construction details.
  • Stable Baselines 3 is supported for user-friendliness, reproducibility, refinement, new-idea development, and documentation.It serves as a base for extensions and comparisons with existing approaches.
  • RLlib supports high-performance and distributed training, including multiprocessing on laptops and scaling across servers.It natively supports TensorFlow, TensorFlow Eager, and PyTorch.
  • ElegantRL emphasizes lightweight code, customization, efficiency, finance-oriented optimization, and tutorials for discrete and continuous DRL.Its core code has fewer than 1,000 lines and uses PyTorch, OpenAI Gym, NumPy, and Matplotlib.

3.4 Environment Layer

The environment layer converts market information into MDP states and provides reusable, customizable training environments from historical data and live trading sources.

  • FinRL’s environment layer observes market information and translates it into MDP states containing market and agent variables.Stock-trading examples include OHLCV data, technical indicators, account balance, and share holdings.
  • The agent interacts with the environment by observing price changes, taking actions, and receiving rewards while learning a strategy that maximizes expected return.FinRL reformats real market data into Gym-style environments using time-driven simulation.
  • Because financial tasks require different market simulators and building them is time-consuming, FinRL supplies representative environments and supports user-imported data.This design targets the environment-building workload in finance-oriented DRL.
  • Users can import datasets, adjust time granularity, split data into training, testing, and trading periods, and calculate statistics and indicators for the state space.FinRL also supports episodic total return and Sharpe-ratio calculations.

3.5 Training-Testing-Trading Pipeline

FinRL extends conventional training and testing with a trading stage that connects trained agents to live trading APIs, addressing the gap between backtests and live markets.

  • Conventional training-testing workflows can exhibit a simulation-to-reality gap because testing is offline backtesting while the goal is live order placement.Financial tasks therefore differ from standard machine-learning evaluation settings.
  • FinRL uses historical time series for training and backtesting, then connects the trained agent to live APIs such as CCXT, Alpaca, and Interactive Brokers.The live stage downloads data, feeds it to the trained model, and obtains trading positions.
  • The pipeline consists of a training window, a testing window for evaluation and iterative hyperparameter tuning, and a trading window for deployment.These stages are organized as sequential training-testing-trading steps.
  • Rolling windows let investors and portfolio managers retrain models periodically as time advances.FinRL supports monthly, quarterly, yearly, and user-specified rolling windows.

4 HANDS-ON TUTORIALS AND BENCHMARK PERFORMANCE

FinRL tutorials guide users through strategy design, evaluation, and risk-aware trading configuration. The framework combines automated backtesting, conventional baselines, performance metrics, and market constraints across trading tasks.

  • Backtesting and constraints: FinRL provides automated backtesting with plots, performance metrics, and configurable market frictions, liquidity, and investor risk-aversion.Transaction costs can be modeled as flat fees or per-share percentages, while bid-ask spreads and risk-adjusted rewards are also supported.
  • Risk-aversion: For extreme market fluctuations, the turbulence index can halt buying and selling above a preset threshold and resume activity below it.The index uses current returns, historical average returns, and the historical covariance matrix.
  • Baselines: Baseline strategies include passive index replication, mean-variance or minimum-variance allocation, and equally weighted portfolios.
  • Performance evaluation: FinRL evaluates trading with final portfolio value, cumulative and annualized returns, volatility, maximum drawdown, and Sharpe ratio.Cumulative return compares the final portfolio value with its initial value, while drawdown measures losses from a historical peak to a trough.
  • Tutorial workflow: Tutorials walk users through state-action-reward specifications, agent-environment interactions, and the stock-trading strategy design pipeline.

4.4 Use Case I: Stock Trading

FinRL reproduces and compares DRL trading strategies across stocks, portfolio allocation, and cryptocurrencies. The reported use cases emphasize reproducibility, plug-and-play algorithm comparison, and competitive backtesting results against conventional baselines.

  • Stock trading: The stock-trading ensemble combines PPO, A2C, and DDPG, while FinRL makes algorithm implementations and preprocessing transparent to users.
  • Stock trading: The stock-trading test covers Dow 30 constituents from 2020/07/01 to 2021/06/30 after training on daily data from 2009/01/01 to 2020/06/30.
  • Stock trading: The ensemble strategy achieves a Sharpe ratio of 2.81 and an annual return of 52.61%, versus DJIA annual return of 32.84% in the reported bullish period.
  • Reproducibility: FinRL supports reproducible comparisons by varying stock pools, initial capital, and hyperparameters without redeveloping neural networks.
  • Portfolio allocation: Portfolio allocation results show A2C leading with a Sharpe ratio of 2.36 and annual return of 42.57%, while each listed DRL agent outperforms DJIA and the min-variance strategy.
  • Cryptocurrency trading: In cryptocurrency trading with transaction costs, ElegantRL PPO attains a cumulative return of 103%, ahead of the equally weighted portfolio at 99% and BTC buy-and-hold at 93%.The experiment uses ten top-market-cap cryptocurrencies with five-minute training and testing periods in October 2021.

5 ECOSYSTEM OF FINRL AND CONCLUSIONS

FinRL extends its ecosystem across users, markets, tools, and scalable training while supporting an open-source DRL finance community.

  • Conclusions: The framework positions open-source DRL as an ecosystem for finance users across academia and industry.Its training-testing-trading pipeline, standardized environments, implemented DRL algorithms, automated backtesting, and benchmark schemes support this ecosystem.
  • Ecosystem of FinRL: FinRL supports entry-level through advanced users with hands-on documents, ElegantRL, and the cloud-native FinRL-Podracer solution.These offerings target beginners, professionals, and investment banks or hedge funds, respectively.
  • Ecosystem of FinRL: The framework covers diverse markets and finance tasks, including portfolio allocation, cryptocurrency trading, high-frequency trading, and live trading.It also incorporates state-of-the-art DRL algorithms for these applications.
  • Ecosystem of FinRL: FinRL develops supporting tools for financial data engineering, explainable portfolio management, and ensemble strategies for stock trading.FinRL-Meta provides a unified data processor and hundreds of market environments.
Loading 2111.09395v1…