Source-linked AI summary
Qlib: An AI-oriented Quantitative Investment Platform
Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, Tie-Yan Liu
TL;DR
AI-driven quantitative investment raises infrastructure and finance-specific adaptation challenges. Qlib combines an AI-oriented platform with financial-data infrastructure and research tools; on a typical task, its infrastructure outperforms most existing solutions.
Problem
AI-driven workflows require upgraded infrastructure, while financial tasks pose domain-specific challenges that generic machine-learning solutions rarely address without adaptation.
Source: §1 · p. 1Method
Qlib is an AI-oriented quantitative-investment platform that combines research workflow support with high-performance infrastructure and financial machine-learning tools.
Source: §1 · p. 1Results
Qlib's infrastructure outperforms most existing solutions on a typical quantitative-investment task.
Source: §1 · p. 1Takeaways & Limitations
Qlib is designed to support AI-based quantitative research and help researchers explore its potential in investment.
Source: §1 · p. 1Array-based HDF5 and pickle data are efficient for scientific computing but difficult to update or query.
Source: Financial data · p. 4
Abstract
from arXiv · showhide
Quantitative investment aims to maximize the return and minimize the risk in a sequential trading period over a set of financial instruments. Recently, inspired by rapid development and great potential of AI technologies in generating remarkable innovation in quantitative investment, there has been increasing adoption of AI-driven workflow for quantitative research and practical investment. In the meantime of enriching the quantitative investment methodology, AI technologies have raised new challenges to the quantitative investment system. Particularly, the new learning paradigms for quantitative investment call for an infrastructure upgrade to accommodate the renovated workflow; moreover, the data-driven nature of AI technologies indeed indicates a requirement of the infrastructure with more powerful performance; additionally, there exist some unique challenges for applying AI technologies to solve different tasks in the financial scenarios. To address these challenges and bridge the gap between AI technologies and quantitative investment, we design and develop Qlib that aims to realize the potential, empower the research, and create the value of AI technologies in quantitative investment.
1 Introduction
AI expands quantitative investment workflows but demands upgraded infrastructure and finance-specific adaptation. Qlib addresses these needs with an AI-oriented platform, and its infrastructure outperforms most existing solutions on a typical task.
1 Introduction
- AI can enable end-to-end portfolio generation, while its data-driven nature calls for upgraded infrastructure and finance-specific adaptation.Applying machine-learning solutions without domain adaptation rarely works in financial scenarios.Source: §1 · p. 1
- Qlib combines an AI-oriented framework, high-performance infrastructure, and quantitative-investment machine-learning tools.The platform aims to accommodate AI-based solutions and support research in financial scenarios.Source: §1 · p. 1
- Qlib's infrastructure outperforms most existing solutions on a typical quantitative-investment task.The authors compare several solutions for that task.Source: §1 · p. 1
2 Background and Related Works
The section contrasts traditional workflows with AI-era demands: flexible, data-driven methods require more infrastructure and finance expertise, while existing tools cover only parts of the workflow. Qlib is presented as an open-source platform for that broader workflow.
Quantitative research workflow revolution
- RL provides an end-to-end route from data to trading actions, requiring a responsive simulator rather than traditional back-testing.AI-era workflows may also use machine learning to construct features and mine trading signals.Source: Quantitative research workflow… · p. 2
High performance requirements for infrastructure
- High-frequency data can reach terabyte scale, and researchers may derive thousands of features from five basic price-and-volume dimensions.The resulting data-processing demands can overburden researchers and make some topics impractical.Source: High performance requirements for… · p. 2
Obstacles to apply machine learning solutions
- Low signal-to-noise data, finance-specific task design, and nondifferentiable objectives complicate machine-learning applications in finance.The section also identifies costly hyperparameter search and limited machine-learning experience as barriers for some researchers.Source: Obstacles to apply machine learning… · p. 2
2.2 Related Work
- Existing quantitative tools focus on selected workflow components, whereas Qlib is presented as the first open-source platform for the modern AI-era workflow.The section describes prior tools spanning portfolio selection, infrastructure, back-testing, and signal or portfolio analysis.Source: §2.2 · pp. 2–3
3.1 Overall Design
Qlib combines a flexible, modular research platform with high-performance financial-data infrastructure and machine-learning guidance. Its design aims to let researchers extend workflows while offering practical defaults and shared resources.
3.1 Overall Design
- Qlib's modular platform lets researchers extend workflows and try ideas efficiently, while sharing code, computation, and data in some modules.The authors contrast this platform model with a toolbox.Source: §3.1 · p. 3
- Qlib's time-series flat-file database is designed for financial scientific computing and outperforms popular general-purpose and time-series databases on typical investment-research data-processing tasks.Its expression engine accelerates factor and feature implementation and computation.Source: §3.1 · p. 3
- Qlib integrates typical quantitative-investment datasets, reasonable feature spaces and target labels, and hyperparameter-optimization tools to guide machine-learning users.The paper says these settings can help models learn generalizable patterns rather than overfit noise.Source: §3.1 · p. 3
3.2 AI-oriented Framework
Qlib organizes quantitative research as flexible modules that map to workflow tasks, with interfaces for extending methods. It also supports dynamic model and strategy updates as financial data change over time.
3.2 AI-oriented Framework
- Qlib's modules map to workflow sub-tasks, letting researchers replace implementations and test ideas alongside other modules.The workflow runs from querying and processing data through dataset construction and model learning to portfolio generation, execution, and analysis.Source: §3.2 · pp. 3–4
- The Order Executor is a responsive simulator rather than a back-testing function, supporting learning paradigms such as RL that need environmental feedback.The Analyser modules produce feedback about the environment.Source: §3.2 · p. 4
- Because financial data and markets change over time, Qlib's Dynamic Modeling modules provide interfaces and infrastructure for updating models and strategies.The paper describes regular model updates as a typical way to leverage new data and dynamic updates as a way to improve performance further.Source: §3.2 · p. 4
3.3 High Performance Infrastructure
Qlib's high-performance infrastructure is tailored to quantitative research's compact, changing financial data, combining flat-file storage with expression and cache systems.
Financial data
- Existing databases impede array-based computation and single-file array formats hinder updates and queries, motivating a customized quantitative-research store.General-purpose and time-series databases use formats that are inefficient for scientific computation; array-based formats are efficient but difficult to update or query.Source: Financial data · p. 4
File storage design
- Qlib organizes data by frequency, instrument, and attribute in compact fixed-width files, with a shared calendar and start indices enabling time alignment and efficient appends.Separate files make adding or removing attributes and instruments straightforward, while the design retains compact-array performance without database overhead.Source: File storage design · pp. 4–5, Figure 2 · p. 5
Expression Engine
- The expression engine represents factors as functions of basic data and lets researchers build them from readable, reusable expressions instead of complicated code.For example, the paper implements the upper Bollinger Band with a single expression.Source: Expression Engine · p. 5
Cache system
- Memory LRU caching reuses computed syntax-tree nodes, while two disk-cache levels store expression results and combined datasets for reuse across time-based queries and appends.Expression caching avoids recomputing the same expression; dataset caching saves data-combination time.Source: Cache system · p. 5, Figure 3 · p. 5
3.4 Guidance for Machine Learning
Qlib provides machine-learning guidance and automates hyperparameter search, adapting the search distribution around the previous best setting for sequential financial tasks.
3.4 Guidance for Machine Learning
- Qlib supplies typical datasets and task settings, giving researchers domain knowledge to start without exploring everything from scratch.Examples include data preprocessing and learning targets.Source: §3.4 · p. 5
- Hyperparameter optimization is necessary for better generalization but repetitive and effortful, so HTE lets users define a search space and search automatically.HTE provides an interface for defining the hyperparameter search space Θ.Source: §3.4 · pp. 5–6
- For sequential financial tasks, HTE concentrates search probability near the previous training's best hyperparameter while keeping the search domain unchanged.The previous best is often close to the new best, and the distribution is intended to improve the chance of finding it with fewer trials.Source: §3.4 · p. 6
4 Use Case & Performance Evaluation
Qlib offers a configurable but flexible research workflow and evaluates its data-processing infrastructure against other storage solutions, reporting time savings from caching and multiple CPUs.
4.1 Use Case
- CDPE lets users define a research workflow in a simple config file while keeping configuration optional and code-built blocks available; Figure 4 gives a configuration example.Source: §4.1 · p. 6, Figure 4 · p. 6
4.2 Performance Evaluation
- The comparison spans HDF5, MySQL, MongoDB, InfluxDb, and Qlib on 14 OHLCV-derived features for 800 daily stocks from 1/1/2007 to 1/1/2020.Source: §4.2 · p. 6
- The breakdown separates loading, expression computation, Qlib-only index conversion, pool filtering, and cross-stock array combination; historical OHLCV is needed before filtering.Source: §4.2 · p. 6
- 80.4% of Compute Expr. time is saved with Qlib +E -D when expression cache hits; Qlib -E -D saves about 24% of Compute Expr. time.With Qlib +E +D, dataset caching further reduces time cost.Source: §4.2 · p. 7
- Multiple CPUs significantly reduce Qlib's time cost, while Qlib +E +D gains no further acceleration because it reads existing cache and computes almost nothing.The latter configuration is already reading cached results rather than doing substantial computation.Source: §4.2 · p. 7
4.3 More about Qlib
- Qlib is open source and continuously developed, with repository features including client-server data service, analysis, and cloud deployment.Source: §4.3 · p. 7
5 Conclusion
The paper presents practical problems facing quantitative researchers in the AI era and designs and implements Qlib to help them realize AI technologies' potential in quantitative investment.
5 Conclusion
- The paper presents practical problems faced by modern quantitative researchers in the age of AI.Source: §5 · p. 7
- Based on those problems, the paper designs and implements Qlib.Source: §5 · p. 7
- Qlib aims to empower every quantitative researcher to realize AI technologies' potential in quantitative investment.Source: §5 · p. 7