Source-linked AI summary
PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, Jimin Huang
TL;DR
Financial AI lacked openly available finance-specific LLMs, instruction-tuning data, and comprehensive benchmarks. PIXIU addresses these gaps with FinMA, FIT, and FLARE, and its evaluations show strong performance on most financial tasks alongside weaknesses in quantitative reasoning and difficult prediction tasks.
Problem
Financial AI lacked open financial LLMs, financial instruction data, and comprehensive benchmarks for assessing financial task abilities.
Method
PIXIU constructs FIT instruction data, fine-tunes LLaMA into FinMA, and builds FLARE to evaluate financial NLP and prediction tasks.
Results
FinMA significantly outperforms other evaluated LLMs on most financial NLP and stock-movement tasks, but underperforms on quantitative question answering and difficult financial prediction.
Takeaways & Limitations
PIXIU provides open financial models, instruction data, benchmarks, and results intended to facilitate further research in financial language understanding and prediction.
Takeaways & Limitations
FinMA models are presented only up to 30B, and FinMA-30B was not fine-tuned on the full dataset because of computational constraints.
Abstract
from arXiv · showhide
Although large language models (LLMs) has shown great performance on natural language processing (NLP) in the financial domain, there are no publicly available financial tailtored LLMs, instruction tuning datasets, and evaluation benchmarks, which is critical for continually pushing forward the open-source development of financial artificial intelligence (AI). This paper introduces PIXIU, a comprehensive framework including the first financial LLM based on fine-tuning LLaMA with instruction data, the first instruction data with 136K data samples to support the fine-tuning, and an evaluation benchmark with 5 tasks and 9 datasets. We first construct the large-scale multi-task instruction data considering a variety of financial tasks, financial document types, and financial data modalities. We then propose a financial LLM called FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. To support the evaluation of financial LLMs, we propose a standardized benchmark that covers a set of critical financial tasks, including five financial NLP tasks and one financial prediction task. With this benchmark, we conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks. The model, datasets, benchmark, and experimental results are open-sourced to facilitate future research in financial AI.
1 Introduction
PIXIU addresses the lack of open financial LLMs, instruction data, and comprehensive evaluation benchmarks by introducing FinMA, FIT, and FLARE. The framework combines multi-task, multi-modal instruction tuning with evaluation across financial understanding and prediction tasks.
- Framework: PIXIU introduces FinMA, an open financial LLM created by fine-tuning LLaMA with multi-task and multi-modal financial instruction data.The framework also releases the model, instruction-tuning data, benchmark datasets, and implementation.
- Evaluation: FLARE evaluates financial LLMs across four financial NLP tasks and one financial prediction task using nine datasets.The benchmark is designed to assess and compare LLM abilities on financial understanding and prediction.
- Results: FinMA outperforms BloombergGPT, ChatGPT, and GPT-4 on most FLARE tasks, including sentiment analysis, headline classification, NER, and stock movement prediction.The reported results support the value of tailoring LLMs specifically to finance.
- Results: FinMA underperforms other evaluated LLMs on quantitative question answering, while all evaluated models show limited stock-movement prediction performance.The analysis attributes FinMA's question-answering weakness to LLaMA's limitations in quantitative reasoning and mathematics.
- Framework: 136K instruction-tuning samples cover five financial tasks and diverse financial data modalities, including texts, tables, and historical stock-price time series.The tasks include sentiment analysis, news headline classification, named entity recognition, question answering, and stock movement prediction.
2 Related Work
Prior work developed financial language models, evaluation benchmarks, and open general-purpose LLMs, but financial models remained limited in openness and instruction following. Existing benchmarks primarily emphasized financial NLP, motivating broader evaluation that includes prediction.
- Financial Language Models: finBERT was introduced as an early financial pretrained language model using open financial corpora, while BloombergGPT used mixed general and financial data at 50 billion parameters.BloombergGPT's model and pretraining datasets were not released.
- Financial Language Models: BloombergGPT and earlier financial pretrained models were not instruction-following models like ChatGPT and GPT-4.This distinguishes domain pretraining from instruction tuning for downstream task following.
- Financial Evaluation Benchmarks: FLUE established a heterogeneous financial benchmark covering five financial NLP tasks, including sentiment analysis, headline classification, NER, structure boundary detection, and question answering.The cited benchmark work focused on financial NLP task coverage.
- Open-Sourced Large Language Models: LLaMA provided an open-source general-purpose LLM family with models ranging from 7B to 65B parameters, followed by efforts to improve instruction following through instruction tuning.These developments supplied the general-purpose foundation for later financial adaptation.
3 FIT: Financial Instruction Tuning Dataset
FIT is an instruction-tuning dataset built from open financial NLP and prediction data, covering diverse tasks, modalities, and financial text types. Domain experts convert these sources into task-specific instruction samples using templates tailored to each task.
- 3 FIT: Financial Instruction Tuning Dataset: FIT is the first finance instruction-tuning dataset and includes stock movement prediction alongside financial NLP tasks.
- 3.1 Raw Data: The raw data cover sentiment analysis, headline classification, named entity recognition, question answering, and stock movement prediction.
- 3.1 Raw Data: Financial sentiment analysis uses the Financial Phrase Bank and FiQA-SA datasets for analyzing sentiment in financial texts.
- 3.1 Raw Data: Gold headline data support binary classification across nine financial information categories, including price movement and asset comparison.
- 3.1 Raw Data: The FIN dataset provides manually annotated LOCATION, ORGANISATION, and PERSON entities from financial agreements and SEC filings.
- 3.1 Raw Data: FinQA and ConvFinQA provide expert-annotated question answering over earnings reports containing unstructured documents, tables, and multi-turn conversations.
- 3.1 Raw Data: Stock movement prediction is framed as binary classification from historical stock prices and tweets, with movements above 0.55% labeled positive and movements below -0.55% labeled negative.
- 3.2 Instruction construction: Domain experts write ten instructions for most datasets, while ConvFinQA uses one because its multi-turn questions already function as instructions.
4 FinMA: Financial Large Language Model
FinMA is created by fine-tuning LLaMA checkpoints on FIT instruction data at multiple model sizes and dataset scopes. The training uses distinct epoch schedules and shared optimization settings, with the full-data model trained for fewer epochs.
- FinMA-7B is trained for 15 epochs, whereas FinMA-7B-full is trained for 3 epochs using AdamW.
- Training uses batch size 32, learning rate 8e-6, weight decay 1e-5, 5% warmup steps, and a maximum input length of 2048.
5 FLARE: Financial Evaluation Benchmark
FLARE evaluates financial language understanding and prediction, extending prior NLP-focused benchmarking with stock movement prediction. Its evaluation protocol uses task-specific metrics and test sets aligned with BloombergGPT’s reported distributions.
- FLARE combines financial NLP tasks with financial prediction tasks to evaluate practical financial-domain applications more comprehensively than FLUE.
- FLARE uses validation sets for checkpoint selection and test sets for evaluation, while matching BloombergGPT’s test-set sizes and data distributions for comparison.
- Evaluation reports ACC and weighted F1 for sentiment, Avg F1 for headline classification, Entity F1 for NER, and exact-match accuracy for question answering.
6 Experiments on FLARE
The experiments compare FinMA with BloombergGPT and general-domain LLMs on FLARE under zero-, few-, and 20-shot settings. FinMA leads on several financial NLP tasks, while complex quantitative reasoning and stock prediction remain difficult.
- Experimental Setup: FLARE reports zero-shot and few-shot performance across different LLMs, with BloombergGPT additionally evaluated in 20-shot settings on selected datasets.BloombergGPT’s unreleased test datasets were reconstructed with the same data distribution for direct comparison.
- Overall Performance: FinMA significantly outperforms BloombergGPT, ChatGPT, and GPT-4 on FPB, FiQA-SA, and Headline datasets.FinMA-30B exceeds GPT-4 by 10% F1 on FPB and BloombergGPT by 37% F1 on FPB.
- Overall Performance: FinMA-7B outperforms BloombergGPT and other LLMs on NER while remaining competitive with ChatGPT and GPT-4.
- Overall Performance: FinMA shows a large performance gap behind GPT models on FinQA and ConvFinQA, which require complex numeric reasoning.The paper attributes this difficulty to LLaMA’s lack of mathematical pre-training data.
- Overall Performance: All evaluated LLMs, including FinMA, ChatGPT, and GPT-4, struggle with stock movement prediction.
- Further Analysis: FinMA-30B is not significantly better than FinMA-7B on most NLP and stock-movement tasks, whereas larger models generally perform better on ConvFinQA.The analysis identifies instruction quality as more important than model size for most tested tasks.
- Further Analysis: FinMA-7B-full, fine-tuned with NLP and prediction data, performs significantly better on ACL18 and comparably on NLP tasks with FinMA-7B and GPT-4.
7 Limitations
The paper identifies constraints in model scale and training coverage, difficult quantitative tasks, resource-dependent generalizability, input length, and potential misuse.
- Model and Training Constraints: FinMA is presented only up to 30B parameters, and FinMA-30B was not fine-tuned on the full dataset because of computational constraints.
- Complex Task Performance: FinMA struggles with quantitative-reasoning tasks such as financial question answering and difficult financial prediction because of limitations in its LLaMA backbone.
- Resource Constraints and Generalizability: Available resources and handcrafted instructions may affect the diversity and generalizability of FinMA, FIT, and FLARE.
- Resource Constraints and Generalizability: FinMA’s maximum input size is limited by the maximum input length supported by the LLaMA backbone.
- Potential Negative Impacts: The paper acknowledges potential negative impacts, including financial misinformation and unethical market influence.
8 Conclusion
PIXIU combines the open-sourced FinMA model, FIT instruction dataset, and FLARE evaluation benchmark. Extensive evaluation supports domain-specific instruction tuning’s potential in finance, while complex tasks and resource constraints remain challenges.
- PIXIU encompasses the open-sourced FinMA financial LLM, FIT instruction-tuning dataset, and FLARE evaluation benchmark.
- Extensive evaluation demonstrated FinMA’s effectiveness across various financial tasks.
- Domain-specific instruction tuning shows potential for financial language understanding, prediction, and LLM applications.
- Improving performance on complex tasks and addressing resource constraints remain challenges.