Source-linked AI summary
FinBERT: A Pretrained Language Model for Financial Communications
Yi Yang, Mark Christopher Siy UY, Allen Huang
TL;DR
Financial sentiment analysis lacks a finance-specific pretrained language model despite abundant financial communications and the domain dependence of sentiment. The paper pretrains FinBERT on representative financial corpora and finds that it outperforms generic BERT models across three financial sentiment classification tasks, supporting its release as a financial NLP resource.
Problem
Financial sentiment analysis is domain dependent, yet no finance-specific pretrained language model is available despite the sector’s large financial communication text collections.
Method
FinBERT is pretrained as a finance-specific BERT on 4.9 billion tokens from corporate reports, earnings call transcripts, and analyst reports.
Results
FinBERT outperforms generic BERT models on three financial sentiment classification tasks.
Takeaways & Limitations
The released FinBERT model and pretrained resources are intended for financial sentiment analysis and broader financial NLP applications.
Abstract
from arXiv · showhide
Contextual pretrained language models, such as BERT (Devlin et al., 2019), have made significant breakthrough in various NLP tasks by training on large scale of unlabeled text re-sources.Financial sector also accumulates large amount of financial communication text.However, there is no pretrained finance specific language models available. In this work,we address the need by pretraining a financial domain specific BERT models, FinBERT, using a large scale of financial communication corpora. Experiments on three financial sentiment classification tasks confirm the advantage of FinBERT over generic domain BERT model. The code and pretrained models are available at https://github.com/yya518/FinBERT. We hope this will be useful for practitioners and researchers working on financial NLP tasks.
1 Introduction
Financial NLP can extract sentiment from communication relevant to market monitoring, but generic pretrained language models overlook the domain-specific nature of financial sentiment. FinBERT addresses this gap by pretraining BERT on large-scale financial communications and evaluating it on three sentiment tasks.
- Financial sentiment from news and social media can serve as a directional signal for trading and has been linked to market returns.The paper motivates financial NLP through practitioners’ interest in real-time sentiment monitoring.
- Generic language models are typically pretrained on corpora such as Wikipedia, while financial sentiment analysis is strongly domain dependent.Financial and business communications provide a large domain-specific text resource for pretraining.
- FinBERT is a finance-specific BERT pretrained on 4.9 billion tokens from corporate reports, earnings call transcripts, and analyst reports.The model is introduced to fill the absence of a finance-specific pretrained language model.
- Experiments on three financial sentiment classification tasks show that FinBERT outperforms generic BERT models.The released model and corpus are presented as resources for financial sentiment analysis.
2 Related Work
Pretrained language models produce contextual token representations that improve many NLP tasks, and domain-specific pretraining can further improve downstream performance. This motivates specialized models for financial language alongside existing biomedical and clinical models.
- Unsupervised pretraining on large corpora has improved NLP performance across tasks from sentence classification to question answering.Unlike traditional word embeddings, these models generate contextualized representations for each token.
- Generic language models are trained on broad-domain sources such as news articles and Wikipedia.Their representations can be fine-tuned for downstream tasks.
- Large-scale domain-corpus pretraining has been shown to improve task performance beyond fine-tuning a generic language model.The related work cites biomedical and clinical BERT models as examples of domain-specific pretraining.
3 Financial Corpora
The paper builds a representative financial communications corpus from corporate reports and earnings call transcripts, using publicly available filings and communications that contain business, risk, and performance information.
- Corporate Reports: Corporate reports are treated as important financial communications because they provide comprehensive information about a company’s business and financial condition.The corpus uses Form 10-K and Form 10-Q filings obtained from the SEC website.
- Corporate Reports: The corpus contains 60,490 Form 10-Ks and 142,622 Form 10-Qs from Russell 3000 firms during 1994–2019.Included sections include Business, Risk Factors, and Management’s Discussion and Analysis.
- Earnings Call Transcripts: Earnings calls are quarterly discussions in which executives and analysts discuss firm performance, forward-looking statements, and clarifying information.Investors can attend to executives’ tones as potential signals of good or bad company news.
- The pretraining corpus size is summarized in Table 1.The supplied caption identifies the table as reporting the size of the pretraining financial corpora.
4 FinBERT Training
FinBERT training adapts BERT’s vocabulary and pretraining procedure to financial communications. The paper compares financial and original vocabularies, cased and uncased variants, and continued pretraining versus training from scratch.
- Vocabulary: FinVocab is a financial WordPiece vocabulary built with SentencePiece in cased and uncased versions.The vocabulary sizes are 28,573 cased and 30,873 uncased tokens, with 41% overlap with BERT’s BaseVocab.
- FinBERT-Variants: FinBERT uses BERT-Base configuration, initially training with maximum sentence length 128 before continuing with lengths up to 512 tokens.Four variants combine cased or uncased modeling with BaseVocab or FinVocab.
- FinBERT-Variants: BaseVocab models are initialized from original BERT-Base and further pretrained on financial corpora for 250K iterations at learning rate 2e−5.This configuration applies to both cased and uncased variants.
- FinBERT-Variants: FinVocab models are trained from scratch for 1M iterations using new financial vocabularies.Training uses multi-GPU hardware with a batch size of 128.
5 Financial Sentiment Experiments
The experiments evaluate FinBERT on three financial sentiment classification datasets using repeated train-test splits and accuracy, comparing it with generic BERT and varying vocabulary, casing, and pretraining corpora. FinBERT improves over generic BERT, with financial-corpus pretraining contributing more than the in-domain vocabulary alone.
- 5.1 Dataset: The experiments use Financial Phrase Bank, AnalystTone, and FiQA for financial sentiment classification.FiQA’s original regression task is converted to binary classification for comparison with the other datasets.
- 5.2 Experimental Setup: Each dataset is randomly split into 90% training and 10% testing 10 times, with average accuracy reported.
- 5.3 Experiment Results: On FiQA, uncased FinBERT-FinVocab achieves 0.844 accuracy, improving 15.6% over uncased BERT and 29.2% over cased BERT.
- 5.3 Experiment Results: FinBERT-FinVocab outperforms BaseVocab models, but the improvement is marginal on PhraseBank and AnalystTone and substantial on FiQA, where accuracy is 0.844 versus 0.796.
- 5.3 Experiment Results: Uncased models perform better than cased models across all tasks, while models trained on all financial corpora achieve the overall best performance.
6 Conclusion
The paper presents FinBERT, a financial-task-oriented BERT model pretrained on representative English financial communications. It reports that FinBERT outperforms generic BERT on three financial sentiment classification tasks and may support broader financial prediction applications.
- FinBERT is a financial-task-oriented BERT model trained on representative English financial communications.
- FinBERT outperforms generic BERT models on three financial sentiment classification tasks.
- The released model is intended for applications involving financial outcomes beyond sentiment, including stock returns, stock volatilities, and corporate fraud.