Source-linked AI summary
WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, Diyi Yang
TL;DR
Financial language-model research had not fully incorporated financial domain knowledge and lacked diverse evaluation benchmarks. The paper introduces FLANG, which uses finance-specific word and phrase masking with a span boundary objective, and FLUE, a five-task financial benchmark suite. FLANG-BERT and FLANG-ELECTRA outperform prior models across most financial benchmarks, while the authors note limitations in benchmark coverage, pre-training data, language scope, and model architectures.
Problem
Prior financial language models did not fully use financial domain knowledge during pre-training, and financial evaluation lacked diverse benchmarks.
Method
FLANG uses finance-specific word and phrase masking with a span boundary objective, while FLUE provides five financial NLP benchmarks.
Results
FLANG-BERT outperforms prior works on nearly all benchmarks, and FLANG-ELECTRA further improves performance, yielding two new state-of-the-art models.
Takeaways & Limitations
FLUE offers comprehensive financial evaluation, while the results support finance-aware pre-training for financial language understanding.
Takeaways & Limitations
The work omits abstractive generation and summarization, social-media pre-training, non-English tasks, and decoder-based architectures.
Abstract
from arXiv · showhide
Pre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. Our models, code and benchmark data are publicly available on Github and Huggingface.
1 Introduction
Financial language differs from general usage, motivating domain-specific pre-training and broader financial evaluation. FLANG combines finance-aware masking and span objectives with FLUE benchmarks, and the resulting models outperform prior systems across most evaluated tasks.
- Motivation: Financial terminology can reverse the sentiment implied by ordinary-language cues, motivating domain-specific language-model training.The example of rising crude oil prices illustrates why financial text requires domain-aware interpretation.
- Research gap: Prior financial pre-training did not fully use financial domain knowledge and lacked diverse evaluation benchmarks.These limitations concern both the training process and the ability to test performance across finance-related tasks.
- Approach: FLANG uses finance-specific word and phrase masking, prediction of phrases, and a span boundary objective to learn multi-word financial representations.The approach includes FLANG-BERT and FLANG-ELECTRA, with the latter adding span-boundary training to ELECTRA.
- Results: FLANG-BERT outperforms prior works on nearly all benchmarks, while FLANG-ELECTRA further improves performance and yields two new state-of-the-art models.The models, code, and benchmark data are publicly released for further research.
- Evaluation: FLUE provides comprehensive financial benchmarks spanning sentiment analysis, headline classification, named entity recognition, structure boundary detection, and question answering.The suite combines new and existing finance benchmarks and is intended for public evaluation subject to licensing and privacy considerations.
2 Related Work
Related work covers general pre-trained language models, masking strategies, domain-specific models, and financial benchmarks. Existing financial evaluation is less exhaustive, motivating the heterogeneous FLUE suite.
- Pre-trained language models: Pre-trained language models use objectives including masked language modeling, denoising autoencoding, span prediction, and token detection.These objectives support different approaches to learning from unlabeled text.
- Masked language modeling: Masking research includes random token masking, rule-based domain strategies, and automatically learned masking policies.The cited approaches differ in how they select content to mask during pre-training.
- Domain-specific language models: Domain-specific models such as BioBERT, ClinicalBERT, and SciBERT show the use of specialized pre-training across biomedical, clinical, and scientific text.Prior financial work also fine-tuned BERT on financial corpora.
- Finance benchmarks: Financial benchmarks previously included datasets for sentiment analysis, question answering, news headlines, named entity recognition, and structure boundary detection.The related-work discussion characterizes the financial benchmark suite as less exhaustive than general-purpose GLUE.
- Finance benchmarks: FLUE assembles heterogeneous benchmark tasks for more comprehensive financial-language evaluation.The suite builds on existing literature while broadening coverage across financial NLP tasks.
3 Benchmarks (FLUE) and Datasets
FLUE is a public suite of five financial NLP tasks built from diverse datasets, while FLANG pre-training combines general English and finance-specific corpora. The benchmarks cover sentiment, headlines, entities, structure, and question answering.
- FLUE overview: FLUE is a comprehensive benchmark suite covering five financial tasks and providing dataset statistics and licensing information.The datasets are described as publicly usable with low ethical risk and no sensitive organizational or individual information.
- Sentiment analysis: Financial sentiment evaluation includes classification on Financial PhraseBank and regression on FiQA 2018 task 1.The datasets cover financial news sequences, headlines, and microblogs.
- News headline classification: The Gold news headline dataset contains 11,412 headlines annotated with nine binary labels.It evaluates dimensions of financial headlines beyond sentiment, including information related to price changes.
- Entity and structure tasks: The NER task identifies financial entities using the dataset released by Alvarado et al., while structure detection uses FinSBD-3 to locate textual and document-component boundaries.FinSBD-3 includes sentences, lists, list items, headers, footers, and tables.
- Question answering and pre-training data: The question-answering task uses FiQA 2018 Task 2, which evaluates answering finance-domain questions over financial data.Pre-training combines general English corpora with finance-specific sources including reports, calls, analyst reports, financial news, and Investopedia.
4 Model
The model combines finance-specific word and phrase masking with span-boundary training to learn representations for financial language. Its training and evaluation setup includes multiple model variants, benchmark tasks, and supervised contrastive fine-tuning.
- Financial masking: Financial terms are preferentially masked using a dictionary of more than 8200 words and phrases, while preserving a 15% total masking rate.Single-word financial tokens receive 30% masking probability, with other tokens masked at 70%.
- Model and evaluation setup: The experiments compare BERT, ELECTRA, finBERT, and FLANG variants across financial benchmarks, with supervised contrastive learning added for classification fine-tuning.The benchmark results are summarized across models, including sentiment regression, named entity recognition, news headline classification, structure boundary detection, and question answering.
- Phrase masking: Phrase masking replaces all tokens in a financial phrase with one [MASK] token and predicts the phrase through masked language modeling.Training first uses single-token masking, then adds phrase masking in a second phase.
- Span boundary objective: FLANG-ELECTRA adds a span boundary objective to the masked-language-modeling loss, alongside word and phrase masking.The combined loss includes the MLM, discriminator, and span boundary components.
- Span boundary objective: The span boundary objective uses representations before and after a masked contiguous span to generate its tokens and learn multi-word financial representations.This is intended to capture financial terms that are not present in the model vocabulary.
5 Experiments
The experiments pre-train and fine-tune ELECTRA-based variants on financial data, progressively adding financial vocabulary, phrase masking, span-boundary training, and contrastive loss. FLANG-ELECTRA is the final variant in this staged comparison.
- Multi-stage pre-training: Each model is further pre-trained for four epochs, using two epochs of single-token masking followed by two epochs of word-and-phrase masking.This multi-stage setup gives the lowest model perplexity in the cited experiment.
- Experiment setup: ELECTRA is used as the base architecture because its discriminator learns from all input tokens and fits preferential masking.The setup compares standard BERT and ELECTRA with financial-domain and FLANG variants.
- Multi-stage pre-training: FLANG-BERT is trained with single-token financial masking followed by word-and-phrase masking for comparison with finBERT.The staged procedure is also used for the ELECTRA-based variants.
- Model variants: The compared variants progressively add additional financial data, financial-vocabulary masking, phrase financial vocabulary, span-boundary objectives, and supervised contrastive loss.FLANG-ELECTRA combines additional data, phrase financial vocabulary, and the span boundary objective; contrastive loss is added during supervised classification fine-tuning.
5.2 Benchmark Results
Across financial benchmarks, FLANG models generally outperform prior baselines, with the strongest gains on tasks using finance-specific language and smaller gains on more domain-agnostic tasks.
- Financial PhraseBank Sentiment Classification: FLANG-ELECTRA achieves state-of-the-art sentiment classification on Financial PhraseBank, while FLANG-BERT substantially improves over FinBERT.The paper uses accuracy and marginal performance improvement over FinBERT because prior results were already in the higher 80s.
- FiQA Sentiment Regression: FLANG-ELECTRA achieves state-of-the-art sentiment regression on FiQA and outperforms both BERT and FinBERT.Evaluation uses Mean Squared Error and R Square.
- News Headline Classification: FLANG-ELECTRA obtains the highest mean F-1 score on nine binary news-headline classification tasks, while FLANG-BERT outperforms BERT.All deep-learning models outperform Support Vector Machines on this benchmark.
- Named Entity Recognition: The models outperform baselines in financial NER under multi-token labeling, but the improvement margin is more muted than on finance-specific tasks.The multi-token setting assigns the same label to all tokens belonging to a split word.
- Structure Boundary Detection: On structure boundary detection, the models perform similarly or slightly better than baselines, while remaining significantly better than FinBERT.The paper attributes the limited gains to SBD relying more on general language cues than finance keywords.
- Question Answering: On question answering, FLANG-BERT and FLANG-ELECTRA outperform other models across Precision, nDCG, and MRR, except SpanBERT with additional data.Higher nDCG and MRR indicate better ranking of relevant documents and earlier retrieval of the first relevant item.
5.3 Ablation Studies
Ablation studies identify preferential financial masking, multi-stage terminology masking, additional data, and the span boundary objective as important components of the training setup.
- Masking Strategy: Masking 30% of financial tokens gives each model the lowest perplexity.The result comes from the comparison of different masking percentages.
- Masking Strategy: Using single-word financial terms in the first two pre-training epochs and multi-word terms in the next two produces the lowest perplexity.This multi-stage schedule is evaluated in the terminology-masking ablation.
- FLUE Performance: Additional data and domain-specific preferential masking substantially increase performance on FLUE tasks.The comparison is reported in the ablation studies summarized in Table 9.
- Span Boundary Objective: Adding the Span Boundary Objective to the ELECTRA generator produces the best-performing model among comparable encoder architectures.The comparison includes SpanBERT, DistilBERT, and BERT.
- Perplexity: The proposed pre-training methodology yields the lowest perplexity compared with prior baselines.Table 12 averages perplexity over validation sentences using three runs.
6 Conclusion
The paper contributes two finance-domain language models and a five-task benchmark suite, reporting broad improvements over prior models and claiming that the masking approach generalizes beyond finance.
- Contributions: The authors contribute two language models using domain-specific word and phrase masking as a pre-training objective.The contribution is paired with a comprehensive finance benchmark suite spanning five natural-language tasks.
- Results: The language model outperforms previous language models on all reported benchmarks.The models, code, and benchmark data are intended for release.
- Generalization: The preferential masking methodology is presented as architecture- and domain-independent and applicable to other language-model training settings.This is stated as a generalization of the method beyond finance.
Ethics Statement
The ethics statement addresses dataset permissions, privacy risks, and the environmental cost of training large language models.
- Data Ethics: The benchmark datasets are reported to have low ethical risks and to contain no sensitive or personally identifiable information.The authors state that they obtained permission for each included dataset.
- Environmental Impact: The authors acknowledge the large carbon footprint of language-model training and report minimizing full-scale pre-training runs.They also state that the added objectives have minimal computational overhead relative to traditional BERT or ELECTRA pre-training.
Limitations
The work leaves several evaluation and modeling boundaries: FLUE omits abstractive generation and summarization, while the models exclude social-media pretraining and focus on English encoder-based settings. The approach may transfer to similarly structured domains, but obtaining suitable vocabulary resources can be difficult.
- FLUE omits abstractive generation and summarization tasks because large annotated datasets are unavailable.
- The pretraining data excludes social-media sources such as Twitter and Reddit because their informal language can impede formal and syntactical correctness.
- The models are trained and tested on English tasks, limiting expected performance on non-English text where domain-specific vocabulary is scarce.
- The methods may work in similarly structured domains such as clinical data, but suitable vocabulary term lists and dictionaries can be difficult to obtain.
- The study limits itself to encoder-based architectures, leaving models such as GPT3 and T5 for future work.
7 Appendix
The appendix documents the financial pretraining corpora, masking and perplexity studies, sentiment ablations, and supervised fine-tuning losses. Together, these details specify the data sources, training diagnostics, and objective functions used in the evaluation pipeline.
- Pretraining datasets: The pretraining-data summary reports dataset sizes and the percentage sampled in each training epoch.
- Pretraining datasets: Investopedia contributes 638 financial-concept articles containing terminology definitions and relationships among concepts.
- Pretraining studies: 30% financial-term masking yields the lowest perplexity in the masking study, while the total masked-token rate remains 15%.
- Pretraining studies: FLANG-BERT lowers validation perplexity relative to BERT and FinBERT, and financial data further lowers perplexity for ELECTRA models.
- Fine-tuning and objectives: Financial pretraining raises sentiment accuracy from 88.1% to 91.1%, financial-vocabulary masking reaches 91.4%, and span-boundary objectives improve it further.
- Fine-tuning and objectives: Supervised classification uses cross-entropy loss over N samples and C classes, with sentence-label pairs and predicted class probabilities.
- Fine-tuning and objectives: Supervised contrastive learning captures similarities among same-class examples and contrasts them with examples from other classes; λ weights the combined losses.