PaperlayerView exact revision on arXiv

Source-linked AI summary

FinBERT: Financial Sentiment Analysis with Pre-trained Language Models

Dogu Araci

arXiv:1908.10063v1cs.CLcs.LG

TL;DR

Financial sentiment analysis is challenged by specialized language and scarce labeled data. The paper introduces FinBERT, a BERT-based model further pre-trained for finance and fine-tuned for sentiment analysis, achieving state-of-the-art results on two datasets, including a 15% accuracy increase for classification.

  • Problem

    Financial sentiment analysis faces specialized language and a shortage of costly expert-labeled financial text.

  • Method

    FinBERT applies BERT to financial NLP by further pre-training on a financial corpus and fine-tuning for sentiment analysis.

  • Results

    15% accuracy increase over the state of the art was achieved for the classification task, with state-of-the-art results on both datasets.

  • Takeaways & Limitations

    Pre-trained language models can perform effectively on financial sentiment analysis with small labeled datasets, including 500 training examples.

  • Takeaways & Limitations

    FinBERT extracts explicit sentiments but modeling implicit information remains a challenging task.

Abstract

from arXiv · show

Financial sentiment analysis is a challenging task due to the specialized language and lack of labeled data in that domain. General-purpose models are not effective enough because of the specialized language used in a financial context. We hypothesize that pre-trained language models can help with this problem because they require fewer labeled examples and they can be further trained on domain-specific corpora. We introduce FinBERT, a language model based on BERT, to tackle NLP tasks in the financial domain. Our results show improvement in every measured metric on current state-of-the-art results for two financial sentiment analysis datasets. We find that even with a smaller training set and fine-tuning only a part of the model, FinBERT outperforms state-of-the-art machine learning methods.

1 INTRODUCTION

Financial sentiment analysis is difficult because financial text requires costly expert labeling and uses specialized, often vague language. The thesis tests whether pre-trained language models, including domain-specific further pre-training, can improve sentiment prediction with limited labeled data.

  • Motivation: Automated analysis is motivated by the volume of financial news, analyst reports, and company announcements that is too large for manual processing by one entity.
  • Motivation: Financial sentiment analysis classifies financial text as positive, negative, or neutral, but requires costly expert labeling and specialized-domain modeling.The thesis focuses on sentiment toward the financial actor depicted in a sentence, using Financial PhraseBank and FiQA sentiment scoring.
  • Related approaches: Word-counting and lexicon-based methods do not adequately capture the deeper semantic meaning of financial text.
  • Approach: Pre-trained language models can reduce labeled-data requirements by learning from large unlabeled corpora before downstream fine-tuning.
  • Approach: Domain-specific further pre-training can teach models semantic relations and vocabulary whose distribution differs substantially from general corpora.
  • Contributions: The thesis introduces FinBERT, evaluates it on two financial sentiment datasets, and reports state-of-the-art performance while examining alternative models and training strategies.

2 RELATED LITERATURE

Prior financial sentiment work includes word-counting, lexicon-based, and neural methods, but limited labeled financial data constrains neural approaches. Pre-trained language models offer transfer learning through large-corpus language modeling and task-specific fine-tuning.

  • Financial sentiment analysis: Word-counting methods cannot represent sequence-dependent semantic information, while deep learning represents text as sequences of embeddings.
  • Financial sentiment analysis: Financial sentiment analysis commonly targets how markets may react to information in financial text, using machine learning and lexicon-based approaches.
  • Financial sentiment analysis: Kraus and Feuerriegel used an LSTM for financial polarity analysis and found that pre-training on a larger corpus improved results, but used labeled pre-training.
  • Research gap: Limited large labeled financial datasets make it difficult to use neural networks fully for financial sentiment analysis.
  • Transfer learning: Initializing nearly the entire model with pre-trained values may address the need to learn complex relations from relatively little labeled data.
  • Transfer learning: Language-model transfer learning pre-trains on large corpora and adds task-specific layers before fine-tuning on target datasets.
  • Pre-trained models: ELMo provides contextualized word representations, whereas ULMFit fine-tunes the whole pre-trained language model with the downstream task layers.
  • Pre-trained models: BERT extends this approach with masked-token prediction, sentence-pair classification, and a large network trained on an unusually large corpus.

3 METHOD

The method section introduces FinBERT as the paper’s BERT-based implementation for financial-domain language modeling and sentiment analysis.

  • FinBERT: FinBERT is the paper’s implementation of BERT for the financial domain.

3.1 Preliminaries

The preliminaries describe recurrent and Transformer-based language-model architectures and the pre-training principles underlying ELMo, ULMFit, and BERT. These models use contextual or bidirectional representations and are adapted for downstream tasks through fine-tuning.

  • LSTM: LSTM is a recurrent architecture that preserves long-term sequence dependencies through forget and update gates.
  • LSTM: GloVe represents words using an unsupervised word-word co-occurrence objective but does not contextualize representations by sequence.
  • ELMo: ELMo generates contextualized word representations with a bidirectional language model built from multiple LSTM layers.
  • ULMFit: ULMFit fine-tunes an entire AWD-LSTM language model together with task-specific layers and uses specialized dropout tuning.
  • ULMFit: ULMFit also provides strategies for further pre-training on domain-specific corpora and fine-tuning downstream tasks.
  • Transformer: The Transformer models sequential information with stacked layers containing multi-headed self-attention and feed-forward networks.
  • BERT: BERT uses stacked Transformer encoders, predicts randomly masked tokens, and trains on next-sentence prediction.
  • BERT: BERT classification inputs include token and position embeddings plus [CLS] and [SEP] markers, with [CLS] used for classification tasks.

3.2 BERT for financial domain: FinBERT

FinBERT adapts BERT to financial NLP through domain-specific further pre-training and task-specific fine-tuning. The procedure also applies techniques intended to reduce catastrophic forgetting during fine-tuning.

  • FinBERT further pre-trains BERT on financial-domain text before training a classifier on labeled sentiment data.The paper experiments with both a relatively large financial corpus and sentences from the target classification dataset.
  • Sentiment classification adds a dense layer after the [CLS] token’s final hidden state and trains it on labeled sentiment data.
  • The fine-tuning procedure uses slanted triangular learning rates, discriminative fine-tuning, and gradual unfreezing to address catastrophic forgetting.
  • Discriminative fine-tuning assigns lower learning rates to lower network layers, reflecting different roles for language and classification information.The method uses α_l−1 = θα_l for the learning rate of the preceding layer.
  • Gradual unfreezing begins with only the classifier trainable and progressively unfreezes layers from the highest upward.This initially protects lower-level language information learned during pre-training from being forgotten.

4 EXPERIMENTAL SETUP

The experiments evaluate FinBERT against alternative transfer-learning methods and state-of-the-art systems across discrete and continuous financial sentiment tasks. They also test domain adaptation, fine-tuning strategies, encoder layers, and the amount of fine-tuning required.

  • The study compares FinBERT with ELMo and ULMFit for short-sentence classification.
  • The research questions include comparisons with state-of-the-art financial sentiment systems for both discrete and continuous targets.
  • The experiments examine how further pre-training on financial or target-domain corpora affects classification performance.
  • The study tests slanted triangular learning rates, discriminative fine-tuning, and gradual unfreezing for their effects on performance and catastrophic forgetting.
  • Additional experiments investigate which encoder layer performs best and how many layers require fine-tuning for comparable performance.

4.2 Datasets

The experiments use a financial news corpus for further pre-training, two sentiment datasets with different target types, and comparison models based on LSTM, ELMo, and ULMFit.

  • TRC2-financial contains 46,143 Reuters documents, more than 29M words, and nearly 400K sentences after financial-keyword filtering.It is a subset of Reuters’ TRC2 corpus, whose articles were published between 2008 and 2010.
  • Financial PhraseBank contains 4,845 English financial-news sentences annotated by 16 finance and business experts.Annotations reflect how each sentence might affect the mentioned company’s stock price, with annotator agreement information also included.
  • FiQA Task 1 contains 1,174 financial headlines and tweets with continuous sentiment scores ranging from −1 to 1.Each example also identifies the financial entity targeted by the sentence.
  • Contrastive baselines include LSTM classifiers with GloVe or ELMo embeddings and a ULMFit classifier.The paper cautions that these baselines were not tested as thoroughly as BERT, so comparisons are not definitive method rankings.

4.4 Evaluation Metrics

The evaluation uses accuracy, cross entropy loss, and macro F1, with weighted loss to address label imbalance. BERT training uses specified regularization, optimization, and staged unfreezing settings.

  • Evaluation uses Accuracy, cross entropy loss, and macro F1 average for classification models.Macro F1 averages the F1 scores computed separately for each class.
  • Cross entropy loss is weighted by the square root of inverse label frequency to address Financial PhraseBank’s label imbalance.A label comprising 25% of examples receives a weight of 2.
  • BERT uses dropout p = 0.1, warm-up proportion 0.2, maximum sequence length 64, learning rate 2e−5, and mini-batches of 64.
  • Training runs for 6 epochs, selects the best validation model, and begins with only the classification layer unfrozen.The next layer is unfrozen after each third of a training epoch.

5 EXPERIMENTAL RESULTS (RQ1 & RQ2)

FinBERT outperforms the implemented and previously reported methods across measured Financial PhraseBank metrics, while ULMFit also benefits from language-model pre-training. Across training-set sizes and on FiQA, the results support strong performance with limited labeled data.

  • Financial PhraseBank results are reported for both the complete dataset and the subset with 100% annotator agreement.
  • FinBERT performs best on all measured Financial PhraseBank metrics among implemented and previously reported methods.The comparison includes LSTM, ULMFit, LPS, HSC, and FinSSLX.
  • ULMFit significantly improves all metrics and outperforms the machine-learning methods LPS and HSC.The authors attribute this result to language-model pre-training and training strategies that help with the small dataset.
  • 80% accuracy is achieved by FinBERT with 250 training examples, while all models improve as training data increases.ULMFit and FinBERT outperform LSTM classifiers trained on the whole dataset at this training size.
  • FinBERT outperforms ULMFit and consequently all other methods in every metric.
  • FinBERT outperforms state-of-the-art models on the FiQA sentiment dataset for both MSE and R2.FinBERT is evaluated with 10-fold cross-validation, whereas the compared papers use the official test set.

6 EXPERIMENTAL ANALYSIS

The experiments evaluate FinBERT across pre-training, fine-tuning, encoder-layer, and failure-analysis settings. Results show benefits from domain pre-training and selected training strategies, while partial fine-tuning can approach full-model performance.

  • Financial sentiment evaluation: FinBERT outperforms state-of-the-art models on FiQA sentiment scoring for both MSE and R2.The evaluation uses 10-fold cross validation rather than the official test set used by the comparison papers.
  • Effects of further pre-training: Domain-corpus pre-training performs best among Vanilla BERT, FinBERT-task, and FinBERT-domain, although the difference is not very high.The authors suggest several possible explanations, including distribution differences and limited room for improvement.
  • Catastrophic forgetting: Applying slanted triangular learning rates, gradual unfreezing, and discriminative fine-tuning together produces the best test loss and accuracy.The authors attribute the strategy design to fine-tuning higher-level features more than lower-level language-model features.
  • Catastrophic forgetting: Without the training strategies, validation loss rises after the first epoch, whereas all three strategies produce a more stable trajectory.The authors identify gradual unfreezing as the most important technique in this setting.
  • Encoder-layer analysis: The last BERT encoder layer contributes most to classification performance across all measured metrics.The comparison includes individual representations from all 12 Transformer encoder layers and their average.
  • Training only a subset of the layers: Fine-tuning after Layer-9 performs virtually the same as broader fine-tuning, while fine-tuning the whole model remains best.The result indicates that full-model training is not mandatory, allowing a trade-off between training time and performance.
  • Where does the model fail?: 73% of failures occur between positive and negative labels, compared with 5% for negative and positive.The analysis also discusses difficulty distinguishing positive outlook from objective observation and relates this to annotator disagreement.

7 CONCLUSION AND FUTURE WORK

FinBERT applies BERT to financial sentiment analysis through domain-specific further pre-training and fine-tuning, achieving state-of-the-art results on both evaluated datasets. The experiments also examine data efficiency, domain pre-training, learning-rate strategies, and partial fine-tuning, while identifying implicit sentiment modeling as an important future challenge.

  • FinBERT achieved state-of-the-art results on both financial sentiment datasets by further pre-training BERT on financial text and fine-tuning it for sentiment analysis.
  • 15% improvement in accuracy over the state-of-the-art was achieved for the classification task.
  • FinBERT surpassed prior state-of-the-art results with a training set as small as 500 examples, despite the complete dataset containing more than 3000 examples.
  • Further domain-specific pre-training was not significantly better in this case, while more aggressive higher-layer fine-tuning performed better and helped prevent catastrophic forgetting.
  • Comparable performance was achieved with much less training time by fine-tuning only BERT’s last 2 layers.
  • FinBERT extracts explicit sentiment, but modeling implicit information and applying it to stock-return data remain proposed extensions.
Loading 1908.10063v1…