PaperlayerView exact revision on arXiv

Source-linked AI summary

Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis

Agam Shah, Suvan Paturi, Sudheer Chava

arXiv:2305.07972v1cs.CLq-fin.CP

TL;DR

FOMC communications are important to financial markets, but traditional positive-versus-negative sentiment does not capture monetary policy stance reliably. The paper builds annotated FOMC text datasets, introduces hawkish-dovish classification, and uses a benchmarked model to construct a stance measure; the authors report relationships with economic and Treasury indicators and a simple strategy that outperforms QQQ over the last decade.

  • Problem

    Positive-versus-negative sentiment can miss whether FOMC text is hawkish or dovish, despite the importance of these communications to markets.

    Source: §1 · p. 1

  • Method

    The authors collect and annotate FOMC minutes, press conference transcripts, and speeches, develop a hawkish-dovish classification task, benchmark models, and use the best-performing model to construct a document-level stance measure.

    Source: §1 · p. 2, §3.1 · p. 3, §5 · p. 6

  • Results

    The stance measure is studied in relation to CPI, PPI, and Treasury yields, and the authors report that a simple trading strategy outperforms QQQ over the last decade.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The paper releases its dataset, models, code, and benchmark data for further FOMC text analysis.

    Source: §6 · p. 9

    The measure uses only FOMC minutes, speeches, and press conferences, excludes audio and video, and relies on rule-based sentence splitting that the authors leave open to improvement.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Monetary policy pronouncements by Federal Open Market Committee (FOMC) are a major driver of financial market returns. We construct the largest tokenized and annotated dataset of FOMC speeches, meeting minutes, and press conference transcripts in order to understand how monetary policy influences financial markets. In this study, we develop a novel task of hawkish-dovish classification and benchmark various pre-trained language models on the proposed dataset. Using the best-performing model (RoBERTa-large), we construct a measure of monetary policy stance for the FOMC document release days. To evaluate the constructed measure, we study its impact on the treasury market, stock market, and macroeconomic indicators. Our dataset, models, and code are publicly available on Huggingface and GitHub under CC BY-NC 4.0 license.

1 Introduction

Large market moves around FOMC announcements motivate measuring policy stance directly from Fed communications, since positive-versus-negative sentiment can miss whether a statement is hawkish or dovish. The paper introduces a hawkish-dovish task and annotated datasets, benchmarks models, and constructs an aggregate stance measure.

1 Introduction

  • An eight-minute Jackson Hole speech was followed by an almost $3 trillion decline in U.S. equity value that day and more than $6 trillion over the next three days.The authors use these market shifts to motivate measuring policy stance in FOMC communications.Source: §1 · p. 1
  • Positive-versus-negative sentiment misses policy stance: “increase” can describe employment or inflation, while finance sentiment models classify both examples as positive.The paper argues this motivates a dedicated hawkish-versus-dovish task with annotated data.Source: §1 · p. 1
  • The authors characterize the traditional rule-based approach as rudimentary for measuring monetary policy stance from text.They use rule-based models as a baseline in performance analysis.Source: §1 · pp. 1–2
  • The paper introduces sentence classification into hawkish versus dovish, rather than positive versus negative, and builds clean, tokenized, annotated datasets for FOMC minutes, press conferences, and speeches.The collected texts span 1996–2022; the paper also benchmarks rule-based through fine-tuned language models and zero-shot ChatGPT.Source: §1 · p. 2
  • The paper constructs an aggregate monetary policy stance measure and evaluates it by its ability to predict financial and economic indicators.The introduction also identifies sentence splitting as a way to handle counterfactual sub-sentences that moderate a statement’s tone.Source: §1 · p. 2

2 Related Work

Prior finance NLP largely studies news, filings, and earnings calls, while financial-language benchmarks have not included macroeconomics-based tasks. FOMC research finds that central-bank communication moves markets, but earlier work used methods such as topic models, graph clustering, and unsupervised approaches.

NLP in Finance

  • Finance NLP commonly analyzes news, SEC filings, and earnings calls; a financial-language benchmark proposed by Shah et al. (2022) does not include macroeconomics-based tasks.Finance-specific language models have advanced benchmark performance, but the cited benchmark omits this task domain.Source: §2 · p. 2

FOMC and Text Analysis

  • Studies report that central-bank communications move markets, while the cited literature does not leverage transformer-based models for that analysis.The reported market effect can vary with communication style in a study of Czech, Hungarian, and Polish central banks.Source: §2 · p. 2
  • LDA studies use FOMC statements or minutes to examine rates, forward guidance, and market-moving information; Hansen and McMahon (2016) suggest minutes and speeches may add insight beyond post-meeting statements.Jegadeesh and Wu (2017) note that minutes released weeks after a meeting still carry pertinent market-moving information.Source: §2 · p. 2
  • Prior FOMC text work also used word embeddings, graph clustering, and unsupervised extraction; the multimodal MONOPOLY dataset includes audio and video but misses the 2000–2002 and 2007–2008 downturns.Zirn et al. (2015) used graph clustering to estimate hawkish-dovish stance in response to scarce annotated data.Source: §2 · pp. 2–3

3 Dataset

The dataset combines FOMC minutes, press-conference transcripts, and speeches, then filters and samples text for manual hawkish-dovish-neutral annotation. It also pairs the text with Treasury, QQQ, CPI, and PPI data.

3.1 FOMC Data

  • Meeting minutes and speeches span January 1, 1996–October 15, 2022; press conferences span April 27, 2011–October 15, 2022.Minutes are reports from eight scheduled annual FOMC meetings; press-conference transcripts cover prepared remarks and Q&A, while speeches are talks by Federal Reserve officials.Source: §3.1 · p. 3
  • The raw corpus is dominated by minutes and speeches; press-conference data are reduced by recency and by retaining only chair statements that are not questions.The authors report that raw text also contains unrelated sentences and is too large for manual labeling, motivating filtration and sampling.Source: §3.1 · p. 3
  • Dictionary filtering selects target sentences; for speeches, the same dictionary is applied to titles to isolate relevant files, yielding the greatest target-sentence count per file.Sentences containing words in specified dictionary panels are retained, and the speech-title filter helps address the prevalence of non-monetary-policy speeches.Source: §3.1 · p. 3
  • The labeled samples contain 1,070 meeting-minutes sentences, 315 press-conference sentences, and 994 speech sentences, annotated independently by two annotators using a predefined guide.Dovish labels indicate future easing, hawkish labels future tightening, and neutral labels mixed tone, no policy change, or text not directly related to stance.Source: §3.1 · p. 4
  • A custom keyword splitter separates mixed-tone statements at contrastive connectives when each resulting segment contains a key word.The authors first tried SentiBigNomics but switched after it performed poorly; Table 4 reports labeled-sentence counts before and after splitting for each event.Source: §3.1 · p. 4, Table 4 · p. 4

3.2 Economic Data

  • The economic data comprise monthly CPI and PPI inflation measured as year-over-year percentage changes, daily Treasury yields across maturities, and daily adjusted QQQ closing prices.CPI and PPI come from FRED, Treasury yields from the U.S. Department of the Treasury, and QQQ prices from Yahoo Finance.Source: §3.2 · p. 5

4 Models

The paper benchmarks a dictionary rule, recurrent neural networks, pretrained transformers, and zero-shot ChatGPT for monetary-stance classification. The provided passages describe model choices and experimental setup, but not comparative results.

4.1 Rule-Based

  • The rule-based benchmark applies a financial-word dictionary to test sets formed with an 80:20 training-test split, assigning stance from combinations of word panels and reversing labels when panel C appears.The approach follows Gorodnichenko et al. (2021).Source: §4.1 · p. 5

4.2 LSTM & Bi-LSTM

  • LSTM and Bi-LSTM models classify sentence vectors built from a 2,000-word vocabulary, with padding, masking, dropout, and varying epochs and batch sizes.The vocabulary covers over 99% of meeting-minutes and press-conference words and around 91% of speech words.Source: §4.2 · p. 5

4.3 PLMs

  • The pretrained-transformer benchmark includes five smaller models and BERT-large and RoBERTa-large, fine-tuned after grid search over learning rates and batch sizes without additional pretraining.The models are initialized from pretrained versions in Hugging Face Transformers.Source: §4.3 · pp. 5–6

4.4 ChatGPT

  • The ChatGPT benchmark uses zero-shot gpt-3.5-turbo with a prompt requesting hawkish, dovish, or neutral labels and a brief explanation.The model is run with temperature 0.0 and a 1,000-token output limit.Source: §4.4 · p. 6

5 Results and Analysis

Fine-tuned PLMs outperform rule-based and recurrent models, with RoBERTa-large leading most dataset categories; the resulting stance measure tracks inflation indicators and is associated with Treasury yields. A QQQ strategy based on the measure reports higher returns than buy-and-hold over the evaluated period.

5.1 Model Performance

  • Rule-based models score around 0.5 F1, while LSTM and Bi-LSTM offer little improvement and struggle most on the smaller press-conference datasets.The RNNs work best on meeting minutes and speeches; the authors identify the limited results as motivation for transformer-based models.Source: §5.1 · p. 6
  • RoBERTa-large leads the large-model and overall comparisons except on PC-S, while fine-tuned PLMs outperform rule-based and LSTM models.RoBERTa-base leads most base-model datasets, with exceptions for after-split meeting minutes and press-conference data.Source: §5.1 · p. 6
  • Sentence splitting improves performance on meeting-minutes and press-conference data, but not speeches; average gains are higher for base than large models.The authors say splitting is intended to improve the document-level stance measure, not classification performance itself.Source: §5.1 · pp. 6–7
  • Zero-shot ChatGPT outperforms rule-based and fine-tuned RNN models but underperforms fine-tuned PLMs across datasets.The authors caution that ChatGPT is not a good baseline.Source: §5.1 · p. 7

5.2 Hawkish Measure Construction

  • RoBERTa-large labels filtered sentences from minutes, speeches, and press conferences to construct a document-level hawkishness measure.The measure is based on the labeled sentences in each document.Source: §5.2 · p. 7

5.3 Market Analysis

  • The measure has statistically significant positive correlations with CPI and PPI percentage changes across all three data classes, with correlations increasing over time.The authors report that the meeting-minutes measure captures inflation and deflation periods fairly well.Source: Table 6 · p. 7, §5.3 · p. 8, Figure 1 · p. 8
  • All reported Treasury-yield regressions are statistically significant, and the 1-year yield is most sensitive to monetary-policy changes.The regressions use measures from meeting minutes, speeches, and testimonies across 3-month, 1-year, and 10-year maturities.Source: Table 7 · p. 8, §5.3 · p. 9
  • 163.4% excess return over buy-and-hold is reported for the measure-based QQQ strategy as of September 21, 2022.The strategy shorts QQQ when the measure is positive and goes long when it is negative; buy-and-hold remains long QQQ.Source: Figure 2 · p. 9, §5.3 · p. 9

6 Conclusion

The paper releases a cleaned, tokenized, and labeled dataset for three types of FOMC text and introduces sentence-level monetary-policy stance classification. It applies a trained model to construct a measure validated against inflation and Treasury yields, and reports that a simple strategy outperforms the QQQ benchmark.

6 Conclusion

  • The paper releases an open-source dataset of FOMC meeting minutes, speeches, and press-conference transcripts, alongside models and code on Hugging Face and GitHub.The dataset is cleaned, tokenized, and labeled.Source: §6 · p. 9
  • The proposed sequence-classification task labels sentences as hawkish, dovish, or neutral and supports a trained-model measure of monetary-policy stance.The measure is studied in relation to CPI, PPI, and Treasury yields.Source: §6 · p. 9
  • The paper reports that its simple trading strategy outperforms the QQQ index benchmark over the last decade.Source: §6 · p. 9

Limitations

The study is limited to selected FOMC text sources and uses a simple rule-based sentence-splitting approach. Its trading strategy excludes transaction fees, while several extensions are left for future work.

Limitations

  • The analysis uses only meeting minutes, speeches, and press conferences, and excludes audio/video features and other potential text sources.The authors identify congressional and senate testimonies, beige books, and green books as possible additions.Source: Limitations · p. 9
  • Sentence splitting uses a simple rule-based approach, leaving better methods for sentences with opposite tones as an open problem.Source: Limitations · p. 9
  • The trading strategy excludes transaction fees; high-frequency strategy construction is suggested as future work.The authors connect the fee exclusion to the strategy's low-frequency trading.Source: Limitations · p. 9
  • The authors also propose a more comprehensive zero-shot and few-shot generative LLM benchmark using open-source models as future work.Source: Limitations · p. 9

Ethics Statement

The study acknowledges geographic and gender biases in its data and limits model development to fine-tuning existing PLMs because of pretraining's carbon footprint. The authors report that the public data and models raise no ethical concerns under their stated conditions.

Ethics Statement

  • The study covers only the U.S. Federal Reserve and notes that a female chair served for 4 of the 27 years studied.Source: Ethics Statement · p. 10
  • Because large-PLM pretraining has a substantial carbon footprint, the authors limit their work to fine-tuning existing PLMs.Source: Ethics Statement · p. 10
  • The authors state that public raw data and publicly available models licensed for their purpose raise no ethical concerns in this study.Source: Ethics Statement · p. 10

A FinBERT Sentiment Analysis

The authors test whether financial sentiment labels can stand in for monetary-policy stance, but find that the FinBERT sentiment model performs poorly on hawkish-dovish classification. They present this result as support for a task-specific dataset and classification task.

A FinBERT Sentiment Analysis

B Transfer Learning

Training on meeting minutes and press conferences yielded a lower speech-test F1 score than training on speech data. The annotation guide shaped labels across categories, and annotator disagreements were resolved through the guide and, when needed, online resources.

B Transfer Learning

C.1 Annotation Agreement

  • Disagreements between annotators were resolved using the annotation guide, with uncovered cases checked against online resources and added to the guide.Source: §C.1 · p. 12

C.2 Annotation Guide

  • The annotation guide organized sentences into categories and was used by two independent annotators to assign dovish, hawkish, or neutral labels.Source: §C.2 · pp. 12, 14

D Robustness check

A time-based evaluation found that RoBERTa-large’s performance was not driven by look-ahead bias. The check trained on 1996–2019 data and tested on 2020–2022 data.

D Robustness check

  • Because the dataset is temporal and RoBERTa was trained on data available before mid-2019, predictions could reflect future knowledge about past sentences.The authors call this potential issue look-ahead bias.Source: D Robustness check · p. 14
  • Average weighted F1 was 0.7114 across three seeds, validating that RoBERTa-large’s performance was not driven by look-ahead bias.The evaluation used a time-based split of Combined-S data.Source: D Robustness check · p. 14
  • The time-based split trained on Combined-S data from 1996 to 2019 and tested on data from 2020 to 2022.Source: D Robustness check · p. 14

Related papers

Loading 2305.07972v1…