PaperlayerView exact revision on arXiv

Source-linked AI summary

BloombergGPT: A Large Language Model for Finance

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, Gideon Mann

arXiv:2303.17564v3cs.LGcs.AIcs.CLq-fin.GN

TL;DR

Financial NLP requires specialized language understanding, yet prior LLM research had not reported a dedicated financial model. BloombergGPT trains a 50 billion parameter model on mixed financial and general-purpose data and evaluates it across financial and general benchmarks. It achieves strong financial-task performance while retaining strong general LLM benchmark results.

  • Problem

    Financial NLP covers complex, specialized tasks, but the literature had not reported an LLM specialized for the financial domain.

  • Method

    BloombergGPT combines curated financial documents in FinPile with public general-purpose data, producing a roughly half-domain-specific, half-general-purpose training corpus.

  • Results

    BloombergGPT outperforms comparable models on financial tasks while achieving strong results on general LLM benchmarks.

  • Takeaways & Limitations

    Mixing domain-specific and general-purpose data balances performance across financial and general domains.

  • Takeaways & Limitations

    The paper situates BloombergGPT within broader ethical, use, and limitation concerns for large language models.

Abstract

from arXiv · show

The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering. Large Language Models (LLMs) have been shown to be effective on a variety of tasks; however, no LLM specialized for the financial domain has been reported in literature. In this work, we present BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data. We construct a 363 billion token dataset based on Bloomberg's extensive data sources, perhaps the largest domain-specific dataset yet, augmented with 345 billion tokens from general purpose datasets. We validate BloombergGPT on standard LLM benchmarks, open financial benchmarks, and a suite of internal benchmarks that most accurately reflect our intended usage. Our mixed dataset training leads to a model that outperforms existing models on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. Additionally, we explain our modeling choices, training process, and evaluation methodology. We release Training Chronicles (Appendix C) detailing our experience in training BloombergGPT.

1 Introduction

Large language models have demonstrated broad capabilities, but financial NLP’s specialized terminology and use cases motivate a dedicated model. BloombergGPT addresses this gap by mixing curated financial and general-purpose data, then evaluating both domain-specific and general performance.

  • 1 Introduction: Large language models gained broad task abilities and few-shot prompting through scale, but most research focused on general-purpose capabilities.Models expanded from GPT-3’s 175 billion parameters to systems reaching 1 trillion parameters, while few-shot prompting broadened supported tasks.
  • 1 Introduction: Financial NLP spans sentiment analysis, named entity recognition, news classification, and question answering, but its terminology and complexity warrant domain-specific systems.The paper frames financial technology as a large, growing area where NLP has an increasingly important role.
  • 1 Introduction: BloombergGPT is a 50 billion parameter model designed to achieve best-in-class financial performance while retaining competitive general-purpose benchmark results.The model targets Bloomberg’s diverse applications, most of which are within finance, while still supporting broad task coverage.
  • 1 Introduction: The paper’s mixed-data strategy combines Bloomberg’s curated financial archives with public datasets into a corpus exceeding 700 billion tokens.Bloomberg’s financial data analysts curated documents over forty years, and the model was trained using part of the resulting corpus.
  • Broader Contributions: BloombergGPT’s alternative to exclusively domain-specific or general-purpose training mixes both data types, achieving strong domain performance while maintaining general benchmark performance.The paper presents this mixed approach as previously unstudied for domain-specific LLMs.
  • Broader Contributions: The work emphasizes curated, reliable training data and evaluates both public financial benchmarks and internal tasks aligned with intended use cases.It contrasts curated sources with common web-scraped corpora and notes that domain evaluations may not match practical usage.

2 Dataset

BloombergGPT’s training corpus combines FinPile, a curated collection of financial documents, with public general-purpose datasets to support both financial and broader language-model capabilities.

  • FinPile: FinPile combines financial news, filings, press releases, web-scraped documents, and social media from Bloomberg archives.The corpus draws on documents acquired through Bloomberg’s business processes and curated over multiple decades.
  • Corpus composition: The full training set is roughly half domain-specific text and half general-purpose text, with each dataset deduplicated to improve data quality.The included datasets are FinPile, The Pile, C4, and Wikipedia.
  • FinPile: FinPile includes public, purchased, and private documents, with cleaning that removes markup, special formatting, and templates.The collection is time-stamped from 2007-03-01 to 2022-07-31, although date information is not used in this work.
  • FinPile: Company filings include dense financial statements such as 10-K and 10-Q reports, providing information considered important for financial decision-making.Most filings come from EDGAR, the SEC’s online database, and are processed and normalized in Bloomberg.
  • FinPile: Bloomberg-authored news and documents such as opinions and analyses focus on content relevant to the financial community and vary in length.Bloomberg News and Bloomberg First Word are identified as the largest sources within this category.
  • Public datasets: Public augmentation uses The Pile, C4, and English Wikipedia to provide cleaned, diverse, and more up-to-date general-language data.The Pile supports domain diversity and generalization, C4 adds separately processed cleaned web text, and Wikipedia adds newer factual content than older copies in The Pile and C4.

3 Model

BloombergGPT is a BLOOM-based decoder-only causal language model whose 50B-parameter design balances compute, data availability, and hardware-efficient architecture choices. Training uses mixed-domain data, fixed-length causal sequences, and distributed-memory techniques for large-scale optimization.

  • Architecture: BloombergGPT uses a decoder-only causal language-model architecture based on BLOOM, with 70 transformer decoder layers.
  • Architecture: The architecture uses self-attention, layer normalization, GELU feed-forward networks, ALiBi positional encoding, tied embeddings, and an added embedding layer normalization.The embedding modification introduces consecutive layer normalizations in its second term.
  • Size: ~700B tokens is insufficient for a Chinchilla-optimal configuration under the compute budget, while FinPile is kept above half of total training data.The authors note that tokenizer-specific scaling-law transfer remains an open question.
  • Size: 50B parameters is selected from Chinchilla scaling calculations under a 1.3M-GPU-hour compute budget, while preserving roughly 30% for failures, retries, and restarts.Activation checkpointing adds 0.33x TFLOPs per iteration, so the calculations use 0.75 × 1.3M GPU hours.
  • Shape: The target shape is 70 layers and a hidden dimension near 7,510, adjusted to 40 attention heads of dimension 192 for divisibility and Tensor Core efficiency.
  • Training: Training uses 2,048-token left-to-right sequences formed by concatenating documents with end-of-text separators, allowing multiple domains per sequence.
  • Optimization: Large-scale training relies on AdamW, learning-rate warmup and cosine decay, ZeRO stage 3, model parallelism, activation checkpointing, and 512 A100 GPUs.ZeRO shards parameters, gradients, and optimizer state across GPUs, while activation checkpointing trades memory for recomputation.

4 Training Run

The training run tracked smoothed training loss and periodically measured validation loss, with optimization interventions after validation loss flattened or increased. Training stopped when validation progress ceased, and the checkpoint selected by validation and downstream evaluations became the final model.

  • Monitoring: Figure 2 plots smoothed training loss as solid lines and held-out validation loss as dotted lines, omitting unsuccessful or superseded runs.
  • Monitoring: Training loss is smoothed with a running average using α = 0.001, whereas validation loss is measured on the entire validation set every 300 steps.
  • Duration: 139,200 steps over approximately 53 days consumed 569B of 709B available tokens, covering about 80% of one data epoch.
  • Interventions: After step 115,500, flat or increasing validation loss prompted learning-rate reductions and the addition of dropout.The learning rate was reduced at steps 115,500, 129,900, and 137,100; dropout with probability 0.1 was added at step 129,900.
  • Stopping criterion: Step 146,000 ended the run because validation loss showed no observable progress, while step 139,200 was selected as the final checkpoint.Selection used validation loss and downstream evaluations.

5 Evaluation

BloombergGPT is evaluated across finance-specific and general-purpose benchmarks against similarly sized and larger models. It performs strongly across financial tasks while remaining competitive on general-purpose evaluations.

  • Evaluation setup: The evaluation covers finance-specific tasks, general-purpose benchmarks, and internal Bloomberg tasks aligned with intended use cases.Finance-specific datasets span multiple NLP tasks, while internal tasks target Bloomberg applications.
  • Heldout loss: BloombergGPT consistently achieves lower heldout bits per byte than other models, with the largest gap on specialized Filings documents.The heldout data is temporally separated and deduplicated from training data, and the Filings category is especially challenging for existing models.
  • External financial tasks: BloombergGPT performs best on four of five external financial tasks and has the highest win rate among tested models.It leads on ConvFinQA, FiQA SA, FPB, and Headline, and ranks second on NER.
  • Internal financial tasks: BloombergGPT outperforms other models by wide margins across most internal sentiment and entity-linking tasks, except social media sentiment.On internal aspect-specific sentiment tasks, gains range from at least 25 to over 60 points on three tasks; on social media data it ranks second behind BLOOM176B.
  • General-purpose evaluation: General-purpose results place BloombergGPT behind some much larger models but generally ahead of similarly sized alternatives.It is often competitive with models containing hundreds of billions of parameters while retaining a smaller parameter count.
  • General-purpose evaluation: Across dozens of tasks, BloombergGPT is the strongest model among compared models with tens of billions of parameters and sometimes matches or exceeds much larger models.It also achieves an 85% win rate among evaluated models.

6 Qualitative Samples

Qualitative examples show BloombergGPT applying financial and news-domain knowledge to practical language interfaces. Demonstrations include generating BQL, suggesting headlines, and answering financial questions.

  • Generation of Bloomberg Query Language: BloombergGPT transforms natural-language requests into valid Bloomberg Query Language, making a powerful but complex data-retrieval tool more accessible.BQL supports different security classes, fields, functions, and parameters.
  • Suggestion of News Headlines: BloombergGPT generates short news-headline suggestions from article text without requiring a dedicated task-specific model.The paper presents headline generation as a use case for journalists and newsletters.
  • Financial Question Answering: BloombergGPT correctly identifies company CEOs in financial question-answering examples where GPT-NeoX and FLAN-T5-XXL fail on some cases.The model still makes mistakes, but the authors found no example that another model solved while BloombergGPT did not.

7 Related Work

Related work traces the rise of large autoregressive language models, domain-specific training, corpus construction, and evolving evaluation practices. It positions BloombergGPT within efforts to combine in-domain capabilities with general-domain performance.

  • Language Models: Large autoregressive language models achieved strong transfer performance across diverse tasks as model size, data, and compute increased.GPT-3 and subsequent work also motivated research on emergent capabilities, robustness, and scaling strategies.
  • Domain-Specific Large Language Models: Domain-specific language models have been developed through from-scratch training or continued pretraining, but large decoder-only examples remain comparatively scarce.This scarcity reflects the substantially higher cost of training decoder-only models above 10B parameters.
  • Domain-Specific Large Language Models: Scientific and other specialized models suggest that in-domain pretraining can improve domain capabilities when sufficient specialized data is available.BloombergGPT extends this direction by augmenting private financial data with public data to preserve general-domain performance.
  • Training Data: Web-derived corpora provide broad coverage but retain artifacts, duplicates, toxic language, and possible evaluation-data contamination despite cleaning.Prior work also reports potential bias from exclusion filters and unintended marginalization of minority voices.
  • Evaluation: BloombergGPT addresses corpus-quality and evaluation gaps by using curated financial data and combining general-purpose benchmarks with targeted financial and internal tasks.The paper notes that financial NLP lacks a standard benchmark and that public evaluations may not match practical use cases.

8 Ethics, Limitations, and Implications

The paper frames BloombergGPT’s use in finance as requiring rigorous oversight because financial accuracy, toxicity, bias, privacy, and release risks have practical consequences. It describes risk controls while acknowledging unresolved questions and restricting model-weight release.

  • Ethical Use: Financial applications require accurate, factual information, so Bloomberg applies annotation guidelines, multilayer pre-launch review, post-launch monitoring, and regulatory compliance.The review process involves central risk and compliance organizations and relevant product leaders.
  • Ethical Use: The effects of FinPile’s cleaner, less overtly biased or toxic data on harmful-language generation remain unknown and require further study.The paper plans to test whether these data properties reduce inappropriate-content generation.
  • Openness: Releasing BloombergGPT carries abuse risks because its press releases, news articles, and filings could support imitation.The paper presents release strategy as a trade-off between community evaluation and potential misuse.
  • Openness: Bloomberg does not release model weights because extraction attacks could expose FinPile and undermine access to proprietary data.The paper states that selective researcher access would not guarantee that the model could not be leaked.

9 Conclusion

The conclusion presents BloombergGPT as a financial NLP model whose mixed-data training balances financial and general-domain performance. It also identifies training practice, logs, alignment, toxicity, bias, and tokenization as continuing research concerns.

  • Conclusion: BloombergGPT is presented as a best-in-class large language model for financial NLP.This is the paper’s stated overall contribution.
  • Conclusion: Mixing domain-specific and general-purpose data produces a model that balances performance across financial and general domains.The conclusion frames this as evidence relevant to training domain-specific models.
  • Conclusion: Training logs are intended to guide researchers developing their own large language models.The paper presents these logs as part of its broader contribution to understanding domain-specific model training.
  • Conclusion: Future work includes financial-domain alignment, testing effects of FinPile’s toxicity and bias profile, and understanding the impact of tokenization.The conclusion states that the effects of cleaner, less biased or toxic data remain unknown.
  • Conclusion: BloombergGPT achieves strong general-benchmark results while outperforming comparable models on financial tasks.The authors attribute the outcome primarily to a curated internal dataset, followed by tokenizer choice and architecture.

Appendix A. Architecture

The architecture appendix establishes notation for scalars, vectors, matrices, sequences, functions, element-wise products, indicators, indexing, and named functions. It also specifies conventions for trainable parameters and standard operations such as addition, softmax, and dropout.

  • Notation: Scalars are unstyled, bold lowercase variables denote column vectors, and bold capitalized variables denote matrices.For example, h_i,j may be an element of vector h_j, which is a column of matrix H.
  • Sequences: A sequence of n elements is represented as {x_i}_i=1^n, and a sequence of column vectors is treated as a matrix.The notation links individual sequence elements to matrix representations.
  • Functions: Vector functions f: R^n → R^n are applied column-wise when their inputs are matrices.The resulting matrix consists of the function applied independently to each column vector.
  • Operations: The Hadamard product A ⊙ B denotes element-wise multiplication of same-shaped matrices or vectors, while 1(P) is an indicator returning one when P is true.The appendix also defines [n] as the positive integers through n.
  • Operations: Adding a vector to a matrix means repeating the vector addition across every matrix column.Named functions such as softmax and FFN use regular non-italicized type, while red denotes trainable parameters or parameterized functions.
  • Operations: Dropout scales a randomly masked input by 1/(1-p), with mask variables independently sampled for each presentation.The mask is applied element-wise to the input.

A.1 Full Architecture

The architecture embeds token sequences, processes them through sequential transformer layers with ALiBi self-attention and feedforward components, and produces logits using the final representation. It omits positional embeddings because ALiBi supplies the positional mechanism.

  • Input tokens are converted into embedding vectors using a shared embedding matrix and an embedding LayerNorm function.
  • SelfAttention with ALiBi: No positional embedding is applied because the architecture uses ALiBi.
  • Layers: Transformer layers sequentially apply self-attention, feedforward, and LayerNorm functions to layer representations.
  • Logits: Final-layer representations are transformed into logits with the shared embedding matrix and a final LayerNorm, without a bias term.
  • SelfAttention with ALiBi: Self-attention uses ALiBi and a causal attention mask, with separate trainable parameters for attention and hidden-unit components.
  • The appendix lists shape, initialization, parameter-size, and tag-documentation details for the architecture and related data.

Appendix B. Details on external financial tasks

The appendix describes external financial benchmarks spanning sentiment, headline classification, named entity recognition, and conversational numerical question answering. It also documents training-data ordering and the initial curriculum-learning run.

  • External financial tasks: Financial Phrasebank classifies approximately 5,000 financial-news sentences as positive, negative, or neutral from an investor perspective.
  • External financial tasks: FiQA sentiment analysis predicts aspect-specific sentiment in financial news and microblog headlines, converting continuous annotations into classes for few-shot evaluation.
  • External financial tasks: Headline is a binary classification task over 11,412 gold-related news headlines, with high-quality category annotations.
  • External financial tasks: Financial NER uses SEC credit-risk documents and reports entity-level F1 after preprocessing and 20-shot context sampling.
  • External financial tasks: ConvFinQA requires conversational numerical reasoning over financial tables and text from S&P 500 earnings reports.
  • Training run v0: The first training run temporally ordered FinPile by month for curriculum learning, but its learning curve showed a large train-validation gap and stalled progress.
  • Training run v1.0: Removing curriculum learning and uniformly shuffling shards led to much faster validation-loss improvement, although the decisive factor was uncertain.

C.1 Elbow

Run v1.0 developed increasing gradient norms, validation-loss jumps, and a distinctive elbow in the first layer’s input LayerNorm multiplier weights. Investigations found incorrect LayerNorm weight decay but did not fully explain the trend.

  • Observed instability: After approximately 12k steps, v1.0 showed steadily increasing gradient norms, occasional spikes, validation-loss jumps, and renewed training-loss plateauing.
  • Observed instability: The first layer’s input LayerNorm was the sole component whose normalized L2 norm developed an elbow and then increased roughly linearly after step ~12k.
  • Observed instability: Individual input LayerNorm multiplier values shrank until steps 11–12k and then shifted upward, matching the component-level trend.
  • Investigation: The investigation found that weight decay had incorrectly been applied to LayerNorm multipliers initialized at 1.
  • Investigation: The extra push from weight decay could explain a downward trend but not the sudden shift toward growth, leaving the elbow unexplained.
  • Attempts to fix the run: Several attempted fixes targeted gradient growth through learning-rate reduction, gradient clipping, FP32 LM-head computation, and other conservative changes.

C.2 Slide

Run v2.0 adopted conservative hyperparameters and additional monitoring after the v1.x instability. It trained smoothly for about 42 days, with stable LayerNorm norms and steadily decreasing validation loss, although the decisive change remains unknown.

  • Run v2.0 setup: Run v2.0 combined conservative hyperparameter changes with query-key scaling, batch-size warm-up, and monitoring of LayerNorm multiplier norms.
  • Run v2.0 outcome: The run remained smooth and uneventful for approximately 42 days, or about 115,500 iterations, with few surprises in performance curves and gradient norms.
  • Run v2.0 outcome: Validation loss decreased smoothly, aside from a few early jumps, before flattening around 2.116 at the end of training.
  • Run v2.0 outcome: Initial-layer LayerNorm weight norms stayed smooth and stable without the trend changes observed in v1.x.
  • Open question: The authors could not determine which individual change was decisive for the successful v2.0 training run.

C.3 Suspense

The team investigated stalled validation loss by changing learning rates, checkpoints, data order, and dropout while monitoring validation and downstream evaluations. Despite several interventions, validation loss eventually flattened or rose, downstream gains remained insignificant, and training ended after 77% of the data had been used.

  • Validation loss stopped improving after about 48 days, while training loss flattened near 2.10, prompting intervention.
  • The team considered changing the maximum learning rate, rolling back to an earlier checkpoint, and re-shuffling the remaining data to alter optimization.
  • The validation setup combined a future-oriented July 2022 set with a second validation set drawn from the final 105M training tokens, while BBH and MMLU provided downstream checks.The future set was intended to avoid leakage, whereas the training-derived set was not held out by time period.
  • After lowering the learning rate and rolling back, validation loss initially improved sharply but soon flattened, leading the team to enable dropout at probability 0.1.
  • With dropout, validation loss initially decreased but later rose to the original run’s value; subsequent learning-rate and dropout combinations likewise failed to produce significant overall gains.
  • Training ended after 77% of the data had been used, motivated by flattening validation loss, limited downstream improvement, budget constraints, and promising initial benchmark results.The remaining unseen data was retained for later training or fine-tuning experiments.
Loading 2303.17564v3…