BloombergGPT: A Large Language Model for Finance

1.1 BloombergGPT

• BloombergGPT is a 50-billion-parameter model trained on mixed financial and general data to achieve strong financial benchmark results without sacrificing general benchmark performance.Its training corpus exceeds 700 billion tokens, and evaluation covers standard LLM, public financial, and Bloomberg-internal benchmarks.Source: §1.1 · pp. 3–4

1.2 Broader Contributions

• The paper’s broader contribution is introducing an LLM trained jointly on domain-specific and general data, an approach not previously studied for domain-specific language modeling.The model combines strong domain-task performance with strong general-purpose benchmark performance.Source: §1.2 · p. 4
• BloombergGPT uses unusually substantial curated data from reliable sources and evaluates both public financial benchmarks and internal tasks aligned with practical use cases.The internal evaluations address the mismatch between standardized benchmarks and how domain-specific models are used in practice.Source: §1.2 · pp. 4–5
• The paper also documents choices and experience concerning model sizing, tokenization, and training challenges to support future domain-specific LLM development.The authors train a 50-billion-parameter model on 569 billion tokens, use a Unigram tokenizer, and leverage BLOOM software to show that a moderately sized team can build a competitive model.Source: §1.2 · p. 5

2 Dataset

• FinPile combines financial news, filings, press releases, web-scraped documents, and social media with public datasets, yielding a corpus roughly half domain-specific and half general-purpose text.Each dataset is deduplicated, and the full training corpus is documented in Table 1.Source: §2 · p. 5

2.1 Financial Datasets (363B tokens – 51.27% of training)

• 363B tokens of FinPile derive from Bloomberg’s curated financial data, including company filings, financial news, and other market-relevant documents.The corpus draws on four decades of structured and unstructured data, with documents created internally or acquired externally.Source: §2.1 · p. 7

2.1.1 Web (298B tokens – 42.01% of training)

• 298B tokens of FinPile come from a financially focused web crawl emphasizing high-quality relevant websites rather than a general-purpose web crawl.Web-source classifications are rough and include regional categories such as US, Asia-Pac, and UK.Source: §2.1.1 · p. 7

2.1.3 Filings (14B tokens – 2.04% of training)

• 14B tokens of filings provide normalized financial statements, including dense tables and charts, that are important for financial decision-making.Most filings come from EDGAR, the SEC’s online database, and include documents such as 10-K and 10-Q reports.Source: §2.1.3 · p. 8

2.2 Public Datasets (345B tokens – 48.73% of training)

• The training corpus uses The Pile, C4, and English Wikipedia: The Pile provides cleaned, diverse domains; C4 is processed differently; Wikipedia is a July 1, 2022 dump.Source: §2.2.1 · p. 9, §2.2.2 · p. 9, §2.2.3 · p. 9

2.3 Tokenization

• The tokenizer uses byte-level Unigram tokenization with greedy pretokenization, separate digit chunks, and spaces retained in alphabetic chunks to increase information density.It is trained on The Pile and uses a 131,072-token vocabulary selected through C4 encoding-efficiency experiments.Source: §2.3 · pp. 9–10

3.1 Architecture

• The architecture is a BLOOM-based decoder-only transformer with 70 layers, multi-head self-attention, GELU feed-forward networks, ALiBi positional encoding, tied embeddings, and an additional embedding layer normalization.The feed-forward networks have one hidden layer, and the additional normalization follows prior work by Le Scao et al. (2022) and Dettmers et al. (2022).Source: §3.1 · p. 11

3.2 Model Scaling

• Data limitations and a 1.3M GPU-hour compute budget led to a 50B-parameter model, with 70 layers, 40 attention heads of dimension 192, and 50.6B total parameters.The ~700B-token dataset was too small for a Chinchilla-optimal configuration under the compute budget, while FinPile was kept from representing less than half of total training data.Source: §3.2 · pp. 12–13

3.3 Training Configuration

• Training uses a standard left-to-right causal objective with concatenated documents chunked into 2,048-token sequences, while ALiBi permits longer sequences at inference.Optimization uses AdamW, cosine learning-rate decay with linear warmup, and batch-size warmup; the initial run set dropout to 0.0.Source: §3.3 · p. 13

3.4 Large-scale Optimization

• Large-scale optimization combines ZeRO stage 3, SageMaker Model Parallelism, activation checkpointing, BF16 mixed precision, and fused kernels, reaching 102 TFLOPs with 32.5-second training steps.The model is sharded across 128 GPUs with four model copies; fused kernels improve speed by 4–5 TFLOPs and avoid out-of-memory errors.Source: §3.4 · pp. 14–15

Training duration and stopping

• 139,200 steps (~53 days) processed 569B of 709B available tokens before early stopping because held-out development loss no longer improved.The authors note that substantially longer training might have produced further improvements.Source: §4 · p. 16

Optimization trajectory

• The run switched from batch size 1,024 to 2,048 at step 7,200, after which training and validation losses generally decreased until validation loss flattened or increased after step 115,500.The batch-size change produced visible curvature in validation loss, while later instability triggered corrective interventions.Source: §4 · p. 16

Optimization interventions

• After step 115,500, the run sequentially reduced the learning rate to two-thirds, halved it while adding 0.1-probability dropout, and halved it again.These interventions occurred at steps 115,500, 129,900, and 137,100, respectively.Source: §4 · p. 16

Checkpoint selection

• The run ended at step 146,000 for lack of observable validation-loss progress, and checkpoint 139,200 became the final model based on validation loss and downstream evaluations.Figure 2 displays the successful run’s smoothed training-loss and held-out validation-loss curves, excluding other attempts and overwritten partial runs.Source: §4 · pp. 15–16

5 Evaluation

• The evaluation covers finance-specific public and Bloomberg-internal tasks alongside general-purpose benchmarks grouped into BIG-bench Hard, Knowledge Assessments, Reading Comprehension, and Linguistic Tasks.The benchmark suite draws from HELM, SuperGLUE, MMLU, the GPT-3 suite, and additional internal and public financial tasks.Source: §5 · p. 16
• All models are compared without prompt tuning, task descriptions, or Chain-of-Thought prompting, using standard prompting and task-dependent few-shot examples.The study reports win rates from side-by-side comparisons for model pairs evaluated under these procedures.Source: §5 · p. 18

5.1 Few-shot Methodology

• Candidate-based tasks use likelihood-based classification with regular, calibration, or normalization methods, while other tasks use greedy generation and standardized dataset splits and inputs.The best classification method is reported per model and task; few-shot examples are sampled from training data to reduce evaluation variance.Source: §5.1 · p. 18

5.2 Heldout Loss

• BloombergGPT consistently outperforms other models on temporally heldout, deduplicated FinPile data, with the largest improvement on specialized documents such as Filings.Evaluation uses bits per byte and a future heldout set unseen during training; Figure 3 presents results by document type.Source: §5.2 · p. 19

5.3.1 External Financial Tasks

• BloombergGPT performs best on four of five external financial tasks—ConvFinQA, FiQA SA, FPB, and Headline—and second on NER, yielding the highest win rate.Its advantage over equally sized models is especially pronounced on ConvFinQA, which requires conversational reasoning over tables and answer generation.Source: §5.3.1 · p. 21, Table 8 · p. 21

5.3.2 Internal Task: Sentiment Analysis

• BloombergGPT outperforms all tested models across four internal aspect-specific sentiment tasks, by at least 25 and up to over 60 points on three tasks.The models perform similarly only on social media sentiment; Table 10 reports these results.Source: §5.3.2 · p. 23, Table 10 · p. 23

5.3.3 Exploratory Task: NER

• Internal NER results are mixed: BLOOM176B wins most NER tasks, while BloombergGPT ranks first once, second four times, third once, and last once among like-sized models.NER is treated as exploratory because it is largely unexplored for generative LLMs and requires extensive prompt engineering and more shots.Source: §5.3.3 · pp. 24–25
• BloombergGPT outperforms all other models by a large margin on internal NER+NED, except on social media data where it ranks second behind BLOOM176B.NER+NED identifies company mentions and generates their corresponding stock tickers; Table 12 reports the results.Source: Table 12 · p. 25, §5.3.3 · p. 26

5.4 BIG-bench Hard

• On BIG-bench Hard, BloombergGPT trails the much larger PaLM540B and BLOOM176B but is best among similarly sized models and leads on three tasks.It performs best in date understanding, hyperbaton, and tracking shuffled objects, indicating that finance-specific training did not sacrifice general-purpose abilities.Source: §5.4 · p. 26, Table 13 · p. 27

5.5 Knowledge Assessments

• On knowledge assessments, BloombergGPT outperforms similarly sized models, achieves the highest win rate among models run by the authors, and performs second best on average.It is below GPT-3 overall, with the largest gap in social sciences and the closest performance in STEM and “Other,” which include finance- and accounting-related questions.Source: §5.5 · pp. 27–28, Table 15 · p. 28

5.6 Reading Comprehension

• BloombergGPT is a close second to GPT-3 in reading comprehension and generally achieves the highest performance among BLOOM176B, GPT-NeoX, and OPT66B.The evaluation covers open-book question answering, including BoolQ, OpenBookQA, RACE, MultiRC, and ReCoRD.Source: §5.6 · pp. 28–29, Table 16 · p. 29

5.7 Linguistic Tasks

• BloombergGPT consistently scores highest among evaluated linguistic-task models, achieving an 85% win rate, while falling slightly behind GPT-3 and outperforming the other models.The tasks assess disambiguation, grammar, entailment, and related language understanding scenarios using accuracy.Source: §5.7 · pp. 29–30, Table 17 · p. 30

5.8 Summary

• Across dozens of tasks, BloombergGPT performs best among compared models with tens of billions of parameters and sometimes matches or exceeds models with hundreds of billions.In few-shot prompting, it uses knowledge of stock tickers and financial terms to compose valid Bloomberg Query Language queries from natural-language requests.Source: §5.8 · p. 30

Generation of Bloomberg Query Language

• BloombergGPT transforms natural-language queries into valid Bloomberg Query Language, making the powerful but complex financial-data tool more accessible.BQL supports different security classes, each with its own fields, functions, and parameters.Source: §6 · p. 31

Financial Question Answering

• BloombergGPT correctly identifies company CEOs in examples where GPT-NeoX fails and FLAN-T5-XXL consistently ignores the company, though BloombergGPT still makes mistakes.The comparison includes outputs from other models and notes no found example where another model succeeded while BloombergGPT failed.Source: §6 · pp. 31–32

Language Models

• Transformer-based language models enabled scaling in data, compute, and parameters, while generative pretraining and subsequent scaling produced strong transfer and emergent capabilities across downstream tasks.This progression spans recurrent models, transformers, generative pretraining, and large-scale models such as GPT-3.Source: §7 · pp. 32–33

Domain-Specific Large Language Models

• Domain-specific training improves performance, but large decoder-only domain models are relatively rare because they are substantially more costly to train than smaller masked language models.Existing approaches adapt general models or train decoder-only models from scratch, with examples spanning biomedical, scientific, protein, and other domains.Source: §7 · p. 34
• BloombergGPT augments private financial data with public data to gain in-domain capabilities without sacrificing general-domain performance, motivated by prior findings on domain-specific training.The paper argues that sufficient in-domain data is available and uses Bloomberg’s curated data to construct FinPile.Source: §7 · pp. 34–35

Training Data

• Web-scale corpora require cleaning and deduplication because they can contain artifacts, toxic language, bias, machine-generated text, and evaluation-set contamination; deduplication can reduce memorization and improve training efficiency without reducing performance.These limitations motivate constructing a high-quality financial corpus from Bloomberg’s curated datasets.Source: §7 · p. 35

Evaluation

• Because no standard financial NLP benchmark adequately covers relevant tasks, BloombergGPT combines few-shot FLUE evaluation with company-internal benchmarks and distinguishes reported from verified numbers to address inconsistent experimental settings.The evaluation discussion contrasts broad automatic benchmarks with product-oriented task evaluations and notes differences in prompts, decoding, and few-shot examples.Source: §7 · pp. 35–36

Model Size and Tokenization

• Scaling studies motivate a nearly Chinchilla-optimal standard architecture, while domain-specific vocabulary findings motivate training a unigram tokenizer for BloombergGPT.Prior work links performance to parameters, data, compute, architecture, and tokenization; domain-specific vocabularies can improve token efficiency and downstream performance.Source: §7 · p. 36, Tokenization · p. 37

8.1 Ethical Use

• Financial applications require accurate information, so Bloomberg uses annotation guidelines, multi-level pre-launch review, post-launch monitoring, and applicable regulatory controls.Reviews involve central risk and compliance organizations and relevant product leaders.Source: §8.1 · p. 37
• The potential for harmful language from BloombergGPT remains difficult to quantify because toxicity and bias depend on application areas, although cleaner FinPile data may reduce inappropriate generations.The authors plan to apply existing testing procedures and risk and compliance controls as products are developed.Source: §8.1 · pp. 37–38

8.2 Openness

• Model release decisions trade off community evaluation against nefarious use, with options ranging from open weights and licensed access to APIs or no access.The authors describe these choices as reflecting model use, potential harms, and business considerations.Source: §8.2 · p. 38
• BloombergGPT is not released because model-weight access could expose FinPile through data leakage, even under selective researcher access.The authors note that LLMs can extract substantial text from weights and therefore prioritize caution without strong privacy guarantees.Source: §8.2 · p. 38
• Bloomberg shares its training and evaluation experience through Training Chronicles, contributing practical knowledge for developers of domain-specific models.The authors identify prior OPT, BLOOM, GPT-3, PaLM, Chinchilla, and Gopher experiences as crucial enablers.Source: §8.2 · p. 38

Conclusion

• The authors present BloombergGPT as a best-in-class LLM for financial NLP.This conclusion summarizes the model’s central contribution.Source: §9 · p. 38
• Mixing domain-specific and general-purpose data balances performance across both domains and contributes to understanding domain-specific training and Chinchilla-optimal model sizing.The training logs are intended to guide others training their own LLMs.Source: §9 · p. 38
• Future work will study financial-domain alignment, the effects of potentially less toxic and biased FinPile data, and how tokenization changes the resulting model.The cited alignment directions include task fine-tuning opportunities (Wei et al., 2021), while the effects of FinPile’s data selection remain unknown.Source: §9 · pp. 38–39
• BloombergGPT achieves strong general-benchmark results and outperforms comparable models on financial tasks, attributed primarily to its curated internal dataset, tokenizer choice, and up-to-date architecture.The authors rank these factors by decreasing impact and plan to continue developing financial applications with BloombergGPT.Source: §9 · p. 39

A.0 Notation

• Scalars are unstyled, bold lowercase variables are column vectors, and bold capitalized variables are matrices.For example, h_i,j can be an element of vector h_j, which is the j-th column of matrix H.Source: §A.0 · p. 61
• Named functions use non-italicized regular typeface, while red denotes trainable parameters or parameterized functions.Examples include softmax(·), W, and FFN(·).Source: §A.0 · p. 61
• A sequence of n elements is denoted by {x_i}^n_i=1, and a sequence of column vectors is represented as a matrix X.Source: §A.0 · p. 61
• Vector functions f: R^n → R^n are applied column-wise when their input is a matrix.Thus, for X = {x_j}^m_j=1, f(X) = {f(x_j)}^m_j=1.Source: §A.0 · p. 61
• The notation defines ⊙ as the element-wise product, 1(P) as a Boolean indicator, and [n] as the positive integers through n.Source: §A.0 · p. 61
• Adding a vector to a matrix means repeating the addition across every column, while exp(·) acts element-wise on vectors.Source: §A.0 · p. 61
• Dropout scales a masked input by 1/(1−p), with independently sampled mask variables for each example presentation.The mask is formed from independently drawn random variables m_i.Source: §A.0 · p. 61

A.1 Full Architecture

• The model maps each input token to an embedding and applies sequential trainable blocks across layers, with separately parameterized LayerNorm applications for different block inputs and outputs.The embedding uses a token embedding matrix and LayerNorm; no positional embedding is applied because ALiBi supplies the positional mechanism.Source: §A.1 · pp. 61–62
• The final layer representation is converted into token logits using the same embedding matrix, a final LayerNorm, and no bias term, yielding next-token distributions conditioned on the prefix.The logits are defined over the vocabulary and sequence positions, with each output column determining the distribution for the corresponding next token.Source: §A.1 · p. 62

A.2 SelfAttention with ALiBi (SA)

• ALiBi self-attention operates over multiple heads with head-specific dimensions, trainable bias parameters, attention and hidden-unit dropout, relative-position biases, and a causal attention mask.The ALiBi matrix and mask are defined over sequence positions, with the mask assigning −∞ to future positions.Source: §A.2 · pp. 62–63

A.3 LayerNorm (LN)

• LayerNorm uses trainable gain and bias parameters with a small constant, and the notation distinguishes the embedding, final, and input LayerNorm instances.The parameterization variable θ is used to identify LNem, LNf, and LNin separately.Source: §A.3 · p. 63

A.4 FeedForwardNetwork (FFN)

• The feed-forward component is a multilayer perceptron using element-wise GELU activation, trainable bias parameters, and component-specific dropout.The GELU expression is specified explicitly in the appendix.Source: §A.4 · p. 63

A.5 List of All Trainable Parameters

• The parameter appendix lists shape hyperparameters and their values for the model’s trainable components.It separately introduces the initialization hyperparameters and the complete parameter list with parameter sizes and element-wise initialization.Source: §A.5 · p. 64
• Initialization uses a default range based on 3D, with a rescaled range for the second feed-forward layer and the final linear map in self-attention.These conventions are presented as part of the parameter initialization specification.Source: §A.5 · p. 64

Appendix B. Details on external financial tasks

• FPB uses investor-oriented positive, negative, and neutral sentiment labels, with a random split, 5-shot prompting, and support-weighted F1 evaluation.The split contains 3,876 training and 970 test sentences.Source: Appendix B · p. 65
• FiQA SA (Maia et al., 2018) converts continuous sentiment scores into negative, neutral, and positive classes, using random splits, 5 shots, and weighted F1.The thresholds are −1 ≤ x < −0.1, −0.1 ≤ x < +0.1, and +0.1 ≤ x ≤ +1.Source: Appendix B · p. 65
• Headline (Sinha and Khandait, 2020) classifies gold-news headlines across nine information tags, using 5-shot prompting and average weighted F1 across categories.The dataset contains 11,412 headlines, with annotator-consensus Cohen’s Kappa of at least 0.85 for each category.Source: Appendix B · pp. 65–66
• NER (Salinas Alvarado et al., 2015) evaluates PER, LOC, and ORG entities after dropping MISC and entity-free sentences, using Fin-5 context, Fin-3 testing, 20 shots, and entity-level F1.The processed data contain 504 training and 98 test sentences.Source: Appendix B · p. 66
• ConvFinQA (Chen et al., 2022) tests conversational numerical reasoning over linearized financial tables, using one gold conversation as context and exact-match accuracy on the dev set.The data contain 3,892 conversations and 14,115 questions; unavailable test labels motivate reporting dev-set results.Source: Appendix B · p. 66

C.0 Still

• Figure 7 shows v0’s training and validation losses becoming nearly flat after step 20k, so the team abandoned temporally ordered curriculum learning as too risky to continue.The final 6k steps lasted about ~2.3 days, while validation progress was very limited between steps 15k–20k.Source: Figure 7 · p. 67, §C.0 · p. 67
• After removing curriculum learning and uniformly shuffling training data at shard level, v1.0 produced much faster validation-loss improvement, though the cause of v0’s plateau remained unresolved.The team could not determine whether curriculum learning or another v1.x issue caused the plateau.Source: §C.0 · p. 68

Observed failure

• After about 12k steps, gradient norms increased with occasional spikes, coinciding with sudden validation-loss jumps and another training-loss plateau.The pattern suggested increasing sensitivity to small weight changes.Source: §C.1 · p. 68

Norm investigation

• Input LayerNorm at layer 1 was the only component whose normalized L2 norm abruptly increased roughly linearly after step 12k, aligning with initial gradient-norm growth.Other components followed similar benign trends.Source: Figure 9 · p. 69, §C.1 · p. 70
• All 60 inspected γin_1 multiplier values shrank until steps 11–12k before shifting upward, producing the same elbow pattern.These values came from one model shard containing 60 such multipliers out of 128.Source: §C.1 · p. 70, Figure 10 · p. 70

Implementation bug

• Weight decay was mistakenly applied to LayerNorm multiplier weights γ, although they are initialized at 1 and should be excluded.The implementation inherited this practice from BERT.Source: §C.1 · p. 70

Interpretation and response

• The weight-decay bug did not explain the elbow artifact, because pushing weights toward zero would predict continued decline rather than a sudden shift to growth.After four failed fixes, the team considered restarting with more conservative hyperparameters and restoring the removed embedding LayerNorm as a precaution.Source: §C.1 · p. 71

• The v2.0 run used conservative hyperparameter choices to stabilize learning dynamics, with query-key layer scaling (Shoeybi et al., 2019) and batch-size warm-up.The batch size was 1024 for 7200 iterations before increasing to 2048, and gradient clipping was set to 0.3 instead of 1.0.Source: §C.2 · pp. 71–72
• Validation loss decreased smoothly before flattening around 2.116, while v2.0 training remained uneventful for approximately 42 days and ~115,500 iterations.Training and validation curves showed few surprises; the job required only one restart after 28 days because of a platform duration limit.Source: §C.2 · p. 72
• Monitoring LayerNorm weight norms showed smooth, stable behavior in the initial layer, providing evidence that the earlier pathological γem behavior was absent.The monitored quantities included γem and γin norms.Source: §C.2 · p. 72
• Although the changes produced a smooth training run, the authors could not conclude which intervention was responsible for v2.0’s success.They deferred isolating the decisive change to future work.Source: §C.2 · p. 72

Training attempts

• After 48 days, validation loss had stalled for a week while training loss flattened around 2.10, indicating that v2.0 was no longer learning properly.The stalled period spanned iterations 115,500–133,200.Source: §C.3 · p. 72
• The team rolled back seven days, shrank the learning rate, reshuffled remaining data, and later enabled dropout at probability 0.1 after v2.1 showed only temporary improvement.With dropout, training loss increased as expected, validation loss initially decreased, then returned to the original run’s value.Source: §C.3 · pp. 73, 75

Validation and evaluation

• Because valfuture contained only July 2022 data and was slightly out of distribution, the team added valpast, a 105M-token offline validation set from the training data.Valpast was not held out by time because the training data had been fully shuffled.Source: §C.3 · pp. 73–74
• The team used BBH’s multiple-choice subset and MMLU alongside validation perplexity to test whether stalled validation loss reflected stalled downstream performance.These evaluations provided additional assurance but required substantially more computation than validation-loss calculation.Source: §C.3 · p. 74

Training attempts

• Every v2 run eventually flattened or increased validation loss, and no change significantly improved validation loss and downstream evaluations.This outcome was summarized across Figure 12 and Table 20.Source: Figure 12 · p. 73, §C.3 · p. 75, Table 20 · p. 76

Stopping decision

• Training ended after 77% of the data was used, preserving unseen data for later training or finetuning experiments.Future stability experiments were proposed with SwiGLU activations, RoPE embeddings, and query-key normalization.Source: §C.3 · p. 75