PaperlayerView exact revision on arXiv

Source-linked AI summary

The Falcon Series of Open Language Models

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, Guilherme Penedo

arXiv:2311.16867v2cs.CLcs.AI

TL;DR

The Falcon series addresses how to scale performant language models using predominantly web data. It introduces 7B, 40B, and 180B models, finding that Falcon-180B nears PaLM-2 Large performance while remaining an openly available model series.

  • Problem

    Training modern language models requires trillions of tokens, motivating investigation into whether rigorously filtered and deduplicated web data can provide performant training data at scale.

  • Method

    The paper develops and evaluates the Falcon-7/40/180B causal decoder-only models, predominantly trains on filtered and deduplicated web data, and studies dataset and architecture choices through ablations.

  • Results

    Falcon-180B nears PaLM-2 Large performance and falls between GPT-3.5 and GPT-4, while the series is trained on up to 3,500B tokens.

  • Takeaways & Limitations

    The Falcon series and a 600B-token RefinedWeb extract are released to support research and adoption of large-scale filtered and deduplicated web data.

  • Takeaways & Limitations

    The evaluations predominantly cover classic natural-language tasks and do not establish human preferences for downstream models after finetuning or reinforcement learning.

Abstract

from arXiv · show

We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest model, Falcon-180B, has been trained on over 3.5 trillion tokens of text--the largest openly documented pretraining run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla, and improves upon concurrently developed models such as LLaMA 2 or Inflection-1. It nears the performance of PaLM-2-Large at a reduced pretraining and inference cost, making it, to our knowledge, one of the three best language models in the world along with GPT-4 and PaLM-2-Large. We report detailed evaluations, as well as a deep dive into the methods and custom tooling employed to pretrain Falcon. Notably, we report on our custom distributed training codebase, allowing us to efficiently pretrain these models on up to 4,096 A100s on cloud AWS infrastructure with limited interconnect. We release a 600B tokens extract of our web dataset, as well as the Falcon-7/40/180B models under a permissive license to foster open-science and accelerate the development of an open ecosystem of large language models.

1 Introduction

The Falcon series comprises three causal decoder-only models built through scalable data, hardware, and training practices. The paper documents their pretraining and releases the models and a large RefinedWeb extract to support open research.

  • The paper documents large-scale pretraining across performance, data, and hardware scalability axes.Falcon-180B follows updated scaling-law recommendations, while the training setup scales to 4,096 A100 40GB accelerators with limited interconnect.
  • The authors extensively document Falcon pretraining to address limited public documentation of recent state-of-the-art models.The documentation covers training decisions, experiments, custom tooling, and the data pipeline.
  • The Falcon series and a 600 billion token RefinedWeb extract are openly released to accelerate research and community-driven improvement.Falcon-7/40B use an Apache 2.0 license, while Falcon-180B uses a dedicated responsible AI use license.

2 State-of-the-art: from language modeling to frontier models

Language modeling progressed from representation learning and transfer learning to Transformer-based generalist models, with scaling laws establishing compute as a systematic route to improvement. This progression led to frontier models, while very large open-source pretraining runs remained uncommon.

  • Word embeddings, recurrent architectures, and transfer learning established successive foundations for language modeling from unstructured text.These approaches progressed from vector-based representations to contextual models and downstream specialization through finetuning.
  • Transformers enabled efficient recipes spanning encoder-only BERT models and causal decoder-only GPT models.GPT-2 helped popularize emergent few-shot generalization abilities.
  • Scaling laws linked increased pretraining compute with commensurate gains in language-modeling performance and fueled a rapid expansion of large language models.GPT-3 demonstrated the scaling approach and contributed to the subsequent scaling frenzy.
  • Frontier models are defined as large-scale machine-learning models exceeding currently advanced capabilities across a wide variety of tasks.GPT-4 and PaLM-2 are identified as early contributions to this category.
  • Open models lagged closed models in pretraining compute by approximately 18 months, although the gap was not widening.Open-source models above 10,000 PF-days remained rare.

3 Design philosophy

The Falcon design philosophy prioritizes scalable methods that leverage compute across performance, data, and hardware. It also emphasizes cost-efficiency through distributed training and memory-saving techniques.

  • Falcon prioritizes scalability across performance, data, and hardware axes, following the view that compute-leveraging methods are ultimately most effective.The design philosophy is inspired by the bitter lesson and favors scalable approaches over occasional paradigm-shifting methods.
  • Scaling pretraining compute is expected to produce systematic and predictable improvements in language-modeling capabilities.The paper presents scaling laws as a more reliable path to improvement than waiting for occasional research breakthroughs.
  • Jointly scaling model size and training data is preferred, while larger datasets increase pretraining capacity without increasing inference costs.The paper contrasts dataset growth with model-size growth, which increases inference burden.
  • Large-scale training depends on distributed methods that efficiently coordinate thousands of hardware accelerators.Hardware-efficient methods are presented as an important determinant of which approaches gain community traction.
  • Falcon combines 3D parallelism and optimizer sharding with memory-saving methods to use cost-efficient AWS infrastructure and 40GB A100 GPUs.The implementation targets limited interconnect and cheaper accelerator configurations.

4 Experiments and motivations for data, architecture, and hyperparameters

The experiments examine data quality, data mixtures, multilinguality, code, architecture, and hyperparameter choices for efficient Falcon pretraining. They find that filtered and deduplicated web data is highly effective, small code or multilingual fractions preserve English performance, and several architecture choices mainly improve scalability rather than task performance.

  • Web data: Filtered and deduplicated web data matches curated-data performance on natural-language tasks, with RefinedWeb becoming the best benchmarked dataset after both processing stages.Filtering closes the gap with The Pile, while stringent deduplication further improves performance.
  • Data mixtures: Curated data added to a strong RefinedWeb baseline never significantly improves zero-shot accuracy, while books and technical data degrade it beyond 50%.Conversations degrade least, possibly because they are more diverse or closer to task distributions.
  • Code and multilingual data: 5-10% code or multilingual data does not broadly affect English zero-shot performance, although multilinguality causes most degradation on HellaSwag in this restricted setup.The authors caution that these small-scale ablations may not represent other multilingual settings or larger models.
  • Architecture: The multigroup attention variant reduces communication requirements during both training and inference while preserving the fixed-factor scaling of the key/value cache.It assigns separate key/value pairs to each tensor-parallel rank, with n_kv = TP.
  • Architecture: RoPE shows a likely advantage over ALiBi at 1B parameters, but that advantage diminishes at 3B, while URPE provides no evidence of improvement over RoPE.SwiGLU likewise shows no clear zero-shot benefit in the reported small-scale experiments.
  • Hyperparameters: Warm-up rankings identify the best learning rate in the tested 1B-model setup, with rankings generally stable through training.Only the second- and third-best learning rates switched places.

5 Implementation

Falcon’s implementation scales pretraining through a predominantly web-based dataset, custom distributed training, and memory- and throughput-efficient kernels. The pipeline emphasizes extensive filtering and deduplication, while the training system combines parallelism, optimizer sharding, FlashAttention, and selective recomputation.

  • Data scale and mixture: 3,000-5,000 billion tokens were targeted for pretraining, with Falcon using a 3,500B-token dataset and no upsampling of curated sources.The same mixture percentages were used for Falcon-7B, 40B, and 180B despite differing training lengths.
  • Data scale and mixture: Nearly 85% of the final Falcon mixture is web data, while curated corpora remain a minority intended to broaden model expressiveness.The curated sources account for 13% of the final dataset and are not upsampled.
  • Macrodata Refinement: RefinedWeb scales web data through stringent filtering and deduplication, using up to 20,000 CPU cores and neutral, mostly non-ML-based filtering strategies.The pipeline applies fuzzy MinHash deduplication, substring deduplication, and heuristics targeting extraction artifacts and machine-generated content.
  • Macrodata Refinement: The Macrodata Refinement pipeline proceeds through document preparation, filtering, and deduplication, removing nearly 90% of originally collected CommonCrawl documents.Reported removal rates include about 50% for non-English documents, 24% of the remainder for insufficient quality, and 12% for duplicates.
  • Distributed training: Gigatron combines 3D distributed parallelism with ZeRO optimizer sharding to reduce memory consumption and improve scalability on AWS infrastructure with limited interconnect.Across 1B to 180B parameters, optimizer sharding had no performance overhead and enabled larger microbatches and higher throughput.
  • Memory and throughput efficiency: FlashAttention improves training throughput and memory availability, while selective recomputation halves decoder-block memory consumption without reducing throughput.The custom FlashAttention implementation avoids materializing the full attention matrix, and selective recomputation recomputes activation functions and layer norms.

6 Results

Falcon evaluations show strong performance across natural-language, commonsense, question-answering, code, and aggregate benchmarks, while also exposing sensitivity to evaluation setup and human preferences.

  • Evaluation and human preference: FLAN fine-tuning produces the best zero-shot SuperGLUE performance but the lowest human preference, whereas self-instruct improves preference more modestly on SuperGLUE.Human ratings were collected from 15 annotators across 250 prompts, making preference evaluation more expensive than SuperGLUE evaluation.
  • Comparisons with PaLM: Falcon-180B recovers 99.5% of PaLM-2 Large’s average performance across tasks, exceeding PaLM-2 Medium and PaLM.It outperforms PaLM-2 on HellaSwag, Winogrande, and PIQA, but lags on RACE and ANLI.
  • Comparisons with GPT-3.5 and GPT-4: Falcon-180B performs between GPT-3.5 and GPT-4 on all evaluated tasks, nearly matching GPT-4 on Winogrande.Its multiple-choice question-answering performance is closer to GPT-3.5, although consistently higher.
  • State-of-the-art comparisons: Falcon-180B significantly improves over other state-of-the-art models on commonsense tasks except BoolQ and strongly outperforms them on question answering.Falcon-40B and Falcon-7B slightly underperform corresponding LLaMA-2 models, with differences associated with pretraining compute and, for Falcon-7B, an aggressive multiquery configuration.
  • State-of-the-art comparisons: Falcon-180B performs best among natural-language-focused models on HumanEval, matching Inflection-1 despite only 3% code in its training data.It nearly matches PaLM-Coder and PaLM-2 S∗ despite their dedicated code specialization.
  • EleutherAI Evaluation Harness: Across model scales, Falcon significantly improves over the comparison models, with Falcon-40B outperforming GPT-3 175B despite a smaller compute budget.Falcon-7B approaches GPT-3 175B, while smaller RefinedWeb-only validation models compare favorably with same-size GPT-3 models.

7 Limitations

The Falcon study identifies limitations arising from outdated research timing, constrained compute, narrow benchmarks, deployment trade-offs, data composition, context length, and limited post-training. These boundaries qualify how the models and ablations should be interpreted and deployed.

  • Scope of findings: The research largely predates major developments after December 2022, while compute constraints limited the scale of some investigations.The authors identify research timing and resource constraints as broad sources of limitation.
  • Ablation and benchmark scope: Ablations used small models and limited zero-shot logprob tasks, which may miss scale-dependent effects, code, multilingual, and human-preference performance.The authors recommend larger-scale ablations and broader benchmarking.
  • Deployment trade-offs: Longer pretraining reduces inference-cost decoupling, making some Falcon deployments more challenging than smaller models trained for fixed token budgets.This trade-off particularly concerns the 40B and 180B models trained closer to pretraining optimality.
  • Data composition: The models predominantly use RefinedWeb with a conservative code fraction, limiting code-oriented performance relative to training regimes with more code data.The authors recommend increasing code prevalence and potentially upsampling domain-relevant sources.
  • Context length: A 2,048-token pretraining sequence length can limit multi-turn chat and code use cases, although later adaptation to longer sequences is possible.The paper describes rotary-embedding-based adaptation and staged longer-context training as future directions.
  • Post-training and safety: The released models are predominantly pretrained rather than aligned, so deployment should include guardrails and target-specific bias and harm evaluations.The authors note that reinforcement learning from human feedback may be relevant before deployment.

8 Conclusion

The paper documents the Falcon series’ data, architecture, ablations, training implementation, and benchmark results, while releasing models and data to support open research. Falcon-180B approaches PaLM-2 Large performance and is reported as the strongest open-source model available at release.

  • Training implementation: The training implementation combines conversational masking, 3D parallelism, ZeRO, and memory-efficient tooling for cost-efficient cloud training.The paper also describes a custom data pipeline and dedicated FlashAttention interventions.
  • Results: Falcon-180B nears PaLM-2 Large performance and falls between GPT-3.5 and GPT-4 on reported key benchmarks.The evaluation primarily focuses on classic natural-language tasks.
  • Open artifacts: The project publicly releases Falcon models and a 600B-token RefinedWeb extract under permissive open-source licenses.The release is intended to foster open research and technological development.
  • Motivation: The paper frames responsible sharing and open sourcing as ways to broaden collaboration on improving large language model safety and reliability.This motivation is presented as part of the project’s rationale for public release.

C Model card

The model card for Falcon follows the framework introduced by Mitchell et al. (2019).

  • The Falcon model card follows the framework introduced by Mitchell et al. (2019).

D Datasheet

The datasheet section refers readers to the dedicated RefinedWeb paper.

E Comparisons with undocumented models

The paper contrasts documented research papers with scant promotional reports, whose sparse evaluations and missing modeling details make comparisons challenging.

  • Recent models have released scant promotional technical reports instead of adequately documented research papers.
  • Sparse evaluations and absent modeling details make comparisons challenging.
  • Details leaked through back-channels are poorly suited to citation and attribution.

F.1 Measurement plan to measure all to all bandwidths/latencies efficiently

The measurement code constructs comparison configurations across powers-of-two dimensions and rotations, then stacks and reshapes the resulting comparisons.

  • The routine initializes comparison configurations and iterates over dimensions that double while remaining below n.
  • For each dimension d, it tests every rotation r in the range from 0 through d-1.
  • Each generated configuration applies op with the selected dimension and rotation, then copies the result into comps.
  • The collected configurations are stacked and reshaped into an array whose final dimension is 2.

F.2 Converting tree token depth into an attention mask:

The attention-mask routine determines visibility by comparing token positions and values, then writes the resulting boolean mask for each position.

  • The routine loops over every position i in the input sequence.
  • For each i, it scans all positions j and marks attention when j equals i + 1.
  • It retains attention only when the value at position i is less than the value at position j.
  • The routine writes the negated attention condition, excluding self-attention, into attention_mask[j, i].

F.3 Zero-1 pseudo-code

The Zero-1 pseudo-code partitions model gradients across data-parallel workers, updates optimizer parameters, and gathers the updated model parameters.

  • reduce_scatter_grads divides the model-gradient buffer by dp_world_size and allocates a bfloat16 destination tensor.
  • The routine reduce-scatters model gradients across the data_parallel_group and copies the result into optimizer gradients.
  • all_gather_opt_params_to_model_params converts sharded optimizer parameters to bfloat16 before gathering them into model parameters.
  • The sequence performs optimizer.step() and then gathers optimizer parameters back into model parameters.

G Prompts

The evaluation prompts use standardized EleutherAI Harness defaults for several sections, with task-specific prompts elsewhere. The tables document task-specific candidate handling and shot conventions across the benchmark suite.

  • Prompt selection: Default EleutherAI Harness prompts reproduce the GPT-3 evaluation setup for Sections 4 and 6.5.Other sections use prompts specified in the paper; unspecified prompts fall back to the harness default.
  • Prompt structure: Candidates receive log-probability evaluation conditioned on the context and provided samples.The context appears once, while samples repeat for each shot and answers are supplied except for the final sample.
  • Task-specific conventions: ARC and OpenBookQA use answer choices directly as candidates when options are absent from the prompt.
  • Task-specific conventions: LAMBADA selects the model's most likely answer without restricting evaluation to predetermined candidates.
  • Task-specific conventions: RACE counts shots at the article level, providing three questions per article even in 0-shot evaluation.
  • Task-specific conventions: COPA and Winograd retain the reference formatting with task-specific instruction or candidate-label changes.COPA adds an instruction, while Winograd changes yes/no candidates to true/false.
Loading 2311.16867v2…