PaperlayerView exact revision on arXiv

Source-linked AI summary

RedPajama: an Open Dataset for Training Large Language Models

Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, Ce Zhang

arXiv:2411.12372v1cs.CLcs.LG

TL;DR

Pretraining data composition and curation remain poorly understood because leading language models disclose few dataset details. RedPajama releases open datasets with quality signals and evaluates filtering strategies, finding that progressively higher-quality subsets produce models with varying quality across benchmarks.

  • Problem

    Most leading language models disclose little about pretraining-data composition and curation, leaving optimal dataset composition and filtering strategies insufficiently understood.

  • Method

    The paper releases RedPajama-V1 and web-only RedPajama-V2, pairing raw text with quality signals and evaluating dataset-filtering heuristics through broad model ablations.

  • Results

    Filtering RedPajama-V2 into successively higher-quality subsets produced language models with varying quality across diverse benchmark tasks and outperformed models trained on other large-scale pretraining corpora.

  • Takeaways & Limitations

    RedPajama provides open, transparent pretraining datasets and quality signals that support research into data curation and filtering for language models.

  • Takeaways & Limitations

    The relatively small models limit the evaluation, requiring further exploration at larger scales.

Abstract

from arXiv · show

Large language models are increasingly becoming a cornerstone technology in artificial intelligence, the sciences, and society as a whole, yet the optimal strategies for dataset composition and filtering remain largely elusive. Many of the top-performing models lack transparency in their dataset curation and model development processes, posing an obstacle to the development of fully open language models. In this paper, we identify three core data-related challenges that must be addressed to advance open-source language models. These include (1) transparency in model development, including the data curation process, (2) access to large quantities of high-quality data, and (3) availability of artifacts and metadata for dataset curation and analysis. To address these challenges, we release RedPajama-V1, an open reproduction of the LLaMA training dataset. In addition, we release RedPajama-V2, a massive web-only dataset consisting of raw, unfiltered text data together with quality signals and metadata. Together, the RedPajama datasets comprise over 100 trillion tokens spanning multiple domains and with their quality signals facilitate the filtering of data, aiming to inspire the development of numerous new datasets. To date, these datasets have already been used in the training of strong language models used in production, such as Snowflake Arctic, Salesforce's XGen and AI2's OLMo. To provide insight into the quality of RedPajama, we present a series of analyses and ablation studies with decoder-only language models with up to 1.6B parameters. Our findings demonstrate how quality signals for web data can be effectively leveraged to curate high-quality subsets of the dataset, underscoring the potential of RedPajama to advance the development of transparent and high-performing language models at scale.

1 Introduction

The paper frames opaque pretraining-data composition and curation as a central obstacle to open language-model development. It introduces RedPajama as a transparent, scalable, versatile dataset family with open reproductions, raw web data, quality signals, and model studies.

  • Motivation: Pretraining-data composition and curation are often insufficiently transparent, even for many reports on state-of-the-art and open-weights language models.The paper identifies transparency as a core challenge in developing open language models.
  • Design principles: More than 100 trillion tokens comprise the RedPajama datasets, supporting the goal of democratizing access to open-source language-model development.The datasets are designed around scale as well as openness.
  • Design principles: Transparency, scale, and versatility guide RedPajama through public curation documentation, broad general-purpose web data, and document-level quality signals.The quality signals let users select data according to their own needs and criteria rather than following a prescribed definition of quality.
  • Contributions: RedPajama-V1 is an open reproduction of LLaMA’s training data, while RedPajama-V2 provides raw, unfiltered web data with 46 quality measures per document.The contributions also include reports on corpus creation and model training, with artifacts intended to enable further curation research.
  • Contributions: Ablation studies use decoder-only Transformer models with 468 million parameters to examine how quality signals produce varying performance on common NLP benchmarks.These studies are based on RedPajama-V2 and focus on creating models with different performance levels through data curation.

2 Related Work

Prior work has developed both web-only and composite pretraining datasets for large language models. C4 established a benchmark for web dataset quality, while open composite datasets such as the Pile and RedPajama-V1 broadened domain coverage and enabled transparent replication.

  • Web-only datasets: 175B tokens comprised C4, a large-scale web corpus filtered from CommonCrawl that remains a benchmark for web dataset quality.C4 was among the first large-scale web datasets.
  • Composite datasets: The Pile was one of the first fully open composite datasets, combining additional domains to enable broader coverage.Composite datasets complement web-only datasets by incorporating multiple domains.
  • Composite datasets: RedPajama-V1 openly replicated the seven-subset, multiple-domain LLaMA recipe and gained widespread adoption.SlimPajama was subsequently derived from RedPajama-V1 through further cleaning and deduplication.

3 RedPajama-V1: An open Reproduction of the LLaMA Training Data

RedPajama-V1 recreates the LLaMA training corpus across seven data sources while resolving gaps in the original documentation through explicit dataset choices and preprocessing. RedPajama-INCITE models were then trained on Summit and evaluated against comparable open models, with instruction tuning producing strong few-shot results.

  • Corpus reconstruction: RedPajama-V1 reproduces the LLaMA corpus from seven sources: CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, and Stack Exchange.The authors document gaps in the original descriptions and explain how they resolve ambiguities during reconstruction.
  • Corpus reconstruction: The reconstruction applies source-specific processing, including CommonCrawl quality filtering, GitHub file heuristics, Wikipedia boilerplate removal, book deduplication, LaTeX cleanup, and Stack Exchange answer ranking.Books3 was removed because of copyright issues, while Stack Exchange posts were grouped into question-answer pairs.
  • Model training: RedPajama-INCITE includes pretrained and instruction-tuned models at 3B and 7B sizes, trained on Summit’s 4608 6xV100-node cluster.The Power9 architecture required a custom training stack, and V100 limitations required fp16 with loss scaling rather than bf16.
  • Evaluation: After 800B tokens, RedPajama-Base-INCITE-3B showed better average few-shot and zero-shot performance than other open models of similar size.Few-shot performance was measured with HELM classic across 16 core scenarios, while zero-shot performance used EleutherAI’s LM evaluation harness.
  • Evaluation: RedPajama-INCITE-7B-Base trailed Falcon-7B by 1.0 points and Llama-7B by 4.1 points on HELM-classic, mainly on tasks requiring logprobs.Its average scores were comparable on tasks that directly generate answers and measure quality.
  • Evaluation: RedPajama-INCITE-7B-Instruct outperformed similarly sized leading open models, including Llama-7B, Falcon-7B, and MPT-7B, on HELM by 2-8 points.The instruction-tuned model was optimized for few-shot performance using diverse tasks from P3 and Natural Instructions.

4 RedPajama-V2

RedPajama-V2 is a transparent, web-only dataset built from large-scale Common Crawl data and enriched with metadata and quality signals for flexible curation. Its ablations show that filtering, especially Gopher rules combined with fuzzy deduplication, can improve downstream performance.

  • Quality signals: The dataset provides metadata and quality signals covering natural language, repetitiveness, harmful content, machine-learned quality, and deduplication.Signals include heuristic measures, blocklist and blocked-domain indicators, fastText and DSIR-based measures, MinHash signatures, and exact-duplicate IDs.
  • Data construction: RedPajama-V2 processes Common Crawl web-extracted text while preserving multiple perplexity buckets and supporting English, French, German, Italian, and Spanish data.The dataset uses all 84 monthly snapshots from 2014 through April 2023 and applies the lightly processing CCNet pipeline.
  • Dataset composition: 113B documents span five languages, with 32.8B documents in the head+middle partitions and 80B in the tail partition.Tail documents are typically shorter at 850 tokens, compared with approximately 1500 tokens for head and middle documents.
  • Ablation results: Gopher filtering rules generally improve performance across the RedPajama-V2 ablations.The evaluation uses decoder-only Llama-2 models with 468M and 1.6B parameters across diverse downstream benchmarks and language modeling.
  • Ablation results: Fuzzy deduplication combined with Gopher filtering achieves the highest aggregated scores across the RedPajama-V2 datasets.Its average and normalized average benchmark scores are second only to RefinedWeb, while its rank-score is higher than RefinedWeb’s.

5 Conclusion · A Intended Uses

RedPajama provides over 100 trillion tokens in large, fully open, transparent datasets for language-model pretraining. Its datasets span multiple sources and domains, with RedPajama-V2 offering web-only data accompanied by quality signals for filtering into higher-quality subsets.

  • 5 Conclusion: Over 100 trillion tokens make RedPajama among the largest fully open and transparent pretraining datasets.The datasets have served as a central building block for strong open-source language models.
  • 5 Conclusion: RedPajama-V2 can be filtered into successively higher-quality subsets for training language models with varying quality levels.The paper presents examples of this filtering process across a diverse set of benchmarks.
  • A Intended Uses: The datasets are intended primarily to provide training data for large language models.RedPajama contains data from different sources and domains.
  • A Intended Uses: RedPajama-V1 includes web scrapes, Wikipedia articles, scientific content from arXiv, and code in various programming languages.These sources support the dataset’s broad domain coverage.
  • A Intended Uses: RedPajama-V2 consists exclusively of data based on web scrapes.Its composition differs from the more varied source collection in RedPajama-V1.
  • A Intended Uses: RedPajama-V2 is accompanied by quality signals intended to support dataset use and filtering.These signals provide information for working with its web-scraped text.

B Dataset Accessibility

RedPajama-V1 and RedPajama-V2 are openly downloadable through the Hugging Face Hub and public HTTPS endpoints. Component-specific URLs provide access to dataset documents and curation artifacts, including quality signals, duplicate IDs, and MinHash signatures.

  • Dataset downloads: Both RedPajama-V1 and RedPajama-V2 are available for download through the Hugging Face Hub.The datasets are hosted at the listed RedPajama-Data-1T and RedPajama-Data-V2 dataset pages.
  • Public endpoints: Public HTTPS endpoints provide URL lists for the components of both RedPajama-V1 and RedPajama-V2.The V1 component list and V2 component lists are available through the specified Together-hosted endpoints.
  • Dataset artifacts: RedPajama-V2 endpoints expose raw documents, quality signals, exact-duplicate document IDs, and MinHash signatures.The passage lists separate URLs for document data, quality-signal files, duplicate identifiers, and MinHash signatures.

B.1 Structure of the datasets … C RedPajama-V1

RedPajama-V1 and V2 are distributed as sharded JSON Lines data, but their structures differ according to each dataset’s contents. V2 pairs raw documents with quality signals, duplicate identifiers, and minhash signatures that support granular quality analysis and deduplication.

  • B.1.1 RedPajama-V1: RedPajama-V1 contains seven domains, with source-specific metadata and a distinct Common Crawl structure.Metadata fields include identifiers, timestamps, language, URLs, source information, and domain-specific attributes.
  • B.1.2 RedPajama-V2: RedPajama-V2’s core consists of text documents accompanied by quality annotations, duplicate IDs, and minhash signatures.For each Common Crawl snapshot, documents are partitioned into 5,000 shards by language and perplexity bucket, with associated artifacts following the same organization.
  • B.1.2 RedPajama-V2: V2 files are organized by snapshot, shard, language, and Wikipedia-perplexity bucket across document, quality-signal, duplicate, and minhash artifacts.Languages include en, de, fr, es, and it; perplexity buckets are head, middle, and tail.
  • B.1.2 RedPajama-V2: V2 documents are Gzip-compressed JSONL records containing crawl metadata, raw content, line information, language fields, perplexity, and bucket assignment.The schema includes fields such as url, date_download, digest, length, nlines, source_domain, title, raw_content, and language_score.
  • B.1.2 RedPajama-V2: V2 quality signals use keyed lists of [start, end, score] tuples to locate signal values within raw_content.This representation supports quality signals at different granularities, including line-level and document-level annotations.
  • B.1.2 RedPajama-V2: V2 duplicate parquet files list documents duplicated at least once, excluding each duplicate cluster’s first occurrence.Dropping every listed document leaves one member of each duplicate cluster in the dataset.
  • C RedPajama-V1: The paper’s RedPajama-V1 section provides additional dataset details and results beyond the structural description.Both datasets are distributed in JSON Lines format and partitioned into shards, while their internal structures differ.

C.1 Filtering Heuristics for Code obtained from GitHub

The GitHub code dataset retains projects under Apache, BSD, or MIT licenses and removes files failing structural, character-composition, token-ratio, or extension-whitelist heuristics. These filters target unusually long or code-unlike files while restricting retained files to recognized programming and markup extensions.

  • Projects are retained only when licensed under Apache, BSD, or MIT, with additional heuristics modeled on The Stack.
  • Files with a maximum line length above 1000 characters or average line length above 100 characters are removed.
  • Files are removed when alphanumeric characters comprise less than 0.25 of their content.
  • Files are removed when the ratio of alphabetical characters to tokens is below 1.5.
  • Files must use one of the whitelisted extensions spanning programming, markup, scripting, configuration, and documentation formats.The whitelist includes extensions such as .c, .cpp, .go, .java, .js, .py, .rs, .sh, .ts, .tsx, .xml, Dockerfile, and Makefile.

C.2 Detailed Evaluations for the RedPajama-INCITE LLMs

This section reports detailed benchmark scores for the RedPajama-INCITE 3B and 7B language models trained on the RedPajama-V1 dataset.

  • C.2 Detailed Evaluations for the RedPajama-INCITE LLMs: The evaluation covers RedPajama-INCITE 3B and 7B LLMs trained on RedPajama-V1.The section provides their detailed benchmark scores.

C.3 Detailed Sources of Uncertainties in the Construction of the RedPajama-V1 Dataset … D.3 Embedding-based Clustering

The paper documents uncertainty in reproducing RedPajama-V1 and attributes performance mismatch with LLaMA-1 partly to likely dataset differences. It then analyzes RedPajama-V2 through deduplication statistics, grouped quality signals, English-only signal distributions, and embedding-based clustering.

  • C.3 Detailed Sources of Uncertainties in the Construction of the RedPajama-V1 Dataset: Uncertainty in RedPajama-V1 construction mainly arose from missing details about the reference dataset.The authors identify a likely mismatch between RedPajama-V1 and the dataset used to train LLaMA-1 models.
  • D RedPajama-V2: The RedPajama-V2 section adds analyses, statistics, and ablations on differently filtered subsets of the web dataset.These analyses provide additional characterization of the dataset and its filtering choices.
  • D.1 Summary Statistics of our Deduplication Approach: Document counts in the head+middle partition remain relatively stable until 2018, while 2014–2016 contains substantially fewer documents, up to 10x for German.The analysis also tracks unique documents after sequential deduplication from the newest snapshot to the oldest.
  • D.2 Quality Signals: RedPajama-V2 includes quality signals computed from the CCNet pipeline and additional signal families for language, repetitiveness, content, and ML-based heuristics.The signals are summarized across Tables 11–15.
  • D.2.1 Overview of Available Quality Signals: The quality-signal overview organizes signals by natural language, text repetitiveness, text content, and ML-based heuristics.CCNet-derived signals are included separately in the overview.
  • D.2.2 Histograms: The quality-signal distributions are further examined through histograms.These statistics use the 2023-06 snapshot and are computed only for English data.
  • D.3 Embedding-based Clustering: Embedding-based clustering sampled 2,000,000 documents from the unfiltered 2021-04 snapshot and embedded each document’s middle 8,192 tokens with gte-large-en-v1.5.Nomic Atlas was used for clustering and topic modeling, with cluster topics illustrated in Figure 8.

D.4 Data Ablations: Detailed Evaluations

This section expands the aggregate benchmark analysis with task-level evaluations of dataset filters and web datasets, covering 468M- and 1.6B-parameter language models across multiple benchmarks.

  • Detailed evaluations: Task-level scores are reported separately in Tables 18, 19, and 20 to provide more detail beyond the aggregate benchmark results.The evaluations cover individual tasks rather than only aggregated benchmark scores.
  • 468M-parameter evaluations: 468M-parameter language models are evaluated across different dataset filters and strong web datasets, with rankings indicated for each metric.Table 18 reports general evaluations, while Tables 19 and 20 cover 5-shot MMLU subtasks and multiple-choice tasks.
  • 1.6B-parameter evaluations: 1.6B-parameter language models are compared on downstream task accuracy after training on different datasets over 350B tokens.These results are summarized in Table 21.

D.5 Evaluations for the 1.6B Parameter Models · E Author Responsibility Statement · F License

The paper evaluates 1.6B-parameter language models trained on 350B tokens using ablations and multiple benchmark settings, then specifies dataset licensing, author responsibility, and user obligations. The code uses Apache 2.0, while dataset components remain subject to their source licenses and applicable legal and ethical responsibilities.

  • D.5 Evaluations for the 1.6B Parameter Models: Each 1.6B-parameter model was trained on 350B tokens for the reported ablation evaluations.The evaluations are presented in Tables 21, 22, and 23.
  • D.5 Evaluations for the 1.6B Parameter Models: The 1.6B-parameter language model was evaluated in a 5-shot setting on MMLU and its subtasks.These results are reported in Table 22.
  • D.5 Evaluations for the 1.6B Parameter Models: The 1.6B-parameter language model was also evaluated on multiple-choice tasks.These results are reported in Table 23.
  • E Author Responsibility Statement: The aggregated dataset is licensed under ODC-By-1.0 and any licenses applying to its constituent parts.The license terms therefore cover both the aggregation and applicable component licenses.
  • E Author Responsibility Statement: The authors disclaim liability for rights violations arising from dataset use, including copyright infringement, privacy violations, or misuse of sensitive information.They also state that the dataset’s scale prevents guarantees of absolute completeness or correctness.
  • E Author Responsibility Statement: Users assume sole responsibility for consequences of using the dataset and must exercise due diligence while following applicable laws, regulations, and ethical guidelines.Accessing, downloading, or using the dataset signifies acceptance of the statement and license terms.
  • F License: The repository code is distributed under an Apache 2.0 license, while datasets derived from Common Crawl follow Common Crawl Foundation Terms of Use.Other datasets follow the licenses under which they were originally distributed.
  • F License: Users are asked to follow each individual license governing the dataset subset they use.Specified sources include C4, restricted GitHub licenses, Arxiv, Wikipedia, and StackExchange terms.
Loading 2411.12372v1…