PaperlayerView exact revision on arXiv

Source-linked AI summary

SlimPajama-DC: Understanding Data Combinations for LLM Training

Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, Eric Xing

arXiv:2309.10818v3cs.CLcs.AI

TL;DR

This paper studies how deduplication scope and combinations of data sources affect LLM pretraining on SlimPajama. It evaluates configurations across benchmarks and finds that most combinations outperform RedPajama-1.3B on average, with benefits also reported for larger models.

  • Problem

    The paper asks how data-source composition and deduplication affect LLM pretraining, including whether different source combinations support broad performance.

    Source: §1 · pp. 3–4

  • Method

    The authors compare SlimPajama data combinations and token-distribution patterns, evaluating trained models across multiple benchmarks and reporting a risk-of-random-guessing metric for MMLU.

    Source: §2.2 · p. 5, §5.1 · p. 10, §5.2 · p. 11

  • Results

    All but DC-4 outperform RedPajama-1.3B in average results on the Huggingface harness, and all but DC-4 exceed GPT-neo-1.3B and RedPajama-1.3B in average performance across additional evaluations.

    Source: §5.1 · p. 10, §5.2 · p. 11

  • Takeaways & Limitations

    The authors conclude that SlimPajama-DC's advantages can transfer from compact models to models several times larger.

    Source: §7 · p. 15

Abstract

from arXiv · show

This paper aims to understand the impacts of various data combinations (e.g., web text, Wikipedia, GitHub, books) on the pretraining of large language models using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source dataset, which has been refined and further deduplicated to 627B tokens from the extensive 1.2T token RedPajama dataset contributed by Together. We have termed our research as SlimPajama-DC, an empirical analysis designed to uncover fundamental characteristics and best practices associated with employing SlimPajama in the training of large language models. During our research with SlimPajama, two pivotal observations emerged: (1) Global deduplication vs. local deduplication. We analyze and discuss how global (across different sources of datasets) and local (within the single source of dataset) deduplications affect the performance of trained models. (2) Proportions of highly-deduplicated multi-source datasets in the combination. To study this, we construct six configurations on SlimPajama dataset and train individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best configuration outperforms the 1.3B model trained on RedPajama using the same number of training tokens by a significant margin. All our 1.3B models are trained on Cerebras 16$\times$ CS-2 cluster with a total of 80 PFLOP/s in bf16 mixed precision. We further extend our discoveries (such as increasing data diversity is crucial after global deduplication) on a 7B model with large batch-size training. Our SlimPajama-DC models are available at: https://huggingface.co/MBZUAI-LLM/SlimPajama-DC and the separate SlimPajama-DC datasets are available at: https://huggingface.co/datasets/MBZUAI-LLM/SlimPajama-627B-DC.

1 Introduction

Pretraining performance depends on the composition of training data, especially its diversity across domains and genres. This study examines how deduplication scope and combinations of highly deduplicated sources relate to model performance.

1 Introduction

  • Training data should span domains, genres, and languages because its composition plays a critical role in LLM performance.The paper focuses on domain/source weightings such as GitHub, Wikipedia, books, and web text.Source: §1 · p. 3
  • Deduplication removes repeated data to address redundancy, which can slow convergence, encourage overfitting to frequent patterns, and reduce efficient use of model capacity.The paper frames deduplication as supporting generalization and reducing undue biases.Source: §1 · p. 3
  • Global deduplication removes overlaps across combined sources, whereas local deduplication removes duplicates within each source and can leave cross-source duplicates.The paper notes that global deduplication requires more hardware memory and is generally preferable for multi-source training.Source: §1 · p. 3
  • The study examines how to combine diverse, highly deduplicated datasets, while noting that shared biases across sources may be amplified.The authors emphasize that data quality can outweigh quantity.Source: §1 · pp. 3–4
  • Combining specialized sources may broaden task coverage while sacrificing some depth on particular tasks compared with specialized training.The study explores this specialization–generalization trade-off using individual and combined sources.Source: §1 · p. 4

2 Dataset Overview

SlimPajama combines multiple data sources and analyzes their token distributions before filtering short documents and deduplicating globally. The processing reduces the original RedPajama collection from 1.2T to 627B tokens.

2.2 Dataset Token Frequency Statistics

  • The dataset comparison uses KL divergence between domain token-count distributions and examines letters, frequent tokens, numbers and operators, whitespace, and non-alphanumeric tokens.The figure presents the KL-divergence comparison as a confusion matrix.Source: §2.2 · p. 5, Figure 1 · p. 6
  • Token distributions are similar among RefinedWeb, Book, C4, and CommonCrawl, and between GitHub and StackExchange; ArXiv differs in non-alphanumeric tokens.RefinedWeb also differs from GitHub and StackExchange in whitespace-token distributions, while number and operator distributions are relatively consistent across datasets.Source: §2.2 · p. 5

2.3 Dataset Processing Procedure

  • SlimPajama filters low-length documents and applies MinHashLSH deduplication to reduce RedPajama from 1.2T to 627B tokens.The dataset overview reports 500M-token validation and test sets, each cleaned to avoid overlap with training data; study configurations use 330B training tokens after tokenization.Source: §2.1 · p. 4, §2.3 · p. 5

2.3.1 Low-length Document Filtering

  • The additional low-length filter removes documents under 200 characters after text cleanup, except in Books and GitHub, where short documents are retained.The authors say such short documents typically contain metadata and no useful information; the additional step removes 1.86% of documents.Source: §2.3.1 · p. 6

2.3.2 Global Deduplication

  • Global deduplication removes duplicates within and across sources, addressing cross-source overlap that local, per-source deduplication cannot remove.The global procedure uses MinHashLSH; its trillion-token scale required memory and parallelization optimizations.Source: §2.3.2 · p. 7

3 Dataset Combination Configurations

The study varies source diversity, source proportions, and the inclusion of individual domains while keeping total training tokens constant. It also uses RefinedWeb as a CommonCrawl-only comparison.

3.1 SlimPajama

  • The configurations vary source diversity, domain proportions, and individual-source inclusion while holding total training tokens constant.The final whole combination uses the largest diversity; ArXiv and StackExchange are omitted from configuration 5 to preserve resources and sufficient CommonCrawl tokens.Source: §3.1 · p. 8, Table 3 · p. 8

3.2 RefinedWeb

  • RefinedWeb is a rigorously filtered and extensively deduplicated English web dataset used as the CommonCrawl-only comparison.Source: §3.2 · p. 8

4 Network Architecture and Training Details

The experiments use Cerebras-GPT decoder models with dense attention, ALiBi, and SwiGLU. Training uses a shared tokenizer and fixed optimization settings across the configurations.

4.1 Network Architecture

  • Cerebras-GPT uses an autoregressive decoder with dense attention in every block and a maximum sequence length of 2,048 tokens.The models follow an aspect ratio of approximately 80 or dimensions congruent with GPT-3 models; ALiBi applies distance-based bias to query-key attention scores instead of adding positional embeddings.Source: §4.1 · pp. 8–9

4.2 Training Details

  • All configurations use an adapted GPT-NeoX BPE tokenizer with a 50,277-token vocabulary and train on 330B tokens for about 2.5 days.Training uses AdamW, a linear learning-rate schedule ending at 10% of peak, weight decay 0.1, gradient clipping at 1.0, and 150 warmup steps; models use SwiGLU, ALiBi, and bfloat16 mixed precision.Source: §4.2 · p. 9, §4.1 · p. 9

5 Results and Analysis

The evaluations compare SlimPajama data combinations across leaderboard and domain-specific benchmarks, examining average performance, task-specific strengths, and training loss. They also introduce RRGS to assess whether MMLU predictions resemble random guessing.

5.1 Huggingface Leaderboard Evaluation with Harness

  • DC-6 has the top average accuracy across SlimPajama combinations and the best HellaSwag result; all but DC-4 outperform RedPajama-1.3B on average.DC-1 leads ARC and MMLU but ranks last on TruthfulQA; the authors report that greater domain diversity brings better overall accuracy.Source: §5.1 · p. 10

5.2 More Evaluations

  • Except for DC-7, adding sources typically improves average performance; all configurations except DC-4 surpass GPT-neo-1.3B and RedPajama-1.3B on average.The evaluation also finds mixture-specific strengths: DC-1 leads ARC challenge and RACE, DC-4 leads Winogrande and Xstory cloze, and DC-5 leads WSC273.Source: §5.2 · pp. 10–11
  • RRGS measures MMLU score variance around the 25% random-guessing baseline, considering above-baseline items, below-baseline items, or all items.The authors state that greater variance suggests less chance-like prediction; generally, higher MMLU average scores correspond to lower random-guessing probability.Source: §5.2 · p. 11

5.3 Training Loss

  • DC-7 has the highest average accuracy despite the highest training loss, while code-heavy DC-3 has the lowest loss.The authors conclude that lower training loss does not necessarily correspond to better model performance, and that more code is associated with lower loss.Source: §5.3 · p. 12

6 Related Work

Related work covers the construction of multi-source corpora, data processing and selection, data-combination strategies, and large-batch training. It situates SlimPajama-DC among approaches to dataset diversity, weighting, and training efficiency.

6.1 RedPajama, SlimPajama and Others.

  • SlimPajama refines and deduplicates RedPajama from 1.2T to 627B tokens, while Pile combines 22 subsets to support dataset diversity.The passage describes SlimPajama as a highly deduplicated, multi-source open dataset and Pile as an English corpus of 825 GiB.Source: §6.1 · p. 13

6.2 Data Processing and Optimization Approaches

  • SlimPajama adapts importance sampling by heuristically valuing unique data points; DSIR selects data using importance-resampling weights in a reduced feature space.The passage also describes a pruning approach that may suppress traditional power-law scaling and potentially reach exponential scaling for pruned dataset sizes.Source: §6.2 · pp. 13–14

6.3 Data Combination for Training Large Language Models

  • Prior data-combination approaches include concatenating diverse corpora, balancing or weighting sources, and combining text with other modalities.DoReMi derives domain weights with a proxy model using group distributionally robust optimization, then resamples data for full-model training.Source: §6.3 · p. 14

6.4 Large Batch Training for Large Language Models

  • Large batches can hasten convergence but may impair generalization; the study explores progressive weight decay as an angle on this generalization gap.The passage notes that gradually enlarging batch size from a smaller starting size is another proposed approach.Source: §6.4 · pp. 14–15

7 Conclusion

SlimPajama-DC studies domain weights and data combinations for LLM training. The authors report that its advantages can transfer from compact models to larger models, supporting further data-centric exploration.

7 Conclusion

  • SlimPajama-DC studies domain weights and data combinations, with reported advantages that transfer from compact models to models several times larger.The authors connect this transfer to faster training on SlimPajama using optimal domain sampling probabilities.Source: §7 · p. 15

A Data Proportion Details

The paper reports data-source proportions for its configurations and examines MMLU results by domain. Its 7B training combination uses SlimPajama with adjusted source proportions and added data sources.

A Data Proportion Details

  • Table 8 presents detailed data-source proportions for various datasets, while Table 10 presents the 7B large-batch training combination.The supplied captions do not specify the proportions or individual table entries.Source: Table 8 · p. 19, Table 10 · p. 21

B MMLU

  • Some configured 1.3B models outperform GPT-3 175B and LLaMA2 7B on individual MMLU subdomains.The provided table caption identifies these as 5-shot results per domain; it does not provide the specific values.Source: B MMLU · p. 19, Table 9 · p. 20

C.1 7B Training Data Combination

  • The 7B training combination adjusts source proportions, increases Github and Wikipedia sampling, and adds S2orc and Stack-Markdown for diversity.It also incorporates Commoncrawl web data from RedPajama to obtain a sufficient proportion of web text.Source: §C.1 · p. 21

C.2 7B Model Training Configurations

The 7B large-batch model uses MPT architecture and a GPT-NeoX-20b-adapted tokenizer. Training uses AdamW and bf16 mixed precision on 232 A100 GPUs.

Architecture

  • The 6.7B-parameter LBS model uses MPT architecture, 2,048-token sequences, Triton with Flash Attention, and Alibi.Alibi is enabled to make the model more flexible for input-length extrapolation.Source: §C.2 · p. 21

Tokenizer

  • The tokenizer is adapted from GPT-NeoX-20b, with vocabulary size adjusted to 50,432.The passage says the adjustment improves mfu and leaves tokens available for subsequent training.Source: §C.2 · pp. 21–22

Optimizer and System

  • Training uses AdamW with β1=0.9 and β2=0.95, cosine learning-rate decay to 10% of maximum, staged weight decay, gradient clipping at 1.0, and 2,000 warmup steps.The system uses 232 NVIDIA A100 80G GPUs, llm-foundry, FSDP with activation checkpointing, and bf16 mixed precision.Source: §C.2 · p. 22

C.3 Fast Training with Large Batch-size

The paper describes large-batch training as a route to faster convergence and more efficient hardware use, while noting risks of overfitting and difficult optimization. Its scheme reports higher throughput and model FLOPs utilization than LLaMA and MPT, with fewer total GPU-hours.

Fast Training with Large Batch-size

  • The large-batch scheme achieves higher throughput and mfu than LLaMA and MPT, with fewer total training GPU-hours.Table 12 is identified as reporting throughput per GPU, mfu, and GPU-hours per trillion training tokens; supplied text gives no specific values.Source: §C.3 · p. 22, Table 12 · p. 22
  • Large batches allow larger learning rates and fewer batches, which the paper associates with faster convergence and more efficient hardware use.The text says large-batch training can also increase overfitting and may leave models stuck or near saddle points in the loss landscape.Source: §C.3 · p. 22
  • The paper introduces an approach to mitigate overfitting in large-batch LLM training, after noting that large-batch minima can differ from those reached with smaller batches.The supplied passage does not describe the approach in this section.Source: §C.3 · p. 22

C.4 Progressive Training on Weight Decay

Progressive Training on Weight Decay (PTWD) divides training into three weight-decay phases to support smooth convergence with large batches and few iterations. After ShareGPT instruction tuning, average accuracy rises, although ARC and HellaSwag decline slightly.

C.4 Progressive Training on Weight Decay

  • Figure 4 is identified as the loss curve for the LBS-7B training.The supplied figure caption does not state the curve’s values or outcome.Source: Figure 4 · p. 23
  • PTWD uses three weight-decay phases: zero until convergence, 0.5 to suppress overfitting, then 0.1 as loss stabilizes.Each phase spontaneously converges to roughly one-third of the total training budget; the authors report smooth convergence with large batch size and extremely small iterations.Source: §C.4 · p. 23

C.5 Results of Pre-training and Instruction Tuning

  • After ShareGPT instruction tuning, average accuracy increases substantially, while MMLU and TruthfulQA improve and ARC and HellaSwag decline slightly.Table 13 presents the pretraining and instruction-tuning results; the supplied passage gives no specific metric values.Source: §C.5 · p. 23, Table 13 · p. 24

Related papers

Loading 2309.10818v3…