PaperlayerView exact revision on arXiv

Source-linked AI summary

SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models

Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, Shreyas Saxena

arXiv:2303.10464v2cs.LGcs.CL

TL;DR

Large-model pre-training is computationally costly, and conventional setups retain the same model capacity across pre-training and fine-tuning. SPDF uses unstructured sparse pre-training followed by dense fine-tuning; at 75% sparsity, GPT-3 XL reduced overall training FLOPs 2.5x while retaining benefits across most tasks and metrics.

  • Problem

    Scaling LLMs has improved quality but substantially increased pre-training computation, while model capacity is generally kept the same across pre-training and fine-tuning.

    Source: §1 · pp. 1–2

  • Method

    SPDF pre-trains with unstructured weight sparsity, then densifies the model for task-specific fine-tuning by allowing zeroed weights to learn.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    At 75% sparsity, GPT-3 XL reduced overall training FLOPs by 2.5x while retaining pre-trained textual-representation benefits across a majority of tasks and evaluation metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report a correlation between optimal pre-training sparsity and downstream fine-tuning dataset size and task difficulty.

    Source: §1 · p. 2

    The study uses simple static sparsity and leaves investigation of how the Chinchilla pre-training schedule transfers to sparse models for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural Language Processing (NLP). Instead of directly training on a downstream task, language models are first pre-trained on large datasets with cross-domain knowledge (e.g., Pile, MassiveText, etc.) and then fine-tuned on task-specific data (e.g., natural language generation, text summarization, etc.). Scaling the model and dataset size has helped improve the performance of LLMs, but unfortunately, this also lead to highly prohibitive computational costs. Pre-training LLMs often require orders of magnitude more FLOPs than fine-tuning and the model capacity often remains the same between the two phases. To achieve training efficiency w.r.t training FLOPs, we propose to decouple the model capacity between the two phases and introduce Sparse Pre-training and Dense Fine-tuning (SPDF). In this work, we show the benefits of using unstructured weight sparsity to train only a subset of weights during pre-training (Sparse Pre-training) and then recover the representational capacity by allowing the zeroed weights to learn (Dense Fine-tuning). We demonstrate that we can induce up to 75% sparsity into a 1.3B parameter GPT-3 XL model resulting in a 2.5x reduction in pre-training FLOPs, without a significant loss in accuracy on the downstream tasks relative to the dense baseline. By rigorously evaluating multiple downstream tasks, we also establish a relationship between sparsity, task complexity and dataset size. Our work presents a promising direction to train large GPT models at a fraction of the training FLOPs using weight sparsity, while retaining the benefits of pre-trained textual representations for downstream tasks.

1 INTRODUCTION

Pre-training costs have risen with LLM scale, while conventional training keeps model capacity fixed across pre-training and fine-tuning. SPDF uses unstructured sparse pre-training followed by dense fine-tuning; the paper reports lower pre-training FLOPs, retained downstream performance on most tasks, and a relationship between useful sparsity, task difficulty, and dataset size.

  • Scaling model and dataset size has improved LLM quality but substantially increased pre-training costs; GPT-3 175B training is estimated to cost millions of dollars.Source: §1 · p. 1
  • Prior LLM sparsification work targets training or inference FLOPs while matching dense training, whereas this work focuses on unstructured sparsity during pre-training.Source: §1 · p. 2
  • SPDF separates model capacity across phases: it pre-trains with unstructured weight sparsity, then densifies for downstream fine-tuning so zeroed weights can learn.Source: §1 · p. 2
  • At 75% sparsity, GPT-3 XL training used 2.5x fewer overall training FLOPs while retaining pre-trained textual-representation benefits across a majority of tasks and evaluation metrics.Source: §1 · p. 2
  • The authors establish a correlation between optimal pre-training sparsity and both fine-tuning dataset size and task difficulty.Source: §1 · p. 2

2 METHODOLOGY

The methodology uses sparse pre-training to reduce compute, then restores model capacity through dense fine-tuning. It motivates this design with hypotheses about downstream task difficulty, dataset size, and model scale, and specifies an autoregressive language-modeling implementation.

2.1 INTUITION AND HYPOTHESES

  • The authors hypothesize that high pre-training sparsity can preserve downstream accuracy with dense fine-tuning, while performance also relates to dataset size, task difficulty, and model size.Source: §2.1 · p. 3

2.2 SPARSE PRE-TRAINING AND DENSE FINE-TUNING

  • The method sparsifies a dense network, pre-trains it on a large unsupervised text corpus, then fine-tunes densely by allowing zeroed weights to learn on labeled task data.Source: §2.2 · pp. 3–4
  • For the motivating use case, autoregressive pre-training maximizes likelihood on an unsupervised token corpus, with each token conditioned on a context window.Source: §2.2 · p. 3
  • A fixed binary mask zeros a target fraction of weights during sparse pre-training; the study uses randomly pruned, uniform static sparsity.Source: §2.2 · pp. 3–4
  • The fine-tuning data are context-target sequence pairs, including linearized tables paired with descriptions and article content paired with summaries.Source: §2.2 · p. 4
  • Dense fine-tuning removes the mask and revives inactive weights initialized at zero, increasing capacity; each downstream task learns its own full-sized parameter increment.Source: §2.2 · p. 4

3 EXPERIMENTAL SETUP AND RESULTS

The section evaluates SPDF across language-model sizes, sparsity levels, and downstream tasks, then examines parameter changes and training FLOPs. It finds that sparse pre-training can preserve downstream performance while reducing training FLOPs, with dense fine-tuning outperforming sparse fine-tuning.

Flop Optimal Pre-training via Chinchilla Scaling Law

  • Pre-training follows Chinchilla scaling at approximately 20 tokens per parameter: 2.5B tokens for GPT-2 Small and 26B for GPT-3 XL, using the Pile and 50% or 75% sparsity unless stated otherwise.The models are pre-trained from scratch.Source: Flop Optimal Pre-training via… · p. 5

3.1 DETAILS ON THE FINE-TUNING DATASETS

  • The evaluation covers three data-to-text NLG datasets and Curation Corpus summarization, selected to vary dataset size, data type, and task difficulty.The datasets are E2E, WebNLG, DART, and Curation Corpus.Source: Flop Optimal Pre-training via… · p. 5, §3.1 · p. 5

3.2 SPARSE FINE-TUNING VS DENSE FINE-TUNING

  • Dense fine-tuning reduces the BLEU-score drop versus sparse fine-tuning across E2E, WebNLG, and DART, especially at 75% sparsity.On WebNLG at 75% sparsity, the BLEU deltas are -1.48 with sparse fine-tuning and -0.78 with dense fine-tuning.Source: Figure 2 · p. 5, §3.2 · p. 6

3.3 SPDF ON NATURAL LANGUAGE GENERATION AND TEXT SUMMARIZATION

  • Across NLG tasks, GPT models tolerate 50%-75% pre-training sparsity without significant accuracy loss, while higher sparsity causes greater degradation on the more difficult Curation Corpus task.At 75% sparsity, GPT-3 XL loses 2.75 perplexity on Curation Corpus; data-to-text NLG tasks show lower degradation.Source: §3.3 · p. 6
  • At 75% sparsity, the performance drop is smaller for GPT-3 XL than GPT-2 Small on Curation Corpus and DART, indicating greater tolerance to sparsity at larger model size.The Curation Corpus perplexity deltas are +3.76 for GPT-2 Small and +2.75 for GPT-3 XL; the DART BLEU deltas are -1.33 and -0.75, respectively.Source: §3.3 · p. 6

3.4 PRE-TRAINING VS FINE-TUNING PARAMETER SUBSPACES

  • Dense pre-trained weights have small cosine distances from fine-tuned weights across nearly all modules and layers, while 75% sparse weights move more in some modules, including attention output and MLP output projections.The analysis covers DART weights in GPT-2 Small and GPT-3 XL.Source: §3.4 · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
  • Dense GPT-3 XL weights show smaller cosine distances across nearly all modules and layers than GPT-2 Small; even at 75% sparsity, GPT-3 XL requires less parameter movement to remain competitive with the dense model.The authors leave investigating whether to freeze modules with small parameter changes for future work.Source: §3.4 · p. 7, Figure 4 · p. 7

3.5 SPDF TRAINING EFFICIENCY

  • ≈2.5x FLOP reduction over end-to-end dense training is reported for GPT-3 XL with 75% SPDF; GPT-2 Small achieves approximately 1.65x at 75% sparsity.The reported totals include pre-training and dense fine-tuning FLOPs; fine-tuning is a small fraction of pre-training FLOPs.Source: §3.5 · p. 7, Table 2 · p. 8

4 RELATED WORK

Related work contrasts zero- and few-shot use with fine-tuning, which supports downstream performance and behavioral updates. It also motivates limiting fine-tuning capacity while SPDF instead reduces pre-training FLOPs and restores model capacity for fine-tuning.

Zero-Shot vs. Fine-tuning

  • Fine-tuning is presented as important for downstream accuracy and behavioral updates, while this work focuses on pre-training and fine-tuning across generation and summarization tasks.The authors note that fine-tuning can help mitigate harmful outputs in deployed models.Source: §4 · p. 8

Efficient Fine-tuning

  • Prior efficient fine-tuning work limits the layers or parameters adapted, whereas this work reduces pre-training FLOPs and restores all modeling capacity during fine-tuning.The authors leave efficient fine-tuning approaches, including alternative sparsity schedules and low-rank structures, for future work.Source: §4 · p. 8

Weight Sparsification Techniques

  • Static sparsity fixes the mask at initialization, while dynamic sparse training prunes and regrows weights; this study uses static sparsity and leaves dynamic methods for future work.The paper also distinguishes its sparse-pre-training-then-dense-fine-tuning approach from prior work focused on dense-to-sparse fine-tuning.Source: §4 · p. 8

5 CONCLUSION AND FUTURE WORK

The conclusion presents SPDF as a way to reduce GPT training FLOPs and reports high-sparsity pre-training without significant downstream metric loss. The study uses static sparsity and Chinchilla-based pre-training schedules, while identifying sparse training methods and scaling as future directions.

5 CONCLUSION AND FUTURE WORK

  • SPDF uses weight sparsity to reduce the computational FLOPs of training GPT models.The paper frames this as its central contribution.Source: §5 · p. 8
  • GPT models were pre-trained at 50%-75% sparsity without significant loss in downstream task metrics.The conclusion presents this as the first such result for a large GPT model, to the authors’ knowledge.Source: §5 · pp. 8–9
  • The study uses simple static sparsity and Chinchilla schedules, and proposes dynamic sparsity, improved sparse optimization, sparse-friendly architectures, and non-Chinchilla scaling for future investigation.The authors say the Chinchilla schedule is FLOP-optimal for dense models and plan to test how it transfers to sparse models.Source: §5 · p. 9

A.1 PRE-TRAINING ON PILE

The appendix describes model training configurations and a uniform static-sparsity setup. Models are trained to Chinchilla-optimal configurations, with approximately 20 tokens per parameter.

A.1 PRE-TRAINING ON PILE

  • The GPT models use AdamW with β1 = 0.9, β2 = 0.999, ϵ = 10−8, global-norm clipping at 1.0, and weight decay 0.1.Learning rate warms up for the first 375M tokens and then cosine-decays to 10% of its peak value.Source: §A.1 · p. 14
  • The models are configured to Chinchilla-optimal training at approximately 20 tokens per parameter, with tabled sizes, architectures, batch sizes, and learning rates.The architecture details include parameter count, decoder layers, model width, feedforward width, and attention-head dimensions.Source: Table 1 · p. 14, §A.1 · p. 14
  • Random static sparsity is distributed uniformly across layers and applied to dense MLP and self-attention linear weights, while embeddings, LayerNorm, and biases remain dense.The MLP matrices are WI and WO; the self-attention matrices are WQ, WK, WV, and WD.Source: §A.1 · p. 14

A.2 NATURAL LANGUAGE GENERATION

The appendix describes fine-tuning optimization and reports the scope of pre-training FLOP accounting. Its table includes forward and backward passes and relative reductions against dense baselines.

A.2 NATURAL LANGUAGE GENERATION

  • Fine-tuning uses AdamW with a linear learning-rate schedule for five epochs, early stopping, and validation-set selection of batch size and learning rate.Batch sizes are selected from {8, 16, 32, 64}, and learning rates from {1e-4, 5e-5, 2.5e-5}.Source: §A.2 · p. 14
  • Table 2 reports total pre-training FLOPs for each pre-trained model and relative FLOP reductions of sparse variants over dense baselines; per-sequence FLOPs include both forward and backward passes.Source: Table 2 · p. 15

A.3 CURATION CORPUS

The Curation Corpus fine-tuning setup uses five epochs with early stopping when overfitting begins. Learning rates are selected by validation perplexity within a fixed compute budget.

A.3 CURATION CORPUS

  • Fine-tuning runs for 5 epochs, with early stopping once models start to overfit.The setup follows [Rae et al., 2021].Source: §A.3 · p. 15
  • A grid search selects the learning rate with the best validation perplexity for a given compute budget.Source: §A.3 · p. 15
  • The dense baseline and sparse variants use batch size 32 and search six learning rates from 3e-3 to 1e-5.The candidates are 3e-3, 1e-3, 3e-4, 1e-4, 3e-5, and 1e-5.Source: §A.3 · p. 15

A.4 TRAINING FLOPS FOR PRE-TRAINING AND FINE-TUNING

The section compares pre-training FLOPs for dense and sparse GPT-2 Small and GPT-3 XL variants with dense fine-tuning FLOPs across four tasks. Fine-tuning accounts for a small fraction of pre-training FLOPs, even when performed densely.

A.4 TRAINING FLOPS FOR PRE-TRAINING AND FINE-TUNING

  • Dense fine-tuning uses only a small fraction of total pre-training FLOPs, despite being performed densely.Source: §A.4 · p. 15
  • Pre-training FLOPs and reductions relative to the dense baseline are reported for dense and sparse GPT-2 Small and GPT-3 XL variants.Source: §A.4 · p. 15
  • Dense fine-tuning FLOPs are reported for GPT-2 Small and GPT-3 XL on E2E, WebNLG, DART, and Curation Corpus.FLOPs per sequence include both forward and backward passes.Source: §A.4 · p. 15, Table 3 · p. 15

B DETAILED EXPERIMENTS ON NLG AND TEXT SUMMARIZATION

Detailed results cover official evaluation metrics for E2E, WebNLG, and DART. Across these metrics, the authors report no significant performance drop for sparse-pre-trained GPT-2 Small and GPT-3 XL models.

B DETAILED EXPERIMENTS ON NLG AND TEXT SUMMARIZATION

  • Across the official evaluation metrics for E2E, WebNLG, and DART, sparse-pre-trained GPT-2 Small and GPT-3 XL show no significant performance drop.Source: B DETAILED EXPERIMENTS ON NLG AND TEXT… · p. 15, Table 4 · p. 16, Table 5 · p. 16
  • E2E results report downstream accuracy for GPT-2 Small and GPT-3 XL across pre-training sparsity levels, with metric arrows indicating the better direction.Source: Table 4 · p. 16
  • WebNLG results cover BLEU, NIST, METEOR, ROUGE-L, and CIDEr for GPT-2 Small and GPT-3 XL at different pre-training sparsity levels.The metric arrows indicate the better direction.Source: Table 5 · p. 16
  • DART results cover BLEU, MET, and TER for GPT-2 Small and GPT-3 XL at different pre-training sparsity levels.The metric arrows indicate the better direction.Source: Table 6 · p. 16

C UNSTRUCTURED SPARSITY ON SPECIALIZED HARDWARE ACCELERATORS

The section uses a GPT-3 layer matrix-multiplication figure to illustrate potential gains from unstructured weight sparsity on the Cerebras CS-2. It compares measured and theoretical speedups across sparsity levels.

C UNSTRUCTURED SPARSITY ON SPECIALIZED HARDWARE ACCELERATORS

Related papers

Loading 2303.10464v2…