PaperlayerView exact revision on arXiv

Source-linked AI summary

Cramming: Training a Language Model on a Single GPU in One Day

Jonas Geiping, Tom Goldstein

arXiv:2212.14034v1cs.CLcs.LG

TL;DR

Because large-scale training has made language models feel inaccessible to many researchers, this paper asks what can be achieved from scratch on one consumer GPU in one day. It tests constrained transformer MLM training and reports decent GLUE performance, while finding that scaling down remains difficult.

  • Problem

    The drive toward extremely large language models has made training feel out of reach for many researchers and practitioners.

    Source: §1 · p. 1

  • Method

    The paper investigates transformer masked-language-model pretraining from scratch under a single-GPU, 24-hour budget, varying architecture, training, and data choices.

    Source: §4.2 · p. 6, §4.3 · p. 8, §4.4 · p. 9

  • Results

    Several strands of modification yield decent GLUE performance, while larger-model gains are evened out by slower speed.

    Source: §7 · p. 12

  • Takeaways & Limitations

    The study offers a baseline for further exploration of cramming and assesses recent transformer improvements under limited compute.

    Source: §7 · p. 12

    The investigation is limited to transformer architectures trained with masked-language-modeling objectives.

    Source: §6 · p. 11

Abstract

from arXiv · show

Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners. While most in the community are asking how to push the limits of extreme computation, we ask the opposite question: How far can we get with a single GPU in just one day? We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU. Aside from re-analyzing nearly all components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, we investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we categorize a range of recent improvements to training and architecture and discuss their merit and practical applicability (or lack thereof) for the limited compute setting.

1 SCALING UP AND SCALING DOWN

Transformer scaling has driven major language-model gains, but the resulting compute demands leave many researchers and practitioners feeling unable to train models. This paper asks what scratch training can achieve with one GPU in one day and investigates which changes help in that setting.

1 SCALING UP AND SCALING DOWN

  • Scaling has become the dominant performance strategy, with model performance improving as parameter counts and data increase, following power laws.The authors frame their question as testing what remains possible when scaling resources down.Source: §1 · p. 1
  • The race toward extremely large models has made language-model training feel inaccessible to many researchers and practitioners.Training runs have reached zettaFLOP and later yottaFLOP scales.Source: §1 · p. 1
  • The paper asks what downstream performance a modest researcher can achieve by training from scratch on one GPU for one day.The authors also ask what trade-offs emerge when scaling down.Source: §1 · p. 1
  • If scaled-down pretraining is a viable analogue to large-compute training, it could support empirical studies of data attribution, security, stability, and generalization.The authors also point to situations where practitioners may want to retrain using specialized or trustworthy data sources.Source: §1 · p. 2
  • The authors use the one-day, single-GPU challenge to benchmark conceptual progress beyond simply increasing compute.They note that BERT-like performance with modest resources would have seemed unthinkable in 2018.Source: §1 · p. 2
  • The authors investigate pipeline modifications, report that scaling laws remain relevant in the constrained setting, and find that some training-recipe changes improve performance without reducing model size.Their models achieve respectable GLUE performance, often close to and sometimes exceeding BERT.Source: §1 · p. 2

2 TYING OUR HANDS BEHIND OUR BACK: A SETUP WITH LIMITED COMPUTE

The setup defines a constrained pretraining and evaluation challenge: train a masked-language model from scratch on one GPU for 24 hours, without relying on pretrained models. It permits raw-text choices while bounding downstream GLUE tuning.

2 TYING OUR HANDS BEHIND OUR BACK: A SETUP WITH LIMITED COMPUTE

  • The challenge trains a transformer language model with masked-language modeling from scratch, using a single GPU for 24 hours.The model may be any size, and existing pretrained models cannot be used in the pipeline.Source: §2 · p. 2
  • Raw text other than downstream data may be used, including sampling choices that do not require a pretrained model.Downloading and preprocessing raw data are outside the compute budget, but representation learning must count toward runtime.Source: §2 · p. 2
  • GLUE evaluation uses at most five finetuning epochs per task, shared global hyperparameters, and excludes finetuning from the compute budget.Only each downstream task's training data may be used for finetuning.Source: §2 · p. 2
  • The setup compares available compute budgets across selected training runs, with table columns for group, accelerator, time limit, and total exaFLOPs.Large-scale language models are included for reference.Source: Table 1 · p. 3, §2 · p. 3
  • The setup excludes pretrained-model distillation and data filtering, and limits downstream gains from extensive training or task-specific hyperparameter tuning.The authors also avoid restricting training data to BERT's original dataset.Source: §2 · p. 3

3 RELATED WORK ON EFFICIENT TRANSFORMERS

Prior work on transformer improvements and BERT training generally targets more compute than this paper's setting. Its review notes that proposed modifications often fail to yield reliable gains, while scaling laws help explain why improvements under fixed compute can be modest.

3 RELATED WORK ON EFFICIENT TRANSFORMERS

  • Prior BERT training estimates vary with hardware and efficiency measures, and published recipes often assume full GPU server nodes or TPU slices.Peak available FLOPs provide an upper bound on a run's compute budget rather than its realized throughput.Source: §3 · p. 3
  • SqueezeBERT used eight Titan RTX cards for four days, while Sellam et al. report that doubling BERT's training time more reliably reproduces its original results.Both examples target compute settings closer to the original BERT run.Source: §3 · p. 3
  • Izsak et al. trained a BERTLARGE variant for 24 hours on eight V100 GPUs; this paper re-evaluates that setup with about 15 times less compute.Their recipe includes sequence length 128 and several training modifications.Source: §3 · pp. 3–4
  • Meta-studies find that many proposed transformer modifications do not reliably improve final accuracy, and only a few outperform the original architecture across scales.These studies evaluate larger compute settings than the limited-compute focus here.Source: §3 · p. 4
  • Under fixed compute, scaling laws predict only mild performance dependence on model size because larger models process less data but improve faster by nearly the same margin.The authors note that the overall logic of these laws appears to hold even at smaller scales.Source: §3 · p. 4

4 INVESTIGATIONS

The investigations test which pipeline changes matter under constrained training and find that scaling laws limit gains from major model-size or architecture changes. Within that constraint, targeted architecture, training, and data choices improve computation or downstream performance.

4.1 IMPLEMENTATION DETAILS

  • Experiments used shared PyTorch-level implementations with automated operator fusion across components, re-enabling efficient attention only after selecting the final architecture; tokenized data used randomized 128-token sequences and a 32,768-word vocabulary.Source: §4.1 · pp. 4–5

4.2 MODIFYING THE ARCHITECTURE

  • Across 24 hours, transformer variants reach MLM loss around 1.9; per-token efficiency depends mainly on model size, limiting gains from major architectural reshaping.Models with more parameters learn faster per gradient, while smaller models process more tokens; after the first 1B tokens, architecture curves differ mainly by a size-dependent shift.Source: §4.2 · pp. 5–6, Figure 1 · p. 5
  • Removing attention and feedforward biases speeds gradient computation with little change in model size; fewer heads speed gradients but hurt fine-tuning, so all 12 remain.A gated linear unit brings small improvements, while the original feedforward block and GELU are largely retained.Source: §4.2 · p. 6
  • Scaled sinusoidal embeddings add incremental gains; pre-normalization stabilizes training and permits larger learning rates and reduced warmup, while other embedding and head variants add little or none.Removing the nonlinear head has no ill effect; a final Layer Norm is added to further stabilize training.Source: §4.2 · pp. 6–7

4.3 MODIFYING THE TRAINING SETUP

  • A one-cycle schedule peaking at 10^-3 gives minimal pretraining loss within the budget, and triangular one-cycle schedules show better end-time behavior.The authors suggest quick annealing may explain that end-time behavior.Source: Figure 2 · p. 7, §4.3 · p. 7
  • For the 2080ti, batch size 1,536 minimizes pretraining loss and 4,032 maximizes downstream performance; linear accumulation helps modestly, and dropout is disabled during pretraining.The linear schedule increases averaged micro-batches over training; dropout is re-enabled for downstream fine-tuning.Source: §4.3 · p. 8

4.4 OPTIMIZING THE DATASET

  • The Pile performs best on MNLI; filtering C4 for hard-to-compress entries improves it, while exact deduplication does not, and token-prevalence sorting and larger final batches also help.The larger final batch is disproportionately effective on C4 compared with bookcorpus-wikipedia.Source: Table 2 · p. 8, §4.4 · p. 9
  • On bookcorpus-wikipedia, larger vocabularies correlate with higher average GLUE scores, but MNLI plateaus near the original 32,768 vocabulary size, which the authors retain.The vocabulary-size comparison is shown for models trained in the cramming regime.Source: Figure 3 · p. 9, §4.4 · p. 9

5 FINETUNING PERFORMANCE ON GLUE

Under the fixed GLUE fine-tuning protocol, the modified recipe achieves decent downstream performance, particularly on larger datasets, while CoLA remains a notable weakness. Ablations attribute the largest gains to architecture changes, and the recipe also generalizes to a larger compute budget with uneven task-level results.

5 FINETUNING PERFORMANCE ON GLUE

  • The crammed recipe performs surprisingly decently on GLUE, especially MNLI, QQP, QNLI, and SST-2, and gains substantially over naive BERT training and the Izsak et al. (2021) recipe.Fine-tuning uses a five-epoch limit and fixed hyperparameters across tasks; the Izsak et al. recipe was designed for an eight-GPU server and was squeezed onto one GPU here.Source: §5 · pp. 9–10, Table 3 · p. 10
  • The average GLUE score is reduced by a significant CoLA drop, which the authors hypothesize may reflect task-specific fine-tuning hyperparameters or insufficient text processing.They also note that BERT can use global hyperparameters, suggesting a possible deficiency in the crammed model; these explanations are hypotheses.Source: §5 · p. 10

5.1 ABLATION - WHICH CHANGES REALLY MATTERED?

  • Ablations attribute about two average GLUE points to architecture changes, one point to data changes, and half a point to training modifications.Training and architecture modifications must co-occur; the aggressive learning-rate schedule requires pre-normalization Layer Norms.Source: §5.1 · p. 10, Table 5 · p. 11

5.2 WHAT HAPPENS WHEN TRAINING LONGER?

  • With 16 times the compute, the recipe performs in a similar range to RoBERTa-base on some tasks, but additional compute barely improves CoLA and RTE.The 48-hour, eight-A6000-GPU runs reuse the same setup and data, with the dataset repeated multiple times.Source: Table 6 · p. 11, §5.2 · p. 11

6 LIMITATIONS

The investigation is limited to transformer architectures trained with masked language modeling, leaving alternative objectives and architectures outside its scope.

6 LIMITATIONS

  • The study covers transformer-based architectures trained with masked language modeling, not other pretraining objectives or model families.The authors identify ELECTRA and non-transformer architectures as possible directions for cramming studies.Source: §6 · p. 11
  • Alternative pretraining objectives have been proposed, and the authors note that ELECTRA might benefit crammed models.They also cite work finding that masked language modeling continues to hold up well as a pretraining objective.Source: §6 · p. 11
  • The authors leave open whether a non-transformer architecture could be optimal for the cramming task.Their investigation itself remains restricted to transformers.Source: §6 · p. 11

7 CONCLUSIONS

The paper reports decent GLUE performance from several modification strands in a very limited-compute setting, while finding that cramming remains hard and scaling-law implications persist. It offers a baseline for further cramming investigations and provides code to reproduce its experiments.

7 CONCLUSIONS

  • Several modification strands yield decent downstream GLUE performance for transformer language models trained under very limited compute.The paper presents this as an investigation of how much performance can be achieved in the crammed setting.Source: §7 · p. 12
  • Cramming remains hard: implications of Kaplan et al. (2020) still appear to hold, and larger-model improvements are evened out by slower speed.Source: §7 · p. 12
  • The authors offer the work as a baseline for further cramming investigations and provide code to reproduce all experiments.Source: §7 · p. 12, REPRODUCIBILITY STATEMENT · p. 12

A OTHER MODIFICATIONS

The paper lists several developments and modification families that its study did not include.

A OTHER MODIFICATIONS

B ADDITIONAL INFORMATION

The appendix directs readers to tables containing additional architecture and training-modification results, while noting that some results are not printed there.

B ADDITIONAL INFORMATION

Related papers

Loading 2212.14034v1…