PaperlayerView exact revision on arXiv

Source-linked AI summary

FLM-101B: An Open LLM and How to Train It with $100K Budget

Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun, Yequan Wang

arXiv:2309.03852v3cs.CLcs.AI

TL;DR

LLM training costs are substantial, while progressive training beyond 100B parameters remains underexplored. This paper trains FLM-101B through sequential model growth and reports that it is competitive and robust at low cost, while noting limited exploration of growth schedules, operators, and data amounts.

  • Problem

    LLM training costs are rising, and progressive training methods for models beyond 100B parameters remain underexplored.

    Source: §1 · p. 1

  • Method

    FLM-101B is trained by sequentially growing 16B, 51B, and 101B models, with each stage inheriting knowledge from its predecessor.

    Source: §2.3 · p. 2

  • Results

    The authors report FLM-101B as competitive and robust despite its low training cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors release the checkpoint, code, and tools to promote research related to progressive training.

    Source: §1 · p. 2

    Resource constraints limited exploration and comparison of growth schedules, growth operators, and amounts of data.

    Source: §6 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) are considered important approaches towards foundational machine intelligence, achieving remarkable success in Natural Language Processing and multimodal tasks, among others. However, the carbon footprints and financial costs originating from heavy pre-training computation is a non-negligible issue. Progressive training methods, inspired by the neurogenesis process that grows neural structures, have shown potential to accelerate LLM pre-training. However, the algorithms, implementation, and practices for progressively training LLMs beyond 100B parameters remain underexplored. In this paper, we show that our model, namely FLM-101B, trained with our growth strategy under a budget of \$100K, reaches 80\% of the baselines' performances with only 10\% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B.

1 Introduction

The introduction frames rising LLM training costs as a challenge for Green AI and explores progressive growth as a way to train a 100B+ model within a $100K budget. It evaluates FLM-101B against earlier large models using knowledge and IQ-inspired tasks.

1 Introduction

  • LLM training costs rise alongside larger data and open-model scales, motivating methods to reduce expenses in line with Green AI.The introduction cites training sets of 1.4T, 2T, and 15T tokens for Llama-1, Llama-2, and Llama-3, respectively.Source: §1 · p. 1
  • FLM-101B uses an aggressive growth strategy adapted from MSG to explore training a 100B+ model with a $100K budget.The authors compare it with first-generation 100B+ models because a strict same-model, fixed-size comparison is impractical at this scale.Source: §1 · pp. 1–2
  • The evaluation pairs knowledge benchmarks with IQ-inspired tasks covering symbolic mapping, rule understanding, pattern mining, and anti-interference.The authors argue knowledge tests alone may not distinguish recall from reasoning or inference, while the additional tasks may be less affected by leakage or memorization.Source: §1 · p. 2
  • The authors report FLM-101B as competitive and robust despite its low training cost, and plan to release its checkpoint, code, and tools.They describe this as their first attempt, to their knowledge, to train a 100B+ LLM from scratch using a growth strategy.Source: §1 · p. 2

2 Design Overview of FLM-101B

FLM-101B combines a modified FreeLM backbone and bilingual pre-training with sequential growth from 16B to 51B to 101B parameters. Its parallel setup exceeds 160 teraFLOPs/sec per GPU in every stage, with utilization of at least 51.3%.

2.1 Architecture

  • The backbone adopts modified FreeLM, a decoder-only GPT-style architecture with language and teacher objectives, and integrates xPos for long-sequence modeling.The design preserves GPT-3-style transformer blocks and uses a GPT-4-derived tokenizer.Source: §2.1 · p. 2

2.2 Pre-Training Setup

  • FLM-101B is English-Chinese bilingual, mixing the corpora at approximately 53.5% : 46.5% and including OIG and COIG instruction data during pre-training.Teacher signals are used only for the 16B eFLM-16B model; its samples are mixed in every batch to improve sampling consistency and training stability.Source: §2.2 · p. 2

2.3 Growth Strategy

  • The model grows progressively from 16B to 51B and then 101B parameters, with each model inheriting knowledge from its predecessor.This contrasts with independently training models of different sizes.Source: §2.3 · p. 2, §2.1 · p. 2
  • Function-preserving growth uses adapted MSG operators, while the schedule allocates 245.37B, 39.64B, and 26.54B tokens to the 16B, 51B, and 101B stages.The 21.54-day schedule is reported as 72% time-saving, or a 3.56x speedup, versus an estimated 76.74 days to train a 101B model from scratch.Source: §2.3 · p. 3

2.4 The Parallelism Setup and Model Configurations

  • Every growth stage exceeds 160 teraFLOPs/sec per GPU, with utilization of at least 51.3%.The cited comparison reports GLM-130B at 135 teraFLOPs/sec and 42.27% utilization.Source: §2.4 · p. 3

3 Training Stability of FLM-101B

The paper addresses instability in models beyond 100B parameters by selecting hyperparameters with a small proxy model and applying µP transfer. The authors report stable training through growth to 101B parameters.

3 Training Stability of FLM-101B

  • Models beyond 100B parameters can encounter loss divergence, gradient explosion, and numerical overflow or underflow, increasing tuning and maintenance costs and making budgets unpredictable.The section presents its stability methods as a response to these problems.Source: §3 · pp. 3–4
  • A grid search on a model under 100M parameters selects learning rate 4e −4, standard deviation 1.6e −2, and softmax temperature 2.0 for transfer via µP.The search varies those three hyperparameters; applying them to FLM-16B is reported to avoid instabilities and growth-related divergence in FLM-51B and FLM-101B.Source: §3 · p. 4
  • Bfloat16 is used instead of FP16 for its precision near zero; the authors report avoiding the FP16 underflow issue and not needing loss-scale adjustments.Mixed-precision training is applied to save runtime memory and reduce time costs.Source: §3 · p. 4
  • The training loss curve steepens after each growth, and the 101B stage remains stable despite the 51B stage using only 40B tokens.The authors describe the training procedure as robust and predictable.Source: §3 · p. 4

4 Benchmark Evaluation

The section compares FLM-101B with existing models on knowledge benchmarks, examines performance with added professional knowledge, and evaluates its progressive growth stages.

4.1 Open LLM Evaluation

  • FLM-101B averages 43.94, reaching over 90% of GLM-130B's performance despite GLM-130B using 7 times more FLOPs.Both models underperform the Llama series; the authors suggest architecture and training-data refinement may contribute.Source: Table 4 · p. 5, §4.1 · p. 5

4.2 Evaluation on the Professional Knowledge-Enhanced Model

  • On MMLU, eFLM-16B scores 44.50 versus GLM-130B's 42.59, and on C-Eval it leads GLM-130B by about 2 points.Vanilla FLM-16B scores 27.02 on MMLU and averages 27.0 on C-Eval, below GLM-130B.Source: Table 5 · p. 5, §4.2 · p. 6

4 Benchmark Evaluation

  • FLM-101B's estimated training cost is 52.76 zettaFLOPs, versus 421.60 zettaFLOPs for GLM-130B; its estimated pre-training carbon footprint is one-tenth of a typical LLM.The cost estimates use FLOPs as an index because infrastructure and prices differ and full configurations are often unavailable.Source: §4 · pp. 4–5

4.1 Open LLM Evaluation

  • With less than 0.16T English data, FLM-101B achieves the best baseline TruthfulQA accuracy, 41.47.The authors report that TruthfulQA, ARC, and HellaSwag performance improves with increased data and reduced training loss.Source: §4.1 · p. 5

4.3 Evaluation of the Growth Strategy

  • Across the 16B, 51B, and 101B stages, performance rises with model size; FLM-101B is best on almost all tasks and improves more than 51B with less data.The authors interpret this as evidence that the model inherits knowledge across growth stages and incorporates new weights after growth.Source: Table 6 · p. 6, §4.3 · p. 6

5 Evaluation Inspired by IQ Tests

The section supplements knowledge benchmarks with IQ-inspired evaluations of inference and generalization in new contexts, covering symbolic mapping, rule understanding, pattern mining, and anti-interference.

Symbolic Mapping

  • Symbolic mapping replaces categorical labels with symbols unlikely to occur in training data, probing language understanding and generalization to a new context.The authors construct SuperGLUE-IQ and CLUE-IQ from SuperGLUE and CLUE.Source: §5 · p. 6
  • With one magnitude less computation, FLM-101B matches GPT-3 on SuperGLUE-IQ and outperforms GLM-130B on CLUE-IQ.The symbolic-mapping evaluations use labels unlikely to appear in training data to probe generalization in a new context.Source: §5 · pp. 6–7, Table 7 · p. 7, Table 8 · p. 7

Rule Understanding

  • FLM-101B scores 69.59% on Counting, about 9 points above GLM-130B; GPT-3 places first on Counting and Replace-Lowercase, and second on Replace-Word.The rule-understanding evaluation includes 0-shot Counting and 4-shot String replacement tasks.Source: §5 · p. 7, Table 9 · p. 7

Pattern Mining

  • On Head & Tail and Full Repeating, FLM-101B trails GPT-3 by a few points, but it outperforms GPT-3 on Head Slicing and GLM-130B on all three tasks.Each pattern-mining task uses 5-shot evaluation with 100 instances.Source: §5 · p. 7, Table 10 · p. 7

Overall results

  • Across four IQ-inspired evaluations, FLM-101B outperforms GLM-130B and is competitive with GPT-3 on some tasks at much lower cost.For anti-interference, it achieves the second-best passing rates, about 7% above GLM-130B.Source: §5 · p. 7, Table 11 · p. 7

Anti-interference

  • FLM-101B achieves the second-best passing rates on anti-interference, with an advantage of around 7% over GLM-130B.The evaluation covers Multiple Key Retrieval, Single Supporting Fact Tracking, and Two Supporting Facts Tracking.Source: §5 · p. 7, Table 11 · p. 7

6 Conclusions, Limitations, and Future Work

The authors present FLM-101B as an open-sourced model trained from scratch for $100,000, with benchmark performance comparable to strong baselines at lower computational cost. They identify limited exploration of growth choices and data as a constraint and suggest further work on growth and training stability for scaling.

Conclusions

  • FLM-101B is an open-sourced LLM trained from scratch within a $100,000 budget, with comparable performance to strong baselines at lower computational cost.The reported comparisons cover knowledge-oriented and IQ-related benchmarks.Source: §6 · p. 7
  • The authors caution that harmful content may be induced from the open-sourced checkpoint and does not represent their opinions.Source: §6 · p. 7

Limitations

  • Resource constraints left growth schedules, growth operators, and data amounts inadequately explored or compared.The authors identify these areas as limitations of the work.Source: §6 · p. 7

Future Work

  • Further exploration of growth strategies and training stability may benefit future attempts to scale LLMs beyond 1T parameters.The authors frame this as a potential benefit for future work.Source: §6 · p. 8

Related papers

Loading 2309.03852v3…