PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Learning Rate Tuning in the Era of Large Language Models

Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, Yanzhao Wu

arXiv:2309.08859v1cs.LG

TL;DR

Existing learning-rate policies may not transfer well from traditional DNN training to costly LLM fine-tuning, where poor choices affect convergence, quality, and tuning expense. The paper reassesses these challenges, introduces LRBench++, and evaluates policies across DNN and LLM settings. Experiments show substantial policy sensitivity and support the need for dedicated learning-rate tuning for LLMs.

  • Problem

    Traditional learning-rate practices may not work well for LLM fine-tuning, while poor rates can impair convergence, quality, and efficiency.

    Source: §I · p. 1, §III · p. 3

  • Method

    The paper introduces LRBench++, a benchmark and tuning system supporting DNNs, LLMs, multiple learning-rate policies, evaluation metrics, and search algorithms.

    Source: §IV · pp. 3–4

  • Results

    Learning-rate choices substantially affect outcomes: ResNet32 reaches 66.96% with fixed k=0.0001 versus over 91% for most other policies, while LLM results vary across policies and tasks.

    Source: §V · pp. 5, 7

  • Takeaways & Limitations

    Learning-rate tuning should be reassessed for LLMs rather than copied directly from traditional DNN training practices.

    Source: §III · p. 3, §VII · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) represent the recent success of deep learning in achieving remarkable human-like predictive performance. It has become a mainstream strategy to leverage fine-tuning to adapt LLMs for various real-world applications due to the prohibitive expenses associated with LLM training. The learning rate is one of the most important hyperparameters in LLM fine-tuning with direct impacts on both fine-tuning efficiency and fine-tuned LLM quality. Existing learning rate policies are primarily designed for training traditional deep neural networks (DNNs), which may not work well for LLM fine-tuning. We reassess the research challenges and opportunities of learning rate tuning in the coming era of Large Language Models. This paper makes three original contributions. First, we revisit existing learning rate policies to analyze the critical challenges of learning rate tuning in the era of LLMs. Second, we present LRBench++ to benchmark learning rate policies and facilitate learning rate tuning for both traditional DNNs and LLMs. Third, our experimental analysis with LRBench++ demonstrates the key differences between LLM fine-tuning and traditional DNN training and validates our analysis.

I. INTRODUCTION

LLM training is prohibitively expensive, making fine-tuning the practical route for adapting released foundation models. Because existing tuning practices inherit traditional DNN assumptions, learning-rate policies require systematic reassessment for LLM fine-tuning.

  • Training LLMs from scratch requires thousands of GPUs and investments exceeding $10 million, unlike generally affordable traditional DNN training.Source: §I · p. 1
  • Fine-tuning is expected to drive LLM adoption, but systematic evidence on achieving high efficiency and accuracy remains limited.Source: §I · p. 1
  • Learning-rate policies directly affect training effectiveness and model accuracy, while unsuitable values can prevent convergence or yield suboptimal performance.Source: §I · p. 1
  • Manual trial-and-error learning-rate tuning is tedious, time-consuming, and especially expensive for LLMs.Source: §I · p. 1
  • The paper studies whether LLM complexity and distinctive characteristics create new learning-rate tuning challenges and opportunities.Source: §I · p. 2

II. RELATED WORK

Prior work organizes learning-rate tuning into formula-, state-, and exploration-based policies, while general hyperparameter frameworks provide broader search tools. The paper identifies a gap in dedicated learning-rate support and positions LLM fine-tuning as an emerging application area.

Learning Rate Tuning

  • Existing learning-rate policies comprise formula-based, state-based, and exploration-based approaches for traditional DNN training.Source: §II · p. 2

Hyperparameter Tuning

  • General hyperparameter frameworks use methods such as Bayesian optimization, bandits, and population-based search, but lack dedicated support for complex learning-rate policies.Source: §II · p. 2

LLM Fine-tuning

  • Fine-tuning adapts pretrained LLMs to new tasks with lower training costs and less data than training from scratch, with Alpaca, Koala, and Vicuna as examples.Source: §II · p. 2
  • The study presents itself as the first to explore learning-rate tuning for LLMs.Source: §II · p. 2

III. PROBLEM STATEMENT

LLM fine-tuning iteratively updates a pretrained model using data, a loss function, an optimizer, and a learning-rate policy. Its billion-parameter scale, high cost, few epochs, initialization, and difficult evaluation make traditional DNN learning-rate practices unreliable.

III. PROBLEM STATEMENT

  • LLM fine-tuning uses inference, loss-based gradient computation, and optimizer-policy updates to improve a pretrained model on a new dataset or task.Source: §III · p. 2
  • LLMs typically contain billions of parameters, while traditional DNNs contain millions; training LLMs from scratch can exceed 80k GPU hours.Source: §III · p. 3
  • LLM fine-tuning starts from pretrained initialization, usually takes 2–3 epochs, and may require costly real-world or LLM-based evaluation.Source: §III · p. 3
  • Traditional cyclic learning rates often use 4–6-epoch step sizes, which does not fit LLM fine-tuning regimes lasting only 2–3 epochs.Source: §III · p. 3
  • Different learning-rate policies produce distinct optimization paths: TRIEXP progresses fastest initially, FIX turns toward the wrong direction, and only NSTEP reaches the global optimum.Source: §III · p. 3, Figure 3 · p. 3
  • Learning-rate research must address policy evaluation, policy selection and composition, and parameter tuning across training stages.Source: §III · p. 3

IV. LRBENCH++ OVERVIEW

LRBench++ is a modular system for evaluating, selecting, and tuning learning-rate policies across DNN training and LLM fine-tuning. It combines monitoring, policy evaluation, search, visualization, benchmarking, and objective-specific tuning support.

  • LRBench++ supports benchmarking, selecting, and tuning learning-rate policies for both DNN training and LLM fine-tuning.Source: §IV · p. 3
  • The system extends LRBench with LLM support, LLM evaluation techniques, grid and random search, and tuning methods for accuracy or cost objectives.Source: §IV · p. 4
  • Its six core components monitor training, store policy results, narrow learning-rate ranges, visualize status, evaluate policies, and optimize tuning.Source: §IV · p. 4, Figure 4 · p. 4
  • Iteration-based tuning provides finer-grained control for LLM use cases that often involve only 2∼3 epochs, alongside NLP tasks such as ARC, MMLU, HellaSwag, and TruthfulQA.Source: §IV · p. 4
  • LRBench++ supports single-policy selection, multi-policy composition, and cost-effective tuning to maximize accuracy or minimize cost under target constraints.Source: §IV · p. 4
  • The platform is intended to let users collect and share tuning results for traditional DNNs and LLMs, with an open-source implementation.Source: §IV · p. 4

V. EXPERIMENTAL ANALYSIS

LRBench++ evaluates learning-rate policies across traditional DNN training and LLM fine-tuning, revealing that policy choice and configuration materially affect accuracy, cost, and task performance. Composite and cyclic policies improve DNN results, while LLM outcomes vary across tasks and may not track training loss.

Accuracy Comparison of LR Policies

  • 92.91% accuracy from the advanced composite LR exceeded SINEXP's 92.81% on CIFAR-10 ResNet32, while a three-FIX composite reached 69.05% Top-1 and 88.76% Top-5 on ImageNet ResNet18.The composite policy adapted learning rates across training stages and produced the strongest reported DNN results.Source: §V · p. 5, Table II · p. 5

LR Tuning for Cost Reduction

  • Cyclic policies reduced iterations needed to reach target accuracy, from 10,000 to 3,500 on MNIST and from 64,000 to 20,000 on CIFAR-10.These correspond to 2.86× and 3.2× cost reductions, respectively.Source: §V · p. 5, Table III · p. 6

LR Tuning by Grid Search

  • Grid search identified TRIEXP with λ=0.05 as the best LeNet-MNIST configuration at 99.35% accuracy, while other policy-coefficient combinations missed or exceeded the target substantially.Further tuning found k1/k0=3 remained optimal, with accuracies ranging from 99.10% to 99.35%.Source: §V · p. 6, Figure 6 · p. 6

Impact of LR Tuning Algorithms

  • Grid search selected LR coefficients more accurately and with lower standard deviation than random search in repeated LeNet-MNIST experiments.Both methods evaluated five coefficient choices over the same [0.001, 0.1] range, but grid choices were based on LRBench++ recommendations.Source: §V · p. 6, Table IV · p. 6

Impact of LRs on LLM Fine-tuning

  • LLM fine-tuning performance varied sharply with learning-rate configuration: 2 × 10^-5 achieved 0.5392 ARC versus 0.4667 for a 6 × 10^-5 linear-decaying LR after two epochs.The latter score was below the pre-trained baseline, showing that learning-rate selection can materially affect task outcomes.Source: §V · pp. 6–7, Table V · p. 7
  • No single configuration outperformed all others across tests; fixed 1 × 10^-5 performed best on ARC after 2 epochs and HellaSwag after 3 epochs, while loss did not reliably predict task performance.At epoch 2, fixed LR 2 × 10−5 produced the highest training loss yet the best ARC score; training loss showed significant drops at epochs 2 and 3.Source: §V · p. 7

VI. DISCUSSION

The discussion reviews common LLM fine-tuning settings and identifies cost-effective tuning, benchmarking, and performance evaluation as key learning-rate challenges.

Common LR Settings for LLM Fine-tuning

  • Most existing LLM fine-tuning studies simply use the default initial learning rate of 2 × 10−5, according to a review of popular LLM fine-tuning practices.Source: §VI · p. 7

Challenges and Opportunities in LR Tuning

  • Cost-effective tuning seeks learning-rate policies or strategies that achieve desired accuracy within the computational budget of LLM training and fine-tuning.Source: §VI · p. 8
  • Benchmarking must capture learning-rate effects across hyperparameter configurations, computational costs, and trained or fine-tuned model quality.Source: §VI · p. 8
  • Because fine-tuning loss may not reflect task-specific performance, loss-monitoring policies such as Reduce-LR-On-Plateau and Change-LR-On-Plateau may be unsuitable.Source: §VI · p. 8

VII. CONCLUSION

The paper identifies core learning-rate challenges across traditional DNNs and LLMs, introduces LRBench++, and empirically validates the need for further study.

VII. CONCLUSION

  • LRBench++ benchmarks and tunes learning-rate policies for traditional DNN training and LLM fine-tuning, while experiments validate the paper’s analysis.Source: §VII · p. 8

Related papers

Loading 2309.08859v1…