PaperlayerView exact revision on arXiv

Source-linked AI summary

QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer

arXiv:2305.14314v1cs.LG

TL;DR

Finetuning very large language models is prohibitively expensive, and existing quantization methods generally fail during training. QLoRA enables 4-bit finetuning with low-rank adapters, reducing 65B-model memory needs below 48GB while reaching 99.3% of ChatGPT’s Vicuna performance.

  • Problem

    Finetuning very large language models requires prohibitive GPU memory, while existing quantization methods reduce inference costs but break down during training.

  • Method

    QLoRA backpropagates through a frozen, 4-bit quantized pretrained model into learnable Low Rank Adapters, using memory-saving quantization and optimization techniques.

  • Results

    QLoRA reduced 65B-model finetuning memory from >780GB to <48GB without degrading performance, while the largest Guanaco reached 99.3% of ChatGPT’s Vicuna performance.

  • Takeaways & Limitations

    QLoRA makes finetuning the largest publicly available models feasible on a single GPU while preserving 16-bit full-finetuning performance.

  • Takeaways & Limitations

    The study does not establish that QLoRA matches full 16-bit finetuning performance at the 33B and 65B scales.

Abstract

from arXiv · show

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.

1 Introduction

QLoRA enables performance-preserving finetuning of 4-bit quantized large language models by backpropagating through frozen quantized weights into learnable low-rank adapters. It reduces 65B-model finetuning memory from more than 780 GB to under 48 GB and supports extensive instruction-tuning and chatbot evaluation studies.

  • Motivation: Regular 16-bit finetuning of a LLaMA 65B model requires more than 780 GB of GPU memory, making very large-model finetuning prohibitively expensive.
  • Method: QLoRA finetunes a pretrained model quantized to 4-bit by learning a small set of Low-rank Adapter weights through gradients backpropagated through the quantized weights.
  • Results: QLoRA reduces 65B-model finetuning memory from >780GB to <48GB without degrading runtime or predictive performance relative to a 16-bit fully finetuned baseline.
  • Innovations: QLoRA combines 4-bit NormalFloat, Double Quantization, and Paged Optimizers to reduce memory without sacrificing performance.Double Quantization saves an average of about 0.37 bits per parameter, approximately 3 GB for a 65B model.
  • Evaluation: QLoRA enables training more than 1,000 models spanning instruction datasets, architectures, and sizes from 80M to 65B parameters, including 16-bit performance and Guanaco chatbot analyses.
  • Evaluation: The study evaluates chatbots through tournament-style matches judged by human annotators or GPT-4, aggregates outcomes into Elo scores, and finds that GPT-4 and human evaluations largely agree.

2 Background

The background introduces block-wise low-bit quantization to improve bin utilization in the presence of outliers, alongside LoRA, which trains small adapters while keeping pretrained weights fixed. It also emphasizes that activation gradients, rather than adapter parameters, dominate memory use during parameter-efficient finetuning.

  • Block-wise k-bit Quantization: Quantization discretizes higher-bit representations into lower-bit data types, commonly rescaling inputs by the absolute maximum to use the target range.The quantization constant or scale governs this rescaling, and dequantization reverses it.
  • Block-wise k-bit Quantization: Block-wise quantization partitions tensors into independently quantized blocks, each with its own constant c, reducing inefficient bin usage caused by outliers.Outliers can leave some low-bit quantization bins sparsely or entirely unused when the whole tensor shares one scale.
  • Low-rank Adapters: LoRA reduces memory by training a small set of adapter parameters while keeping the full pretrained model fixed and passing gradients through it.The adapters augment linear projections through an additional factorized projection.
  • Memory Requirement of Parameter-Efficient Finetuning: LoRA’s minimal footprint allows using more adapters to improve performance without significantly increasing total memory.The section frames adapter count and adapter size as key memory considerations in parameter-efficient finetuning.
  • Memory Requirement of Parameter-Efficient Finetuning: 567 MB of LoRA input gradients versus 26 MB of LoRA parameters shows that activation gradients dominate memory for a 7B LLaMA model trained with LoRA.This example uses FLAN v2, batch size 1, and LoRA weights equivalent to 0.2% of the original model weights.

3 QLORA Finetuning

QLoRA enables memory-efficient 4-bit finetuning by combining NF4 quantization, Double Quantization, and Paged Optimizers while dequantizing weights to BFloat16 for computation and updating only LoRA parameters.

  • QLoRA Finetuning: QLoRA stores weights in usually 4-bit precision, dequantizes them to BFloat16, and performs matrix multiplications in 16-bit.The forward and backward passes use the computation type, while only LoRA parameter gradients are computed.
  • 4-bit NormalFloat Quantization: NF4 derives an information-theoretically optimal k-bit datatype for zero-mean normal distributions by equalizing quantization-bin occupancy and explicitly representing zero.It normalizes theoretical normal-distribution quantiles and weight tensors into the [−1, 1] range before quantization.
  • 4-bit NormalFloat Quantization: Approximate quantile estimation can produce large errors for outliers, whereas fixed distributions up to a quantization constant permit exact quantile estimation.Pretrained neural-network weights are treated as zero-centered normally distributed and rescaled to the datatype range.
  • Double Quantization: Double Quantization reduces quantization-constant overhead from 0.5 bits per parameter to 0.127 bits, saving 0.373 bits per parameter.The second quantization uses 8-bit floats with blocksize 256, while the first uses blocksize 64.
  • Paged Optimizers: Paged Optimizers use NVIDIA unified memory to evict optimizer states to CPU RAM when GPU memory is exhausted and page them back when needed.This mechanism is intended to prevent occasional out-of-memory errors during large-model finetuning.

4 QLoRA vs. Standard Finetuning

Across encoder, encoder-decoder, and decoder-only experiments, 4-bit QLoRA with NF4 matches 16-bit LoRA and full-finetuning performance, while NF4 outperforms alternative 4-bit types. Matching full-finetuning with LoRA requires adapters on all transformer-block linear layers, and paged optimizers enable large-model tuning on single GPUs without reducing training speed at batch size 16.

  • Experimental setup: QLoRA was evaluated against 16-bit adapter-finetuning and full-finetuning across three architectures, GLUE, Super-NaturalInstructions, and 5-shot MMLU.The experiments covered models up to 3B parameters in the general setup, with LLaMA evaluations after finetuning on Flan v2 and Alpaca.
  • Memory and runtime: With a batch size of 16, paged optimizers provide the same training speed as regular optimizers for 65B models on 48GB GPUs.Paged optimizers are critical for 33B/65B QLoRA tuning on single 24/48GB GPUs, although hard measurements were not provided because paging is rare for typical mini-batches.
  • LoRA configuration: Using LoRA only on query and value projections fails to match full-finetuning performance for large base models, whereas adapters on all transformer-block linear layers are required.For LLaMA 7B finetuning on Alpaca, the total number of LoRA adapters was the most critical hyperparameter.
  • Quantization data types: NF4 outperforms regular 4-bit floating-point data types on mean zero-shot accuracy across Winogrande, HellaSwag, PiQA, Arc-Easy, and Arc-Challenge.The evaluated model families included OPT, BLOOM, Pythia, and LLaMA at sizes from 125M to 65B.
  • Performance comparison: 4-bit QLoRA with NF4 matches 16-bit full-finetuning and 16-bit LoRA performance on established academic benchmarks.RoBERTa and T5 experiments from 125M to 3B parameters found that 16-bit, 8-bit, and 4-bit adapter methods replicated the fully finetuned 16-bit baseline; LLaMA 7B–65B experiments tested MMLU after Alpaca and Flan v2 finetuning.

5 Pushing the Chatbot State-of-the-art with QLoRA

The study evaluates QLoRA instruction finetuning across language-understanding and chatbot benchmarks, comparing Guanaco with research and commercial systems. Guanaco 65B achieves near-ChatGPT performance, while pairwise Elo evaluation addresses uncertainty in absolute chatbot ratings.

  • Experimental setup: The study compares QLoRA models with Vicuna, Open Assistant, GPT-4, GPT-3.5-turbo, and Bard.Open Assistant is a LLaMA 33B model trained with RLHF on OASST1, while Vicuna fully fine-tunes LLaMA 13B on ShareGPT conversations.
  • Evaluation benchmarks: MMLU reports 5-shot accuracy across 57 language-understanding tasks, while chatbot evaluation uses curated Vicuna and multilingual OASST1 queries.The Vicuna set contains 80 prompts; the OASST1 procedure produces 953 unique queries.
  • Chatbot results: 30%: Guanaco 65B and 33B have this expected win probability against GPT-4, the highest reported at that time from human system-level pairwise comparisons.The result is based on Elo ratings from human annotators and reflects performance competitive with ChatGPT.
  • Chatbot results: 99.3%: Guanaco 65B achieves this performance relative to ChatGPT on Vicuna, becoming the best model after GPT-4.Guanaco 33B improves over Vicuna 13B by three percentage points while using 21 GB versus 26 GB; Guanaco 7B has a 5 GB footprint.
  • Evaluation reliability: GPT-4 and human system-level rankings show moderate agreement, with Kendall τ = 0.43 and Spearman r = 0.55, while example-level agreement is weaker at Fleiss κ = 0.25.The authors recommend Elo rankings because absolute ratings have wide confidence intervals and unclear scale interpretation.
  • Elo evaluation: Elo rankings show Guanaco 33B and 65B outperforming every model except GPT-4 on both Vicuna and OASST1, with performance comparable to ChatGPT.The Vicuna benchmark favors open-source models, whereas the larger OASST1 benchmark favors ChatGPT.

6 Qualitative Analysis

The qualitative analysis uses representative, adversarially elicited examples to contextualize quantitative results, revealing strengths in factual recall, misinformation resistance, and recognizing unanswerable questions alongside weaknesses in obscure factual recall, refusals, secret keeping, and mathematics. It also emphasizes that benchmark, human, and automated evaluations have important limitations, while noting open questions about multilingual training and the absence of RLHF.

  • Qualitative methodology: Representative samples from Vicuna and OpenAssistant data were used to adversarially elicit incorrect response patterns, but the study was not comprehensive because response distributions and other variables were uncontrolled.The authors present these examples as context for earlier quantitative evidence and hope open-sourced models and code will enable future work.
  • Observed behaviors: Guanaco consistently answers common factual questions correctly but becomes confidently unreliable on obscure questions, producing both the wrong popularizer and wrong birthday in a HotPotQA example.The birthday was correct for the incorrectly identified person, Al Jolson.
  • Observed behaviors: Guanaco resists assumed misinformation and recognizes questions requiring real-time information, but it sometimes refuses benign instructions for seemingly random reasons.It rejects the premise that scientists confirmed the earth is flat and explains that it lacks access to the current time, yet refuses to reverse a sentence.
  • Observed behaviors: Secret keeping is initially successful under direct questioning but fails after a tiny amount of trickery, causing Guanaco to reveal the secret word “banana.”The desired direct-response behavior is refusal to disclose or repeat the secret word.
  • Observed behaviors: Mathematics is Guanaco’s biggest weakness: shown work tends to be accurate, but the model can fail on simple problems when it does not reason step-by-step and can produce nonsensical premises.The analysis specifically notes an incorrect claim that 1833 is prime and warns that the model’s inferences are unreliable.
  • Evaluation and training considerations: Fleiss κ = 0.42 among human annotators and Fleiss κ = 0.25 between GPT-4 and humans expose limitations in chatbot evaluation, including subjective preferences, order effects, and misaligned evaluator preferences.The authors also note that GPT-4 assigns higher scores to systems appearing first in its prompt.

7 Related Work

Related work spans inference-time quantization, parameter-efficient adapter methods, instruction finetuning, and dialogue-based chatbot training. QLoRA positions LoRA as achieving full 16-bit finetuning performance without using reinforcement learning.

  • Quantization of Large Language Models: LLM quantization research largely targets inference-time quality, addressing outlier features, grouping methods, regular rounding, or optimized rounding decisions.Examples include SmoothQuant, LLM.int8(), and other grouping and rounding approaches.
  • Finetuning with Adapters: PEFT methods include prompt tuning, embedding or hidden-state tuning, added layers, bias tuning, Fisher-based masks, and combined approaches; QLoRA uses LoRA adapters.The paper reports that LoRA adapters reach full 16-bit finetuning performance.
  • Instruction Finetuning: Instruction finetuning trains pretrained LLMs on input-output pairs so they generate outputs conditioned on prompts, with datasets including MetaICL, FLAN, Alpaca, Vicuna, and others.The related approaches also include InstructGPT, PromptSource, Super-NaturalInstructions, Self-instruct, and OPT-IML.
  • Chatbots: Dialogue-based chatbots often use RLHF or RLAIF, with related datasets and systems including Anthropic-HH, Open Assistant, LaMDA, and Sparrow.QLoRA does not use reinforcement learning; Guanaco is finetuned on multi-turn chat interactions from Open Assistant.

8 Limitations and Discussion

The study supports QLoRA’s ability to reproduce 16-bit finetuning performance with a 4-bit base model and LoRA, but leaves important scale, benchmark, responsible-AI, and design-space questions unresolved.

  • Scale limitations: QLoRA was not shown to match full 16-bit finetuning performance at 33B and 65B scales because of immense resource costs.The authors leave this comparison to future work.
  • Evaluation limitations: Evaluations covered MMLU, Vicuna, and OA, but not BigBench, RAFT, or HELM, so generalization to those benchmarks remains unestablished.The study nevertheless conducts a broad MMLU analysis and develops new chatbot-evaluation methods.
  • Evaluation limitations: Benchmark performance appears to depend on similarity between finetuning data and benchmark data, requiring better evaluation and care about what is being measured.FLAN v2 resembles MMLU, whereas Chip2 resembles chatbot benchmarks, and the models score accordingly on MMLU and Vicuna.
  • Responsible-AI limitations: Guanaco received only a limited responsible-AI evaluation, although Guanaco-65B showed a much lower average social-bias score than other raw pretrained models.The authors suggest that finetuning on OASST1 reduces bias in the LLaMA base model.
  • Design-space limitations: The study did not compare 3-bit base models or alternative adapters, leaving unclear whether other PEFT methods scale to large models or outperform LoRA.LoRA was selected because prior results established its robustness, while other adapters might yield better performance.

9 Broader Impacts

QLoRA broadens access to high-quality LLM finetuning by enabling large-model training on single GPUs and potentially mobile devices. Its broader impact is dual-use: increased access may support independent auditing, but finetuning can also be abused.

  • 9 Broader Impacts: QLoRA enables finetuning 33B models on a single consumer GPU and 65B models on a single professional GPU without degrading performance relative to full finetuning.The best 33B model trained on Open Assistant can rival ChatGPT on the Vicuna benchmark.
  • 9 Broader Impacts: QLoRA could enable finetuning LLMs on phones and other low-resource settings, including 7B models previously shown runnable on phones.With an iPhone 12 Plus, the authors estimate 3 million tokens finetuned per night while charging.
  • 9 Broader Impacts: Finetuning is dual-use: widespread LLM use has known dangers, but broader access could enable more independent analysis than concentrating LLM power in corporations without releasable models or source code.The authors frame equalized access as a way to support auditing of increasingly ubiquitous technology.
  • 9 Broader Impacts: The authors expect QLoRA to broadly improve access to finetuning high-quality LLMs by making it more widely and easily accessible.This conclusion summarizes the method’s anticipated overall societal impact.

A QLoRA vs Standard Finetuning Experimental Setup Details … B.2 Hyperparameters

The experiments tune LoRA configurations, use dataset-specific setups, and evaluate QLoRA across diverse instruction and preference datasets. Core training uses NF4 with double quantization, bf16 computation, and LoRA modules on all linear layers.

  • A.1 Hyperparameters for QLORA: The LoRA search varies dropout, rank, and layer placement while tuning learning rate with fixed LoRA α because α remains proportional to learning rate.The search covers dropout values 0.0, 0.05, and 0.1; ranks 8 through 256; and five layer-placement options.
  • A.1 Hyperparameters for QLORA: LoRA dropout 0.05 helps 7B and 13B models but not 33B and 65B models, while rank is unrelated to final performance when applied to all layers.For LLaMA 7B models finetuned on Alpaca, performance for specific ranks appears independent of other hyperparameters.
  • A.2 Super-Natural Instructions Experimental Setup Details: T5 experiments reuse Wang et al.’s preprocessing and training hyperparameters, add validation-based tuning and early stopping, and use rank 16 for small-to-large models versus 64 for xl and xxl.LoRA α is 64 across these T5 experiments.
  • B.1 Datasets: The dataset suite spans crowd-sourced conversations, human preference data, task mixtures, distilled instruction datasets, hybrid long-form corpora, and code-oriented examples.Examples include OASST1, HH-RLHF, FLAN v2, Self-Instruct, Alpaca, Unnatural Instructions, Longform, and Chip2.
  • B.1 Datasets: OASST1 contributes 9,209 examples after retaining only top replies, while HH-RLHF contributes 160,800 examples using only preferred replies from combined helpfulness and harmlessness data.OASST1 originally contains 161,443 messages across 66,497 conversations and 35 languages; HH-RLHF pairs two assistant replies with human preferences.
  • B.1 Datasets: FLAN v2 contains over 15M examples across 1836 tasks, while Self-Instruct, Alpaca, and Unnatural Instructions contain 82,612, 51,942, and 240,670 examples respectively.The FLAN v2 mixtures follow the authors’ task mixtures except for unavailable datasets.
  • B.2 Hyperparameters: Across QLoRA experiments, hyperparameters are tuned on the MMLU 5-shot development set, using NF4 with double quantization, bf16 computation, rank 64, α 16, and all linear layers.Adam beta2 is 0.999, maximum gradient norm is 0.3, and dropout is 0.1 for models up to 13B and 0.05 thereafter.

B.3 Ablations · B.4 What is more important: instruction finetuning dataset size or dataset quality? · D Pairwise Evaluation with GPT-4

The ablations find that training only on target responses benefits MMLU, while dataset suitability matters more than dataset size. GPT-4 pairwise judgments are well-ordered and transitive when averaged across presentation orders.

  • B.3 Ablations: Only training on target responses benefits MMLU across four instruction-tuning datasets using 52,000 examples and a 7B model.The experiments compare training on responses alone with training on instructions plus responses.
  • B.3 Ablations: The study did not evaluate whether this training choice affects chatbot performance on Vicuna or OA benchmarks.This limits the ablation’s conclusions to MMLU performance.
  • B.4 What is more important: instruction finetuning dataset size or dataset quality?: Dataset suitability is more important than dataset size for instruction finetuning outcomes.The analysis compares subsampled Chip2, FLAN v2, and Unnatural Instructions datasets.
  • B.4 What is more important: instruction finetuning dataset size or dataset quality?: 0.0 - 0.5 MMLU: increasing dataset size and training epochs improves MMLU only marginally.The experiments use dataset sizes of 50,000, 100,000, and 150,000 examples.
  • D Pairwise Evaluation with GPT-4: GPT-4 pairwise results are well-ordered after averaging judgments across both system-presentation options.Evaluation outcomes differed depending on which system appeared first.
  • D Pairwise Evaluation with GPT-4: The aggregated GPT-4 pairwise judgments are transitive: if System A beats B and B beats C, A also beats C.Transitivity yields a complete ordering of the evaluated systems.

E NormalFloat 4-bit data type

This section presents the exact values defining the NF4 data type.

  • E NormalFloat 4-bit data type: The NF4 data type is specified by an explicit list of exact values.

F Normality of Trained Neural Network Weights · G Memory Footprint

Statistical testing finds that most LLaMA weights are normally distributed, but 7.5% of neurons are exceptions. QLoRA’s 33B model requires paged optimizers to fit within 24 GB under the tested training setup.

  • F Normality of Trained Neural Network Weights: Almost all pretrained LLaMA weights appear normally distributed, supporting the characterization of trained neural network weights as mostly normal.The analysis uses Shapiro-Wilk testing on individual hidden units.
  • F Normality of Trained Neural Network Weights: 7.5% of neurons are non-normally distributed under a 5% significance threshold, exceeding the expected false-positive rate by about 2.5%.Weights are tested per hidden unit because different units have different normal distributions.
  • G Memory Footprint: Figure 6 decomposes memory usage across LLaMA models, including adapter and base-model weights plus input gradients, while excluding attention.Input gradients are estimated for batch size 1 and sequence length 512.
  • G Memory Footprint: Paged optimizers provide enough memory headroom for models that otherwise do not quite fit on certain GPUs.This addresses memory spikes during QLoRA training.
  • G Memory Footprint: 33B LLaMA does not quite fit into a 24 GB GPU during QLoRA training, so paged optimizers are needed.The setup uses batch size 1, sequence length 512, and gradient checkpointing.
  • G Memory Footprint: Batch size and sequence length can substantially increase activation-gradient memory beyond the illustrated footprint.The memory estimate assumes batch size 1 and sequence length 512; larger settings may consume considerable additional memory.
Loading 2305.14314v1…