PaperlayerView exact revision on arXiv

Source-linked AI summary

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

Md Romyull Islam

arXiv:2608.23816v1cs.LG

TL;DR

QLoRA reduces memory but incurs a training-speed penalty from repeated 4-bit dequantization. AQLoRA replaces search with one data-free CPU pass for precision allocation and adapter placement, achieving faster training while preserving near-baseline accuracy, with limitations around layer-selection evidence and hardware scope.

  • Problem

    QLoRA makes large-model fine-tuning memory-efficient but remains slower than fp16 LoRA because every 4-bit weight block is dequantized during training.

  • Method

    AQLoRA uses one calibration-free CPU pass to rank layers by NF4 reconstruction error, protect top-K layers in fp16, and place adapters on every layer or only top blocks.

  • Results

    Across six models and four architecture families, AQLoRA matches QLoRA accuracy and comes within a point of fp16 LoRA, while its speed setting is 11.1 ± 2.7% faster than well-tuned QLoRA.

  • Takeaways & Limitations

    AQLoRA provides a cheap, reproducible zero-search way to recover training speed on the stock stack, with speed driven by the number of protected layers and early backward stopping.

  • Takeaways & Limitations

    NF4-error ranking does not outperform budget-matched random protected-layer selections, and reported speedups target the stock stack on pre-Hopper or edge GPUs.

Abstract

from arXiv · show

Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass over the weights sets everything, with no search and no calibration data. The pass ranks layers by NF4 reconstruction error and keeps the top-K in fp16 under a memory budget. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The rule reproduces Unsloth's hand-curated dynamic-4bit selection exactly, in seconds, where search-based allocation needs repeated calibration passes. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.1 +/- 2.7% faster than well-tuned QLoRA and gives up about one accuracy point. It was faster in all nine independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.4% faster. Its accuracy is level with QLoRA on every model and within a point of fp16 LoRA, for 0.2 GiB more memory. These error bars are measured between independent sessions, not within one. Earning them taught us three rules for timing on shared hardware. Fix the measurement duration, not the step count. Measure the noise floor from a duplicated arm, not a nearly identical method. Repeat whole sessions: a floor computed inside one sweep understates the real uncertainty several times over, and the random seed controls almost none of it. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either. The count of protected layers, not their identity, carries the speed effect.

1 Introduction

AQLoRA targets QLoRA’s dequantization speed penalty with a deterministic, data-free allocation and adapter-placement recipe. Across diverse models, it preserves QLoRA-level accuracy while improving training speed on the stock software stack.

  • Motivation: QLoRA saves memory but trains more slowly than fp16 LoRA because it dequantizes every 4-bit weight block during forward and backward passes.NF4 QLoRA has been reported at 0.7 to 0.8 times plain LoRA throughput.
  • Method: AQLoRA uses one CPU pass to rank layers by NF4 reconstruction error, protect the top-K in fp16 under a memory budget, and place adapters for quality or speed.The pass requires no calibration data or forward pass.
  • Results: The quality setting matches QLoRA accuracy across six models and four architecture families, while coming within a point of fp16 LoRA and matching QLoRA on GSM8K.The evaluation covers Commonsense-170K models from 1.4B to 14B parameters.
  • Controls: Controls show that weight-density adapter selection and NF4-error protected-layer selection are no better than random, whereas adapting only top blocks provides the speed effect.The authors attribute the speed effect to the number of protected layers and early backward stopping, not layer identity.
  • Results: 11.1 ± 2.7% faster training is achieved by the speed setting than well-tuned QLoRA, with about one accuracy point given up and nearly unchanged memory.The result is reported across nine timing sessions on five models.
  • Results: 4.8 ± 2.4% faster training is achieved by the quality setting, which remains within a point of fp16 LoRA accuracy.Both AQLoRA settings were faster in every reported session.

2 Related Work

Prior work addresses quantized fine-tuning through improved initialization, information retention, mixed precision, adapter selection, kernels, sparsity, and on-device methods. AQLoRA is positioned as a stock-stack, zero-search approach for per-layer precision and adapter placement, especially on less specialized hardware.

  • Quantized PEFT: Existing quantized PEFT methods improve initialization, mergeability, information retention, or quantizer modularity rather than directly targeting training speed.Several are reported as slower than QLoRA.
  • Mixed precision: Sensitivity-based mixed-precision allocation is established in PTQ and deployed by llama.cpp and Unsloth, but prior practice lacks a stated selection rule or systematic study.QEFT and OWQ protect sensitive columns using custom layouts.
  • Adapter placement: AdapterDrop and later methods reduce computation by selecting layers or parameters through output, spectral, magnitude, probe, or rank statistics.AQLoRA’s top-block placement follows the selective-backpropagation line of work.
  • Fast fine-tuning systems: Kernel-level, FP8, and sparsity systems set higher speed benchmarks but require custom kernels, Hopper-class hardware, or specialized computation.AQLoRA instead targets the stock stack on pre-Hopper and edge devices.
  • On-device fine-tuning: On-device fine-tuning methods generally use uniform base precision or fixed heuristic mixes rather than measuring each model’s per-layer precision needs.AQLoRA differs by deriving precision from a model-specific measurement.

3 Method

AQLoRA profiles NF4 reconstruction error in a data-free CPU pass, protects the worst layers in fp16, and controls speed through adapter placement. Its zero-search precision rule is cheap and reproducible, but the reported control does not show that its selected layer identities outperform random choices.

  • 3.1 Profiling pass: The profiling pass evaluates each linear layer’s weights in 64-element blocks, matching bitsandbytes’ NF4 layout.The supplied equation defines the per-layer reconstruction-error calculation.
  • 3.1 Profiling pass: The CPU pass runs on the fp16 checkpoint without data or a forward pass, taking about 1.7 seconds at 0.5B and 2.5 seconds at 1.4B on warm checkpoints.Cold disk loading adds a few seconds, while calibration-based allocators require repeated forward passes.
  • 3.2 Precision allocation: On phi-1.5, NF4 errors span about 12× across 144 linear layers, with the top fifth containing roughly 29% of total error.This motivates allocating a small fp16 budget to the highest-error layers.
  • 3.2 Precision allocation: A fixed top-K cut loads the layers with largest NF4 error in fp16 and keeps the rest in NF4, with K determined by the protection budget.Protected fp16 layers skip the per-step dequantization that slows QLoRA.
  • 3.2 Precision allocation: The NF4-error ranking does not outperform budget-matched random selections, so the evidence supports choosing the protection count rather than the selected identities.The rule is retained for determinism, data freedom, low cost, and reproduction of a deployed hand-curated selection.
  • 3.3 Adapter placement: Adapter placement uses all layers for quality or top blocks for speed, allowing autograd and checkpoint recomputation to stop below the cut block.The adapter fraction α therefore acts as a speed-quality knob.
  • 3.3 Adapter placement: AQLoRA-q uses ρ=.2 and α=1.0, whereas AQLoRA-s uses ρ=.2, α=.85, and early backward stopping.The settings are two points on one budget knob, not separate methods.

4 Experimental setup

The experiments span six models and four architecture families, using a controlled timing protocol with repeated sessions to separate method effects from shared-hardware noise. The setup also tests how architecture-dependent layer layouts affect memory costs.

  • Models and benchmark: Commonsense-170K is evaluated on six models across four architecture families spanning 1.4B to 14B parameters.The benchmark uses eight held-out task test sets and greedy generation.
  • Architecture-dependent budgets: A ρ=.20 layer budget protects different parameter fractions across architectures, making equal-ρ comparisons unequal in memory.The protected fraction is 4.3% on Llama-3.2-1B versus 16.3% on Phi-3-mini.
  • Timing protocol: Each configuration is run seven times in an interleaved session on one GPU, with speed estimated from the fastest half of samples.The protocol excludes the interference-slowed tail rather than using a plain mean.
  • Timing protocol: 1.77% to 0.10% within-session floors suggest roughly 2% measurement noise, but duplicated-arm sessions show that this floor understates between-session uncertainty.The duplicated QLoRA arm removes dependence on a nearly identical proxy method.
  • Timing protocol: AQLoRA-s averages 11.1% faster than QLoRA across nine clean sessions, while AQLoRA-q averages 4.8% faster.AQLoRA-s has a between-session standard deviation of 2.7%; AQLoRA-q's is 2.4%.
  • Timing protocol: Single-session efficiency results can look several times more precise than justified because clean floors are necessary but not sufficient.Across clean Phi-3-mini sessions, measured AQLoRA-s speedups span 7.0 to 15.2%.

5 Main results

Across the evaluated models and tasks, AQLoRA-q largely preserves QLoRA and fp16-LoRA accuracy while AQLoRA-s trades some accuracy for speed. At larger scale, quantization becomes necessary because fp16 weights exceed commodity-GPU memory.

  • Accuracy: Repeated phi-1.5 runs put AQLoRA-q, QLoRA, and LoftQ at 74.8, matching fp16 LoRA and supporting parity rather than a win.The gaps among the first four methods are smaller than the seed spread of any one method.
  • Accuracy-speed trade-off: AQLoRA-s finishes about 1.5 points below QLoRA on phi-1.5, about 1.0 on Llama-3.2-3B, and about 0.4 on Phi-3-mini.The speed setting is therefore a speed-quality dial rather than a uniform accuracy improvement.
  • Scale: At 7B, AQLoRA-q matches fp16 LoRA at 89.0 accuracy while using 10.6 GiB versus 19.1 GiB.At 14B, fp16 LoRA requires 27.5 GiB for weights alone, beyond a 24 GB card.
  • Scale: At 14B, QLoRA reaches 91.1 at 12.4 GiB, while AQLoRA-s reaches 91.3 at 14.2 GiB.The reported AQLoRA-s wall-clock comparison is indicative because 14B was not timed under the clean protocol.
  • Accuracy: On GSM8K, the three methods land within 0.5 points while quantized runs use about half the memory of fp16.This extends the reported parity pattern beyond Commonsense-170K.

6 Where the speed comes from

AQLoRA’s speed comes from combining early backward stopping with protected fp16 layers, measured across independent timing sessions rather than a single sweep.

  • Early backward stopping accounts for about 6% of the speed gain by preventing computation below the adapter-placement cutoff.With gradient checkpointing off, the gap over QLoRA grows to 42% at 23% less memory.
  • Length-grouped batching cuts padding waste from 39.8% to 3.7%, making it the largest shared speed lever applied to all baselines.AQLoRA’s own components add speed on top of this common optimization.
  • The timing protocol uses independent sessions and excludes sessions failing the duplicate-arm floor test.Single-session rows have no deviation to report.

7 Ablations

The ablations show that speed depends mainly on the number of protected or adapted layers, while the proposed NF4-error ranking does not reliably improve accuracy and its budget is model- and composition-sensitive.

  • Adapter placement: Random adapter selection beats density ranking by 1.1 points at α=.5 and 0.4 points at α=.7, while density ranks last in all four cells.The comparison used two budgets and two seeds.
  • Adapter placement: Reducing the adapter budget from every layer to α=.7 costs about 0.8 points and cuts training time, whereas top-block placement costs more accuracy but enables an explicit backward-pass cut.The shipped speed setting uses α=.85.
  • Protection budget: At ρ=.05, the protection budget adds 0.06 GiB over QLoRA and retains a 4.1% step-time advantage against a 2.8% noise floor.The three timed budgets span only 1.1%, so no speed trend among them is claimed.
  • Scope and robustness: The cheap ρ=.05 budget is not a universal default: it underperforms ρ=.20 on Phi-3-mini and remains unresolved under rsLoRA.The authors retain ρ=.20 in the main tables and recommend sweeping the budget per model.
  • Precision selection: Eight of ten random selections beat NF4-error top-K at fixed K=29, while downstream accuracy was indistinguishable across top-K, random, and bottom-K selections.The study therefore finds no evidence that NF4 reconstruction error selects better layers than chance.
  • Additional controls: A second NF4 stage over protected-layer residuals also works but is not worth using, and the formalized rule matches Unsloth’s hand-curated selection at 89.0.The Unsloth comparison used an otherwise identical Qwen2.5-7B run.
  • Reproducibility: Repeated Qwen2.5-7B runs show AQLoRA-q at 89.08 ± 0.15 versus QLoRA at 88.80 ± 0.09, while repeated phi-1.5 runs tie at 74.78 versus 74.75.These repetitions illustrate that small single-seed gaps may not persist.
  • Reproducibility: Seed does not control the reported variability: same-seed AQLoRA-q runs spread as widely as different-seed runs because training lacks deterministic kernels.The reported standard deviations are 0.24 and 0.25, respectively.

8 Limitations

The main limitation is that NF4-error ranking did not outperform budget-matched random layer selections in the tested setting. The evidence supports protecting a chosen number of layers for speed, but not the ranking criterion’s quality advantage, and large-model hardware introduces an additional precision concern.

  • Ranking limitation: NF4-error ranking did not outperform budget-matched random selections in the tested model and budget.The authors tested ten random draws at one model and budget, which withdraws the claim without ruling it out more broadly.
  • Ranking limitation: The speed effect depends on the count of fp16-protected layers, not their identity.The authors therefore retain the rule as a deterministic, data-free way to fix protection count, not because sensitivity ranking is validated.
  • Hardware boundary: On 14B models, protected layers can overflow fp16 backward on pre-bf16 GPUs and silently collapse the AMP loss scaler.Computing protected layers in fp32 at unchanged storage mitigates the issue; it does not arise on bf16 hardware.
  • Hardware boundary: AQLoRA is not expected to win at matched memory because fp16 protection may require shrinking the batch or sequence.

9 Conclusion

AQLoRA combines zero-search precision selection with adaptive adapter placement for quantized LoRA fine-tuning. Across the evaluated models, it preserves QLoRA-level accuracy while reducing training time, with the effect attributed to the number of protected layers rather than their identity.

  • 9 Conclusion: AQLoRA sets per-layer precision and adapter placement from one CPU pass over the weights with no search.The pass formalizes a zero-search allocation rule and ties precision protection to placement choices.
  • 9 Conclusion: AQLoRA’s accuracy matches QLoRA’s and trails fp16 LoRA by less than a point across six models and four architecture families.The same conclusion is reported for the evaluated benchmark scope, including the stated cross-model comparison.
  • 9 Conclusion: The count of protected layers, not their identity, carries the measured speed effect.The conclusion also reports that protected-layer identity was not supported as the source of the quality effect.
  • 9 Conclusion: AQLoRA lowers the time and hardware needed for language-model fine-tuning, potentially broadening access beyond current datacenter hardware.The passage also notes a corresponding dual-use concern because cheaper fine-tuning can reduce the cost of harmful adaptation.

A On the shared levers and the ranking

Native-recipe controls show that shared settings slightly reduce accuracy across methods but largely preserve the ordering of QLoRA, LoftQ, and AQLoRA-q. The section also documents method-specific exceptions and compares allocation and timing evidence.

  • Shared levers: Every method scores higher under its native recipe, showing that the shared levers cost some accuracy across the board.
  • Shared levers: +0.27 to +0.40 points was the native-recipe shift for QLoRA, LoftQ, and AQLoRA-q, with their order unchanged.QDoRA instead gains +0.89 and moves from last under shared levers to second under its native recipe.
  • Evaluation scope: The tables report 8-task Commonsense-170K accuracy for Llama-3.2-3B and Phi-3-mini, and GSM8K accuracy for Qwen2.5-7B.These table captions identify the evaluated metrics and model groupings; they do not by themselves state a comparative outcome.
  • Comparison scope: Peak memory is comparable within a model but not between models because vocabulary sizes differ by 4×.
  • Ranking: Figure 3 sorts per-layer NF4 error on phi-1.5 and marks layers kept in fp16.

C Rank sweep

The rank sweep finds that AQLoRA-q remains near QLoRA across tested ranks on phi-1.5, while rank effects and scaling failures can dominate method differences. Conclusions beyond phi-1.5 are limited by model-specific and learning-rate-dependent behavior.

  • Rank sweep on phi-1.5: +0.26 at r=16, +0.03 at r=8, and +0.01 at r=32 are AQLoRA-q’s reported differences over QLoRA on phi-1.5.AQLoRA-q is at or above QLoRA at all four tested ranks, though never by much.
  • Rank sweep on phi-1.5: 0.22 GiB is AQLoRA-q’s constant memory premium over QLoRA across ranks.Because the protected set is fixed while adapters grow with rank, the premium falls from 5.7% at r=8 to 5.3% at r=32.
  • Rank sweep on phi-1.5: About 1.8 points separates r=4 from r=32 for the working methods, exceeding any fixed-rank gap among them.
  • Rank sweep on Llama-3.2-1B: AQLoRA-q, QLoRA, and LoftQ remain within 0.5 points at r=16 on Llama-3.2-1B, with AQLoRA-q 0.33 below QLoRA.This result is one seed per cell and is scoped to the reported configuration.
  • Rank sweep on Llama-3.2-1B: At r=32 on Llama-3.2-1B, QLoRA and AQLoRA-q fall to chance while LoftQ remains at 63.2 under the shared learning rate.The authors attribute the collapse to the baseline’s learning-rate configuration and treat the r=32 result as unusable for rank conclusions.
  • rsLoRA control: 37.47 is rsLoRA’s r=32 score after collapsing from 75.16 at r=8 under the shared learning rate.The reported cause is rsLoRA scaling: the update grows with rank and overshoots at the largest tested value.
  • Evaluation convention: The r=8 column is a three-seed mean, while the other ranks are single runs.This convention affects how the rank comparisons should be interpreted.

D The rsLoRA composition

The rsLoRA composition does not support a general accuracy win: its observed margin is sensitive to session count, model, and rank. The section also situates AQLoRA’s broader trade-offs in feasibility, timing, and protection-budget experiments.

  • The rsLoRA composition: At six seeds, AQLoRA+rs averages 75.49 ± 0.31 versus 75.16 ± 0.37 for the rsLoRA baseline, with p=.14 and overlapping ranges.The earlier three-seed result had p=.005, but the larger run widened the estimated spread.
  • The rsLoRA composition: The authors do not claim an rsLoRA win because its positive margin is comparable to arm spread and disappears on Llama-3.2-1B and at r=32 on phi-1.5.They recommend treating +rs as a composition to try with a tuned learning rate, not as a property of AQLoRA.
  • Edge feasibility: The edge-profile section projects a phone-sized phi-1.5 run at roughly 1.3 h per 1000-example epoch by scaling V100 throughput with DRAM bandwidth.The section labels this a feasibility estimate: its device figures are bandwidth-scaled projections from V100 measurements.
  • Timing methodology: Fixed-duration timing is preferable to fixed step counts because equal jitter occupies a larger fraction of shorter measurements.A 60-step window took 164 s on Llama-3.2-1B but 297 s on Qwen2.5-7B; a noisy 1006 s sweep was discarded after interference widened its floor to 5.15%.
  • Protection budget: A larger protection budget improves base-model perplexity under NF4 and fp4, but that improvement does not carry to downstream task accuracy.A residual NF4 protection scheme also failed to justify its added complexity: it matched plain fp16 protection while costing 3.6% step time.

L Full lever data

The full lever data uses sequential configurations to attribute throughput changes to individual components, while distinguishing the attribution setup from the shipped speed setting. The measurements also note run-to-run noise when interpreting individual lever steps.

  • Full lever data: Sixteen phi-1.5 configurations—fourteen lever rungs and two edge profiles—provide the measured source data for the full lever analysis.All configurations use 220 steps on one V100 with the node’s other GPUs held idle.
  • Full lever data: The lever ladder is a single sequential pass designed to attribute speedup to its components, with measured comparisons reported separately.The comparisons are identified as Tables 3 and 4.
  • Full lever data: The attribution ladder uses α=.7, whereas the shipped speed setting uses α=.85; rsLoRA scaling is included but does not change step time.The scalar affects the update rather than the measured step duration.
  • Full lever data: One lever row was slower than the preceding row within single-pass run-to-run noise, so that local reversal is not treated as a robust effect.The figure tracks throughput as shared and AQLoRA-specific levers are added, alongside an fp16 LoRA reference.
Loading 2608.23816v1…