Bayesian Low-rank Adaptation for Large Language Models

1 INTRODUCTION

• Fine-tuned LLMs often exhibit overconfidence, a concern for safety-critical and limited-data decisions such as medical diagnosis, finance, and experimental design.The introduction argues for calibration strategies that make predictions more trustworthy.Source: §1 · p. 1
• Bayesian language-model research has mostly focused on pretraining, where its advantages are less clear and scaling costs may impede adoption.The authors contrast this with fine-tuning, where calibration problems and smaller datasets make Bayesian methods more pertinent.Source: §1 · p. 1
• Bayesian fine-tuning may address poor calibration and limited-data uncertainty while adding less computational burden than Bayesian pretraining.The authors note that fine-tuning typically uses less data than pretraining and has lower computational cost.Source: §1 · p. 1
• Laplace-LoRA applies a post-hoc Laplace approximation to only LoRA parameters, preserving standard efficient pretraining and fine-tuning pipelines.The authors report improved calibration on LLaMA2-7B across six common-sense reasoning tasks and out-of-distribution settings.Source: §1 · p. 2

2 RELATED WORK

• Earlier Bayesian language-model work usually addressed pretraining, while this paper targets fine-tuning, where the authors see clearer calibration benefits.The cited motivation is that even large-scale instruction fine-tuning can yield poor calibration.Source: §2 · p. 2
• Unlike Fan et al. (2020) and Zhang et al. (2021), Laplace-LoRA places its posterior over parameters and keeps pretraining and fine-tuning unchanged.The authors contrast roughly 6 million LoRA parameters with about one billion attention weights in their illustrative setting.Source: §2 · p. 2
• Prior Laplace work on language models used DistillBERT as a feature extractor and applied inference only to its final linear layer; this work uses LoRA across all layers.The authors describe their models as 100 times larger than that prior application.Source: §2 · p. 2
• Fine-tuning regularizers such as Mixout, Mixup, and KL regularization alter weight training, whereas Laplace-LoRA estimates uncertainty around the chosen weights.The authors describe the approaches as orthogonal and say Laplace-LoRA can use weights from standard fine-tuning or these methods.Source: §2 · pp. 2–3

3.1 LOW-RANK ADAPTATION (LORA)

• LoRA keeps the pretrained weight matrix fixed and represents its perturbation as the product of two low-rank matrices.With rank n_lr much smaller than the matrix dimensions, this requires n_lr(n_in + n_out) parameters rather than n_in n_out.Source: §3.1 · p. 3

3.2 LAPLACE APPROXIMATIONS

• Laplace approximation finds a MAP solution, expands the log-joint to second order, and yields a Gaussian posterior whose covariance is the inverse Hessian.The paper uses Fisher information to estimate curvature while ensuring a positive-definite covariance.Source: §3.2 · pp. 3–4
• The full Hessian or Fisher matrix is intractable at LLM scale, so the paper considers last-layer or Kronecker-factored structure for linear-layer blocks.KFAC approximates the Fisher using blocks for each linear layer.Source: §3.2 · p. 4
• Post-hoc Laplace does not require a separate validation set, an advantage when training data is scarce.The paper describes optimizing prior precision using the Laplace marginal likelihood on the training dataset.Source: §3.2 · p. 4
• The method predicts with a linearized model and computes the Bayesian model average by averaging softmax probabilities across Gaussian-noise samples.The authors report that generalized probit and Laplace bridge approximations perform considerably worse than naive Monte Carlo sampling, potentially because they ignore covariance terms.Source: §3.2 · p. 4

4 METHODS

• Laplace-LoRA treats each LoRA adapter BA as two linear layers with weights A and B, then applies post-hoc Kronecker-factored Laplace approximations.The large d × d Kronecker factor would erase LoRA's memory advantage if represented explicitly, so the method uses a rank-n_kfac approximation.Source: §4 · p. 5

5 RESULTS

• The study fine-tunes LLaMA2-7B on common-sense reasoning tasks with LoRA on queries, values, and the output layer for 10,000 iterations, then applies full or output-layer-only Laplace-LoRA at checkpoints.It assesses NLL and ECE, focusing primarily on datasets with fewer than 10,000 training examples.Source: §5 · p. 5

5.1 IN-DISTRIBUTION FINE-TUNING AND EVALUATION

• At 5,000 steps, LA achieves the best NLL across all six datasets and considerable ECE gains over MAP while maintaining similar ACC.Figure 1 also shows calibration benefits across training checkpoints, including at the checkpoint with optimal MAP performance.Source: §5.1 · p. 6, Figure 1 · p. 6, Table 1 · p. 7
• With a validation split, LA gives slightly improved NLL and usually improved ECE compared with temperature scaling at the best MAP validation-accuracy checkpoint.The split uses 80% of training data for training and 20% for validation, which usually slightly decreases performance relative to training on all data.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 EVALUATIONS UNDER DISTRIBUTION SHIFT

• Under distribution shift, LA substantially improves ECE and NLL over other methods while maintaining similar ACC, with a slight ACC improvement in computer science.Across out-of-distribution scenarios, both LLLA and LA typically improve ECE and NLL.Source: §5.2 · pp. 7–8, Table 3 · p. 8
• Across RoBERTa, LLaMA, and Mistral tasks, full LA consistently improves uncertainty estimation; diagonal Fisher approximations do not consistently improve ECE and NLL.The authors identify KFAC's modeling of weight correlations as important to Laplace-LoRA's calibration results.Source: §5.2 · pp. 8–9

5.3 MEMORY AND RUNTIME COST

• Laplace-LoRA adds around 1–5% memory overhead and about 10% measured time overhead when low-rank Kronecker factors are accumulated ten times.The authors estimate that accumulating the factors once in practice would bring time overhead to around 1%.Source: Table 4 · p. 9, §5.3 · p. 9

6 CONCLUSION

• Laplace-LoRA focuses on LoRA weights for scalability and applies Laplace post-hoc, requiring no changes to standard fine-tuning.Experiments report gains in expected calibration error and negative log-likelihood.Source: §6 · p. 9

A.1 MONTE CARLO SAMPLING

• Monte Carlo sampling averages class probabilities after passing sampled logits through softmax, with Gaussian noise generated from IID standard-normal variables.The logits’ Gaussian covariance is decomposed using Cholesky factorization.Source: §A.1 · p. 15

A.2 PROBIT APPROXIMATION

• Generalized probit approximates the classification predictive posterior by integrating over the posterior on weights.Its decision-boundary guarantee for binary sigmoid classification does not extend to softmax multiclass classification, as stated in the passage.Source: §A.2 · p. 15

A.3 LAPLACE BRIDGE

• The Laplace bridge maps a Gaussian distribution over logits to a Dirichlet distribution over classes.It uses only the diagonal of the covariance, ignoring covariance terms.Source: §A.3 · p. 15

A.4 EMPIRICAL COMPARISON

• MC joint consistently achieves the highest accuracy and is among the best for NLL, while bridge is often worst.Probit and MC indep can also be suboptimal; the passage attributes the pattern as likely due to their omission of covariance between logits.Source: §A.4 · p. 15, Figure 2 · p. 16

B PROMPT TEMPLATES FOR LLAMA2 COMMON SENSE REASONING TASKS

• For consistency, ARC-C and ARC-E examples with three or five choices were removed from training and evaluation.The passage reports small counts of such examples for both datasets.Source: B PROMPT TEMPLATES FOR LLAMA2 COMMON… · p. 16

C HYPERPARAMETERS

• The hyperparameter passage points to Table 6 for RoBERTa-base and RoBERTa-large LoRA settings and Table 7 for Llama-7B settings.It says the two settings differ in batch size and maximum sequence length.Source: C HYPERPARAMETERS · p. 16, Table 6 · p. 17, Table 7 · p. 17

D METRICS FOR UNCERTAINTY QUANTIFICATION

• NLL reflects the negative log probability assigned to true labels, while ECE measures alignment between confidence and accuracy across bins.ECE cannot be optimized directly like NLL; a completely random model can achieve zero ECE.Source: D METRICS FOR UNCERTAINTY QUANTIFICATION · p. 17

E LOW-RANK COMPUTATIONS

• For LoRA adapters, the smaller covariance factor is represented as full-rank and the larger factor as low-rank.The resulting precision has a diagonal term plus a low-rank term, which the method uses for computation.Source: E LOW-RANK COMPUTATIONS · p. 18

E.1 INCREMENTAL COMPUTATION OF LOW-RANK FACTOR

• An incremental low-rank SVD estimates the factor without computing the large d × d matrix, retaining the top nkfac components as new activities or gradients arrive.The update combines the current estimate with new vectors, then truncates its SVD.Source: §E.1 · p. 18

E.2 MARGINAL LIKELIHOOD OPTIMIZATION WITH THE MATRIX DETERMINANT LEMMA

• The matrix determinant lemma reduces the posterior log-determinant computation to a smaller matrix; the stated setting has nlorankfac = 40.The appendix also describes optimizing prior precision using training-set model evidence.Source: §E.2 · pp. 18–19

E.3 LINEARIZED PREDICTION WITH WOODBURY

• Linearized prediction uses Woodbury to compute predictive covariance from output gradients, which are treated as matrices across LoRA parameters in a layer.The passage defines each gi as the gradient of a network output for a particular parameter.Source: §E.3 · p. 19

F OPTIMIZING LAPLACE PRIOR PRECISION

• Prior precision is optimized with model evidence when only training data are available, or validation log-likelihood when a validation set is introduced.For validation-set optimization, the mean and Jacobian are precomputed before mini-batch gradient descent on λ.Source: F OPTIMIZING LAPLACE PRIOR PRECISION · p. 19

G.1 LLAMA2-7B EVALUATIONS UNDER DISTRIBUTION SHIFT

• Algorithm 3 tunes Laplace prior precision by maximizing mini-batch validation log-likelihood; Table 8 compares methods on six OOD datasets using LlaMA2-7B fine-tuned on OBQA.Source: §G.1 · p. 20, Table 8 · p. 20

G.2 FINE-TUNING ROBERTA FOR TEXT CLASSIFICATION

• Full Laplace-LoRA delivers substantial gains across tested RoBERTa and LlaMA2 models and is usually the most robust method, including with validation-set tuning.The reported comparisons cover RoBERTa-base and RoBERTa-large with and without a validation split.Source: §G.2 · pp. 20–21, Table 9 · p. 21, Table 10 · p. 22

G.3 FINE-TUNING LLAMA2-7B FOR COMMON SENSE REASONING

• Splitting the training data into 80% training and 20% validation slightly affects LlaMA2-7B accuracy; LLLA also misses some zero-checkpoint evaluations because of Cholesky errors.The split is used to tune temperature scaling and Laplace prior precision on held-out validation data.Source: §G.3 · p. 21

G.4 FINE-TUNING MISTRAL-7B FOR COMMON SENSE REASONING

• For Mistral-7B, Laplace-LoRA achieves the best NLL on all six datasets and the best ECE on three.The authors describe this as evidence of robustness across decoder-only backbones.Source: §G.4 · p. 22, Table 13 · p. 25

G.5 LAPLACE-LORA ON DIFFERENT SUBSETS OF LAYERS

• There is no conclusive layer-count trend, although applying Laplace approximation to more than eight layers usually improves NLL.The compared settings include the top 8, 16, and 24 layers and all layers.Source: §G.5 · pp. 22–23, Table 14 · p. 26

G.6 DIAGONAL LAPLACE APPROXIMATION

• The diagonal Hessian approximation is generally less effective than K-FAC, which approximates the block-diagonal Hessian.Source: §G.6 · p. 23

G.6.1 ROBERTA

• On RoBERTa-base and RoBERTa-large, diagonal LLLA gives tiny ECE and NLL advantages over MAP, while diagonal LA has mixed performance.Source: §G.6.1 · p. 24

G.6.2 LLAMA2-7B

• On LlaMA2-7B, diagonal LA and LLLA lack consistent gains; diagonal LA is much worse than MAP in ARC accuracy and has worse ECE on ARC-E, OBQA, and BoolQ.Validation-set tuning of prior precision using validation log-likelihood yields much better results.Source: §G.6.2 · p. 25, Table 14 · p. 26

• With model evidence, nearly all full-LA logit uncertainty comes from lower layers, leaving LLLA with far less uncertainty and helping explain its poor performance.LLLA captures only the last layer, whereas full LA also captures earlier layers.Source: H LA VS LLLA, OR WHICH LAYERS ARE… · p. 26
• Validation-log-likelihood optimization gives LLLA much higher logit variance and reasonable performance, but its posterior differs sharply from the expected Bayesian allocation of uncertainty.The expected allocation has much less uncertainty in last-layer weights and most logit uncertainty arising from lower layers.Source: H LA VS LLLA, OR WHICH LAYERS ARE… · pp. 26–27

• The distribution-shift evaluations use specified MMLU datasets, with each subject’s task splits selected from those assigned by Hendrycks et al. (2020).Source: I EVALUATION DATASETS UNDER LARGER… · p. 27

• The authors released an original implementation based on Laplace Redux and ASDL, plus a newer standalone implementation intended for ongoing support.Source: J CODE IMPLEMENTATIONS · p. 28