PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-Tuning

M. Berk Sahin, Ege C. Kaya, Arda Fazla, Behzad Sharif, Abolfazl Hashemi

arXiv:2610.12865v1cs.LG

TL;DR

Non-log-concave SG-LMC guarantees typically assume global smoothness and uniform noise variance, conditions that can fail even in elementary models. The paper proposes robust and adaptive methods with non-asymptotic FI guarantees; RA-LMC matches or outperforms leading baselines in LLM and VLM accuracy and uncertainty quantification.

  • Problem

    Non-log-concave SG-LMC guarantees typically rely on globally smooth gradients and uniformly bounded noise variance, assumptions that can fail even in elementary models.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes a state-dependent finite-p-moment oracle and robust, adaptive LMC methods that combine robust gradient estimation with adaptive steps for the two relaxations.

    Source: §3.1 · p. 4, §3.2 · p. 5

  • Results

    RA-LMC matches or outperforms leading baselines in accuracy and uncertainty quantification on LLM and VLM experiments, while synthetic experiments verify FI convergence of all three methods.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Bayesian LoRA fine-tuning experiments provide empirical evidence supporting the BG-p oracle model.

    Source: §5 · p. 8

    The oracle assumes a finite p-th moment for 1<p≤2; when p<2, it permits but does not require infinite variance.

    Source: §2 · p. 4

Abstract

from arXiv · show

Existing convergence guarantees for non-log-concave sampling with stochastic gradient Langevin Monte Carlo (SG-LMC) typically assume globally Lipschitz gradients and uniformly bounded noise variance. Both assumptions can fail even in elementary sampling problems, and relaxing them better reflects practical applications such as Bayesian fine-tuning of large language models (LLMs). We relax both assumptions through generalized smoothness and our proposed state-dependent heavy-tailed oracle with finite $p$-th moments, $1<p\leq 2$. The oracle permits infinite variance when $p<2$ and includes the Blum-Gladyshev (BG-0) condition. We study these relaxations separately and jointly, proposing three methods. Under globally Lipschitz gradients, robust LMC (R-LMC) handles the proposed oracle through geometric-median centering and residual clipping. With exact gradients, adaptive LMC (A-LMC) controls discretization error under generalized smoothness. Robust adaptive LMC (RA-LMC) addresses both relaxations through robust estimation and a noise-aware step size. For all three methods, we establish non-asymptotic convergence guarantees in Fisher information (FI) without assuming dissipativity or a functional inequality. Synthetic experiments demonstrate FI convergence and the benefit of each mechanism. Bayesian fine-tuning experiments with LLMs and vision-language models support our oracle model and show that RA-LMC matches or outperforms state-of-the-art baselines in accuracy and uncertainty quantification.

1 INTRODUCTION

The introduction motivates relaxing global smoothness and uniform noise assumptions for non-log-concave sampling, then presents three methods and their theoretical and empirical contributions.

1 INTRODUCTION

  • Existing non-log-concave guarantees often require functional inequalities, dissipativity or tail-growth conditions, or explicit mixture structure.Relative FI measures stationarity but can be small even when a law assigns incorrect weights to well-separated modes.Source: §1 · p. 1
  • Global smoothness excludes superquadratic potentials, while minibatching in Bayesian linear regression can produce noise variance that grows quadratically with distance from the posterior mode.The latter state-dependent variance is ensured by the Blum–Gladyshev (BG-0) condition.Source: §1 · p. 2
  • Bayesian LLM fine-tuning raises whether LoRA training exhibits heavy-tailed gradient noise and how LMC can retain non-asymptotic FI guarantees under that noise and nonsmooth potentials.Experiments provide evidence of heavy-tailed noise and state-dependent growth consistent with BG-0.Source: §1 · p. 2

1.1 Contributions

  • R-LMC, A-LMC, and RA-LMC provide FI guarantees for heavy-tailed noise, generalized smoothness, and their combination, respectively.RA-LMC controls the dependence between gradient-estimation errors and adaptive step sizes.Source: §1.1 · pp. 2–3
  • RA-LMC matches or improves upon leading baselines in predictive accuracy and uncertainty quantification on LLM and VLM experiments.Synthetic experiments verify FI convergence of all three methods and demonstrate benefits from each mechanism and their combination.Source: §1.1 · p. 3

2 PRELIMINARIES

The preliminaries set the target and baseline LMC assumptions, then introduce gradient-dependent smoothness and state-dependent, finite-moment noise models.

2 PRELIMINARIES

  • The target is π(dx)=Z^-1e^-V(x)dx, and complexity bounds assume initial KL divergence K0=O(d), retaining K0 in finite-time bounds and parameter choices.The text distinguishes the deterministic reference point xref from the random initial state X0.Source: §2 · p. 3
  • Under global L-smoothness, O(L^2d^2/ε^2) iterations suffice for FI(µ̄T∥π)≤ε; the stated SG-LMC oracle complexity is O(L^2d^2σ^2/ε^3).The SG-LMC bound assumes unbiased gradients with uniformly bounded second moment, and averages b independent oracle evaluations per update.Source: §2 · p. 3
  • Symmetric generalized smoothness lets local curvature grow with gradient magnitude and covers superquadratic potentials, polynomial double-well models, and generalized Gaussian distributions.Its segment-based gradient supremum treats the two endpoints symmetrically and reduces to global L0-smoothness when L1=0.Source: §2 · p. 3
  • Prior reports of state-dependent variance and heavy-tailed gradient noise motivate relaxing both uniform variance and tail assumptions.The cited examples include Bayesian linear regression, BERT, CLIP, ALBERT, and RoBERTa.Source: §2 · pp. 3–4
  • The BG-p moment model combines state-dependent noise growth with a finite p-th moment bound, allowing infinite variance when 1<p<2.At p=2 it recovers BG-0; setting B=0 recovers a state-independent finite-p-moment bound.Source: §2 · p. 4

3 METHODS

The methods address state-dependent heavy-tailed noise and generalized smoothness separately, then combine robust estimation with adaptive discretization in RA-LMC. Each method has a non-asymptotic Fisher-information guarantee under its stated conditions.

3 METHODS

  • The methods share an Euler interpolation with a force and step size selected before each Brownian increment, and establish non-asymptotic FI guarantees for their respective settings.The framework develops robust estimation under global smoothness, adaptive steps with exact gradients, and then their joint treatment of stochastic estimation error and local curvature.Source: §3 · p. 4

3.1 Robust LMC under State-Dependent Heavy-Tailed Noise

  • R-LMC uses geometric-median centering and residual clipping to control estimation error under BG-p noise, including state-dependent noise and potentially infinite variance.For K0 = O(d), it achieves FI(µ̄T ∥ π) ≤ ε with NT = O(L2d2/ε2); the BG-p oracle’s state-dependent noise scale determines the leading high-accuracy oracle rate when B > 0.Source: §3.1 · pp. 4–5

3.2 Adaptive LMC under Generalized Smoothness

  • A-LMC adapts step sizes to gradient magnitude, using shorter steps where generalized smoothness permits more rapid gradient change.With exact gradients it has FI guarantees under Assumption 2.3; when L1 = 0, its complexity recovers standard LMC under global L0-smoothness.Source: §3.2 · p. 5

3.3 Robust Adaptive LMC (RA-LMC)

  • RA-LMC combines robust gradient estimation with gradient- and noise-aware step sizes to guarantee non-asymptotic Fisher-information convergence under generalized smoothness and BG-p noise.The noise-aware step controls the coupling between estimation error and the duration over which it acts; when L1 = κ = 0, RA-LMC reduces to R-LMC, and exact gradients recover A-LMC.Source: §3.3 · p. 6

4 EXPERIMENTS

Synthetic experiments test the three methods under heavy-tailed noise, generalized smoothness, and both challenges together; Bayesian fine-tuning experiments examine oracle behavior and compare RA-LMC with baselines. The experiments support the robust-estimation and adaptive-step mechanisms and show favorable RA-LMC baseline comparisons.

4.1 Synthetic Experiments

  • R-LMC closely follows exact-gradient LMC’s FI decrease on a Gaussian mixture, whereas SG-LMC diverges under the state-dependent Student oracle with infinite variance.Both stochastic methods use 16 oracle queries per update and the same fixed step size.Source: §4.1 · p. 6, Figure 1 · p. 7
  • On a radial quartic target with generalized smoothness, A-LMC reaches lower FI than fixed-step LMC over the observed horizon, supporting adaptive steps to control discretization error.The experiment uses exact gradients and γ = 2/3.Source: §4.1 · p. 6, Figure 1 · p. 7
  • RA-LMC closely tracks exact-gradient A-LMC on the radial cubic target, while fixed-step SG-LMC and R-LMC encounter numerical failures and A-SG-LMC plateaus at higher FI.The comparison uses the same state-dependent heavy-tailed oracle and 16 queries per stochastic update.Source: §4.1 · pp. 6–7, Figure 1 · p. 7

4.2.1 Empirical Evidence for BG-p Oracle

  • Minibatch-gradient error norms from Qwen3-4B LoRA fine-tuning resemble synthetic norms with an infinite second moment, suggesting heavy-tailed stochastic gradient errors.Figure 2 compares errors at step 2,500 on OpenBookQA with norms from a one-sided Lévy-stable reference construction.Source: Figure 2 · p. 7, §4.2.1 · p. 8
  • During Bayesian fine-tuning of Qwen3-4B on OpenBookQA and Qwen3-VL-4B-Instruct on SLAKE, empirical first moments of minibatch-gradient error norms increase with distance from the final LoRA iterate, supporting state-dependent noise.Each plotted point averages error norms from 32 independent minibatches of size four.Source: Figure 3 · p. 7, §4.2.1 · p. 8

4.2.2 Comparison with Baselines

  • RA-LMC has the highest accuracy and lowest Brier score on both datasets, the lowest NLL on OpenBookQA, and remains competitive on other metrics.Table 1 reports OpenBookQA and SLAKE test results.Source: §4.2.2 · p. 8, Table 1 · p. 8

5 CONCLUSION

The paper extends FI convergence analysis for non-log-concave Langevin sampling beyond globally Lipschitz gradients and uniformly bounded noise variance. It reports guarantees without dissipativity or functional inequalities and favorable Bayesian LoRA fine-tuning comparisons for RA-LMC.

5 CONCLUSION

  • The work extends the FI framework beyond globally Lipschitz gradients and uniformly bounded noise variance, with non-asymptotic guarantees for three methods and no dissipativity or functional-inequality assumption.RA-LMC’s analysis controls the interaction between robust estimation and adaptive step sizes; synthetic and Bayesian LoRA experiments support the mechanisms and oracle model.Source: §5 · p. 8

AI Use Statement

  • The authors report using generative AI for proof writing, method implementation, qualitative analysis, code, literature analysis, and manuscript editing, with human review of the assisted work.They state that AI did not develop theoretical models, provide critical proof ingredients, or design experiments.Source: AI Use Statement · p. 9

Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-Tuning: Supplementary Materials

The supplement formalizes the algorithmic laws and constants used in its convergence theorems, then gives guarantees for fixed-batch robust, exact-gradient adaptive, and robust adaptive LMC. Its proof outline connects robust drift-error control to time-averaged Fisher information and complexity bounds.

A.1 Information and interpolation conventions

  • The appendix defines the time-averaged law as a deterministic mixture of the process laws and assumes fresh, mutually independent oracle calls at each stochastic event.For fixed step size and horizon T = Nh, the event count is N.Source: §A.1 · p. 14

A.2 Notation, constants, and deterministic schedules

  • The notation section defines coefficients for discretization-error bounds and records that the squared local Lp noise scale is nonnegative, nondecreasing, and concave in squared distance.This concavity lets a trajectory second-moment bound control the average squared local noise scale.Source: §A.2 · pp. 15–16

A.3 Complete convergence guarantees

  • Theorem A.2 specifies fixed-batch robust LMC with a state-scaled clipping radius and states a finite-time convergence bound under its assumptions.When B = 0, the proof removes the need for an initial second-moment bound; when B = G = 0, the construction reduces to exact-gradient LMC.Source: §A.3 · p. 16, Then the updates in (6) satisfy · pp. 16–17
  • Theorem A.4 chooses an adaptive step size so the time-averaged law has relative Fisher information at most ε, and provides complexity specializations for generalized and global smoothness.The global-smoothness case uses a constant step size and deterministic oracle count.Source: Then the updates in (6) satisfy · pp. 17–18
  • Theorem A.6 gives a finite-horizon guarantee for robust adaptive LMC, with a noise-aware step size, independent robust-estimator batches, and a clipping radius scaled to the current state.The last occupied interval is truncated at T, and the number of events through T is almost surely finite.Source: Then the updates in (6) satisfy · p. 18
  • For p = 2, robust adaptive LMC has oracle complexity scaling as ε^-5 under BG-0 noise and ε^-3 under state-independent finite-variance noise.The stated powers hold with other problem parameters fixed.Source: Then the updates in (6) satisfy · p. 19

A.4 Proof of Theorem A.2

  • The fixed-batch proof combines a robust-estimator mean-squared-error bound, KL dissipation, and trajectory-moment control to obtain a time-averaged Fisher-information bound.The trajectory estimate controls how estimation error depends on the state-dependent noise scale.Source: §A.4 · p. 19

A.4.1 Mean-squared error of the robust estimator

The robust estimator combines a geometric-median center with residual clipping to turn finite p-th-moment oracle noise into mean-squared error control. Independent replicates and a state-dependent clipping radius balance bias and variance.

Tail amplification by the geometric-median center

  • A geometric median of three independent oracle evaluations has a finite second moment even when each call has only a finite p-th moment, because a large center error requires at least two large errors.Independence squares the tail probability for the joint event.Source: §A.4.1 · pp. 19–20

Conditional mean-squared error of the robust estimator

  • Each robust replicate centers three oracle calls geometrically and clips the residual of a fourth; averaging b independent replicates reduces variance by a factor of 1/b.The clipping radius balances squared bias against variance after averaging.Source: §A.4.1 · p. 20
  • For a deterministic state x, Lemma A.9 bounds the conditional mean-squared error of the averaged robust estimator using the local noise scale and clipping radius.Choosing λ = s b^1/p balances the squared-bias and averaged-variance terms; the zero-noise case is exact.Source: §A.4.1 · p. 20

Concavity of the squared local Lp noise scale

  • For 1 < p ≤ 2, the squared local Lp noise scale is nonnegative, nondecreasing, and concave in squared distance from the reference point.This property supports controlling the average local noise scale using a trajectory second-moment bound.Source: §A.4.1 · p. 21

A.4.2 From drift error to relative Fisher information

This proof section transfers drift-estimation error into relative Fisher-information control, then closes the bound using trajectory moments and an accuracy schedule. It also identifies the initial-moment assumption needed when noise grows with state and gives the route to iteration and oracle complexities.

A.4.2 From drift error to relative Fisher information

  • A transfer proposition bounds time-averaged relative Fisher information using the mean-squared drift error, under global smoothness and a finite initial second moment.The proof uses KL dissipation for an interpolation with a possibly random frozen gradient vector.Source: §A.4.2 · p. 22

A.4.3 Trajectory moment control and completion of the finite bound

  • The trajectory argument bounds average squared distance to the reference point, uses concavity of the local noise scale to control estimation error, and closes the finite-time Fisher-information bound.The proof then verifies the accuracy schedule and derives iteration and stochastic-oracle complexities.Source: §A.4.3 · pp. 24–25, §A.4.4 · p. 25
  • For state-dependent noise with B > 0, the proof retains a known finite initial second-moment bound; finite initial KL alone does not imply it for general non-log-concave targets.When B = 0, the local noise scale is constant, and the proof removes the initial second-moment condition.Source: §A.4.3 · p. 25

A.4.4 Accuracy schedule and oracle complexity

  • The accuracy schedule establishes Fisher information at most ε, while the proof derives a finite stochastic-oracle bound for the resulting procedure.The complexity derivation treats state-dependent and state-independent noise separately.Source: §A.4.4 · pp. 25–26

A.5 Proof of Corollary A.3

This section develops the estimates and discretization bounds used to prove adaptive LMC and RA-LMC convergence under generalized smoothness, including state-dependent noise. It establishes Fisher-information accuracy and complexity guarantees, along with explicit schedules and relevant specializations.

A.5 Proof of Corollary A.3

  • When K0 = O(d), the iteration bound follows from (50), and fixed G > 0 absorbs the lower-order iteration complexity as ε decreases.With B = G = 0, all four robust-replicate calls equal the exact gradient, so replacing them with one evaluation leaves the LMC iterates unchanged and QT = NT.Source: §A.5 · pp. 26–27

A.6 Minibatched SG-LMC under uniform variance

  • For fixed σ² > 0 and K0 = O(d), the accuracy-based SG-LMC batch choice yields a leading complexity term O((K0 + 1)L^2dσ²/ε^3), avoiding the extra Ld factor in the compared schedule.The batch size is proportional to σ²/ε, without an additional Ld factor; Sahin et al. (2026b, Appendix B.3) use a schedule with oracle work O(L^3d^3σ²/ε^3).Source: §A.6 · p. 28

A.7 Proof of Theorem A.4

  • The adaptive-step proof sets up an interpolation over intervals that end at T, shortening only the final occupied interval when necessary.It defines each interval length as the minimum of the adaptive step and the remaining time.Source: §A.7 · p. 28

A.7.1 Endpoint and gradient moment estimates

  • Under generalized smoothness, ∇V is locally Lipschitz and its Jacobian satisfies an almost-everywhere bound by L0 + L1∥∇V(x)∥^γ.The endpoint gradient-variation estimates are treated separately for polynomial and exponential cases because the displacement exponent diverges as γ approaches 1.Source: §A.7.1 · pp. 28–29

A.7.2 Gradient second moment bound

  • For measures with finite relative Fisher information, the squared gradient moment is bounded by 2 FI(µ∥π) plus dimension- and smoothness-dependent terms.This estimate extends globally smooth gradient-moment bounds to generalized smoothness.Source: §A.7.2 · p. 29

A.7.3 Exact continuation and KL dissipation

  • The exact Langevin diffusion has a unique strong solution for all t ≥ 0, preserves π, and can continue the adaptive interpolation after any finite number of intervals.An adapted-drift KL-dissipation inequality provides the analytical bridge from drift approximation to Fisher information and accommodates progressively selected drifts.Source: §A.7.3 · p. 30

A.7.4 Gaussian estimates and a reusable approximate-drift lemma

  • The appendix establishes an adaptive-step discretization bound for approximate drifts under generalized smoothness, extending a frozen-gradient argument to general approximate drifts.The bound’s coefficients separate constant terms from terms involving the population gradient and approximate drift; it is used directly in the robust adaptive method’s proof.Source: §A.7.4 · pp. 32–33
  • The reusable approximate-drift coefficients are nondecreasing in h and, for 0 < h ≤ 1, are bounded by h times their values at one.The proof treats the regimes 0 < γ < 1, γ = 1, and L1 = 0.Source: §A.7.4 · pp. 33–34

A.7.5 Discretization error for adaptive LMC with exact gradients

  • For exact gradients, adaptive LMC’s interval discretization error is bounded using coefficients A1(h), B1(h), and Q1(h) applied to the starting gradient norm.The bound is ℓ{A1(h) + B1(h)g^2 + Q1(h)g} for an interval of length ℓ.Source: §A.7.5 · p. 36

A.7.6 Closure, nonaccumulation, and finite horizon bounds

  • Finite-horizon Fisher-information bounds extend to any initialization with finite KL divergence, and adaptive intervals do not accumulate in finite time.The proof uses finite-step truncation and approximation to pass the bounds to the full process; each occupied interval uses one exact gradient query.Source: §A.7.6 · pp. 38–39

A.7.7 Accuracy schedule and complexity

  • The explicit accuracy schedule yields a Fisher-information bound of at most ε, with its horizon and step size chosen using the model parameters.The schedule sets K0/T = ε^4 and bounds the discretization contribution by 3ε/8.Source: §A.7.7 · pp. 39–40
  • At γ = 1, the accuracy endpoint is proved directly, without taking a limit in the γ < 1 constants, completing the accuracy and complexity claims in Theorem A.4.The expanded bounds use S_γ = O((dΛ)^(2/(2−γ))) for 0 < γ < 1, with Λ = max{1, L_0, L_1}.Source: §A.7.7 · p. 41

A.8 Proof of Corollary A.5

  • In the globally smooth exact-gradient case, the specialization recovers the globally smooth exact-gradient LMC order, and neither the algorithm nor its accuracy schedule depends on γ.Here L_1 = 0, so the adaptive interval is deterministic.Source: §A.8 · p. 41

A.9 Proof of Theorem A.6

  • The robust estimator and adaptive step size control different errors: estimation yields a finite conditional second moment, while adaptivity controls gradient variation and accumulated estimator error.The step-size denominator includes κ_sp(X_τk)^2 to bound that accumulated error.Source: §A.9 · p. 41

A.9.1 Finite iteration localization and proof quantities

  • For finite-iteration localization, the proof follows RA-LMC for at most m iterations and then uses exact Langevin diffusion as a proof-only continuation to establish the needed moment bounds.The localization argument starts with bounded initial support while retaining the actual finite KL divergence.Source: §A.9.1 · pp. 41–42

A.9.2 Discretization, FI, and gradient and drift moment bounds

  • The reported bounds are g ≤ 9MT + 40Dγ(h) + 21e and a ≤ 18MT + 80Dγ(h) + 44e.Source: §A.9.2 · p. 43

A.9.4 From per-update estimation error to error over elapsed time

The proof turns per-update estimation-error control into finite-horizon bounds, then removes iteration truncation and bounded-support restrictions. An accuracy schedule gives an FI guarantee below ε and bounds expected estimation error.

A.9.4 From per-update estimation error to error over elapsed time

  • The scalar feedback term can be absorbed when 2ρbκ−1χT,h ≤ 1, a sufficient restriction verified for the theorem’s batch size.Source: §A.9.4 · p. 45
  • The proof converts per-update estimation error into elapsed-time control, giving χT,he ≤ h/T + JT,h and finite-horizon bounds uniform in truncation level m.Source: §A.9.4 · p. 45

A.9.5 Removing the iteration truncation and the bounded support restriction

  • Removing iteration truncation preserves the finite-horizon bounds: update times do not accumulate before T, and convergence arguments pass the count, error, and FI bounds to the original process.Source: §A.9.5 · p. 45
  • Removing the temporary bounded-support condition retains the same noise, step-size, and count bounds without recomputing algorithm parameters.The limiting FI inequality retains the actual duration-weighted estimation error.Source: §A.9.5 · p. 46
  • Each occupied stochastic interval uses exactly 4b oracle evaluations, so the expected oracle-call count is 4b times the expected interval count.Source: §A.9.5 · p. 46

A.9.6 Accuracy schedule and general complexity

  • The prescribed accuracy schedule yields E_T ≤ ε/10 and an FI bound of 57ε/64 < ε.It chooses h with Dγ(h) ≤ ε/64 and a prescribed batch size.Source: §A.9.6 · pp. 46–47

A.10 Proof of Corollary A.7

The proof specializes RA-LMC’s oracle complexity and verifies reductions to simpler methods. It also makes explicit the scope of its compact-rate and initialization claims.

A.10 Proof of Corollary A.7

  • For p = 2, EQT is O(ε−5) under BG-0 noise (B > 0) and O(ε−3) under uniformly bounded variance (B = 0).Other problem parameters are fixed.Source: §A.10 · p. 48
  • With K0 = O(d), the proof makes dimension and smoothness dependence explicit using Λ = max{1, L0, L1}, with hidden constants depending only on γ.Source: §A.10 · p. 48
  • The compact B > 0 rate applies only below a small-accuracy threshold that may depend on problem parameters; finite K0 alone does not imply a finite initial second moment.Source: §A.10 · p. 48
  • The compact oracle-formula constants are pointwise in γ, not uniform as γ approaches 1.Source: §A.10 · p. 48

A.11 Verification of the algorithmic reductions

  • When L1 = 0 and κ = 0, the adaptive duration reduces to fixed-step robust LMC; when B = G = 0, exact gradients make robust estimation redundant and recover adaptive exact-gradient LMC.Source: §A.11 · p. 49

B.1.1 Experimental Setting

The experiments use synthetic target potentials and compare methods under controlled noise, smoothness, and step-size settings. Fisher information is estimated from time-averaged trajectory laws using uniform time samples and smoothed density estimates.

B.1.1 Experimental Setting

  • The synthetic experiments use target potentials listed in Table B.1, including two radial potentials that are not globally smooth.Source: Table B.1 · p. 50, §B.1.1 · p. 50
  • The state-dependent heavy-tailed oracle uses a normalized spherical Student random vector and has finite p-th moment but infinite variance, with BG-p parameters B = G = 8 and xref = 0.The setting uses p = 1.2 and ν = 1.4.Source: §B.1.1 · p. 50
  • LMC uses one exact-gradient evaluation per update, while SG-LMC and R-LMC each use 16 oracle queries; the primary comparison fixes physical time, with fixed-update results reported complementarily.Source: §B.1.1 · p. 50

B.1.2 Calculation of Fisher information

  • The primary Fisher-information estimate samples a uniform time from [0, T] for each of 8,192 trajectories to approximate the time-averaged law.Source: §B.1.2 · p. 51
  • For numerical stability, both simulated and target laws are smoothed with N(0, 0.08I2), and FI is evaluated at specified time or update checkpoints.The reported time checkpoints range from T = 0.25 to 32; iteration checkpoints range from N = 32 to 8,192.Source: §B.1.2 · p. 51

B.1.3 Additional Results

At larger step sizes, the additional experiments preserve the qualitative method comparisons from the primary settings. The fixed-update results also show that these advantages persist when adaptive methods may have reached different physical times.

B.1.3 Additional Results

  • The fixed-N comparison complements fixed-time results, but adaptive methods may reach different physical times after the same number of updates.Source: §B.1.3 · p. 51, Figure B.3 · p. 52
  • The plotted curves show median FI across ten seeds, with interquartile-range bands.Source: §B.1.3 · p. 51
  • At larger step sizes, R-LMC remains closer to exact-gradient LMC than SG-LMC in the heavy-tailed setting, while RA-LMC stays close to A-LMC and below fixed-step LMC and A-SG-LMC in the combined setting.In the combined setting, fixed-step stochastic methods become numerically unstable, leaving curves absent or truncated.Source: §B.1.3 · p. 52, Figure B.2 · p. 52, Figure B.3 · p. 52
  • Under generalized smoothness, A-LMC achieves lower FI than fixed-step LMC at later iterations in the larger-step experiments.Source: §B.1.3 · p. 52

B.2.1 Experimental Setting

The fine-tuning appendix separates measurements used to support the BG-p oracle from the RA-LMC baseline comparison. It describes a fitted heavy-tail reference model and evaluates how gradient-error magnitudes vary along fine-tuning trajectories.

B.2.1 Experimental Setting

  • Oracle-evidence trajectories use minibatch-mean gradients, whereas the Table 1 RA-LMC comparison replaces that mean with a geometric-median-centered, residual-clipped estimate.Source: §B.2.1 · p. 52
  • The experiments fine-tune frozen Qwen3-4B and Qwen3-VL-4B-Instruct backbones with rank-8 LoRA factors for 5,000 AdamW updates.The datasets are OpenBookQA and the SLAKE yes/no subset.Source: §B.2.1 · p. 52
  • The empirical clipping radius uses an online dispersion proxy because the theoretical local p-moment noise scale depends on unknown quantities; a lagged scale prevents a current outlier from raising its own threshold.Source: §B.2.1 · pp. 53–54

B.2.2 Gradient-Error Histograms

  • The heavy-tail analysis fits a positive β-stable reference model to squared gradient-error norms; its infinite first moment implies an infinite second moment for the modeled error norms.For Figure 2, the fitted β = 0.2 is at the lower search bound.Source: §B.2.2 · pp. 54–55

B.2.3 State Dependence of Gradient-Error Moments

  • Across 50 checkpoints, the study estimates first-moment gradient-error magnitudes from groups of independent minibatches; the resulting plots provide evidence of oracle-noise state dependence.The first moment is used because it is well defined for BG-p models with p > 1 and is less sensitive to extreme errors than higher empirical moments.Source: §B.2.3 · p. 55

B.2.4 Additional Results

Additional fine-tuning analyses find heavy-tailed gradient-error distributions and increasing empirical moments with distance from the final iterate. Related-work comparisons distinguish Fisher-information guarantees from global mixing guarantees and identify obstacles in extending optimization techniques to Langevin sampling.

B.2.4 Additional Results

  • Across OpenBookQA and SLAKE, empirical gradient-error histograms closely resemble fitted β-stable reference distributions, especially from step 1,250 onward.The figures compare empirical and synthetic distributions at five checkpoints for each fine-tuning setting.Source: §B.2.4 · p. 55, Figure B.4 · p. 56, Figure B.5 · p. 56
  • Empirical gradient-error moments across displayed orders generally increase with distance from the final iterate, extending the state-dependence pattern beyond first moments.The analyses cover moment orders p ∈ {1, 1.25, 1.5, 1.75, 2} in both fine-tuning settings.Source: §B.2.4 · pp. 55, 57, Figure B.6 · p. 57, Figure B.7 · p. 57

C.1 Non-Log-Concave Langevin Sampling

  • Fisher-information convergence does not by itself guarantee global mixing or correct probability mass across well-separated modes.Distributional guarantees address a more global question and typically rely on additional target structure, such as functional inequalities or dissipativity.Source: §C.1 · p. 58
  • Prior relaxations of smoothness or variance assumptions generally retain other controls, while existing FI extensions do not directly cover both gradient-dependent curvature and state-dependent finite-p-moment noise.The cited state-dependent-variance result assumes finite second moments, unlike BG-p when p < 2.Source: §C.1 · p. 58

C.2 Generalized Smoothness and Heavy-Tailed Noise in Optimization

  • Optimization methods under generalized smoothness or heavy-tailed noise target stationarity or objective quality, not the distribution of Langevin iterates.The cited stochastic optimization results permit finite variance or use clipping, normalization, and related mechanisms for heavy-tailed noise.Source: §C.2 · pp. 58–59
  • Clipping or normalizing Langevin drift without changing the Brownian term generally alters the target distribution, so optimization descent alone does not establish sampling correctness.The modified generator generally does not leave the target density invariant.Source: §C.2 · p. 59
  • For FI guarantees, centering and clipping residuals separate bias from squared-error control, while adaptive steps require controlling error weighted by elapsed time.The paper also distinguishes averages over physical time from averages over iteration indices.Source: §C.2 · p. 59

C.3 Bayesian Fine-Tuning and Experimental Baselines

The section describes Bayesian LoRA baselines that represent predictive uncertainty through single adapters, repeated predictions, ensembles, local approximations, or distributions over low-rank updates. It distinguishes these approaches from the paper’s RA-LMC comparator and from BayesTune’s parameter-selection use of Langevin sampling.

C.3 Bayesian Fine-Tuning and Experimental Baselines

  • MLE and BAG MAP each return one adapter rather than a posterior, while temperature scaling adjusts MLE logits and Monte Carlo dropout averages ten stochastic predictions.The BAG dropout implementation uses probability 0.1.Source: §C.3 · p. 59
  • Deep Ensemble averages predictions from ten independently trained adapters, whereas Laplace LoRA uses a local Gaussian approximation around a fine-tuned adapter.These approaches obtain predictive diversity through independent optimization or a local posterior approximation.Source: §C.3 · p. 59
  • BLoB, ScalaBL, and Training-Free Bayesianization construct uncertainty over low-rank updates using variational, latent-space, or Gaussian-noise approaches.Their BAG implementations average ten predictive samples.Source: §C.3 · p. 59
  • The SG-LMC comparator uses minibatch-averaged gradients, fixed step sizes, and discounted Gaussian perturbations, unlike RA-LMC’s estimator and step-size choice.BayesTune instead samples sparsity-scale variables for selecting weights and does not produce posterior samples of the final fine-tuned adapter.Source: §C.3 · pp. 59–60

Related papers

Loading 2610.12865v1…