PaperlayerView exact revision on arXiv

Source-linked AI summary

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

Hang Zou, Chao Zhang, Yuzhi Yang, Yu Tian, Samson Lasaulce, Mérouane Debbah

arXiv:2610.01537v1cs.LG

TL;DR

In federated LLM fine-tuning, LoRA creates a trade-off between aggregation accuracy and preserving low-rank adapters. FedFit combines shared vector banks with alternating updates and residual spectral correction; on Qwen2.5 models, it reports improved benchmark performance alongside sharply reduced communication.

  • Problem

    In FedLLM, combining LoRA client updates must balance accurate aggregation against retaining a single low-rank adapter, while separate factor averaging discards cross-client terms.

    Source: A. Motivation · p. 1

  • Method

    FedFit represents adapters with fixed shared projections and disjoint banks, alternates single-bank and joint updates with residual spectral correction, and supports asymmetric uplink and downlink quantization.

    Source: A. Disjoint Shared Vector-Bank… · p. 4, 1) Decoupled uplink and downlink… · p. 7, §VII · pp. 12–13

  • Results

    Across tested LLM sizes, FedFit achieves lower evaluation perplexity and higher average scores; for Qwen2.5-7B-Instruct, it communicates about 190 KB versus 20 MB for standard LoRA methods, roughly 100× compression.

    Source: §VII · p. 9

  • Takeaways & Limitations

    The authors frame the findings as a first step toward deploying billion-parameter models over highly constrained wireless networks.

    Source: §VIII · p. 14

    The study leaves client heterogeneity and partial participation, transmission errors and delays, and acquisition of new data for future investigation.

    Source: §VIII · p. 14

Abstract

from arXiv · show

Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental "aggregation dilemma" between the accurate Sum-of-Products (SoP) and the communication-efficient Product-of-Sums (PoS) implementations. To tackle these challenges, we propose FedFit. First, to significantly reduce communication overhead, we introduce a disjoint shared vector-bank parameterization that reconstructs high-dimensional adapter matrices from two compact and disjoint global vector banks. Second, to address the aggregation dilemma, we devise an alternating optimization schedule. By cycling between decoupled single-bank updates (which allow for accurate aggregation) and joint updates corrected by a Residual Spectral Aggregation mechanism, we resolve the conflict between SoP and PoS. Additionally, we integrate blockwise quantization with client-side error feedback to further compress the transmitted vectors. Furthermore, we establish theoretical convergence guarantees for the proposed algorithm. Extensive experiments on Qwen2.5 models demonstrate that FedFit achieves perplexity performance comparable to standard federated LoRA methods, while providing compression ratios up to 100x higher.

I. INTRODUCTION

The introduction frames federated LLM fine-tuning as a communication problem compounded by LoRA’s aggregation trade-off. FedFit combines shared vector banks, alternating updates, and quantization to address these challenges.

A. Motivation

  • Full-parameter FedLLM is prohibitively expensive, and although LoRA reduces costs, communication remains a bottleneck and its SoP and PoS aggregation rules trade low-rank structure against accuracy.Averaging products can lose exact low-rank structure, whereas averaging factors separately discards cross-client cross terms.Source: A. Motivation · p. 1

B. Related Works

  • Prior work studies federated LoRA, aggregation, and quantization, but the cited literature does not address shared vector banks with communication-constrained, quantized FedLLM updates.The related-work discussion distinguishes these gaps from work on raw LoRA adapters and other federated compression methods.Source: B. Related Works · p. 2

C. Contributions

  • FedFit uses disjoint global vector banks, alternating single-bank and residual-corrected joint updates, and post-training bank quantization with client-side error feedback.The single-bank phases are described as ensuring perfect aggregation with halved transmission; the joint phases use residual spectral correction to mitigate the aggregation gap.Source: C. Contributions · p. 2
  • FedFit reports better performance on Qwen2.5 models while reducing communication costs to approximately 1–3% of standard federated LoRA methods.The contribution summary reports this comparison across Qwen2.5 models.Source: C. Contributions · p. 2

II. SYSTEM MODEL

  • The system model has a server and K clients with private labeled data collaborating over T communication rounds to fine-tune a pretrained LLM.The model adopts LoRA adapters while keeping the pretrained weights frozen.Source: §II · p. 3

III. PROPOSED FEDFIT FRAMEWORK

  • FedFit is presented as a framework combining disjoint shared vector-bank parameterization with an alternating optimization schedule for wireless federated learning.The framework addresses communication overhead and aggregation discrepancy in bandwidth-constrained settings.Source: §III · p. 3

A. Disjoint Shared Vector-Bank Parameterization

FedFit reconstructs layerwise adapters from compact, disjoint shared banks, then alternates decoupled and joint updates to balance exact aggregation with full-space optimization.

A. Disjoint Shared Vector-Bank Parameterization

  • Two disjoint, low-dimensional global banks reconstruct layerwise A and B adapters through fixed projections, replacing repeated transmission of high-dimensional adapter parameters with compact shared parameters.The projections are fixed and seeded across clients; compute-heterogeneous clients can train on a sub-bank that the server zero-pads before aggregation.Source: A. Disjoint Shared Vector-Bank… · pp. 3–4

B. Alternating Optimization Strategy

  • A-only and B-only phases freeze the other bank for exact aggregation and half uplink cost, while joint A+B phases train the full space but can incur aggregation errors.The schedule cycles between solo and joint phases to balance exact aggregation with capturing correlations between A and B.Source: B. Alternating Optimization Strategy · p. 4

IV. AGGREGATION APPROACHES

  • LoRA’s bilinear factorization breaks the linearity assumed by standard federated aggregation; FedFit uses exact decoupled updates and residual spectral correction for joint updates.The correction mechanism is introduced to rectify aggregation errors during joint update phases.Source: §IV · p. 4, B. Alternating Optimization Strategy · p. 4

A. The Aggregation Dilemma

Federated LoRA must reconcile efficient aggregation that preserves low rank with accurate aggregation that can produce high-rank updates. FedFit uses exact single-bank phases and residual spectral correction in joint phases to address this conflict.

A. The Aggregation Dilemma

1) Product of Sums (PoS):

  • Separate averaging of A and B is efficient but approximate because the product of averaged factors generally differs from the average of client products.The mismatch reflects missing cross-client coupling between A and B.Source: 1) Product of Sums (PoS) · p. 4

2) Sum of Products (SoP):

  • Summing clients’ adapter products gives the exact arithmetic mean of local updates, but can yield rank up to K × r.Returning that update to rank-r factors can require costly SVD and introduce truncation errors.Source: 2) Sum of Products (SoP) · p. 4

B. Perfect Aggregation in the A/B-only Phase

C. Aggregation Strategy in the A+B phase: From Separate Averaging to Spectral Approximation

  • In joint phases, FedFit adds a rank-1 approximation of the residual product error to stable FedAvg bank vectors, restoring coupled information while keeping the model in a stable parameter region.The residual is the difference between the true sum of products and the product of FedAvg sums; a projected QR decomposition computes its correction with linear complexity O((dA + dB)K^2).Source: C. Aggregation Strategy in the A+B… · pp. 5–6

A. Blockwise Round-To-Nearest (RTN) quantizer

FedFit quantizes the compact vector-bank representation blockwise rather than quantizing heterogeneous adapter matrices directly. Its RTN encoder represents the vector with integer codes and per-block scales.

A. Blockwise Round-To-Nearest (RTN) quantizer

  • The vector-bank representation turns transmitted state into a single 1-D vector with near-Gaussian, homogeneous entries, making blockwise quantization suitable.The paper presents vector-bank parameterization and quantization as a co-design, rather than two independent compression stages.Source: A. Blockwise Round-To-Nearest (RTN)… · p. 6
  • FedFit partitions the vector into contiguous blocks and uses a per-block scale with a symmetric integer dynamic range for RTN quantization.The range is C(b) = 2^(b−1) − 1 for bitwidth b.Source: A. Blockwise Round-To-Nearest (RTN)… · p. 6
  • The quantized representation consists of integer codes and their block scales, which the decoder uses to reconstruct the vector blockwise.The representation is denoted Q(v; b, D).Source: A. Blockwise Round-To-Nearest (RTN)… · p. 6

1) Decoupled uplink and downlink quantizers:

FedFit configures uplink and downlink quantizers independently, allowing asymmetric compression. On the uplink, clients use error feedback to carry forward quantization residuals into later updates.

1) Decoupled uplink and downlink quantizers:

2) Client-side error feedback on the uplink:

  • Client-side error feedback adds the accumulated quantization residual to the next bank difference before encoding it.The accumulator stores the residual not transmitted in the prior round, and error feedback is used only on the uplink.Source: 2) Client-side error feedback on the… · p. 7

B. Compression Ratio and Communication Cost Analysis

FedFit combines quantization and shared vector banks to reduce communication, while its analysis characterizes how bank size affects optimization and the conditions under which its objectives are smooth. The theoretical guarantees assume exact gradients and full-precision communication.

B. Compression Ratio and Communication Cost Analysis

A. Effect of Vector Banks in Convexity

  • The induced vector-bank objective is almost blockwise smooth and satisfies a mixed-smoothness property under the stated scaling-law approximation.The analysis assumes exact gradients and full-precision communication; minibatch sampling and quantization are outside the proof and add error terms under bounded variance and distortion.Source: §VI · p. 7, A. Effect of Vector Banks in Convexity · p. 8
  • Smaller dA and dB worsen effective smoothness and weak-convexity constants, trading lower computation and communication costs against performance.The constants worsen as the vector-bank dimensions shrink.Source: A. Effect of Vector Banks in Convexity · p. 8
  • The analyzed landscape has many local minima and saddle points, so convergence to a stationary point alone does not ensure good performance.The text identifies alternating freeze-and-update schemes as especially prone to becoming trapped there.Source: A. Effect of Vector Banks in Convexity · p. 8

B. Convergence of the Proposed Method

The convergence analysis establishes stationary-point convergence for alternating single-bank updates and for joint A+B updates under their respective learning-rate conditions. The joint-only approach has practical constraints that may lead to training failure.

A-Only Phase and B-Only Phase

  • O(1/T) convergence holds for alternating A-only and B-only phases with constant η, under 0 < ηt < 2/max{βA, βB}.The theorem states convergence to a vector bank with ∇f(vA, vB) = 0.Source: B. Convergence of the Proposed Method · p. 8

A+B Phase

  • The joint A+B phase also converges to a stationary point with sufficiently small ηt under the stated blockwise- and mixed-smoothness assumptions.For constant η, the stated condition is 0 < η < 2/(max{βA, βB} + βmix).Source: B. Convergence of the Proposed Method · p. 9
  • Joint-only updates require a smaller learning rate and frequent aggregations, conditions the authors say are hard to satisfy and may cause training failure.The method is said to converge to the same suboptimal set even when SoP ≠ PoS.Source: B. Convergence of the Proposed Method · p. 9

VII. NUMERICAL RESULTS

FedFit outperforms federated LoRA baselines across Qwen2.5 model sizes while sharply reducing communication, and its experiments examine quantization, scheduling, aggregation correction, and robustness across settings.

Training dynamics

  • On Qwen2.5-0.5B-Instruct, FedFit consistently beats the plotted baselines in evaluation perplexity and keeps improving after FedIT and Flex-LoRA plateau after round 15.Fig. 2 shows evaluation perplexity during training.Source: Figure 2 · p. 9, §VII · p. 9

Comparison with SOTA FL methods for LLMs

  • For Qwen2.5-7B, FedFit communicates about 190 KB versus 20 MB for standard LoRA methods, roughly 100× less, while outperforming baselines overall.The reported comparison spans Qwen2.5 model sizes; full-precision FedFit outperforms baselines on seven common-sense benchmarks, with comparable performance under 4-bit RTN transmission.Source: §VII · p. 9, Table I · p. 10, Table II · p. 11

Impact of vector bank quantization

  • With 4-bit RTN on both directions, compression is approximately 4× with negligible performance degradation; 2-bit downlink instead significantly raises perplexity.Quantization effects are asymmetric: 2-bit uplink remains acceptable when downlink uses at least 4 bits.Source: §VII · pp. 9–10, Figure 3 · p. 12

Impact of training part scheduler

  • The one-round both-phase plus one-round solo-phase schedule has the best tested training curve and balances benchmark performance, communication cost, and stability.A non-alternating schedule has slightly lower evaluation perplexity but a lower common-sense average, suggesting potential overfitting.Source: §VII · pp. 10–12, Table IV · p. 11, Figure 4 · p. 12

System-level overhead and wall-clock breakdown

  • FedFit takes 23.8 versus FedIT’s 24.5 s/round; at 7B, its uplink is about 1.12 s, or 0.30 s with 4-bit RTN, on a 5 Mbps link.Local SFT dominates wall-clock time, while the reported uplink time remains below it.Source: §VII · pp. 12–13, Table V · p. 13, Table VI · p. 13

Impact of vector-bank dimension and LoRA rank

  • Doubling bank dimension from 524K to 1.05M improves evaluation perplexity from 5.62 to 5.61 and the seven-task average from 60.61 to 60.68; rank changes move metrics by at most 0.3–0.5 points.At fixed communication budget, the results favor larger banks over larger LoRA ranks.Source: §VII · p. 13, Table VII · p. 13

Impact of residual spectral correction

  • The both-bank RSC variant attains the best seven-task average, while one-sided variants trail no RSC; at K=16 and K=32, the rank-1 gate returns FedAvg.The core matrix’s top mode holds 7.9% of residual energy at K=16 and 4.1% at K=32, versus 36.3% at K=4.Source: §VII · pp. 13–14, Table VIII · p. 13, Table XII · p. 14

Generalization and robustness

  • On MetaMathQA fine-tuning evaluated with GSM8K and MMLU, FedFit tracks FedIT while sending 13.2/21.4× fewer bytes; results also remain stable across tested non-IID partitions and K=4–32.The math-task comparison covers Qwen2.5-1.5B/3B-Instruct; non-IID common-sense averages span 60.60–60.93, and client-count tests use full participation.Source: §VII · pp. 13–14, Table IX · p. 13, Table XI · p. 14

VIII. CONCLUSIONS

FedFit combines shared vector-bank parameterization with alternating low-rank-factor updates to address communication and aggregation bottlenecks in federated LLM fine-tuning. The authors report communication costs of 1–3% of standard methods, while identifying several unstudied settings.

VIII. CONCLUSIONS

  • FedFit reconstructs client models from a shared, ultra-compact parameter space and uses alternating low-rank-factor updates to pursue accurate aggregation.The authors state that the shared representation does not sacrifice expressivity and that sequential factor updates are viable for accurate aggregation in distributed settings.Source: §VIII · p. 14
  • Communication costs of 1–3% of standard methods are achievable by coupling structural parameter sharing with post-training quantization.The authors describe these results as an important first step toward deploying billion-parameter models over constrained wireless networks.Source: §VIII · p. 14
  • The paper leaves client heterogeneity, partial participation, transmission errors and delays, and new-data acquisition for future investigation.It also calls for deeper analysis of how the LLM model affects the results.Source: §VIII · p. 14

APPENDIX A PROOF TO THEOREM 1

The proof of Theorem 1 represents bank-vector updates as linear maps into the full update space, bounds their smoothness, and derives mixed smoothness using the chain rule and symmetry.

APPENDIX A PROOF TO THEOREM 1

  • With vA fixed, the proof treats the mapping from vB to the full stacked update space as a linear operator.The proof states that the remaining case follows by symmetry.Source: Appendix A · p. 14
  • The operator P_B is formed from randomly generated one-hot rows, and the proof estimates its operator norm under the stated large-dimension condition.It assumes the full update dimension is much larger than the bank dimension and sufficiently large.Source: Appendix A · p. 14
  • The proof obtains smoothness for matrix B by combining its preceding estimates, with the result for matrix A following by symmetry.Source: Appendix A · pp. 14–15
  • Using the chain rule, the proof derives mixed smoothness from the bilinear LoRA mapping and the unit operator-norm bounds for P_A and P_B.It invokes bounded-gradient and full-space β-smoothness assumptions, then gives the corresponding mixed-gradient bound symmetrically.Source: Appendix A · p. 15

APPENDIX B PROOF TO THEOREM 3

Theorem 3’s proof bounds the objective gap under a constant learning rate and concludes an O(1/T) rate, with gradient norms vanishing along the generated sequence.

APPENDIX B PROOF TO THEOREM 3

  • O(1/T) convergence to a stationary point follows for a constant learning rate below 2/(max{βA, βB} + βmix).If the iterates converge, continuity of the gradient implies that the limiting point has zero gradient.Source: Appendix B · p. 15
Loading 2610.01537v1…