PaperlayerView exact revision on arXiv

Source-linked AI summary

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

Juseok Jeon, Ramy E. Ali, Doyun Kwon, Myungbeom Her, Jinhwi Kim, Jinhyun So

arXiv:2608.15381v1cs.AIcs.LG

TL;DR

Federated LoRA must balance accurate aggregation against continuity of locally optimized factors. FedPA-LoRA preserves local factors while aligning and aggregating their products, and experiments report consistent gains across data- and resource-heterogeneous settings.

  • Problem

    Federated LoRA faces a tension between accurate aggregation of local updates and continuity of locally optimized factors.

  • Method

    FedPA-LoRA preserves client factors, guides their products toward rank-specific global references, and aggregates heterogeneous-rank updates in common product space.

  • Results

    FedPA-LoRA consistently outperforms representative baselines across diverse fine-tuning tasks and data- and resource-heterogeneous settings.

  • Takeaways & Limitations

    The framework supports client-specific computation and communication budgets while retaining convergence guarantees for homogeneous and heterogeneous client ranks.

  • Takeaways & Limitations

    The convergence analysis relies on LoRA factor regularity and Lipschitz continuity assumptions.

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

1 Introduction

The introduction motivates federated LoRA for privacy-constrained large-model adaptation, identifies aggregation and resource heterogeneity challenges, and presents FedPA-LoRA as a product-aligned solution preserving local factors while supporting rank-aware aggregation.

  • Motivation: Federated learning coordinates model training without transferring raw data, addressing privacy and regulatory constraints on data held by devices or institutions.
  • LoRA background: LoRA represents each weight update as ΔW = BA, substantially reducing trainable and communicated parameters for federated fine-tuning of large models.
  • Aggregation challenges: Factor-wise averaging preserves compact adapters but creates aggregation mismatch, whereas product-space aggregation avoids cross-client terms before low-rank reconstruction.
  • Resource heterogeneity: Different client capabilities require different LoRA ranks, creating incompatible factor dimensions and necessitating rank-aware aggregation and client-specific redistribution.
  • FedPA-LoRA: FedPA-LoRA retains locally optimized factors, aligns their product with a rank-specific global reference, and aggregates heterogeneous-rank updates in the common product space.
  • Contributions: FedPA-LoRA jointly addresses aggregation and factor-level initialization mismatches, supports client-specific budgets, and is proven to converge to a stationary point of its product-guided local objective.

2 Related Work

Federated LoRA methods differ mainly in how they aggregate and redistribute low-rank factors, with prior approaches trading off factor continuity, aggregation fidelity, adaptation capacity, and communication. FedPA-LoRA retains client-specific factors while aggregating heterogeneous-rank updates in a common product space and decoupling local computation from communication rank.

  • Factor-wise aggregation: Factor-wise methods preserve fixed-rank global adapters but can restrict joint factor optimization or introduce aggregation mismatch.FedIT averages B and A directly, while FedRot-LoRA aligns client factors before averaging; freezing one factor can slow convergence or limit adaptation capacity.
  • Partial aggregation: FedSA-LoRA trains both factors locally but aggregates only A, producing an incomplete global adapter.
  • High-communication aggregation: FedEx-LoRA achieves exact aggregation through additional communication by transmitting the residual between averaged local products and the factor-wise average.
  • Heterogeneous-rank methods: HetLoRA supports heterogeneous ranks through zero-padding and truncation, but independently averaging padded factors still introduces aggregation mismatch.FlexLoRA instead aggregates local products and uses SVD-based reconstruction to redistribute rank-specific factors.
  • Our work: FedPA-LoRA retains client-specific factors, aggregates heterogeneous-rank updates in a common product space, and decouples local computation rank from communication rank.This design jointly preserves local optimization continuity and improves aggregation fidelity.

3 Motivating Example

The motivating example establishes a trade-off between aggregation fidelity and continuity of locally optimized LoRA factors: factor-wise methods preserve continuity but incur aggregation mismatch, whereas product reconstruction improves aggregation at the cost of factor-level initialization mismatch. FedPA-LoRA addresses both by combining product-guided local alignment with optimal server-side reconstruction.

  • Aggregation and continuity trade-off: Factor-wise aggregation introduces aggregation mismatch because products of independently averaged factors contain cross-client terms absent from locally optimized updates.FedIT preserves rank by averaging local factors independently, but the resulting product is generally not the ideal average of local products.
  • Representative baselines: FedRot-LoRA reduces aggregation error through client-specific orthogonal rotations, yet neither exactly recovers the average local products nor preserves the locally optimized factors.Rotation preserves each local product and improves factor consistency, but rotated factors are still independently averaged and later replaced by aggregated global factors.
  • Aggregation and continuity trade-off: Product-space reconstruction yields the optimal rank-r approximation of the ideal update, but replacing local factors with reconstructed factors causes factor-level initialization mismatch.FlexLoRA uses truncated SVD reconstruction; low-rank factorization is non-unique, so product-level equivalence does not ensure factor consistency.
  • FedPA-LoRA: FedPA-LoRA achieves the lowest observed aggregation error by combining optimal server-side reconstruction of averaged products with product-guided local alignment.The alignment keeps local products close to a global reference while avoiding the aggregation mismatch associated with factor-wise averaging.
  • Empirical trade-off: FedIT has the smallest factor-level initialization mismatch, FlexLoRA the largest, and FedRot-LoRA an intermediate value because their factor updates range from averaging to refactorization.The larger initialization errors for B than A are consistent with B capturing more client-specific adaptation under non-IID data.

4 Proposed Method: FedPA-LoRA

FedPA-LoRA preserves locally optimized LoRA factors while guiding their products toward rank-specific global references, mitigating initialization mismatch and client drift. It aggregates updates in product space and reconstructs a rank-constrained global adapter efficiently, supporting heterogeneous ranks and client-specific budgets.

  • Local factor preservation: FedPA-LoRA preserves each client’s locally optimized factors across communication rounds instead of replacing them with newly reconstructed global factors.This preserves optimization continuity and avoids factor-level initialization mismatch.
  • Product-level alignment: Product-level regularization aligns each local product with a global reference, mitigating client drift without replacing preserved factors or requiring matching local and reference ranks.The reference rank is determined by the client’s communication budget, while the local rank determines computation and uplink costs.
  • Product-space aggregation: The server aggregates LoRA updates directly in the common product space, avoiding factor-wise aggregation mismatch and naturally supporting heterogeneous local ranks.All client products have the adapted weight matrix’s dimensions.
  • Efficient global reconstruction: Reduced-QR and core-SVD reconstruction recovers the same optimal rank-Rg approximation while reducing dominant server complexity from O(d3) to O(N 2dr2).A randomized extension further reduces complexity to O(Ndr2), but gives up exact rank-constrained optimality.
  • Convergence analysis: Under Assumptions 1–4, Theorem 1 establishes convergence of FedPA-LoRA in the homogeneous-rank setting, while the analysis extends to heterogeneous ranks with an additional bounded reference-truncation error term.Appendix E further extends the convergence result to partial client participation with K active clients per round.

5 Experiments

Experiments show that FedPA-LoRA achieves the strongest performance across homogeneous and heterogeneous ranks, diverse data heterogeneity and client scales, rank variations, and both understanding and generation tasks. Ablations further show that local factor preservation and product-guided regularization jointly support accuracy and robustness.

  • Homogeneous ranks: 0.9146 average GLUE accuracy makes FedPA-LoRA the best method on all five homogeneous-rank tasks, outperforming RoLoRA by 4.86 percentage points.The result highlights the benefit of local factor preservation and product-space alignment.
  • Data heterogeneity: FedPA-LoRA achieves the highest MNLI accuracy across all evaluated heterogeneity levels, maintaining strong performance from near-IID to highly non-IID data.Product-guided regularization promotes global consistency while local factor preservation retains client-specific knowledge.
  • Client scale: 0.906 to 0.916 accuracy as clients increase from 3 to 50 shows that FedPA-LoRA consistently outperforms baselines and remains stable across client scales.The evaluated setting uses MNLI with rank r = 4 and β = 0.5.
  • Rank variation: 0.906–0.910 accuracy across evaluated ranks demonstrates that FedPA-LoRA remains robust to rank variation, unlike baselines with noticeable rank-sensitive fluctuations.FedSA-LoRA drops substantially at r = 24.
  • Heterogeneous resources: 6.82 percentage points is FedPA-LoRA’s improvement over HetLoRA in heterogeneous-rank GLUE, where it achieves 0.9226 average accuracy across all five tasks.It also achieves the best heterogeneous-rank generation results: 0.5018 accuracy on GSM8K and 0.4048 pass@1 on HumanEval.
  • Ablation and sensitivity: Moderate product-guided regularization improves and stabilizes performance, while weak guidance under non-IID data and overly strong regularization under IID data reduce accuracy.Under non-IID data, performance remains consistently strong for λ ∈ {1, 2, 5, 10}; under IID data, λ = 0.01 is insufficient and λ = 10 restricts local adaptation.

6 Conclusion

FedPA-LoRA preserves local factors while aggregating in product space to mitigate aggregation and factor-level initialization mismatches. It provably converges across homogeneous and heterogeneous client ranks and delivers consistent gains under data- and resource-heterogeneous settings.

  • 6 Conclusion: FedPA-LoRA preserves local factors while aggregating them in product space, jointly mitigating aggregation and factor-level initialization mismatches.The framework is designed to address both mismatch types simultaneously.
  • 6 Conclusion: FedPA-LoRA provably converges under homogeneous and heterogeneous client ranks.The convergence guarantee covers both rank configurations.
  • 6 Conclusion: FedPA-LoRA supports client-specific computation and communication budgets.This enables adaptation to differing client resources.
  • 6 Conclusion: Experiments confirm consistent gains across data- and resource-heterogeneous settings.The evaluation covers natural language understanding and generation tasks.

Appendices … C.2 Local Objectives of the Ablation Variants

The appendices provide related-work discussion, the complete FedPA-LoRA algorithm, theoretical extensions, experimental configurations, efficiency analyses, and ablation objectives. They also explain that FedPA-LoRA preserves and aggregates client-specific factors while regularizing the LoRA product directly.

  • B FedPA-LoRA: Detailed Algorithm: Appendix B formalizes FedPA-LoRA’s local preservation, alignment, and aggregation procedures in complete algorithmic pseudocode.The algorithm supports client-specific computation ranks, preserves local factors across rounds, sends client factors to the server, and reconstructs global LoRA factors.
  • C Experimental Details and Extra Results: Appendix C documents hyperparameter configurations, rank settings, detailed results, convergence proofs, partial-participation extensions, regularization schedules, and efficiency evaluations.The appendices cover homogeneous and heterogeneous ranks, bounded reference-truncation error, K active clients per round, λ(t) adaptation, runtime, approximation quality, and client-side wall-clock results.
  • A Additional Discussion of Related Work: Appendix A contrasts FedDPA-LoRA’s global-only transmission and personalized-adapter initialization variants with FedSA-LoRA’s factor asymmetry analysis.FedDPA-F re-initializes the personalized adapter from the freshly aggregated global adapter, whereas FedDPA-T uses its previous-round value; FedSA-LoRA identifies A as task-general and B as client-specific.
  • A Additional Discussion of Related Work: FedPA-LoRA addresses client-specificity by preserving local factors while continuing to include B in the global model, rather than permanently excluding an adapter or factor from federation.This distinguishes its approach from FedDPA-LoRA and FedSA-LoRA, which withhold client-specific components from federation.
  • C.1 Experimental Details: The experiments average results over three random seeds (0, 13, 123), with configurations selected by validation performance for homogeneous- and heterogeneous-rank settings.Experiments use FederatedScope-LLM and PyTorch, and heterogeneous-rank studies use client ranks ri ∈{2, 4, 16} for FlexLoRA, HetLoRA, and FedPA-LoRA.
  • C.1 Experimental Details: FedPA-LoRA’s product-guided regularization strength is searched over λ ∈{0.1, 1.0, 2.0, 5.0, 10.0}, with Figure 4 additionally evaluating λ = 0.01.The learning-rate search uses η ∈{5e-4, 1e-3, 5e-3, 1e-2, 2e-2}.
  • C.2 Local Objectives of the Ablation Variants: The ablation variants share FedPA-LoRA’s task loss and server-side aggregation but differ in regularization space: the first three regularize B, A, or both separately, while FedPA-LoRA regularizes their product.These are the local objectives defined in Appendix C.2.

C.3 Extra Results … D.2.1 Setup and Additional Assumptions

The paper adds rank- and task-specific experimental coverage, while its convergence analysis establishes homogeneous stationarity and extends the framework to heterogeneous local and reference ranks. The heterogeneous analysis uses product-space aggregation, rank-specific references, dense-free projections, and bounded truncation error assumptions.

  • C.3 Extra Results: Additional experiments cover MNLI across LoRA ranks and generation performance on GSM8K and HumanEval under homogeneous- and heterogeneous-rank settings.MNLI uses β = 0.5 and N = 3 clients; the generation settings specify task-specific client counts and data partitions.
  • D.1 Convergence Analysis: Homogeneous Setting: The homogeneous convergence proof assumes equal local and reference ranks, full client participation, and λ > 0, with expectations conditioned on current iterates.The proof develops stochastic and deterministic objectives, lower-bounded task loss, bounded initial objective gap, smoothness, and gradient assumptions.
  • D.1.4 Proof of Theorem 1: FedPA-LoRA’s homogeneous proof derives one-step and one-round descent by expanding factor updates, bounding interaction terms, and accounting for local factor preservation.The analysis then compares product-level regularization before and after server aggregation and telescopes the objective across communication rounds.
  • D.1.4 Proof of Theorem 1: FedPA-LoRA achieves the O((Tτ)^-1/2) averaged stationarity rate stated in Theorem 1 for 0 < η ≤ 1.The proof concludes by telescoping communication-round inequalities and dividing by the accumulated local-update horizon.
  • D.2.1 Setup and Additional Assumptions: The heterogeneous setup averages preserved local products, projects onto rank Rg = maxi Ri, and sends each client its best rank-Ri approximation of the global product.Reduced-QR and core-SVD compute these projections without forming dense matrices, while clients optimize regularized objectives from preserved local products.
  • D.2.1 Setup and Additional Assumptions: The heterogeneous analysis assumes bounded LoRA factors and regularized gradients, together with a bounded truncation error δ for client-specific references.The truncation error is small when the global product is approximately low-rank or reference ranks are sufficiently large; δ = 0 when the relevant singular-value tails vanish.
  • D.2.1 Setup and Additional Assumptions: The heterogeneous proof introduces an ideal local objective using the full global product to analyze mismatch from per-client truncated references.This objective supplies a repeated relation for handling the difference between client-specific references and the global product.

D.2.2 Convergence Result … D.2.5 Proof of Proposition 2

The convergence analysis establishes FedPA-LoRA’s heterogeneous-rank theorem under stated boundedness, smoothness, and lower-loss assumptions, while proving descent through intermediate lemmas and controlling the global LoRA loss via truncation error. The proofs combine local factor preservation, auxiliary full-product objectives, rank-specific projections, and Lipschitz bounds.

  • D.2.2 Convergence Result: Theorem 2 establishes convergence for FedPA-LoRA with heterogeneous client ranks under Assumptions 1, 2, 4, 5, and 6.The theorem assumes a uniform lower bound Ψ⋆ on average task loss and an initial gap bounded by D.
  • D.2.2 Convergence Result: FedPA-LoRA retains the homogeneous case’s O averaged stationarity rate up to an additive constant governed by δ.The theorem introduces constants independent of T and τ for step sizes satisfying 0 < η ≤ 1.
  • D.2.5 Proof of Proposition 2: Proposition 2 bounds the global LoRA loss by combining client-wise Lc-Lipschitz continuity, the triangle inequality, and the truncation error from Assumption 6.Averaging the client inequalities yields the stated relation to Ψ(t).
  • D.2.3 Preliminary Lemmas: The preliminary lemmas establish uniform boundedness, smoothness, and bounded stochastic factor gradients across rounds, clients, and local steps.The actual and ideal objectives are L-smooth, while factor-gradient bounds follow from bounded factors and products.
  • D.2.3 Preliminary Lemmas: Lemma 2 converts one local SGD update on LoRA factors into a weight-space descent inequality for the ideal objective.Its proof controls expected inner-product and stochastic-interaction terms using smoothness, unbiased gradients, factor bounds, and η ≤ 1.
  • D.2.4 Proof of Theorem 2: Theorem 2’s proof applies Lemma 2 over local steps, introduces an auxiliary full-global-product objective, and uses truncated rank-specific references to derive round-wise descent.The argument explicitly preserves local factors and uses the best rank-Rg approximation in the projection step.
  • D.2.4 Proof of Theorem 2: The proof bounds the initial-to-final auxiliary-objective gap by eD = D + LcCACB using the lower loss bound and bounded global products.This controls the cumulative descent after summing the round-wise inequality over t = 1, . . . , T.
  • D.2.4 Proof of Theorem 2: The final theorem argument relates actual and ideal local gradients, then combines η- and δ-dependent terms through constants M̄λ and M̃λ.The constants are set as M̄λ = 8C2 and M̃λ = λ(8C1 + 2c).

E Convergence with Partial Participation · F Adaptive Regularization Strength

FedPA-LoRA extends its convergence guarantee to uniformly sampled partial participation, where inactive clients retain their factors and the convergence bound includes a participation-dependent term. The paper also motivates client- and round-specific regularization weights because suitable regularization may vary with data heterogeneity.

  • E Convergence with Partial Participation: Theorem 3 establishes convergence under partial participation when a uniformly sampled subset of K clients is active each round.The guarantee assumes smooth, Lipschitz local losses, unbiased bounded stochastic gradients, a gradient-preservation property, and a lower-bounded average task loss.
  • E Convergence with Partial Participation: Inactive clients perform no local updates and retain their factors, while the server reconstructs a rank-constrained global update from the sampled clients’ average product.The sampled subset is drawn independently of mini-batch noise and prior history, enabling the virtual local trajectory used in the analysis.
  • E Convergence with Partial Participation: When K = N, the additional participation term vanishes and the convergence bound reduces exactly to the full-participation bound.The proof obtains the result by telescoping the expected objective decrease over T rounds.
  • E Convergence with Partial Participation: The proof applies one-round descent to the sampled joint objective and uses unbiased mini-batch gradients and uniform client sampling to derive the partial-participation bound.Non-participating clients’ task-loss terms cancel because their factors remain unchanged.
  • E Convergence with Partial Participation: For K < N, convergence is guaranteed only to a neighborhood of stationarity, whose radius is governed by K/N and B.The authors note that tighter sampling-concentration analysis might reduce or remove this term.
  • F Adaptive Regularization Strength: The shared regularization weight λ is selected offline by grid search, but its suitable strength may vary with data heterogeneity.Near-IID data may need weaker regularization, whereas non-IID data may require stronger regularization to control client drift.
  • F Adaptive Regularization Strength: This motivates client- and round-specific weights λ(t) based on quantities available locally, presented as illustrative directions rather than a definitive adaptive rule.The paper discusses three possible alternatives without selecting one as the final method.

F.1 Task-loss-based adaptation … G.1.2 Randomized Server-Side Reconstruction

The paper proposes adaptive regularization based on client loss, gradient magnitude, or gradient direction, then analyzes exact and randomized server-side reconstruction for efficient product-space aggregation. Randomized sketches reduce client-count dependence from quadratic to linear while trading exact optimality for an in-expectation guarantee.

  • F.1 Task-loss-based adaptation: Task-loss adaptation weakens regularization for clients with larger initial loss and strengthens global guidance when the loss is smaller.The resulting weight remains bounded by λ0, but behavior can depend on loss scale and the sampled mini-batch.
  • F.2 Gradient-aware drift adaptation: Gradient-aware adaptation adjusts regularization according to estimated local update magnitude relative to the client’s product deviation from the global reference.The adaptive weight is clipped within λmin and λmax, though robustness may be limited by small factor norms and sampled variability.
  • F.3 Gradient-direction-based adaptation: Gradient-direction adaptation decreases regularization when task and product-regularization gradients agree, and increases it when their cosine similarity is small or negative.With stabilization, the weight lies strictly within (0, 2λ0).
  • F.3 Gradient-direction-based adaptation: These adaptive alternatives use global hyperparameters while dynamically changing effective regularization from each client’s local loss or gradient state.For example, a larger local loss permits greater local adaptation, whereas a smaller loss provides stronger global guidance.
  • G Complexity Analysis: The complexity analysis considers N clients, d × d weights, homogeneous rank r = Rg, concatenated rank rtot = Nr, and the low-rank regime rtot ≪ d.Server memory analysis assumes all client uploads remain available during aggregation.
  • G.1.1 Server-Side Asymptotic Complexity: FedPA-LoRA avoids the dense d × d aggregate by applying reduced QR to concatenated factors and SVD to an Nr × Nr core.Under the homogeneous setting, its dominant costs are O(N^2dr^2) computation and O(Ndr) memory, while recovering the dense aggregate’s optimal rank-r approximation.
  • G.1.2 Randomized Server-Side Reconstruction: Randomized reconstruction sketches the factored aggregate with a Gaussian test matrix, finds its range, and performs rank-Rg truncated SVD in the sampled subspace.With Rg + p = O(r), it avoids both the dense aggregate and the full rtot-dimensional basis.
  • G.1.2 Randomized Server-Side Reconstruction: O(Ndr^2) computation replaces O(N^2dr^2) when Rg + p = O(r), trading the exact optimality guarantee for an in-expectation bound.Setting Rg + p = rtot instead captures the column space almost surely and recovers the exact reduced-QR/core-SVD result, with O(N^2dr^2) dominant computation.

G.1.3 Server-Side Wall-Clock Evaluation · G.1.4 Effect of the Sketch Dimension

The evaluation shows that factor-wise averaging has the lowest server-side aggregation cost, while randomized reconstruction offers tunable lower-cost operating points. Across sketch dimensions, FedPA-LoRA remains robust in downstream MNLI accuracy while reducing server-side computation.

  • G.1.3 Server-Side Wall-Clock Evaluation: Factor-wise averaging incurs the lowest server-side aggregation cost, whereas dense product aggregation and SVD become substantially more expensive as model dimensions grow.The wall-clock evaluation uses RoBERTa-Large and Llama 3-8B under homogeneous ranks in the same environment.
  • G.1.3 Server-Side Wall-Clock Evaluation: Table 11 evaluates average per-round server-side aggregation wall-clock time for RoBERTa-Large with N = 10, Rg = 4 and Llama 3-8B with N = 6, Rg = 8.These settings yield rtot = 40 and rtot = 48, respectively, with randomized reconstruction using ℓ = Rg + p.
  • G.1.3 Server-Side Wall-Clock Evaluation: SVD avoids a full d × d decomposition by operating only on the smaller core matrix.This reduces the decomposition burden relative to applying SVD directly to the dense product aggregate.
  • G.1.3 Server-Side Wall-Clock Evaluation: Randomized reconstruction provides lower-cost server operating points through the sketch dimension ℓ, with wall-clock time generally increasing as ℓ approaches the full concatenated rank rtot.The evaluation also includes p = 0 as an empirical minimum-sketch configuration, although the expectation bound in Remark 2 has additional requirements.
  • G.1.4 Effect of the Sketch Dimension: The sketch-dimension study evaluates MNLI with N = 3 clients, r = Rg = 4, β = 0.5, and ℓ ∈ {4, 5, 6, 8, 12}.The corresponding oversampling values are p ∈ {0, 1, 2, 4, 8}, and ℓ = 12 = rtot uses the full concatenated rank.
  • G.1.4 Effect of the Sketch Dimension: Figure 6 includes p = 0 and p = 1 empirically to assess further cost reductions, despite Remark 2 requiring p ≥ 2 for its bound.The evaluated sketch dimensions correspond to ℓ = Rg + p ∈ {4, 5, 6, 8, 12}.
  • G.1.4 Effect of the Sketch Dimension: FedPA-LoRA remains robust to randomized approximation across sketch dimensions, preserving downstream accuracy while reducing server-side computation.Figure 6 reports MNLI accuracy for the homogeneous-rank setting and compares results against baseline accuracies averaged over three random seeds.
  • G.1.4 Effect of the Sketch Dimension: Client-side wall-clock time is averaged across participating clients within each round and then across rounds, excluding the first round’s one-time initialization overhead.This reporting procedure is specified for the client-side timing results in Table 12.

G.2 Client-Side Wall-Clock Evaluation

FedPA-LoRA adds client-side computation through product-space regularization, with modest overhead on RoBERTa-Large but substantially higher relative overhead on Llama 3-8B. It remains less expensive than FedDPA-LoRA on RoBERTa-Large, while the larger-model cost is a limitation.

  • Evaluation setup: The evaluation averages client-side wall-clock time across participating clients within each round and then across rounds for RoBERTa-Large and Llama 3-8B.Strategies include single-factor and full LoRA updates, FedRot-LoRA, FedDPA-LoRA, and FedPA-LoRA.
  • RoBERTa-Large: 9.4603 seconds: FedPA-LoRA incurs a 2.8% overhead over the full LoRA update on RoBERTa-Large and remains less expensive than FedDPA-LoRA.FedRot-LoRA and full LoRA require 9.2121 and 9.2053 seconds, respectively, while FedDPA-LoRA requires 10.3431 seconds.
  • Llama 3-8B: 5.1343 seconds: FedPA-LoRA represents a 34.0% increase over the full LoRA update on Llama 3-8B, with an absolute overhead of about 1.30 seconds per client and round.FedRot-LoRA and FedDPA-LoRA require 3.9323 and 3.8371 seconds, respectively.
Loading 2608.15381v1…