PaperlayerView exact revision on arXiv

Source-linked AI summary

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Junkang Liu

arXiv:2610.01620v1cs.AI

TL;DR

Federated foundation-model training faces client-memory and communication costs, while fixed-rank adapters can limit accumulated updates. FedLore shares and refreshes a low-rank basis for exact aggregation and reduced projection bias. Across vision and language tasks, including federated pre-training, it outperforms evaluated low-rank adapter baselines and matches or exceeds full-parameter training while reducing communication and optimizer-state memory.

  • Problem

    Federated foundation-model training must limit client optimizer-state memory and repeated high-dimensional update exchange, while standard LoRA's accumulated updates remain rank-constrained.

    Source: INTRODUCTION · p. 1

  • Method

    FedLore shares a low-rank optimization basis within each round and refreshes it across rounds, enabling exact coordinate aggregation and removing projection–heterogeneity coupling.

    Source: INTRODUCTION · pp. 2–3

  • Results

    FedLore outperforms evaluated low-rank adapter baselines and matches or exceeds full-parameter training across vision, language, and federated pre-training; its language benchmark average accuracy is 90.44%.

    Source: INTRODUCTION · p. 3, RESULTS ON FINE-TUNING · p. 8

  • Takeaways & Limitations

    A shared, refreshed basis permits accumulated updates beyond rank r while retaining low-rank communication and optimizer-state memory.

    Source: INTRODUCTION · p. 3

    Experiments use benchmark partitions simulating federated clients rather than data collected from real users, and FedLore establishes no formal privacy guarantee.

    Source: ETHICS STATEMENT · p. 13

Abstract

from arXiv · show

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term \emph{subspace fragmentation}: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose \texttt{FedLore}, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an $O(T^{-1/2})$ stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that \texttt{FedLore} outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

INTRODUCTION

Federated foundation-model training must manage client memory and communication costs, while fixed-rank adapters can constrain accumulated updates. FedLore shares and refreshes a low-rank basis to address subspace fragmentation and enable exact coordinate aggregation.

Motivation

  • Federated training also faces coupled costs: adaptive optimizer states must fit on clients, and high-dimensional model updates must be repeatedly exchanged.Full-parameter federated optimization incurs substantial costs on both fronts.Source: INTRODUCTION · p. 1

Problem

  • Client-specific projections can interact with gradient heterogeneity, suppressing useful global directions even when local gradient compression is accurate.The paper calls this mismatch subspace fragmentation; the coupling vanishes with shared projectors.Source: INTRODUCTION · p. 2
  • Aggregating rank-r updates from distinct client subspaces can produce rank up to min{m, n, Nr}, increasing exact communication costs.Local updates may each remain low-rank while their aggregate does not.Source: INTRODUCTION · p. 2

Approach

  • FedLore shares optimization coordinates within each round and refreshes the basis across rounds, enabling exact aggregation and accumulated changes beyond one rank-r correction.The basis is constructed from leading eigenspaces of averaged candidate projectors.Source: INTRODUCTION · pp. 2–3, Figure 2 · p. 2

Contributions

  • The paper establishes an O(T^-1/2) stationarity bound for projected SGD under smoothness, variance, bounded-heterogeneity, and global-gradient-coverage assumptions.Agreement from shared projections alone does not ensure global-gradient coverage.Source: INTRODUCTION · p. 3

RELATED WORK

Related work covers parameter-efficient fine-tuning and federated LoRA methods. FedLore differs by optimizing projected gradients in a shared, evolving basis.

Parameter-Efficient Fine-Tuning for Large Models

  • LoRA reduces gradient and optimizer-state memory with trainable low-rank factors, but accumulated updates remain rank-constrained, potentially limiting adaptation under distribution shift or from-scratch training.Adaptive rank allocation and alternative parameterizations extend standard LoRA (Zhang et al., 2023b; Liu et al., 2024b).Source: RELATED WORK · p. 3

Extending LoRA to Federated Learning

  • Federated LoRA methods refine aggregation, initialization, or factor sharing, while FedLore uses projected gradients in a shared, evolving basis for exact low-rank-coordinate aggregation.Examples include FedIT (Zhang et al., 2023a), FLoRA, and LoRA-FAIR.Source: RELATED WORK · p. 3

WHY LOCAL GRADIENT PROJECTION FAILS IN FEDERATED LEARNING

The section moves from federated optimization setup to local gradient projection, then explains why client-specific projections can impair descent and exact low-rank communication.

WHY LOCAL GRADIENT PROJECTION FAILS IN FEDERATED LEARNING

  • Federated optimization minimizes a global objective built from client losses, with each client loss defined over its local data distribution.The setup identifies the client count and global model alongside the local losses.Source: WHY LOCAL GRADIENT PROJECTION FAILS IN… · p. 3

FL WITH GRADIENT LOW-RANK PROJECTION IN LOCAL

  • Local GaLore constructs a rank-r basis from the leading singular components of a client's gradient, projects gradients into that basis, and maintains optimizer states in projected coordinates.The client uploads its accumulated model update after local steps.Source: FL WITH GRADIENT LOW-RANK PROJECTION… · pp. 3–4

CHALLENGES OF GRADIENT LOW-RANK PROJECTION IN FL

  • Heterogeneous clients can select different subspaces, coupling projection disagreement with gradient heterogeneity and weakening global descent; client-specific updates can also aggregate to rank above r.Shared projections eliminate the coupling term, while global descent additionally requires sufficient gradient coverage; distinct client subspaces can also make exact communication costly.Source: CHALLENGES OF GRADIENT LOW-RANK… · p. 4

FEDERATED TRAINING VIA SHARED GRADIENT LOW-RANK PROJECTION

FedLore uses a shared, evolving gradient subspace: clients optimize and communicate in low-rank coordinates, while the server aggregates updates and constructs the next basis.

Overview

  • FedLore maintains full model weights while storing optimizer states and communicating updates in low-rank coordinates; refreshing the shared subspace lets accumulated updates exceed rank r.The described algorithm assumes full participation and a constant learning rate.Source: FEDERATED TRAINING VIA SHARED GRADIENT… · p. 4

Shared-subspace construction

  • The server forms client candidate bases from leading left singular vectors of the last local gradients and aggregates their projectors to construct the next shared basis.Projector aggregation is invariant to sign changes and within-subspace rotations of candidate bases.Source: FEDERATED TRAINING VIA SHARED GRADIENT… · p. 5

Local optimization in shared coordinates

Exact low-rank aggregation and synchronization

  • With a fixed within-round basis, coordinate aggregation exactly matches endpoint averaging without extra compression; under full participation, communication costs r(m+n) scalars per matrix per client per round in each direction.The two optimizer moments occupy 2rn scalars, excluding weights, gradients, bases, and activations.Source: FEDERATED TRAINING VIA SHARED GRADIENT… · p. 6

THEORETICAL ANALYSIS

The analysis identifies projection disagreement as a source of aggregation bias under heterogeneous client gradients. Shared projections remove that bias, with an O(T^-1/2) stationarity guarantee for projected SGD under the stated assumptions, including global-gradient coverage.

Assumptions

  • The analysis assumes differentiable L-smooth local objectives, a lower-bounded global objective, bounded heterogeneity, and conditionally uncorrelated stochastic-gradient noise.It compares projected-SGD variants under the stated client-sampling and conditioning convention.Source: THEORETICAL ANALYSIS · p. 6

Shared-projection convergence

  • With a shared projector, the coupling bias vanishes and FedLore-SGD’s averaged squared gradient norm is O(T^-1/2) under the stated assumptions.The guarantee assumes global-gradient coverage alongside smoothness, variance, and bounded heterogeneity conditions.Source: THEORETICAL ANALYSIS · p. 6

Projection disagreement

  • Client-specific projections couple projection disagreement with gradient heterogeneity, and exact local SVD projections can stall at a nonstationary point.The paper distinguishes this possible obstruction from an unavoidable error floor implied merely by a nonvanishing upper-bound term.Source: THEORETICAL ANALYSIS · p. 7

Coverage condition

  • Shared projections control projected stationarity without coverage, but full-gradient stationarity additionally requires control of the uncovered gradient component.The full-gradient bound uses a bound on the projected global gradient at every round.Source: THEORETICAL ANALYSIS · p. 7

EXPERIMENTS

FedLore is evaluated on vision and language benchmarks and federated LLaMA pre-training. It achieves strong benchmark results, while ablations examine how rank, subspace sharing, refresh, communication, and optimizer memory relate to performance.

EXPERIMENTS SETTINGS

  • Experiments cover vision classification, C4 pre-training, and language understanding, with results averaged over five runs.The reported benchmarks use Dirichlet client partitions to simulate data heterogeneity.Source: EXPERIMENTS SETTINGS · pp. 7–8

RESULTS ON LLAMA PRE-TRAINING

  • FedLore lowers LLaMA-60M validation PPL from 6707.62 with FedIT to 46.20 and beats FedFull on LLaMA-1B, 28.36 versus 35.20.Table 3 compares low-rank federated algorithms on C4 pre-training and reports validation loss, validation perplexity, communication cost, and local training time.Source: Table 3 · p. 8, RESULTS ON LLAMA PRE-TRAINING · p. 9

RESULTS ON FINE-TUNING

  • FedLore achieves 90.09% average accuracy on the vision benchmarks, compared with 89.99% for FedFull, and is best or statistically competitive across all six dataset combinations.The reported advantages include 90.55% on CIFAR-100 and 87.11% on Tiny-ImageNet for ViT-Base.Source: RESULTS ON FINE-TUNING · p. 8
  • On six language tasks, FedLore achieves 90.44% average accuracy, 0.26 points above FedFull and more than 1 point above the best LoRA-based baseline.The passage reports that FedLore ranks first on all six tasks.Source: RESULTS ON FINE-TUNING · p. 8

ABLATION STUDY

  • Removing the shared projector raises validation loss from 3.554 to 3.890 and communication from 0.36 to 0.68 GB; fixing the subspace raises loss to 5.952.The reported C4 pre-training ablation supports complementary roles for sharing and subspace refresh.Source: Table 4 · p. 9, ABLATION STUDY · p. 9
  • FedLore uses 0.36 GB communication and 0.54 GB optimizer memory, about 47% lower communication than FedFull or Local GaLore.The comparison is for LLaMA 350M; Local GaLore uses 0.54 GB optimizer memory but communicates full updates.Source: Table 4 · p. 9, ABLATION STUDY · p. 9
  • On LLaMA-350M, larger projection ranks accelerate convergence and reduce final validation loss, with marginal gains from r = 256 to r = 384.The reported improvement is significant from r = 64 to r = 256.Source: ABLATION STUDY · p. 9

CONCLUSION

FedLore is a communication- and memory-efficient framework for federated foundation-model training. Its shared, evolving basis supports exact low-rank aggregation, while the paper analyzes projection–heterogeneity coupling and projected-SGD convergence.

CONCLUSION

  • FedLore uses a shared, evolving basis for exact low-rank aggregation without fixing the rank of accumulated updates.The paper characterizes projection–heterogeneity coupling and establishes projected-SGD convergence under global-gradient coverage.Source: CONCLUSION · p. 9
  • Experiments across vision, language, and federated pre-training support the approach.Source: CONCLUSION · p. 9
  • Future work will develop coverage-aware subspace selection and efficient synchronization for intermittent participation.Source: CONCLUSION · p. 9

ETHICS STATEMENT

The study evaluates FedLore on existing vision and language benchmarks using simulated federated clients. It does not establish formal privacy guarantees, and sensitive-data deployment requires additional safeguards.

ETHICS STATEMENT

  • Experiments use existing vision and language benchmarks, with federated clients simulated from benchmark partitions rather than real-user data.The benchmarks include CIFAR-100, Tiny-ImageNet, Food-101, C4, SNLI, AG News, QQP, DBPedia-14, QNLI, and MNLI.Source: ETHICS STATEMENT · p. 13
  • FedLore establishes neither differential privacy nor resistance to inference attacks; communicated updates and shared subspaces may still expose local-data information.Federated training and low-rank projection alone provide no formal privacy guarantee.Source: ETHICS STATEMENT · p. 13
  • Sensitive-data deployment requires additional privacy safeguards and application-specific evaluation because models may inherit dataset or pretrained-model biases and harmful content.Source: ETHICS STATEMENT · p. 13

REPRODUCIBILITY STATEMENT

The paper documents the algorithm, theory, and experimental configurations across the main text and appendices. It reports hardware and repeated-experiment seeds and states that implementation materials will be released upon acceptance.

REPRODUCIBILITY STATEMENT

  • The paper provides the FedLore algorithm and theoretical assumptions, with detailed derivations and proofs in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 13
  • Experimental descriptions cover datasets, architectures, client partitions, participation, communication rounds, local steps, projection ranks, optimizer settings, and hyperparameter selection.Source: REPRODUCIBILITY STATEMENT · p. 13
  • The paper reports hardware and five random seeds for repeated experiments, and says implementation and experiment scripts will be released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 13

AI USE STATEMENT

Large language models assisted with language editing, not research design, implementation, or results interpretation. The authors reviewed the assisted text and take responsibility for the manuscript.

AI USE STATEMENT

  • Large language models assisted with rephrasing, grammar correction, and readability improvements.Source: AI USE STATEMENT · p. 13
  • The tools were not used for research ideation, methodology or experiment design, derivations, implementation, data processing, or results interpretation.Source: AI USE STATEMENT · p. 13
  • The authors reviewed and revised all AI-assisted text and take responsibility for the manuscript’s claims, analyses, and conclusions.Source: AI USE STATEMENT · p. 13

APPENDIX CONTENTS

The appendix contents list experimental details for pre-training and fine-tuning, including a vision benchmark and hyperparameter selection.

Details of Pre-Training Experiments

  • Pre-training details are organized around architecture and hyperparameters, federated setup, optimization, communication and memory, hardware and reproducibility, and learning-rate selection.Source: APPENDIX CONTENTS · p. 14

Details of Fine-Tuning Experiments

  • Fine-tuning contents include a vision benchmark, hyperparameter selection, and learning-rate selection.Source: APPENDIX CONTENTS · p. 14

Language Benchmark .

The listed material spans language-benchmark tuning, communication and memory overhead, shared projections in federated optimization, and a baseline algorithm.

Language Benchmark

Communication and Memory Overhead

Shared Projections in Federated Optimization

  • The shared-projections material lists assumptions, projection disagreement and aggregation bias, and convergence under partial participation.Source: Language Benchmark · p. 14
  • Additional listed topics cover exact local SVD projections, coverage residuals, approximate shared-subspace coverage, synchronization and optimizer scope, and the baseline algorithm.Source: Language Benchmark · p. 14

DETAILS OF PRE-TRAINING EXPERIMENTS

This section specifies FedLore’s federated pre-training and fine-tuning setups, synchronization procedure, and Local GaLore baseline, and analyzes communication and optimizer-memory costs alongside shared and client-specific projections. It develops convergence, gradient-coverage, and residual-certificate analyses under stated assumptions, and describes FedLore-Random’s shared-basis training procedure without reporting pre-training results.

ARCHITECTURE AND HYPERPARAMETERS

HYPERPARAMETER SELECTION

VISION BENCHMARK

  • Fine-tuning experiments use heterogeneous client partitions: vision benchmarks cover CIFAR-100, Tiny-ImageNet, and Food-101, while language benchmarks cover six text-classification datasets.Vision uses Swin-Base and ViT-Base with Dirichlet α = 0.1, 50 clients, and 100 rounds; language uses RoBERTa-Base with Dirichlet α = 0.5, 20 clients, and 100 rounds.Source: VISION BENCHMARK · p. 17

COMMUNICATION COST

  • FedFull communication costs O(2mn), versus O(2r(m + n)) for LoRA; FedLore communicates projected updates and has complexity comparable to LoRA-based methods.For W ∈ R^(m×n), FedLore sends the projected update while the server maintains the shared projection matrix.Source: COMMUNICATION AND MEMORY ANALYSIS · p. 16, COMMUNICATION COST · p. 19

MEMORY ESTIMATES

  • FedLore retains full-parameter model weights while maintaining optimizer states only in the shared projected space, reducing optimizer memory relative to full-parameter training.Memory estimates assume BF16, with 2 bytes per parameter and two AdamW moment states for trainable parameters.Source: MEMORY ESTIMATES · p. 19

SHARED PROJECTIONS IN FEDERATED OPTIMIZATION

  • The analysis identifies client-specific projector coupling with gradient heterogeneity, while shared projectors permit a sharper projected-stationarity analysis without a coverage assumption and expose uncovered gradients as a residual.It bounds that residual for FedLore’s candidate-projector average; exact coordinate aggregation is established for adaptive updates, but stationarity results do not analyze coordinatewise Adam.Source: SHARED PROJECTIONS IN FEDERATED… · p. 20

PROJECTION DISAGREEMENT AND AGGREGATION BIAS

  • For every zero-mean collection of client gradient perturbations, aggregation bias vanishes if and only if all client projectors coincide.The analysis also characterizes the bias’s directional effect on descent; for a shared projector, the directional-bias threshold is zero.Source: PROJECTION DISAGREEMENT AND… · pp. 22–23

CONVERGENCE UNDER PARTIAL PARTICIPATION

  • Projected-SGD convergence separates projection bias, client sampling, oracle noise, and local drift, with full-gradient convergence requiring sufficient coverage of the global gradient.The bound assumes smoothness, bounded gradient heterogeneity, and the stated coverage condition; sampling and accumulated oracle noise have separately identified contributions.Source: SETTING AND ASSUMPTIONS · p. 21, CONVERGENCE UNDER PARTIAL PARTICIPATION · pp. 23–24

AUXILIARY ESTIMATES

  • Local gradient and noise sums may be correlated, so the auxiliary estimates avoid assuming independence and use martingale-difference structure to control noise cross terms.The passages explicitly note that the sums are generally correlated; the noise-sum cross terms vanish by the martingale-difference property.Source: AUXILIARY ESTIMATES · p. 25

PROOF OF THE CONVERGENCE THEOREM

A SHARPER BOUND FOR SHARED PROJECTIONS

  • Shared projections yield a bound on projected stationarity without a lower bound on global-gradient coverage, while the residual tracks the missing full-gradient component.Without coverage, the result is a residual bound, not an unconditional full-stationarity guarantee; expected coverage suffices for the stated coverage conclusion.Source: A SHARPER BOUND FOR SHARED PROJECTIONS · pp. 27–28

CONSEQUENCES FOR SHARED AND CLIENT-SPECIFIC PROJECTIONS

  • Under expected coverage, shared projections achieve O(T −1/2) convergence for fixed problem parameters, K, S, c, and µ; the admissible stepsize is independent of µ, while the coverage factor enters as 1/µ.This improvement uses the common subspace of the entire server direction and does not automatically extend to client-specific projections.Source: CONSEQUENCES FOR SHARED AND… · p. 29

A STRICT SEPARATION WITH EXACT LOCAL SVD PROJECTIONS

  • With rank 1 and full participation, exact local leading-SVD projections can stall at W=0 with squared global-gradient norm 1/2 despite retaining 4/5 of each local gradient’s energy, while shared projections reach the minimizer in two rounds at γ=1.The example uses exact gradients and any fixed K≥1; its shared comparator selects a leading singular subspace of the global gradient.Source: A STRICT SEPARATION WITH EXACT LOCAL… · pp. 30–31

COVERAGE RESIDUAL OF THE IMPLEMENTED PROJECTOR AVERAGE

  • For FedLore’s projector average, the residual certificate analyzes the implemented shared basis from candidate projectors, and an exact leading eigenspace has εeig=0.The shared basis is obtained from candidate projectors rather than the SVD of an averaged gradient.Source: COVERAGE RESIDUAL OF THE IMPLEMENTED… · p. 31
  • The implemented projector-average certificate separates truncation, subspace disagreement, and gradient mismatch, but does not guarantee unconditional O(T^-1/2) stationarity when residuals persist.Agreement alone can fail to provide coverage: exact local projectors may agree while selecting a direction with zero coverage at a nonstationary point.Source: COVERAGE RESIDUAL OF THE IMPLEMENTED… · pp. 32–33

COVERAGE OF AN APPROXIMATE SHARED SUBSPACE

  • A leading rank-r projector of the global gradient maximizes the certified smoothness-bound decrease, with retained fraction at least r/m; approximate estimates preserve coverage under bounded relative error.With uniform true-gradient coverage μ⋆ and estimation error ϵ̄ < √μ⋆/2, the shared method has coverage μ = (√μ⋆−2ϵ̄)^2. Under additive estimation error, average squared error of order T^-1/2 suffices to retain the stationarity rate; fixed error contributes an explicit average penalty.Source: COVERAGE OF AN APPROXIMATE SHARED… · pp. 33–35

EXACT SYNCHRONIZATION AND OPTIMIZER SCOPE

  • With a fixed shared basis and common decay, clients can aggregate endpoint updates exactly in low-rank coordinates, using r(m+n) communicated scalars per direction per round under full participation.The two coordinate moment tensors use 2rn scalars, not a total peak-memory bound; Adam synchronization does not eliminate bias from independently normalized directions, and stationarity results concern projected SGD without weight decay.Source: EXACT SYNCHRONIZATION AND OPTIMIZER… · pp. 35–36

C4 PARTITIONING PROTOCOL

  • C4 clients are partitioned by normalized hostname source groups as a proxy for content and writing-style differences, with source metadata excluded from model inputs.Twenty clients participate in every round with 50 local steps and batch size 16; documents are packed per client into length-256 sequences, and all methods share each seed's partition and validation corpus.Source: C4 PARTITIONING PROTOCOL · pp. 36–37

BASELINE ALGORITHM

  • The federated Local GaLore baseline constructs a local projector from a separate round-start gradient batch, then performs low-rank Adam updates and aggregates client updates in ambient space.The round-start construction batch is independent of subsequent local optimization draws.Source: BASELINE ALGORITHM · p. 38

RELATED WORK

  • Related work addresses communication or solution alignment, whereas this paper focuses on gradient-evaluation geometry; cited examples include FedBCGD, FedSWA and FedMoSWA, and FedNSAM.CTP-FL uses a shared server-defined predictive trajectory so clients evaluate gradients at common points, allowing their average to estimate the global-objective gradient there.Source: RELATED WORK · p. 38
  • FedLore-Random initializes a shared rank-r basis from a server-sampled seed that clients use to reconstruct the same basis.The procedure specifies global weights, learning rate, scale, Adam coefficients, communication rounds, local steps, clients, and weight decay as inputs.Source: RELATED WORK · p. 40
  • At each later round, the server samples a fresh seed independently of training randomness, broadcasts it with the previous update, and clients synchronize using that update.Clients reconstruct the new shared basis from the seed; the excerpt shows model synchronization using the previous basis and update.Source: RELATED WORK · p. 40
  • The procedure begins local projected Adam updates by sampling a fresh mini-batch at each local step.Source: RELATED WORK · p. 40

Related papers

Loading 2610.01620v1…