PaperlayerView exact revision on arXiv

Source-linked AI summary

FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning

Nam Hyeon-Woo, Moon Ye-Bin, Tae-Hyun Oh

arXiv:2108.06098v3cs.LGcs.CV

TL;DR

Federated learning repeatedly transfers models under heterogeneous, bandwidth-constrained conditions. The paper proposes FedPara, which combines low-rank factors through a Hadamard product, and extends it to personalized FL as pFedPara. FedPara achieves comparable or better performance with substantially lower communication, while pFedPara improves personalized FL robustness and parameter efficiency.

  • Problem

    Federated learning still transfers substantial data during training, burdening edge devices and low-quality communication infrastructure.

  • Method

    FedPara re-parameterizes fully connected and convolutional layers as Hadamard products of low-rank inner matrices, while pFedPara separates global and local parameters.

  • Results

    2.8 to 10.1 times fewer communication costs and energy consumption are required by VGG16FedPara than VGG16ori. for the same target accuracy.

  • Takeaways & Limitations

    FedPara enables communication-efficient FL without a low-rank limitation, and pFedPara provides robust personalized performance against competing methods.

  • Takeaways & Limitations

    FedPara is slower than original parameterization and low-rank approaches because training requires additional computation to re-compose the original weight structure.

Abstract

from arXiv · show

In this work, we propose a communication-efficient parameterization, FedPara, for federated learning (FL) to overcome the burdens on frequent model uploads and downloads. Our method re-parameterizes weight parameters of layers using low-rank weights followed by the Hadamard product. Compared to the conventional low-rank parameterization, our FedPara method is not restricted to low-rank constraints, and thereby it has a far larger capacity. This property enables to achieve comparable performance while requiring 3 to 10 times lower communication costs than the model with the original layers, which is not achievable by the traditional low-rank methods. The efficiency of our method can be further improved by combining with other efficient FL optimizers. In addition, we extend our method to a personalized FL application, pFedPara, which separates parameters into global and local ones. We show that pFedPara outperforms competing personalized FL methods with more than three times fewer parameters.

1 INTRODUCTION

Federated learning must reduce repeated communication under heterogeneous data and systems. FedPara addresses this bottleneck by re-parameterizing layers with low-rank Hadamard products while preserving capacity, and pFedPara extends the approach to personalization.

  • Motivation: Heterogeneous data and systems create convergence challenges and substantial communication burdens for federated learning.Client data are non-IID and unbalanced, while connectivity may be slow, costly, or unavailable.
  • FedPara: FedPara re-parameterizes fully connected and convolutional layers into a small factorized form using a low-rank Hadamard product.The method reduces bits transferred per round without changing the optimization procedure.
  • FedPara: FedPara can span full-rank matrices and tensors despite using fewer parameters than conventional low-rank parameterization.Its Hadamard-product structure avoids the usual low-rank restriction.
  • Personalized FL: pFedPara separates global and local inner matrices to learn shared and client-specific knowledge in personalized federated learning.The paper reports robustness and performance improvements against competing personalized FL algorithms.

2 METHOD

The method replaces conventional low-rank structures with Hadamard products of low-rank factors, achieving higher expressiveness for fully connected and convolutional layers. It also defines a global-local personalized variant for heterogeneous clients.

  • Conventional low-rank parameterization: Conventional low-rank decomposition reduces matrix parameters from O(mn) to O((m + n)r) but constrains the representation to rank r.Tensor decompositions such as Tucker and CP provide analogous parameter reductions for convolutional kernels.
  • Conventional low-rank parameterization: Post-decomposition compression does not reduce federated training communication because original-size parameters are transferred before decomposition.This motivates parameterizing the trainable model in reduced form during FL itself.
  • FedPara: FedPara constructs each layer as the Hadamard product of two low-rank inner matrices, trained from scratch across fully connected and convolutional layers.The approach redefines layer internal structure without changing the optimization component of FL.
  • FedPara: When inner ranks satisfy r1r2 ≥ min(m, n), FedPara has no low-rank restriction and can span a full-rank matrix with high chance.With r1 = r2 = R, Proposition 2 gives the parameter count 2R(m + n) under the target-rank criterion.
  • Convolutional layers: For convolutional layers, Proposition 3 applies without reshaping and requires 3.8 times fewer parameters than Proposition 1 in the Table 1 example.The method uses tensor structure directly because it is more effective for common convolutional models.
  • Personalized FL: pFedPara constructs personalized layers as W = W1 ⊙(W2 + 1), with W1 global and W2 local to each client.The global component is transferred and shared, while the local component remains private.

3 EXPERIMENTS

Experiments evaluate FedPara across datasets, architectures, communication costs, parameter ratios, optimizer combinations, and personalized FL settings. Results show comparable accuracy with lower communication costs, stronger capacity than conventional low-rank parameterization, compatibility with other optimizers, and favorable personalized performance.

  • Setup: Experiments use CIFAR-10, CIFAR-100, CINIC-10, Shakespeare, VGG, ResNet, LSTM, and multilayer perceptron models under IID and non-IID settings.FedPara evaluations use FedAvg as the backbone optimizer except in compatibility experiments.
  • Setup: FedPara uses a rank ratio γ to control parameter counts, with ranks interpolated between the minimum full-rank-enabling value and the maximum original-model-equivalent value.The same γ is fixed across layers for simplicity, although it can be tuned independently per layer.
  • Capacity: FedPara surpasses conventional low-rank parameterization across the reported IID and non-IID benchmarks using comparable parameter budgets.Table 2 compares VGG16FedPara and VGG16low at approximately 10% of VGG16ori. parameters and also reports the corresponding LSTM comparison.
  • Communication Cost: 10.1%, 29.4%, and 21.8% of VGG16ori. parameters are used for CIFAR-10, CIFAR-100, and CINIC-10, respectively.Communication cost is measured as 2 × (#participants) × (model size) × (#rounds), including both uplink and downlink.
  • Communication Cost: 2.8 to 10.1 times fewer communication costs and energy consumption are required to reach the same target accuracy as VGG16ori.Across Figures 3a-3f, VGG16FedPara maintains comparable accuracy while requiring much lower communication costs.
  • Quantitative Results: VGG16FedPara accuracy mostly increases with parameter count and can exceed VGG16ori., while combining FedPara with FedDyn gives the best reported optimizer combination.The experiments also evaluate compatibility with other FL algorithms and personalized models.
  • Personalization: pFedPara performs better or favorably against competing personalized methods in most evaluated scenarios involving sufficient, scarce, or highly skewed local data.pFedPara uses global and local components to train personalized models collaboratively.
  • Personalization: 3.4 times fewer parameters are transferred by pFedPara than by the original model per round, compared with 1.07 times fewer for FedPer.pFedPara factorizes all layers, whereas FedPer transfers all layers except the last one.

4 RELATED WORK

Prior federated and distributed learning work reduces communication through algorithmic optimization, compression, and low-rank decomposition, but low-rank approaches can require extra processing or lose accuracy at high compression rates.

  • FedAvg reduces communication costs by averaging the global model after many local SGD iterations per round.
  • Distributed learning uses quantization, sparsification, low-rank decomposition, and adaptive compression to address communication bottlenecks.
  • Low-rank decomposition methods can require post-decomposition and fine-tuning, making them inappropriate for federated learning.
  • Training federated models with low-rank constraints can degrade accuracy when the compression rate is high.

5 DISCUSSION AND CONCLUSION

FedPara and pFedPara reduce federated-learning communication overhead while retaining comparable or better performance, but they introduce training-time computation and open initialization questions.

  • FedPara and pFedPara significantly reduce communication overhead with minimal performance degradation or sometimes better performance than the original counterpart.
  • FedPara avoids the low-rank limitation and can achieve full-rank matrices and tensors despite using low-rank factors.
  • Repeated multiplications during FedPara training may increase susceptibility to exploding or vanishing gradients, dead neurons, or numerical instability.The experiments did not observe these issues with He initialization.
  • Analyzing statistical properties of composited weights and activations remains an unexplored future direction.
  • FedPara requires additional computation to re-compose layer weights during training, making it slower than original and low-rank parameterizations.

ETHICS STATEMENT

FedPara addresses communication-related infrastructure and energy concerns, while sharing federated learning’s privacy and security risks. Its training-time computation is noted elsewhere as a practical trade-off.

  • Privacy and Security: FedPara shares federated learning’s communication-related privacy risks, including membership inference and reconstruction attacks.
  • Privacy and Security: pFedPara’s local parameters could function as a private key for acquiring the complete personal model.
  • Infrastructure Level Gap: FedPara may broaden access to federated learning in regions with inferior communication infrastructure by reducing required bandwidth.
  • Energy Consumption: FedPara’s communication efficiency directly leads to energy savings, including lower battery consumption for IoT devices and reduced electricity-related fossil-fuel use.
  • Energy Consumption: Under fixed bandwidth, FedPara can allow more clients to participate in each learning round than optimization-based approaches that reduce communication rounds.The passage connects this participation pattern with improved convergence speed and accuracy.

A.1 PROOFS

The proofs characterize the rank and parameter-count properties of Hadamard products of low-rank factors, showing when FedPara can attain maximal rank and how balanced inner ranks minimize parameters.

  • Propositions and Corollaries: W is formed from the Hadamard product of two factorized matrices, and its rank is bounded above by r1r2.
  • Propositions and Corollaries: r1 = r2 = R uniquely minimizes (r1 + r2)(m + n) subject to r1r2 ≥ R2, with optimal value 2R(m + n).
  • Propositions and Corollaries: R2 ≥ min(m, n) is necessary and sufficient for W to achieve maximal rank.
  • Propositions and Corollaries: When R2 ≥ min(m, n), W has no low-rank restriction, and the minimum inner rank is rmin := min(⌈√m⌉, ⌈√n⌉).
  • Tensor Rank Analysis: For convolution kernels, the first and second unfoldings satisfy rank(W(1)) = rank(W(2)) ≤ R2.

A.2 ANALYSIS OF THE RANK PROPERTY

FedPara’s low-rank Hadamard parameterization can represent full-rank matrices with fewer entries, supported by repeated random experiments on 100 × 100 weights.

  • 100% of the 1,000 sampled parameterizations achieved full rank while using 2.5 times fewer entries than the original 100 × 100 matrix.The experiment sampled X1, X2, Y1, and Y2 from a standard Gaussian distribution with rank parameters set to 10.

B ADDITIONAL TECHNIQUES

The paper adds optional nonlinearity and Jacobian correction techniques to improve FedPara’s accuracy, stability, and optimization behavior.

  • Non-linear Function: FedPara can inject a nonlinearity before the Hadamard product by computing W = σ(W1)⊙σ(W2).The paper discusses ReLU, Tanh, and Sigmoid, with Tanh favored for its bounded range containing both negative and positive values.
  • Jacobian Correction: Jacobian correction regularization minimizes the difference between the constructed weight after one update and W − ηJW.This regularizer encourages the parameterization to follow the update direction of W−ηJW.
  • Results: The combined Tanh and regularization configuration achieves higher accuracy and lower variation than the base model.Using only one technique improves variance, while the combined configuration improves both accuracy and variance.
  • Results: The additional techniques are optional and may be selected when devices have enough computing power.They are described as nonessential for FedPara to work.

C.1 DATASETS

The experiments use image and language datasets across IID and non-IID federated settings, with model architectures and hyperparameters specified for FedPara and pFedPara.

  • Datasets: The benchmark suite includes CIFAR-10, CIFAR-100, CINIC-10, MNIST, FEMNIST, and Shakespeare.These datasets cover image classification, handwritten-character recognition, and next-word prediction in federated settings.
  • Federated settings: The federated partitions include randomly assigned IID splits and heterogeneous non-IID splits controlled by Dirichlet or label-skew procedures.MNIST clients have at most two classes, while CIFAR non-IID partitions use Dirichlet distributions.
  • Models: VGG16 uses group normalization and modified fully connected layers, while personalization uses two fully connected layers with dimensions 256–⟨#classes⟩.The last three VGG16 fully connected layers are not parameterized by FedPara; pFedPara sets γ = 0.5.
  • Optimization: FedAvg is the primary backbone optimizer, while compatibility experiments combine FedPara with FedProx, SCAFFOLD, FedDyn, and FedAdam.The experiments use SGD for local client updates and specify optimizer-specific hyperparameters.

D.1 TRAINING TIME

FedPara reduces communication and total training time in simulated network settings despite additional computation for weight composition.

  • Per-round time: Communication is the bottleneck because communication time exceeds computation time in the VGG16 experiments.The comparison evaluates network speeds of 2, 10, and 50 Mbps.
  • Per-round time: FedPara decreases communication time by about ten times and total time by 4.80 to 9.52 times versus the original VGG16 model.FedPara requires more computation time because of weight composition.
  • Accuracy–communication trade-off: Figure 7 compares accuracy against communication costs for VGG16ori. and FedPara across low, mid, and high γ settings.The original model is shown with a broken black line, while FedPara variants use red, blue, and yellow solid lines.

D.2 OTHER MODELS

Across VGG16, ResNet18, LSTM, and quantization experiments, FedPara generally preserves or improves accuracy while reducing parameters or communication costs. Its benefits also extend to comparisons with PufferFish and combinations with FedPAQ.

  • VGG16: VGG16FedPara achieves comparable or higher accuracy than VGG16ori. while requiring fewer communication costs.Higher γ uses more parameters and can produce higher accuracy.
  • ResNet18: 1.17 to 5.1 times fewer communication costs are required by ResNet18FedPara than ResNet18ori. at the same target accuracy.Figure 8a also reports comparable accuracy with fewer communication costs.
  • PufferFish: FedPara achieves higher accuracy with fewer parameters than PufferFish on VGG16 trained with CIFAR-10 IID data.The comparison evaluates models across varying parameter counts.
  • LSTM: 19% of original parameters lets LSTMFedPara outperform LSTMlow and LSTMori. on IID data, while remaining slightly below LSTMori. on non-IID data.On non-IID data, LSTMFedPara remains higher than LSTMlow.
  • Quantization: 1.96 times lower bits per round are transferred by FedPara than FedPAQ, while their combination cuts FedPara communication costs by 25% further.The integrated method has a minor 0.1% accuracy drop from FedAvg.
Loading 2108.06098v3…