PaperlayerView exact revision on arXiv

Source-linked AI summary

Communication-Efficient Personalized Federated Learning via Layer-Wise Multi-Threshold Random Sketching

Xu Zhang, Xingyu Hou, Jiacheng Cheng, Kaiyuan Feng, Maoguo Gong

arXiv:2609.04830v1cs.LG

TL;DR

PFL still faces costly bidirectional exchange of high-dimensional models, while one-bit methods use a single layer-agnostic threshold that misses layer-specific and fine-grained parameter variation. The paper proposes pFedLMS, using layer-wise multi-threshold random sketching for compact bidirectional communication. It reports a favorable communication-accuracy balance against representative baselines, with gains over pFed1BS in several settings.

  • Problem

    PFL reduces data-heterogeneity concerns but remains communication-intensive, while single-threshold one-bit compression overlooks layer-wise parameter differences and provides coarse quantization.

  • Method

    pFedLMS assigns independent threshold sets to network layers, forms multi-threshold random sketches, and uses them for bidirectional low-bit consensus and personalized training.

  • Results

    pFedLMS achieves a favorable communication-accuracy balance against representative federated learning baselines and improves over pFed1BS by 9.17% on CIFAR-10 and 13.49% on CIFAR-100 when α = 0.5.

  • Takeaways & Limitations

    Layer-wise multi-threshold sketches provide more expressive low-bit representations while preserving personalized model training under compact uplink and downlink transmission.

  • Takeaways & Limitations

    The theoretical cross-round drift bounds may overestimate actual drift when consensus and thresholds vary slowly during training.

Abstract

from arXiv · show

Personalized federated learning (PFL) is a promising paradigm for collaborative learning over distributed devices, where edge nodes collaboratively train personalized models without sharing raw data. Although PFL addresses data heterogeneity by learning client-specific models, it still suffers from substantial uplink and downlink communication costs when exchanging high-dimensional parameters in bandwidth-constrained systems. Recent one-bit methods achieve extreme compression, but they usually rely on a single thresholding rule applied to the whole model. This design has two limitations. First, it overlooks layer-wise differences in parameter distributions and quantization sensitivities. Second, a single threshold provides only coarse binary information and cannot capture fine-grained variations in parameter distributions. To address these issues, we propose a communication-efficient PFL framework via layer-wise multi-threshold random sketching. In the proposed method, each layer is assigned its own set of quantization thresholds, so that the compressed representation can adapt to layer-specific statistics while using multiple intervals to provide a finer low-bit description of sketched parameters. The proposed method supports bidirectional communication using compact low-bit sketches and improves the communication-accuracy tradeoff compared with existing one-bit compression approaches.

1. Introduction

PFL reduces data-heterogeneity concerns through personalized models, but exchanging high-dimensional parameters remains costly. The paper proposes pFedLMS, which replaces layer-agnostic one-bit compression with layer-wise multi-threshold random sketching.

  • Motivation: PFL learns client-specific models for heterogeneous local data, but repeated exchange of high-dimensional parameters creates a communication bottleneck.Existing compression methods reduce overhead but commonly use layer-agnostic binary rules.
  • Motivation: Layer parameter distributions differ and evolve during training, making one compression rule mismatched across the model.The shallow fc1.weight distribution is more concentrated than the wider fc2.weight distribution.
  • Motivation: A single threshold is too coarse for broad layer distributions, whereas multiple thresholds can improve intra-layer quantization resolution.The proposed design addresses both inter-layer heterogeneity and within-layer variation.
  • Proposed approach: pFedLMS introduces layer-wise multi-threshold random sketching with an independent threshold set for each network layer.This produces structure-aware, fine-grained quantization of sketched representations.
  • Proposed approach: The framework supports bidirectional low-bit communication by aggregating compact quantized sketches and broadcasting a concise global consensus signal.The consensus guides local personalization without transmitting full-precision models.
  • Contributions: The paper combines layer-wise sketching with weighted server aggregation, a multi-threshold consensus regularizer, convergence analysis, and empirical evaluation against federated and personalized baselines.The stated baselines include FedAvg, OBDA, OBCSAA, FedProto, EDEN, zSignFed, and pFed1BS.

2. Problem Formulation

The formulation represents each client model through layer-wise random sketches and threshold comparisons, then aggregates them by weighted majority voting. A multi-threshold regularizer aligns personalized models with the consensus while increasing thresholds improves alignment resolution.

  • System model: Each client owns local data and a personalized model θk ∈ R^n decomposed into L layers, while minimizing its local expected loss.The formulation considers K clients over R communication rounds.
  • Random sketching: For each layer, a sketching operator with mℓ ≪ nℓ produces a lower-dimensional representation, and an ordered threshold set generates one-bit comparisons.The threshold-wise comparisons form a multi-threshold sketch Bk,ℓ.
  • Server aggregation: The server maintains layer-wise threshold consensus variables and updates them independently across layers and thresholds using weighted majority voting.The resulting consensus is broadcast to clients for local updates.
  • Client alignment: The multi-threshold alignment regularizer induces a unique consistency interval for each coordinate, and increasing T refines the partition.The refinement improves alignment resolution within the threshold-covered region.
  • Server aggregation: The server objective measures weighted disagreement between consensus signs and client sketches, and majority voting provides its coordinatewise closed-form solution.This preserves the stated aggregation objective while enabling compact communication.
  • Resolution analysis: Under quantile thresholds, the finite-resolution refinement has order O(1/T) when the minimum internal interval width is bounded away from zero.The analysis distinguishes the finite internal region from the two unbounded tail intervals.
  • Optimization: Nesterov smoothing makes the nonsmooth alignment regularizer differentiable, with uniform approximation error of order O(ρ).The resulting alternating framework separates discrete server aggregation from smooth client optimization.

3. Algorithm Design

The algorithm combines layer-wise thresholding, random sketches, and weighted consensus to support compact bidirectional communication. Clients compute local sketches and statistics, while the server updates thresholds and aggregates threshold comparisons into a broadcast consensus.

  • Communication-efficient implementation: The communication protocol encodes each layer's interval-valued multi-threshold sketch into a binary bitstream for transmission.The server decodes, aggregates, re-encodes, and broadcasts the result; communication scales logarithmically with the number of thresholds.
  • Server update: Clients compute one-bit sketches by comparing each layer's sketched model with its prescribed threshold levels.The resulting multi-threshold sketches are sent to the server for aggregation.
  • Server update: The server aggregates threshold comparisons independently across layers and thresholds using weighted majority voting.This produces the compact global representation V^r broadcast to clients for local updates.
  • Client update: Clients perform E local stochastic-gradient steps on personalized models after receiving the broadcast consensus.The local update includes the consensus-dependent training objective and alignment terms.
  • Threshold update: Layer-wise sketched means and variances are computed over layer entries and sent to the server for threshold updates.The server aggregates these statistics and constructs the next threshold sets using Gaussian quantiles.

4. Theoretical Analysis

The analysis establishes average-stationarity guarantees for pFedLMS under smoothness, stochastic-gradient, sketching, and threshold assumptions. The bound decreases with optimization progress but includes stochastic-noise and cross-round drift terms caused by changing consensus, thresholds, and sketching operators.

  • Analytical setting: The analysis models client objectives as time-varying because they depend on sketch comparisons and the threshold sequence.This distinguishes the setting from optimization of a fixed objective.
  • Assumptions: The framework assumes bounded random sketching operators and ordered, bounded threshold sets.These assumptions control perturbation amplification and keep threshold-dependent regularization terms bounded.
  • Main result: Theorem 1 bounds the average squared gradient norm of client objectives across communication rounds under Assumptions 1–6 and η ≤ 1/L_F.The result measures average stationarity rather than convergence to a fixed objective.
  • Main result: The optimization term decreases at rate O(1/(ηER)), while stochastic gradient noise and cross-round objective drift contribute residual error.Drift reflects changes in consensus, thresholds, and sketching operators.
  • Limitations of the bound: The drift bound can overestimate actual variation when consensus and thresholds change slowly during training.Its sketch-refresh component vanishes when a layer's sketching operator remains fixed.
  • Parameter effects: The regularization parameter λ trades off optimization stability, personalization, and alignment with the global low-bit consensus.Changing λ affects both the drift term and the smoothness-dependent stochastic-gradient term.

5. Experiments

Experiments evaluate pFedLMS across heterogeneous federated settings, datasets, accuracy, communication cost, convergence, and component ablations. The results show that layer-wise multi-threshold sketching improves the accuracy–communication tradeoff and that both proposed components contribute to performance.

  • Experimental setup: Experiments use five datasets with 20 non-IID clients, evaluating maximum accuracy and per-round bidirectional communication cost.The study uses Dirichlet settings α = 0.1 and α = 0.5, with results averaged across three to five runs for maximum accuracy.
  • Accuracy results: At α = 0.5, pFedLMS improves over pFed1BS by 9.17% on CIFAR-10 and 13.49% on CIFAR-100.It achieves high accuracy across all datasets and outperforms most communication-efficient baselines.
  • Convergence: On CIFAR-10, pFedLMS achieves the best performance among compressed methods and approaches FedAvg with much lower communication cost.Its accuracy curve remains above the listed compressed baselines through most of training under α = 0.5.
  • Communication efficiency: pFedLMS reduces communication cost relative to FedAvg by approximately 98.8% on MNIST/FMNIST, 98.3% on SVHN/CIFAR-10, and 98.3% on CIFAR-100.Its cumulative costs are 0.38 MB, 0.72 MB, and 25.05 MB, respectively, compared with FedAvg costs of 31.06 MB, 42.68 MB, and 1495.34 MB.
  • Accuracy–cost tradeoff: On MNIST, pFedLMS quickly stabilizes above 95% accuracy with a small communication budget and achieves the best overall curve among low-communication methods.FedOBDA remains around 40% at approximately 10 MB, while pFed1BS and FedProto finish slightly below pFedLMS.
  • Ablation study: Combining layer-wise sketching and multi-threshold quantization raises SVHN accuracy from 89.80% to 91.94%, while CIFAR-10 accuracy increases from 78.68% to 82.61%.On SVHN, the combined design yields an overall gain of 2.14% over the refined pFed1BS baseline; on CIFAR-10, both components show complementary benefits.

6. Conclusion

The paper presents pFedLMS, a communication-efficient personalized federated learning framework that uses layer-wise multi-threshold random sketching, compact bidirectional transmission, and consensus regularization. It reports reduced communication cost while preserving personalized model training and balancing communication efficiency with accuracy.

  • pFedLMS uses layer-wise multi-threshold random sketching to obtain more expressive low-bit representations than conventional one-bit compression.
  • The framework combines compact uplink and downlink transmission with a consensus regularizer for personalized model training.
  • pFedLMS reduces communication cost while preserving personalized model training.
  • The paper provides convergence analysis for the resulting nonconvex stochastic optimization problem.
  • Experimental results show a favorable balance between communication cost and accuracy compared with representative federated learning baselines.

CRediT authorship contribution statement

The supplied material describes a compact communication protocol for layer-wise multi-threshold random sketching, encoding threshold comparisons as (T + 1)-ary symbols for bidirectional binary transmission. Server decoding, aggregation, re-encoding, and client recovery preserve the original threshold-wise semantics and regularization.

  • The implementation transmits a compact (T + 1)-ary representation encoded into a binary bitstream instead of the theoretical collection of T one-bit comparison vectors.
  • Each (T + 1)-ary symbol identifies the interval containing a sketched coordinate, and the collected symbols form the sketch vector q_k,ℓ.
  • Base-(T + 1) encoding converts the sketch vector into an integer and then a binary bitstream for transmission.
  • The server decodes received bitstreams, recovers threshold-wise comparison results, and aggregates them by weighted majority voting.
  • The aggregated consensus is re-encoded for downlink transmission, while decoding exactly recovers the threshold-wise information used by client regularization.
  • The protocol supports both uplink and downlink communication costs based on the encoded layer-wise representations.

Appendix B.1. Effect of Sketching Operator Refresh Interval

The appendix examines sketching-operator refresh frequency and regularization sensitivity. More frequent refreshes improve MNIST accuracy, while pFedLMS remains relatively stable across a broad range of regularization values on the evaluated datasets.

  • Sketching operators are refreshed every h communication rounds, with h tested from 1 to 50 on MNIST.
  • 97.85% accuracy is achieved when sketching operators are refreshed every round on MNIST.
  • 96.44% accuracy is obtained when the refresh interval increases to 50 rounds.
  • The results indicate that more frequent refreshes generally provide better performance by supplying more diverse measurements for global consensus.
  • pFedLMS remains relatively stable across a broad range of λ values, particularly on MNIST, FMNIST, and CIFAR-10.

Appendix C.1. Proof of Theorem 1

The proof derives a recursion for the objective by combining local-step descent with the change between consecutive communication rounds. Summing the recursion and applying the lower-bound assumption yields the stated result.

  • The proof begins by summing the one-step descent bound across local steps and clients with client weights.
  • It then accounts for objective changes between consecutive communication rounds using the bound from Lemma 7.
  • Combining the descent and inter-round-change bounds produces a recursion for the objective.
  • Summing the recursion over communication rounds yields the proof's cumulative inequality.
  • Assumption 4 supplies a lower bound on the objective, completing the argument.

Appendix C.2. Proof of Lemma 1

The appendix proves Lemma 1 by reducing the objective to separable coordinate-wise problems across layers and thresholds, then characterizing the optimal sign pattern. It also develops supporting bounds for regularization gradients, smoothness, stochastic updates, and related quantities.

  • Coordinate-wise reduction: The proof reduces the problem to independent subproblems across layers, thresholds, and coordinates.The resulting coordinate-wise optimization is analyzed using weighted sign sums.
  • Optimal sign pattern: Ordered thresholds make the sign sequence monotone, with a single transition from +1 to −1 at an index s_ℓ,i.The convention sign(0) = 1 resolves ties when the weighted sum vanishes.
  • Threshold spacing: Quantile-selected thresholds assign probability mass 1/(T + 1) to each internal interval.The bound follows by taking the maximum over the internal threshold intervals.
  • Auxiliary bounds: The remaining bounds follow by combining client-update relations, expectation arguments, Jensen’s inequality, and the assumptions.The appendix repeatedly concludes intermediate results by substitution, averaging over thresholds and layers, and induction over local steps and communication rounds.
  • Auxiliary bounds: The supporting analysis bounds the regularizer gradient using clipping, triangle inequalities, and the stated assumptions.The argument stacks layer blocks after establishing coordinate-wise bounds in [−2, 2].
  • Convergence-related bounds: The proof establishes smoothness of F_k and derives descent and update bounds under η ≤ 1/L_F.It uses the smoothness of f_k, Lipschitz clipping, the regularization term, conditional expectations, and a descent lemma.
Loading 2609.04830v1…