PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhanced Privacy and Communication Efficiency in Non-IID Federated Learning with Adaptive Quantization and Differential Privacy

Emre Ardıç, Yakup Genç

arXiv:2604.23426v1cs.CVcs.LG

TL;DR

Federated learning must reduce communication costs while protecting sensitive information, especially with non-IID data. This paper combines Laplacian differential privacy with adaptive quantization, reducing communication by up to 52.64% on MNIST and 45.06% on CIFAR10 versus 32-bit float training while maintaining competitive accuracy.

  • Problem

    Prior work had not established combined differential privacy and adaptive quantization for large-scale, non-IID federated learning, where privacy and communication efficiency are key challenges.

  • Method

    The framework combines Laplacian-based differential privacy with cosine-annealed global scheduling and entropy-based client-adaptive quantization.

  • Results

    Adaptive quantization reduced communication by up to 52.64% on MNIST, 45.06% on CIFAR10, and 31%–37% on medical imaging datasets versus 32-bit float settings while maintaining competitive accuracy.

  • Takeaways & Limitations

    The approach supports more communication-efficient and privacy-preserving federated learning across non-IID datasets and varying client scales.

  • Takeaways & Limitations

    Adaptive schedulers may not always outperform a well-chosen fixed bit-length setting in performance or communication overhead.

Abstract

from arXiv · show

Federated learning (FL) is a distributed machine learning method where multiple devices collaboratively train a model under the management of a central server without sharing underlying data. One of the key challenges of FL is the communication bottleneck caused by variations in connection speed and bandwidth across devices. Therefore, it is essential to reduce the size of transmitted data during training. Additionally, there is a potential risk of exposing sensitive information through the model or gradient analysis during training. To address both privacy and communication efficiency, we combine differential privacy (DP) and adaptive quantization methods. We use Laplacian-based DP to preserve privacy, which is relatively underexplored in FL and offers tighter privacy guarantees than Gaussian-based DP. We propose a simple and efficient global bit-length scheduler using round-based cosine annealing, along with a client-based scheduler that dynamically adapts based on client contribution estimated through dataset entropy analysis. We evaluate our approach through extensive experiments on CIFAR10, MNIST, and medical imaging datasets, using non-IID data distributions across varying client counts, bit-length schedulers, and privacy budgets. The results show that our adaptive quantization methods reduce total communicated data by up to 52.64% for MNIST, 45.06% for CIFAR10, and 31% to 37% for medical imaging datasets compared to 32-bit float training while maintaining competitive model accuracy and ensuring robust privacy through differential privacy.

I. INTRODUCTION · II. RELATED WORKS · A. COMMUNICATION EFFICIENCY

The paper frames federated learning as a privacy-preserving collaborative method constrained by heterogeneous communication, sensitive gradient exposure, and non-IID data. It addresses these challenges through Laplacian-based differential privacy and adaptive quantization using global cosine annealing and client-level entropy analysis.

  • I. INTRODUCTION: Federated learning coordinates distributed model training through a central server while keeping client data local and transferring only model updates.
  • I. INTRODUCTION: Heterogeneous device capabilities and connectivity create communication bottlenecks, making both communication rounds and transmitted package sizes important optimization targets.
  • I. INTRODUCTION: Gradient analysis can expose sensitive information, while non-IID client data introduces bias that may slow convergence or cause divergence.
  • I. INTRODUCTION: The paper identifies a gap in combining differential privacy and adaptive quantization for large-scale FL with non-IID data, including settings with 1000 clients.
  • I. INTRODUCTION: A round-based cosine annealing scheduler starts server-to-client transmission at 32-bit precision and gradually reduces bit-length during training.
  • I. INTRODUCTION: The proposed framework combines Laplacian-based differential privacy with adaptive quantization to address privacy preservation and communication efficiency simultaneously.
  • A. COMMUNICATION EFFICIENCY: Communication overhead can be reduced through longer local-training intervals, allowing clients to perform multiple epochs before transmitting updates to the server.
  • A. COMMUNICATION EFFICIENCY: Sparsification, subsampling, quantization, and adaptive methods such as AdaQuantFL and FedDQ reduce transmitted information by restricting or lowering update precision.

B. PRIVACY · III. PROPOSED METHODS

The paper frames privacy as protecting client data from gradient and model-update leakage, then proposes Laplacian differential privacy followed by adaptive quantization for non-IID federated learning. The methods combine entropy-based client importance estimation and cosine annealing with formal notation for training, quantization, sensitivity, and DP noise.

  • B. PRIVACY: Client gradients and model updates can expose sensitive training data, motivating privacy mechanisms that keep raw data local.The cited example shows that analyzing trained recurrent neural networks can reveal sensitive text such as credit card numbers.
  • B. PRIVACY: Differential privacy reduces the relationship between model inputs and outputs by hiding whether a particular sample participated in training.The approach treats sample-level privacy as a way to limit output-distribution changes caused by one input feature.
  • B. PRIVACY: Privacy in federated learning is categorized as global or local, depending on whether updates remain hidden from untrusted parties alone or from the server as well.Secure multiparty computation can provide lossless, highly secure protection without compromising learning accuracy.
  • B. PRIVACY: Prior privacy-communication methods include vector-quantization with local DP and model splitting with dynamic quantization, but existing approaches do not fully tailor quantization to client or parameter importance.The passage specifically contrasts adaptive strategies with JoPEQ’s static quantization.
  • III. PROPOSED METHODS: The proposed method applies Laplace noise to model updates before adaptive quantization, using ℓ1-sensitivity to support sharper privacy guarantees for bounded federated-learning updates.This ordering incorporates the DP noise into the quantized updates while balancing privacy and communication efficiency.
  • III. PROPOSED METHODS: Adaptive quantization uses Shannon entropy to estimate client importance and cosine annealing to adjust communication bit-lengths during non-IID federated training.The proposed-methods overview identifies entropy and cosine annealing as the mechanisms for improving communication efficiency between the server and clients.
  • III. PROPOSED METHODS: The method formalizes client selection, local training, quantization and dequantization, scale factors, bit-length, importance scores, sensitivity, gradients, smoothness, and DP-generated noise.These definitions cover the variables needed to specify the training rounds, quantization operations, client contributions, and privacy perturbations.

A. FEDERATED LEARNING · B. DATASETS

The study uses synchronous FedAvg federated learning, where selected clients train locally and periodically exchange model parameters with a central server. Experiments use MNIST and CIFAR10 under deliberately non-IID client partitions, including digit-limited and Dirichlet-based allocations.

  • A. FEDERATED LEARNING: The synchronous FL system consists of a central server and N clients, each holding a local dataset of labeled samples.The global objective is trained across many clients without directly transferring their underlying data.
  • A. FEDERATED LEARNING: FedAvg aggregates local models while the server periodically transfers model parameters between remote clients and the central coordinator.Training repeats client selection, broadcasting, local updating, and server aggregation.
  • A. FEDERATED LEARNING: Client selection considers battery level, network quality, and CPU load to limit adverse effects on device performance and usability.Selected clients retrieve current weights and parameters, update models on local data, and return updates for aggregation.
  • A. FEDERATED LEARNING: Each client minimizes its local loss over its dataset, using stochastic gradient descent with Cross-Entropy loss during local training.The client loss is defined from sample-level loss over the client’s n_i = |D_i| examples.
  • B. DATASETS: MNIST contains 28x28 grayscale handwritten-digit images from 1,000 individuals, distributed across 1000 clients with each client holding two digits.This setup creates a naturally non-IID partition based on users and digit availability.
  • B. DATASETS: CIFAR10 contains 32x32 color images across 10 vehicle and animal categories, creating greater visual variability than MNIST.Images are labeled by category, with dataset statistics reported in Table 2.
  • B. DATASETS: CIFAR10 is distributed with a Dirichlet alpha parameter of 0.5 to simulate realistic non-IID heterogeneity across clients.The allocation can give some clients disproportionate shares of particular classes.
  • B. DATASETS: When fewer than 1,000 MNIST clients are used, images from multiple users are randomly combined to form the required client population.Assigning each user as a client otherwise preserves user-based non-IID partitioning.

C. MODEL ARCHITECTURES · D. STOCHASTIC UNIFORM QUANTIZATION

The paper uses dataset-specific CNN architectures for MNIST and CIFAR10, then applies stochastic uniform quantization to map model weights into compact integer representations. The method combines symmetric stochastic rounding, clipping, dequantization, and per-tensor parameter sharing to reduce representation size while preserving robustness.

  • C. MODEL ARCHITECTURES: The MNIST model is a two-layer CNN with 32 and 64 filters, 5x5 kernels, stride 1, padding 2, ReLU activations, and 2x2 max-pooling.This architecture is also used in the cited FedAvg study.
  • C. MODEL ARCHITECTURES: The CIFAR10 model is a four-layer CNN with 32, 64, 128, and 128 filters, 3x3 kernels, stride 1, padding 1, ReLU, batch normalization, and adaptive average pooling.The architecture is designed for 32x32 color images and is identified as VGG7.
  • D. STOCHASTIC UNIFORM QUANTIZATION: Stochastic uniform quantization maps continuous model weights to discrete levels with randomness, producing uniformly distributed quantization error.The approach is intended to provide robust representations when precision and generalization are important.
  • D. STOCHASTIC UNIFORM QUANTIZATION: Quantization converts real values into integer representations such as FP32 to INT8, while dequantization converts integers such as INT8 back to real values.Uniform quantization maps x ∈[β, α] into the signed integer range determined by target bit-width b and clips out-of-range inputs.
  • D. STOCHASTIC UNIFORM QUANTIZATION: The method uses symmetric quantization with z = 0, applying scale transformation, stochastic rounding, and clipping around zero.The scale factor depends on target bit-length b and the tensor maximum α.
  • D. STOCHASTIC UNIFORM QUANTIZATION: Stochastic rounding maps a floating-point value to one of its nearest integers with probabilities proportional to the distances from those values.The operation rounds to ⌊x⌋ with probability ⌈x⌉−x and to ⌈x⌉ with probability x−⌊x⌋.
  • D. STOCHASTIC UNIFORM QUANTIZATION: The implementation uses per-tensor granularity, sharing quantization parameters across each three-dimensional tensor in every model layer.This avoids the computational expense of assigning individual parameters to every tensor element.

E. ADAPTIVE QUANTIZATION

The paper proposes adaptive quantization that dynamically reduces bit-length during training, using cosine annealing to preserve early-training precision while lowering communication later. A hybrid variant further adapts client-to-server precision using client data entropy and importance.

  • Adaptive quantization: Cosine annealing progressively reduces quantization precision from bmax toward bmin, smoothing the transition to mitigate accuracy loss.The schedule dynamically adjusts bit-length throughout training.
  • Adaptive quantization: Higher precision accelerates early loss reduction, whereas fewer bits later reduce bit volume after model stabilization.Descending-trend quantization is therefore suited to federated learning communication.
  • Adaptive quantization: The hybrid strategy combines cosine annealing with client-based Shannon entropy for client-to-server transmissions, while server-to-client transmissions use cosine annealing alone.Entropy-based adjustments allocate higher bit-lengths to clients with larger, diverse, and balanced datasets.
  • Adaptive quantization: 32 bits gradually reduces to 2 bits under the cosine-annealing bit-length schedule.This schedule defines the overall downward precision trend across communication rounds.

F. DIFFERENTIAL PRIVACY

The section introduces local differential privacy to mitigate leakage from federated model updates, while controlling the accuracy cost of injected noise through sensitivity-aware Laplace perturbation. Gradient clipping and client-specific smoothness estimates determine noise levels for non-IID local updates.

  • Differential privacy is added because federated model updates can leak sensitive information, although injected noise may reduce model accuracy.
  • A mechanism is (ϵ, δ)-differentially private when neighboring datasets satisfy the specified randomized-function condition, with larger ϵ increasing privacy-violation risk.Here, ϵ is the privacy budget and δ bounds the probability of privacy leakage.
  • The method employs Local DP with δ = 0, perturbing client model weights before transmission to the central server.Central DP instead applies perturbation at the server before global-model broadcast.
  • Gradient clipping bounds each client’s l1-norm gradients, enabling sensitivity analysis under non-IID data distributions.Client sensitivity depends on clipping, learning rate, local epochs, dataset size, and regularization-related quantities.
  • Client-specific noise is calibrated with Laplace perturbation and sensitivity scaling, while λi controls gradient norms and the required noise magnitude.λi is estimated from batch-wise gradient and parameter differences, using the largest observed value across batches and epochs.

IV. EXPERIMENTS

The experiments evaluate the proposed federated learning approach across model architectures, client counts, privacy budgets, and non-IID datasets. They assess convergence, communication efficiency, accuracy, and privacy, first testing differential privacy and quantization separately before evaluating their combined effects.

  • Experimental Scope: The evaluation spans model architectures, client counts, privacy budgets, and non-IID datasets.These experiments comprehensively assess the approach defined in Algorithm 1.
  • Evaluation Goals: The method is evaluated for convergence with significantly fewer communicated bits while maintaining model accuracy and enhancing privacy.
  • Evaluation Design: Differential privacy and quantization are first evaluated individually, followed by assessment of their combined impact across client counts and privacy budgets.

A. DEPLOYMENT

The methods were evaluated in a FedML-based federated learning environment supporting single-machine simulation, distributed computation, and edge-device training. Experiments used varying client counts on a single machine with specified hardware and software configurations.

  • A. DEPLOYMENT: FedML provided the evaluation framework, including baseline implementations for optimizers, models, and datasets.The library also offers a versatile programming interface for federated learning experiments.
  • A. DEPLOYMENT: Experiments varied the number of clients while running on a single machine.The system was established using FedML for federated learning evaluation.
  • A. DEPLOYMENT: The deployment used an NVIDIA RTX 3090 GPU, Ryzen 5900X CPU, 32GB RAM, and 1TB SSD.The software environment included Python 3.6, Scikit-Learn 0.24.2, and PyTorch 1.

B. TRAINING CONFIGURATION · C. LAPLACIAN-BASED DP

Training uses a fixed FedAvg configuration over 1000 rounds, while the Laplacian-based DP evaluation studies gradient clipping, client counts, and privacy budgets on non-IID MNIST and CIFAR10. Accuracy generally declines with stronger noise and more clients, with dataset-specific privacy budgets needed for stable convergence.

  • B. TRAINING CONFIGURATION: FedAvg trains models for 1000 rounds, using SGD with learning rate 0.1, weight decay 0.001, batch size 64, and five local epochs.Train and test accuracies are calculated every 10 rounds, and rounds with the highest test accuracy are recorded.
  • C. LAPLACIAN-BASED DP: The DP experiments vary client counts and privacy budgets ϵ on non-IID datasets, first analyzing gradient norm effects and then privacy-budget effects on accuracy.The evaluation is designed to assess both model accuracy and privacy across different client configurations.
  • C. LAPLACIAN-BASED DP: Gradient clipping bounds individual influence for DP, and experiments with 100 clients vary gradient norm ξ to identify an accuracy-preserving setting.The supplied results state that accuracy diminishes as the gradient norm varies, with Table 3 reporting the best accuracies.
  • C. LAPLACIAN-BASED DP: With ξ = 100, test accuracy increases as the total client count decreases because each client receives a larger and more diverse local training dataset.These larger local datasets improve convergence of local models under the non-IID distribution.
  • C. LAPLACIAN-BASED DP: Higher model complexity increases sensitivity to noise, so selecting privacy budgets requires balancing accuracy against privacy across client counts.Experiments varying privacy budgets and client counts are summarized in Table 5.
  • C. LAPLACIAN-BASED DP: For 1000 clients, convergence fails at ϵ ≤103 on MNIST and ϵ ≤105 on CIFAR-10, while CIFAR10 performs more appropriately at ϵ ≥5×105.Increasing client counts reduce samples per client and produce higher noise through the sensitivity formulation, widening the accuracy gap.
  • C. LAPLACIAN-BASED DP: For 100 clients, CIFAR10 accuracy stabilizes around 70% at ϵ = 104, whereas ϵ = 5000 causes a noticeable drop; MNIST reaches approximately 80% at higher budgets.MNIST exhibits larger accuracy fluctuations across privacy budgets during the 1000 training rounds.

D. ADAPTIVE QUANTIZATION

The adaptive quantization evaluation combines cosine-annealed global scheduling with entropy-based client importance to balance accuracy and communication in non-IID federated learning. Across client scales, Dynamic and Cosine generally preserve competitive accuracy while substantially reducing communication relative to 32-bit training.

  • Scheduling method: Cosine annealing schedules global bit-lengths by communication round, while Dynamic further weights client bit-lengths using importance derived from local-dataset homogeneity and size.The server quantizes the global model before broadcasting, and clients either use the global bit-length or adjust it by client importance.
  • Client-importance weighting: λh changes accuracy by at most 1.12% on CIFAR10 but up to 9.75% on MNIST, with λh = 0.75 or λh = 0.5 generally balancing accuracy and communication.For 1000 MNIST clients, λh = 0.5 achieves the highest accuracy.
  • Accuracy–communication trade-off: 8-bit quantization offers a practical accuracy–communication balance, whereas 4-bit quantization causes severe errors and training instability, especially with non-IID data.Cosine and Dynamic maintain similar or slightly better accuracies while reducing total communicated gigabytes.
  • Communication efficiency: 37.46% is the Cosine reduction in total communicated gigabytes versus 32-bit training, while Dynamic reduces communication by 49.54% to 52.64% on MNIST.These reductions are reported across the evaluated client counts and datasets.
  • Large-scale scalability: 52.64% is Dynamic’s MNIST communication reduction and 45.06% its CIFAR10 reduction with 1000 clients, while accuracies remain comparable to full-precision training.Dynamic reaches 65.55% accuracy on MNIST and 59.51% on CIFAR10 in this setting.
  • Practicality: Linear-time entropy calculation and straightforward adaptability make the proposed schedulers fast, efficient, and broadly applicable, although the Dynamic scheduler has additional considerations.The supplied passage introduces a limitation but does not provide its complete description.

E. PERFORMANCE ON MEDICAL IMAGE DATASETS

On non-IID medical image datasets, adaptive quantization substantially reduces communication versus 32-bit training while retaining competitive F1 and BACC, but fixed 16-bit quantization can sometimes perform better. Adaptive methods offer flexibility, yet their effectiveness depends on balancing noise and precision.

  • Communication and performance: 37% communication reduction on PAP-Smear was achieved by Dynamic, with F1 decreasing from 90.94% to 89.41% and BACC from 91.03% to 89.59%.The comparison uses the 32-bit baseline.
  • Communication and performance: 31% communication reduction was reported for Pneumonia with Cosine while preserving near-baseline performance.The passage identifies Cosine as the method achieving near-baseline performance, but the supplied excerpt does not include its F1 value.
  • Communication and performance: Approximately 39% communication reduction on BreakHisV1 was achieved by Dynamic, with F1 decreasing from 89.80% to 87.03% and BACC from 90.64% to 88.36%.The comparison uses the 32-bit baseline.
  • Trade-offs and limitations: 16-bit static quantization sometimes outperformed Cosine and Dynamic in both model performance and total communicated gigabytes, including on PAP-Smear.The passage attributes this trade-off to fixed, moderate precision.
  • Trade-offs and limitations: Adaptive quantization adjusts bit-lengths according to training dynamics and client importance, but performance variability increases with interacting client number and diversity.The supplied passage states that careful management of noise and precision trade-offs is required.

V. CONCLUSION

The paper combines adaptive quantization with Laplacian-based differential privacy to improve communication efficiency and privacy in non-IID federated learning. Experiments on medical imaging datasets showed reduced communication overhead while maintaining competitive accuracy, while future work targets improved client-importance estimation and stronger privacy mechanisms.

  • Contributions: The proposed federated learning approach combines adaptive quantization with Laplacian-based differential privacy across varying client counts and non-IID datasets.Laplacian-based privacy is described as relatively underexplored in federated learning and offering tighter guarantees than Gaussian-based privacy.
  • Experimental Results: 31% to 37% communication-overhead reduction was achieved on PAP-Smear, Pneumonia, and BreakHisV1 while maintaining competitive accuracy versus 32-bit float settings.These medical image datasets pose challenges due to complex image structures and critical classification requirements.
  • Future Work: Future work will refine client-importance estimation and investigate secure multiparty computation to further balance communication efficiency, model performance, and privacy guarantees.The planned enhancements target large-scale federated learning.
Loading 2604.23426v1…