PaperlayerView exact revision on arXiv

Source-linked AI summary

FedFed: Feature Distillation against Data Heterogeneity in Federated Learning

Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, Bo Han

arXiv:2310.05077v1cs.LG

TL;DR

Federated learning struggles with heterogeneous client data, while sharing information creates a privacy–performance dilemma. FedFed shares performance-sensitive features while retaining performance-robust features locally, and experiments show improved accuracy and robustness under varied heterogeneity settings.

  • Problem

    Data heterogeneity causes unstable convergence and degraded prediction accuracy, while sharing client information raises privacy concerns and may reduce performance when noise is added.

  • Method

    FedFed partitions data into performance-sensitive features shared across clients and performance-robust features kept locally for training with local and shared data.

  • Results

    FedFed consistently improves model accuracy across evaluated settings, with improvements of at most 40.67% reported for CIFAR-10 and CIFAR-100.

  • Takeaways & Limitations

    FedFed makes FL models more robust to data heterogeneity, including different Non-IID partition strategies.

  • Takeaways & Limitations

    FedFed adds communication and storage overheads and leaves potential privacy risks for future investigation.

Abstract

from arXiv · show

Federated learning (FL) typically faces data heterogeneity, i.e., distribution shifting among clients. Sharing clients' information has shown great potentiality in mitigating data heterogeneity, yet incurs a dilemma in preserving privacy and promoting model performance. To alleviate the dilemma, we raise a fundamental question: \textit{Is it possible to share partial features in the data to tackle data heterogeneity?} In this work, we give an affirmative answer to this question by proposing a novel approach called {\textbf{Fed}erated \textbf{Fe}ature \textbf{d}istillation} (FedFed). Specifically, FedFed partitions data into performance-sensitive features (i.e., greatly contributing to model performance) and performance-robust features (i.e., limitedly contributing to model performance). The performance-sensitive features are globally shared to mitigate data heterogeneity, while the performance-robust features are kept locally. FedFed enables clients to train models over local and shared data. Comprehensive experiments demonstrate the efficacy of FedFed in promoting model performance.

1 Introduction

Federated learning suffers from data heterogeneity, while sharing information to address it creates a privacy–performance dilemma. FedFed asks whether sharing only performance-sensitive features can mitigate heterogeneity while keeping performance-robust information local.

  • Non-IID client data and diverse computing capabilities cause unstable convergence and degraded prediction accuracy in federated learning.
  • Information sharing can mitigate heterogeneity, but sharing raw, synthesized, or statistical information raises privacy concerns, while noise can degrade performance.
  • FedFed asks whether partial feature sharing can resolve this dilemma by keeping performance-robust features local and sharing performance-sensitive features across clients.
  • The approach is motivated by information bottleneck ideas, where features discarded from generalization-oriented representations may contain private information unnecessary for heterogeneity mitigation.

Q.3 What if performance-sensitive features contain private information?

FedFed protects shared performance-sensitive features by adding noise before constructing a global dataset. It combines partial feature sharing with differential privacy so clients can train on local and shared data.

  • FedFed decomposes data into performance-robust and performance-sensitive features, then shares protected sensitive features while retaining robust features locally.
  • FedFed deploys feature sharing across FedAvg, FedProx, SCAFFOLD, and FedNova under varied client counts, heterogeneity levels, and four datasets.
  • Its contribution is to share partial features with random noise, enabling models to train over local and shared data while targeting convergence and generalization improvements.
  • Differential privacy quantifies privacy preservation through ε and δ, with smaller ε representing a stronger privacy guarantee.

3 Methodology

FedFed uses information-bottleneck-inspired feature distillation to separate locally retained performance-robust features from globally shared performance-sensitive features. Noise protection and composition analysis address privacy during sharing.

  • FedFed shares minimal sufficient features while keeping other features at clients, combining feature partitioning with information-bottleneck principles.
  • 3.1 Motivation: A valid partition separates X into independent parts, with performance-sensitive features containing label information and performance-robust features containing the remaining data information.
  • 3.2 Feature Distillation: Feature distillation learns performance-sensitive features that predict labels with minimal ℓ2-norm, while a generative model produces performance-robust features.
  • 3.2 Feature Distillation: FedFed shares x − q(x; θ) as performance-sensitive features and trains models using both local private data and the shared data.
  • 3.3 Protection for Performance-Sensitive Features: Because sensitive and robust features can overlap, FedFed adds noise before sharing and constructs a global dataset from protected features.
  • 3.3 Protection for Performance-Sensitive Features: For identical ε and δ, FedFed requires a smaller noise scale than raw-data sharing, with σ proportional to ∥x_s∥2, and provides a composed FL privacy guarantee.

4 Experiments

FedFed is evaluated across datasets, federated algorithms, heterogeneity settings, noise levels, and privacy attacks. Results generally show improved accuracy and convergence, while gains are limited in settings near performance bottlenecks and extra overheads remain a limitation.

  • 4.2 Main Results: β-VAE performs better than the ResNet generator for FedFed, so the remaining experiments mainly use β-VAE.The comparison is shown in Figure 2(a).
  • 4.2 Main Results: FedFed consistently improves model accuracy and convergence across CIFAR-10, CIFAR-100, FMNIST, and SVHN under varied settings.The evaluation covers four datasets and applies FedFed to multiple federated learning algorithms.
  • 4.2 Main Results: FedFed accelerates convergence for different federated algorithms, including a notable gain on FMNIST with α = 0.05, E = 1, K = 10.Additional convergence-rate results are provided in the appendix.
  • 4.2 Main Results: FedFed can incur limited gains when existing methods approach a performance bottleneck, and full-data sharing with differential privacy degrades FL performance.The authors attribute the latter degradation to the larger noise required to protect performance-sensitive features in the full data.
  • 4.2 Main Results: Up to 40.67% improvement is observed with 100 clients on CIFAR-10 and CIFAR-100, while FedFed also improves robustness across Non-IID partitions.The authors attribute the improvement to replenishing missed data knowledge and report similar prediction accuracies across paired methods.
  • 4.3 Sensitivity Study: Prediction accuracy decreases as noise increases, while Laplacian-noise experiments support FedFed's robustness to the noise distribution.The noise study examines privacy protection and prediction accuracy under different noise levels.
  • 4.4 Privacy Analysis: Model-inversion and membership-inference experiments indicate protection of globally shared data, with σ2 = 0.3 on raw data achieving protection similar to σ2 = 0.15 on shared data.The reported comparison is consistent with Theorem 3.3.
  • 6 Conclusion: FedFed introduces communication and storage overheads, and the authors identify unresolved privacy concerns requiring future exploration.A hardware-friendly version and cryptographic integration are proposed as future directions.

B.1 Implementation Details

The implementation specifies parameters, sampling settings, and how FedFed is integrated with several federated learning algorithms.

  • B.1 Implementation Details: The experiments tune learning rates from {0.0001, 0.001, 0.01, 0.1} and report the best result with its corresponding rate.The stated default learning rate is 0.01, with additional dataset- and algorithm-specific settings.
  • B.1 Implementation Details: Batch size is 64 for K = 10 and 32 for K = 100, while aggregation selects 5 and 10 clients, respectively.These choices correspond to sampling rates of 50% and 10%.
  • B.1 Implementation Details: Table 7 lists the paper’s experimental parameters, and “#” denotes a number while “/” indicates tried symbolic values.The table is identified as the source for the parameter values used throughout the paper.
  • B.1 Implementation Details: FedFed is deployed with FedAvg, FedProx, SCAFFOLD, and FedNova, with algorithm-specific local objectives or control mechanisms.FedAvg and FedProx differ in their local objective functions; SCAFFOLD and FedNova integrations are also described.

C Full Analysis on Information Theory Perspective

The information-theoretic analysis defines the two feature types and formulates FedFed as an information-bottleneck-style compression problem integrated into federated training algorithms.

  • C Full Analysis on Information Theory Perspective: FedAvg, FedProx, SCAFFOLD, and FedNova versions train clients using local data together with shared data, then aggregate client updates.The supplied pseudocode shows server initialization, client sampling, model communication, local epochs, and algorithm-specific update equations.
  • C Full Analysis on Information Theory Perspective: Performance-sensitive features contain label information, whereas performance-robust features contain the remaining data information.The partition is intended to preserve the information of the original variable without adding or losing information.
  • C Full Analysis on Information Theory Perspective: FedFed defines z as the residual after feature extraction, z = x − q(x; θ), and optimizes predictive information in that residual.The objective uses expected negative log-likelihood and constrains ||x − q(x; θ)||2^2 ≤ ρ.
  • C Full Analysis on Information Theory Perspective: FedFed’s information-bottleneck formulation minimizes I(X; Y | Z) subject to a complexity constraint on Z, equivalently expressed through entropy.The analysis uses information entropy H(·) and mutual information I(·; ·).

D Full Analysis on Differentially Private Features

The privacy analysis establishes differential-privacy conditions for sharing raw features and performance-sensitive features under FedFed’s feature-partitioning framework.

  • D Full Analysis on Differentially Private Features: The security analysis treats client-level differential privacy and the overall FedFed privacy guarantee as two separate steps.The derivation invokes prior work and obtains the FedFed result by taking σ_r → ∞.
  • D Full Analysis on Differentially Private Features: Lemma D.1 states an (ϵ, δ)-DP condition for sharing raw features x.The supplied passage introduces the corresponding privacy bound without displaying its complete expression.
  • D Full Analysis on Differentially Private Features: Lemma D.2 states an (ϵ, δ)-DP condition for sharing performance-sensitive features x_s.This bound is the privacy condition analyzed for FedFed’s selectively shared features.

D.1 Analysis and Proof for Theorem 3.3

Theorem 3.3 compares FedFed with raw-data sharing under equal privacy guarantees and attributes FedFed’s advantage to its lower required noise scale.

  • D.1 Analysis and Proof for Theorem 3.3: FedFed adds differential-privacy noise to x_s while keeping x_r locally, effectively treating the unreleased component as having σ_r = ∞.The local component is not exposed to an adversary, while ℓ2-norm clipping is used for x_s.
  • D.1 Analysis and Proof for Theorem 3.3: The residual-feature norm satisfies (1 − ρ)||x|| ≤ ||x_r|| ≤ (1 + ρ)||x|| when ||x_s|| = ρ||x||.This relation supports the subsequent comparison between noise levels for selective and raw-data sharing.
  • D.1 Analysis and Proof for Theorem 3.3: The analysis assumes data are rescaled into [0, 1] and clips each sample so its norm is bounded by ρM.It also assumes the full-data norm is less than M > 0.
  • D.1 Analysis and Proof for Theorem 3.3: For equal noise scales, raw-data sharing has greater privacy loss than FedFed; for equal privacy, FedFed can use less noise.The analysis links this difference to improved model performance under identical privacy guarantees.
  • D.1 Analysis and Proof for Theorem 3.3: Given identical ϵ and δ, FedFed requires σ < σ′, where σ and σ′ are the noise scales for sensitive-feature and raw-data sharing.The theorem states that FedFed’s noise scale is proportional to the norm of x_s.

D.2 Analysis and Proof for Theorem 3.4

The analysis applies privacy-loss and composition tools to derive FedFed’s overall differential-privacy guarantee. It concludes that FedFed uses different noise levels for performance-sensitive and performance-robust features while attaining stronger security and higher model performance.

  • Composition analysis: FedFed’s composition analysis combines client-side (ε_k, δ)-DP mechanisms to characterize overall privacy across repeated accesses.The proof models privacy regions and composes the mechanisms’ output distributions across clients.
  • Privacy definitions: The proof defines sensitivity as the maximum norm difference of a query over neighboring datasets and uses privacy loss to analyze randomized mechanisms.Sensitivity is defined with an L1 or L2 norm, while privacy loss is associated with the accumulated random noise of the mechanism.
  • Privacy-region analysis: FedFed’s privacy-region analysis assumes neighboring datasets, output distributions, and a permutation relating the two distributions.The resulting privacy region is expressed through divergence-style constraints over measurable output subsets.
  • Theorem conclusion: The theorem analysis concludes that FedFed protects performance-sensitive features with small noise and performance-robust features with extremely large noise.This design is reported to provide higher model performance and stronger security simultaneously.

E More Results on Attack

The attack experiments examine model inversion and membership inference under different attackers and shared-information choices. They report that partial feature sharing with differential privacy reduces leakage relative to raw-data sharing, while shared features remain unrecoverable in the generative attack setting.

  • Model inversion: White-box inversion results show that performance-sensitive features xs can leak privacy, supporting the need for DP protection on xs.The globally shared data is represented as xp = xs + n with Gaussian noise.
  • Generative model inversion: A client-side generative inversion attacker performs better than a server-side attacker but still cannot recover data from the shared features.The attack uses globally shared data as auxiliary data and local private data as public data.
  • Attack protocol: The attack evaluation treats membership inference as binary classification using top-k shadow-model outputs and performs attacks every 10 communication rounds.The global model is the target model and the shadow model is trained with globally shared data.
  • Membership inference: Partial-data sharing produces lower membership-inference leakage than raw-data sharing under the same DP level.FedFed achieves comparable protection with a relatively small noise σ.

F.4 More experimental results

Additional experiments evaluate FedFed across datasets, sampling rates, heterogeneity levels, DP noise, and feature generalization. They report consistent accuracy and convergence improvements, with benefits diminishing as client data become more homogeneous and increasing noise trading performance for privacy.

  • Sampling-rate analysis: Higher client sampling rates improve model performance and accelerate convergence on CIFAR-10.The paper attributes this to more participating clients producing more consistent aggregated update directions.
  • Heterogeneity analysis: FedFed’s performance enhancement decreases as α increases because client data become less heterogeneous, becoming much less effective when α reaches 1.0.The corresponding comparisons are reported for α = 0.5 and α = 1.0.
  • DP-noise analysis: Increasing DP noise decreases the privacy budget and model performance while increasing information protection.The trade-off is evaluated across additional datasets in Figure 16.
  • Feature generalization: Classifiers trained on performance-sensitive features retain generalization ability comparable to classifiers trained on raw data, unlike performance-robust features.The feature-type comparisons use classifiers trained and tested across raw, performance-sensitive, and performance-robust features.

F.7 Overheads Analysis of FedFed

FedFed adds computation, communication, and memory overheads, but the generator is substantially cheaper than the classifier and generated data are sent only once. The main practical limitation is the storage and communication burden of the globally shared dataset.

  • Training time: The generator takes less than 10% of the classifier’s training time.The comparison is reported in Table 13.
  • Computation and rounds: The generator requires approximately 8.2% of the classifier’s computation and 15 communication rounds versus 1000 for the classifier.These figures quantify the generator’s lower model complexity and communication-round requirement.
  • Memory and bandwidth: The globally shared dataset can require substantially more memory than a classifier, such as 586MB versus 48MB for CIFAR-10.The dataset’s communication is equivalent to sending a classifier for approximately 14 communication rounds.
  • Communication overhead: Extra communication costs are approximately 4.54% when K = 10 and β = 50%, and 22.07% when K = 100 and β = 10%.Both examples use T_d = 15 and T_r = 1000.
  • Deployment strategies: FedFed offers one-time and partial-download strategies to trade storage requirements against communication cost.The partial-download strategy uses storage comparable to local private data but incurs approximately 1.5× the one-time strategy’s communication cost.

G.1 Federated Learning with Heterogeneous Data.

Federated learning protects data access across clients but suffers performance degradation when client distributions are heterogeneous. Existing remedies adjust local training, aggregation, or shared information, yet data sharing introduces privacy concerns.

  • Heterogeneous client distributions make FedAvg diverge from centralized training, with the gap potentially accumulating during aggregation and hurting performance.
  • Existing approaches address heterogeneity through regularization, normalized weighting, control of prior knowledge, contrastive learning, or revised server aggregation.
  • Sharing client data can mitigate drift by replenishing missing information, but synthesized data may remain similar to raw data and create privacy leakage.
  • Information-sharing methods transmit or generate alternative signals—including logits, intermediate activations, batch-normalization statistics, generators, and unlabeled-data distillation—to transfer knowledge across clients.
  • Differential privacy can reduce the risk that an adversary identifies whether a client’s data was used, although it may reduce utility.
Loading 2310.05077v1…