PaperlayerView exact revision on arXiv

Source-linked AI summary

FedMix: Approximation of Mixup under Mean Augmented Federated Learning

Tehrim Yoon, Sumin Shin, Sung Ju Hwang, Eunho Yang

arXiv:2107.00233v1cs.LGcs.AIcs.CVcs.DC

TL;DR

Federated learning needs better performance under heterogeneous, non-iid client data without exposing raw local data. The paper proposes MAFL, which exchanges averaged data, and FedMix, which approximates global Mixup from those averages; the methods improve benchmark performance, particularly in non-iid settings. MAFL nevertheless introduces privacy risks because local distributions or data ownership may be inferred from averaged data.

  • Problem

    Federated-learning performance degrades with heterogeneous non-iid client data, while directly accessing external raw data conflicts with privacy constraints.

  • Method

    MAFL exchanges model parameters and averaged local data, while FedMix approximates global Mixup’s loss through Taylor expansion using averaged external data.

  • Results

    FedMix improves performance over existing algorithms across federated-learning benchmarks, particularly in non-iid environments, with smaller accuracy drops under greater heterogeneity or fewer updating clients.

  • Takeaways & Limitations

    Averaged-data exchange enables Mixup-based cross-client augmentation without accessing other clients’ raw data, supporting improved non-iid federated learning while preserving privacy.

  • Takeaways & Limitations

    Averaged data can allow local distributions, individual data, ownership, or changes over time to be inferred, especially when averages contain too few instances or client-specific information.

Abstract

from arXiv · show

Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device, thus preserving privacy and eliminating the need to store data globally. While there are promising results under the assumption of independent and identically distributed (iid) local data, current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases. To resolve this issue, we propose a simple framework, Mean Augmented Federated Learning (MAFL), where clients send and receive averaged local data, subject to the privacy requirements of target applications. Under our framework, we propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup, but does not require local raw data to be directly shared among devices. Our method shows greatly improved performance in the standard benchmark datasets of FL, under highly non-iid federated settings, compared to conventional algorithms.

1 INTRODUCTION

Federated learning struggles with heterogeneous client data, motivating MAFL, which exchanges averaged data to preserve privacy while reducing cross-client bias. FedMix approximates privacy-violating global Mixup using averaged data and improves performance in difficult non-iid settings.

  • Federated learning algorithms commonly average client-trained model parameters, but non-iid client data makes efficient training harder.FedAvg averages model parameters element-wise, while data heterogeneity across clients remains a major practical challenge.
  • Partial client participation can exacerbate heterogeneity in weight updates across clients, worsening the non-iid challenge.This issue arises even though partial participation has little effect on test performance in ideal iid settings.
  • MAFL exchanges model parameters alongside averaged local data, allowing a privacy-information trade-off and enabling Mixup with external averaged data.The amount of information and privacy depends partly on how many data instances contribute to each average.
  • FedMix approximates global Mixup’s loss function through Taylor expansion using averaged data from other clients rather than their raw data.The method is designed to retain the effect of global Mixup without direct access to private external examples.
  • FedMix shows better performance and smaller accuracy drops as data heterogeneity increases or fewer clients update per communication round.The evaluation focuses on standard federated-learning benchmark datasets and increasingly difficult non-iid settings.

2 RELATED WORK

Related work establishes FedAvg and its variants as parameter-averaging approaches, while non-iid data can produce suboptimal performance. Existing alternatives face privacy or baseline-improvement limitations, motivating privacy-preserving Mixup adaptations.

  • FedAvg and related federated-learning methods build on averaging model parameters updated by individual clients.FedAvg weights element-wise parameter averages by client dataset size, while FedProx adds a proximal objective term.
  • Non-iid client distributions can yield suboptimal performance, and convergence guarantees do not ensure iid-level performance.The related-work discussion distinguishes convergence from achieving performance comparable to iid settings.
  • Prior approaches address non-iid federated learning with limitations including privacy violation through partial local-data sharing or no demonstrated improvement over FedAvg.These limitations motivate methods that improve non-iid performance without compromising privacy.
  • Mixup generates additional data by linearly interpolating actual instances and has been applied across image, speech, and language tasks.Related extensions include latent-space Manifold Mixup and region-replacement CutMix.
  • Federated Mixup studies have sent blended data to servers, but doing so can create privacy risks and large communication costs.The paper instead modifies Mixup for federated-learning constraints and non-iid clients.

3 MEAN AUGMENTED FEDERATED LEARNING (MAFL) AND FEDMIX

MAFL augments federated learning by exchanging averaged client data alongside model parameters, with privacy and communication controlled by averaging choices. FedMix approximates global Mixup using averaged data and Taylor expansion, targeting non-iid settings without sharing raw data.

  • NaiveMix: NaiveMix combines local examples with other clients’ averaged inputs and labels, extending Mixup without directly accessing external raw data.The mixed loss uses a local pair and the mean input-label pair from another client.
  • MAFL: MAFL exchanges model parameters and averaged local data between clients and the server, extending the standard federated learning paradigm.Clients compute and send averaged inputs and labels, which the server aggregates and returns to selected clients.
  • MAFL: The averaging batch size M_k controls the privacy–information–communication trade-off: smaller values provide more relevant information but reduce privacy and increase communication.At M_k = 1, raw data is thoroughly exchanged and privacy is not protected.
  • FedMix: FedMix’s input-derivative loss is designed to reproduce global Mixup effects using averaged data from other clients rather than their raw examples.This approach is presented as the more systematic improvement over directly using mashed data.
  • FedMix: FedMix approximates the loss of privacy-insecure global Mixup by applying a Taylor expansion and ignoring the second-order O(λ^2) term.The approximation uses the linearity of the resulting expression to replace multiple external instances with their averaged inputs and labels.
  • Costs and privacy: The framework introduces privacy risks because averaged data may reveal local distributions, while FedMix may require extra memory for input-derivative calculations.Additional communication is considered small relative to model-parameter exchange because input dimensions are typically much smaller than parameter counts.

4 EXPERIMENTS

The experiments evaluate MAFL-based NaiveMix and FedMix against federated baselines across image and language benchmarks, with emphasis on heterogeneous settings. FedMix generally converges faster and maintains stronger performance as heterogeneity, client count, participation, and data conditions vary.

  • Learning curves: FedMix outperforms FedAvg, FedProx, LocalMix, and NaiveMix and converges faster across all tested datasets.The comparison uses learning curves measured by communication rounds.
  • Language benchmark: FedMix and NaiveMix outperform baseline algorithms on the heterogeneous Shakespeare next-character prediction task.Mixup is applied to embeddings because input-level Mixup cannot be performed on discrete character labels.
  • Varying federated settings: FedMix maintains superior performance across varying numbers of clients, local data amounts, and local training epochs.The paper reports these comparisons in Tables 3 and 4 and Appendix F.
  • Mixup ratio: As λ increases, global Mixup improves while FedMix’s approximation becomes less accurate; high λ decreases performance in both methods.The authors attribute the decrease to overrepresentation of external, out-of-distribution data during local updates.
  • Mean computation: Using all local data to compute each mean provides privacy and good performance, while reducing M_k increases computation burden and weakens privacy preservation.Across FEMNIST and CIFAR10, enhanced privacy produces only a small performance decline.
  • Hidden-state Mixup: Hidden-state Mixup variants do not meaningfully advance over FedMix with input Mixup and impose additional communication burden.The comparison is discussed for the language-model and hidden-state Mixup variants.
  • Non-iidness and participation: FedMix outperforms other methods and suffers less performance decline as CIFAR10 client heterogeneity increases or fewer clients train per round.The fewer-client setting reduces communication burden at a performance cost, with MAFL-based methods showing less decline.

5 CONCLUSION

The paper concludes that MAFL exchanges averaged local data to provide relevant cross-client information while preserving privacy. FedMix approximates global Mixup using averaged data and improves benchmark performance, particularly in heterogeneous settings, while broader non-iid environments and tasks remain future work.

  • Conclusion: MAFL exchanges averaged local data to gain relevant information while preserving privacy.The framework is presented as a novel approach for federated learning.
  • Conclusion: FedMix approximates global Mixup using only averaged data rather than accessing other clients’ raw data.The method is developed within the MAFL framework.
  • Conclusion: MAFL, particularly FedMix, improves performance over existing algorithms across benchmarks, especially in heterogeneous non-iid environments.The conclusion identifies heterogeneous client distributions as the setting where the improvement is particularly observed.
  • Future work: Future work should address varied non-iid environments, stronger privacy, and tasks beyond image classification.These are stated as directions for extending the method.

B EXPERIMENTAL DETAILS

The experiments use federated benchmarks with heterogeneous client partitions, dataset-specific architectures, and fixed training hyperparameters. The section also outlines the mathematical proof strategy for FedMix under a small-λ approximation.

  • Datasets: FEMNIST assigns each client data from one writer within a 62-class handwritten-character dataset, creating a realistic non-iid setting.
  • Models and training: Client training uses dataset-specific LeNet, modified VGG, or two-layer LSTM architectures, with SGD learning rate 0.01, decay 0.999, and batch size 10.
  • Datasets: CIFAR10 and CIFAR100 use class-skewed client partitions, with each client receiving data from 2 or 20 randomly chosen classes, respectively.
  • Datasets: Shakespeare partitions conversations by speaking role across 60 clients, and applies Mixup at the embedding layer because character labels are discrete.
  • Proof setup: FedMix's proof starts from cross-entropy loss linearity in y, then uses independent Taylor expansions under λ ≪1 and discards O(λ2) terms.

D COMPARISON OF FEDMIX WITH BASELINE MIXUP SCENARIOS

The comparison tests whether MAFL's averaged data contributes relevant information beyond generic augmentation. FedMix outperforms substitutes based on random noise or locally generated images.

  • Baseline comparison: The comparison is framed as a test of whether MAFL gains reflect information received from other clients rather than data augmentation alone.
  • Baseline comparison: FedMix retains higher performance than Mixup using randomly generated noise or locally generated images as averaged data.
  • Figure 4: Figure 4 reports learning curves for hidden-state Mixup variants after different layer depths, including comparisons among FedMix variants.

E VARIANT OF FEDMIX AND NAIVEMIX WITH MIXUP BETWEEN HIDDEN STATES

Hidden-state variants of FedMix and NaiveMix are evaluated as alternatives to input Mixup. They improve over existing methods but incur communication, computation, and privacy trade-offs that favor input Mixup.

  • Design trade-offs: Fixing the hidden-state layer k reduces communication because randomized layer selection would require clients to send multiple large hidden states.
  • Design trade-offs: Hidden-state Mixup cannot backpropagate through the complete encoder-decoder graph in a typical federated setting, so vanilla SGD updates are added after decoder-only Mixup.
  • Privacy and cost: Hidden representations may permit model inversion when clients know encoder weights, so hidden-state Mixup does not ensure privacy, unlike decoder-phase application.
  • Results: Hidden-state FedMix and NaiveMix improve over existing methods but do not outperform FedMix based on input Mixup on CIFAR10.
  • Results: With hidden-state Mixup, k = 4 learns fastest, while k = 4 and k = 2 converge similarly and both are slightly below input Mixup.
  • Privacy and cost: Input Mixup is preferred because hidden states are larger than raw inputs and must be communicated every round as encoder weights change.
  • Local epochs: Across tested local-epoch values, MAFL-based methods outperform existing algorithms, while FedMix generally performs best except for E = 10.

G ADDITIONAL COMPUTATION COST INCURRED BY MAFL

FedMix adds gradient-related computation and increases memory use, but its training time is comparable to FedAvg. The main memory increase comes from doubling the Mixup input dimension.

  • Computation: FedMix's loss includes additional terms such as the gradient by input compared with vanilla FedAvg.
  • Training time: FedMix reaches 70% CIFAR10 accuracy in 1.94 hours versus 1.95 hours for FedAvg under the same main-experiment settings.
  • Memory: FedMix requires about twice the GPU memory allocation of FedAvg because Mixup doubles the input dimension.
  • Memory: FedAvg uses 46.00MB, while LocalMix uses 94.00MB and FedMix uses 98.00MB in the reported allocation comparison.

H INTRODUCTION OF CUT-OFF THRESHOLD IN MAFL

The study evaluates privacy-oriented cut-off thresholds for averaged data in MAFL, using FEMNIST clients with heterogeneous local data sizes. A moderately small threshold performs best, while varying λ has limited impact when client data sizes differ.

  • Threshold and λ evaluation: N = 300 clients are used to test multiple cut-off threshold levels and FedMix λ values on FEMNIST.The procedure is designed for settings with heterogeneous local client-data sizes.
  • Threshold results: A threshold of 100 produces the best performance among the tested threshold levels.The authors report that threshold changes do not hugely affect performance overall.
  • Threshold trade-off: Increasing the threshold reduces overfitting to clients with small local datasets but also decreases the averaged data each client receives.The threshold therefore trades off protection against small-client effects and the amount of exchanged averaged data.
  • Effect of client data size: Changing λ causes little performance variation when clients have different data amounts, especially compared with the corresponding equal-size-client results.Individual-client test accuracy also shows no noticeable pattern by local data size.

I MAFL IN CONJUNCTION WITH GAUSSIAN NOISE

The paper examines Gaussian noise as an additional privacy measure for averaged data in MAFL. Noise generally causes a small performance decline, while a larger noise level can also provide an augmentation effect, and differential-privacy integration remains future work.

  • Privacy motivation: Small averaged-data groups may create privacy concerns despite providing little performance improvement.The paper motivates adding Gaussian noise as an additional privacy measure.
  • Noise evaluation: Gaussian noise is evaluated by varying its standard deviation σ in FedMix experiments.The corresponding results are reported for FedMix with Gaussian noise.
  • Performance effect: Gaussian noise generally reduces performance, although the reported decline is very small.The performance effect is documented in Table 14.
  • Privacy scope: The method does not directly align with differential privacy, and combining FedMix smoothly with DP-related methods is left for future work.The paper also notes that noise alone does not guarantee differential privacy without linking its level to ε and δ.

J ADDITIONAL EXPERIMENTS: VARIATIONS OF FEDMIX

Additional experiments vary averaging, data splitting, Mixup ratios, and label-skew conditions. FedMix can benefit from moderate extra averaging and remains strongest under label skew, while same-class splitting and excessive averaging hurt performance.

  • Additional averaging: Moderate m-fold extra averaging increases performance, but performance quickly declines as m becomes too large.Extra averaging extends the viable range of M_k beyond n_k while reducing the number of averaged entries.
  • Data splitting: Same-class splitting causes a significant performance decline compared with random splitting at M_k = 0.5n_k.The authors find no advantage for same-class splitting over the random split used in the main results.
  • NaiveMix ratio: NaiveMix has an intermediate optimal λ, but its performance drop at λ = 0.5 is more dramatic than FedMix’s.The paper attributes this behavior to NaiveMix assigning more weight to averaged data, especially for large M_k.
  • Label-skew heterogeneity: FedMix outperforms all other tested algorithms under label-skewed heterogeneity generated with Dirichlet α = 0.2 and 0.5.The experiments use uneven label distributions without limiting the number of classes per client.
Loading 2107.00233v1…