PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning on Non-IID Data: A Survey

Hangyu Zhu, Jinjin Xu, Shiqing Liu, Yaochu Jin

arXiv:2106.06843v1cs.LGcs.DC

TL;DR

Federated learning must preserve privacy while learning from decentralized data, yet Non-IID client distributions produce local-model divergence and performance degradation. This survey analyzes those effects across parametric and non-parametric models in horizontal and vertical FL, reviews mitigation techniques and their trade-offs, and identifies future directions. It concludes that Non-IID data mainly affects parametric models in horizontal FL, while complex models such as DNNs are more sensitive to client distributions.

  • Problem

    Non-IID data can cause federated-learning accuracy deterioration through local-model weight divergence, while existing reviews had not examined these effects comprehensively.

  • Method

    The paper provides a comprehensive survey categorizing Non-IID distributions, analyzing their effects across model types and FL settings, and reviewing mitigation techniques.

  • Results

    Non-IID distributions mainly affect parametric models in horizontal FL, and complex models such as DNNs are more sensitive to client data distributions.

  • Takeaways & Limitations

    Understanding Non-IID effects requires distinguishing horizontal from vertical FL and considering both model class and client data distributions.

  • Takeaways & Limitations

    Federated neural architecture search remains sparsely studied, with little work considering the influence of Non-IID distributions.

Abstract

from arXiv · show

Federated learning is an emerging distributed machine learning framework for privacy preservation. However, models trained in federated learning usually have worse performance than those trained in the standard centralized learning mode, especially when the training data are not independent and identically distributed (Non-IID) on the local devices. In this survey, we pro-vide a detailed analysis of the influence of Non-IID data on both parametric and non-parametric machine learning models in both horizontal and vertical federated learning. In addition, cur-rent research work on handling challenges of Non-IID data in federated learning are reviewed, and both advantages and disadvantages of these approaches are discussed. Finally, we suggest several future research directions before concluding the paper.

1. Introduction

Federated learning trains a global model across devices while keeping user data local, but Non-IID data creates local-model divergence and degrades accuracy. This survey addresses the limited detailed analysis of Non-IID effects by reviewing distributions, impacts, mitigation techniques, and future challenges.

  • Motivation: Federated learning trains models across devices while keeping user data locally stored instead of uploading sensitive data to a centralized server.This reduces direct exposure of local data, although federated learning still faces communication, device, and security challenges.
  • System challenges: Federated learning also must address communication costs, device computing and energy constraints, heterogeneous participation, and adversarial threats.Existing research targets these challenges through communication reduction, hardware-aware methods, and additional privacy protections.
  • Non-IID challenge: Non-IID data causes local models initialized identically to converge toward different parameters, increasing global-model divergence and slowing convergence.The resulting divergence contributes to an almost inevitable accuracy deterioration relative to centralized learning on heterogeneous data.
  • Survey gap: Existing reviews had not examined the impact of Non-IID data in federated learning in comparable detail.Earlier work included general FL reviews and a brief discussion of personalization approaches, but not a comprehensive treatment of Non-IID effects.
  • Survey scope: The survey categorizes Non-IID distributions, analyzes their effects on model aggregation, reviews mitigation techniques and trade-offs, and identifies future research directions.Its scope covers parametric and non-parametric models in both horizontal and vertical federated learning.

2. Federated Learning

Federated learning partitions data across clients horizontally by samples or vertically by features, then uses distinct coordination procedures to train models without centralizing raw data. Horizontal FedAvg aggregates weighted local models, whereas vertical learning exchanges local outputs and can match centralized parametric-model performance in principle.

  • Federated learning types: Federated learning is categorized as horizontal or vertical according to whether clients differ mainly in sample space or feature space.Horizontal FL uses shared features with different samples, while vertical FL uses shared samples with different features.
  • Horizontal Federated Learning: FedAvg selects participating clients, sends them the global model, performs local updates, and returns models for weighted aggregation.The aggregation weights each local model according to its client data size.
  • Horizontal Federated Learning: Horizontal FL keeps training data on client devices and communicates global and local model parameters, but repeated parameter exchange consumes communication resources.Client subsampling, quantization, and reduced deep-layer communication are proposed to lower this cost.
  • Privacy protection: Uploaded model parameters or gradients can still leak private data information despite raw data remaining inaccessible to third parties.Homomorphic encryption, differential privacy, and secure aggregation are used as additional protections; differential privacy can reduce model performance.
  • Vertical Federated Learning: Vertical FL assigns labels commonly to one guest client, combines local model outputs to construct the loss, and lacks a central server or shared global model.Clients must align batch sample identities, while local models corresponding to separate feature spaces remain local.
  • Vertical Federated Learning: Parametric models in vertical FL should in principle match centralized performance because their computed loss function is the same.Communication is often lower than in horizontal FL, though it can exceed horizontal FL when the data size is extremely large.

3. Categories of Non-IID Data

Non-IID data arises when client distributions differ, including differences in features, samples, or shared attributes. Attribute skew ranges from mutually exclusive features to partially overlapping features whose distributions may be consistent or inconsistent.

  • Non-IID data means the local distribution P_k(x, y) differs across clients.
  • Attribute skew: Attribute skew occurs when feature distributions P_k(x) differ across clients, with attributes that may be non-overlapping, partially overlapping, or identical.
  • Non-overlapping attribute skew: Non-overlapping attribute skew assigns mutually exclusive features to clients and can represent vertical FL when samples and labels align across clients.
  • Partial overlapping attribute skew: In partial overlapping attribute skew, clients share some features, whose distributions may be consistent and therefore not enlarge Non-IID divergence.
  • Partial overlapping attribute skew: Inconsistent distributions of shared attributes add Non-IID divergence, while horizontal FL partitions data by samples across clients.

3.2. Label skew

Label skew describes client-to-client differences in label distributions or labeling preferences. It includes fixed-class label imbalance, Dirichlet-based distribution imbalance, and differing annotations for shared samples, while crowdsourcing can introduce noisy or missing labels.

  • Label skew occurs when label distributions differ across clients through label distribution skew or label preference skew.
  • Label distribution skew: Label distribution skew changes P_k(y) across clients while sharing P_k(x|y), and includes label size imbalance and label distribution imbalance.
  • Label size imbalance: In label size imbalance, each client owns samples from c fixed classes, with smaller c indicating stronger label imbalance.
  • Label distribution imbalance: Dirichlet label distribution imbalance assigns class instances to clients with probabilities p_c sampled from Dir_k(β), where β controls imbalance.
  • Label preference skew: Label preference skew allows clients to assign different labels to the same input when P_k(y|x) varies despite shared P_k(x).
  • Label preference skew: Crowdsourcing can produce incorrect, noisy, or missing labels because multiple workers may annotate local data.

3.3. Temporal skew

Temporal skew concerns changing data distributions over time, including time differences between clients and clients observing different time periods. It applies to spatio-temporal and time-series data common in real-world FL applications.

  • Temporal skew is variation in time-stamped data distributions P_k(x, y|t) across clients and over time.
  • Webcam example: Webcam clients can exhibit temporal skew because their recordings differ by collection time, even when observations overlap across the full period.
  • Stock-data example: Stock-data temporal skew can result when one client stores the first 60 months while another stores the last 60 months.

3.4. Other scenarios

Other Non-IID scenarios include attribute-and-label skew, differing data types, quantity skew, and model divergence illustrations. Differing data types can make global aggregation extremely difficult when local model structures differ.

  • Some client-distribution scenarios do not fit the previously discussed Non-IID categories.
  • Attribute-and-label skew: Attribute-and-label skew combines different client features with different labels, integrating characteristics of horizontal and vertical FL.
  • Different data types: When client data types differ, such as images versus speech, local model types or structures may be totally different and global aggregation becomes extremely hard.
  • Quantity skew: Quantity skew occurs when the number of training samples varies across clients and can accompany all previously discussed scenarios.
  • Model divergence: Figure 10 illustrates an apparent divergence between the global model and averaged local model in horizontal FL for both IID and Non-IID data.

4. Challenges of Non-IID Data to Model Training

Non-IID data affects model training differently across parametric and non-parametric models and across horizontal and vertical federated learning. Horizontal parametric models experience divergence, while tree models and vertical linear models are described as unaffected by Non-IID data; vertical neural-network training remains an open challenge.

  • Horizontal FL: Parametric Models: Non-IID data causes global model divergence in horizontal federated learning for parametric models.Averaged local parameters can deviate further from the global parameters as local update epochs increase, and divergence may accumulate across communication rounds.
  • Horizontal FL: Parametric Models: Linear models in horizontal federated learning can be trained directly with FedAvg but expose local inputs through communicated gradients.The passage states that homomorphic encryption is usually required to provide additional privacy protection.
  • Horizontal FL: Non-Parametric Models: Non-IID data does not cause training divergence for horizontal federated decision trees because summed gradients and Hessians remain unchanged for each split.Local histograms may differ across clients, but the server aggregates them before selecting split points.
  • Vertical FL: Parametric Models: Non-IID data does not affect vertical federated learning performance for linear models because their logistic-regression loss function matches centralized learning.Vertical federated logistic regression keeps model gradients and training data on local devices, with encrypted outputs used in some schemes.
  • Vertical FL: Parametric Models: Vertical federated neural networks differ from centralized networks because splitting creates separate subnetworks and removes some global connections.Training vertical neural networks remains an open challenge, and existing work has not discussed Non-IID effects on model-learning performance.
  • Vertical FL: Non-Parametric Models: Non-IID data does not influence vertical federated decision-tree performance because impurity values depend on gradients and Hessians rather than client feature distributions.The generated tree model is paired with a lookup table for retrieving locally stored feature and split information.

5. Main Approaches to Handling Non-IID Data

The paper groups approaches for handling Non-IID data in horizontal federated learning into data-based and algorithm-based categories. These approaches address divergence that FedAvg cannot resolve, particularly for complex models such as neural networks.

  • Approach Categories: Horizontal federated-learning methods for Non-IID data are classified into data-based and algorithm-based approaches.The classification targets divergence caused mainly by label-distribution skew, which FedAvg cannot adequately handle for complex models.

5.1. Data based approach

Data-based approaches modify local or shared data distributions to mitigate Non-IID effects. Data sharing and augmentation can improve performance, but commonly introduce privacy or distribution-construction difficulties.

  • Data Sharing: Data sharing warms up and updates client models using a server-held globally shared dataset with a uniform distribution.Clients combine shared global data with their local training data during model updates.
  • Data Sharing: Data sharing can significantly improve Non-IID performance but requires a difficult-to-construct uniform dataset and may violate privacy by distributing global data.The passage describes this privacy conflict as a fundamental shortcoming of the approach.
  • Data Augmentation: Federated data augmentation uses vanilla transformations, mixup, or generative adversarial networks to address local data imbalance.These methods differ in whether they transform local samples, reconstruct balanced data from encoded samples, or train a generator for distribution to clients.
  • Data Augmentation: Vanilla augmentation uses client label counts to generate additional samples for underrepresented classes before local model updates.When a class count falls below the mean, the client generates augmentations according to a server-derived ratio and a tunable augmentation parameter.
  • Data Augmentation: Data augmentation can significantly improve learning on Non-IID data by replenishing imbalanced local datasets, but often depends on data sharing that increases privacy-leakage risk.MultFAug is identified as an approach intended to protect both data privacy and label-distribution information.

5.2. Algorithm based approach

Algorithm-based personalization methods adapt federated models to local tasks by combining local and global information or preserving client-specific components. The survey covers fine-tuning, meta-learning, regularization, personalization layers, multi-task learning, and related approaches for heterogeneous data.

  • Personalization adjusts federated models according to local tasks through fine-tuning, personalization layers, multi-task learning, and knowledge distillation.
  • Local fine-tuning: Local fine-tuning uses the received global model and local data to find a suitable shared initialization while combining local and global information.
  • Local fine-tuning: Per-FedAvg uses MAML to learn an initial global model that enables local clients to obtain good performance with little computation.
  • Local fine-tuning: Per-FedAvg achieves first-order optimality with convergence guarantees and better performance on heterogeneous data than FedAvg, but approximate gradients significantly affect results.
  • Regularization and interpolation: Regularization and interpolation combine local and global information, with regularization minimizing disparity between global and local models.
  • Personalization layers: LG-FEDAVG reports much better local test accuracy and slightly better new test accuracy than FedAvg.
  • Personalization layers: Personalization layers can improve Non-IID learning and reduce communication costs by sharing only base layers, but clients must permanently store their personalization layers.
  • Multi-task learning: Multi-task and curvature-based methods generate separate client models or protect important task parameters to address Non-IID learning challenges.

5.3. System based approach

System-based approaches address Non-IID data by replacing one global model with multiple cluster models and assigning clients according to data or model similarity. These methods improve specialization and flexibility, but require additional training and testing resources.

  • Client clustering: Client clustering constructs multiple model centers by grouping clients with similar local training data instead of forcing one global model to learn all client information.
  • Client clustering: Clustering can use loss similarity or model-weight similarity while keeping clients’ sensitive data distributions private from the server.
  • Loss-based clustering: Loss-based clustering sends multiple global models to clients, which select the model with the smallest local empirical loss before cluster aggregation.
  • Tree-based clustering: Tree-based clustering produces general models near the root and personalized models at deeper layers, allowing model selection or ensembling for different applications.
  • Client clustering: Client clustering is motivated by negative knowledge transfer from aggregating models trained on considerably different data distributions.
  • Client clustering: Multiple cluster models improve scalability and flexibility, but training and testing consume additional computation and communication resources.

6. Remaining Challenges and Future Directions

The survey identifies unresolved challenges in privacy measurement, benchmarking, personalization, vertical FL, and federated neural architecture search, then proposes corresponding research directions.

  • Privacy and benchmarking: Privacy-preserving methods such as data sharing and knowledge distillation can increase privacy exposure, but quantitative measures of leakage remain unavailable.The survey calls for criteria that bound the maximum amount of shared data.
  • Personalized federated learning: Personalized federated learning remains open on system design, deployment, communication reduction, and task adaptation under limited computation and unstable wireless networks.These constraints are especially relevant to IoT edge devices.
  • Vertical federated learning: Vertical FL needs broader investigation of overlapping features, attribute and label skew, different features and labels, and crowdsourcing skew.The survey recommends comparing non-standard and standard Non-IID settings and developing algorithms for different cases.
  • Federated neural architecture search: The influence of Non-IID distributions on federated neural architecture search remains unclear, making it an open future direction.Only limited research on federated NAS has been reported.
  • Privacy and benchmarking: Federated learning lacks universal homogeneous and heterogeneous benchmarks and standard hyperparameter settings for fair evaluation of Non-IID methods.Arbitrary dataset partitioning may not effectively evaluate methods designed for Non-IID data.

7. Conclusion

The paper surveys how Non-IID data affect federated learning and reviews techniques for addressing those effects. It reports trade-offs among convergence, resource use, privacy, and whether a shared global model remains possible.

  • Scope and findings: The survey categorizes Non-IID distributions and analyzes their effects on parametric and non-parametric models in horizontal and vertical federated learning.It reports that parametric models in horizontal FL are mainly affected, with DNNs more sensitive to client data distributions.
  • Trade-offs of mitigation methods: Local fine-tuning and data sharing can improve convergence but may increase local computation, communication, or data-privacy costs.The conclusion presents these as trade-offs of existing approaches for handling Non-IID data.
  • Trade-offs of mitigation methods: Personalization and client clustering require changes to vanilla FL and may prevent generation of one global model for all clients.The survey identifies remaining challenges and future research directions after reviewing these approaches.
Loading 2106.06843v1…