Hybrid-FL for Wireless Networks: Cooperative Learning Mechanism Using Non-IID Data

Naoya Yoshida; Takayuki Nishio; Masahiro Morikura; Koji Yamamoto; Ryo Yonetani

Source-linked AI summary

Hybrid-FL for Wireless Networks: Cooperative Learning Mechanism Using Non-IID Data

Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto, Ryo Yonetani

arXiv:1905.07210v3May 17, 2019cs.LGcs.DCstat.ML

TL;DR

Non-IID client data can degrade federated-learning model performance, while prior approaches did not consider resource scheduling. Hybrid-FL lets a limited number of clients upload data, trains a server model, and aggregates it with distributed client models using heuristic selection algorithms; it achieved higher accuracy than prior methods, including a 13.5% gain on CIFAR-10.

Problem
Non-IID client data can degrade federated-learning model performance, and prior approaches did not consider resource scheduling.
Method
Hybrid-FL uses data from a limited number of clients to update a server model, aggregates it with models trained on non-IID client data, and heuristically selects clients and upload data.
Results
Hybrid-FL achieved 13.5% higher CIFAR-10 accuracy and 12.5% higher Fashion MNIST accuracy than FedCS under non-IID data.
Takeaways & Limitations
Hybrid-FL improved classification accuracy across CIFAR-10 and Fashion MNIST and remained more accurate than the conventional method across different mixtures of client non-IIDness.

Abstract

from arXiv · show

This paper proposes a cooperative mechanism for mitigating the performance degradation due to non-independent-and-identically-distributed (non-IID) data in collaborative machine learning (ML), namely federated learning (FL), which trains an ML model using the rich data and computational resources of mobile clients without gathering their data to central systems. The data of mobile clients is typically non-IID owing to diversity among mobile clients' interests and usage, and FL with non-IID data could degrade the model performance. Therefore, to mitigate the degradation induced by non-IID data, we assume that a limited number (e.g., less than 1%) of clients allow their data to be uploaded to a server, and we propose a hybrid learning mechanism referred to as Hybrid-FL, wherein the server updates the model using the data gathered from the clients and aggregates the model with the models trained by clients. The Hybrid-FL solves both client- and data-selection problems via heuristic algorithms, which try to select the optimal sets of clients who train models with their own data, clients who upload their data to the server, and data uploaded to the server. The algorithms increase the number of clients participating in FL and make more data gather in the server IID, thereby improving the prediction accuracy of the aggregated model. Evaluations, which consist of network simulations and ML experiments, demonstrate that the proposed scheme achieves a 13.5% higher classification accuracy than those of the previously proposed schemes for the non-IID case.

I. INTRODUCTION

Federated learning preserves client data locally while leveraging distributed computation, but wireless heterogeneity and non-IID client data complicate coordination and can degrade model performance. Hybrid-FL addresses this by combining limited server-side data uploads with local model training and reports higher accuracy than FedCS in non-IID image classification.

Federated learning trains models on mobile clients’ local data, reducing coordinator computation and keeping confidential data on clients.
Wireless FL must coordinate client participation and bandwidth while also accounting for client data distributions and training-data representativeness.
Non-IID client data can degrade FL model performance, creating a resource-coordination problem absent from conventional cooperative network computing.
Hybrid-FL lets a very small fraction of clients upload data, trains a server model from the gathered data, and aggregates it with locally trained models.The proposal considers cases where fewer than 1% of clients permit data uploading and schedules both data- and model-uploading clients.
Hybrid-FL achieves higher classification accuracy than FedCS in a cellular-network simulation when client data is non-IID.The evaluation uses image-classification tasks.

II. SYSTEM MODEL

The system models FL over a cellular MEC platform with managed radio resources and assumes stable, lightly loaded communication conditions. It also assumes that only a very small fraction of clients upload data and restricts evaluation to classification tasks.

The MEC platform consists of a server and base station in a cellular network, with an operator managing FL behavior and radio-block resources.Training and communication are scheduled during midnight or early morning to avoid network congestion.
Client radio throughput varies with channel conditions despite constant allocated resource blocks, while channel state and throughput are assumed stable.
Only a very small fraction of clients, such as less than 1%, are assumed to permit data uploads; the study considers classification tasks only.The paper motivates participation through incentives and notes that larger participation could make federated learning unnecessary.
FedCS iteratively adds clients with the lowest estimated incremental upload time until the round deadline is reached.
FedCS selects clients under heterogeneous computation and wireless conditions, but practical non-IID local datasets can significantly degrade model performance.

IV. HYBRID FEDERATED LEARNING

Hybrid-FL extends federated learning with centralized training on selectively uploaded client data alongside distributed local model training. Its protocol coordinates client requests, selection, parallel updates and uploads, then aggregates the resulting parameters.

Hybrid-FL combines centralized model training with distributed model training to operate efficiently under non-IID data distributions.
The protocol requests resource and data information from randomly selected clients before choosing participating clients and uploaded data.
Selected clients either update models locally or upload class-specific data in parallel, with uploads scheduled using operator-allocated resource blocks.
The server averages updated parameters to replace the global model and repeats the protocol until the desired performance or deadline is reached.

A. Hybrid-FL Protocol

Hybrid-FL constructs an approximately IID server dataset by combining data uploaded by multiple clients, while other clients train models locally. The operator selects both client groups and schedules their communication before server-side updating and aggregation.

Hybrid-FL combines data from multiple clients to approximately construct IID server data even when individual clients hold non-IID datasets.
The operator selects clients for local model updates and clients for data uploads, using reported data, communication, and computation information.
The protocol distributes the global model by multicast, while local-update and data-upload clients operate in parallel for selected classes.
After collecting uploaded data, the server updates the global model and incorporates it with the locally trained model updates.

B. Client and Data Selection in Hybrid-FL

Hybrid-FL uses separate heuristic procedures to select local-training clients and server-uploaded data, balancing training time against data-distribution bias and server-side IID composition.

Selection overview: Hybrid-FL separates client selection from data selection to coordinate local model updates and data uploads.The protocol applies both methods within a Client and Data Selection step.
Client selection: Client selection can maximize the number of clients completing training by a deadline or choose clients whose class-data amounts are more balanced.The second method is intended to make Hybrid-FL more robust to non-IID data.
Client selection: The client-selection objective combines estimated completion time with the coefficient of variation of accumulated class-data amounts.The stated objective is f(S, k) = Tinc(S, k)· CV(Nr).
Data selection: Data selection operates within the time available before new model parameters begin uploading.The operator estimates model-update time, then selects uploadable data from clients that permitted data sharing.
Data selection: The data-selection alternatives either maximize server data volume through high-throughput clients or construct server-side IID data by adding classes iteratively until the upload-time limit.The class-balanced procedure adds similar amounts of each class while tDUL remains within tUD.

V. PERFORMANCE EVALUATION

The evaluation combines MEC simulation with realistic machine-learning experiments under IID and multiple non-IID data distributions to assess protocol performance.

Evaluation design: The study evaluates Hybrid-FL through an MEC-environment simulation and realistic ML experiments using publicly available large-scale datasets.Performance is tested with IID data and various non-IID data distributions.
Evaluation design: Performance is evaluated across both IID and various non-IID data distributions.This design directly tests the protocol under the distributional conditions motivating Hybrid-FL.

A. Simulation Settings

The simulation models a cellular MEC environment with 1,000 clients, a small data-uploading subset, constrained upload selection, and heterogeneous wireless and computation capabilities.

Simulation environment: The simulated MEC environment contains an edge server, base station, and K = 1000 clients, with a small subset permitted to upload data.The uploading clients are randomly selected, and the server and base station are co-located in a 2-km-radius cell.
Data-upload simulation: The upload-data algorithm initializes an empty upload set and repeatedly processes classes while the time budget permits adding data.The procedure stops when adding data would exceed tUD or no eligible client data remains.
Wireless model: Wireless communication follows an LTE urban-channel model with clients uniformly distributed in a 2-km-radius cell.The simulation uses an ITU-R M.2135-1 Micro NLOS model and samples throughput variation around average client throughput.
Client heterogeneity: The simulation models client computation capability as samples processed per second, with fluctuating capacity sampled around client-specific averages.Average capabilities are randomly selected from 10 to 100 samples per second for use by the selection algorithms.

B. Experimental Setup for ML Tasks

The ML evaluation uses two image-classification benchmarks and a standard convolutional neural network shared with prior work.

Datasets: The experiments use CIFAR-10 and Fashion MNIST, each providing 10-class image-classification tasks.CIFAR-10 contains 50,000 training and 10,000 testing images; Fashion MNIST contains 60,000 training and 10,000 testing images.
Model: The model is a standard convolutional neural network with six convolution layers followed by three fully connected layers.The convolution layers use ReLU activations, batch normalization, and max pooling; the final layer has 10 soft-max units.

C. Data Distribution

The evaluation constructs client data distributions by varying how many classes each client possesses, with μ controlling the prevalence of limited-class clients and σ controlling heterogeneity. Smaller μ produces more clients with data from fewer classes.

The training data contain images from 10 classes distributed among K = 1000 clients, with r_l denoting the fraction of clients possessing data from l classes.Each client receives a randomly determined 100–1,000 images sampled from l randomly selected classes.
When r_10 = 1, clients can represent an IID distribution, whereas r_1 = 1 gives every client data from only one class.
Figure 3 examines prediction accuracy versus μ for CIFAR-10 and Fashion MNIST at σ = 0.7.A smaller μ indicates that clients tend to have data from fewer classes.
The client-class distribution is parameterized using a truncated normal construction over l = 1,...,10 classes.The setup fixes a = 0.5 and b = 10.5 while varying μ and σ.
Smaller μ means more clients have data from limited classes, while larger σ increases the variety among clients’ class coverage.

D. Evaluation Results

The evaluation compares four Hybrid-FL client-selection variants with FedCS and centralized training across non-IIDness, uploading-client counts, and resource fluctuations. Hybrid-FL generally maintains higher accuracy under non-IID data, while performance depends on available uploading clients and resource stability.

Evaluation setup: The study evaluates four Hybrid-FL variants formed by combining two data-uploading-client selectors with two model-uploading-client selectors.The variants use maxThroughput or IID for data-uploading clients and maxClient or minCV for model-uploading clients.
Evaluation setup: Mean accuracy is averaged over the final 100 minutes, from T = 300 to T = 400 minutes, across 10 trials.Hybrid-FL is compared with FedCS and centralized model training.
Non-IID data: Hybrid-FL achieved higher accuracy than the conventional method across different σ values representing coexistence of clients with different non-IIDness.The comparison uses μ = 4 on CIFAR-10 and Fashion MNIST.
Data-uploading clients: Increasing the data-uploading-client ratio increased prediction accuracy for both Hybrid-FL and centralized training on CIFAR-10.When r_UL was below 0.01, Hybrid-FL outperformed the existing methods.
Resource fluctuations: Large resource-fluctuation values decreased accuracy for all methods because selections became less aligned with the utility-maximizing resource values.The methods could still train ML models under these fluctuations.

VI. CONCLUSION

Hybrid-FL extends FedCS to mitigate non-IID data by constructing an approximately IID server dataset and aggregating its updated model with distributed client models.

Hybrid-FL extends FedCS to mitigate the model-performance degradation caused by non-IID data.
The protocol gathers data from a limited number of uploading clients to construct an approximately IID server dataset.It then updates a model with this data before aggregation.
Heuristic algorithms select model-uploading clients, participating clients, and uploaded data.
With 1% of data-uploading clients, Hybrid-FL significantly improved classification accuracy under non-IID data in MEC simulations and realistic ML experiments.