PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-objective Evolutionary Federated Learning

Hangyu Zhu, Yaochu Jin

arXiv:1812.07478v2cs.LGcs.AIstat.ML

TL;DR

Federated learning preserves privacy by keeping data distributed across client devices, but its communication demands and heterogeneous data distributions pose important challenges. This paper uses multi-objective evolutionary optimization with scalable network-connectivity encoding to jointly reduce model complexity and test error. Experiments on MLPs and CNNs find models with substantially fewer connections that can maintain or improve federated-learning performance.

Abstract

from arXiv · show

Federated learning is an emerging technique used to prevent the leakage of private information. Unlike centralized learning that needs to collect data from users and store them collectively on a cloud server, federated learning makes it possible to learn a global model while the data are distributed on the users' devices. However, compared with the traditional centralized approach, the federated setting consumes considerable communication resources of the clients, which is indispensable for updating global models and prevents this technique from being widely used. In this paper, we aim to optimize the structure of the neural network models in federated learning using a multi-objective evolutionary algorithm to simultaneously minimize the communication costs and the global model test errors. A scalable method for encoding network connectivity is adapted to federated learning to enhance the efficiency in evolving deep neural networks. Experimental results on both multilayer perceptrons and convolutional neural networks indicate that the proposed optimization method is able to find optimized neural network models that can not only significantly reduce communication costs but also improve the learning performance of federated learning compared with the standard fully connected neural networks.

I. INTRODUCTION

Federated learning protects privacy by keeping training data on clients, but its communication demands and non-IID data create optimization challenges. The paper proposes evolutionary optimization of neural-network connectivity to reduce communication costs while maintaining learning performance.

  • Federated learning keeps training data on client devices while exchanging locally updated models with a central server.Source: §I · p. 1
  • Non-IID local data distributions create statistical challenges because they cause weight divergence and make client participation difficult to select.Source: §I · p. 1
  • Federated learning requires substantial communication resources, motivating methods that reduce communication rounds or transmitted model information.Source: §I · p. 1
  • The paper formulates federated learning as bi-objective optimization of communication cost and global learning accuracy using a multi-objective evolutionary algorithm.Source: §I · p. 2
  • A modified SET algorithm reduces neural-network connections and indirectly lowers the number of parameters transmitted between clients and the server.Source: §I · pp. 1–2

II. PRELIMINARIES

The preliminaries describe MLP and CNN architectures, feed-forward and convolutional computation, and the optimization procedures used to train their parameters. They also distinguish trainable layers from pooling operations during CNN back-propagation.

Multilayer perceptrons

Optimization

Convolutional neural networks

Optimization

  • Back-propagation updates weights in convolutional and fully connected layers, while pooling layers receive no derivative updates because they have no trainable parameters.Source: B. Convolutional neural networks · p. 3

C. Federated learning

Federated learning trains local models on client data and aggregates their parameters into a shared global model. Its effectiveness depends on communication choices and data distributions, with non-IID clients creating weight divergence and convergence difficulties.

Federated-learning characteristics

  • Federated learning demands more local computation and communication resources than traditional centralized learning.Source: C. Federated learning · p. 3

Optimization objective

  • The global objective is the weighted average of local losses, with client weights determined by local data sizes.Source: C. Federated learning · p. 3

Federated-learning procedure

  • Each communication round sends global parameters to clients, which train locally and return updated parameters for server aggregation.Source: C. Federated learning · pp. 3–4

Federated averaging

  • FedAvg reduces communication rounds by increasing local training epochs and decreasing local mini-batch sizes.Source: C. Federated learning · p. 4

Client participation

  • The client-selection fraction C determines how many of the K clients participate in updating the shared global model.Source: C. Federated learning · p. 4
  • Non-IID client data cause weight divergence and are harder to train than IID data, while choosing the right participating clients remains challenging.Source: C. Federated learning · p. 4

D. The elitist non-dominated sorting genetic algorithm

The paper uses NSGA-II to optimize federated neural-network connectivity and hyperparameters for low communication cost and high global accuracy, producing diverse Pareto-optimal solutions. Its main computational limitation is the substantial cost of repeatedly evaluating trained models.

III. PROPOSED ALGORITHM

The proposed algorithm combines scalable network encoding with a modified sparse evolutionary training procedure for federated neural networks. It starts from sparse random connectivity, removes weak weights only after training, and thereby reduces transmitted parameters.

B. The objective functions and encoding of the neural networks

Federated learning is formulated as a two-objective problem balancing global test error and model complexity. The evolutionary chromosome encodes architecture, learning rate, and sparsity-related variables for MLP and CNN models.

MLP encoding

CNN encoding

C. The modified SET federated averaging algorithm

The modified SET FedAvg procedure trains sparse candidate networks locally, aggregates client updates, evaluates global test error, and tracks transmitted model weights. Evolutionary solutions are assessed after repeated communication rounds.

D. Multi-objective evolutionary optimization

The paper formulates federated learning as a bi-objective optimization problem and uses NSGA-II to evolve neural-network solutions balancing communication cost and global accuracy.

A. Experimental settings

Experiments evaluate sparse and fully connected MLP and CNN models on MNIST under IID and non-IID federated data settings, using specified training and evolutionary parameters.

  • The study evaluates multilayer perceptrons and convolutional neural networks trained and tested on MNIST.The original MLP has two 200-node hidden layers and 199,210 parameters; the CNN settings are also specified in the experimental setup.Source: A. Experimental settings · p. 7
  • Federated learning uses 100 clients with all clients participating in each communication round, and each local update runs 5 epochs with batch size 50.Source: A. Experimental settings · p. 7
  • MNIST is partitioned into IID and non-IID client datasets to assess performance under different data distributions.Source: A. Experimental settings · p. 7
  • NSGA-II uses a population size of 20 and runs for 20 IID generations or 50 non-IID generations.The larger non-IID budget reflects greater interest in learning performance in that setting.Source: A. Experimental settings · p. 7
  • Fitness evaluations use limited communication rounds because larger-round evaluations were infeasible under the available computation resources.Source: A. Experimental settings · p. 8
  • The sparse models adopt SET parameters ε = 20 and ξ = 0.3, previously used for both MLPs and CNNs.Source: A. Experimental settings · p. 8

B. Influence of the neural network sparsity on the performance

Increasing network sparsity reduces model connectivity but can lower global test accuracy, revealing a trade-off between communication-oriented model complexity and federated learning performance.

C. Evolved federated learning models

The study evolves Pareto-optimal federated MLP and CNN structures that balance global accuracy against communication cost. Validation shows strong compression is possible, but structures evolved on IID data may transfer poorly to non-IID data.

Multi-objective optimization

  • NSGA-II evolves Pareto-optimal neural-network structures by balancing global learning performance and communication cost.High-accuracy and knee solutions are selected from the non-dominated frontiers for validation against fully connected networks.Source: C. Evolved federated learning models · pp. 9–10

Computational cost

  • The evolutionary search is computationally intensive, with one CNN run taking more than one week on a GTX 1080Ti computer.Limited computation also constrains the number of generations and objective-evaluation communication rounds.Source: C. Evolved federated learning models · p. 9

MLP validation

  • 98.16% and 97.42% are High1 MLP’s global test accuracies on IID and non-IID data, respectively, exceeding the fully connected MLP with about 46% of its connections.High1 averages 91,933 connections versus 199,210 for the fully connected network.Source: C. Evolved federated learning models · pp. 10–11

Cross-distribution robustness

  • Knee1 and Knee2 MLP accuracies fall from 96.84% and 94.24% on IID data to 93.77% and 90.77% on non-IID data.The results indicate that IID-evolved knee structures may not suit non-IID data, whereas non-IID-evolved structures are more robust across settings.Source: C. Evolved federated learning models · pp. 11–12

CNN validation

  • 98.79% is High2 CNN’s non-IID test accuracy, 0.04% above the fully connected model, while using only around 10% of its connections.On IID data, High1 and High2 reach 99.07% and 98.96%, respectively, both above the fully connected CNN.Source: C. Evolved federated learning models · p. 11

Practical implications

  • The selected sparse solutions reduce transferred model parameters without seriously deteriorating performance, enabling communication-efficiency improvements of at least 50%.The proposed algorithm supports multiple solutions for different learning-task preferences and can allow clients to use different model sizes.Source: C. Evolved federated learning models · pp. 11–12

V. CONCLUSIONS AND FUTURE WORK

The paper concludes that multi-objective evolutionary optimization can produce federated neural networks with fewer connections and competitive or improved accuracy. It also identifies limits involving dataset complexity, network depth, communication loss, and adversarial robustness.

Conclusions

  • The modified SET-based evolutionary method generates federated models with better global accuracy and fewer connections on IID and non-IID datasets.The method simultaneously maximizes learning performance and minimizes communication cost.Source: §V · p. 12

Limitations and future work

  • The modified SET FedAvg and FedAvg algorithms do not work very well on complicated datasets such as non-IID CIFAR-10.This marks a dataset-complexity boundary for the reported results.Source: §V · p. 12
  • The studied networks are fairly simple, leaving performance on deep networks with dozens of hidden layers unresolved.The paper also identifies package loss and adversarial attacks as open concerns for federated-learning robustness.Source: §V · p. 12

Related papers

Loading 1812.07478v2…