PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-objective Evolutionary Federated Learning

Hangyu Zhu, Yaochu Jin

arXiv:1812.07478v2cs.LGcs.AIstat.ML

TL;DR

Federated learning preserves privacy by keeping data distributed across client devices, but its communication demands and heterogeneous data distributions pose important challenges. This paper uses multi-objective evolutionary optimization with scalable network-connectivity encoding to jointly reduce model complexity and test error. Experiments on MLPs and CNNs find models with substantially fewer connections that can maintain or improve federated-learning performance.

  • Problem

    Federated learning avoids uploading private training data but consumes substantial communication resources, while non-IID client data complicate optimization.

  • Method

    The paper jointly optimizes global model test error and model complexity with a multi-objective evolutionary algorithm using modified SET connectivity encoding.

  • Results

    The method finds MLP and CNN solutions with fewer connections that can improve test accuracy relative to fully connected models on IID and non-IID datasets.

  • Takeaways & Limitations

    Reducing neural-network connections can improve communication efficiency while preserving or improving federated-learning performance.

Abstract

from arXiv · show

Federated learning is an emerging technique used to prevent the leakage of private information. Unlike centralized learning that needs to collect data from users and store them collectively on a cloud server, federated learning makes it possible to learn a global model while the data are distributed on the users' devices. However, compared with the traditional centralized approach, the federated setting consumes considerable communication resources of the clients, which is indispensable for updating global models and prevents this technique from being widely used. In this paper, we aim to optimize the structure of the neural network models in federated learning using a multi-objective evolutionary algorithm to simultaneously minimize the communication costs and the global model test errors. A scalable method for encoding network connectivity is adapted to federated learning to enhance the efficiency in evolving deep neural networks. Experimental results on both multilayer perceptrons and convolutional neural networks indicate that the proposed optimization method is able to find optimized neural network models that can not only significantly reduce communication costs but also improve the learning performance of federated learning compared with the standard fully connected neural networks.

I. INTRODUCTION

Federated learning protects privacy by keeping training data on clients, but its communication demands and non-IID data create optimization challenges. The paper proposes evolutionary optimization of neural-network connectivity to reduce communication costs while maintaining learning performance.

  • Federated learning keeps training data on client devices while exchanging locally updated models with a central server.
  • Non-IID local data distributions create statistical challenges because they cause weight divergence and make client participation difficult to select.
  • Federated learning requires substantial communication resources, motivating methods that reduce communication rounds or transmitted model information.
  • The paper formulates federated learning as bi-objective optimization of communication cost and global learning accuracy using a multi-objective evolutionary algorithm.
  • A modified SET algorithm reduces neural-network connections and indirectly lowers the number of parameters transmitted between clients and the server.

II. PRELIMINARIES

The preliminaries describe MLP and CNN architectures, feed-forward and convolutional computation, and the optimization procedures used to train their parameters. They also distinguish trainable layers from pooling operations during CNN back-propagation.

  • Multilayer perceptrons: MLPs are feedforward networks with input, hidden, and output layers whose interlayer neurons are typically fully connected.
  • Multilayer perceptrons: During feed-forward propagation, neurons apply weighted sums and biases followed by nonlinear activation functions to produce predictions and losses.
  • Optimization: Training minimizes the loss over N training samples by optimizing the trainable parameters θ.
  • Convolutional neural networks: CNNs combine convolutional, pooling, and fully connected layers and are suited to high-dimensional inputs such as image-classification data.
  • Convolutional neural networks: CNN convolution uses kernel filters as trainable weights, with relu for hidden neurons and softmax for multiclass output nodes.
  • Optimization: Back-propagation updates weights in convolutional and fully connected layers, while pooling layers receive no derivative updates because they have no trainable parameters.

C. Federated learning

Federated learning trains local models on client data and aggregates their parameters into a shared global model. Its effectiveness depends on communication choices and data distributions, with non-IID clients creating weight divergence and convergence difficulties.

  • Federated-learning characteristics: Federated learning demands more local computation and communication resources than traditional centralized learning.
  • Optimization objective: The global objective is the weighted average of local losses, with client weights determined by local data sizes.
  • Federated-learning procedure: Each communication round sends global parameters to clients, which train locally and return updated parameters for server aggregation.
  • Federated averaging: FedAvg reduces communication rounds by increasing local training epochs and decreasing local mini-batch sizes.
  • Client participation: The client-selection fraction C determines how many of the K clients participate in updating the shared global model.
  • Client participation: Non-IID client data cause weight divergence and are harder to train than IID data, while choosing the right participating clients remains challenging.

D. The elitist non-dominated sorting genetic algorithm

The paper uses NSGA-II to optimize federated neural-network connectivity and hyperparameters for low communication cost and high global accuracy, producing diverse Pareto-optimal solutions. Its main computational limitation is the substantial cost of repeatedly evaluating trained models.

  • NSGA-II optimizes neural-network connectivity and hyperparameters to minimize communication costs while maximizing global learning accuracy.
  • NSGA-II generates diverse Pareto-optimal solutions by comparing dominance relationships and crowding distances within the population.
  • Each generation creates offspring through crossover and mutation, combines parent and offspring populations, and selects the next population by non-dominated fronts and crowding distance.
  • The framework can be extended with many-objective evolutionary algorithms or more efficient non-dominated sorting when objectives or population sizes grow.
  • O(mN^2) sorting and repeated model-training evaluations increase computational complexity, especially with large datasets.

III. PROPOSED ALGORITHM

The proposed algorithm combines scalable network encoding with a modified sparse evolutionary training procedure for federated neural networks. It starts from sparse random connectivity, removes weak weights only after training, and thereby reduces transmitted parameters.

  • Direct binary encoding is not scalable for deep networks with many layers and connections, motivating a more scalable connectivity encoding scheme.
  • SET initializes connectivity as an Erdős–Rényi random graph between neighboring layers instead of directly encoding the entire network.
  • When ε is much smaller than the neuron counts in adjacent layers, the connection probability becomes significantly lower.
  • The modified SET algorithm removes a fraction ξ of the smallest-updating weights only at the last training epoch, avoiding severe federated-training fluctuations.
  • The resulting sparse networks contain fewer parameters to download or upload, reducing federated-learning communication cost.

B. The objective functions and encoding of the neural networks

Federated learning is formulated as a two-objective problem balancing global test error and model complexity. The evolutionary chromosome encodes architecture, learning rate, and sparsity-related variables for MLP and CNN models.

  • The objectives are global model test error E_t and model complexity Ω_t, while evolution changes both neural-network hyperparameters and connectivity.
  • Encoded hyperparameters include hidden-layer count, neurons per hidden layer, and mini-batch SGD learning rate η.
  • Integer variables use binary coding, whereas real-valued parameters such as η and SET variables use real encoding.
  • MLP encoding: The MLP example uses ξ = 0.3, η = 0.1, ε = 20, and two hidden layers containing five and four neurons.
  • CNN encoding: The CNN encoding represents convolutional layers, output channels, fully connected layers, and kernel size, with kernel size randomly chosen from integers 3 to 5.
  • Model complexity Ω_t is measured using the average number of weights uploaded by clients during communication round t.

C. The modified SET federated averaging algorithm

The modified SET FedAvg procedure trains sparse candidate networks locally, aggregates client updates, evaluates global test error, and tracks transmitted model weights. Evolutionary solutions are assessed after repeated communication rounds.

  • The modified SET algorithm is integrated with FedAvg to reduce shared-model connectivity while test error evaluates global learning performance.
  • For each communication round, the procedure selects m = C × K clients and trains their models over local epochs and mini-batches.
  • After local training, a fraction ξ of the smallest parameter values is removed before the updated models are processed further.
  • The algorithm calculates each client's connection count, aggregates the counts across clients, evaluates global test accuracy, and returns the two objectives.
  • Each candidate solution specifies a neural-network model with modified SET topology used as the FedAvg global model.
  • Weights are trained with mini-batch SGD and the global model is updated repeatedly for a fixed number of communication rounds before objective calculation.

D. Multi-objective evolutionary optimization

The paper formulates federated learning as a bi-objective optimization problem and uses NSGA-II to evolve neural-network solutions balancing communication cost and global accuracy.

  • NSGA-II is used to obtain a set of Pareto-optimal solutions for minimizing communication costs and maximizing global learning accuracy.
  • Each generation creates offspring through crossover and mutation from a randomly initialized parent population.Binary chromosomes use one-point crossover and flip mutation, while real-valued chromosomes use SBX and polynomial mutation.
  • Parent and offspring populations are merged, ranked by non-dominance and crowding distance, and reduced to M high-ranking parents.
  • Repeating this evolutionary procedure produces a set of non-dominated solutions validated in IID and non-IID environments.

A. Experimental settings

Experiments evaluate sparse and fully connected MLP and CNN models on MNIST under IID and non-IID federated data settings, using specified training and evolutionary parameters.

  • The study evaluates multilayer perceptrons and convolutional neural networks trained and tested on MNIST.The original MLP has two 200-node hidden layers and 199,210 parameters; the CNN settings are also specified in the experimental setup.
  • Federated learning uses 100 clients with all clients participating in each communication round, and each local update runs 5 epochs with batch size 50.
  • MNIST is partitioned into IID and non-IID client datasets to assess performance under different data distributions.
  • NSGA-II uses a population size of 20 and runs for 20 IID generations or 50 non-IID generations.The larger non-IID budget reflects greater interest in learning performance in that setting.
  • Fitness evaluations use limited communication rounds because larger-round evaluations were infeasible under the available computation resources.
  • The sparse models adopt SET parameters ε = 20 and ξ = 0.3, previously used for both MLPs and CNNs.

B. Influence of the neural network sparsity on the performance

Increasing network sparsity reduces model connectivity but can lower global test accuracy, revealing a trade-off between communication-oriented model complexity and federated learning performance.

  • 500 communication rounds are used for MLPs and 200 for CNNs across IID and non-IID settings while varying SET parameters ε and ξ.The figures report global model test accuracies and average client accuracies for the tested sparsity settings.
  • Client training accuracy reaches nearly 100% within a few non-IID rounds and exceeds 95% within 25–50 IID rounds, while server convergence is slower.Server learning is particularly challenging on non-IID data.
  • Reducing network connectivity may degrade global test accuracy on both IID and non-IID datasets.
  • 96.93% global test accuracy is achieved with ε = 50, ξ = 0 and 72,051 average connections, versus 96.54% with ε = 100, ξ = 0.3 and 87,300 connections.
  • Sparse MLPs retain global accuracies about 2% below fully connected models with roughly 10% as many connections, while sparse CNNs are 0.45% worse with about 12%.Accuracy deteriorates more quickly on non-IID data as sparsity increases.
  • Modified SET FedAvg alone cannot simultaneously maximize global learning accuracy and minimize communication costs.

C. Evolved federated learning models

The study evolves Pareto-optimal federated MLP and CNN structures that balance global accuracy against communication cost. Validation shows strong compression is possible, but structures evolved on IID data may transfer poorly to non-IID data.

  • Multi-objective optimization: NSGA-II evolves Pareto-optimal neural-network structures by balancing global learning performance and communication cost.High-accuracy and knee solutions are selected from the non-dominated frontiers for validation against fully connected networks.
  • Computational cost: The evolutionary search is computationally intensive, with one CNN run taking more than one week on a GTX 1080Ti computer.Limited computation also constrains the number of generations and objective-evaluation communication rounds.
  • MLP validation: 98.16% and 97.42% are High1 MLP’s global test accuracies on IID and non-IID data, respectively, exceeding the fully connected MLP with about 46% of its connections.High1 averages 91,933 connections versus 199,210 for the fully connected network.
  • Cross-distribution robustness: Knee1 and Knee2 MLP accuracies fall from 96.84% and 94.24% on IID data to 93.77% and 90.77% on non-IID data.The results indicate that IID-evolved knee structures may not suit non-IID data, whereas non-IID-evolved structures are more robust across settings.
  • CNN validation: 98.79% is High2 CNN’s non-IID test accuracy, 0.04% above the fully connected model, while using only around 10% of its connections.On IID data, High1 and High2 reach 99.07% and 98.96%, respectively, both above the fully connected CNN.
  • Practical implications: The selected sparse solutions reduce transferred model parameters without seriously deteriorating performance, enabling communication-efficiency improvements of at least 50%.The proposed algorithm supports multiple solutions for different learning-task preferences and can allow clients to use different model sizes.

V. CONCLUSIONS AND FUTURE WORK

The paper concludes that multi-objective evolutionary optimization can produce federated neural networks with fewer connections and competitive or improved accuracy. It also identifies limits involving dataset complexity, network depth, communication loss, and adversarial robustness.

  • Conclusions: The modified SET-based evolutionary method generates federated models with better global accuracy and fewer connections on IID and non-IID datasets.The method simultaneously maximizes learning performance and minimizes communication cost.
  • Limitations and future work: The modified SET FedAvg and FedAvg algorithms do not work very well on complicated datasets such as non-IID CIFAR-10.This marks a dataset-complexity boundary for the reported results.
  • Limitations and future work: The studied networks are fairly simple, leaving performance on deep networks with dozens of hidden layers unresolved.The paper also identifies package loss and adversarial attacks as open concerns for federated-learning robustness.
Loading 1812.07478v2…