PaperlayerView exact revision on arXiv

Source-linked AI summary

From Federated Learning to Federated Neural Architecture Search: A Survey

Hangyu Zhu, Haoyu Zhang, Yaochu Jin

arXiv:2009.05868v1cs.DC

TL;DR

The paper addresses the challenge of searching for optimized neural architectures when privacy-preserving federated learning restricts access to raw data and adds communication and resource constraints. It surveys federated learning, NAS, and federated NAS, organizing the latter by optimization timing and objective structure. The survey concludes that federated NAS has practical significance despite substantial unresolved challenges.

  • Problem

    Federated NAS remains difficult because federated learning introduces constraints that make many centralized NAS techniques unsuitable, while privacy-preserving training limits data exchange.

  • Method

    The survey synthesizes federated learning and NAS research, categorizing federated NAS into offline or online optimization and single- or multi-objective approaches.

  • Results

    The survey identifies online federated NAS as more challenging and outlines remaining research challenges and promising research topics.

  • Takeaways & Limitations

    Federated NAS is practically significant for privacy-sensitive real-world problems where handcrafted neural networks may not work properly.

  • Takeaways & Limitations

    Homomorphic encryption is computationally intensive for federated NAS because modern DNNs contain millions of parameters.

Abstract

from arXiv · show

Federated learning is a recently proposed distributed machine learning paradigm for privacy preservation, which has found a wide range of applications where data privacy is of primary concern. Meanwhile, neural architecture search has become very popular in deep learning for automatically tuning the architecture and hyperparameters of deep neural networks. While both federated learning and neural architecture search are faced with many open challenges, searching for optimized neural architectures in the federated learning framework is particularly demanding. This survey paper starts with a brief introduction to federated learning, including both horizontal, vertical, and hybrid federated learning. Then, neural architecture search approaches based on reinforcement learning, evolutionary algorithms and gradient-based are presented. This is followed by a description of federated neural architecture search that has recently been proposed, which is categorized into online and offline implementations, and single- and multi-objective search approaches. Finally, remaining open research questions are outlined and promising research topics are suggested.

1 Introduction

Neural architecture search automates the design of deep neural networks, while federated learning addresses privacy concerns by keeping raw data local. Combining them is important but particularly demanding because federated settings constrain architecture search.

  • Designing high-quality DNN architectures manually requires trying many hyperparameters and broad expertise.
  • NAS aims to automatically search for good neural architectures instead of relying on manual design.
  • Federated learning protects data privacy by communicating model parameters or other model information rather than raw data.
  • Federated NAS is particularly important yet challenging because centralized NAS techniques may not suit federated environments.

Horizontal federated learning

The survey focuses on federated neural architecture search and organizes it by offline versus online optimization and by single- versus multi-objective search. It emphasizes that online search faces stricter performance and resource constraints.

  • The survey overviews federated learning and NAS while focusing on the emerging area of federated neural architecture search.
  • Online federated NAS is more challenging than offline approaches because networks must perform during search under tighter computational constraints.
  • Federated NAS can handle multiple objectives including accuracy, communication costs, model complexity, and local-device memory requirements.

2 Federated Learning

Federated learning distributes training while preserving privacy across differing data-partition settings. The survey describes horizontal, vertical, and hybrid forms, their workflows, and their communication and privacy challenges.

  • Federated learning prioritizes privacy, cannot freely determine client data distributions, and may involve millions of unbalanced clients with difficult connectivity.
  • Horizontal federated learning: Horizontal federated learning uses clients with the same feature space but different samples, often producing non-IID data across clients.
  • Horizontal federated learning: FedAvg initializes and distributes a global model, trains it locally, uploads updates, aggregates them on the server, and repeats until convergence.
  • Horizontal federated learning: Horizontal federated learning must reduce communication, improve convergence, and prevent privacy leakage during model-information exchange.
  • Vertical federated learning: Vertical federated learning applies when clients share samples but have different features, with parties coordinating encrypted losses and gradients.
  • Hybrid federated learning: Hybrid federated learning combines different sample and feature spaces, requiring privacy-preserving identity alignment and often accommodating asymmetric participants.

3 Neural Architecture Search

Neural architecture search automates the design of high-quality deep neural networks through reinforcement learning, evolutionary algorithms, or gradient-based optimization. These strategies differ in search representation and computational cost, with fitness evaluation creating a major bottleneck for evolutionary methods.

  • Overview: NAS automates neural-network architecture design to reduce the expertise and manual experimentation required for selecting effective architectures and hyperparameters.Manual DNN design requires trying many hyperparameters and substantial expertise in both machine learning and the application area.
  • Reinforcement Learning: RL-based NAS treats architecture generation as an agent action, trains the resulting child model, and uses validation performance as the reward.Policy-gradient and Q-learning methods generate architectures sequentially or through controller-based decisions.
  • Reinforcement Learning: Parameter sharing accelerates RL-based NAS by representing candidate architectures as subgraphs of a supernet with shared weights.ENAS reduces repeated training by forcing subgraphs to share weights on common edges.
  • Evolutionary Algorithms: EA-based NAS searches populations of architectures with gradient-free heuristics, including genetic encodings, coevolution, variable-length representations, and genetic programming.These methods can optimize topology and, in some cases, weight initialization or connectivity patterns.
  • Evolutionary Algorithms: AmoebaNet extended large-scale evolution with a micro search space and achieved better ImageNet results than hand-designed methods for the first time.The result demonstrates the potential of evolutionary search despite its substantial computational demands.
  • Evolutionary Algorithms: 27 GPU days for AE-CNN, 35 GPU days for CNN-GA, 2750 GPU days for large-scale evolution, and 3150 GPU days for AmoebaNet illustrate evolutionary NAS's fitness-evaluation bottleneck on CIFAR10.The cost of lower-level weight optimization substantially limits practical usability under constrained search budgets.

4 Federated Neural Architecture Search

Federated NAS optimizes neural architectures while respecting the distributed and privacy-constrained setting of federated learning. The survey distinguishes offline search from online simultaneous search, and describes federated implementations based on evolutionary and gradient-based methods.

  • Overview: Federated NAS optimizes neural-network architectures in federated learning, where distributed training is more difficult than centralized training.The survey organizes approaches by offline versus online optimization and single- versus multi-objective search.
  • Offline Federated NAS: Offline NAS separates architecture search from weight training, so only the final optimized network needs acceptable performance.Population-based evolutionary and pruning-based federated approaches evaluate candidate models before selecting or storing models for later training.
  • Offline Federated NAS: Using all participating clients to train every individual substantially increases computation and communication costs, motivating client sampling for individual model training.Client subsets can contribute to one candidate's training instead of requiring every client to train all N individuals.
  • Offline Federated NAS: Some population-based offline frameworks remove subsets of global models to reduce communication, but pre-aggregation accuracy can misrepresent true accuracy under particularly non-IID client data.The limitation arises because each global model's test accuracy is calculated before model aggregation.
  • Offline Federated NAS: Offline federated NAS faces an unresolved communication-round trade-off: too few rounds bias fitness evaluations through under-training, whereas too many consume communication resources.Training candidate models also adds communication costs that federated learning seeks to avoid.
  • Online Federated NAS: Online federated NAS trains weights and optimizes architecture simultaneously; the surveyed approaches use either federated gradient-based search or an evolutionary method.The gradient-based approach uses a supernet whose architecture parameters and model weights are alternately updated and aggregated.
  • Online Federated NAS: Online search avoids separate candidate-model training and its additional communication, but jointly optimizing the supernet requires more computation and memory on local clients.This resource demand is poorly suited to edge devices such as mobile phones.
  • Online Federated NAS: RT-FedEvoNAS reduces local memory, computation, and uploading costs by sampling one path from the repeated-cell supernet for each client.It combines lightweight model sampling with an evolutionary federated NAS framework for real-time operation.

5 Open Challenges

Federated NAS remains preliminary, with unresolved challenges spanning non-IID data, privacy leakage, adversarial robustness, vertical settings, and encrypted computation. Existing remedies often involve trade-offs in privacy, generality, computation, or communication.

  • Non-IID Learning: Federated NAS lacks a general solution for non-IID learning degradation, and shared-data mitigation intrinsically violates privacy.Alternative aggregation methods outperform FedAvg on non-IID data but remain limited to specific models and datasets, with unclear applicability to federated NAS.
  • Vertical Federated NAS: Current federated NAS methods use horizontal federated learning, while vertical federated NAS remains largely unexplored.Vertical settings make IID versus non-IID status difficult to determine because data are partitioned across features rather than samples.
  • Security and Robustness: Federated learning remains vulnerable to data reconstruction and model attacks because gradients and parameters contain local data information.Potential adversaries include participating clients and the central server, while defenses include gradient compression, differential privacy, secure multiparty computation, and robust aggregation.
  • Security and Robustness: Finding robust model architectures that defend against adversarial attacks remains difficult in federated learning.The survey identifies robust architecture search as an unresolved challenge rather than presenting a general solution.
  • Encrypted Federated NAS: Homomorphic encryption is computationally expensive for federated NAS because modern neural networks contain millions of parameters requiring costly encoding and modulus calculations.Secret-sharing-based encryption also increases communication costs because encrypted gradients must be frequently transferred and partially decrypted.

6 Conclusion

The survey reviews federated learning, neural architecture search, and their combination in federated NAS. It emphasizes practical significance alongside substantial challenges, especially the constraints imposed by privacy-preserving environments and online deployment.

  • Conclusion: The survey introduces federated NAS by combining an overview of federated learning with neural architecture search.It frames federated NAS as a challenging setting where centralized NAS techniques may no longer be suitable.
  • Conclusion: Offline evolutionary NAS can require substantial communication because architecture search and weight training are performed separately.RT-FedEvoNAS addresses the stated challenges, although its search space is highly constrained.
  • Conclusion: Federated NAS is practically significant for real-world problems where handcrafted deep neural networks may fail to work properly.The survey presents this significance despite many remaining grand challenges.
Loading 2009.05868v1…