PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Distributed Machine Learning

Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, Jan S. Rellermeyer

arXiv:1912.09789v1cs.LGcs.DCstat.ML

TL;DR

Growing training datasets and model sizes motivate distributing machine-learning workloads across multiple machines, creating challenges in parallelization and coherent model construction. This survey synthesizes those challenges, techniques, architectures, and systems, concluding that distributed machine learning is increasingly becoming the norm while performance, privacy, and resource trade-offs remain open concerns.

  • Problem

    Training-data demands have outpaced computing-power growth, requiring distributed machine learning while introducing challenges in parallelization and coherent model construction.

  • Method

    The paper provides a holistic survey of distributed machine learning, covering system challenges, techniques, reference architectures, and available systems and libraries.

  • Results

    Distributed machine learning is increasingly becoming the norm, with industry-grade systems addressing the parallel processing and model-combination challenges.

  • Takeaways & Limitations

    The field has a broad ecosystem, but future systems must continue addressing performance, scalability, failure resilience, security, and privacy.

  • Takeaways & Limitations

    Current frameworks provide little support for even basic forms of privacy.

Abstract

from arXiv · show

The demand for artificial intelligence has grown significantly over the last decade and this growth has been fueled by advances in machine learning techniques and the ability to leverage hardware acceleration. However, in order to increase the quality of predictions and render machine learning solutions feasible for more complex applications, a substantial amount of training data is required. Although small machine learning models can be trained with modest amounts of data, the input for training larger models such as neural networks grows exponentially with the number of parameters. Since the demand for processing training data has outpaced the increase in computation power of computing machinery, there is a need for distributing the machine learning workload across multiple machines, and turning the centralized into a distributed system. These distributed systems present new challenges, first and foremost the efficient parallelization of the training process and the creation of a coherent model. This article provides an extensive overview of the current state-of-the-art in the field by outlining the challenges and opportunities of distributed machine learning over conventional (centralized) machine learning, discussing the techniques used for distributed machine learning, and providing an overview of the systems that are available.

1 INTRODUCTION

Machine learning is increasingly applied to complex, data-intensive decision problems, but sophisticated applications require distributed processing and system designs that support parallelism, data distribution, and failure resilience.

  • Machine learning analyzes growing datasets to build decision-making systems for complex problems such as autonomous driving, speech recognition, and consumer-behavior prediction.
  • Training data for sophisticated applications can reach terabyte scale, while some data is inherently distributed or too large to store on one machine.
  • Distributed machine learning requires algorithms and implementations that enable parallel computation, data distribution, and resilience to failures.
  • The survey takes a holistic, distributed-systems view of state-of-the-art machine learning, contrasting it with prior surveys and related work.
  • The article covers system challenges, HPC-based acceleration, a reference architecture, available systems and libraries, and continuing challenges.

2 MACHINE LEARNING - A HIGH PERFORMANCE COMPUTING CHALLENGE?

Distributed machine learning exploits hardware acceleration and scale-out resources for large linear-algebra workloads, but performance gains involve architectural, communication, cost, and resilience trade-offs.

  • Machine learning workloads largely perform vector, matrix, and tensor transformations, linking them to long-established high-performance-computing optimization techniques.
  • GPUs are the most common scale-up method, while ASICs such as TPUs specialize in highly optimized machine-learning operations.
  • DianNao achieved a performance speedup of three orders of magnitude and an energy reduction of more than 20x across layers of several large neural networks.
  • Deep compression produced 2.9 times higher throughput and 19x improved energy efficiency compared to DianNao by placing weights in SRAM and accelerating sparse matrix-vector multiplication.
  • Scale-out designs can reduce equipment costs, improve failure resilience, and increase aggregate I/O bandwidth compared with single-machine approaches.
  • Distributed deployment introduces performance, scalability, failure-resilience, and security challenges, making workload parallelization essential at large scale.

3 A REFERENCE ARCHITECTURE FOR DISTRIBUTED MACHINE LEARNING

The reference architecture organizes distributed machine learning around algorithm choice, parallelism, and network topology, whose interactions determine communication and system behavior.

  • A generic distributed machine-learning system is difficult to design because each algorithm has a distinct communication pattern.
  • The architecture separates decisions about machine-learning algorithms, parallelism, and topology while covering the distributed design space.
  • Training optimizes a model using data and hyperparameters, after which the trained model is deployed to predict outcomes for new data.
  • Data parallelism applies the same algorithm to partitioned datasets, while model parallelism distributes parts or paths of a model across nodes.
  • Distributed systems may train separate model instances and aggregate their outputs through methods such as ensembling.
  • Topology choices affect node roles, inter-node communication, and failure resilience, while the three architectural layers jointly affect training communication volume.

3.1 Machine Learning Algorithms

The survey categorizes machine-learning algorithms by feedback, purpose, and model evolution, then describes representative learning procedures and model families.

  • Machine-learning algorithms are categorized by the feedback they receive, their purpose, and how their models evolve.
  • Feedback: Supervised learning maps labeled inputs to desired outputs, while reinforcement learning trains agents through reward or cost feedback from an environment.
  • Purpose: Machine-learning purposes include anomaly detection, classification, clustering, dimensionality reduction, and representation learning.
  • Method: Evolutionary algorithms iteratively modify model genotypes according to fitness scores, whereas gradient-based training adjusts parameters in the negative-gradient direction.
  • Method: SGD is identified as the most commonly used training method for a variety of machine-learning models.
  • Model families: Neural-network families include multilayer perceptrons, deep and convolutional networks, recurrent networks, Hopfield networks, self-organizing maps, stochastic networks, auto-encoders, and GANs.

3.2 Hyperparameter Optimization

Hyperparameter optimization addresses the problem-specific choice of settings that strongly affects machine-learning performance. The survey groups reusable optimization approaches by the order or strategy used to search parameter space.

  • Optimal hyperparameter values often differ across problem domains, machine-learning models, and datasets.Examples include batch size, learning rate, and model initialization in stochastic gradient descent.
  • Automatic optimization algorithms can tune machine-learning parameters and be reused across different algorithm families.
  • First-order methods use at least one derivative of parameter-dependent accuracy, including stochastic gradient descent and conjugate-gradient methods.
  • Second-order methods use second derivatives of parameter-dependent accuracy, with Newton’s method as an example.
  • Coordinate descent optimizes one variable per iteration, while Markov-Chain Monte-Carlo proposes random parameter updates based on likelihood.

3.3 Combining Multiple Algorithms: Ensemble Methods

Ensemble learning combines multiple models to improve accuracy, particularly when data is inherently distributed and centralization is unavailable. The survey describes several ensemble strategies and their distinct training or selection procedures.

  • Ensemble learning combines multiple models when a single model is not accurate enough for the application.
  • For inherently distributed data, ensemble training can proceed locally at data sites before a global site aggregates the individual results.
  • Bagging builds multiple classifiers and combines them, whereas boosting trains new models on data misclassified by previous models.
  • Bucketing trains many models and selects the one with the best performance.
  • Random Forests use multiple decision trees and average their predictions to increase overall accuracy.

3.4 Topologies

Distributed machine-learning topologies organize cluster computers according to the system’s degree of distribution. They range from centralized aggregation to decentralized and fully distributed arrangements, with trees, rings, and parameter servers among popular choices.

  • Topology design is determined partly by the degree of distribution, with Figure 3 presenting four possible arrangements.
  • Centralized systems aggregate hierarchically at one location, while decentralized systems support intermediate aggregation through trees or partitioned parameter servers.
  • Fully distributed systems use independent nodes that ensemble the solution without assigning specific roles to particular nodes.
  • Tree topologies are easy to scale and manage because each node communicates only with its parent and child nodes.
  • Ring topologies simplify AllReduce communication by requiring synchronization only between neighboring nodes.
  • Parameter Server systems use decentralized workers and centralized masters that store model-parameter shards as a shared key-value state.

3.5 Communication

Distributed training balances accuracy, computation time, and communication cost, because partitioning and synchronization choices affect both convergence and data exchange. The survey presents scheduling concerns and communication models that trade synchronization and convergence guarantees against fresher or faster updates.

  • Data or program partitioning directly affects the communication required to train a distributed model.
  • Parallel learning can reduce computation time, but communication costs may become dominant as model size and training complexity increase.
  • Separate local models avoid communication but can reduce individual accuracy, while ensembling improves overall accuracy without typically reducing computation time substantially.
  • Synchronizing models during training can accelerate convergence, but communication cost increases with model size.
  • Practical deployments seek the communication required to reach target accuracy within an acceptable computation time.
  • Workload scheduling must identify parallel tasks, determine execution order, and balance loads across machines.
  • BSP synchronizes every computation and communication phase, guaranteeing a correct solution but making faster workers wait at barriers.
  • SSP lets faster workers advance within a staleness bound, retaining convergence guarantees while excessive staleness deteriorates convergence rates.

3.6 Discussion

As machine learning applications have expanded, systems concerns have become increasingly important, but understanding of distributed machine-learning workloads remains preliminary. This motivates co-designing algorithms and systems around resource use.

  • Recent advances have driven neural networks into increasingly important applications, while non-functional system aspects received less attention.
  • The community therefore has only a preliminary understanding of how distributed machine-learning algorithms and systems behave as workloads.
  • Resource consumption and return on investment are making systems aspects more important as machine-learning technology matures.
  • Machine-learning algorithms and systems are increasingly co-designed by adapting algorithms to system resources and designing systems for algorithm classes.

4 THE DISTRIBUTED MACHINE LEARNING ECOSYSTEM

The distributed machine-learning ecosystem combines general-purpose distributed platforms, native machine-learning systems, and cloud-delivered services. Native systems are organized around specific algorithmic and operational models.

  • General-purpose distributed platforms provide part of the foundation for processing large datasets, while native systems target distributed machine-learning workloads.
  • Native distributed machine-learning systems are designed around models such as distributed ensemble learning, parallel synchronous SGD, and parameter servers.
  • Although most systems are user-operated and on-premise, machine-learning services delivered through the cloud are increasingly diverse.

4.1 General Purpose Distributed Computing Frameworks

General-purpose distributed frameworks scale computation across commodity machines, using fault tolerance and distributed storage to handle large workloads. MapReduce and Spark provide different execution models, with Spark supporting iterative machine-learning workloads through resilient datasets and MLlib.

  • General Purpose Distributed Computing Frameworks: Commodity-server clusters can be more affordable than specialized large machines when software provides sufficient fault tolerance.
  • General Purpose Distributed Computing Frameworks: Distributed storage commonly uses block-based architectures with replicated chunks, while task scheduling and resource distribution retain competing designs and trade-offs.
  • MapReduce: MapReduce separates parallel map, shuffle, and reduce phases, moving computation toward data and rescheduling failed tasks.
  • MapReduce: Hadoop recreates MapReduce’s architecture with HDFS, while later systems support more flexible forest and DAG data flows.
  • Apache Spark: Spark addresses iterative workloads by avoiding repeated intermediate storage and using RDD lineage for recovery; MLlib adds scalable algorithms and workflow utilities.

4.2 Natively Distributed Machine Learning Systems

Natively distributed systems parallelize machine-learning training through ensemble, synchronous, asynchronous, and model-parallel strategies, each trading communication, scalability, efficiency, and fault tolerance differently.

  • Distributed Ensemble Learning: Ensemble learning trains independent models on data subsets and combines their outputs at prediction time.The approach can use existing machine-learning and distribution frameworks, but depends on appropriate data subdivision.
  • Distributed Ensemble Learning: Data subdivision must preserve independence and identical distribution because biased subsets can produce biased ensemble outputs.This requirement is difficult when the dataset is inherently distributed.
  • Parallel Synchronous Stochastic Gradient Descent: Synchronous approaches aggregate workers’ gradients through operations such as AllReduce after training iterations.AllReduce combines local gradients into a common model state and propagates the result back to workers.
  • Parallel Synchronous Stochastic Gradient Descent: 90% efficiency was achieved for Caffe2 training of ResNet-50 on up to 352 GPUs, with linear scaling but no fault tolerance.The system uses Ring AllReduce and recursive halving and doubling for different bandwidth and latency conditions.
  • Parallel Synchronous Stochastic Gradient Descent: 85%+ speedups were achieved on a Microsoft speech LSTM with up to 16 GPUs, but scalability fell below 70% beyond that.Direct comparison with other synchronous frameworks is uncertain because LSTMs have temporal dependencies unlike ordinary deep neural networks.
  • Parallel Asynchronous Stochastic Gradient Descent and Parameter Servers: DistBelief reported more than 12x speedup on 81 machines for a 1.7-billion-parameter model, while using over 500 machines reduced ImageNet training time by 80%.These gains required substantial computational resources, and model-parallel efficiency depended strongly on architecture and cross-partition communication.
  • Parallel Asynchronous Stochastic Gradient Descent and Parameter Servers: TensorFlow achieved about 50% efficiency on a four-node ResNet-50 cluster and about 75% on GoogleNet, while MXNet achieved almost linear speedup on ten GPU machines.These results indicate that communication overhead and neural-network architecture affect distributed-training efficiency.

4.3 Machine Learning in the Cloud

Cloud operators provide distributed machine-learning services spanning infrastructure-level virtual machines to software-level managed offerings, while cloud deployments also stimulate new system development.

  • Machine Learning in the Cloud: Cloud providers offer machine-learning execution options ranging from IaaS virtual machines with prepackaged software to SaaS-level Machine Learning as a Service.Much of the offered technology consists of standard distributed machine-learning systems and libraries.
  • Machine Learning in the Cloud: Cloud platforms both consume distributed machine-learning technology and fuel the development of systems designed for large-scale deployments.The cloud therefore participates in the broader distributed-machine-learning ecosystem beyond simply hosting workloads.

5 CONCLUSIONS AND CURRENT CHALLENGES

Distributed machine learning has become an established ecosystem, but its long-term success remains constrained by performance-efficiency trade-offs, fault tolerance, privacy, and portability challenges.

  • Distributed machine learning now offers diverse solutions differing in architecture, algorithms, performance, and efficiency, while several open challenges remain.The field has industry-grade systems, but single-machine solutions are increasingly becoming the exception.
  • 5.1 Performance: Adding resources can reduce wall-clock training time while increasing aggregate processing time, energy consumption, and reducing efficiency.TensorFlow GPU deployments often exhibit efficiency below 75%, while some synchronous SGD frameworks achieve linear benchmark speedups at smaller scales.
  • 5.2 Fault Tolerance: Synchronous AllReduce scales better than parameter servers up to a certain cluster size, but one machine failure can block the entire training process.Fault-tolerant implementations remain unavailable in production-ready form, and checkpointing requires significant effort.
  • 5.3 Privacy: Distributed privacy remains difficult: federated learning can keep data local, but record-level differential privacy is generally ineffective against GAN-based attacks.The paper concludes that current frameworks provide little support even for basic privacy forms.
  • 5.4 Portability: Model portability is limited by framework-specific storage formats and increasing hardware diversity across CPUs, GPUs, and custom ASICs.ONNX is presented as a framework-independent specification supported directly or through converters by several frameworks.
Loading 1912.09789v1…