Source-linked AI summary
Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
Ruben Mayer, Hans-Arno Jacobsen
TL;DR
As DL models and datasets grow, scalable distributed infrastructures must address coordination, resource, and data-management challenges. The survey synthesizes infrastructure, parallelization, scheduling, elasticity, and data-management techniques, compares 11 open-source tools, and identifies future directions including data management, decentralized training, heterogeneous infrastructures, serving, and privacy.
Problem
Growing DL models and datasets make distributed scalability necessary, while shared infrastructures must manage parallelization, scheduling, elasticity, and training and model data.
Method
The survey classifies and compares distributed DL infrastructures and techniques, then analyzes 11 open-source frameworks and tools in relation to parallel and distributed training research.
Results
The survey identifies infrastructure, parallelization, scheduling, and data management as central scalable-DL concerns and synthesizes how current systems address them.
Takeaways & Limitations
Future DL tools need stronger training-data and model-data management, support for decentralized training, and the ability to exploit infrastructure heterogeneity.
Takeaways & Limitations
Data parallelism does not solve models that exceed one device’s memory, while model parallelism can incur communication and synchronization delays that prevent speedup.
Abstract
from arXiv · showhide
Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.
1 INTRODUCTION
Deep learning’s progress is linked to scaling models, training data, and infrastructure, but distributed deployments create challenges across parallelization, scheduling, elasticity, and data management. This survey organizes and compares research and open-source tools addressing these challenges.
- Motivation: DL success is driven by scaling model complexity, training data, and infrastructure, including 10s to 100s of TB of training data.Programmable highly parallel hardware, especially GPUs, enables training large models with substantial data in shorter times.
- Scope: Managing large distributed DL infrastructures requires solutions for parallelization, resource scheduling, elasticity, data management, and portability.The survey brings together contributions from distributed systems, data management, and machine learning communities.
- Related Work: Existing surveys cover DL architectures, applications, history, hyper-parameter selection, or Big Data challenges but are not focused on scalable distributed DL systems.Their coverage therefore differs from this survey’s distributed-infrastructure perspective.
- Contribution: This survey also covers resource scheduling, multi-tenancy, data management, and in-depth comparisons of open-source DL frameworks and tools.These aspects are presented alongside research on parallel and distributed DL training.
- Structure: The survey discusses distributed infrastructures, parallel DL training, resource scheduling, data management, and 11 open-source frameworks and tools.It concludes with current trends and open problems requiring further research.
2 FOUNDATIONS
The foundations describe neural networks as layered functions trained to approximate target functions, then introduce common architectures and their limitations. Fully connected MLPs can be memory-intensive and ignore local input structure, motivating more specialized designs such as CNNs.
- AI, ML, and DL: AI research includes symbolic, sub-symbolic, and statistical machine learning methods, with DL situated within this broader AI and ML landscape.
- Neural Networks: A neural network transforms input signals into outputs through interconnected artificial neurons arranged between input and output layers.The network represents a function mapping inputs x to outputs y.
- Training: Training adjusts weights, biases, and thresholds so the network output approximates a target function, commonly using back-propagation on loss gradients.
- Neural Network Architectures: A multi-layer perceptron uses fully connected layers, but its many weights increase memory and sample requirements and make it insensitive to input topology and geometric variation.These limitations are problematic for structured inputs such as images and speech.
- Neural Network Architectures: CNNs use convolutional and sub-sampling layers with local receptive fields and shared weights instead of fully connected connections.
3 DISTRIBUTED DEEP LEARNING
Distributed DL systems address the difficulty of training large models on vast datasets across multiple compute nodes and GPUs. The section introduces system-level challenges and research directions spanning infrastructure, parallel training, scheduling, and data management.
- Challenges: Training large DL models with vast datasets across multi-GPU compute nodes must avoid costly resource stalls caused by communication bottlenecks.
- Challenges: Shared compute, storage, and network resources introduce multi-user and multi-process management requirements associated with cloud-style elasticity.
- Research Directions: DL systems research examines major challenges, state-of-the-art approaches, and open problems at the intersection of computing systems and deep learning.
3.1 Infrastructure
Scalable DL infrastructure spans hardware, networking, software, and data-center-scale system design. Real deployments are heterogeneous, while specialized components and communication protocols are used to improve performance.
- Infrastructure scope: The survey organizes infrastructure coverage around hardware innovations, data-center-scale deployments, networking architectures, and low-level DL software.These infrastructure layers support later discussions of parallelization, scheduling, and data management.
- Hardware components: DL infrastructure evolved from multicore CPU clusters toward highly parallel GPUs and specialized hardware for training and serving.GPU benefits over CPUs depend on workload characteristics, implementation efficiency, and hardware.
- Hardware components: FPGAs can improve DL energy efficiency, but limited memory capacity requires techniques such as tiling, FIFO buffering, and pipelined processing.Caffeine reports up to 43.5x lower energy than CPU and up to 1.5x lower energy than GPU execution; DLAU reports up to 10x less energy than GPUs.
- Hardware components: Neuromorphic hardware uses in-memory computing inspired by brain physiology, but remains experimental and not widely available.Neurostream is described as a processor-in-memory solution tailored to CNN training.
- Large-scale infrastructure: Large-scale deployments combine CPUs, GPUs, custom hardware, and HPC infrastructure, with systems specialized to available hardware and communication technologies.Examples include Google’s TPUs and specialized interconnects, while HPC systems use protocols such as RDMA, NCCL, and MPI.
- Performance considerations: Infrastructure performance depends on throughput, latency, energy consumption, communication protocols, and hardware composition that avoids bottlenecks.The survey identifies RDMA and component composition as important factors beyond raw hardware performance.
3.2 Parallelization Methods
DL training uses data, model, pipeline, and hybrid parallelism, each distributing computation or model state differently. Their benefits are balanced by synchronization, communication, memory, and partitioning challenges.
- Data parallelism: Data parallelism replicates the model across workers, splits training data, and synchronizes parameter updates.It applies broadly to model architectures and scales well for compute-intensive operations with few parameters.
- Data parallelism: Data parallelism becomes limited when parameter synchronization dominates, large batch sizes increase staleness, or the model cannot fit on one device.Many schemes also assume independently and identically distributed training data so worker updates can be summed.
- Model parallelism: Model parallelism partitions a DL model across workers, propagating activations forward and gradients backward between assigned layers.Its main benefit is reducing each worker’s memory footprint when the complete model exceeds one device’s capacity.
- Model parallelism: Model parallelism can stall because effective partitioning is difficult and communication or synchronization overhead may offset training speedups.Reinforcement learning is commonly used to search for improved model partitions.
- Pipeline parallelism: Pipeline parallelism combines model and data parallelism by streaming multiple micro-batches through model partitions, increasing worker utilization while preserving memory savings.GPipe and PipeDream are cited as pipeline-parallel approaches.
- Hybrid parallelism: Hybrid approaches mix parallelization methods because complex models contain layers with different architectures and parallelization needs.Mesh-TensorFlow splits data into shards and tensors along attributes to combine data and model parallelism; automated methods use cost models, graph search, or simulation.
3.3 Optimizations for Data Parallelism
Data-parallel DL systems must choose how and when workers synchronize parameters while controlling communication costs. Centralized, decentralized, and federated architectures offer different scalability, fault-tolerance, bandwidth, and implementation trade-offs.
- System Architecture: Parameter synchronization requires decisions about architecture, timing, and communication-overhead reduction.Workers may use centralized or decentralized synchronization, synchronize after each batch or tolerate stale parameters, and reduce communication costs.
- System Architecture: Decentralized training exchanges updates through allreduce; fully connected communication costs O(n^2), motivating ring and alternative topologies.Ring-allreduce is implemented by Horovod using NCCL, while alternative topologies trade lower communication costs against slower update propagation.
- System Architecture: Decentralized architectures avoid parameter-server configuration and single points of failure, simplifying deployment and enabling workload takeover after node failures.They also avoid heavyweight checkpointing of parameter-server state.
- System Architecture: Decentralized communication can grow quadratically with worker count, while topology changes and gradient partitioning introduce additional complexity and trade-offs.The survey concludes that no universal solution exists for synchronizing parallel parameter updates.
- System Architecture: Federated learning keeps training data on users’ mobile devices and trains a global model from locally computed updates rather than assuming a controlled, homogeneous data-center environment.Federated averaging selects a fraction of clients each round and centrally aggregates their gradients; it performs comparably to centralized learning with i.i.d. data but worse with non-i.i.d. data.
- Synchronization: Bounded asynchronous synchronization is not widely implemented in DL frameworks, with limited reported benefits on uniformly performing GPU-intensive workloads.Exceptions include BMUF in CNTK and bounded-asynchronous implementations in Petuum, while some earlier systems are no longer actively developed.
3.4 Scheduling and Elasticity
Scheduling and elasticity map training processes and jobs onto shared or changing resources while accounting for placement, execution order, scaling, fairness, locality, and model dependencies. The survey covers single-tenant, multi-tenant, and architecture or hyper-parameter search scheduling.
- Scheduling Scope: DL scheduling covers single-tenant placement and elasticity, multi-tenant allocation and fairness, and scheduling the jobs created by model and hyper-parameter search.These three aspects are coupled because search repeatedly creates training jobs that compete for shared distributed resources.
- Single-tenant: Single-tenant scheduling decides process placement, execution order on shared resources, and adaptation of process or resource counts for one training job.Resources may include compute nodes, CPUs, and GPUs, and processes may include data-parallel workers, model-parallel workers, and parameter servers.
- Single-tenant: Model-parallel scheduling must coordinate model partitioning, partition placement, and training order because each partition depends on receiving its input data.Dynamic control flow can make static scheduling infeasible, while proposed heuristics have not been implemented in DL systems.
- Elasticity: Some elastic systems dynamically schedule micro-tasks or workers and adapt resources, using dependency and consistency information or resource-availability monitoring.Litz supports interruption-free elasticity through over-partitioned input data, while CROSSBOW tunes worker counts until throughput no longer increases.
- Multi-tenant: Multi-tenant scheduling must handle competing jobs whose convergence rates vary over time, creating requirements beyond general-purpose resource scheduling.Additional concerns include fairness, locality, queuing delays, co-location interference, and failures.
- Model and Hyper-parameter Search: Hyper-parameter and architecture search repeatedly spawns differently configured training jobs that must be scheduled on shared distributed resources.Search may be random or guided by random forests, Bayesian optimization, or reinforcement learning.
3.5 Data Management
Large-scale DL requires coordinated management of training data and model artifacts across distributed infrastructure. The survey discusses data acquisition, preprocessing, provisioning, model management, and interoperability.
- Training and model data must be managed alongside distributed infrastructure, active training processes, and data-center resource scheduling.
- Large labeled datasets can be built through manual labeling, but this approach is expensive and time-consuming.The passage contrasts manual labeling with training on more readily available noisy data.
- Preprocessing includes normalization and augmentation, while distorted training data can improve robustness to noisy inputs.Examples include cropping, resizing, and creating speech spectrograms.
- Object storage can provide training data cheaply, but its provisioning bandwidth may be far below multi-GPU training throughput.A single object-storage request is limited to about 35 MB/s, whereas a four-GPU machine can consume up to 570 GB/s.
- Model management tracks, stores, and indexes trained models to support their sharing, querying, and analysis.
- ONNX is presented as a de-facto exchange format that facilitates interoperability between DL frameworks.Several frameworks natively support ONNX, while converters exist for additional tools.
4 COMPARISON OF DEEP LEARNING FRAMEWORKS
The survey compares open-source DL frameworks and libraries by APIs, distributed and parallel training support, community activity, and related capabilities. It explicitly excludes framework performance evaluation.
- The comparison covers open-source DL frameworks and tools because their differing parallelization and distribution concepts support scalable DL practice.
- Evaluation criteria: Frameworks are assessed using APIs, distribution and parallelization support, and community activity.Community activity is measured through recent GitHub commits and StackOverflow topic counts.
- Evaluation criteria: The survey does not compare framework performance because comprehensive performance evaluation is outside its scope.
- Framework capabilities: Caffe lacks out-of-the-box distributed training, whereas Caffe2 natively supports synchronous decentralized data-parallel training.
- Framework capabilities: Chainer enables distributed multi-node training through ChainerMN, while CNTK supports distributed data-parallel training and integrates 1-bit stochastic gradient descent.
- Framework capabilities: Deeplearning4j supports distributed and parallel training through Spark, while Keras provides GPU parallelism but delegates higher-level parallelization to its host framework.
- Framework capabilities: MXNet, PyTorch, SINGA, and TensorFlow support combinations of data-parallel, model-parallel, centralized, decentralized, synchronous, or asynchronous training.Their supported architectures and training modes differ across frameworks.
5 CONCLUSIONS AND OUTLOOK
The survey synthesizes scalable DL challenges, techniques, infrastructure, and tools, then identifies future priorities including data management, multi-tenant scheduling, serving, heterogeneous systems, and privacy. It reports broad implementation of scalable-DL techniques in open-source tools.
- The survey reviews distributed infrastructures, training parallelization, resource scheduling, data management, and open-source DL systems and tools.
- Open-source DL frameworks implement a wide range of scalable-DL techniques, indicating interaction between research and practical applications.
- Training and model data management becomes a larger challenge as the number of training datasets and DL models grows.
- Heterogeneous infrastructures require DL tools that can handle and exploit heterogeneity to optimize training.
- Future infrastructure research should address data management, multi-tenant scheduling, DL serving, and privacy alongside training parallelization.The survey distinguishes serving requirements from training requirements and notes a trade-off between more training data and data avoidance for privacy.