Source-linked AI summary
SparkNet: Training Deep Networks in Spark
Philipp Moritz, Robert Nishihara, Ion Stoica, Michael I. Jordan
TL;DR
Training large deep networks is costly, while batch frameworks are poorly suited to the asynchronous, communication-intensive workloads of distributed deep learning. SparkNet combines Spark and Caffe with a communication-efficient parallel SGD scheme, achieving speedups on ImageNet and remaining effective under high communication overhead.
Problem
Training large deep networks can take multiple days on a single GPU, while popular batch frameworks are not designed for distributed deep-learning workloads.
Method
SparkNet integrates Caffe-based deep-network training into Spark using parallel SGD that synchronizes worker models after local updates.
Results
SparkNet achieves speedups over single-node training on ImageNet and maintains relatively consistent speedup when communication overhead is 100 times minibatch computation cost.
Takeaways & Limitations
SparkNet provides an easy-to-use way to train existing Caffe models within Spark, including in highly bandwidth-limited settings.
Takeaways & Limitations
Naive parallelization is strongly constrained by communication overhead, with maximum speedup bounded by C(b)/S.
Abstract
from arXiv · showhide
Training deep networks is a time-consuming process, with networks for object recognition often requiring multiple days to train. For this reason, leveraging the resources of a cluster to speed up training is an important area of work. However, widely-popular batch-processing computational frameworks like MapReduce and Spark were not designed to support the asynchronous and communication-intensive workloads of existing distributed deep learning systems. We introduce SparkNet, a framework for training deep networks in Spark. Our implementation includes a convenient interface for reading data from Spark RDDs, a Scala interface to the Caffe deep learning framework, and a lightweight multi-dimensional tensor library. Using a simple parallelization scheme for stochastic gradient descent, SparkNet scales well with the cluster size and tolerates very high-latency communication. Furthermore, it is easy to deploy and use with no parameter tuning, and it is compatible with existing Caffe models. We quantify the dependence of the speedup obtained by SparkNet on the number of machines, the communication frequency, and the cluster's communication overhead, and we benchmark our system's performance on the ImageNet dataset.
1 INTRODUCTION
SparkNet addresses the difficulty of training large deep networks in batch-processing frameworks by combining distributed SGD with Spark’s data-processing infrastructure. It is designed to remain effective under bandwidth-limited communication while integrating training with existing data pipelines.
- Large models and datasets can require multiple days to train on a single GPU, motivating cluster-based distributed optimization.
- Batch frameworks simplify distributed data analytics but were not designed for asynchronous, communication-intensive deep-learning workloads.
- SparkNet parallelizes SGD in batch frameworks such as Spark and MapReduce while targeting bandwidth-limited environments.
- Integrating training with batch data pipelines keeps data in memory and supports workflows spanning SQL or graph outputs through distributed predictions.
- On a five-node EC2 cluster, SparkNet handles parameter broadcasts and collections taking about 20 seconds while a minibatch gradient computation takes about 2 seconds for AlexNet.
2 IMPLEMENTATION
SparkNet combines Spark and Caffe through Scala interfaces, tensor utilities, and a lightweight synchronization scheme for distributed SGD. Its implementation supports existing Caffe models and reduces communication by synchronizing after multiple local updates.
- SparkNet builds on Spark and Caffe, exposing Scala access to Caffe data and weights for distributed deep-network training.
- The Net API specifies architectures, training and validation data, testing, weights, and weight retrieval for Caffe-backed networks.
- WeightCollection maps layer names to weight lists, while NDArray provides lightweight multidimensional tensors without copying Caffe memory.
- Existing Caffe model definitions and solver files are automatically compatible with SparkNet, and Spark supplies in-memory data loading and preprocessing.
- Each worker runs SGD on its data subset for a fixed number of iterations or time, after which worker models are averaged and redistributed.
- Synchronizing after multiple local updates reduces communication relative to exchanging model parameters after every SGD update.
3 EXPERIMENTS
The experiments analyze SparkNet’s speedup as a function of synchronization frequency, cluster size, and communication overhead, then benchmark it on ImageNet. SparkNet remains effective under communication-heavy conditions and achieves substantial multi-node speedups.
- Naive parallelization limits: Naive parallelization is constrained because effective minibatch scaling requires K much smaller than b, larger batches provide modest benefit, and communication overhead limits speedup.Its theoretical speedup is C(b)/(C(b)/K + S), which is at most C(b)/S.
- SparkNet parallelization: SparkNet synchronizes worker models in rounds: each worker runs τ SGD iterations, after which models are averaged and broadcast.This reduces synchronization frequency relative to updating after every SGD iteration.
- Experimental setup: The zero-communication-overhead analysis measures speedup across choices of τ and K using modified AlexNet on a 100-class ImageNet subset.Speedup is measured relative to single-machine training for a fixed target accuracy.
- Dependence on τ and K: Modest delays between synchronizations can improve speedup, while synchronizing every iteration is equivalent to serial SGD with batch size Kb.With K = 1, synchronization frequency has no effect and the speedup is approximately 1.
- Communication overhead: On a five-node cluster, SparkNet maintains relatively consistent speedup even when communication overhead is 100 times the cost of a minibatch computation.By contrast, naive parallelization provides no speedup when communication overhead is comparable to or greater than minibatch computation cost.
- ImageNet benchmarks: For GoogLeNet, SparkNet achieves speedups of 2.7 and 3.2 over a four-GPU single-node baseline on 3-node and 6-node clusters, respectively.Relative to Caffe on a single GPU, the corresponding speedups are roughly 9.4 and 11.2.
4 RELATED WORK
Prior distributed deep-learning systems achieved high performance through custom, communication-intensive designs, while general-purpose frameworks offered easier large-scale data processing but limited deep-network support. SparkNet addresses this gap by training deep networks in Spark while tolerating lower-bandwidth communication and integrating with existing data pipelines.
- Custom distributed systems: Custom systems such as DistBelief and MPI-based approaches target high-performance distributed deep-network training with fine-grained scheduling and low-latency communication.These systems use stochastic or batch optimization and exploit data or model parallelism.
- Pipeline integration: Training deep networks within data-processing pipelines matters because data may originate from streaming sources, SQL queries, or graph computations.Custom systems can require separate data-transfer steps when training uses outputs from these sources.
- SparkNet’s distinction: SparkNet differs from prior Spark training work by tolerating low-bandwidth communication and running out of the box on Amazon EC2.Prior work used remote direct memory access over InfiniBand to exchange parameters quickly between GPUs.
- Complementary parallelism: Single-node CPU/GPU parallelism methods are compatible with SparkNet and can be used together with its cross-node parallelization.This makes within-node and across-node parallelism complementary rather than competing approaches.
- General-purpose frameworks: General-purpose computational frameworks support many machine-learning models, but demanding communication requirements have limited their extension to deep networks.This creates a gap between accessible batch-processing infrastructure and distributed deep-learning workloads.
5 DISCUSSION
SparkNet integrates deep-network training with Spark and existing Caffe models, enabling models to combine data from diverse distributed sources. Experiments quantify speedup across cluster size, communication frequency, and communication overhead, showing effectiveness in bandwidth-limited settings and speedup over a single node on full ImageNet.
- Contribution: SparkNet provides an easy-to-use Spark implementation that parallelizes existing Caffe models with minimal modification.Its integration supports features from distributed files, SQL or graph queries, and streaming sources.
- Evaluation: SparkNet quantifies speedup as functions of cluster size, communication frequency, and cluster communication overhead using a smaller ImageNet benchmark.The evaluation also tests highly bandwidth-limited settings.
- Evaluation: On full ImageNet, SparkNet achieves a sizable speedup over a single-node experiment using few GPUs.The passage reports this as a benchmark result without specifying the numerical speedup.
- Availability: SparkNet’s code is publicly available and intended to support contributions to the Spark deep-learning community.The project invites contributions from users developing diverse applications.