PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication

Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, Wojciech Samek

arXiv:1805.08768v1cs.LGcs.AIcs.DCstat.ML

TL;DR

Distributed training is constrained by the communication required between clients, especially for large models and bandwidth-limited systems. The paper proposes sparse binary compression, combining communication delay, gradient sparsification, binarization, and weight-update encoding. Across convolutional and recurrent networks, SBC reduces communication by more than four orders of magnitude while preserving approximately baseline convergence and accuracy.

  • Problem

    Communication cost and limited bandwidth constrain distributed deep-learning training, particularly as models, datasets, and numbers of computation nodes grow.

  • Method

    Sparse binary compression combines communication delay, gradient sparsification, sparse binarization, and weight-update encoding to reduce distributed-training communication.

  • Results

    More than four orders of magnitude of communication reduction is achieved across convolutional and recurrent architectures without significantly harming convergence speed in forward-backward passes.

  • Takeaways & Limitations

    Gradient sparsity and communication delay can be treated as independent forms of sparsity, enabling their trade-off to adapt compression to training requirements.

Abstract

from arXiv · show

Currently, progressively larger deep neural networks are trained on ever growing data corpora. As this trend is only going to increase in the future, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes or prohibitive communication cost in general. These challenges become even more pressing, as the number of computation nodes increases. To counteract this development we propose sparse binary compression (SBC), a compression framework that allows for a drastic reduction of communication cost for distributed training. SBC combines existing techniques of communication delay and gradient sparsification with a novel binarization method and optimal weight update encoding to push compression gains to new limits. By doing so, our method also allows us to smoothly trade-off gradient sparsity and temporal sparsity to adapt to the requirements of the learning task. Our experiments show, that SBC can reduce the upstream communication on a variety of convolutional and recurrent neural network architectures by more than four orders of magnitude without significantly harming the convergence speed in terms of forward-backward passes. For instance, we can train ResNet50 on ImageNet in the same number of iterations to the baseline accuracy, using $\times 3531$ less bits or train it to a $1\%$ lower accuracy using $\times 37208$ less bits. In the latter case, the total upstream communication required is cut from 125 terabytes to 3.35 gigabytes for every participating client.

I. INTRODUCTION

Distributed SGD enables clients to train a shared deep-learning model from separate data, but communication can become a major bottleneck. Prior approaches reduce communication frequency, update sparsity, or value precision, each targeting different components of total communication.

  • Distributed training: DSGD lets multiple clients jointly train a deep-learning model using separate local data and periodically exchange weight updates to produce a shared master model.Clients may communicate through a centralized server or an all-reduce operation.
  • Communication bottleneck: Gigabyte-scale models and hundreds of thousands of iterations can make communicating full updates prohibitively expensive for every client.Vanilla DSGD communicates a full gradient update during every iteration, with each update matching the full model size.
  • Communication bottleneck: Communication cost limits distributed-learning performance in both high-frequency cluster training and bandwidth-constrained mobile or embedded-device settings.These settings also motivate communication reduction for device-level parallelism and privacy-preserving learning.
  • Communication bottleneck: Total communication depends multiplicatively on training iterations, communication frequency, nonzero update count, position and value bits, and receiving nodes.For dense updates, positions are predetermined and require no position bits.
  • Prior compression approaches: Sparsification reduces the number of transmitted nonzero updates, but fixed magnitude thresholds can be difficult to choose across architectures and layers.Threshold-based residual aggregation has achieved compression rates of up to 3 orders of magnitude on acoustic modeling.
  • Prior compression approaches: Communication-delay methods reduce communication frequency, while dense quantization methods reduce value bits but may impose accuracy or assumption trade-offs.Federated Averaging reports ×10 - ×100 fewer communication rounds, whereas TernGrad reports ×16 compression with noticeable accuracy drops on large architectures.

II. SPARSE BINARY COMPRESSION

Sparse Binary Compression combines communication delay, sparse updates, binarization, residual accumulation, and position encoding to reduce distributed-training communication while preserving the optimization path.

  • Communication Delay: Communication delay lets clients perform multiple local SGD iterations before exchanging more informative weight-updates.For n = 1, the generalized update becomes regular DSGD; prior empirical analysis found drastic delay with marginal accuracy degradation.
  • Sparse Binarization: Sparse binarization retains only the largest-magnitude fraction p of updates, then represents the remaining nonzero values using one positive or negative mean.The retained fraction is selected by magnitude, and the sign with the larger absolute mean determines which values remain nonzero.
  • Residual Accumulation: Residual accumulation keeps the compressed optimization path as close as possible to the path produced by non-compressed weight-updates.The paper states that the orthogonal projection uniquely minimizes accumulated error in the relevant subspace.
  • Optimal Position Encoding: For sparsity rate p = 0.01, Golomb position encoding requires an average of 8.38 bits and provides ×1.9 compression versus 16-bit naive distance encoding.The encoding communicates distances between nonzero elements rather than their absolute positions.
  • Momentum Correction, Warm-up Training and Momentum Masking: The approach adopts momentum masking, with momentum correction implicit in the proposed method.These modifications are included to improve convergence speed relative to vanilla Gradient Dropping.

III. TEMPORAL VS GRADIENT SPARSITY

SBC allows temporal and gradient sparsity to be traded against one another while preserving similar validation error at comparable total sparsity. This flexibility supports adaptation to changing communication constraints.

  • Temporal versus gradient sparsity: SBC smoothly trades temporal sparsity against gradient sparsity, allowing configurations to adapt to varying communication constraints.Communication conditions may change during distributed training, motivating adjustable combinations of frequent small updates and delayed larger updates.
  • Temporal versus gradient sparsity: Validation error remains more or less constant along off-diagonals where the product of temporal and gradient sparsity is fixed.The comparison uses ResNet32 trained on CIFAR for 60000 iterations.
  • Temporal versus gradient sparsity: Figure 3 maps validation error across temporal and gradient sparsity, with brighter colors indicating lower error.The plot concerns ResNet32 trained on CIFAR.
  • Temporal versus gradient sparsity: Figure 4 compares classification error across total sparsity and training iterations, distinguishing purely temporal, purely gradient, and hybrid methods by color.Purple denotes purely temporal sparsified SGD, yellow denotes purely gradient sparsified SGD, and interpolated colors denote hybrids.

A. Networks and Datasets

The evaluation uses convolutional and recurrent neural networks on five established datasets, with four clients receiving balanced data partitions. The models are sufficient to evaluate compression and its compatibility with batch normalization and dropout, although they do not fully reach state-of-the-art results.

  • Networks and Datasets: Experiments use four clients with balanced, homogeneous data partitions across the participating clients.The training data are split so clients receive equal numbers and distributions of samples.
  • Networks and Datasets: Image-classification benchmarks include LeNet5-Caffe2 on MNIST, ResNet32 on CIFAR-10, and ResNet50 on ImageNet.MNIST uses Adam, while the other image-classification models use momentum SGD.
  • Networks and Datasets: Language-modeling benchmarks use multilayer sequence-to-sequence LSTMs on Penn Treebank and Shakespeare for next-word and next-character prediction.The PTB setup includes a two-layer WordLSTM with 650 hidden units and a 10000-word vocabulary plus an additional rare-word token.
  • Networks and Datasets: The evaluated models do not fully achieve state-of-the-art results but are sufficient for compression evaluation and work with batch normalization and dropout.The paper provides complete model and hyperparameter descriptions in the supplement.

B. Results

Across five datasets with a fixed iteration count, SBC methods achieve validation performance comparable to the baseline while communicating substantially fewer bits. Results also show that different SBC configurations can preserve similar convergence and final accuracy, with communication measured for both image and language models.

  • B. Results: SBC configurations include no communication delay with 0.1% gradient sparsity, or delays of 10 or 100 iterations with 1% gradient sparsity.The configuration choices are described as somewhat arbitrary points on the two-dimensional design grid.
  • B. Results: Table II reports final accuracy or perplexity and compression rate for each evaluated compression scheme.The table compares different compression methods under the reported benchmark settings.
  • B. Results: SBC methods perform comparably to the baseline on all five benchmarks while communicating significantly fewer bits.Table II reports final accuracy or perplexity together with compression rates for different compression schemes.
  • B. Results: For ResNet50 on ImageNet, Figure 5 plots top-1 validation accuracy against epochs and top-1 validation error against transferred bits on logarithmic axes.Learning-rate reductions at epochs 30 and 60 are marked.
  • B. Results: SBC (2) has similar convergence speed and final accuracy to SBC (1) while retaining a compression edge of about ×1.3 - ×2.2.This comparison is reported across the experiments.
  • B. Results: On PTB WordLSTM, all models converge to approximately the same perplexity after around 60 epochs, although Federated Averaging and SBC (3) initially converge more slowly in iterations.Figure 6 reports convergence against both epochs and communicated bits.

V. CONCLUSION

SBC combines multiple compression strategies to reduce distributed-training communication by up to four orders of magnitude with only a slight accuracy decrease. Greater compression may be possible by adapting sparsity type and rate to the training phase.

  • Four orders of magnitude is the reported maximum compression gain, accompanied by only a slight decrease in accuracy.
  • Communication delay and gradient sparsity behave as two independent sparsity types with similar effects on convergence speed.
  • SBC requires no changes to the baseline models’ hyperparameters, supporting direct applicability across experiments.
  • Higher compression rates may be possible by adapting sparsity type and rate to the current training phase.Identifying heuristics and theoretical guidance for this adaptation remains future work.

A. Momentum Correction, Warm-up Training and Momentum Masking:

The paper discusses momentum correction, warm-up training, and momentum masking as modifications related to sparse gradient updates. Its approach uses momentum correction while omitting warm-up training because it does not improve final accuracy.

  • Momentum Correction, Warm-up Training and Momentum Masking: Momentum correction adds the momentum-corrected gradient to the residual and is used implicitly in the paper’s approach.
  • Momentum Correction, Warm-up Training and Momentum Masking: Warm-up training increases sparsity exponentially from 25% to 0.1% during the first epochs.
  • Momentum Correction, Warm-up Training and Momentum Masking: Warm-up training can speed early convergence but ultimately has no effect on final model accuracy.
  • Momentum Correction, Warm-up Training and Momentum Masking: Momentum masking sets momentum to zero for updated weights to prevent stale momentum from directing optimization incorrectly.
  • Golomb Position Decoding can omit tensor-shape information because server and clients already know all weight-tensor shapes.

C. Model Specification

The experiments use convolutional and recurrent models drawn from established implementations, including LeNet5-Caffe, ResNet32/50, WordLSTM, and CharLSTM. The supplied passages also state that tensor projections are uniquely defined in a metric subspace.

  • Model Specification: The experiments include LeNet5-Caffe, featuring convolutional and fully connected layers with pooling.
  • Model Specification: ResNet32 and ResNet50 use the official TensorFlow ResNet implementation and include skip-connections and batch-normalization.
  • Model Specification: WordLSTM uses the official TensorFlow PTB implementation with trainable embeddings, multilayer LSTM cells, and dropout.
  • Model Specification: CharLSTM adapts WordLSTM to a 98-symbol vocabulary by reducing the embedding size.
  • A projection onto the metric subspace S uniquely solves the minimization problem in S.

E. Additional Results

Additional experiments show that SBC can reduce communicated bits while preserving near-baseline performance across convolutional and recurrent models. They also show that the best balance between gradient and temporal sparsity changes across training phases.

  • ResNet32 on CIFAR: ×32300 less bits lets SBC (3) train ResNet32 on CIFAR to almost baseline accuracy, with 0.4% degradation.
  • ResNet32 on CIFAR: Figure 7 compares ResNet32’s Top-1 accuracy and error against epochs and transferred bits on CIFAR.
  • CharLSTM on Shakespeare: ×2572 less bits is achieved by SBC (1) while attaining the highest accuracy on CharLSTM trained on Shakespeare.
  • CharLSTM on Shakespeare: Figure 8 compares CharLSTM perplexity against epochs and transferred bits using log-log plots.
  • WordLSTM on PTB: Figure 9 evaluates WordLSTM validation error across gradient and temporal sparsity levels, with constant total sparsity along matrix diagonals.
  • WordLSTM on PTB: Different forms of sparsity perform best at different training stages, so no single sparsity setup is optimal throughout training.
Loading 1805.08768v1…