PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Greedy Learning of CNNs

Eugene Belilovsky, Michael Eickenberg, Edouard Oyallon

arXiv:1901.08164v4cs.LGstat.ML

TL;DR

Back-propagation’s sequential gradient processing limits parallel training, and auxiliary gradient-prediction approaches can scale poorly for CNNs. This paper optimizes a greedy learning objective to decouple module training and extends it with replay buffers for asynchronous settings. The authors report convergence, better generalization than sequential greedy optimization, and favorable comparisons with alternative approaches across CIFAR-10 and ImageNet.

  • Problem

    Back-propagation’s sequential gradient processing limits parallelization, while auxiliary gradient-prediction methods can scale poorly in computation or accuracy, especially for CNNs.

    Source: §1 · p. 1

  • Method

    DGL optimizes a greedy relaxation of the joint objective with decoupled module updates, and replay buffers extend it to asynchronous operation.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Results

    The authors report that DGL matches or exceeds state-of-the-art approaches and scales to larger datasets, with better generalization than sequential greedy optimization.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    The authors present DGL as a strong baseline for parallelizing per-layer and per-module computation in CNN training.

    Source: §6 · p. 8

    The convergence analysis assumes finite variance.

    Source: §3 · p. 4

Abstract

from arXiv · show

A commonly cited inefficiency of neural network training by back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full network before updating. Several alternatives that can alleviate this issue have been proposed. In this context, we consider a simpler, but more effective, substitute that uses minimal feedback, which we call Decoupled Greedy Learning (DGL). It is based on a greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification. We consider an optimization of this objective that permits us to decouple the layer training, allowing for layers or modules in networks to be trained with a potentially linear parallelization in layers. With the use of a replay buffer we show this approach can be extended to asynchronous settings, where modules can operate with possibly large communication delays. We show theoretically and empirically that this approach converges. Then, we empirically find that it can lead to better generalization than sequential greedy optimization. We demonstrate the effectiveness of DGL against alternative approaches on the CIFAR-10 dataset and on the large-scale ImageNet dataset.

1. Introduction

Back-propagation’s sequential gradient processing limits parallelization, while DNI’s auxiliary gradient prediction can scale poorly in accuracy and computation. The paper proposes optimizing a greedy learning objective to decouple module updates and, with replay buffers, support asynchronous operation.

1. Introduction

  • Sequential gradient processing in back-propagation prevents parallelization of constituent-module computations.The paper identifies forward, update, and backward locking as distinct forms of this constraint.Source: §1 · p. 1
  • DNI uses auxiliary networks to predict backward-pass gradients, but this approach scales poorly in computation and accuracy, especially for CNNs.The predicted weight gradients can be high-dimensional for larger models and inputs.Source: §1 · p. 1
  • DGL optimizes a greedy learning objective to decouple module computations, achieve update unlocking, and use replay buffers to address forward unlocking.The paper presents this as a simpler alternative to feedback-dependent approaches.Source: §1 · p. 1
  • The paper extends DGL with replay buffers for asynchronous training and reports convergence guarantees and empirical scalability, stability, and robustness to delays.The reported experiments include CIFAR-10 and ImageNet.Source: §1 · pp. 1–2

2. Parallel Decoupled Greedy Learning

Parallel Decoupled Greedy Learning trains modules using local greedy objectives rather than requiring updates to wait for signals from later modules. A replay-buffer extension reduces dependence on preceding modules, while auxiliary-network design is considered in light of parallel training speed.

2.1. Preliminaries

  • DNI uses auxiliary networks to predict gradients from subsequent modules, whereas the adapted greedy objective avoids that upper-module feedback and permits update unlocking.DNI’s auxiliary models are trained against true gradients when they become available, requiring cached activations.Source: §2.1 · p. 2

2.2. Optimization for Greedy Objective

  • Sync-DGL performs module updates in parallel using local losses, allowing a subsequent layer to begin computing before the preceding layer’s update finishes.The greedy objective uses per-module empirical risk with cross-entropy loss and auxiliary networks that predict the final target.Source: §2.2 · pp. 2–3
  • The greedy objective can be optimized with Sync-DGL instead of sequential layerwise solving, although each subproblem’s input distribution changes over time.The authors solve subproblems by back-propagation but note that other iterative solvers can also be used.Source: §2.2 · p. 3

2.3. Asynchronous DGL with Replay

  • Asynchronous DGL shares replay buffers between adjacent modules so they can reuse older samples and operate despite communication delays or speed differences.The algorithm simulates delays by selecting workers according to a probability distribution; experiments vary worker-selection distributions and buffer size.Source: §2.3 · p. 3
  • Asynchronous DGL requires only local communication, no master node, and each node maintains and updates only its own module parameters.The authors contrast this arrangement with common data-parallel asynchronous algorithms and decentralized SGD.Source: §2.3 · p. 3

2.4. Auxiliary and Primary Network Design

  • For parallel training, auxiliary networks must be designed with their speed relative to the primary module in mind; the paper aims to keep them below 5% of primary-network FLOPs.The authors also note that auxiliary-network design may require different architecture principles but is not inherently prohibitive.Source: §2.4 · p. 3
  • The authors describe greedy objectives as applicable beyond distributed learning and note that single-worker DGL reduces memory.They cite recent uses of greedy objectives in other settings.Source: §2.4 · pp. 3–4

3. Theoretical Analysis

The analysis establishes convergence for DGL and provides a convergence rate for expected gradient norms. It handles module inputs whose distributions vary with, and depend on, the convergence of preceding modules.

3. Theoretical Analysis

  • Module input distributions vary over time and depend on previous-module convergence; the analysis assumes standard conditions and convergence of the previous layer.The previous-layer convergence assumption is specific to the time-varying input setting.Source: §3 · p. 4
  • Under Assumptions 1, 2, 3 and 4, each term of the following equation converges; the DGL scheme also converges in the sense of Bottou et al. (2018) and Huo et al. (2018a).Source: §3 · p. 4
  • The analysis gives a rate for the sequence of expected gradient norms accumulating around zero.The rate result follows the convergence statement.Source: §3 · p. 4

4. Experiments

The experiments compare DGL with alternative training methods on CIFAR-10 and ImageNet, and test its memory use and replay-buffer asynchrony.

4.1. Other Approaches and Auxiliary Network Designs

  • On CIFAR-10, DGL outperforms DNI and cDNI in test accuracy and training loss, and can generalize better than backpropagation.With more common CNN optimization settings, DNI diverged at larger learning rates while DGL sub-problem optimization remained effective.Source: §4.1 · p. 5, Figure 2 · p. 5
  • After spatial averaging, MLP-aux and MLP-SR-aux reduce auxiliary-network computation to less than 5% of the primary network’s FLOPs, with minimal accuracy loss.MLP-aux also drastically reduces FLOPs compared with CNN-aux with minimal accuracy loss.Source: §4.1 · pp. 5–6, Table 1 · p. 5
  • Parallel DGL quickly catches up with sequential greedy training and sometimes generalizes better; after a few epochs, layers show progressive improvement with depth.The comparison trains layers for 50 epochs in both schemes and examines learning curves at layer 4.Source: §4.1 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

4.2. Large-scale Experiments

  • On ImageNet, DGL performs as well as or sometimes better than end-to-end-trained models across the tested architectures, while permitting parallel training.The experiments include VGG-family models and ResNet architectures.Source: §4.2 · p. 7
  • ResNet-152 DGL with K=2 fits 38% more samples on one 16GB GPU than standard end-to-end training.The paper also reports favorable synchronous-DGL memory use relative to DDG and DNI.Source: §4.2 · p. 7

4.3. Asynchronous DGL with Replay

  • With a slowed layer at S=2.0, that layer reaches only epoch 25 while downstream layers reach epoch 50; accuracy is maintained until approximately 1.2× slowdown.The experiment limits each layer to 50 epochs, giving layers identical numbers of gradient updates.Source: §4.3 · p. 7
  • At S=1.2× slowdown, even a tiny replay buffer yields only a slight loss in accuracy.The experiments examine performance across buffer sizes.Source: §4.3 · p. 7

5. Related work

The section contrasts update unlocking with alternatives that address backward locking or preserve dependencies between layers. It also distinguishes model-parallel learning from data-parallel and systems-level pipelining approaches.

5. Related work

  • Ideally, backward unlocking scales as O(LTF), while update unlocking could scale as O(TF) with L parallel nodes.The comparison assumes equal forward-processing time TF across L layers and a backward pass that takes a constant multiple of forward time.Source: §5 · p. 7
  • With a single layer slowed, Async DGL maintains accuracy until approximately 1.2× slowdown; beyond that, the accuracy difference remains small.The figure compares against the synchronous case.Source: Figure 5 · p. 8
  • For Async DGL, Figure 6 compares buffer size with accuracy and reports that smaller buffers produce only a small loss in accuracy.Source: Figure 6 · p. 8
  • Alternative feedback methods address backward unlocking but not update or forward locking; their ImageNet result was 17.5% top-5 accuracy versus 59.8% for the reference model.The passage attributes the 17.5% result to these approaches and cites Bartunov et al. (2018).Source: §5 · p. 8
  • DGL with VGG and ResNet obtains similar or better ImageNet accuracies than end-to-end backprop while enabling parallelization and reduced memory.The caption identifies Table 3 as the comparison using the Xiao et al. (2019) training schedule.Source: Table 3 · p. 8
  • ADMM-style methods optimize the joint objective and retain links between successive layers, so they do not achieve update or forward unlocking.Some allow simultaneous weight updates, but most require batch gradient descent and have not been demonstrated on large-scale datasets; none had been combined with CNNs.Source: §5 · p. 8
  • Distributed synchronous and asynchronous SGD are data-parallel approaches, while this work is closer to model parallelism and can be combined with data-parallel methods.Pipelining targets system latency but does not remove the algorithmic locking constraints of the learning objective and backpropagation.Source: §5 · p. 8

6. Conclusion

The paper presents per-layer and per-module computation as a simple, strong baseline for parallelizing CNN training. It reports matching or exceeding state-of-the-art approaches and scaling to larger datasets.

6. Conclusion

  • The work introduces a simple, strong baseline for parallelizing per-layer and per-module computations in CNN training.The conclusion says it matches or exceeds state-of-the-art approaches addressing these problems and scales to much larger datasets than others.Source: §6 · pp. 8–9
  • Future work can develop improved auxiliary problem objectives and combine them with delayed feedback.Source: §6 · p. 9

A. Proofs

The proofs establish a convergence proposition under Assumptions 1–4. The proof fragments invoke standard inequalities and a telescoping sum for the non-negative loss.

A. Proofs

  • The proof invokes Fubini’s theorem, Cauchy–Schwarz, Jensen’s inequality, Fatou’s lemma, and Assumptions 3 and 4 in its convergence steps.Source: A. Proofs · pp. 11–12
  • Under Assumptions 1, 2, 3, and 4, Proposition 4.2 states that each term in the displayed equation converges.Source: A. Proofs · p. 12
  • The proof applies Lemma 3.2 across iterations and uses a telescoping sum for the non-negative loss.Source: A. Proofs · p. 13

B. Additional Descriptions of Experiments

The appendix gives experimental implementation details, including a DNI replication caveat and training schedules for auxiliary-network and ImageNet experiments.

Comparisons to DNI

  • The reproduced DNI baseline is worse than the original report, possibly because of minor implementation differences.Backpropagation and cDNI baseline accuracies were close to those reported in Jaderberg et al. (2017); the authors used a popular PyTorch DNI implementation.Source: B. Additional Descriptions of… · p. 13

Auxiliary network study

Imagenet

  • ImageNet training uses 50 epochs, batch size 256, SGD with momentum 0.9, weight decay 10−4, and learning rate 0.1 reduced tenfold every 10 epochs.The schedule follows Xiao et al. (2019).Source: B. Additional Descriptions of… · p. 13

C. Detailed Discussion of Relative Speed of Competing Methods

The paper compares methods’ speed by their computation and communication overheads, concluding that DGL is faster by construction under matched training iterations and network splits. It also notes that multi-GPU communication must be well optimized and that optimized distributed implementations are outside the work’s scope.

C. Detailed Discussion of Relative Speed of Competing Methods

  • Relative speed depends on auxiliary computation, activation and feedback communication, and waiting for signals to reach the final module.Activation-transfer time should be equal across methods under the same implementation and hardware; final-module waiting applies to DDG/FA, but not DNI or DGL.Source: C. Detailed Discussion of Relative… · p. 13
  • Under the same number of epochs or iterations and the same network splits, DGL is faster by construction than methods that rely on feedback.DGL has negligible auxiliary-network overhead and no feedback communication; DDG/FA also wait for the signal to reach the final module.Source: C. Detailed Discussion of Relative… · p. 13
  • Multi-GPU use requires well-optimized communication, while optimized distributed implementations are outside this work’s scope.The paper includes a parallel implementation based on software from Huo et al. (2018b).Source: C. Detailed Discussion of Relative… · pp. 13–14
  • Figure 7 illustrates signal propagation for three mini-batches, assuming equal forward and backward speeds and negligible auxiliary-network computation.Each module begins forward and backward processing as soon as possible.Source: Figure 7 · p. 14

C.1. Auxiliary Network Sizes and FLOP comparisons on ImageNet

For the ImageNet VGG-13 example, the paper describes two auxiliary-network designs and characterizes their size as negligible.

C.1. Auxiliary Network Sizes and FLOP comparisons on ImageNet

  • For VGG-13, the MLP auxiliary network pools the first-layer output to 2 × 2 × 64, uses three fully connected layers of size 256, and projects to 1000 image categories.The first-layer output is 224 × 224 × 64.Source: §C.1 · p. 14
  • The MLP-SR auxiliary network reduces to 56 × 56 × 64, applies three 1 × 1 convolutions of width 64, then reduces to 2 × 2 and uses the MLP design’s fully connected layers.Source: §C.1 · p. 14
  • The paper describes both auxiliary networks as negligible in size for the ImageNet experiments.Source: §C.1 · p. 14

D. Additional pseudo-code

The additional pseudocode makes worker behavior explicit for parallel DGL and asynchronous buffered DGL implementations.

D. Additional pseudo-code

  • Algorithm 3 presents a parallel implementation equivalent in output to Algorithm 1, with worker behavior specified explicitly.Its initialization includes parameters {θj, γj}j≤J.Source: D. Additional pseudo-code · pp. 14–15
  • Algorithm 4 presents a parallel implementation of Algorithm 2 using buffers and parameters initialized for each worker.The delay distribution is an input, and the buffer size is M.Source: D. Additional pseudo-code · pp. 14–15
  • Algorithm 4’s probabilities are derived from communication and computation speed differences, but are not included in the pseudocode.Source: D. Additional pseudo-code · p. 14

Related papers

Loading 1901.08164v4…