Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities

1 Introduction

• Scaling deep learning drives financial, environmental, and inclusion concerns, while current training strategies are described as becoming unsustainable.The introduction cites the cost of GPT-3 training and projected growth in training costs as examples.Source: §1 · p. 2
• Algorithmic-efficiency methods change training semantics to reduce resource use and can compound with software and hardware acceleration.The methods may alter data or sample order, model structure, or optimization.Source: §1 · p. 2
• Fragmented metrics, narrow or poorly characterized evaluations, and the absence of a taxonomy impede consistent comparisons and navigation of the speedup landscape.The paper links disparate metrics to inconsistent rankings and limited evaluations to incorrect or overly broad conclusions.Source: §1 · p. 2
• The paper presents evaluation practices and a practitioner’s guide that connect method choice and mitigation to evaluation quality and compute-platform bottlenecks.Its holistic framing emphasizes that actual speedup depends on interactions between the method and the compute platform; the stated scope is training algorithms for language and computer vision, not inference, hardware, or other task-specific areas.Source: §1 · pp. 2–3
• The paper formalizes algorithmic speedup and classifies over 200 papers using five speedup actions across three training-pipeline components.The taxonomy is intended to help practitioners select methods, readers digest the literature, and researchers identify opportunities.Source: §1 · p. 3

2.1 Overview of DNN Training Process

• A training iteration loads and preprocesses a batch, transfers it to an accelerator for forward and backward passes, then updates model parameters and optimizer state.Iterations continue until the desired model quality or a predefined iteration count is reached.Source: §2.1 · pp. 4–5

2.2 Metrics to Quantify Training Efficiency

• Training time directly reflects practical cost but depends on hardware and software, while FLOPs and parameter counts enable hardware-independent comparisons yet omit important runtime factors.Electricity and carbon metrics also have hardware, location, time, or lifecycle dependencies; no single metric captures all relevant costs.Source: §2.2 · p. 6

2.3 Pitfalls of Different Metrics

• Input-output operand size predicts runtime more consistently than FLOPs across operations and ResNet-50 variants, but remains imperfect; the pattern also holds across hardware and architectures.The results point to memory bandwidth as a frequent bottleneck, while factorization and batch-normalization removal affect data movement differently.Source: §2.3 · pp. 7–9

2.4 Algorithmic Speedup: Motivation and Definition

• Algorithmic speedup changes training semantics to reduce the time required to reach a target quality, through fewer iterations, shorter iterations, or both.A faster recipe at equal or better quality is strictly better; when time and quality conflict, the comparison becomes a tradeoff.Source: §2.4 · p. 10
• The taxonomy organizes speedup methods through components, actions, and mechanisms, with methods represented as paths connecting elements across these building blocks.New approaches can come from changing the selected elements or their connections.Source: §2.4 · p. 11

3 A Unifying Perspective for Speedup Methods: Taxonomy

• The taxonomy classifies speedup methods along three building blocks: training-pipeline components, actions, and mechanisms.Components identify where changes occur, actions identify what changes, and mechanisms describe when and how they are made.Source: §3 · p. 11, §3.1 · p. 11

3.1 Component Categorization

• The function component includes architecture and model parameters, while training data and optimization are other pipeline components.Derived data arises during forward and backward passes; optimization includes the objective and training algorithm.Source: §3.1 · p. 11, §3.1.1 · p. 12, §3.1.2 · p. 12

3.2.1 Remove

• Remove is subtractive, typically aiming to reduce per-iteration work by removing component elements.Representative remove methods are illustrated across subcomponents.Source: §3.2.1 · p. 13

3.2.2 Restrict

• Restrict shrinks the range of possible component values, usually reducing per-iteration work and sometimes the number of iterations.Representative restrict methods are illustrated across subcomponents.Source: §3.2.2 · p. 13

3.2.3 Reorder

• Reorder changes when or where component elements are introduced or used, potentially reducing per-iteration work or the number of iterations.Progressively increasing problem complexity and reordering samples are examples.Source: §3.2.3 · p. 13

3.2.4 Replace

• Replace substitutes a component element or subcomponent for another, potentially reducing both per-iteration work and the number of iterations.Representative replace methods are illustrated across subcomponents.Source: §3.2.4 · p. 14

3.2.5 Retrofit

• Retrofit adds to components and typically seeks fewer training iterations rather than less work per iteration.It is described as the opposite of remove.Source: §3.2.5 · p. 14

3.3 Mechanism Categorization

• Mechanisms distinguish static changes made at initialization from dynamic changes during training, and domain-knowledge-based from learning-based decisions.The taxonomy groups methods to help reason about potential effectiveness for a bottleneck and identify themes and opportunities, despite not being exhaustive or mutually exclusive.Source: §3.3 · p. 15, §3.3.1 · p. 15, §3.3.2 · p. 15

4 Categorization of Existing Work: Survey

• The section reviews representative literature on algorithmic speedup methods.Source: §4 · p. 16

4.1.1 Model Parameters

• Structured pruning removes parameter blocks rather than individual weights, and PruneTrain reports speedups without significant accuracy loss.PruneTrain uses group lasso to encourage sparsity during training, then removes filters with the smallest maximum absolute parameter values.Source: §4.1.1 · p. 16
• 1.37× faster ImageNet training with little impact on final test accuracy is reported for models using sample-based tensor-operation approximations.The method dynamically prunes rows and columns in matrix multiplications and convolutions.Source: §4.1.1 · p. 17
• Pixelated Butterfly combines block-sparse and low-rank weight matrices and yields high accuracy with speedups on Transformer and MLP-Mixer models.The paper reports that the combination is more expressive than either sparse-only or low-rank-only matrices.Source: §4.1.1 · p. 18
• Parameter-restriction methods for fast training are mostly static, while dynamically finding layer ranks adds cost and has not been shown to speed training in the reviewed examples.The review notes a potential exception for pipelines involving retraining.Source: §4.1.1 · p. 18
• Switch Transformers have up to a trillion parameters and are reported to train seven times faster than a T5 transformer.Their routing mechanism conditionally uses subsets of the larger parameter set, and the paper attributes lower training times to fewer iterations for a given model quality.Source: §4.1.1 · pp. 19–20

4.1.2 Architecture

• Architecture-removal methods often require domain knowledge and dynamic compensation, while layerwise training can reduce time but sometimes yields worse quality than end-to-end optimization.Removing batch normalization, for example, is paired with regularization and stability measures.Source: §4.1.2 · pp. 20–22
• 40–85% faster ViT ImageNet training is reported for AutoProg, which progressively grows a ViT from subnetworks without accuracy drop relative to the large-ViT baseline.Training subnetworks lowers per-iteration work while keeping the number of iterations constant.Source: §4.1.2 · p. 22
• FNet replaces transformer attention with a 2D DFT and has lower training times than BERT without heavy accuracy loss; at larger sizes, BERTs and FNet hybrids are preferable.FNet is on the speed-accuracy Pareto frontier at small model sizes; the review also describes EfficientNetV2's training-aware architecture search and faster-training design choices.Source: §4.1.2 · pp. 22–23

4.2.1 Training Data

• Dataset pruning selects smaller training subsets, but large-scale pruning remains difficult, especially on ImageNet.Methods use static or dynamic selection; learning-based approaches often perform best, while cheaper methods such as random pruning can be effective when applied dynamically.Source: §4.2.1 · pp. 24–25
• Restricting training data can reduce iterations by controlling its statistical properties, but these approaches are mostly static and practical use is limited by lack of control over data generation.Theoretical work links dataset properties such as spectral structure and cover complexity to generalization or error.Source: §4.2.1 · pp. 25–26
• Curricula and progressive resizing reorder training data to improve learning efficiency, with curriculum speed benefits generally modest but more notable under limited iteration budgets or noisy labels.Easy-then-hard ordering helps in those settings; progressive resizing lowers per-step computation by starting with smaller images while maintaining or improving accuracy.Source: §4.2.1 · p. 26
• Replacing training data with compressed representations or distilled samples can reduce step time, but preserving the quality attainable with the original data remains a major challenge.Dubois et al. (2021) report 98.7% accuracy on STL using directly learned encodings in under five minutes; distilled data can improve performance per step versus non-distilled data of equal size, yet may not reach the quality of large non-distilled datasets.Source: §4.2.1 · p. 27
• Retrofitting training data through augmentation, repeated augmentations, or pretraining can improve accuracy or training efficiency, while learning augmentation policies is computationally demanding and hard to transfer.For language-model training with transformers, scaling dataset and model size roughly equally yields the best quality for a fixed compute budget.Source: §4.2.1 · pp. 27–28

4.2.2 Derived Data

• Removing derived data can lower per-step cost by pruning activations or gradients; Selective-Backprop achieves a 3.5x speedup over standard SGD at the cost of lower accuracy.Selective-Backprop (Jiang et al., 2019) omits backward-pass computation for selected samples, while sparse attention methods target the quadratic sequence-length cost and are most useful for long sequences.Source: §4.2.2 · p. 29
• Restricting derived data through normalization, quantization, or low-rank approximations can reduce step time or iterations, but approximation overhead and instability can erase gains.Gradient compression is useful when communication is the bottleneck; Batch Normalization’s standardization cost is one example of potentially large overhead.Source: §4.2.2 · pp. 29–31
• Derived-data reordering has few known speedup methods, which use static, domain-knowledge-based mechanisms; replacing derived data mainly targets step time but must preserve model quality.The scarcity of reordering approaches suggests room for further work.Source: §4.2.2 · p. 31
• Kernel-based attention methods, including Linear Transformer (Katharopoulos et al., 2020), Performer (Choromanski et al., 2020), and Scatterbrain (Chen et al., 2021b), yield linear sequence-length complexity by reordering kernelized attention.They replace or approximate softmax attention with φ(Q)(φ(K)^T V).Source: §4.2.2 · p. 31
• Attention-matrix changes are framed as dynamic modifications of derived activations despite being static architectural replacements, suggesting efficient transformer approaches could draw on other derived-data ideas.The review emphasizes their effect on derived activations when classifying them as dynamic modifications.Source: §4.2.2 · pp. 31–32
• Annealed Gaussian gradient noise reduces iterations to a target quality (Neelakantan et al., 2015), while ALiBi supports competitive longer-sequence evaluation after shorter-sequence training (Press et al., 2021).The noise variance decays over time; ALiBi penalizes attention activations to encourage different length scales across heads.Source: §4.2.2 · p. 32
• The review finds retrofit derived-data methods mostly use dynamic, domain-knowledge mechanisms and expects learning-based mechanisms could yield more accurate methods.Source: §4.2.2 · p. 32

4.3 Optimization Speedup Strategies

• Most optimization-related methods seek to reduce niteration, though added per-step computation can be useful when communication or data loading leaves accelerators underused.The review frames the methods around optimization objectives and algorithms, with a trade-off between fewer iterations and longer steps.Source: §4.3 · p. 32

4.3.1 Training objective

• Removing objective computation approximates expensive losses, but preserving original model quality is a key challenge; hierarchical softmax and negative sampling target large-vocabulary output costs.Hierarchical softmax computes word probabilities with work logarithmic in the number of classes; negative sampling excludes most word probabilities from the objective.Source: §4.3.1 · p. 32
• Reordering image resolution, dropout, and augmentation during training can lower niteration without accuracy loss, but the reviewed knowledge is somewhat task-specific.The review identifies generalizing such reorder methods beyond the tasks considered as a possible opportunity.Source: §4.3.1 · p. 33
• Replacement objectives seek greater model quality per step; reviewed methods are mostly static, while data- or training-state-dependent losses remain a proposed opportunity.The review describes replacement objectives as a common speedup technique and suggests dynamic mechanisms for further improvements.Source: §4.3.1 · p. 34
• Objective retrofits rely heavily on domain knowledge; SAM improves quality per step but adds gradient-computation time, and restrict-objective work remains early-stage.The review suggests deeper understanding of learning principles could improve retrofitted objectives and of loss landscapes could improve restricted objectives.Source: §4.3.1 · pp. 33, 35

4.3.2 Algorithm

• Algorithm restrictions use domain knowledge to keep training in efficient regimes; adaptive batch size or momentum and schedule adjustments are possible speedup directions.For shortened cosine schedules, at least 80% of the schedule must be completed during reduced training time to avoid accuracy loss, and best performance occurs when the schedule ends with training.Source: §4.3.2 · pp. 36–37
• Replacing algorithms can reduce niteration, but learned optimizers require task-specific training with unclear cross-task generalization, while removing algorithm steps often degrades accuracy at scale.The review notes significant accuracy degradation for removal methods, especially on moderate- to large-scale data such as ImageNet.Source: §4.3.2 · pp. 36–37
• Retrofitting algorithms can improve performance per step, but added computation must not outweigh iteration savings; the review suggests learned mechanisms could help manage that overhead.The reviewed retrofit methods use dynamic, domain-knowledge-based mechanisms.Source: §4.3.2 · p. 38

4.4 Other Directions

• Compute-platform approaches include faster implementations or kernels, better hardware, and parallelism.These approaches are distinguished from algorithmic speedup methods.Source: §4.4 · p. 38
• FFCV speeds preprocessing with JIT-compiled augmentations and data-pipeline optimizations, while NoPFS targets distributed-training data pipelines.The passage presents both as approaches that can accelerate training without changing the hardware.Source: §4.4 · p. 38
• Large-scale parallel-training trade-offs depend on accelerator count, interconnect, memory, batch and input sizes, and model structure.The review notes multiple approaches to large-scale parallel training.Source: §4.4 · p. 38

• Define the intended problem scope and report the experimental setup, hyperparameters, and relevant variation.Hold hyperparameters constant as far as feasible; sweep them when claiming robustness.Source: §5 · p. 40
• Compare methods under identical hardware and software where feasible, and disclose comparison sources and baseline uncertainties.Training time and accuracy can vary with hardware, libraries, and software support.Source: §5 · pp. 40, 42
• Show sufficiently granular accuracy-efficiency tradeoff curves beside competing methods and prioritize high-accuracy operating points.The guidance recommends at least three operating points per dataset-architecture pair.Source: §5 · pp. 40–41
• Adjusting the learning-rate schedule lets shorter training outperform nearly all tested speedup methods; simply aborting a longer run is less effective.The results support including reduced epoch count as a baseline alongside early stopping.Source: §5 · p. 41, Figure 13 · p. 41
• Include reduced epoch count and smaller models as strong baselines in speed-accuracy tradeoff curves.ResNet-18 or ResNet-34 can outperform some speedup methods under certain hyperparameters.Source: §5 · pp. 41–42
• Composing methods can greatly improve the time-accuracy curve, and methods’ effects can differ with training duration.The compared regularization methods harm accuracy at short durations but help at long durations.Source: §5 · p. 42
• Use multiple runs with separate initializations and seeds, reporting central tendency and variation when feasible.Clearly defined error bars are recommended when computationally feasible.Source: §5 · p. 42

• Identifying hardware bottlenecks matters because accelerating a non-bottleneck operation provides no benefit.The section uses data loading as a motivating example of a bottleneck that can distort speed comparisons.Source: §6 · pp. 42–43
• Different combinations of speedup methods work best at different training budgets, so no combination is universally best.The figure contrasts method compositions across the time-accuracy curves.Source: Figure 14 · p. 43
• A data-loader bottleneck can make ResNet-18 appear little faster than ResNet-34, obscuring the effect of model size.With faster data loading, ResNet-18 could be much faster.Source: §6 · pp. 42–43

6.1.1 GPU compute

• Compute is usually a bottleneck mainly for large matrix multiplications or convolutions; reducing tensor sizes or using structured sparsity can mitigate it.The overview distinguishes operations on data from moving data between devices or cache levels.Source: §6.1.1 · pp. 43–44

6.1.2 GPU memory

• GPU memory use is often dominated by optimizer state at small batch-times-input sizes and by activations at larger sizes.Mitigations include sharding or offloading state, smaller-state optimizers, activation checkpointing, sharding, or compression.Source: §6.1.2 · pp. 44–45

6.1.3 GPU memory bandwidth

• GPU memory bandwidth can be reduced by compressing tensors or improved in use by increasing compute per byte.Increasing a matrix multiplication’s contraction dimension can raise arithmetic intensity more than increasing its row or column counts.Source: §6.1.3 · p. 45

6.1.4 CPU compute

• CPU preprocessing and augmentation can bottleneck training, especially in vision; simplification, optimized loaders, or accelerator offloading can help.The discussion also describes hiding this bottleneck with more GPU compute per loaded batch.Source: §6.1.4 · p. 45

6.1.5 CPU memory

• CPU memory can constrain dataset caching, model initialization, or tensor offloading; fast storage and fewer or smaller offloaded tensors can help.Careful tool configuration can also avoid unnecessary copies of a model in memory.Source: §6.1.5 · p. 46

6.1.6 CPU memory bandwidth

• CPU memory bandwidth is not typically a training bottleneck on accelerators; when training on CPUs, GPU-bandwidth mitigations apply.Source: §6.1.6 · p. 46

6.1.7 Inter-GPU interconnect

• Inter-GPU gradient synchronization can be reduced through compression or less frequent synchronization, with possible accuracy tradeoffs.Larger batches or input sizes and different parallelism schemes are additional options, each with tradeoffs.Source: §6.1.7 · p. 46

6.1.8 CPU-GPU interconnect

• Repeated CPU-GPU transfers can bottleneck training; avoid unnecessary movement and offload less data when possible.Offloading only optimizer states is given as an example.Source: §6.1.8 · pp. 46–47

• Peak throughput across computational resources differs by many orders of magnitude, and workload-resource imbalance can leave some resources underutilized while others bottleneck.The paper provides representative hardware peak-throughput values in Table 4.Source: §6.2 · p. 48, Table 4 · p. 48
• A single A100 can perform 249600 multiply-adds for every 16-bit value transmitted over a 10 gigabit network connection.Source: §6.2 · p. 48, Table 4 · p. 48
• These throughput disparities make hardware understanding important when evaluating and applying algorithmic speedup methods.Source: §6.2 · p. 48

7 Summary and Path Forward

• The paper combines a formalized speedup problem and taxonomy with a survey of over 200 approaches, evaluation guidance, and practical application guides.The authors say these connections help organize literature and identify trends and future opportunities.Source: §7 · p. 49
• Speedup methods often target computer vision; transferring them to LLM training, generative modeling, reinforcement learning, and other problems could benefit the community.Source: §7 · p. 49
• Many training subcomponent-action pairs remain insufficiently explored, while methods targeting more than one or two subcomponents could be useful.Source: §7 · p. 49
• New ranking metrics, potentially drawing on multi-objective optimization, could support more comprehensive comparisons of speedup methods.Source: §7 · p. 49
• A comprehensive Model Zoo with publicly available pretrained models following FAIR principles could avoid duplicated work in many cases.Source: §7 · p. 49
• Algorithm-hardware co-design may let hardware-unfriendly methods such as unstructured sparsity attain the training-efficiency improvements they suggest.Source: §7 · p. 49
• A comprehensive, fair, and reliable benchmarking leaderboard is identified as a crucial building block for speedup research.The paper cites DawnBench and MLCommons Algorithmic Efficiency as related efforts.Source: §7 · p. 49
• Cross-pollination among deep learning, blackbox and multi-objective optimization, approximate computing, and hardware design is described as critical to developing the speedup-method toolbox.Source: §7 · p. 49

Appendix A. Motivation for Compute-Efficient Deep Learning

• Since 2010, ML training compute growth has outpaced Moore’s Law, with large-model-era requirements increasing 10 to 100-fold and current large models using 600,000 times AlexNet’s computing power.Source: Appendix A · p. 50
• Compute growth makes training more time-consuming and expensive; GPT-3 training was estimated at about $1.65 million at advertised maximum Google TPU v3 performance.The passage also reports an estimate of around $35 million to replicate AlphaGo Zero experiments.Source: Appendix A · p. 50
• Training energy use can contribute to carbon emissions; one cited estimate puts a transformer model trained with neural architecture search at 626,155 lbs of CO2.Source: Appendix A · p. 51
• Compute-intensive training is not yet feasible at the edge, limiting applications that depend on resource-constrained devices and locally adaptable models.Source: Appendix A · p. 51
• The growing computational intensity of deep learning favors resource-rich organizations, and the cited analysis links access to computing power with a widening compute divide.Source: Appendix A · p. 51

Appendix B. Experimental Details

• The experiments used one machine with eight A100s and two 32-core AMD EPYC 7513 processors; microbenchmarks used one A100 and five trials of ten executions after four warmups.Microbenchmarks timed only the forward pass and used half-precision weights and activations.Source: Appendix B · p. 52
• Figure 16 maps the speedup methods used for the different Figure 14 curves, while the selected recipe-based hyperparameters may not be optimal.The authors caution that the comparisons support only that certain baselines can outperform these methods, not that they always will.Source: Appendix B · p. 52, Figure 16 · p. 52
• Mosaic ResNet results came from the authors’ public data.Source: Appendix B · p. 52

Appendix C. Additional Experiments

• On CPU, FLOP count can be a poor proxy for wall time, while memory-bandwidth consumption can be more helpful but remains imperfect.The experiments cover ConvNeXt-B and Swin-B on CPU, complementing the paper’s GPU results.Source: Appendix C · p. 52, Figure 17 · p. 53, Figure 18 · p. 53