PaperlayerView exact revision on arXiv

Source-linked AI summary

Searching for Fast Model Families on Datacenter Accelerators

Sheng Li, Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc Le, Norman P. Jouppi

arXiv:2102.05610v1cs.CVeess.IV

TL;DR

Existing NAS and model scaling do not sufficiently account for datacenter accelerator architecture, despite the growing performance gap on these devices. The paper combines accelerator-tailored architecture search with latency-aware compound scaling, producing EfficientNet-X, which is faster than existing model families with comparable accuracy in the reported comparisons.

  • Problem

    Existing NAS and model scaling lack sufficient visibility into datacenter accelerator architecture, while the root cause of FLOPs-latency noncorrelation on these accelerators had not been analyzed in depth.

    Source: §2 · p. 2

  • Method

    The paper analyzes accelerator bottlenecks, adds accelerator-friendly operations to the NAS search space, and uses LACS to scale models against accuracy and latency objectives.

    Source: §1 · pp. 1–2

  • Results

    EfficientNet-X is up to more than 2X faster than EfficientNet with comparable accuracy on TPUv3 and GPUv100, and up to 7X faster than RegNet on TPUv3.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method produces accelerator-specific model families designed for better accuracy and speed through optimized FLOPs, operational intensity, and efficiency.

    Source: §8 · p. 9

    The fused-convolution analysis assumes equal input and output depth and spatial dimensions, with stride-1 square kernels.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Neural Architecture Search (NAS), together with model scaling, has shown remarkable progress in designing high accuracy and fast convolutional architecture families. However, as neither NAS nor model scaling considers sufficient hardware architecture details, they do not take full advantage of the emerging datacenter (DC) accelerators. In this paper, we search for fast and accurate CNN model families for efficient inference on DC accelerators. We first analyze DC accelerators and find that existing CNNs suffer from insufficient operational intensity, parallelism, and execution efficiency. These insights let us create a DC-accelerator-optimized search space, with space-to-depth, space-to-batch, hybrid fused convolution structures with vanilla and depthwise convolutions, and block-wise activation functions. On top of our DC accelerator optimized neural architecture search space, we further propose a latency-aware compound scaling (LACS), the first multi-objective compound scaling method optimizing both accuracy and latency. Our LACS discovers that network depth should grow much faster than image size and network width, which is quite different from previous compound scaling results. With the new search space and LACS, our search and scaling on datacenter accelerators results in a new model series named EfficientNet-X. EfficientNet-X is up to more than 2X faster than EfficientNet (a model series with state-of-the-art trade-off on FLOPs and accuracy) on TPUv3 and GPUv100, with comparable accuracy. EfficientNet-X is also up to 7X faster than recent RegNet and ResNeSt on TPUv3 and GPUv100.

1. Introduction

The paper targets the gap between available and achieved performance on datacenter accelerators, arguing that existing NAS and model scaling lack sufficient hardware detail. It combines accelerator-focused architecture search with latency-aware scaling to build EfficientNet-X model families.

1. Introduction

  • NAS and model scaling leave a performance gap on specialized datacenter accelerators because they lack sufficient visibility into hardware architecture details.The paper notes the gap between achieved and available performance as accelerator and model architectures grow more complex.Source: §1 · p. 1
  • The accelerator analysis finds that state-of-the-art CNNs have low operational intensity and parallelism, contributing to low computation rate and suboptimal inference latency and throughput.The paper describes this analysis as revealing a root cause of FLOPs-latency nonproportionality.Source: §1 · pp. 1–2
  • The accelerator-optimized search space includes space-to-depth, space-to-batch, fused convolution structures, and block-wise searchable activations.These operations are intended to improve CNN operational intensity and efficiency for better accuracy-speed trade-offs.Source: §1 · p. 2
  • Latency-aware compound scaling (LACS) searches scaling factors using both accuracy and inference speed as objectives.The paper presents LACS as the first compound scaling method with both latency and accuracy in its multi-objective.Source: §1 · p. 2
  • EfficientNet-X is up to more than 2X faster than EfficientNet with comparable accuracy on TPUv3 and GPUv100, and 7X faster than RegNet on TPUv3.On GPUv100, it is on average 82% faster than RegNet and 48% faster than ResNeSt; on TPUv3, it is 48% faster than ResNeSt.Source: §1 · p. 2

2. Rethink model speed on DC accelerators: Why FLOPs and latency do not correlate

FLOPs alone do not explain accelerator inference latency: the paper’s performance model makes latency depend on computation, operational intensity, and execution efficiency. Its roofline analysis identifies low operational intensity and insufficient parallelism as key CNN bottlenecks on TPUs and GPUs.

2. Rethink model speed on DC accelerators: Why FLOPs and latency do not correlate

  • Latency depends on computation, operational intensity, and execution efficiency—not FLOPs alone—explaining FLOPs-latency nonproportionality.The model defines operational intensity as computation per memory traffic and execution efficiency as achieved computation rate divided by the ideal rate.Source: §2 · pp. 2–3
  • TPUs and GPUs need high operational intensity to reach compute-bound performance because their peak computation grows faster than memory bandwidth, placing their ridge points farther right than CPUs.The analysis reports roughly 35X higher TeraOps/sec for GPUv100 and TPUv3 than typical CPUs.Source: §2 · p. 3
  • EfficientNet’s operational intensity is an order of magnitude below the TPU/GPU ridge point, despite roughly 10X fewer FLOPs than comparable-accuracy models such as ResNet.The roofline analysis places models below the ridge point in the memory-bandwidth-bound region; TPU and GPU rooflines overlap because their peak computation and memory bandwidth are similar.Source: §2 · p. 3, Figure 2 · p. 3
  • Convolutions also need sufficient depth, batch, and spatial dimensions for matrix-unit parallelism, while parallelism between matrix and vector units matters for overall performance.Elementwise operations such as activations run on vector units rather than matrix units.Source: §2 · p. 3

3. Optimize search space for DC accelerators

The search space is tailored to accelerator bottlenecks by adding operations and structures intended to improve parallelism, operational intensity, and execution efficiency. It also makes activation functions searchable at the block level.

3. Optimize search space for DC accelerators

  • The accelerator-optimized search space adds space-to-depth/batch operations, fused convolution structures, and block-wise searchable activations.These components target operational intensity and parallelism while exposing architecture choices to NAS.Source: §3 · p. 3

3.1. Efficient space-to-depth and space-to-batch

  • Space-to-depth and space-to-batch increase depth and batch dimensions while keeping tensor volume unchanged, addressing insufficient parallelism on matrix units.Space-to-depth uses a stride-n convolution to reshape the tensor; this can improve operational intensity and efficiency while contributing to model capacity. Space-to-batch uses memory-intensive copy-reshape operations.Source: §3.1 · pp. 3–4

3.2. Fused convolution structures

  • Fused MBConv offers higher operational intensity but more FLOPs than MBConv, so either structure may be faster depending on activation and weight shapes.The search space adds fused MBConv because its speed and accuracy trade-offs are difficult to optimize manually; the analysis also notes depthwise convolutions have fewer FLOPs but lower operational intensity.Source: §3.2 · p. 4

3.3. Block-wise searchable activation functions

  • Activations are memory-bound and can slow end-to-end inference, so the search makes ReLU and swish selectable per block, with one activation shared across each block’s layers.Fusing activations with convolutions can avoid extra memory access; practical outcomes depend on the software stack.Source: §3.3 · p. 4

4. Latency-aware compound scaling (LACS)

Existing compound scaling searches for accuracy without accounting for the speed impact of the resulting family, while directly searching a whole family is costly. LACS searches depth, width, and resolution scaling using accuracy and latency together.

4. Latency-aware compound scaling (LACS)

  • Searching an entire model family directly is costly; the paper gives searching EfficientNet B0–B7 as approximately 100X more expensive than searching its B0 base model.Prior compound scaling uses accuracy as its sole objective and therefore does not consider the performance impact of the resulting family.Source: §4 · p. 5
  • LACS searches scaling factors for depth, width, and resolution with accuracy and latency as joint objectives for each accelerator.The scaling coefficients preserve the basic network architecture and are selected to target successive model-family latency budgets.Source: §4 · pp. 4–5
  • Unlike prior accuracy-only compound scaling, LACS treats φ as a latency budget and allocates it across depth, width, and resolution using α, β, and γ.The coefficients can be determined by grid search; the paper describes LACS as the first multi-objective compound scaling method.Source: §4 · p. 5

5. Search and scaling optimized model families on DC accelerator

The section searches for accelerator-specific base models with NAS, then uses LACS to scale them into model families on TPUs and GPUs. The resulting scaling favors depth growth, and direct search finds the same architectures as LACS at the same latency target with fewer accelerator resources.

5.1. NAS for base models

  • Using the original EfficientNet search space, latency- and FLOPs-optimized searches both converge to EfficientNet-B0 with ReLU, with no better model found.That space lacks TPU/GPU-oriented operations such as space-to-depth/batch, fused MBConv, and searchable activation functions.Source: §5.1 · p. 5
  • NAS on the accelerator-optimized search space produces EfficientNet-X-B0 with space-to-depth, hybrid MBConv, and accelerator-specific activation choices.TPU and GPU architectures differ mainly in activation choices; the GPU model uses ReLU throughout, while the TPU model mixes swish and ReLU.Source: §5.1 · p. 6
  • TPU NAS pairs ReLU with depthwise-convolution stages because both depthwise convolution and swish use vector units, creating contention.On GPUs, swish is not fused with convolutions in the software stack, while TPU XLA fuses it; the searched GPU model therefore uses ReLU throughout.Source: §5.1 · p. 6

5.2. Scaling to form model families with LACS

  • LACS finds that network depth should grow much faster than image resolution and width, with image resolution growing slowest.Faster depth growth can slow memory inflation from activation and intermediate tensors, making models more compute-bound.Source: §5.2 · p. 7
  • At the same latency target, LACS finds the same model architectures as direct multi-objective NAS while using far fewer accelerator resources.The comparison was run on TPUv3 and GPUv100.Source: §5.2 · p. 7

6. Experiments

EfficientNet-X delivers faster inference than competing model families at comparable accuracy on TPUv3 and GPUv100. The experiments attribute its speed to a balance between computation load and operational intensity, and report gains from both the search space and LACS.

6. Experiments

  • EfficientNet-X models are fastest within each comparable-accuracy group on TPUv3 and GPUv100, balancing FLOPs and operational intensity between competing model-family extremes.EfficientNet-X is reported as 82% and 48% faster than RegNet and ResNeSt on GPUv100, and 7X and 48% faster than RegNet and ResNeSt on TPUv3, respectively.Source: §6 · p. 7
  • EfficientNet-X achieves up to 2X+ speedup over EfficientNet on TPUv3 and GPUv100, with geometric-mean speedups of 56% on TPUs and 91% on GPUs, respectively.Source: §6 · p. 7
  • On migration from TPUv2 to TPUv3, EfficientNet-X achieves about 1.9X average speedup versus about 1.5X for EfficientNet, a roughly 30% better speedup.TPUv3 has 2X the peak performance of TPUv2.Source: Figure 4 · p. 7, §6 · p. 7
  • The new search space yields about 50% speedup on TPUv3 and GPUv100, and LACS adds 14% and 25% geometric-mean performance gains on TPUs and GPUs, respectively.The LACS gains are relative to the single-objective-scaled family built from EfficientNet-X-B0.Source: §6 · p. 7, Figure 3 · p. 7

7. Related work

Related work spans NAS methods, efficiency objectives, targeted model optimizations, model scaling, and datacenter accelerators. The paper distinguishes its approach from prior work that uses high-level efficiency signals without accelerator-architecture visibility.

7. Related work

  • Prior NAS methods include reinforcement learning, evolutionary search, differentiable search, and other approaches, with work also reducing search cost and improving inference efficiency.For fast inference, prior multi-objective NAS combines accuracy with performance or efficiency, but uses high-level signals such as model size and latency.Source: §7 · p. 8
  • Targeted efficiency work includes automated pruning and quantization as well as manual platform-specific optimizations, especially for mobile devices.The cited automated approaches include model pruning and quantization.Source: §7 · pp. 8–9
  • Model scaling progressed from separately or jointly changing depth and width to compound scaling across depth, width, and resolution.More recent work combines scaling with NAS to construct larger models.Source: §7 · p. 9
  • TPUs and GPUs provide computing power for machine-learning training and inference at scale.Source: §7 · p. 9

8. Conclusions

The conclusion presents a CNN-family search method that combines accelerator-tailored NAS with latency-aware compound scaling. EfficientNet-X reports faster speed at comparable accuracy on TPUv3 and GPUv100, with stronger speedup than EfficientNet when moving from TPUv2 to TPUv3.

8. Conclusions

  • The paper presents a datacenter-accelerator-targeted method for searching CNN model families for high accuracy and efficient inference.The method combines accelerator-tailored NAS and latency-aware compound scaling, and EfficientNet-X is reported up to 2X+ faster with comparable accuracy on TPUv3 and GPUv100.Source: §8 · p. 9

A. Ablation study on the DC accelerator optimized search space and the searched EfficientNet-X-B0 base model

The ablation finds that space-to-depth and fused convolutions improve accuracy and speed, while activation choices trade accuracy for speed differently across accelerators.

A. Ablation study on the DC accelerator optimized search space and the searched EfficientNet-X-B0 base model

  • The activation strategy improves speed but lowers accuracy on both accelerators; TPUv3's hybrid ReLU/swish selection yields higher accuracy than GPUv100's ReLU-only selection.TPUv3 uses ReLU with depthwise-convolution blocks and swish with vanilla-convolution blocks; GPUv100 selects ReLU throughout because non-fused swish degrades performance.Source: A. Ablation study on the DC… · p. 11
  • Space-to-depth and fused convolutions improve both accuracy and speed on TPUv3 and GPUv100, offsetting activation-related accuracy losses.Table 4 reports the EfficientNet-X-B0 ImageNet ablation; its inference latency uses 128-image batches, and GPUv100 models run in FP16.Source: A. Ablation study on the DC… · p. 11, Table 4 · p. 12
  • On TPUv3, the search-space enhancements contribute almost equally to inference speed, whereas activation choices produce a larger speedup and accuracy drop on GPUv100.The authors attribute the difference to the software stack and expect fused swish on GPUv100 to make its behavior more like TPUv3.Source: A. Ablation study on the DC… · p. 12

B. Ablation study on latency-aware compound scaling and the EfficientNet-X family

Latency-aware compound scaling (LACS) produces hardware-specific EfficientNet-X scaling factors and favors deeper, slimmer families than accuracy-only scaling. Its low-to-high scaling closely matches direct high-end search, while avoiding the cost of searching directly at that level.

B. Ablation study on latency-aware compound scaling and the EfficientNet-X family

  • Searching for coefficients at a lower target latency is more cost-effective, but nonlinear accuracy and latency growth can make scaled models deviate from direct-search optima.The ablation tests this risk by comparing scaled X-B7 models with direct X-B7 search.Source: B. Ablation study on latency-aware… · p. 12
  • The X-B1 LACS coefficients applied at X-B7 yield network dimensions within 6% of those from direct X-B7 Pareto search on GPUv100.The comparison tests whether scaling from a lower target latency deviates from the empirical high-end optimum.Source: B. Ablation study on latency-aware… · pp. 12–13
  • 60% ∼70% deeper and ∼40% smaller input resolutions: LACS favors deeper, slimmer models than single-objective compound scaling on GPU and TPU.The comparison covers the EfficientNet-X family and its model dimensions, including the shared B0 base model.Source: Table 6 · p. 12, B. Ablation study on latency-aware… · p. 13
  • EfficientNet-X has much faster inference speed with comparable accuracy to EfficientNet under the LACS scaling factors.LACS produces different scaling factors for TPUv3 and GPUv100, unlike the accuracy-only method.Source: B. Ablation study on latency-aware… · p. 13
  • Accuracy-only compound scaling retains EfficientNet's factors even with EfficientNet-X-B0 as its base, while LACS chooses different factors for TPUv3 and GPUv100.The LACS changes are attributed to its accuracy-latency objective and greater visibility into hardware details.Source: B. Ablation study on latency-aware… · p. 13

Related papers

Loading 2102.05610v1…