PaperlayerView exact revision on arXiv

Source-linked AI summary

The Efficiency Misnomer

Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, Yi Tay

arXiv:2110.12894v2cs.LGcs.AIcs.CLcs.CVstat.ML

TL;DR

Model efficiency affects training costs and inference experience, yet common cost indicators need not agree. The paper reviews these indicators and demonstrates that incomplete reporting can produce partial comparisons, recommending broader, setup-specific reporting.

  • Problem

    Common efficiency indicators are often assumed to correlate, although each captures different aspects and can mislead when used alone.

    Source: §1 · p. 1

  • Method

    The paper reviews common cost indicators, analyzes experiments and literature practices, and offers reporting recommendations.

    Source: §1 · p. 2

  • Results

    Efficiency rankings differ by metric: EfficientNets lie on the accuracy-parameter and accuracy-FLOPs Pareto frontiers, while SwinTransformers dominate the accuracy-throughput curve.

    Source: §3.1 · p. 7

  • Takeaways & Limitations

    Report and plot available cost indicators together, and limit efficiency claims to the evaluated setup.

    Source: §4 · p. 9

    A training-cost comparison at fixed model M and quality X does not establish comparisons for another model or a higher target quality.

    Source: §2.1 · p. 5

Abstract

from arXiv · show

Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.

1 INTRODUCTION

Model efficiency matters because training affects financial and environmental costs, while inference speed can shape user experience. The paper argues that no single cost indicator gives a complete comparison and recommends broader, setup-specific reporting.

1 INTRODUCTION

  • FLOPs alone can mislead because they omit parallelism and hardware details such as memory-access costs.A low-FLOPs model may not be fast; the paper notes FLOPs is commonly used to quantify efficiency, especially in computer vision.Source: §1 · p. 1
  • Parameter count can indicate memory fit but is unreliable alone: shared parameters across many computational steps can leave a model slow.Parameter-matched models may still differ in speed.Source: §1 · p. 1
  • Efficiency comparisons can change with the chosen indicator: Figure 1 shows model rankings reversing across cost metrics.The paper identifies parameters, FLOPs, and throughput as common indicators.Source: §1 · p. 1, Figure 1 · p. 2
  • Throughput comparisons tightly couple implementation, hardware optimization, and infrastructure details, limiting apples-to-apples comparisons across setups.Source: §1 · p. 2
  • Efficiency trade-offs depend on whether costs are measured for training or inference and on the application’s needs.For example, parameter sharing can save inference memory while training still requires storing activations for the backward pass.Source: §1 · p. 2
  • The paper reviews cost indicators, demonstrates how incomplete reporting can misrepresent efficiency, and recommends considering multiple indicators.It calls this phenomenon the “efficiency misnomer.”Source: §1 · pp. 1–2

2 A PRIMER ON COST INDICATORS

Model efficiency has no universally sufficient cost indicator: measures can disagree, and the relevant trade-offs depend on whether training or inference is being considered. The section reviews common indicators and shows how architecture, training choices, and hardware can change what appears efficient.

2 A PRIMER ON COST INDICATORS

  • FLOPs, parameter count, speed, and memory-related measures capture different costs, and theoretical FLOPs omit parallelism and other practical factors.The paper focuses on parameters, FLOPs, and speed as common indicators, while noting other measures may matter by use case.Source: §2 · pp. 3–4
  • Speed measures include throughput, latency, and runtime; unlike throughput, latency does not account for batching parallelism, and speed depends on hardware and implementation.The section also identifies pipeline bubble and memory access cost as measures relevant to runtime.Source: §2 · p. 3

2.1 TRAINING OR INFERENCE COST?

  • Training and inference costs can diverge: parameter sharing may reduce inference memory, while training activations and optimizer memory can remain substantial.Training efficiency matters for iteration and retraining, while inference efficiency is often treated as the deployed cost.Source: §2.1 · p. 4
  • Training-time comparisons depend on the full recipe, and step counts alone can be stretched, so meaningful claims compare accuracy with total training cost.The section cautions that a method with lower apparent training cost may not have been optimized for that cost.Source: §2.1 · p. 4
  • Short schedules and lower weight decay can converge faster initially yet finish below slower schedules or higher-weight-decay settings.The cited examples show that early progress is not a reliable proxy for final performance.Source: Figure 2 · p. 4, §2.1 · p. 5, Figure 3 · p. 5

2.2 POTENTIAL DISAGREEMENT BETWEEN COST INDICATORS

  • Parameter sharing reduces trainable parameters without changing FLOPs or speed, while sparse models can cut theoretical FLOPs without comparable speed-ups.Switch Transformer offers a better quality trade-off by FLOPs and speed than by parameter count; routing overhead and hardware constraints limit sparse-model speed-ups.Source: §2.2 · pp. 5–6
  • For ViTs, width can be favorable at smaller FLOPs or parameter budgets and depth at larger budgets, while speed can favor width over depth.W3072 is not necessarily slower than D48 because it has fewer sequential and more parallelizable operations.Source: §2.2 · p. 6, Figure 4 · p. 6
  • Hardware and implementation can reverse apparent efficiency: weight decomposition may reduce FLOPs yet run slower on GPUs, and architecture speed can differ across platforms.The section also describes implementation choices that affect memory cost on specific hardware.Source: §2.2 · p. 6

3 DISCUSSION

Efficiency rankings vary with the cost indicator, and neither parameter-matched nor compute-matched comparisons are universally fair. The discussion gives examples of these trade-offs and urges caution when choosing indicators for architecture search.

3.1 ON PARTIAL CONCLUSIONS FROM INCOMPLETE COMPARISONS

  • EfficientNets lead accuracy–parameter and accuracy–FLOPs trade-offs, while SwinTransformers dominate accuracy–throughput, so no model is clearly most efficient across metrics.At similar FLOPs, EfficientNet has fewer parameters but slower throughput than RegNet and SwinTransformer.Source: §3.1 · p. 7
  • Transformer variants with equal parameter counts can differ in GFLOPs, throughput, and accuracy when input resolution changes.Within-family cost indicators often correlate, but this is not always true when scaling depth.Source: §3.1 · pp. 7–8

3.2.1 THE ISSUES WITH PARAMETER MATCHED COMPARISONS

  • Token-free models drop subword-embedding parameters, but scaling the Transformer stack to compensate substantially reduces speed.The added stack parameters incur more compute than embedding parameters; ideally, comparisons would be both parameter- and compute-matched.Source: §3.2.1 · p. 8
  • An encoder-decoder with L layers in each stack has approximately the parameters of a 2L-layer decoder-only model, but half the compute and twice the speed.This parameter-matched comparison may be unfair when speed matters more than memory consumption.Source: §3.2.1 · p. 8
  • Sparse models remain compute-matched while scaling to many parameters, so parameter-matching them can unfairly downplay their strengths.In the cited Switch Transformer comparison, matching parameter counts favors the vanilla Transformer.Source: §3.2.1 · p. 8
  • For ViT, larger patch sizes have more parameters but fewer FLOPs and faster inference.Table 1 reports parameters, FLOPs, and inference speed across patch sizes.Source: §3.2.1 · pp. 8–9, Table 1 · p. 9

3.2.2 THE ISSUES WITH COMPUTE MATCHED COMPARISONS

  • Compute-matched comparisons can handicap a baseline in capacity; the Perceiver IO example shrinks BERT to 20M parameters versus 425M for the proposed approach.The baseline also has 6 layers versus 40, while a speed-matched comparison would make it substantially faster than the proposed method.Source: §3.2.2 · p. 9

3.3 COST INDICATORS FOR ARCHITECTURE SEARCH

  • Architecture searches that assume cost indicators are interchangeable risk finding impractical models; indicator choice also affects search sensitivity and computational cost.The paper recommends extra caution when choosing indicators as resource constraints in architecture search.Source: §3.3 · p. 9

4 SUGGESTIONS AND CONCLUSION

The conclusion cautions against relying on a single efficiency indicator and recommends reporting multiple indicators while limiting claims to the evaluated setup.

4 SUGGESTIONS AND CONCLUSION

  • Parameter count is particularly misleading as a standalone efficiency measure, and estimating capacity across different architectures remains an open research problem.The paper notes that parameter count is often used to imply model capacity, but that inference is wrong when architectures vary nontrivially.Source: §4 · p. 9
  • Report and plot curves using all available cost indicators rather than highlighting results with only one.Each indicator captures a different aspect and has its own advantages and disadvantages.Source: §4 · p. 9
  • For ViT-B/64, the patch-embedding parameter count is 64×64×3×768.The general expression given is n×n×3×emb dim for patch size n×n.Source: §4 · p. 9

A EXPERIMENTAL SETUP: SCALING DEPTH VS. SCALING WIDTH

The appendix presents Vision Transformer experiments comparing model scaling by depth with scaling by width.

A EXPERIMENTAL SETUP: SCALING DEPTH VS. SCALING WIDTH

B ADDITIONAL COST INDICATORS

The appendix describes additional cost indicators beyond parameter count, FLOPs, and speed, including training data, environmental impact, runtime-related activations, and memory use.

B ADDITIONAL COST INDICATORS

  • Sample efficiency can be expressed as the training data needed to reach reasonable performance, such as data points or language-model tokens.It depends on factors including inductive biases and the training curriculum.Source: B ADDITIONAL COST INDICATORS · p. 16
  • A model’s carbon footprint during training or inference proxies environmental impact, but quantifying it directly is not straightforward.The cited estimate combines training and query energy with datacenter carbon emissions per kilowatt-hour.Source: B ADDITIONAL COST INDICATORS · p. 16
  • Monetary cost can represent training or serving expenses; training cost may be computed from total training time, chip count, and cloud prices.The passage says this indicator is more common in business documents than scientific articles.Source: B ADDITIONAL COST INDICATORS · p. 16
  • Activation size counts elements in building-block output tensors and has been reported as more strongly correlated with runtime than FLOPs on memory-bandwidth-limited hardware.This runtime relationship is attributed to Dollár et al. (2021).Source: B ADDITIONAL COST INDICATORS · p. 16
  • Memory reporting can use peak usage or the maximum batch size that fits a specific task and device.Unlike activation size, memory footprint depends on hardware and implementation and can determine whether a model fits on a device.Source: B ADDITIONAL COST INDICATORS · p. 16

Related papers

Loading 2110.12894v2…