PaperlayerView exact revision on arXiv

Source-linked AI summary

FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment

Enrique Barba Roque, Luís Cruz

arXiv:2608.14550v1cs.AIcs.PF

TL;DR

FLOPs are widely used to assess AI computational cost, but their relationship to execution time may not hold across operations or newer hardware. This paper replicates prior experiments and finds that raw FLOPs remain an inadequate execution-time proxy, while the alpha-FLOPs formula generally underestimates latency on modern hardware.

  • Problem

    The paper asks whether prior findings about FLOPs and execution time, including the alpha-FLOPs estimation, remain applicable on newer hardware.

  • Method

    The study replicates prior data collection, hardware-specific regression, and execution-time measurements for convolutional configurations.

  • Results

    Raw FLOPs do not predict equal execution times; spatial FLOPs are generally more parallelizable than kernel FLOPs, while alpha-FLOPs usually underestimates latency.

  • Takeaways & Limitations

    The findings support hardware-aware efficiency assessment and rigorous, complete replication packages for AI research.

  • Takeaways & Limitations

    The original replication materials omit code, regression data, dependency details, and other methodological information needed for full replication.

Abstract

from arXiv · show

AI efficiency has recently taken the spotlight in both academy and industry due to massive model scales, high energy demands, and environmental costs. While reporting Floating Point Operations (FLOPs) is a traditional approach for assessing computational costs, the relationship between FLOPs and execution time is not straightforward, as layers with the same number of FLOPs may not have the same execution time because some operations are more easily parallelized than others. This paper sets out to replicate the original experiments from a study that proposed the $α-FLOPs$ estimation formula to verify whether the results remain applicable on newer, more powerful hardware. During the replication process, we identify limitations in the replication materials provided by the original study, including a lack of specific dependency details and transparency regarding regression data. Our results validate the thesis that raw FLOPs alone are not an appropriate metric for execution time, as spatial dimensions remain more easily parallelized than kernel dimensions. However, fine-grained measurements reveal that the relationship is much less straightforward than previously shown, with newer hardware exhibiting instabilities and discontinuities in execution time, including jumps and oscillations, that the $α-FLOPs$ formula generally underestimates. Ultimately, this work validates the empirical findings from the original study but shows negative results when applying the $α-FLOPs$ estimation. We also highlight the critical need for complete and accurate replication packages for research on hardware-dependent efficiency assessment and provide a complete replication package for our implementation to facilitate further study.

1 Introduction

The introduction motivates hardware-aware AI efficiency assessment because FLOPs do not straightforwardly predict execution time, and presents a replication study of the α-FLOPs approach. It also identifies incomplete replication materials and defines objectives concerning FLOPs, parallelizability, and hardware-specific estimation.

  • Motivation: AI’s growing scale creates substantial computational, energy, and environmental costs, increasing the need to optimize inference time and energy consumption.LLMs may contain billions of parameters trained on terabytes of code, while projected AI energy consumption is expected to double over five years.
  • Efficiency metrics: FLOPs are easy to derive but do not reliably represent execution time because equally costly layers may differ in parallelizability.Execution time and energy are more direct metrics but require specialized profilers and bare-metal access during development.
  • Implications: Hardware-aware execution-time estimates could support model-design decisions and extend parallelization analysis and the α-FLOPs formula across AI architectures.The introduction frames these extensions as potential implications for Software Engineering for AI.
  • Replication rationale: The study replicates prior experiments to test whether their findings remain applicable on newer hardware and exposes limitations in the original replication package.The package lacks experiment-running code, specific dependency details, regression-data transparency, and consistency between paper descriptions and implementations.
  • Replication objectives: The replication objectives test whether raw FLOPs inadequately proxy CNN execution time, whether spatial FLOPs are more parallelizable than kernel FLOPs, and whether α-FLOPs enables accurate hardware-specific estimates.These objectives correspond to RO1, RO2, and RO3, respectively.

2 Background

FLOPs provide an intuitive operation-count measure, but they do not directly determine execution time because parallel operations are not additive. Convolutional-layer experiments show that equal-FLOP configurations can run at very different speeds, motivating a hyperparameter-aware correction.

  • 2 Background: FLOPs-based cost comparison is naive because parallel operations can execute concurrently rather than contributing additively to runtime.The impact depends on operation characteristics and GPU capacity.
  • 2 Background: Experiments vary convolutional input dimensions, channel counts, and kernel sizes while maintaining the same theoretical FLOPs.The measurements compare forward execution times across different CNN hyperparameter configurations.
  • 2 Background: Layers with the same theoretical FLOPs exhibit vastly different execution times depending on the hyperparameters and where those FLOPs are applied.Spatial-dimension FLOPs are more easily parallelized, producing lower execution times as input size increases.
  • 2 Background: Increasing kernel dimensions while reducing size or channel counts increases execution time for layers with the same FLOPs.This contrasts with the favorable parallelization of spatial dimensions.
  • 2 Background: The original study proposes a FLOPs correction that estimates convolutional-layer execution time from FLOPs and hyperparameters.Its parameters are obtained by regression from empirical convolutional configurations and are hardware-dependent, although preliminary results found little variance across hardware.

3 Limitations to Replicability

Replication on newer hardware exposed substantial limitations in the original materials, including unspecified dependencies, missing regression data and configuration, and inconsistencies between the paper’s formulas and supplied code. These gaps hinder reproducibility and require assumptions in the replication methodology.

  • Motivation: Because the methodology is hardware-dependent, replication on newer GPUs is necessary to verify whether the original findings remain applicable.The empirical approach could also be extended to different AI architectures to identify FLOP and execution-time patterns across hyperparameters.
  • Replication-material limitations: The provided dataset supports recreating plots but does not constitute a complete replication package because timing and regression code are omitted.The original authors explicitly state that they do not provide code for timing execution or regression over data.
  • Replication-material limitations: Unspecified AI-library, library-version, GPU-driver, and CUDA details introduce methodological variability and can change execution time through newer processing optimizations.The paper does not identify whether TensorFlow or PyTorch implements the convolutional layers.
  • Replication-material limitations: Estimating α-FLOPs parameters β_k and γ_k requires regression data and configuration details that the original study does not provide.The regression uses convolutional configurations represented by (S, C, K) and execution time T, where S, C, and K denote input, channel, and kernel sizes.
  • Replication-material limitations: The artifact omits the FLOPs formula’s constant 2 and applies an unjustified final = 0.0375 scaling factor after estimating execution time.The omitted constant can create inconsistencies when regression and estimation use different formulas.

4 Methodology

The replication follows the original study’s three-step workflow: collecting convolutional execution times, fitting α-FLOPs parameters, and comparing measured times with formula estimates. It adapts the configuration sampling, regression, measurement, and software setup to newer hardware while documenting assumptions where replication materials are incomplete.

  • Replication workflow: The replication collects convolutional execution times, regresses α-FLOPs parameters, and compares measured times with α-FLOPs estimates.These are the three steps defined in the original study.
  • Data collection: Sampling spans non-square input/output sizes, selected channel ranges, and square kernels K∈[1, 14], with ranges chosen to cover configurations analyzed later.Input and output sizes use {1, 2, 4, 8, 16, 32, 64, 96, 128, 256}; channels follow the specified piecewise sets.
  • Regression: Regression minimizes Mean Squared Error using least squares, fits β_k and γ_k, and adds f_i_n_a_l_k as a scaling parameter.The implementation uses scipy’s curve_fit with initial values β_k=0.01, γ_k=0.8, and f_i_n_a_l_k=0.3, bounds [0, 2], and separate regressions for K=1 and K>1.
  • Measurement set: Experiments test constant-FLOPs and increasing-FLOPs configurations to examine whether equal-FLOPs layers differ in execution time because of parallelization.Each configuration is run 2000 times, and the average execution time is reported; increasing-FLOPs variables span 1 to 650 in D, E, F and 1 to 1000 in G.

5 Results

The results confirm that equal theoretical FLOPs can yield different execution times because channel-dimension work is more computationally intensive than spatial-dimension work. On newer hardware, execution-time behavior is often discontinuous and hardware-specific, causing the α-FLOPs formula to underestimate measurements despite some qualitative agreement with prior results.

  • Regression results: The regression produced β1 = 0.00972514, γ1 = 1.1779239, and final1 = 0.00549217 for K=1, versus βK = 0.00785036, γK = 1.0601711, and finalK = 0.00401956 for K>1.These values were obtained from the collected data for the authors’ hardware.
  • Experiments B–C: Equal theoretical FLOPs do not ensure equal forward-pass time: introducing FLOPs through channels is more computationally intensive than introducing them through spatial dimensions.The comparison holds configurations on the same theoretical-FLOPs vertical line.
  • Replication comparison: The empirical trends broadly match the original study with a ∼10× speedup, but α-FLOPs estimates are less accurate on the newer hardware.Original execution times ranged from ∼0.2 to 8 milliseconds, while the replicated setup followed similar evolution with increasing size.
  • Experiment C: α-FLOPs reproduces the qualitative shape but underestimates execution time, with RMSE between 0.2 and 0.6 ms and around 47% deviation from the measured range.The estimates remain distant from empirical results in the reported configuration.
  • Experiments C–E: Execution time becomes unstable across kernel, spatial, and channel dimensions, showing configuration reversals, discrete jumps, and oscillations that are more erratic than in the original measurements.Largest reported jumps occur at W = {164, 246, 328, 492} and Cout = {101, 192, 257, 385, 513}; some configurations run 17−20% faster.
  • Experiments F–G: The α-FLOPs formula generally fails to capture hardware-specific behavior, including nonlinearities for K=3 and K=5, while latency can jump after Cout exceeds 500 for H=300.Experiment F is less unstable than earlier experiments but still shows substantially worse estimation accuracy.

6 Discussion

The replication confirms that raw FLOPs do not reliably predict convolutional execution time, while spatial dimensions are generally more parallelizable than kernel dimensions. However, newer hardware produces unstable timing behavior and largely invalidates accurate α-FLOPs estimation, underscoring the need for complete replication materials.

  • Empirical replication: Equal-FLOPs convolution configurations can have significantly different execution times because parallelization effectiveness depends on which dimensions increase.Spatial dimensions are generally more easily parallelized than kernel dimensions.
  • α-FLOPs estimation: Most α-FLOPs configurations are underestimated, and the approach fails to reproduce accurate estimations despite modeling spatial parallelization and kernel-related costs.The correction considers input size S and kernel size K but omits input and output channels, which materially affect execution time on the newer setup.
  • Replication limitations: Missing replication materials forced assumptions about hardware, regression data, and author choices, preventing clear separation of hardware effects from setup discrepancies.These assumptions are identified as threats to replication validity and may explain the negative estimation results.
  • Hardware-dependent behavior: ∼10× faster empirical measurements than the original study are attributed mainly to differences in GPU hardware, including cores, memory, compute units, and TF32 support.The comparison specifically contrasts the RTX 4090’s Ada Lovelace architecture with the original study’s Turing architecture.
  • Hardware-dependent behavior: Fine-grained measurements reveal large execution-time variance between consecutive configurations, including unstable behavior that lower-resolution experiments can conceal.The newer hardware’s behavior only partially resembles the original results, with some experiments showing similar trends and others greater instability.
  • Replication materials: The paper provides a complete α-FLOPs replication package to support testing on other hardware and the development of extensions or fixes.The authors argue that complete and accurate packages would facilitate reuse of efficiency-assessment techniques and results.

7 Related Work

Related work questions FLOPs’ validity as a proxy for AI efficiency because hardware optimizations can decouple logical FLOPs from execution time and energy. Other studies instead emphasize parallelization and careful replication of hardware-bound benchmarks.

  • FLOPs as proxy for AI model efficiency: FLOPs are easy to calculate from architecture but may fail to capture GPU and TPU optimizations affecting hardware execution.This creates a gap between logical FLOPs and hardware FLOPs in efficiency assessment.
  • FLOPs as proxy for AI model efficiency: 36% higher inference throughput on GPU and up to 2.8× faster performance were achieved by optimizing parallelization rather than minimizing FLOPs.The cited work reports that reducing FLOPs does not necessarily produce a similar latency reduction.
  • FLOPs as proxy for AI model efficiency: FLOPs showed little correlation with energy across architectures but significant correlation within the same architecture, supporting relative architectural ordering.The findings come from repeated inference measurements on pretrained image-classification models.
  • Replication of hardware-bound benchmarks: Poor replication scripts can obscure whether negative reproducibility results arise from software or hardware differences in HPC benchmarking.Pearce et al. propose an open-source framework with step-by-step replication support.

8 Conclusion

The replication confirms that raw FLOPs inadequately represent CNN execution time because equal-FLOP configurations can run differently and efficiency depends on parallelization. Although spatial FLOPs remain easier to parallelize than kernel FLOPs, newer hardware exhibits discontinuities and oscillations, while α-FLOPs generally underestimates latency.

  • Replication findings: Equal-FLOP CNN configurations do not exhibit equal execution times, confirming that FLOPs are not a good proxy for execution time.The replication used modern hardware, including an NVIDIA RTX 4090, and found a consistent 10× speedup over the original study’s results.
  • Replication findings: Spatial FLOPs are generally easier to parallelize than kernel FLOPs, replicating the original study’s overall trend.Newer hardware additionally shows discontinuities and oscillations that were not clearly visible in the original results.
  • Estimation results: The α-FLOPs model underestimates latency in most configurations and does not consistently provide accurate estimates.The replication therefore does not reproduce the original result of accurate time estimation.
  • Reproducibility: The study provides a fully documented replication package to improve transparency and support extending the estimation approach to newer architectures.It emphasizes the critical necessity of complete and accurate replication packages in AI efficiency research.
Loading 2608.14550v1…