PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting the Computational Cost of Deep Learning Models

Daniel Justus, John Brennan, Stephen Bonner, Andrew Stephen McGough

arXiv:1811.11880v1cs.LGcs.AIstat.ML

TL;DR

Deep-learning training time is difficult to predict because FLOP-based linear estimates miss effects such as data movement and imperfect parallelism. The paper learns timing models for individual layers and combines their predictions to estimate complete network execution time. In the reported comparisons, the deep-learning predictor achieves lower RMSE than linear regression.

  • Problem

    The paper addresses limited accurate prediction of deep-learning execution time, especially because linear FLOP-based models do not capture nonlinear execution effects.

    Source: §I · p. 1

  • Method

    The paper trains data-driven deep-learning models on layer, network, data, and hardware features, then combines layer or batch timings to estimate whole-network execution time.

    Source: §I · pp. 1–2, §IV · p. 3

  • Results

    The deep-learning predictor reports RMSEs of 0.83 ms versus 2.42 ms for linear regression and 3.18 ms versus 8.95 ms in another forward-and-backward comparison.

    Source: B. Comparison with other prediction… · p. 6

  • Takeaways & Limitations

    The model can support informed choices about hardware, model design, and execution-time prediction for complete deep-learning networks.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Deep learning is rapidly becoming a go-to tool for many artificial intelligence problems due to its ability to outperform other approaches and even humans at many problems. Despite its popularity we are still unable to accurately predict the time it will take to train a deep learning network to solve a given problem. This training time can be seen as the product of the training time per epoch and the number of epochs which need to be performed to reach the desired level of accuracy. Some work has been carried out to predict the training time for an epoch -- most have been based around the assumption that the training time is linearly related to the number of floating point operations required. However, this relationship is not true and becomes exacerbated in cases where other activities start to dominate the execution time. Such as the time to load data from memory or loss of performance due to non-optimal parallel execution. In this work we propose an alternative approach in which we train a deep learning network to predict the execution time for parts of a deep learning network. Timings for these individual parts can then be combined to provide a prediction for the whole execution time. This has advantages over linear approaches as it can model more complex scenarios. But, also, it has the ability to predict execution times for scenarios unseen in the training data. Therefore, our approach can be used not only to infer the execution time for a batch, or entire epoch, but it can also support making a well-informed choice for the appropriate hardware and model.

I. INTRODUCTION

The paper targets a priori prediction of deep-learning execution time, focusing on epoch time rather than the number of epochs needed for accuracy. It proposes learning layer-level timing models whose batch predictions can be combined for complete networks and varied hardware or architectures.

Motivation

  • Predicting execution time can inform training-cost estimates, experimentation choices, and hardware or strategy decisions before training begins.The paper frames cost broadly, including financial and execution cost.Source: §I · p. 1
  • Training cost depends on both execution time per epoch and the number of epochs needed to reach a target accuracy.This work addresses estimating execution time for a single epoch; predicting the required number of epochs is left for future work.Source: §I · p. 1

Approach

  • The approach trains a data-driven model on features from computational resources, networks, and training data.This is intended to capture execution-time behavior beyond a simple linear model.Source: §I · pp. 1–2
  • Layer-level timing predictions are combined into a batch estimate, then multiplied by the number of batches to estimate a full epoch.The framework predicts forward or forward-and-backward execution for individual layers and supports convolutional, pooling, and fully connected layers.Source: §I · p. 2, Figure 1 · p. 2

Contributions

  • The contributions include categorizing performance-influencing features, predicting network execution time with deep learning, and comparing predictions across hardware.The paper also presents the framework as applicable to previously unseen hardware, networks, data, or combinations of these.Source: §I · p. 2

II. RELATED WORK

Earlier approaches estimate execution time mainly from floating-point-operation counts or limited benchmark architectures, but nonlinear effects and hardware differences weaken such generalization. The paper motivates a feature-based prediction framework designed to model these effects and extend to unseen models or hardware.

Prior approaches

  • Prior epoch-time prediction commonly uses Big O-style estimates based primarily on floating-point operations.PALEO multiplies an epoch FLOP count by a system-specific floating-point speed scaling factor.Source: §II · p. 2
  • Benchmarking initiatives quantify hardware performance using a few reference model architectures, limiting coverage of new architectures.The paper contrasts these reference-architecture benchmarks with prediction for broader model and hardware combinations.Source: §II · p. 2

Limits of linear models

  • Execution time scales only approximately linearly with FLOPs because GPU parallelism, non-floating-point work, and GPU–memory transfers introduce additional effects.These effects limit the accuracy of FLOP-based scaling assumptions.Source: §II · p. 3
  • Activation functions, optimizers, and incomplete utilization of compute resources or memory bandwidth introduce nonlinearities into execution time.The cited discussion states that sophisticated linear approaches can generalize badly across hardware types.Source: §III · p. 3

Proposed direction

  • The proposed framework uses features from computational resources, networks, and data to address nonlinearities and generalize to unseen models or hardware.The stated aim is accurate prediction while extending beyond previously observed configurations.Source: §III · p. 3

IV. TRAINING FEATURES

The paper organizes execution-time predictors around layer, layer-specific, implementation, and hardware features. It defines a core subset while noting that the feature space is extensible and that exhaustive feature analysis is outside the work’s scope.

Feature categories

  • Features are categorized into layer, layer-specific, implementation, and hardware features.The paper describes these categories as potentially containing an almost endless list of features.Source: §IV · p. 3
  • The proposed feature set is a core subset, while additional features can be incorporated later.A complete analysis of all available features and their effects on prediction accuracy is beyond the paper’s scope.Source: §IV · p. 3, A. Layer Features · p. 3

Implementation features

  • Activation functions such as ReLU, softmax, sigmoid, and tanh can be represented with one-hot encoding.The listed activation choices also include no activation.Source: A. Layer Features · p. 3
  • The optimizer is included as an implementation feature, with choices including Gradient Descent, Adadelta, Adagrad, Momentum, Adam, and RMS Prop.These optimizer categories can also be encoded using one-hot encoding.Source: A. Layer Features · p. 3

Layer-specific features

  • Batch size records the number of training samples processed together, and values may differ across independently predicted layers.The paper states that layer-wise prediction makes differing feature values permissible.Source: A. Layer Features · p. 3
  • For fully connected layers, relevant structural features include the input count and the number of neurons in the layer.Inputs correspond to outputs of the preceding layer, while neurons correspond to outputs of the current layer.Source: B. Layer Specific Features · p. 3, 1) MLP features · p. 3

2) Convolutional features:

The convolutional feature set represents input dimensions, filter geometry, channel depths, stride, and padding; pooling features use kernel size, stride, and padding.

2) Convolutional features:

4) Recurrent features:

Recurrent and hardware features extend the predictor beyond basic network dimensions by encoding recurrence behavior, GPU characteristics, and interconnects.

4) Recurrent features:

  • Recurrent networks retain MLP features and additionally encode recurrence type and bidirectionality.Recurrence types include default, LSTM, and GRU; bidirectionality is represented as a binary value.Source: 4) Recurrent features · p. 3
  • Hardware features describe GPU technology, count, memory, clock speed, bandwidth, core count, peak performance, and card connectivity.Source: C. Hardware Features · p. 4
  • GPU technology is one-hot encoded by chip manufacturer and technology, allowing cards from the same generation to be grouped.Source: C. Hardware Features · p. 4
  • GPU peak performance is recorded in GFLOPS and derives from GPU clock speed and GPU core count.Source: C. Hardware Features · p. 4
  • Card connectivity uses one-hot encoding for interconnects such as PCIe3.0 x16, PCIe3.0 x4, and NVLink.Source: C. Hardware Features · p. 4

D. Training Space

The approach samples atomic layer operations, predicts their execution times with a feed-forward network, and combines batch and layer estimates into network-level timing.

D. Training Space

  • The feature space contains too many combinations for exhaustive training, so predictor networks use a random subsample.Source: D. Training Space · p. 4
  • Each deep learning network is decomposed into individual layers treated as atomic operations, whose forward or forward-and-backward times are measured repeatedly.Source: A. General Considerations · p. 4
  • Predicted operation times are combined across layers to estimate overall network performance.Source: A. General Considerations · p. 4
  • Adding a layer type requires benchmarking batches of that type and retraining the predictor with the added feature and timing information.The predictor is described as relatively simple, making retraining time insignificant.Source: A. General Considerations · p. 4
  • The execution-time predictor is a fully connected feed-forward network trained on network, hardware, data, and actual training-run features.Source: B. Model Architecture and Training… · p. 4
  • A single batch time is estimated from layer predictions, while epoch time is obtained by combining layer estimates across the network and the required number of batches.Source: C. Full Model Predictions · p. 4
  • Predicting total training time also requires the number of epochs needed to reach target accuracy, which remains outside this work’s scope.The paper notes that fixed epoch counts can still support an estimate.Source: C. Full Model Predictions · p. 4

D. Comparison Metrics

Evaluation compares predicted and actual execution times on unseen data using correlation and RMSE, across individually trained and combined GPU models and sampled hardware configurations.

D. Comparison Metrics

  • Evaluation compares predicted and actual execution times on an unseen test dataset.Source: D. Comparison Metrics · p. 4
  • Correlation assesses agreement between predicted and actual times, while lower RMSE indicates better predictions.Source: D. Comparison Metrics · p. 4
  • Table I lists tested hardware by name, provisioning source, and main system specifications.Source: Table I · p. 5, A. Hardware · p. 5
  • Models are trained on individual GPU cards and on combined GPU sets because the available hardware platforms are limited.Source: §VI · p. 5
  • Fully connected-layer evaluation tested 25,000 parameter combinations from a search space of 30,064,771,072 combinations.Batch sizes ranged from 1 to 64, while input and output dimensions ranged from 1 to 4096.Source: B. Test Case: Fully Connected Layer · p. 5

C. Test case: Convolutional Neural Layer

The convolutional-layer benchmarks vary operation and training settings across a large parameter space, using repeated measurements and held-out data to train and evaluate the predictor.

Benchmark design

  • 10,038,745,006,080 parameter combinations define the tested convolutional-layer space across batch, matrix, kernel, stride, padding, bias, and input-output dimensions.The benchmark sampled combinations involving batch sizes 1–64, matrix sizes 1x1–512x512, kernel sizes 1x1–7x7, and additional convolution settings.Source: C. Test case: Convolutional Neural Layer · p. 5
  • Five benchmark runs per randomly selected feature set were summarized by the median before model training.Feature sets were sampled uniformly without dependence on previously selected features.Source: D. Data Collection and Preparation · p. 5

Evaluation protocol

Predictor configuration

Training settings

  • Experiments included randomly selected activation functions and optimizers, with 50% of runs using pure forward passes without backpropagation.The optimizer choices included Gradient Descent, Adadelta, Adagrad, Momentum, Adam, and RMSProp.Source: F. Implementation Details · pp. 5–6

VII. RESULTS

The results show that the deep-learning predictor estimates layer execution times across GPUs and outperforms linear regression when execution time depends on nonlinear feature relationships.

Individual-GPU predictions

  • 1.73 ms, 3.32 ms, 6.07 ms, 7.84 ms, 11.90 ms, and 2.55 ms are the convolutional prediction RMSEs for the V100, P100, M60, K80, K40, and GTX1080Ti, respectively.Older GPUs have longer execution times and larger absolute prediction deviations.Source: A. Estimating the training time for a… · p. 6, Figure 3 · p. 7
  • 0.048 ms, 0.033 ms, 0.031 ms, 0.145 ms, 0.167 ms, and 0.034 ms are the fully connected-layer RMSEs for the six GPUs, respectively.The relatively small errors correspond to the smaller average execution times of the tested fully connected layers.Source: A. Estimating the training time for a… · p. 6

Comparison with linear regression

  • 0.83 ms versus 2.42 ms is the RMSE for the deep-learning predictor versus linear regression on forward-pass convolution predictions.The linear model could not use one-hot encoded features and was supplemented with floating-point-operation counts.Source: B. Comparison with other prediction… · p. 6
  • 3.18 ms versus 8.95 ms is the RMSE for the deep-learning predictor versus linear regression on forward-and-backward passes with stochastic gradient descent.The linear regression error increased with actual execution time, missing nonlinear effects of longer runtimes.Source: B. Comparison with other prediction… · p. 6
  • Feature engineering beyond floating-point-operation counts may address some linear-model problems but cannot be expected to capture the full complexity of execution-time prediction.Source: B. Comparison with other prediction… · p. 6
  • The deep-learning approach can handle different optimizers simultaneously, unlike an approach restricted to forward passes or a single optimizer.Source: B. Comparison with other prediction… · pp. 6–7

C. A general model constructed from multiple cards

A multi-GPU model incorporates hardware features to predict convolution times across cards, including held-out hardware, and combines layer estimates to analyze complete-network execution.

Multi-GPU model

Unseen hardware

Full-network prediction

  • The VGG-16 evaluation predicted each layer’s forward or forward-and-backward time and combined those estimates for the complete network.The model was trained using data from all six GPUs.Source: D. Inferring a full model prediction · p. 8
  • Fully connected layers dominate at small batch sizes, whereas convolutional layers dominate at larger batch sizes because convolution time grows more strongly with batch size.Source: D. Inferring a full model prediction · p. 8

Unseen hardware

  • 3.65 ms, 4.32 ms, 8.09 ms, 7.94 ms, 7.00 ms, and 4.24 ms were the RMSEs for held-out V100, P100, M60, K80, K40, and GTX1080Ti predictions, respectively.The authors report small systematic discrepancies and greater uncertainty outside the range of known hardware specifications.Source: C. A general model constructed from… · p. 7, Figure 7 · p. 9

Full-network prediction

VIII. CONCLUSIONS AND FUTURE DIRECTIONS

The work demonstrates component-level execution-time prediction that extends to complete neural networks and supports hardware and model-design decisions. The framework is extensible, but broader hardware benchmarking and feature investigation remain future directions.

  • The model accurately predicts execution time for frequently used neural-network components, including nonlinear components previously largely ignored.Source: §VIII · p. 10
  • These component predictions can provide a foundation for selecting training or inference hardware and informing neural-network model design.Source: §VIII · p. 10
  • The framework supports incorporating custom model architectures with hardware-specific training data and extending to other frameworks, languages, and numerical precisions.Source: §VIII · p. 10
  • Wider hardware benchmarking and investigation of additional training features are identified as future work to improve generalization across networks, data sizes, and hardware configurations.Source: §VIII · p. 10

Related papers

Loading 1811.11880v1…