PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting the Computational Cost of Deep Learning Models

Daniel Justus, John Brennan, Stephen Bonner, Andrew Stephen McGough

arXiv:1811.11880v1cs.LGcs.AIstat.ML

TL;DR

Deep-learning training time is difficult to predict because FLOP-based linear estimates miss effects such as data movement and imperfect parallelism. The paper learns timing models for individual layers and combines their predictions to estimate complete network execution time. In the reported comparisons, the deep-learning predictor achieves lower RMSE than linear regression.

  • Problem

    The paper addresses limited accurate prediction of deep-learning execution time, especially because linear FLOP-based models do not capture nonlinear execution effects.

  • Method

    The paper trains data-driven deep-learning models on layer, network, data, and hardware features, then combines layer or batch timings to estimate whole-network execution time.

  • Results

    The deep-learning predictor reports RMSEs of 0.83 ms versus 2.42 ms for linear regression and 3.18 ms versus 8.95 ms in another forward-and-backward comparison.

  • Takeaways & Limitations

    The model can support informed choices about hardware, model design, and execution-time prediction for complete deep-learning networks.

Abstract

from arXiv · show

Deep learning is rapidly becoming a go-to tool for many artificial intelligence problems due to its ability to outperform other approaches and even humans at many problems. Despite its popularity we are still unable to accurately predict the time it will take to train a deep learning network to solve a given problem. This training time can be seen as the product of the training time per epoch and the number of epochs which need to be performed to reach the desired level of accuracy. Some work has been carried out to predict the training time for an epoch -- most have been based around the assumption that the training time is linearly related to the number of floating point operations required. However, this relationship is not true and becomes exacerbated in cases where other activities start to dominate the execution time. Such as the time to load data from memory or loss of performance due to non-optimal parallel execution. In this work we propose an alternative approach in which we train a deep learning network to predict the execution time for parts of a deep learning network. Timings for these individual parts can then be combined to provide a prediction for the whole execution time. This has advantages over linear approaches as it can model more complex scenarios. But, also, it has the ability to predict execution times for scenarios unseen in the training data. Therefore, our approach can be used not only to infer the execution time for a batch, or entire epoch, but it can also support making a well-informed choice for the appropriate hardware and model.

I. INTRODUCTION

The paper targets a priori prediction of deep-learning execution time, focusing on epoch time rather than the number of epochs needed for accuracy. It proposes learning layer-level timing models whose batch predictions can be combined for complete networks and varied hardware or architectures.

  • Motivation: Predicting execution time can inform training-cost estimates, experimentation choices, and hardware or strategy decisions before training begins.The paper frames cost broadly, including financial and execution cost.
  • Motivation: Training cost depends on both execution time per epoch and the number of epochs needed to reach a target accuracy.This work addresses estimating execution time for a single epoch; predicting the required number of epochs is left for future work.
  • Approach: The approach trains a data-driven model on features from computational resources, networks, and training data.This is intended to capture execution-time behavior beyond a simple linear model.
  • Approach: Layer-level timing predictions are combined into a batch estimate, then multiplied by the number of batches to estimate a full epoch.The framework predicts forward or forward-and-backward execution for individual layers and supports convolutional, pooling, and fully connected layers.
  • Contributions: The contributions include categorizing performance-influencing features, predicting network execution time with deep learning, and comparing predictions across hardware.The paper also presents the framework as applicable to previously unseen hardware, networks, data, or combinations of these.

II. RELATED WORK

Earlier approaches estimate execution time mainly from floating-point-operation counts or limited benchmark architectures, but nonlinear effects and hardware differences weaken such generalization. The paper motivates a feature-based prediction framework designed to model these effects and extend to unseen models or hardware.

  • Prior approaches: Prior epoch-time prediction commonly uses Big O-style estimates based primarily on floating-point operations.PALEO multiplies an epoch FLOP count by a system-specific floating-point speed scaling factor.
  • Prior approaches: Benchmarking initiatives quantify hardware performance using a few reference model architectures, limiting coverage of new architectures.The paper contrasts these reference-architecture benchmarks with prediction for broader model and hardware combinations.
  • Limits of linear models: Execution time scales only approximately linearly with FLOPs because GPU parallelism, non-floating-point work, and GPU–memory transfers introduce additional effects.These effects limit the accuracy of FLOP-based scaling assumptions.
  • Limits of linear models: Activation functions, optimizers, and incomplete utilization of compute resources or memory bandwidth introduce nonlinearities into execution time.The cited discussion states that sophisticated linear approaches can generalize badly across hardware types.
  • Proposed direction: The proposed framework uses features from computational resources, networks, and data to address nonlinearities and generalize to unseen models or hardware.The stated aim is accurate prediction while extending beyond previously observed configurations.

IV. TRAINING FEATURES

The paper organizes execution-time predictors around layer, layer-specific, implementation, and hardware features. It defines a core subset while noting that the feature space is extensible and that exhaustive feature analysis is outside the work’s scope.

  • Feature categories: Features are categorized into layer, layer-specific, implementation, and hardware features.The paper describes these categories as potentially containing an almost endless list of features.
  • Feature categories: The proposed feature set is a core subset, while additional features can be incorporated later.A complete analysis of all available features and their effects on prediction accuracy is beyond the paper’s scope.
  • Implementation features: Activation functions such as ReLU, softmax, sigmoid, and tanh can be represented with one-hot encoding.The listed activation choices also include no activation.
  • Implementation features: The optimizer is included as an implementation feature, with choices including Gradient Descent, Adadelta, Adagrad, Momentum, Adam, and RMS Prop.These optimizer categories can also be encoded using one-hot encoding.
  • Layer-specific features: Batch size records the number of training samples processed together, and values may differ across independently predicted layers.The paper states that layer-wise prediction makes differing feature values permissible.
  • Layer-specific features: For fully connected layers, relevant structural features include the input count and the number of neurons in the layer.Inputs correspond to outputs of the preceding layer, while neurons correspond to outputs of the current layer.

2) Convolutional features:

The convolutional feature set represents input dimensions, filter geometry, channel depths, stride, and padding; pooling features use kernel size, stride, and padding.

  • 2) Convolutional features:: Convolutional layers use matrix size, kernel size, input depth, output depth, stride size, and input padding as features.
  • 2) Convolutional features:: Matrix size represents the size of the input data being trained.
  • 2) Convolutional features:: Kernel size represents the size of the filter applied to image data.
  • 2) Convolutional features:: Input and output depth represent the numbers of channels or layers in convolutional input and output data.
  • 2) Convolutional features:: Pooling layers use kernel size, stride size, and input padding as distinctive features.

4) Recurrent features:

Recurrent and hardware features extend the predictor beyond basic network dimensions by encoding recurrence behavior, GPU characteristics, and interconnects.

  • 4) Recurrent features:: Recurrent networks retain MLP features and additionally encode recurrence type and bidirectionality.Recurrence types include default, LSTM, and GRU; bidirectionality is represented as a binary value.
  • 4) Recurrent features:: Hardware features describe GPU technology, count, memory, clock speed, bandwidth, core count, peak performance, and card connectivity.
  • 4) Recurrent features:: GPU technology is one-hot encoded by chip manufacturer and technology, allowing cards from the same generation to be grouped.
  • 4) Recurrent features:: GPU peak performance is recorded in GFLOPS and derives from GPU clock speed and GPU core count.
  • 4) Recurrent features:: Card connectivity uses one-hot encoding for interconnects such as PCIe3.0 x16, PCIe3.0 x4, and NVLink.

D. Training Space

The approach samples atomic layer operations, predicts their execution times with a feed-forward network, and combines batch and layer estimates into network-level timing.

  • D. Training Space: The feature space contains too many combinations for exhaustive training, so predictor networks use a random subsample.
  • D. Training Space: Each deep learning network is decomposed into individual layers treated as atomic operations, whose forward or forward-and-backward times are measured repeatedly.
  • D. Training Space: Predicted operation times are combined across layers to estimate overall network performance.
  • D. Training Space: Adding a layer type requires benchmarking batches of that type and retraining the predictor with the added feature and timing information.The predictor is described as relatively simple, making retraining time insignificant.
  • D. Training Space: The execution-time predictor is a fully connected feed-forward network trained on network, hardware, data, and actual training-run features.
  • D. Training Space: A single batch time is estimated from layer predictions, while epoch time is obtained by combining layer estimates across the network and the required number of batches.
  • D. Training Space: Predicting total training time also requires the number of epochs needed to reach target accuracy, which remains outside this work’s scope.The paper notes that fixed epoch counts can still support an estimate.

D. Comparison Metrics

Evaluation compares predicted and actual execution times on unseen data using correlation and RMSE, across individually trained and combined GPU models and sampled hardware configurations.

  • D. Comparison Metrics: Evaluation compares predicted and actual execution times on an unseen test dataset.
  • D. Comparison Metrics: Correlation assesses agreement between predicted and actual times, while lower RMSE indicates better predictions.
  • D. Comparison Metrics: Table I lists tested hardware by name, provisioning source, and main system specifications.
  • D. Comparison Metrics: Models are trained on individual GPU cards and on combined GPU sets because the available hardware platforms are limited.
  • D. Comparison Metrics: Fully connected-layer evaluation tested 25,000 parameter combinations from a search space of 30,064,771,072 combinations.Batch sizes ranged from 1 to 64, while input and output dimensions ranged from 1 to 4096.

C. Test case: Convolutional Neural Layer

The convolutional-layer benchmarks vary operation and training settings across a large parameter space, using repeated measurements and held-out data to train and evaluate the predictor.

  • Benchmark design: 10,038,745,006,080 parameter combinations define the tested convolutional-layer space across batch, matrix, kernel, stride, padding, bias, and input-output dimensions.The benchmark sampled combinations involving batch sizes 1–64, matrix sizes 1x1–512x512, kernel sizes 1x1–7x7, and additional convolution settings.
  • Benchmark design: Five benchmark runs per randomly selected feature set were summarized by the median before model training.Feature sets were sampled uniformly without dependence on previously selected features.
  • Evaluation protocol: The collected data used an 80% training, 10% test, and 10% validation split.
  • Predictor configuration: The predictor used fully connected feed-forward networks with dropout, ReLU activations, and L2 regularisation at 10^-5 to improve generalisability and accuracy.
  • Training settings: Experiments included randomly selected activation functions and optimizers, with 50% of runs using pure forward passes without backpropagation.The optimizer choices included Gradient Descent, Adadelta, Adagrad, Momentum, Adam, and RMSProp.

VII. RESULTS

The results show that the deep-learning predictor estimates layer execution times across GPUs and outperforms linear regression when execution time depends on nonlinear feature relationships.

  • Individual-GPU predictions: 1.73 ms, 3.32 ms, 6.07 ms, 7.84 ms, 11.90 ms, and 2.55 ms are the convolutional prediction RMSEs for the V100, P100, M60, K80, K40, and GTX1080Ti, respectively.Older GPUs have longer execution times and larger absolute prediction deviations.
  • Individual-GPU predictions: 0.048 ms, 0.033 ms, 0.031 ms, 0.145 ms, 0.167 ms, and 0.034 ms are the fully connected-layer RMSEs for the six GPUs, respectively.The relatively small errors correspond to the smaller average execution times of the tested fully connected layers.
  • Comparison with linear regression: 0.83 ms versus 2.42 ms is the RMSE for the deep-learning predictor versus linear regression on forward-pass convolution predictions.The linear model could not use one-hot encoded features and was supplemented with floating-point-operation counts.
  • Comparison with linear regression: 3.18 ms versus 8.95 ms is the RMSE for the deep-learning predictor versus linear regression on forward-and-backward passes with stochastic gradient descent.The linear regression error increased with actual execution time, missing nonlinear effects of longer runtimes.
  • Comparison with linear regression: Feature engineering beyond floating-point-operation counts may address some linear-model problems but cannot be expected to capture the full complexity of execution-time prediction.
  • Comparison with linear regression: The deep-learning approach can handle different optimizers simultaneously, unlike an approach restricted to forward passes or a single optimizer.

C. A general model constructed from multiple cards

A multi-GPU model incorporates hardware features to predict convolution times across cards, including held-out hardware, and combines layer estimates to analyze complete-network execution.

  • Multi-GPU model: 6 hidden layers yielded the best results after adding GPU memory bandwidth, clock frequency, and CUDA-core count as features.
  • Multi-GPU model: 3.88 ms RMSE was obtained when predicting convolution times with a model trained on data from all available GPUs.This experiment used batch sizes up to 32 because some GPUs had limited memory.
  • Unseen hardware: The held-out-GPU evaluation trained on five GPUs and predicted execution time for the sixth, testing performance on unseen hardware.Each GPU was evaluated separately using data from the other five GPUs.
  • Full-network prediction: The VGG-16 evaluation predicted each layer’s forward or forward-and-backward time and combined those estimates for the complete network.The model was trained using data from all six GPUs.
  • Full-network prediction: Fully connected layers dominate at small batch sizes, whereas convolutional layers dominate at larger batch sizes because convolution time grows more strongly with batch size.
  • Unseen hardware: 3.65 ms, 4.32 ms, 8.09 ms, 7.94 ms, 7.00 ms, and 4.24 ms were the RMSEs for held-out V100, P100, M60, K80, K40, and GTX1080Ti predictions, respectively.The authors report small systematic discrepancies and greater uncertainty outside the range of known hardware specifications.
  • Full-network prediction: Combining individual layer times supports estimating one-epoch training time and assessing effects of hardware, batch size, model characteristics, and optimizer.

VIII. CONCLUSIONS AND FUTURE DIRECTIONS

The work demonstrates component-level execution-time prediction that extends to complete neural networks and supports hardware and model-design decisions. The framework is extensible, but broader hardware benchmarking and feature investigation remain future directions.

  • The model accurately predicts execution time for frequently used neural-network components, including nonlinear components previously largely ignored.
  • These component predictions can provide a foundation for selecting training or inference hardware and informing neural-network model design.
  • The framework supports incorporating custom model architectures with hardware-specific training data and extending to other frameworks, languages, and numerical precisions.
  • Wider hardware benchmarking and investigation of additional training features are identified as future work to improve generalization across networks, data sizes, and hardware configurations.
Loading 1811.11880v1…