Source-linked AI summary
NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks
Ermao Cai, Da-Cheng Juan, Dimitrios Stamoulis, Diana Marculescu
TL;DR
NeuralPower addresses the need to predict CNN serving energy before deployment and training. It uses layer-wise sparse polynomial regression to predict power, runtime, and energy, achieving strong network-level accuracy and improving runtime prediction over prior work by up to 68.5%.
Problem
CNN serving energy must be predicted before deployment because runtime and energy are critical when architectures are invoked repeatedly across devices and services.
Method
NeuralPower uses layer-wise learning-based polynomial regression with architecture and target-platform features to predict and break down CNN power, runtime, and energy.
Results
NeuralPower achieves average network-level accuracies of 88.24% for runtime, 88.34% for power, and 97.21% for energy, while improving runtime prediction accuracy by up to 68.5% over prior work.
Takeaways & Limitations
The framework helps machine learners identify layer-level bottlenecks and select energy-efficient CNN architectures for deployment.
Takeaways & Limitations
Network-level runtime prediction assumes that summing predicted layer runtimes provides a good estimate of total runtime.
Abstract
from arXiv · showhide
"How much energy is consumed for an inference made by a convolutional neural network (CNN)?" With the increased popularity of CNNs deployed on the wide-spectrum of platforms (from mobile devices to workstations), the answer to this question has drawn significant attention. From lengthening battery life of mobile devices to reducing the energy bill of a datacenter, it is important to understand the energy efficiency of CNNs during serving for making an inference, before actually training the model. In this work, we propose NeuralPower: a layer-wise predictive framework based on sparse polynomial regression, for predicting the serving energy consumption of a CNN deployed on any GPU platform. Given the architecture of a CNN, NeuralPower provides an accurate prediction and breakdown for power and runtime across all layers in the whole network, helping machine learners quickly identify the power, runtime, or energy bottlenecks. We also propose the "energy-precision ratio" (EPR) metric to guide machine learners in selecting an energy-efficient CNN architecture that better trades off the energy consumption and prediction accuracy. The experimental results show that the prediction accuracy of the proposed NeuralPower outperforms the best published model to date, yielding an improvement in accuracy of up to 68.5%. We also assess the accuracy of predictions at the network level, by predicting the runtime, power, and energy of state-of-the-art CNN architectures, achieving an average accuracy of 88.24% in runtime, 88.34% in power, and 97.21% in energy. We comprehensively corroborate the effectiveness of NeuralPower as a powerful framework for machine learners by testing it on different GPU platforms and Deep Learning software tools.
1. Introduction
NeuralPower addresses the need to predict CNN serving energy before deployment by modeling power, runtime, and energy from the architecture and target GPU platform. It also helps select energy-efficient architectures and locate layer-level bottlenecks.
- Motivation: CNN serving energy matters because deployment can invoke the same architecture billions of times across millions of devices.The paper focuses on testing or service rather than training because training is usually performed once.
- Motivation: More than 40× energy variation can occur among CNN architectures with similar test error around 20%.The comparison uses randomly sampled architectures on CIFAR-10 with TensorFlow on an Nvidia Titan X GPU.
- NeuralPower: NeuralPower predicts CNN power, runtime, and energy from the architecture and target deployment platform without actually running the CNN there.The framework is intended for the testing or deployment phase.
- Results: 88.34% average accuracy is reported for GPU power prediction, while runtime accuracy improves by up to 68.5% over the best previously published work.The runtime comparison is against state-of-the-art analytical models.
- Results: 97.21% average accuracy is reported for energy prediction using power and runtime predictions.NeuralPower also provides layer-level runtime and power breakdowns to identify bottlenecks.
- NeuralPower: The Energy-Precision Ratio guides trade-offs between CNN classification accuracy and energy efficiency.This metric is proposed to support energy-efficient architecture selection.
2. Background and Related Work
Prior work addresses CNN runtime and energy constraints through hardware, software, and architecture-oriented methods, but lacks a comprehensive model of runtime, power, and energy across platforms. NeuralPower is positioned against these limited or inflexible approaches.
- Design constraints: Runtime and power overhead are significant design concerns for accurate, power-efficient CNNs, especially as architectures grow larger or target mobile platforms.Existing approaches therefore accelerate execution or reduce power and energy consumption.
- Runtime-efficient CNNs: Prior runtime-efficient CNN research explores hardware platforms such as FPGAs and ASIC-like designs alongside software acceleration libraries.These approaches address execution speed through hardware- and software-based methodologies.
- Power- and energy-aware CNNs: Power- and energy-aware CNN research includes hardware-adaptive architecture customization and trainer-trainee CNN methods.These approaches aim to limit energy or power while minimally affecting inference accuracy.
- Metric modeling: Prior modeling methods use proxies such as memory consumption, runtime, or network-weight counts rather than comprehensively modeling runtime, power, and energy.The paper identifies this as a gap in existing CNN metric modeling.
- Metric modeling: Paleo analytically estimates runtime across platforms but requires detailed platform optimization knowledge and cannot predict power or energy.This limits its flexibility across platforms with different optimization libraries.
3. Methodology: Power and Runtime Modeling
NeuralPower builds layer-level polynomial models for CNN runtime and power, then composes them into network-level runtime, power, and energy predictions for deployment platforms.
- Layer-level modeling: NeuralPower models convolutional, fully connected, and pooling layers because they carry the main computation load during CNN execution.The framework aims to remain flexible across software and hardware platforms without requiring platform details.
- Layer-level modeling: Learning-based polynomial regression estimates layer runtime and power from configuration features and physically meaningful operation and memory-access terms.Runtime features include parameters such as batch size, tensor sizes, kernel shape, stride, and padding.
- Layer-level modeling: Polynomial models can represent non-monotonic runtime effects, such as increased stride reducing total convolutional operations and runtime.This behavior motivates polynomial formulations over posynomial models for runtime prediction.
- Layer-level modeling: Power models augment runtime features with logarithmic terms to capture power’s bounded, non-linear response to increasing computational load.The extended power feature vector includes logarithms of features such as batch size, input size, and output size.
- Network-level modeling: Network-level runtime sums predicted layer runtimes, while average power uses layer runtimes and powers, and total energy sums layer-wise energy consumption.The network models apply layer predictions as building blocks for architectures whose layers are connected in series.
- Data collection: The evaluation collects layer power and runtime data on a fixed GPU configuration, using 1 ms instantaneous-power measurements across CNN architectures.Training data include 858 convolution, 216 pooling, and 116 fully connected layer samples.
4. Experimental Results
The evaluation assesses NeuralPower’s power, runtime, and energy prediction accuracy at both layer and network levels, and tests its Energy-Precision Ratio and cross-platform applicability.
- Evaluation scope: NeuralPower is evaluated for power, runtime, and energy prediction accuracy at both layer and network levels.The evaluation also examines the Energy-Precision Ratio as a guide toward energy-efficient CNNs.
- Evaluation scope: The experiments use the Energy-Precision Ratio to guide machine learners toward energy-efficient CNNs.The same evaluation section also validates the framework on other hardware and software platforms.
- Evaluation scope: The framework is validated across other hardware and software platforms in addition to the primary evaluation setting.
4.1. Layer-Level Model Evaluation
Layer-level evaluation selects sparse polynomial models by cross-validation and compares runtime and power predictions against available baselines or measured values.
- Model selection: NeuralPower varies polynomial order, applies Lasso, and selects the model with the lowest cross-validation RMSE.Figure 3 reports the best-performing model for each polynomial order.
- Model selection: Polynomial order two achieves the best RMSE for both runtime and power modeling while also attaining the lowest RMSPE.
- Runtime models: Up to 68.5% improvement in accuracy over Paleo is achieved by the layer-level runtime models.The comparison uses RMSE and RMSPE against the state-of-the-art analytical model.
- Runtime models: Convolutional runtime modeling uses a degree-three polynomial with batch size, tensor, kernel, stride, padding, and output features, while fully connected modeling uses degree two.Pooling models use input tensor, stride, kernel, and output tensor features, with degree three selected by Lasso and cross-validation.
- Power models: Power prediction errors remain below 9% for all three evaluated layer types, with logarithmic and operation or memory-access terms included in the model.The model captures power-consumption trends across layer sizes and types.
4.2. Network-level Modeling Evaluation
NeuralPower is evaluated for network-level runtime, power, and energy prediction, including layer-wise bottleneck profiling. Across CNN architectures, it generally outperforms Paleo for runtime and achieves accurate power and energy estimates.
- Layer-level profiling: NeuralPower provides layer-wise runtime and power profiling to identify bottlenecks across CNN components.The framework is intended to support benchmarking and architecture analysis at the layer level.
- Runtime evaluation: Our model captures the dominant conv4 layer in NIN, whereas Paleo incorrectly identifies conv2 as dominant.For VGG-16, NeuralPower predicts fc6 with -2.53% error, compared with Paleo’s -96.16% error.
- Runtime evaluation: Runtime predictions remain within 24% of actual values across five networks, while prior art underestimates overall runtime by up to 42%.The layer-runtime summation also approximates total runtime, supporting the network-level formulation.
- Power evaluation: Convolutional layers dominate power consumption, while fc6 has high runtime but relatively low power, showing runtime alone can misrepresent energy contribution.This comparison motivates modeling power and runtime jointly when estimating energy.
- Power evaluation: Our power predictions have 11.66% RMSPE across five CNNs, including 10.47% error for AlexNet and 20.48% for NIN.The larger errors in AlexNet and NIN are attributed to underestimated runtime in the power formulation.
- Energy evaluation: NeuralPower predicts total energy with an average RMSPE of 2.79% across the evaluated CNNs.Energy is predicted from the model’s per-layer runtime and power estimates.
4.3. Energy-Precision Ratio
The Energy-Precision Ratio is introduced to help select CNN architectures that trade classification accuracy against energy consumption. Lower values indicate a better supported trade-off, and the preferred architecture can change with α.
- Metric definition: Energy-Precision Ratio guides selection of accurate yet energy-efficient CNN architectures.It is intended for machine learners choosing models for energy-constrained deployment.
- Metric definition: The metric combines classification error and energy consumption per classified data item, with α controlling the weight on error.A larger α places more importance on classification accuracy.
- Metric interpretation: Lower M values indicate a better trade-off between energy efficiency and CNN accuracy.For α = 1, AlexNet and NIN have smaller M values than VGG-16, despite VGG-16 having the lowest error.
- Metric interpretation: Network choices vary with α: AlexNet is selected for α = 1, 2, 3, while VGG-16 is selected for α = 4.The metric therefore reflects how the desired accuracy–energy balance changes with α.
4.4. Models on other platforms
NeuralPower is evaluated beyond the original setting on an Nvidia GTX 1070 and with Caffe. These experiments retrain platform- and software-specific models and evaluate runtime and power predictions for CNN layers and networks.
- Cross-platform evaluation: The framework is extended to Nvidia GTX 1070 and the Caffe deep-learning software tool.This tests the intended flexibility across hardware and software platforms.
- GTX 1070 evaluation: On GTX 1070 with TensorFlow, the authors repeat runtime and power data collection and train models for that platform.The layer-wise and overall network evaluations are reported in Tables 8 and 9.
- Caffe evaluation: Caffe evaluation uses layer-by-layer benchmarking with extended instrumentation to collect runtime and power data.The default Caffe version reports timing only, so the implementation was extended for power measurement.
- Caffe evaluation: For Caffe on GTX 1070, the authors report per-layer runtime and power accuracy and evaluate AlexNet and NIN.Whole-network execution is not directly comparable under Caffe, so the evaluation instead uses the corresponding model equations.
4.5. Discussion
NeuralPower’s one-time training overhead is small relative to its intended use in CNN architecture search and bottleneck analysis.
- Overhead: Training on a new GPU platform takes around 30 minutes, including under 10 minutes for data collection and less than 20 minutes for model training.The process is performed once for a new platform.
- Overhead: The overhead can be offset when the CNN architecture search space is large, while detailed bottleneck breakdowns remain available for smaller searches.The breakdown covers runtime, power, and energy.
5. Conclusion
NeuralPower estimates CNN runtime, power, and energy during testing, outperforming the current state-of-the-art runtime model and providing network-level accuracy estimates. It also breaks down CNN behavior to help identify design bottlenecks.
- 88.24% runtime, 88.34% power, and 97.21% energy average accuracy were achieved for state-of-the-art CNN configurations.
- NeuralPower provides an accurate breakdown of CNN networks to help machine learners identify runtime, power, and energy bottlenecks.
- The framework’s runtime model outperforms the current state-of-the-art predictive model in accuracy.