Optimizing CNN Model Inference on CPUs

Yizhi Liu; Yao Wang; Ruofei Yu; Mu Li; Vin Sharma; Yida Wang

Source-linked AI summary

Optimizing CNN Model Inference on CPUs

Yizhi Liu, Yao Wang, Ruofei Yu, Mu Li, Vin Sharma, Yida Wang

arXiv:1809.02697v3Sep 7, 2018cs.DC

TL;DR

CNN inference on CPUs remains difficult to optimize end to end because library-based operation tuning limits coordinated graph-level optimization. NeoCPU uses operation templates and joint operation–graph optimization without third-party libraries, achieving up to 3.45× speedup over state-of-the-art solutions across popular CPUs. Its scope primarily covers direct convolution and shared-memory processors without NUMA optimization.

Problem
Library-based CNN inference optimizes individual operations but limits coordinated graph-level optimization, leaving end-to-end performance gains unrealized on widely used CPUs.
Method
NeoCPU uses flexible operation templates and jointly optimizes operations and graph data layouts without relying on third-party high-performance libraries.
Results
Up to 3.45× speedup over state-of-the-art solutions was achieved on 15 popular CNN models across Intel Skylake, AMD EPYC, and ARM Cortex A72 CPUs.
Takeaways & Limitations
NeoCPU provides flexible end-to-end CNN inference optimization across multiple CPU architectures and workloads.
Takeaways & Limitations
The paper primarily addresses direct convolution and restricts optimization to shared-memory processors, leaving Winograd, FFT, and NUMA beyond its main scope.

Abstract

from arXiv · show

The popularity of Convolutional Neural Network (CNN) models and the ubiquity of CPUs imply that better performance of CNN model inference on CPUs can deliver significant gain to a large number of users. To improve the performance of CNN inference on CPUs, current approaches like MXNet and Intel OpenVINO usually treat the model as a graph and use the high-performance libraries such as Intel MKL-DNN to implement the operations of the graph. While achieving reasonable performance on individual operations from the off-the-shelf libraries, this solution makes it inflexible to conduct optimizations at the graph level, as the local operation-level optimizations are predefined. Therefore, it is restrictive and misses the opportunity to optimize the end-to-end inference pipeline as a whole. This paper presents \emph{NeoCPU}, a comprehensive approach of CNN model inference on CPUs that employs a full-stack and systematic scheme of optimizations. \emph{NeoCPU} optimizes the operations as templates without relying on third-parties libraries, which enables further improvement of the performance via operation- and graph-level joint optimization. Experiments show that \emph{NeoCPU} achieves up to 3.45$\times$ lower latency for CNN model inference than the current state-of-the-art implementations on various kinds of popular CPUs.

1 Introduction

NeoCPU addresses the limits of library-based CNN inference by jointly optimizing operations and computation graphs without third-party kernel libraries. It combines flexible operation templates with global data-layout planning and achieves strong performance across CPUs and models.

Motivation: Third-party kernel libraries optimize individual CNN operations but restrict coordinated graph-level optimization, leaving end-to-end performance gains unrealized.Framework-level fusion and layout planning are limited when operation implementations are predefined in external libraries.
Approach: NeoCPU performs full-stack operation- and graph-level joint optimization without relying on third-party high-performance libraries.Its operation templates support flexible optimization across workloads and CPU architectures.
Approach: NeoCPU globally selects data layouts across CNN operations to reduce transformation overhead while preserving individual-operation performance.The scheme coordinates operation implementations for end-to-end inference rather than optimizing each operation in isolation.
Scope: The paper primarily addresses direct convolution computation, while remaining compatible with Winograd and FFT-based convolution optimizations.These alternative convolution algorithms are identified as complementary optimization work.
Results: NeoCPU delivers the best performance for 13 of 15 networks on Intel Skylake, 14 of 15 on AMD EPYC, and all 15 on ARM Cortex A72.The evaluation spans x86 and ARM CPUs and compares against framework-specific and framework-agnostic solutions.
Deployment: NeoCPU produces a small standalone module independent of frameworks and high-performance kernel libraries, supporting deployment across platforms.It is used in Amazon SageMaker Neo and has supported production CNN deployments on multiple platforms.

2 Background

CNN inference is modeled as computation-graph execution dominated by convolution, while associated operations are often memory-bound. Efficient inference therefore depends on architecture-aware computation and data-layout management across the graph.

CPU scope: The scope excludes NUMA systems and avoids hyper-threading because an additional thread on a fully utilized physical core can reduce performance.The optimization targets shared-memory programming on processors.
CNN computation graphs: CNN models are represented as directed acyclic computation graphs whose nodes are operations and whose edges carry dependencies.Inference flows input data through this graph to produce the output.
Convolution workloads: Convolutions account for most CNN inference computation and can exploit CPU parallelization, vectorization, and fused multiply-add features.Architecture-friendly data layouts and computation arrangements are established techniques for high-performance convolution.
Associated operations: The remaining CNN operations are mostly memory-bound tasks associated with convolution, including normalization, pooling, activation, and element-wise addition.These operations are commonly fused with convolutions to increase arithmetic intensity.
Training applicability: CNN inference optimization can also apply to training because training uses the same essential computation graph with additional backward operations.Training graphs are larger and include operations such as loss functions.

3 Optimizations

NeoCPU optimizes CNN inference end to end through configurable convolution templates, custom parallelization, and graph-aware layout and schedule searches. These components jointly target CPU-specific computation, memory access, and transformation overheads.

Overview: NeoCPU extends TVM with end-to-end optimization that jointly considers operation schedules and graph-level decisions.The approach avoids relying on third-party high-performance kernel libraries and searches for a globally best optimization.
Single thread optimization: NCHW[x]c and KCRS[x]c[y]k layouts reorganize tensors and kernels for improved memory locality and vectorization.The output layout uses a potentially different split factor y, while register blocking uses reg_n to split output width.
Operation optimization: Convolution templates combine dimension ordering, register blocking, parallel execution, and configurable blocking and unrolling parameters.The template exposes x, y, reg_n, and unroll_ker so schedules can adapt to CPU cache and vector-width characteristics.
Thread-level parallelization: A customized thread pool assigns disjoint convolution work to physical cores and coordinates workers with atomics and lock-free queues.Thread binding reduces hardware contention, while cache-line padding addresses false sharing; hyper-threading is not used.
Layout transformation elimination: Graph-level layout planning reduces costly transformations between convolutions and other operations that require different layouts.Because transformations can introduce significant overhead, the graph inserts LayoutTransform nodes only where the selected layout plan requires them.
Search-based optimization: Local search selects convolution schedules from (ic_bn, oc_bn, reg_n, unroll_ker), while global search combines candidates to minimize end-to-end execution time.The approximation algorithm achieves at least 88% of the best available result and completes quickly when dynamic programming is impractical.

4 Evaluation

NeoCPU is evaluated across popular CNN models, CPU architectures, baselines, and individual optimization techniques. It generally achieves strong performance, with gains attributed to joint operation- and graph-level optimization, layout management, global search, and customized threading.

Overall performance: NeoCPU achieves 0.94–1.15× performance on Intel Skylake, 0.92–1.72× on AMD EPYC, and 2.05–3.45× on ARM Cortex A72 versus the best baseline for each model.The comparison averages execution times over 1000 single-image inference runs.
Overall performance: NeoCPU avoids third-party high-performance libraries, enabling model-wide optimization that outperforms baselines across different CPU architectures.OpenVINO does not work on ARM because it relies on MKL-DNN, while NeoCPU is independent of these libraries.
Layout optimization of CONV: 1.1–1.5× additional acceleration comes from eliminating unnecessary data layout transformations after CONV layout optimization.NeoCPU infers layouts across the computation graph and inserts transformations only when needed.
Optimization scheme search: 1.1–1.5× additional performance over manually selected layouts comes from the global search algorithm, with larger gains on more complex networks such as ResNet-50.The search also removes tedious manual parameter selection and accelerates SSD through an approximation algorithm.
Multi-thread parallelization: NeoCPU’s custom thread pool scales better than OpenMP because repeated thread-launch overhead is larger across the model’s multiple parallelization regions.Figure 4 reports single-image inferences per second as a function of thread count.

5 Related Works

Related work spans optimized deep-learning frameworks, compiler stacks, operation-level kernels, and auto-tuning systems. NeoCPU distinguishes itself by extending optimization across the entire CNN model rather than focusing only on individual operations.

Deep-learning acceleration: Modern deep-learning frameworks accelerate workloads across CPUs, GPUs, FPGAs, and specialized hardware by leveraging optimized implementations for each target.The related-work landscape includes both framework-based and hardware-specific acceleration efforts.
Deep-learning compilers: NeoCPU builds on TVM, an end-to-end compiler stack that represents models in intermediate representations and compiles them to machine code.Other related compiler systems include TensorFlow XLA, Tensor Comprehensions, Glow, and DLVM.
Operation-level optimization: Prior convolution and matrix-multiplication optimizations mainly target individual operations without maintaining data layouts across the entire network.These approaches carefully optimize computational kernels but do not address end-to-end layout coordination.
Global optimization: NeoCPU extends auto-tuning from single-operation performance to global search across the entire CNN model.Its non-greedy and approximation-based search addresses models with complicated structures.

6 Conclusion

The paper presents an end-to-end compiler and optimization approach for CNN inference on modern CPUs. Across Intel Skylake, AMD EPYC, and ARM Cortex A72, the experiments report up to 3.45× speedup over state-of-the-art solutions.

Conclusion: NeoCPU compiles and optimizes CNNs end-to-end for efficient inference on modern CPUs.The approach targets Intel Skylake, AMD EPYC, and ARM Cortex A72 systems.
Conclusion: Up to 3.45× speedup is achieved on 15 popular CNN models compared with state-of-the-art solutions.The reported evaluation spans Intel Skylake, AMD EPYC, and ARM Cortex A72 CPUs.
Future work: Future work includes Winograd and FFT convolutions, quantized inference, other hardware platforms, and dynamic-shape models.The paper specifically mentions INT8 inference, NVIDIA GPUs with TensorRT, and RNNs as directions.