PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors

Siqi Wang, Gayathri Ananthanarayanan, Yifan Zeng, Neeraj Goel, Anuj Pathania, Tulika Mitra

arXiv:1903.05898v3cs.LGcs.DCcs.PF

TL;DR

Efficient CNN inference on ARM big.LITTLE processors is hindered by inter-cluster communication overhead when kernels are parallelized across heterogeneous clusters. Pipe-it uses layer-level pipelining, performance prediction, and design-space exploration to balance stages, improving throughput by 39% on average over the baseline.

  • Problem

    Inter-cluster communication overhead limits CNN inference throughput when heterogeneous ARM big.LITTLE cores parallelize convolution kernels across clusters.

  • Method

    Pipe-it predicts layer execution times across core configurations and uses heuristic design-space exploration to allocate CNN layers among balanced pipeline stages.

  • Results

    39% average throughput improvement over using only the homogeneous high-performance Big cluster is achieved by Pipe-it.

  • Takeaways & Limitations

    Layer-level splitting can use heterogeneous multi-core resources more effectively than kernel-level splitting for CNN inference throughput.

Abstract

from arXiv · show

IoT Edge intelligence requires Convolutional Neural Network (CNN) inference to take place in the edge devices itself. ARM big.LITTLE architecture is at the heart of prevalent commercial edge devices. It comprises of single-ISA heterogeneous cores grouped into multiple homogeneous clusters that enable power and performance trade-offs. All cores are expected to be simultaneously employed in inference to attain maximal throughput. However, high communication overhead involved in parallelization of computations from convolution kernels across clusters is detrimental to throughput. We present an alternative framework called Pipe-it that employs pipelined design to split convolutional layers across clusters while limiting parallelization of their respective kernels to the assigned cluster. We develop a performance-prediction model that utilizes only the convolutional layer descriptors to predict the execution time of each layer individually on all permitted core configurations (type and count). Pipe-it then exploits the predictions to create a balanced pipeline using an efficient design space exploration algorithm. Pipe-it on average results in a 39% higher throughput than the highest antecedent throughput.

I. INTRODUCTION

CNN inference is increasingly important for real-time edge applications, but kernel-level parallelization on ARM big.LITTLE processors suffers from inter-cluster communication overhead. Pipe-it addresses this limitation by pipelining CNN layers across homogeneous core clusters using performance prediction and design-space exploration, improving throughput by 39%.

  • I. INTRODUCTION: CPU remains a practical edge-inference platform because dedicated accelerators have constrained applicability, while CNN applications increasingly require concurrent use of available resources.The motivation includes real-time vision workloads such as ADAS, VR, and AR, as well as multi-task systems such as smart classrooms and autonomous drones.
  • I. INTRODUCTION: Kernel-level Heterogeneous Multi-Processing throughput drops sharply when Small cores from another cluster are added because of inter-cluster communication overhead.The limitation stems from the kernel-level strategy rather than implementation quality.
  • I. INTRODUCTION: Pipe-it partitions CNN layers across heterogeneous core clusters so each homogeneous core group processes a fixed set of layers as a pipeline stage.This layer-level strategy avoids assigning kernels from the same layer across clusters, while the mapping space grows exponentially with network depth.
  • I. INTRODUCTION: 39% average throughput improvement is achieved over using only the high-performance homogeneous Big cluster by employing the entire heterogeneous multi-core with Pipe-it.Pipe-it combines predicted convolutional-layer performance with design-space exploration to select the pipeline configuration and layer allocation.
  • I. INTRODUCTION: Pipe-it predicts convolutional-layer performance on permitted core types and counts from network structure descriptors, then uses those predictions to find a balanced pipeline configuration.The framework targets the diverse resource requirements of CNN layers while limiting kernel parallelization to assigned homogeneous clusters.

II. BACKGROUND · III. CO-EXECUTION AT DIFFERENT LEVELS · A. Kernel-Level Splitting

The background establishes ARM-CL as an optimized ARM CNN framework whose graph nodes execute convolution kernels across cores. Kernel-level splitting across heterogeneous clusters does not improve throughput over Big-cluster-only execution for most CNNs, due to cache-related communication costs.

  • II. BACKGROUND: ARM-CL, Tencent NCNN, and TVM [6] provide competing ARM CNN-inference frameworks, with ARM-CL and NCNN supporting NEON acceleration for intensive convolution kernels.The comparison is described for multi-threaded inference on the Big cluster using ARM-CL version 18.05.
  • II. BACKGROUND: ARM-CL represents CNN workloads as graph nodes connected in sequence, with frontend graph construction and backend-managed execution.Its graph API supports complex networks and maps weighted layers such as convolutional and fully connected layers into implementations summarized in Table I.
  • II. BACKGROUND: Within each ARM-CL node, convolution uses NEON-accelerated im2col and GEMM kernels whose computations can be distributed across multiple cores.The runtime scheduler dispatches kernels to processing units during execution.
  • II. BACKGROUND: The extended implementation runs multiple independent image graphs concurrently while sharing read-only weights and biases across graphs.Execution uses one thread per core with thread pinning and minimal migrations.
  • III. CO-EXECUTION AT DIFFERENT LEVELS · A. Kernel-Level Splitting: Adding cores within one cluster improves kernel parallelization, whereas extending kernel splitting across heterogeneous clusters fails to improve throughput.The observed limitation is reported for ARM-CL thread-pool execution and summarized in Figure 3.
  • A. Kernel-Level Splitting: For most CNNs, no Big/Small kernel-workload split yields statistically higher throughput than running kernels exclusively on the Big cluster.This conclusion follows exhaustive search over workload ratios and is illustrated by the normalized throughput comparison.
  • A. Kernel-Level Splitting: Cross-cluster execution increases average on-chip L2 access latency because cache conflict misses may be served through the interconnect from another cluster’s L2 cache.Within-cluster parallel L2 accesses are handled by the cluster’s SCU without being overwhelmed, but splitting the working set across clusters introduces additional cache interactions.

B. Layer-Level Splitting · IV. DESIGN SPACE · A. Split Points at Convolutional Layers

Pipe-it assigns convolutional layers to pipeline stages because convolution dominates CNN inference and deeper layers generally require less processing. Optimal split points vary substantially across networks and pipeline configurations, with a three-stage ResNet50 pipeline reaching 5.6 Img/s and exceeding the best two-stage pipeline by 7%.

  • B. Layer-Level Splitting: Convolutional processing dominates total forward-pass time for all evaluated networks except AlexNet, where fully connected layers dominate.This makes convolutional-layer allocation the primary focus for most networks.
  • B. Layer-Level Splitting: Convolutional-layer processing generally decreases with network depth because filtering reduces the spatial size and dimensionality of intermediate data.Initial layers process the largest original inputs, whereas later layers receive smaller feature maps.
  • B. Layer-Level Splitting: Pipe-it can balance heterogeneous clusters by assigning intensive initial layers to high-performance Big cores and less intensive deeper layers to low-performance Small cores.Kernels within assigned layers may still be split among homogeneous cores in each cluster, while non-convolutional layers are associated with preceding convolutional layers.
  • B. Layer-Level Splitting: Layer-level splitting reduces inter-cluster L2 conflict misses and CCI load by keeping dependent layers together, while allowing different images to execute concurrently across clusters.For example, the Big cluster can process image Z+1 while the Small cluster processes image Z.
  • IV. DESIGN SPACE: Because convolutional-layer structures and cluster performance vary substantially, selecting split points is a non-trivial Pipe-it design-space decision.A two-stage B4-s4 pipeline must choose one split point X among W−1 possible locations, while a three-stage pipeline must choose two split points.
  • A. Split Points at Convolutional Layers: 5.6 Img/s is achieved by the optimal ResNet50 B4-s2-s2 pipeline with split points at Layers 33 and 45, delivering 7% higher throughput than the corresponding optimal two-stage pipeline.The first split separates Big and Small clusters, while the second separates the two Small-core stages.

B. Stages of Pipelines

Pipe-it restricts pipeline stages to homogeneous core types, assigning Big cores to initial convolutional layers and Small cores to later layers. Even under these constraints, the layer-level splitting design space is too large for exhaustive exploration.

  • Pipeline design constraints: Pipeline stages use homogeneous core types, with Big cores assigned to initial convolutional layers and Small cores to subsequent layers.The design excludes heterogeneous core types within a stage because kernel-level splitting between clusters is not helpful.
  • Pipeline design space: The number of pipelines is enumerated by varying the number of Big and Small stages subject to feasible bounds on pB and ps.Equation (1) counts pipeline configurations, while Equation (2) counts layer-level design points for a CNN with W convolutional layers.
  • Pipeline design space: 64 pipelines are possible for the eight-core prototype board when using 2 to 8 stages.The count is calculated with Equation (1).
  • Pipeline design space: 5,379,616 distinct design points exist for MobileNet’s 28 convolutional layers under layer-level splitting.The design space grows further for larger CNNs such as GoogLeNet and ResNet50, making exhaustive exploration infeasible.

C. The Pipe-it Framework · V. LAYER-WISE PERFORMANCE ESTIMATION · A. Convolution as GEMM

Pipe-it combines layer-wise execution-time prediction from static convolution descriptors with heuristic design-space exploration to find near-optimal pipelined configurations and workload allocations. Its estimation basis is GEMM-based convolution, whose dimensions determine arithmetic work while execution time also depends on memory access and parallelism.

  • C. The Pipe-it Framework: Pipe-it predicts every layer’s execution time across possible core configurations, then heuristically searches the design space for a near-optimal pipeline and workload allocation.This two-part framework uses static network-layer configuration descriptors for prediction and predicted timing information for configuration search.
  • V. LAYER-WISE PERFORMANCE ESTIMATION: Convolutional layers dominate CNN execution time because they perform extensive tensor-filter calculations and require hardware-dependent optimization.Each convolution produces an output tensor that feeds subsequent layers.
  • V. LAYER-WISE PERFORMANCE ESTIMATION: ARM-CL implements convolution with Im2col, GEMM, and Col2Im, while prior work [22] links convolution execution time linearly to matrix dimensions.Pipe-it builds on correlating statically available convolution-layer descriptors with layer execution time.
  • A. Convolution as GEMM: A convolution layer maps an input tensor and filters to an output tensor, subject to matching input and filter depth, padding, and stride.The passage also notes that input and output tensors are usually square.
  • A. Convolution as GEMM: ARM-CL rearranges input patches into rows and filters into columns, transforming convolution into GEMM between an [N×K] image matrix and a [K×M] filter matrix.The resulting matrix is later resized into the output tensor.
  • A. Convolution as GEMM: The GEMM transformation performs N × K × M arithmetic operations for each convolution.This operation count follows from the dimensions of the input, filter, and result matrices.
  • A. Convolution as GEMM: GEMM execution time depends jointly on memory accesses, arithmetic computations, and exploitable parallelism in the convolution kernel.Therefore, matrix dimensions alone do not exhaust the factors governing GEMM performance.

B. Single Core Estimation · C. Multi-core Estimation

The framework estimates convolutional-layer execution time from GEMM behavior, first for single-core configurations and then for multi-core allocations. Across five benchmark CNNs, the resulting model predicts layer execution accurately, with average errors of 13.2% on Big cores and 11.4% on Small cores.

  • B. Single Core Estimation: Micro-benchmarks measure representative convolutional layers and GEMM configurations using randomly generated inputs and filter parameters.Measurements use ARM-CL and vary input sizes, filter sizes, and other configuration points.
  • B. Single Core Estimation: The single-core model predicts convolutional-layer time by linearly regressing on GEMM matrix dimensions N, K, and M.The regression coefficients are fitted from measurements, and interaction terms represent matrix-size products such as NK, KM, and NM.
  • C. Multi-core Estimation: ARM-CL assigns tiled GEMM work across H threads by dividing the image matrix rows into iterations, with niter = N/ts.The tile size ts is selected according to cache sizes, and the threads collectively process all iterations.
  • C. Multi-core Estimation: For single-threaded execution, all iterations are processed sequentially, providing the basis for estimating each iteration’s time.The model assumes identical processing time across iterations when extending the single-thread estimate.
  • C. Multi-core Estimation: For multi-threaded execution, total time is determined by the slowest thread, with coefficients fitted by linear regression.The model estimates multi-threaded execution time from workload distribution across H threads.
  • C. Multi-core Estimation: The multi-core model combines iteration timing with an equal workload split, using matrix size N, tile size ts, and core count H.For homogeneous cores, the expected allocation is itert = niter/H = N/(ts ∗ H).
  • C. Multi-core Estimation: 13.2% and 11.4% are the overall average prediction errors for Big and Small cores, respectively, across all homogeneous core allocations and five benchmark CNNs.Table III reports prediction error averaged across all convolutional layers for each possible homogeneous allocation.

D. Fully-connected Layers … B. Work-Flow Split Determination

Pipe-it models fully-connected layers alongside convolutions and searches heterogeneous pipeline configurations to maximize throughput. Its workflow-splitting heuristic balances adjacent-stage latency by moving layers toward deeper stages under an ordered core-capability assumption.

  • D. Fully-connected Layers: Pipe-it treats fully-connected layers as major components because AlexNet spends substantial execution time on them, while their many parameters cause excessive memory transfers.Newer CNNs commonly omit fully-connected layers or retain only one classifier layer.
  • D. Fully-connected Layers: Micro-benchmarks show execution time varies linearly with input tensor size for a fixed neuron count, enabling regression-based prediction for fully-connected layers.The evaluated neuron counts are 4096 and 1000.
  • VI. DESIGN SPACE EXPLORATION: Pipe-it uses a heuristic design-space search because pipeline stage counts, core combinations, and layer allocations create many possible heterogeneous pipeline configurations.The objective is to quickly obtain a high-performing layer-level split.
  • A. Definitions: A pipeline is represented by stages defined by core type and count, with each stage assigned an ordered subset of convolutional layers.Only homogeneous Big or Small cores are used within an individual stage.
  • B. Work-Flow Split Determination: Pipe-it orders stages from more to less compute-capable based on the assumption that initial CNN layers require more processing power than deeper layers.This ordering produces increasing layer-processing times toward deeper stages and supports one-way workload flow.
  • B. Work-Flow Split Determination: Because the slowest pipeline stage determines throughput, Pipe-it seeks balanced stage workloads by minimizing the maximum stage latency.The layer execution-time matrix supplies predicted times for each layer and core configuration.
  • B. Work-Flow Split Determination: For adjacent stages, Pipe-it starts with all layers on the faster stage and moves trailing layers deeper until the next stage becomes the bottleneck.It repeats this pairwise balancing across successive stages until allocation stabilizes.

C. Pipeline Stage Merging

Pipe-it merges pipeline stages to alleviate bottlenecks while matching stage sizes to layer-specific multi-threading speedups. Merging is restricted to same-type cores, accepted only when performance improves, and followed by workload reallocation until no beneficial merge remains.

  • C. Pipeline Stage Merging: Different layers benefit unequally from multi-threading, while saturating TLP makes speedup gains concave as core allocation increases.Stage sizes must therefore be matched to the speedup characteristics of their allocated layers.
  • C. Pipeline Stage Merging: Algorithm 3 merges pipeline stages to create more compute-capable stages for layers that bottleneck the initial one-core-per-stage pipeline.The process starts with an (H_B + H_s)-stage pipeline and uses Algorithm 2 to search workload splits before merging.
  • C. Pipeline Stage Merging: A merge combines adjacent stages only when they use the same core type, producing a stage whose core count is the sum of both inputs.Stages P_i and P_i+1 become P_i′ with count_i + count_i+1 cores and retain their originally allocated layer sets.
  • C. Pipeline Stage Merging: Merging continues only when the combined stage outperforms at least one input stage; otherwise, concavity implies further merging is unhelpful.After each successful merge, Algorithm 2 reoptimizes layer allocation, and Algorithm 3 iterates until no beneficial merge remains.

D. An Example

The example applies Pipe-it to ResNet50 on an eight-core heterogeneous processor, using predicted layer times to iteratively balance and merge pipeline stages. The procedure ultimately selects a three-stage configuration spanning four Big and four Small cores.

  • Example setup: The example predicts execution times for 54 ResNet50 layers across eight possible core combinations, forming a time matrix T of size (54,8).The architecture contains four Big and four Small cores, enabling eight different pipeline-stage core combinations.
  • Pipeline construction: Algorithm 3 initializes eight single-core stages and invokes Algorithm 2 to split and balance the workload across them.Algorithm 2 initially assigns all layers to P1, then uses Algorithm 1 to move layers into successive stages until workloads are balanced.
  • Stage merging: After balancing adjacent stages, Algorithm 3 tests mergers when Equation (12) holds, recalculating allocations and stopping further merging when a merger is unhelpful.In the example, merging begins with the first two Big-core stages and later proceeds through the Small cluster.
  • Final configuration: Pipe-it ultimately selects P = {(B, 4), (s, 2), (s, 2)} with workload allocation L = {l1−35, l36−44, l45−54}.The algorithm recalculates workload allocation after each stage update and merges stages in the Big cluster before applying similar rules to the Small cluster.

VII. EXPERIMENTAL EVALUATION · A. Resultant Configurations

On the HiKey 970, Pipe-it uses predicted layer timings to construct balanced heterogeneous pipelines and improves CNN inference throughput over the four-Big-core baseline. The evaluation covers five CNNs using 50-image streams, with resultant pipeline configurations and layer allocations reported for comparison.

  • VII. EXPERIMENTAL EVALUATION: Experiments run on the HiKey 970’s ARM big.LITTLE octa-core CPU, comprising four A73 Big cores at 2.4 GHz and four A53 Small cores at 1.8 GHz.The evaluation uses the platform’s maximum core frequencies and five CNN models specified in Table I.
  • VII. EXPERIMENTAL EVALUATION: Each data point averages throughput over a continuous stream of 50 images, with approximately 10 seconds of runtime after cooling between runs.The runtime makes exhaustive search for an optimal configuration impractical because searching an average CNN would require about five million points and hundreds of days.
  • VII. EXPERIMENTAL EVALUATION: The baseline is kernel-level splitting on four homogeneous Big cores because splitting across all eight heterogeneous cores performs worse and default ARM-CL achieves its best throughput there.This baseline is used for the reported throughput comparisons.
  • A. Resultant Configurations: Pipe-it’s best configurations are reported as pipeline stages and layer allocations, including B4-s2-s2 for ResNet50 across layers 1–35, 36–44, and 45–54.The example uses three stages containing four Big cores, two Small cores, and two Small cores, respectively.
  • A. Resultant Configurations: Throughput is compared for homogeneous execution and Pipe-it heterogeneous pipelines using measured and predicted layer execution times.Table IV presents the throughput of the respective pipelines, while Tables V and VI summarize configurations derived from predicted and measured timings.
  • A. Resultant Configurations: 39% average throughput improvement over the baseline demonstrates Pipe-it’s overall benefit, while even LeNet gains 20.6% from its three-stage pipeline.The benefit is attributed to deep, balanced pipelines that distribute major layers across stages.

B. Layer Performance-Prediction Model … E. Quantization Considerations

Pipe-it uses layer-time predictions to balance pipelines across heterogeneous clusters, generalizes to non-standard big.LITTLE configurations, and improves quantized MobileNet throughput. Its benefits are shaped by prediction accuracy, implementation-dependent quantization overheads, and added inter-cluster memory-coherency power.

  • C. General Applicability: Pipe-it applies to heterogeneous multi-cores with at least two clusters and identifies pipelines engaging both clusters on a simulated three-Big/two-Small configuration.The configuration was created by disabling one Big and two Small cores on the HiKey 970; its benefit was less significant than on the standard platform.
  • D. Power Efficiency: Power measurements use whole-board socket power because the development board lacks individual CPU-component sensors, while unused clusters are turned off during homogeneous runs.The measured socket power includes non-CPU board components, and active memory power cannot be separated from CPU power.
  • D. Power Efficiency: Lower-than-expected Small-cluster power efficiency for memory-intensive CNNs such as AlexNet is attributed to memory power, while Pipe-it adds coherency-related memory consumption.The limitation arises because active memory power cannot be separated from CPU power in the measurements.
  • E. Quantization Considerations: Quantization is orthogonal to Pipe-it, but de-quantization and re-quantization overheads can reduce its benefits; ARM-CL supports 8-bit asymmetric integers (QASYMM8).With ARM-CL, quantized MobileNet convolutional-layer execution improves by 14% in the reported comparison.
  • E. Quantization Considerations: 24% faster convolutional layers and 19% faster overall execution are obtained with MobileNet quantization on ARM-CL v18.11 versus its F32 implementation.F32 MobileNet itself executes 20% faster on v18.11 than on v18.05, showing that quantization benefits depend on the implementation version.
  • E. Quantization Considerations: 31 Img/sec is achieved by quantized MobileNet with Pipe-it on ARM-CL v18.11, which improves performance across all evaluated implementations.The comparison uses effective per-frame latency, the inverse of throughput, and includes original and quantized MobileNet across ARM-CL versions.

F. Comparison with Other Frameworks · VIII. RELATED WORK · IX. CONCLUSION

Pipe-it achieves the highest measured MobileNet performance among the compared frameworks and comparable energy-efficiency to DeepX at much higher throughput. The paper positions this result within CPU-, accelerator-, and resource-aware CNN deployment research, concluding that layer-level splitting improves heterogeneous multi-core inference throughput.

  • F. Comparison with Other Frameworks: Pipe-it provides the highest MobileNet performance among the compared CNN frameworks, based on experiments and approximately platform-scaled results from prior sources, [13].Figure 15 compares TVM, NCNN, Pipe-it, and Pipe-it** using experiments, while remaining framework values are taken from other sources and scaled approximately for platform differences.
  • F. Comparison with Other Frameworks: Pipe-it achieves 1.8 Img/J versus DeepX’s 2.2 Img/J for AlexNet, while delivering much higher throughput under the reported comparison conditions.DeepX’s 2.2 Img/J uses 444 mJ at a 500 ms latency requirement, or 2 Img/s, on a Qualcomm Snapdragon 800 four-core CPU.
  • VIII. RELATED WORK: CNN deployment research targets increasingly compact models, with model size falling from 250MB for AlexNet to under 0.5MB for SqueezeNet [13] while retaining accuracy.These advances enable deployment on mobile platforms with limited computational and memory resources.
  • VIII. RELATED WORK: Accelerator-oriented approaches use embedded GPUs and heterogeneous processors, while DeepX combines runtime layer compression with workload decomposition across processors.DeepX targets edge neural-network execution through co-execution on CPUs, GPUs, and low-power processors.
  • VIII. RELATED WORK: Other work characterizes CNN resource requirements, [22], develops efficient edge libraries,, [6], [19], and automatically generates platform-specific C or GPU code.These approaches address resource constraints, implementation support, and hardware-specific optimization requirements.
  • VIII. RELATED WORK: CPU-only approaches remain important for platforms lacking capable GPUs or accelerators; Graphi [30] exploits layer-level parallelism by scheduling independent neural-network layers concurrently.Graphi is particularly beneficial for networks such as LSTM and GoogLeNet that have high layer-level parallelism.
  • IX. CONCLUSION: The conclusion identifies kernel-level splitting across heterogeneous core types as throughput-detrimental and presents Pipe-it’s layer-level splitting as a way to minimize cross-cluster coherency.Pipe-it efficiently uses entire heterogeneous multi-cores to improve CNN inference throughput.
Loading 1903.05898v3…