PaperlayerView exact revision on arXiv

Source-linked AI summary

Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs

Yujie Zhang, Huiying Lan, Ehsan Aghapour, Zhiyuan Ning, Peng Zan, Weidong Shao, Anuj Pathania, Tulika Mitra

arXiv:2609.04168v1cs.DCcs.LGcs.PF

TL;DR

Heterogeneous SoCs and operator-rich neural networks create a latency–throughput trade-off that existing execution approaches do not resolve. Para-Pipe combines hierarchical operator parallelism with pipelining, producing hybrid configurations that balance these objectives and identify energy-efficient mappings.

  • Problem

    Existing ML frameworks often underutilize heterogeneous SoC compute units, while pipelining favors throughput and purely parallel execution favors latency at the expense of the other metric.

  • Method

    Para-Pipe hierarchically maps neural-network subgraphs with different operator-parallelism levels to heterogeneous processors for parallel or sequential execution across pipeline stages.

  • Results

    Para-Pipe generates Pareto-optimal configurations balancing latency and throughput across Amlogic and BST SoCs; on Amlogic, hybrid-L reduces latency by 36.0% with a 12.4% throughput decrease versus pipe-only.

  • Takeaways & Limitations

    The framework identifies energy-efficient mappings for neural networks with dense inter-operator parallelism while providing configurable latency–throughput trade-offs.

Abstract

from arXiv · show

As edge-based deep learning applications become more complex, optimizing performance on heterogeneous System-on-Chips (SoCs) presents unique challenges. Traditional pipelining techniques distributing the computation across different on-chip processing units, while effective for throughput, do not address the latency demands posed by modern neural networks with complex interdependencies and extensive operator parallelism. There is a potential in leveraging operator parallelism to enable concurrent execution across multiple processing units, thereby reducing inference latency. However, prioritizing pipelining or parallel execution often necessitates a compromise, where optimizing one performance metric adversely impacts the other. This paper introduces Para-Pipe, a hierarchical mapping framework that integrates intra- and inter-stage operator parallelism within a pipelined architecture. Para-Pipe navigates the trade-off between throughput and latency by selectively fine-tuning parallelism levels within and across pipeline stages. This strategy can significantly reduce inter-processor communication overhead, significantly improving energy efficiency. Our evaluation demonstrates that Para-Pipe generates multiple Pareto-optimal configurations, achieving a balance between throughput and latency on an Amlogic SoC equipped with ARM big.LITTLE CPUs and GPU, as well as the Black Sesame Technology SoC featuring a deep learning accelerator and two DSPs. More importantly, throughput-optimized configurations under Para-Pipe on Amlogic SoC show an average energy efficiency improvement of 11.0% over purely pipelined strategies and 23.3% relative to non-pipelined parallel execution.

I. INTRODUCTION

Para-Pipe targets heterogeneous SoCs where conventional sequential execution, pipelining, or parallel execution alone cannot jointly optimize latency and throughput. It hierarchically combines pipeline and operator parallelism to produce configurable, energy-efficient mappings.

  • Motivation: Modern heterogeneous SoCs contain diverse processors, but common ML frameworks often underuse simultaneous execution across them.Sequentially favoring one compute unit simplifies deployment but misses potential speed and energy benefits from multiple units.
  • Motivation: Modern neural networks expose operator parallelism through parallel branches and attention heads, yet latency-focused execution can undermine pipelined throughput.InceptionNet branches and transformer attention heads provide examples of concurrent operator execution.
  • Para-Pipe: Para-Pipe separates sequential and parallel subgraphs into pipeline stages and tunes intra- and inter-stage parallelism to balance latency and throughput.It also regulates inter-processor communication costs through hierarchical mapping.
  • Evaluation: 11.0% average energy-efficiency improvement over purely pipelined strategies and 23.3% over non-pipelined parallel execution were achieved by throughput-optimized Amlogic configurations.The evaluation used Amlogic and Black Sesame Technology SoCs and produced Pareto-optimal mapping options.
  • Motivating Example: 115% higher latency for pipe-only and 26% lower throughput for para-only illustrate the cost of optimizing one metric in isolation.Hybrid strategies combine pipelining and parallel execution to address this trade-off.
  • Contributions: Two ILP-based mapping algorithms provide distinct granularities for balancing mapping precision and mapping time.The framework was implemented with ARM Compute Library and evaluated on Amlogic and BST SoCs.

II. OVERVIEW

Para-Pipe uses pipeline-level and operator-level mapping tailored to the structured subgraphs of ML computational graphs. Its operator mapper assigns operations to compute engines using ILP-based coarse- and fine-grained strategies, while a cost estimator supports Pareto-optimal selection.

  • Mapping Overview: Para-Pipe has two mapping levels: pipeline-level mapping and operator-level mapping.The approach exploits regular ML graph structures to establish clearer pipeline-stage boundaries than generic DAG mapping.
  • Operator Mapping: The operator mapper assigns stage operations to compute engines to maximize parallelism and reduce latency.It minimizes mapping costs through integer linear programming.
  • Operator Mapping: Coarse-grained mapping treats a branch as the mapping unit, whereas fine-grained mapping assigns individual operators.The two granularities balance mapping generality, precision, and efficiency.
  • Cost Estimation: A cost estimator models computational and communication costs for the ILP formulation.These costs support the mapping optimization problem.
  • Strategy Selection: Para-Pipe evaluates candidate strategies for latency, throughput, and energy efficiency, then selects Pareto-optimal mappings for user requirements.The current process is static and excludes runtime dynamicity before validation on real SoC platforms.

A. Graph Partitioning

Para-Pipe partitions irregular ML graphs into ordered subgraphs and enumerates pipeline configurations before mapping operators within stages. Its ILP formulations assign branches or operators while accounting for execution and communication costs.

  • Graph Partitioning: The graph partitioner creates non-overlapping, topologically ordered subgraphs containing linear chains or fan-in structures.Fan-in structures capture patterns common in InceptionNet and transformer-based models.
  • Graph Partitioning: The partitioning algorithm merges operators where feasible and traces graph outputs to identify subgraphs across irregular structures.It tracks extensible and non-extensible nodes during traversal.
  • Pipeline Configuration: Para-Pipe enumerates configurations from 1 stage for para-only execution to m stages for pipe-only execution across m compute engines.Processor combinations are assigned to distinct stages without overlap.
  • Pipeline Configuration: Candidate pipeline configurations are generated by enumerating partition points and the sequential subgraphs assigned to each stage.The enumeration can produce a pipe-only configuration with nearly maximal throughput.
  • Operator Mapping: Coarse-grained and fine-grained ILP algorithms trade faster, structure-aware solutions against more flexible mappings with longer solving times.Fine-grained mapping can segment branches and distribute operators across CPU clusters.
  • Operator Mapping: The mapper minimizes the longest execution time by combining operator computation and communication costs.Branches are assigned to processors, while unused processors can handle nested branches outside the computationally demanding path.

B. Fine-grained Mapping

The fine-grained mapping formulates operator placement, execution timing, communication costs, and processor availability as an optimization problem. Binary variables and big-M constraints encode assignments and non-overlapping execution, enabling the mapping strategy to minimize graph execution time.

  • The objective minimizes the maximum execution time of all last operators in the graph.
  • Each operator is assigned to exactly one processor using binary assignment variables.
  • Operator execution time combines start-time dependencies with computation costs selected by processor assignment.
  • Communication time is modeled by selecting processor pairs for graph edges and summing their corresponding communication costs.
  • Processor-availability constraints prevent two parallel operators assigned to the same processor from overlapping.
  • Big-M constraints encode the exclusive ordering condition between operators assigned to the same processor.

V. COST ESTIMATION & STRATEGY SELECTION

Para-Pipe estimates performance and energy for mapping options, then identifies Pareto-optimal configurations for user selection. Performance uses stage execution and communication costs, while energy efficiency is evaluated from throughput and active processor power on Amlogic SoC.

  • Para-Pipe selects Pareto-optimal mapping options after pipeline mapping, parallel-operator mapping, and performance and energy estimation.
  • Latency is the cumulative execution time of all stages, while throughput is the reciprocal of the longest stage’s execution time.
  • Amlogic performance costs are profiled offline, including operator computation, data-transition, data-conversion, SDRAM-access, and address-mapping costs.
  • Energy efficiency is measured as inference requests per unit of energy by dividing throughput by total platform active power.
  • Energy efficiency is evaluated exclusively on Amlogic SoC because BST does not provide an energy estimation model.
  • Processor power is estimated with a model combining dynamic and static power, whose platform-specific parameters are refined using utilization measurements and linear regression.

B. Strategy Selection

Para-Pipe estimates and filters mapping configurations to retain Pareto-optimal choices across latency and throughput, while also considering energy efficiency. The resulting options expose trade-offs among pipe-only, para-only, and hybrid strategies.

  • Strategy filtering: Configurations with inferior estimated latency and throughput are excluded, while Pareto-optimal options are retained.The selection process uses estimated performance to remove dominated mappings.
  • Trade-off options: Pipe-only maximizes throughput but incurs impractical latency, whereas para-only minimizes latency at substantially lower throughput.Hybrid configurations provide intermediate trade-offs between these extremes.
  • Balanced options: Para-Pipe provides multiple balanced configurations when latency and throughput receive equal importance.Users can choose among several configurations with moderate performance on both metrics.
  • Estimation accuracy: The cost estimator reports RMSPE values of 15.33% for latency, 15.25% for throughput, and 6.40% for energy efficiency.Despite approximately 15% latency and throughput prediction error, it identifies relative mapping-strategy rankings effectively.
  • Evaluation context: Para-Pipe mapping options are evaluated on Amlogic and BST platforms, including runtime-management implementations and visual comparisons of execution strategies.The supplied figures cover representative mappings, prediction errors, hardware boards, sequential runtime, and coarse- versus fine-grained mapping.

A. Implementation on Amlogic SoC

Para-Pipe is implemented across heterogeneous SoC processing units by assigning graph nodes to processor-specific backends and coordinating concurrent execution. Runtime support spans Amlogic CPUs and GPU as well as BST NPU and DSP resources.

  • Amlogic platform: On Amlogic, Para-Pipe targets Cortex-A73 and Cortex-A53 CPU clusters together with an ARM G52 MP4 GPU.The heterogeneous platform combines high-performance and power-efficient CPU resources with GPU computation.
  • Runtime scheduling: Custom CPU and GPU backends distribute graph nodes according to the mapping strategy, with separate schedulers enabling concurrent operator execution.CPU schedulers use thread pools bound to the corresponding cluster cores.
  • Pipeline execution: Each pipeline stage is represented by an ARM-CL graph and executed on a dedicated thread, with additional threads when multiple engines operate within a stage.Stage outputs are transferred through buffers and notify successor stages.
  • Data movement: CPU-to-GPU transfers require data conversion into OpenCL tensor formats, whereas transfers between CPU clusters require no data conversion.Semaphores manage dependencies during concurrent execution.
  • BST platform: The BST A1000 platform combines an NPU with DSP resources, while the evaluation uses one NPU and two DSPs because two DSPs are reserved for other autonomous-driving functions.Runtime and communication costs are estimated using an operator simulator, historical measurements, and kmeans clustering for unseen settings.

A. Experimental Setup

The experiments assess six modern models and compare Para-Pipe strategies with sequential and DAG-mapping baselines on real and simulated SoC settings. Selected configurations emphasize latency, throughput, and pipeline behavior across heterogeneous processors.

  • Benchmarks: Six models include four Inception-family networks and two networks with complex, irregular connections.The benchmark suite targets dense operator parallelism beyond the linear models studied previously.
  • Amlogic setup: Amlogic measurements use a real Khadas Vim3 Pro platform with CPU and GPU frequencies configured at peak operating levels.The platform includes big and small CPU clusters plus an ARM GPU.
  • Metrics: Throughput is measured in frames per second and latency per frame over a continuous stream of 50 frames, while power measurements yield active power and energy efficiency.The board is cooled between runs to prepare for subsequent tests.
  • Baselines: Layer-switched provides a sequential baseline using all on-chip resources, while HEFT and CPOP provide DAG-mapping comparisons.The selected HEFT or CPOP variant is the one with the best latency for each model.
  • Strategies: Hybrid-L and hybrid-T use dual-stage Amlogic configurations spanning CPU clusters and GPU, with coarse-grained parallel operator mapping within stages.The study also compares pipe-only, para-only, and hybrid strategies targeting latency or throughput.
  • BST configurations: On BST, hybrid-T balances pipeline workloads to reduce the longest-stage delay, whereas hybrid-L accelerates frames through NPU allocation or greater intra-stage parallelism.These configurations are detailed for four representative benchmarks.

C. Latency-Throughput Trade-off

Para-Pipe exposes a latency-throughput trade-off by combining pipeline and operator parallelism, with configurations tailored to different platform constraints and priorities. Hybrid strategies can improve energy efficiency while balancing the extremes of pipe-only and para-only execution.

  • Baseline comparison: Para-only improves over HEFT or CPOP by 10.9% latency and 12.5% throughput on Amlogic, and 15.5% latency and 18.8% throughput on BST.These comparisons use the better-performing HEFT or CPOP baseline.
  • Latency-throughput extremes: On Amlogic, pipe-only increases latency by 113.8% versus para-only, while para-only reduces throughput by 26.6% versus pipe-only.The two configurations represent opposing latency and throughput extremes.
  • Hybrid trade-offs: Relative to pipe-only, hybrid-L reduces throughput by 12.4% and latency by 36.0%, while hybrid-T reduces throughput by 7.3% and latency by 26.8%.For Inception-v4, hybrid-T exceeds pipe-only throughput while halving latency.
  • Platform adaptation: On BST, hybrid-T improves throughput by 16.2% over para-only, while hybrid-L and para-only achieve comparable latency.Hybrid-T uses one DSP for operators unsupported by the NPU alone.
  • Efficiency mechanisms: Para-only improves energy efficiency through optimized workload mapping that reduces synchronization contention and processor idle time.Relative to HEFT & CPOP, it increases active power by 3.3% while reducing mutex contention by 11.7%.
  • Energy efficiency: Hybrid-L and hybrid-T improve energy efficiency over para-only by 16.7% and 23.3%, and over pipe-only by 8.7% and 11.0%, respectively.Inception-v4 is an exception, with hybrid-T reaching 0.53 fps/J versus pipe-only’s 0.54 fps/J.

1) Coarse-grained Mapping:

Para-Pipe’s mapping analysis compares coarse- and fine-grained execution choices across heterogeneous processors, exposing trade-offs among latency, throughput, energy efficiency, and scheduling overhead. Fine-grained mapping improves performance for compatible CPU architectures but incurs implementation and synchronization costs.

  • Coarse-grained mapping: 16.4% lower latency and 19.8% higher throughput result from parallel processing with two CPU clusters versus the big CPU cluster.Adding a GPU provides only marginal improvement for some models because communication overhead offsets co-execution benefits.
  • Coarse-grained mapping: 14.2%, 17.6%, and 31.3% average energy-efficiency gains come from combining the big CPU with the small CPU, GPU, or both, respectively.The gains reflect reduced reliance on the power-intensive big CPU and shorter execution times, although CPU-GPU combinations dilute some GPU power-efficiency benefits.
  • Coarse-grained mapping: Pipe-only and hybrid Para-Pipe configurations preserve GPU power efficiency while compatible CPU clusters improve throughput with minimal latency trade-offs.These choices also reduce unnecessary communication and support sequential processing stages that include the GPU.
  • Fine-grained mapping: 5.9% average program overhead accompanies fine-grained CPU-GPU mapping, while synchronization overhead averages 21.7% with a big CPU and GPU but falls to 4.7% when a small CPU is added.Inception-ResNet-v2 reaches 11.7% program overhead.
  • Fine-grained mapping: 3.35% lower latency and 4.28% higher throughput are achieved by fine-grained mapping over coarse-grained mapping when two CPU clusters co-execute.Fine-grained mapping is especially effective for irregular models and units with similar architectures and data formats.
  • Mapping overhead: Fine-grained mapping requires longer resolution times than coarse-grained mapping, although subgraph-based ILP solutions are generated rapidly for most networks.The largest PETR-based subgraph requires approximately 6 hours of ILP solution time.

VIII. RELATED WORK

The paper situates Para-Pipe among heterogeneous-SoC methods for pipelining, operator parallel execution, intra- and inter-operator parallelism, and multiple workloads. Its contribution is hierarchical operator parallelism within a pipeline, producing Pareto-optimal strategies for latency, throughput, and energy efficiency.

  • Pipelining: Existing pipelining methods primarily target throughput by partitioning models across heterogeneous CPU clusters or other processors.Prior work includes binary search, dynamic programming, and Bayesian optimization for pipeline configuration and partitioning.
  • Operator parallel execution: Prior operator-parallel execution methods use optimization and scheduling techniques such as ant colony optimization, mixed ILP, and DAG ranking.These approaches account for heterogeneous task mapping and, in some cases, processor data-exchange overhead.
  • Intra- and inter-operator parallelism: Earlier intra- and inter-operator approaches combine operator-level CPU pipelines, GPU or NPU acceleration, multithreading, or dynamic work stealing.The related methods include TensorRT-based parallelization and runtime workload remapping on SoCs.
  • Multiple workloads: Prior multiple-workload methods address energy-aware scheduling, DVFS, or dynamic task management, often for simple linear graphs.The reviewed work therefore differs from Para-Pipe’s focus on complex and irregular model architectures.
  • Para-Pipe: Para-Pipe hierarchically tunes intra- and inter-stage operator parallelism within a pipelined architecture to map irregular neural-network graphs.It decomposes graphs into subgraphs and assigns them to parallel or sequential execution in distinct pipeline stages.
  • Para-Pipe: Evaluations on Amlogic and BST SoCs produce Pareto-optimal configurations that balance latency and throughput while identifying energy-efficient mappings for dense inter-operator parallelism.The conclusion frames this balance as Para-Pipe’s central outcome on heterogeneous embedded platforms.
Loading 2609.04168v1…