Source-linked AI summary
DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion
Wei Niu, Jiexiong Guan, Yanzhi Wang, Gagan Agrawal, Bin Ren
TL;DR
DNNFusion addresses restrictive pattern-based and low-level fusion approaches by analyzing DNN operators and their combinations. Its framework expands fusion opportunities and outperforms four state-of-the-art execution frameworks, while enabling efficient mobile execution for many previously unsupported models.
Problem
Existing pattern-based and low-level loop fusion approaches can miss opportunities across diverse DNN operators and layer connections, while deep models require substantial memory and computation.
Method
DNNFusion classifies operators and their combinations, uses an Extended Computational Graph with mathematical-property-based rewriting, and integrates fusion-plan generation with lightweight profiling.
Results
DNNFusion achieves up to 8.8× more loop fusions and 9.3× speedup compared with four state-of-the-art end-to-end DNN execution frameworks across 15 diverse models.
Takeaways & Limitations
DNNFusion enables many cutting-edge models unsupported by prior end-to-end frameworks to execute efficiently on mobile devices, even in real-time.
Takeaways & Limitations
For some remaining cases involving changed temporary output sizes, fusion is not applied and more sophisticated methods are left for future work.
Abstract
from arXiv · showhide
Deep Neural Networks (DNNs) have emerged as the core enabler of many major applications on mobile devices. To achieve high accuracy, DNN models have become increasingly deep with hundreds or even thousands of operator layers, leading to high memory and computational requirements for inference. Operator fusion (or kernel/layer fusion) is key optimization in many state-of-the-art DNN execution frameworks, such as TensorFlow, TVM, and MNN. However, these frameworks usually adopt fusion approaches based on certain patterns that are too restrictive to cover the diversity of operators and layer connections. Polyhedral-based loop fusion techniques, on the other hand, work on a low-level view of the computation without operator-level information, and can also miss potential fusion opportunities. To address this challenge, this paper proposes a novel and extensive loop fusion framework called DNNFusion. The basic idea of this work is to work at an operator view of DNNs, but expand fusion opportunities by developing a classification of both individual operators and their combinations. In addition, DNNFusion includes 1) a novel mathematical-property-based graph rewriting framework to reduce evaluation costs and facilitate subsequent operator fusion, 2) an integrated fusion plan generation that leverages the high-level analysis and accurate light-weight profiling, and 3) additional optimizations during fusion code generation. DNNFusion is extensively evaluated on 15 DNN models with varied types of tasks, model sizes, and layer counts. The evaluation results demonstrate that DNNFusion finds up to 8.8x higher fusion opportunities, outperforms four state-of-the-art DNN execution frameworks with 9.3x speedup. The memory requirement reduction and speedups can enable the execution of many of the target models on mobile devices and even make them part of a real-time application.
1 Introduction
DNNFusion addresses restrictive operator-fusion patterns by combining operator-level classification, extended graph analysis, mathematical graph rewriting, profiling, and optimized code generation. It is evaluated across diverse DNN models and reports substantially more fusion and faster mobile execution than existing frameworks.
- DNNs require substantial computation and memory, creating a deployment trade-off between higher accuracy and resource-constrained mobile execution.
- Existing frameworks use fixed fusion patterns that may not cover diverse operators and layer connections, while low-level loop fusion can miss operator-level opportunities.ONNX lists 167 distinct operators, making exhaustive pattern construction impractical.
- DNNFusion classifies operators by input-output mappings and defines combination rules instead of relying on specific operation patterns.The framework uses mapping types such as One-to-One and One-to-Many to guide fusion decisions.
- DNNFusion extends computational graphs with explicit operation types and properties, then applies mathematical-property-based rewriting to simplify computations and facilitate fusion.Its rewriting rules operate on tensor operations and go beyond traditional scalar strength reduction.
- The framework integrates machine-independent mapping analysis, light-weight profiling, fusion-plan generation, and optimized fused-code generation.Its contributions include ECG-based abstractions, profiling-informed planning, and integration into an end-to-end DNN execution framework.
- 8.8× more loop fusions and 9.3× speedup are achieved across 15 DNN models compared with four state-of-the-art end-to-end frameworks.The evaluation covers five task types, varied model sizes and layer counts, and mobile devices.
2 Blessing and Curse of Deep Layers
Increasing DNN depth can impair mobile execution even when total computation is similar, because deeper models create more intermediate data and insufficient work per layer. Operator fusion is presented as a way to reduce memory pressure and improve efficiency.
- Deep DNNs are difficult to execute efficiently on resource-constrained mobile devices because they require substantial memory and computation.
- 457 layers and 78 GFLOPs/S characterize DistilBERT, while VGG-16 has 51 layers and 320 GFLOPs/S despite similar computation amounts.The comparison supports layer depth as an important execution-efficiency factor.
- More layers usually generate more intermediate results, increasing memory and cache pressure.
- Deeper models often contain too little computation per layer, degrading processor utilization, particularly on GPUs.
- Operator fusion can reduce memory requirements and improve execution efficiency.
3 Classification of DNN Operators and Fusion Opportunity Analysis
DNNFusion classifies operators by input-output mapping relationships and analyzes operator-type combinations to determine fusion legality, profitability, and resulting mappings.
- DNN Operators Classification: DNNFusion classifies operators into five abstract mapping types based on relationships between input and output elements.The types are One-to-One, Reorganize, Shuffle, One-to-Many, and Many-to-Many, ordered by increasing complexity.
- Fusion Opportunity Analysis: Mapping type analysis predicts the resulting fused operator type and simplifies profitability evaluation and correct fusion implementation.The analysis uses the mapping types of two candidate operators to infer their fused mapping.
- Fusion Opportunity Analysis: Green combinations are directly profitable, red combinations are illegal or unprofitable, and yellow combinations require profiling.This classification avoids runtime analysis or autotuning for green and red cases.
- Fusion Opportunity Analysis: Transformation impedance qualitatively measures fusion difficulty: One-to-One has the lowest impedance, while One-to-Many and Many-to-Many have the strongest.The resulting mapping type is increasingly determined by higher-impedance operators during fusion.
- Fusion Opportunity Analysis: One-to-One operators can fuse with any operator type when their element mapping functions are known, enabling access mapping from downstream outputs to inputs.Tensor-based DNN operators restrict dependencies and expose mapping information that supports this analysis.
4.1 Overview of DNNFusion
DNNFusion extends a compiler-generated computational graph with operator information, then combines graph rewriting, profile-driven fusion planning, and fused-code generation.
- Overview: DNNFusion takes a computational graph from compiler-based DNN frameworks and augments it into an Extended Computational Graph.The ECG provides the basis for subsequent optimization and code generation.
- Overview: The overview connects the Extended Computational Graph to the framework’s fusion planning and code-generation stages.Figure 1 summarizes this end-to-end organization.
- Overview: Its optimization pipeline contains mathematical-property-based graph rewriting, lightweight profile-driven fusion-plan exploration, and fusion code generation with additional optimizations.These three components form the main compiler optimization and code-generation stage.
4.2 Mathematical-Property-Based Graph Rewriting
DNNFusion rewrites tensor computation graphs using mathematical properties to reduce unnecessary work and prepare graphs for more effective operator fusion.
- Graph Rewriting: 18% fewer fused layers remained after graph rewriting on GPT-2.The rewriting pass removes unnecessary operations, eliminates redundant intermediate copies, and replaces costly operator combinations with more efficient ones.
- Mathematical Properties: Distributive rewriting simplifies common operator combinations, while commutative rewriting reorders operators to reduce overall computation.Figure 2 illustrates these three mathematical-property-based transformations.
- Rewriting Rules: DNNFusion derives 45 Associative, 38 Distributive, and 66 Communicative graph-rewriting rules.Table 4 lists representative rules and focuses mainly on One-to-One and selected Many-to-Many reduction operators.
- Rewriting Metric: Graph rewriting primarily uses #FLOPs as its optimization metric because temporary output sizes usually remain unchanged or are addressed by subsequent fusion.Cases where temporary output size changes and fusion is not applied require more sophisticated future methods.
- Mathematical Properties: Associative rewriting changes execution order to replace expensive operator combinations with cheaper ones and reduce intermediate results.One example replaces two Recip and two Mul operators with Recip, Square, and Mul, eliminating a Mul operator.
- Pattern Matching: DNNFusion partitions the graph before pattern matching because associative and commutative matching is NP-complete.Within each resulting subgraph, it explores possible patterns and pattern combinations.
4.3 Light-Weight Profile-Driven Fusion Plan Exploration
DNNFusion generates fusion plans by selecting promising One-to-One seed operators, then recursively exploring successors and predecessors using mapping analysis, constraints, and lightweight profiling. This approach controls the NP-complete search while expanding candidate fusion blocks.
- Fusion-plan generation uses a greedy approach over the Extended Computational Graph and operation mapping types to control an NP-complete search space.The method is designed to keep fusion-plan exploration computationally manageable.
- Step I: Fusion seed operator selection: One-to-One operators with the minimum intermediate-result size are selected as fusion seeds because they can enable broader fusion with lower memory and register requirements.The algorithm identifies One-to-One operators and chooses the minimum intermediate-result size as the seed heuristic.
- Steps II–III: Propagated exploration: Starting from each seed, DNNFusion recursively explores successors and predecessors, considering fusion candidates until mapping or constraint checks stop expansion.Predecessor exploration can fuse some, but not necessarily all, immediate predecessors when multiple predecessors exist.
- Candidate analysis: Mapping analysis classifies candidate combinations as fuse_break, fuse_through, or fuse_depend, with the last case requiring a profile-data lookup.Constraint checks can reject candidates likely to incur excessive overheads such as register spills.
- Illustrative fusion plan: In the illustrative graph, successor exploration produces a five-operator Many-to-One fusion block containing Add, Conv, Relu, Mul, and Sub.The example assumes identical output shapes and intermediate-result sizes for the operators.
4.4 Fusion Code Generation and Optimizations
DNNFusion constructs fused code from a data-flow tree and predefined rules, targeting mobile CPUs and GPUs. It also applies intra- and inter-fusion-block optimizations before and after code generation.
- Data-flow-tree construction: DNNFusion builds a data-flow tree from the Extended Computational Graph to represent outputs, intermediate results, and inputs for fused-code generation.The tree reverses the ECG edge direction so parent nodes depend on child nodes during traversal.
- Rule-based fusion: Predefined code-generation rules recursively fuse operator pairs while preserving mapping-type information as combinations such as One-to-One plus One-to-One or Many-to-One.The generated code targets C++ for mobile CPUs and OpenCL for mobile GPUs.
- Mapping-aware code generation: For Shuffle and One-to-One combinations, DNNFusion permutes loops before fusion; it then maps Many-to-Many outputs to destinations determined by the resulting Shuffle operator.The example combines GEMM, Div, and Transpose through successive mapping-aware fusion steps.
- Post-fusion optimizations: Additional optimizations operate within fusion blocks on the ECG and across blocks on generated code, including data movement, layout, vectorization, unrolling, tiling, and memory/register optimizations.The framework also supports auto-tuning of these optimizations through PatDNN.
5 Evaluation
DNNFusion is evaluated across diverse DNN models, tasks, devices, and competing frameworks. It achieves higher fusion rates and lower execution latency, with especially strong benefits for transformer-based and extremely deep models.
- Evaluation setup: DNNFusion is evaluated on 15 mainstream DNN models spanning image classification, detection, action recognition, and language tasks with varied sizes and layer counts.The study compares fusion rates and execution latency across mobile CPU and GPU settings.
- Model coverage: DNNFusion is the only evaluated end-to-end framework supporting all target models on both mobile CPU and mobile GPU.A dash in the evaluation tables denotes unsupported models for a framework.
- Fusion rate: Fusion rates improve by 1.3×–2.9× over MNN, 1.3×–8.1× over TVM, 1.3×–8.8× over TFLite, and 1.3×–2.8× over Pytorch.R-CNN and transformer-based models obtain 3.9×–10.0× fusion rates over original models, compared with 1.7×–3.6× for 2D/3D CNNs.
- Execution latency: On mobile CPU, DNNFusion achieves speedups of 1.4×–2.6× over MNN, 1.5×–3.5× over TVM, 1.4×–3.3× over TFLite, and 1.6×–9.3× over Pytorch.On mobile GPU, speedups reach 1.7×–2.6× over MNN, 2.0×–3.1× over TVM, and 1.6×–4.0× over TFLite.
- Comparison with TASO: Compared with TASO-optimized graphs executed on TFLite, DNNFusion delivers 1.4×–2.6× speedup on mobile CPU.The paper attributes the difference to graph rewriting designed specifically to work with operator fusion.
- Optimization breakdown: Graph rewriting, fusion, and other optimizations contribute additional CPU speedups of 1.2×–1.5×, 1.6×–2.2×, and 1.3×–1.8×, respectively.On mobile GPU, the corresponding ranges are 1.3×–1.5×, 2.1×–3.3×, and 1.7×–2.1×.
- Memory and utilization: DNNFusion reduces memory accesses and consumption on YOLO-V4 and achieves the highest reported CPU and GPU utilization among the compared frameworks.The paper links these outcomes to reduced intermediate-result materialization and more coarsely grained execution.
- Compilation time: TVM tuning dominates YOLO-V4 compilation on mobile CPU, lasting around four hours, whereas DNNFusion reports a separate compilation-time comparison.The supplied evaluation passage does not state DNNFusion’s exact compilation duration.
6 Related Work
Related work spans fixed-pattern operator fusion, polyhedral loop-fusion methods, and other fusion-based machine-learning frameworks, with DNNFusion differing in abstraction, targets, and techniques.
- Operator fusion in end-to-end mobile DNN frameworks: Mobile DNN frameworks such as MNN, TVM, TensorFlow-Lite, and Pytorch use fixed-pattern fusion that may miss diverse operator and layer connections.Examples include unsupported ConvTranspose + ReLU + Concat in TVM, while DNNFusion handles additional combinations in GPT-2 and TinyBERT.
- Operator fusion in end-to-end mobile DNN frameworks: DNNFusion classifies both operators and their combinations to enable a larger set of optimizations than these frameworks.
- Other operator-fusion frameworks: Other systems, including Rammer, Cortex, and TensorFlow XLA, apply distinct fusion techniques or target different machine-learning optimization settings.
- Polyhedral-based and other loop fusion methods: Polyhedral analysis provides a general and rigorous foundation for loop transformation and optimization, including optimized loop fusion.
- Polyhedral-based and other loop fusion methods: Polyhedral approaches do not directly address DNNFusion’s problem of selecting fusion plans amid large numbers of operators in target models.
7 Conclusions and Future Work
DNNFusion combines high-level operator abstractions, mathematical-property-based graph rewriting, and integrated fusion planning, and is evaluated across diverse mobile DNN workloads. The reported results include up to 8.8× speedup over four state-of-the-art frameworks and future integration with model pruning.
- Conclusions: DNNFusion combines mapping-type abstractions, the Extended Computational Graph, mathematical-property-based graph rewriting, and integrated fusion-plan generation.
- Conclusions: 15 diverse DNN models evaluated on multiple mobile devices showed up to 8.8× speedup over four state-of-the-art DNN execution frameworks.
- Conclusions: DNNFusion enabled many cutting-edge models previously unsupported by end-to-end frameworks to execute efficiently on mobile devices, including in real time.The paper also reports improved cache performance, device utilization, restricted-resource execution, and reduced compilation tuning time.
- Future work: Future work will combine DNNFusion with model pruning to pursue better performance than either approach alone.The authors note that dense models with fusion can outperform pruned efforts by having fewer layers.