Source-linked AI summary
The Deep Learning Compiler: A Comprehensive Survey
Mingzhen Li, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Lin Gan, Guangwen Yang, Depei Qian
TL;DR
Deploying diverse DL models across heterogeneous hardware is difficult, and existing surveys had not comprehensively analyzed DL compiler architecture. This paper surveys compiler frontends, multi-level IRs, and backends, compares existing systems, and reports performance evaluations. It also identifies research directions including dynamic shapes, auto-tuning, unified optimization, and privacy protection.
Problem
Deploying diverse DL models efficiently on heterogeneous hardware is difficult, while prior surveys had not comprehensively analyzed DL compiler design architecture.
Method
The paper dissects existing DL compilers into frontend, multi-level IR, and backend components, then analyzes their designs, optimizations, taxonomy, and performance.
Results
The survey provides a comprehensive comparison and analysis of DL compiler architectures, IR designs, optimization techniques, and CNN performance.
Takeaways & Limitations
Future DL compiler research should address dynamic shapes, advanced auto-tuning, unified optimization, subgraph partitioning, quantization, differentiable programming, and privacy protection.
Takeaways & Limitations
Frontend and backend optimizations are tightly coupled in existing DL compilers, making their individual effectiveness difficult to measure accurately.
Abstract
from arXiv · showhide
The difficulty of deploying various deep learning (DL) models on diverse DL hardware has boosted the research and development of DL compilers in the community. Several DL compilers have been proposed from both industry and academia such as Tensorflow XLA and TVM. Similarly, the DL compilers take the DL models described in different DL frameworks as input, and then generate optimized codes for diverse DL hardware as output. However, none of the existing survey has analyzed the unique design architecture of the DL compilers comprehensively. In this paper, we perform a comprehensive survey of existing DL compilers by dissecting the commonly adopted design in details, with emphasis on the DL oriented multi-level IRs, and frontend/backend optimizations. Specifically, we provide a comprehensive comparison among existing DL compilers from various aspects. In addition, we present detailed analysis on the design of multi-level IRs and illustrate the commonly adopted optimization techniques. Finally, several insights are highlighted as the potential research directions of DL compiler. This is the first survey paper focusing on the design architecture of DL compilers, which we hope can pave the road for future research towards DL compiler.
1 INTRODUCTION
DL compilers address the difficulty of deploying diverse models across heterogeneous hardware by translating framework models into optimized hardware code. This survey dissects their architecture, compares existing systems, evaluates performance, and identifies future research directions.
- Motivation: Diverse DL models and hardware make programming and efficient deployment increasingly difficult.The paper motivates compiler support as models such as CNNs, RNNs, LSTMs, and GANs proliferate across applications.
- Motivation: DL compilers translate model definitions from DL frameworks into efficient implementations for various DL hardware targets.They are presented as a way to reduce the burden of manually optimizing models for each hardware platform.
- Survey scope: The survey analyzes DL compiler frontends, multi-level IRs, and backends, emphasizing IR design and frontend/backend optimization methods.Its architecture-oriented analysis covers graph transformations, IRs, hardware-specific optimization, auto-tuning, and optimized kernel libraries.
- Contributions: The paper provides a taxonomy of existing DL compilers to guide practitioner selection and summarize compiler research.The taxonomy is organized around the key design components analyzed in the survey.
- Evaluation: The evaluation compares end-to-end and per-layer performance on full-fledged and lightweight CNN models, with scripts and results open sourced.Convolution layers receive per-layer analysis because they dominate inference time.
- Future directions: The survey highlights dynamic shapes, pre-/post-processing, advanced auto-tuning, polyhedral models, subgraph partitioning, quantization, unified optimization, differentiable programming, and privacy protection as future directions.These directions are proposed as opportunities for continued DL compiler research.
2 BACKGROUND
DL frameworks simplify model development but differ in programming paradigms, interfaces, and hardware support, while ONNX improves interoperability. DL hardware spans general-purpose, dedicated, and neuromorphic categories, motivating specialized code generation.
- Deep learning frameworks: Popular DL frameworks include TensorFlow, PyTorch, MXNet, and CNTK, each exposing different design tradeoffs and interfaces.The background also discusses Keras, Caffe/Caffe2, PaddlePaddle, and historical frameworks.
- Deep learning frameworks: ONNX defines a scalable computation-graph format that enables model conversion between frameworks.It integrates directly with several frameworks and provides converters for frameworks such as TensorFlow and Keras.
- Deep learning frameworks: Historical frameworks became inactive or were replaced as the DL ecosystem evolved.Examples include Torch being replaced by PyTorch, Theano no longer being maintained, and Chainer being replaced by later systems.
- Deep learning frameworks: Prior studies compare DL framework performance, whereas this survey focuses on DL compilers as a general approach for executing models across diverse hardware.The distinction concerns the object of study rather than a direct performance result.
- DL hardware: DL hardware is categorized as general-purpose, dedicated, or neuromorphic according to its generality.General-purpose processors use hardware and software optimization, dedicated hardware uses customized circuits, and neuromorphic hardware mimics the human brain.
- Hardware-specific DL code generators: FPGA code generators target processor-like or streaming architectures with different scheduling and resource assumptions.Processor-like designs use software-controlled static scheduling, while streaming designs pipeline layer-specific hardware blocks but may require sufficient on-chip resources.
3 COMMON DESIGN ARCHITECTURE OF DL COMPILERS
A common DL compiler architecture separates frontend and backend responsibilities around multi-level IRs. The frontend performs hardware-independent graph processing, while the backend lowers IR and applies hardware-specific code-generation optimizations.
- Architecture overview: DL compilers primarily contain a frontend and backend, with IR spanning both components at multiple abstraction levels.High-level IR resides in the frontend and low-level IR in the backend.
- Intermediate representations: High-level IR represents computation and control flow independently of hardware and supports graph-level optimization.It establishes operator and data dependencies while preserving semantic information for transformations.
- Intermediate representations: Low-level IR supports hardware-specific optimization and code generation by expressing fine-grained hardware characteristics.It can also accommodate mature toolchains such as Halide, polyhedral models, and LLVM.
- Compiler frontend: The frontend converts framework models into computation graphs and applies format conversion plus hardware-independent graph optimizations.These optimizations combine general-purpose compiler techniques with DL-specific transformations.
- Compiler backend: The backend lowers high-level IR to low-level IR or third-party toolchains and applies customized optimizations using DL and hardware knowledge.This supports more efficient code generation than relying only on general-purpose compiler infrastructure.
4.1 High-level IR
High-level IRs, or graph IRs, express DL computation and control flow while remaining flexible across models and hardware backends. Existing systems use DAG, let-binding, tensor, shape, layout, and operator representations to support analysis and optimization.
- Graph IR role: High-level graph IR abstracts complex DL computation and control flow while providing an interface for graph-level optimization.Its design must capture diverse models and establish dependencies among operators and data.
- Graph IR representation: DAG-based IRs represent operators as nodes and tensors as edges, but plain IR can suffer semantic ambiguity when computation scope is unspecified.Let-binding addresses this issue by assigning expressions restricted scopes and maintaining variable-result mappings; TVM Relay combines both approaches.
- Tensor computation: Tensor computations are represented through function-based forms, lambda expressions, or Einstein notation.These alternatives trade encapsulation, concise index-based specification, and notation simplicity against constraints such as associativity and commutativity for reductions.
- Data representation: Tensor data may use explicit or unknown shapes, layouts, and bound inference to support symbolic computation and optimization.Placeholders separate computation definitions from concrete values, while dynamic shapes support models with unknown dimensions.
- Operators supported: Graph IR operators cover algebraic, neural-network, tensor, broadcast, reduction, and control-flow workloads.Broadcast relaxes shape constraints, while control flow supports recurrent and data-dependent models such as RNNs and reinforcement learning.
- Common design philosophies: Most DL compilers use hardware-independent high-level IRs with DAG or let-binding structures and flexible tensor-computation representations.These IRs support diverse models and can target different hardware backends.
4.2 Low-level IR
Low-level IRs provide fine-grained representations that expose computation and memory-access details for hardware-specific optimization and code generation. Existing designs mainly use Halide-based, polyhedral-based, or customized IRs, often lowering eventually to LLVM IR.
- Low-level IR represents DL computations more finely than high-level IR, enabling target-dependent tuning of computation and memory access.
- Halide-based IR: Halide-based IR separates computation from scheduling, supporting parameterized schedules and tuning for regular DL computations but not complicated non-rectangular patterns.DL compilers must modify Halide’s original representation to handle exact tensor shapes.
- Polyhedral-based IR: Polyhedral-based IR uses affine transformations and linear programming for deeply nested loops, enabling fusion, tiling, sinking, and mapping.Its flexible polyhedral boundaries support generic patterns but hinder integration with tuning mechanisms.
- Hybrid IR designs: TC combines Halide-based computation representation with polyhedral loop structures, using specialized nodes to describe index domains, hardware-related variables, and code-generation instructions.
- Polyhedral-based IR: PlaidML’s Stripe IR hierarchically organizes parallelizable polyhedral blocks and maps nested polyhedrons to nested memory units.Hardware configuration is separated from kernel code, while tiles are sized to fit local hardware resources.
- LLVM integration: Most low-level IRs eventually lower to LLVM IR, while DL compilers add upper-IR loop transformations or hardware information to avoid poor direct LLVM code generation.Frontend-oriented compilers emphasize upper-IR transformations, whereas backend-oriented systems emphasize target information.
4.3 Frontend Optimizations
Frontend optimizations operate on computation graphs before backend implementation, using global shape and graph information to perform hardware-independent transformations. They span node-level, block-level, and dataflow-level techniques, including algebraic simplification, fusion, memory planning, and layout handling.
- Frontend graph optimizations use a global computation view and remain hardware-independent because they transform the graph rather than backend implementations.
- Frontend passes capture graph features, rewrite graphs, and can use known tensor shapes to guide optimization.Developers may define customized passes in addition to predefined passes.
- The survey classifies frontend optimizations into node-level, block-level, and dataflow-level categories.
- Node-level optimizations: Node-level optimizations eliminate unnecessary nodes or replace them with lower-cost alternatives, including no-op and zero-dimension-tensor cases.Examples include removing single-input sums, zero-padding nodes, and operations on empty tensors.
- Block-level optimizations: Block-level optimizations simplify operator sequences through algebraic identities, strength reduction, constant folding, sinking, and operator fusion.Fusion can share computation, remove intermediate allocations, combine loop nests, and reduce launch and synchronization overhead.
- Dataflow-level optimizations: Dataflow-level optimizations eliminate repeated computations and unused code, while static memory planning reuses buffers to reduce memory requirements.Memory-aware scheduling targets peak activation footprint on edge devices.
- Layout transformation: Layout transformation selects tensor layouts and inserts conversion nodes, but the backend performs the actual transformations because layouts and conversion costs affect hardware performance.CPU-oriented TVM work globally explores NCHW[x]c channel splits through auto-tuning using hardware details.
4.4 Backend Optimizations
Backend optimizations generate hardware-specific code through target transformations, scheduling, memory management, intrinsic mapping, and optimized libraries. Auto-tuning searches large configuration spaces, but library calls can restrict cross-operator optimization.
- DL compiler backends combine hardware-specific optimizations, auto-tuning, and optimized kernel libraries to generate efficient code for diverse targets.
- Hardware-specific optimization: Hardware-specific optimization either lowers low-level IR through LLVM or applies customized transformations using DL and target-hardware knowledge.The survey presents five widely adopted approaches because target-specific techniques cannot be covered exhaustively.
- Memory allocation and fetching: Memory-scope scheduling allocates shared or thread-local GPU memory, adds cooperative fetching and barriers, and supports accelerator buffering.
- Hardware-specific optimization: Backend techniques include loop fusion, sliding windows, tiling, loop reordering, loop unrolling, parallelization, intrinsic mapping, and memory allocation.
- Loop-oriented optimizations: Sliding windows trade parallelism for data reuse by computing values when needed and retaining them only while required.
- Loop-oriented optimizations: Loop reordering improves memory access and spatial locality but is unsafe when dependencies follow the iteration order.
- Auto-tuning: Auto-tuning addresses enormous hardware-optimization search spaces through parameterization, cost models, search techniques, and acceleration.TVM, TC, and XLA support auto-tuning; their cost models include black-box, machine-learning-based, and predefined approaches.
- Optimized kernel libraries: Optimized kernel libraries can improve performance for matching primitives but may constrain control flow and prevent cross-operator optimizations such as fusion.
5 TAXONOMY OF DL COMPILERS
The survey’s taxonomy organizes selected DL compilers by frontend, backend, IR, and optimization features. It is intended to support compiler selection by practitioners and provide researchers with a systematic design summary.
- The survey examines TVM, nGraph, Tensor Comprehension, Glow, and XLA because they are well-known, maintained, and widely used.
- The taxonomy covers frontend, backend, IR, and optimization perspectives corresponding to the survey’s key design components.
- Compiler descriptions include feature support, programming interfaces, development status, and reasons particular features are unsupported.
- Table 1 summarizes framework support, training and quantization support, compilation methods, and supported devices.These features are presented because they affect compiler usage in particular scenarios.
- Table 1 and Figure 2 provide a systematic summary for identifying supported features and compiler components.
6 EVALUATION
The evaluation compares DL compilers through end-to-end and convolution-layer performance across CPUs and GPUs, while documenting compatibility constraints and measurement challenges. Results show compiler performance depends strongly on hardware, model type, tuning, and optimization coupling.
- Experimental Setup: 19 ONNX neural-network models, spanning full-fledged and lightweight CNNs, were used to evaluate TVM, nGraph, TC, Glow, and XLA on CPU and GPU.End-to-end comparisons used TVM, nGraph, Glow, and XLA; TC was omitted because it requires manually implementing all model layers and supports only GPUs.
- Compatibility: nGraph and XLA exhibited ONNX compatibility problems, including nGraph failures on models with dynamic-shape tensors and XLA replacement with TensorFlow Hub models.With TensorFlow Hub models, XLA became two orders of magnitude faster and competitive with other compilers.
- End-to-end Performance: On CPU, Glow performed worse because it lacked thread parallelism, whereas TVM, nGraph, and XLA could leverage all CPU cores.The evaluation therefore identified thread parallelism as a key difference in CPU utilization.
- End-to-end Performance: On CPU, TVM and nGraph generally outperformed other compilers; TVM matched nGraph on full-fledged models and surpassed it on lightweight models.nGraph benefited from offloading optimized subgraphs to DNNL and its Intel-CPU instruction-level JIT optimizations.
- End-to-end Performance: Tuned TVM, using 200 trials, almost achieved the best performance across CPU and GPU models, with a 41.26× average speedup over untuned TVM on GPU.The larger GPU effect was attributed to its more complicated thread and memory hierarchy, which requires finer-grained scheduling.
- Per-layer Performance: Per-layer results favored nGraph on CPU and TVM on GPU, while Glow lagged on selected 1×1 and depth-wise separable convolutions.TVM’s MobileNetV2_1.0 performance was unstable on some layers because autotuning could derive imprecise or negative scheduling parameters under interference from other processes.
7 CONCLUSION AND FUTURE DIRECTIONS
The survey identifies unresolved challenges and research directions for DL compilers, spanning dynamic models, quantization, unified optimization, auto-tuning, polyhedral compilation, and subgraph partitioning. It also connects compiler design to privacy protection, differentiable programming, and compiler–hardware co-design.
- Dynamic shape and pre/post processing: Dynamic shapes and complex pre/post-processing remain difficult because runtime-unknown inputs and Python execution can limit DL compiler efficiency.Supporting complete control flow could allow compilers to express and optimize pre/post-processing with DL models.
- Advanced auto-tuning: Advanced auto-tuning should optimize combinations of operators and account for memory footprint and energy consumption, not only execution time.ML-based tuning may also select compiler options and schedules, while feature engineering could incorporate domain knowledge.
- Polyhedral model: Combining polyhedral models with auto-tuning could reduce tuning search and compilation overhead, but sparse tensors create non-affine indexing challenges.Polyhedral scheduling can reduce the search space, whereas auto-tuning can reuse configurations to reduce JIT compilation overhead.
- Subgraph partitioning: Subgraph partitioning can integrate optimized graph libraries and distribute subgraphs across heterogeneous hardware units for parallel execution.The survey cites nGraph–DNNL integration and heterogeneous edge devices containing CPUs, GPUs, DSPs, and NPUs as examples.
- Quantization: Quantization requires reusable operator implementations and coordination with graph optimizations such as operator fusion.The survey identifies choosing the quantization stage and implementing new quantized operators without heavy engineering as open challenges.
- Unified optimizations: Unified optimization infrastructure could enable reuse of compiler optimizations across DL compilers and support efficient compiler–hardware co-design.Google MLIR provides multi-level IR infrastructure and flexible dialects for transformations across high- and low-level IRs.
- Other directions: Further directions include compiler support for privacy protection and differentiable programming, both of which involve difficult placement or semantic-integration problems.Privacy methods must determine where to insert noise, while differentiable programming requires handling differences in control flow, operation abstractions, and differentiation semantics.