Source-linked AI summary
Glow: Graph Lowering Compiler Techniques for Neural Networks
Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein, Jack Montgomery, Bert Maher, Satish Nadathur, Jakob Olesen, Jongsoo Park, Artem Rakhov, Misha Smelyanskiy, Man Wang
TL;DR
Neural-network compilers must efficiently target heterogeneous hardware while handling expanding architectures and operators. Glow addresses this by progressively lowering neural-network graphs through strongly typed intermediate representations and backend-specific code generation. The design supports multiple targets, with reported speedups over TensorFlow and TVM and commitments from several hardware companies.
Problem
Neural-network frameworks face a scalability challenge because new operators and architectures require repeated implementations across supported targets.
Method
Glow progressively lowers neural-network graphs through target-independent and target-specific strongly typed intermediate representations for optimization and code generation.
Results
Glow is up to 2.7x faster than TensorFlow and up to 1.3x faster than TVM in the reported evaluation.
Takeaways & Limitations
Glow provides a compiler toolkit intended to automate instruction selection, memory allocation, and graph scheduling across diverse hardware backends.
Takeaways & Limitations
Glow’s strict type system requires frameworks to specialize computations before graph construction, potentially generating separate graphs for different batch sizes.
Abstract
from arXiv · showhide
This paper presents the design of Glow, a machine learning compiler for heterogeneous hardware. It is a pragmatic approach to compilation that enables the generation of highly optimized code for multiple targets. Glow lowers the traditional neural network dataflow graph into a two-phase strongly-typed intermediate representation. The high-level intermediate representation allows the optimizer to perform domain-specific optimizations. The lower-level instruction-based address-only intermediate representation allows the compiler to perform memory-related optimizations, such as instruction scheduling, static memory allocation and copy elimination. At the lowest level, the optimizer performs machine-specific code generation to take advantage of specialized hardware features. Glow features a lowering phase which enables the compiler to support a high number of input operators as well as a large number of hardware targets by eliminating the need to implement all operators on all targets. The lowering phase is designed to reduce the input space and allow new hardware backends to focus on a small number of linear algebra primitives.
1 Introduction
Glow is an open-source compiler framework designed to optimize neural networks for heterogeneous hardware. It lowers graph-based models into a software layer that automates compilation tasks across diverse accelerators.
- Motivation: Glow provides compiler techniques for executing neural networks efficiently on domain-specific architectures.The paper describes these techniques as implemented in an open-source framework for heterogeneous hardware.
- Motivation: Traditional node-by-node execution is inefficient, motivating compiler-based execution of neural-network graphs.The paper connects this need to energy efficiency in data centers and mobile devices and to growing architectural diversity.
- Glow’s role: Glow lowers the framework-level graph into a low-level graph and code generator without replacing the machine-learning high-level graph.The low-level graph serves a role analogous to a compiler intermediate representation beneath an abstract syntax tree.
- Glow’s role: Glow automates instruction selection, memory allocation, and graph scheduling so hardware developers can focus on accelerator implementation.The compiler toolkit is open-source and publicly available.
- Adoption: Five companies committed to supporting Glow in future silicon products, whose accelerators are expected to differ in capabilities.The cited companies are Cadence, Esperanto, Habana, Intel, and Qualcomm Technologies.
2 Related Work
Neural-network frameworks provide useful modeling interfaces, but supporting new operators and architectures requires repeated implementations. Related compiler systems address this through graph lowering, intermediate representations, backend libraries, or search-based code generation.
- Framework scalability: Adding an operator requires implementation on each supported architecture, while adding an architecture requires implementing all operators for it.Glow is designed to optimize and generate code for diverse backends more scalably.
- Framework interoperability: ONNX provides an open-source format for representing and serializing AI models across different neural-network frameworks.It supports conversion of compute graphs between frameworks such as PyTorch and CNTK.
- Compiler systems: XLA lowers neural-network nodes into primitive linear-algebra operations and uses backend-specific libraries for different targets.Its backends include Eigen for CPUs and cuDNN for GPUs.
- Compiler systems: Figure 1 highlights that compilers struggle when two loops originate from different dataflow-graph nodes.This illustrates a challenge in analyzing and optimizing graph-derived code.
- Compiler systems: TVM/NNVM lowers nodes into a Halide-based low-level representation for loop optimization before generating LLVM or accelerator source code.DLVM instead lowers its IR into LLVM IR, using LLVM’s optimizer and code generator.
- Compiler systems: nGraph represents framework compute graphs in a single-level IR before lowering them to backends such as cuDNN and MKL-DNN.This contrasts with systems using multiple IR levels.
- Compiler systems: Tensor Comprehensions lets developers specify networks so a JIT compiler can search for efficient execution plans and generate backend-specific code.It targets programmers creating operators that may not already exist.
- Compiler systems: Glow uses multiple levels of its own IR, with tensors represented as first-class values carrying shape and element type.Backends may perform additional lowering, while Glow’s CPU backend uses low-level instructions and its own standard library.
3 Intermediate Representation
Glow uses multiple strongly typed intermediate representations to optimize neural-network graphs at different abstraction levels. The high-level IR supports tensor- and operation-level reasoning, while the low-level IR exposes addresses for memory optimizations and enables retargetable code generation.
- 3.1 Motivation: The high-level IR enables domain-specific optimization because tensors and operations remain explicit rather than being immediately translated into low-level loops.Glow argues that general-purpose compiler analyses struggle to recover neural-network structure from low-level code.
- 3.1 Motivation: The compiler is retargetable: its first two IR levels are shared across targets, while backends may add more target-specific representations near instruction selection.This separates target-independent optimization from increasingly hardware-specific code generation.
- 3.2 High-Level IR: Glow’s high-level IR is a strongly typed, node-based dataflow graph whose tensor shapes and element types are verified by the compiler.It represents neural-network operations such as convolution and matrix multiplication while allowing graph transformations and compile-time tensor handling.
- 3.2 High-Level IR: Glow distinguishes Constant nodes, which the optimizer can inspect and transform, from Placeholder nodes, whose backing tensors may change after compilation.Inputs and outputs are modeled as Placeholders, while compile-time weights can undergo transformations such as quantization and constant propagation.
- 3.4 Node Lowering: Glow lowers high-level operators into smaller linear-algebra operators, so backends can implement primitives such as matrix multiplication instead of every high-level opcode.For example, a FullyConnected layer becomes matrix multiplication followed by broadcasted addition.
- 3.5 Low-Level IR: The low-level IR is instruction-based and address-oriented, enabling memory optimizations such as in-place buffer transformations that are unavailable when memory is implicit.These optimizations include transformations for element-wise arithmetic and rely on direct tensor-address representation.
4 Quantization
Glow converts floating-point networks to signed 8-bit integer networks using profile-guided ranges and typed quantized tensors. It then optimizes the quantized graph to reduce conversions, eliminate rescaling, and enable efficient integer operations.
- Profile-Guided Quantization: Glow converts floating-point-based networks into signed 8-bit integer networks using profile-guided quantization.Inference profiling estimates numeric ranges for each network stage; training-based quantization is future work.
- Tensor Representation: Quantized tensor types combine an Int8 element type with scale and offset fields describing the represented numeric range.Glow uses these fields to convert values from the integer range [-128..127] to floating point.
- Profile-Guided Quantization: Per-stage ranges are necessary because network values can span substantially different numeric scales.A single network-wide scale can be imprecise for small values or truncate large values.
- Profile-Guided Quantization: Profile-guided conversion instruments the network, records activation ranges during inference, and recompiles it into a statically optimizable quantized form.The process uses profiling nodes in the first compilation and profile information during recompilation.
- Quantized Graph Optimizations: Quantization optimizations minimize floating-point/integer conversions, fold rescale nodes into producing operations, and normalize max operands to a shared scale.Shared scales allow hardware to perform simple comparisons for quantized max operations.
5 CPU Backend
Glow’s CPU backend lowers and specializes networks into optimized x86 and ARM64 code. It combines target-specific convolution layouts and tiling with memory planning and operator stacking to reduce execution overhead.
- Backend and Code Generation: The CPU backend compiles Glow’s low-level IR into optimized instructions using LLVM for x86 and ARM64 targets.It can emit stand-alone object files or execute code just in time.
- Standard Library: Glow specializes CPU operator implementations using compilation-time tensor dimensions and buffer addresses.Its compiler-linked standard library lets LLVM optimize implementations for the specific context.
- ResNet50 Optimization: A five-dimensional convolution-filter layout enables consecutive SIMD memory access, with tiling selected according to the processor’s first-level cache.The backend replaces selected convolutions with this specialized implementation.
- Operator Stacking: Operator stacking executes consecutive data-parallel operations on the same memory location instead of repeatedly traversing the buffer.This reduces repeated memory loads and cache invalidation; LLVM cannot perform the optimization alone.
- Operator Stacking: Glow’s stacking automatically creates fast kernels for all permutations of consecutive data-parallel nodes without requiring backend-specific kernels for each permutation.This differs from manually fused operators, which require implementations for each supported combination.
- Operator Stacking: Stacking lets Glow lower high-level operators into primitives while recovering performance through backend fusion, and can reduce GPU kernel-launch overhead.The paper gives SGD lowering into addition, subtraction, and multiplication as an example.
- ResNet50 Optimization: ResNet50 optimization combines high-level graph transformations with target-specific convolution strategies for different channel, activation-buffer, and weight-buffer sizes.The transformations include removing redundant transposes and merging batch normalization with convolution.
- Memory Optimization: The low-level optimizer shortens activation allocation lifetimes and statically places the network in a single buffer, reducing mutable memory footprint.After static allocation, compiled code can refer directly to memory pointers.
6 Glow Runtime
Glow’s runtime manages partitioning, compilation, loading, and asynchronous execution of neural networks across heterogeneous accelerator devices. It hides hardware differences behind host- and device-level abstractions.
- Runtime Overview: The runtime partitions models, queues requests, and executes models across multiple devices through a common host-level interface.It uses knowledge of the available accelerator cards to support partitioning decisions.
- Partitioning: The Partitioner divides a network into sub-networks using memory, estimated execution-time, and inter-device communication costs.Partitioning can help accommodate accelerator memory limits and saturate multiple accelerators.
- Provisioning: The Provisioner assigns partitioned sub-graphs to devices and invokes the backend and Device Manager to compile and load them.Figure 10 depicts assignments in which networks may be partitioned or duplicated across accelerators.
- Device Management: Each Device Manager abstracts a physical accelerator by handling network loading, memory transfers, execution, and hardware state.Glow provides a DeviceManager class per device type and instance per physical accelerator.
- Execution: The Executor tracks sub-network state, propagates dependencies, and asynchronously handles inference requests before returning collated results.Execution proceeds through a scheduled directed graph of partitioned sub-networks.
- Execution: Inference handling creates an execution graph, launches the first sub-network, transfers inputs, signals completion, triggers satisfied dependencies, and returns outputs.These steps are coordinated by the HostManager, Executor, and DeviceManager.
7 Evaluation
Glow was evaluated against TensorFlow-1.7 and TVM on a single-threaded Intel Kaby Lake CPU using ResNet50 and VGG19. It achieved higher reported throughput in the tested configuration, with the strongest comparison against TensorFlow.
- Setup: The evaluation compares Glow, TensorFlow-1.7, and TVM on one Intel Core i7-7600U CPU core using ResNet50 and VGG19.All frameworks target the native architecture, and results use batch size 8.
- Results: 2.7x faster than TensorFlow: Glow achieves this reported maximum speedup in the evaluation.The paper attributes the difference to Glow’s direct convolution and avoidance of TensorFlow’s im2col overhead.
- Interpretation: Glow combines direct convolution with shape-aware code generation and efficient memory-access patterns in the evaluated CPU backend.The paper notes that TVM and Glow both generate efficient patterns such as tiling.
- Results: 1.3x faster than TVM: Glow achieves this reported maximum speedup under the tested configuration.The comparison did not use TVM autotuning or improved schedules, which the paper expects would improve TVM’s performance.
8 Conclusion
Glow is a machine learning compiler for heterogeneous hardware that lowers neural-network compute graphs into multilevel strongly typed intermediate representations. This structure supports level-specific analyses and optimizations for efficiently targeting many backends.
- Glow lowers neural-network compute graphs into multilevel strongly typed intermediate representations.
- Each representation level enables analyses and optimizations suited to efficient and scalable compilation across many hardware backends.
- The project aims to enable further research in machine learning acceleration.