Source-linked AI summary
Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning
Scott Cyphers, Arjun K. Bansal, Anahita Bhiwandiwalla, Jayaram Bobba, Matthew Brookhart, Avijit Chakraborty, Will Constable, Christian Convey, Leona Cook, Omar Kanawi, Robert Kimball, Jason Knight, Nikolay Korovaiko, Varun Kumar, Yixing Lao, Christopher R. Lishka, Jaikrishnan Menon, Jennifer Myers, Sandeep Aswath Narayana, Adam Procter, Tristan J. Webb
TL;DR
Deep-learning performance optimization across many frameworks and hardware platforms currently requires framework-specific changes. nGraph introduces a framework-independent IR and backend transformers; the paper describes compiler optimizations and execution paths spanning its supported frameworks and backends.
Problem
Improving training performance for new deep-learning topologies across frameworks and hardware requires deep framework changes and O(fp) effort.
Source: §1 · p. 1Method
Results
The described transformers provide pattern matching, liveness analysis, memory management, and tensor layout and shape management combined with backend kernel selection.
Source: §4 · p. 2Takeaways & Limitations
A fixed, extensible IR operation set distinguishes nGraph from NNVM's unspecified operator set, which the paper says makes frontends and backends incompatible.
Source: §1.1 · p. 2
Abstract
from arXiv · showhide
The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call "direct optimization", requires deep changes within each framework to improve the training performance for each hardware backend (CPUs, GPUs, FPGAs, ASICs) and requires $\mathcal{O}(fp)$ effort; where $f$ is the number of frameworks and $p$ is the number of platforms. While optimized kernels for deep-learning primitives are provided via libraries like Intel Math Kernel Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired ways in which performance can be further optimized. Building on our experience creating neon (a fast deep learning library on GPUs), we developed Intel nGraph, a soon to be open-sourced C++ library to simplify the realization of optimized deep learning performance across frameworks and hardware platforms. Initially-supported frameworks include TensorFlow, MXNet, and Intel neon framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R) Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported compiler optimizations include efficient memory management and data layout abstraction. In this paper, we describe our overall architecture and its core components. In the future, we envision extending nGraph API support to a wider range of frameworks, hardware (including FPGAs and ASICs), and compiler optimizations (training versus inference optimizations, multi-node and multi-device scaling via efficient sub-graph partitioning, and HW-specific compounding of operations).
1 INTRODUCTION
nGraph places a framework-independent intermediate representation between framework computation definitions and backend execution. The related-work discussion contrasts its fixed, extensible operation set and broader training and execution support with other developing IR efforts.
1 INTRODUCTION
- nGraph converts framework-specific computations into a framework-independent IR that backends compile or interpret for execution.Framework bridges act as framework backends, while backend transformers provide allocation and execution APIs.Source: §1 · p. 1
1.1 Related Work
- NNVM leaves its operator set unspecified, whereas nGraph, XLA, and LLVM use a fixed, extensible IR operation set.The passage says NNVM's unspecified set makes different frontends and backends incompatible.Source: §1.1 · p. 2
- DLVM proposes an LLVM-inspired modular IR with full control flow and a side-effect-free representation, whose optimization potential remains uncertain.The passage leaves open whether its flexibility can support optimizations enabled by simpler data-flow graph IRs.Source: §1.1 · p. 2
- ONNX is a cross-industry effort to standardize an inference IR, while nGraph includes training support, optimization passes, and execution backends.The authors say they aim for ONNX interoperability.Source: §1.1 · p. 2
- The authors expect nGraph to interoperate with developing standards through framework bridges and nGraph backends.They describe the compiler and IR efforts as complementary.Source: §1.1 · p. 2
2 INTERMEDIATE REPRESENTATION
The nGraph IR represents computation as a graph of stateless operations over tensors. It leaves tensor axis order independent of element layout, except for tensors directly accessible to framework users.
2 INTERMEDIATE REPRESENTATION
- An nGraph IR is a directed acyclic graph of stateless operation nodes, with inputs, outputs, and optional constant attributes.A node's inputs and attributes determine output shapes and element types.Source: §2 · p. 2
- Node inputs and constant attributes determine output shape and element types.Attributes can specify details such as which axes to sum over.Source: §2 · p. 2
- nGraph nodes operate on multidimensional arrays called tensors, whose axis order is not fixed to their element layout except for tensors directly accessible to framework users.This differs from framework-specific axis conventions, which can require reshaping and reordering for video or high-dimensional time-series data.Source: §2 · p. 2
3 FRAMEWORK BRIDGES
Framework bridges translate framework computations into nGraph IR, allowing framework-specific execution paths to use nGraph functions. The described bridges cover MXNet, TensorFlow through XLA, and a planned Python binding for neon.
3 FRAMEWORK BRIDGES
- Framework bridges use framework computational graphs to construct nGraph IR.Framework graphs can also support autodiff, either directly or by constructing a derivative computation graph.Source: §3 · p. 2
- The MXNet-nGraph bridge translates the NNVM inference graph into nGraph IR and selects the largest possible computation for the backend.It uses autodiff on nGraph IR for derivatives, and compiled functions interface with standard MXNet execution engines.Source: §3 · p. 2
- The TensorFlow bridge registers as an XLA device, maps HLO IR to nGraph IR, and returns a compiled function for TensorFlow execution.TensorFlow invokes the function on input data and receives the resulting nGraph output.Source: §3 · p. 2
- For neon, the authors are creating a Python binding for the nGraph API, with possible use in other Python-based frameworks..Source: §3 · p. 2
4 TRANSFORMERS
nGraph transformers generate backend-specific optimized code from the IR, combining graph analyses and memory management with tensor layout and kernel selection. The section describes CPU, NNP, GPU, and communication support paths.
4 TRANSFORMERS
- Transformers generate backend-specific optimized code and provide pattern matching, liveness analysis, memory management, and combined tensor layout, shape, and kernel selection.These facilities combine tensor-element layout and shape management with backend kernel selection.Source: §4 · p. 2
- The CPU transformer uses MKL-DNN for optimized kernel sequences and can handle subgraphs containing operations unsupported by other backends.MKL-DNN optimizations operate at a finer granularity than nGraph's.Source: §4 · p. 2
- The NNP transformer aims to use the hardware fully and falls back to the CPU transformer for unsupported operations..Source: §4 · p. 2
- The cuDNN transformer dynamically generates links to cuDNN for common kernels and uses LLVM to compile graph portions into PTX assembly.The cuDNN-linked kernels include convolution and softmax.Source: §4 · p. 2
- nGraph plans collective and point-to-point communication primitives as core graph operations, with transformers generating corresponding library calls.Planned options include vanilla MPI or optimized methods that may impose restrictions, such as requiring a homogeneous cluster topology.Source: §4 · p. 2