Source-linked AI summary
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, Xiaoqiang Zheng
TL;DR
TensorFlow addresses the challenge of expressing and executing machine-learning computations across heterogeneous systems at different scales. The paper presents a flexible dataflow-based interface and implementation, which supports research and production workloads ranging from mobile inference to large-scale neural-network training.
Problem
Machine-learning systems need a flexible way to express and execute computations across heterogeneous hardware, from mobile devices to large distributed deployments.
Method
TensorFlow uses a dataflow-like programming model with implementations for single-machine and distributed systems, mapping computations across varied hardware and supporting parallel execution.
Results
Dozens of internal clients switched to TensorFlow for research and production tasks spanning mobile computer-vision inference and large-scale neural-network training.
Takeaways & Limitations
TensorFlow provides a flexible platform used across diverse machine-learning applications and released as an open-source system for broader community use.
Takeaways & Limitations
Automatic gradient construction can cause memory-management heuristics to break down, retaining tensors in scarce GPU memory and limiting computation size.
Abstract
from arXiv · showhide
TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.
1 Introduction
TensorFlow is Google’s second-generation system for implementing and deploying large-scale machine-learning models, developed from experience with DistBelief. It supports low-effort experimentation with parallelism and serves research and production workloads ranging from mobile inference to massive distributed training.
- 1 Introduction: TensorFlow applications span diverse areas including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery.The system was used both for research and for deploying machine-learning systems into production.
- 1 Introduction: TensorFlow is Google’s second-generation system for implementing and deploying large-scale machine-learning models, built from experience with DistBelief.DistBelief supported extensive research and product deployments before TensorFlow was developed.
- 1 Introduction: Clients can express multiple forms of parallelism by replicating and executing a core model dataflow graph across collaborating computational devices.Modest computation-description changes enable different parallel approaches with low effort, and some uses exploit relaxed synchronization requirements.
- 1 Introduction: Dozens of DistBelief clients had switched to TensorFlow for research and production across workloads including mobile computer-vision inference and large-scale neural-network training.The reported training workloads used hundreds of billions of parameters and example records across many hundreds of machines.
2 Programming Model and Basic Concepts
TensorFlow expresses computations as directed dataflow graphs whose nodes implement operations over tensors, with support for state and control flow. Clients build graphs and execute requested outputs through sessions, while variables provide mutable state across executions.
- Graph-based computation: TensorFlow represents computation as a directed graph of nodes, with extensions for persistent state and branching or looping control structures.Clients typically construct these graphs using C++ or Python.
- Graph-based computation: Nodes instantiate operations, and normal edges carry tensors while control dependencies enforce ordering without carrying data.Tensors are arbitrary-dimensional arrays whose element types are specified or inferred during graph construction.
- Operations and kernels: Operations are named abstract computations with construction-time attributes, while kernels provide particular implementations and attributes can support polymorphism across tensor element types.Examples include matrix multiplication, addition, and addition over float or int32 tensors.
- Sessions and execution: Sessions manage graphs through Extend and execute requested outputs through Run, which computes the required dependency closure and schedules nodes in dependency-respecting order.Run can feed tensors into the graph in place of certain node outputs.
- Persistent state: Variables return handles to persistent mutable tensors that survive graph executions and can be modified by operations such as Assign and AssignAdd.Most other tensors do not survive beyond a single graph execution.
3 Implementation
TensorFlow implements graph execution through clients, masters, workers, and heterogeneous devices in both local and distributed settings. Its runtime maps graph nodes to devices, handles cross-device communication through explicit Send/Receive nodes, and supports scalable distributed execution with checkpoint-based recovery.
- System architecture: Clients communicate with a master through the Session interface, while workers arbitrate access to CPUs or GPUs and execute graph nodes as instructed.TensorFlow provides both local and distributed implementations of this interface.
- Devices and tensors: Devices expose typed identities and manage memory allocation, deallocation, and kernel execution; TensorFlow includes CPU and GPU implementations and supports registering other device types.Tensors are typed multidimensional arrays whose device-specific backing storage is reference counted.
- Device placement: The placement algorithm uses cost-model estimates and greedy simulation to assign each graph node to a feasible device before real execution.The cost model estimates tensor sizes and operation times using heuristics or measurements from earlier placements.
- Communication and scheduling: Cross-device edges are replaced with Send/Receive node pairs that isolate communication, transmit each needed tensor once per source–destination pair, and allocate destination memory once.The same mechanism provides synchronization between workers and devices while simplifying runtime scheduling.
- Communication and scheduling: A single Run request per participating worker replaces master scheduling of every node and communication, enabling more scalable and finer-granularity execution.Distributed Send/Receive pairs move data across machine boundaries using mechanisms such as TCP or RDMA.
- Fault tolerance: Distributed failures abort and restart graph execution, while periodically saved Variable-node state supports consistent checkpointing and recovery.Failures are detected through Send/Receive communication errors or periodic master health checks of workers.
4 Extensions
TensorFlow extends its basic programming model with automatic differentiation, partial graph execution, device-placement constraints, control flow, and asynchronous queues. These features support flexible execution and more concise or efficient machine-learning representations while introducing memory-management challenges for gradients.
- Automatic gradient computation: TensorFlow automatically computes gradients by extending the computation graph with nodes that compose operation-specific partial gradients along backward paths using the chain rule.Given tensor C and dependent tensors {Xk}, the built-in function returns {dC/dXk}.
- Automatic gradient computation: Automatic gradients can undermine memory-reuse heuristics because reversing forward computation order gives users less control over execution ordering.Users can change graph-construction order or add control dependencies when forward-execution heuristics are ineffective, but automatically added gradient nodes reduce that control.
- Partial execution: The Run method executes arbitrary graph subgraphs, accepts fed tensors on edges, and returns tensors flowing along requested output edges.Inputs and outputs define the exact subgraph through inserted feed and fetch nodes, after which dependencies determine the nodes to execute.
- Device placement: Clients can constrain node placement by device type, job location, or colocation, while TensorFlow computes feasible device sets and colocation components.Union-find identifies components that must be placed together, and their feasible sets are intersected before assignment.
- Control flow: Switch, Merge, Enter, Leave, and NextIteration operators extend TensorFlow to cyclic dataflow graphs supporting conditionals and loops, with concurrent loop iterations.The runtime uses tags and frames to identify iterations and represent their execution state.
- Queues: Queues enable asynchronous graph portions to run at different cadences and transfer data through blocking Enqueue and Dequeue operations.They can prefetch input data from disk while a previous batch is being processed.
5 Optimizations
TensorFlow improves performance and resource usage through graph deduplication, operation scheduling, non-blocking kernels, optimized numerical libraries, and reduced-precision communication. These optimizations target redundant computation, memory and transfer costs, thread-resource overhead, kernel efficiency, and communication efficiency.
- Graph optimization: A common subexpression pass canonicalizes operations with identical inputs and types, replacing redundant graph copies with a single node.The pass redirects graph references to the canonicalized node.
- Scheduling: ASAP/ALAP scheduling analyzes graph critical paths to delay Receive nodes, reducing unnecessary early remote-value reads and improving memory and transfer efficiency.Scheduling can shorten intermediate-result lifetimes, lower peak memory consumption, and reduce cross-device communication contention, especially on GPUs.
- Kernel execution: Non-blocking kernels invoke a continuation upon completion, avoiding the resource costs of maintaining many active threads.They use a different interface from synchronous kernels, whose execution completes within the Compute method.
- Numerical libraries: TensorFlow kernel implementations often wrap optimized device-specific libraries, including BLAS, cuBLAS, cuda-convnet, and cuDNN.The system also extensively uses Eigen and extended it to support arbitrary-dimensionality tensor operations.
- Communication optimization: For noise-tolerant algorithms, TensorFlow can lossy-compress higher-precision representations for communication and restore them to 32-bit values after transfer.The 32 →16 →32-bit conversion fills lost mantissa bits with zeroes rather than using probabilistic rounding.
6 Status and Experience
TensorFlow was released as an Apache 2.0 open-source system with documentation, tutorials, examples, and Python and C++ front-ends. Porting the Inception model exposed difficult validation challenges, but systematic debugging strategies enabled a 6-fold training-time improvement over DistBelief.
- Release and usability: TensorFlow was open sourced under an Apache 2.0 license with documentation, tutorials, examples, and availability at www.tensorflow.org.Examples covered varied machine learning tasks, including MNIST handwritten-digit classification.
- Release and usability: The system provided front-ends for specifying computations in Python and C++, with additional front-ends expected over time.Future additions were expected to respond to internal Google users and the broader open-source community.
- Outcome: 6-fold speed improvement in training time versus the existing DistBelief implementation resulted from the validation strategies used to instantiate Inception in TensorFlow.The speed gains proved indispensable for training a new class of larger-scale image models.
- Inception migration: Porting Inception was challenging because assembling 36,000 operations correctly required validation in an inherently stochastic system.The model classified 224 × 224 pixel images into 1000 labels, comprised 13.6 million learnable parameters, and required 2 billion multiply-add operations per image during inference.
- Inception migration: Parameter-counting tools revealed subtle architecture flaws, including incorrectly instantiated operations and variables caused by automatic broadcasting.These tools were among the critical strategies used to port Inception to TensorFlow.
- Inception migration: Starting with a small CIFAR-10 convolutional network exposed subtle operation edge cases that were difficult to decipher in complex models.The strategy supported progressively scaling the migration after debugging simpler networks.
7 Common Programming Idioms
TensorFlow’s dataflow graph model expresses several techniques for accelerating SGD-based training of computationally intensive neural networks. These include synchronous and asynchronous data parallelism, model parallelism, and pipelined concurrent execution.
- Data parallelism: Synchronous data parallelism splits a mini-batch across model replicas, combines their gradients, and applies updates so execution matches sequential SGD on the full batch.For a batch of 1000, 10 replicas each process 100 examples before synchronously combining gradients and updating parameters.
- Data parallelism: Asynchronous data parallelism uses multiple graph replicas that independently apply parameter updates, with one client thread per replica.The replicas execute the bulk of model computation and update shared model parameters asynchronously.
- Model parallelism: Model parallel training assigns different portions of the same batch’s model computation to different devices simultaneously.TensorFlow expresses a recurrent deep LSTM sequence-to-sequence model parallelized across three devices.
- Pipelined execution: Pipelined execution improves device utilization by running a small number of concurrent model-computation steps on the same devices.This resembles asynchronous data parallelism, but parallelism occurs within the same devices rather than through replicated computation graphs.
8 Performance
The paper defers comprehensive evaluation of TensorFlow’s single-machine and distributed implementations to a future version of the white paper.
- A future white-paper version will comprehensively evaluate both the single-machine and distributed implementations.The current section does not provide those performance results.
9 Tools
TensorFlow includes TensorBoard, an open-source visualization tool for computation graphs and model behavior, plus an internal EEG tool for fine-grained execution tracing and performance analysis. TensorBoard organizes large graphs and displays changing summaries, while EEG reconstructs detailed execution behavior across single-machine and distributed systems.
- TensorBoard: TensorBoard helps users understand computation-graph structure and overall model behavior through an open-source companion visualization tool.It is included in the TensorFlow open-source release.
- TensorBoard: TensorBoard reduces graph-visualization clutter by collapsing nodes into high-level blocks and separating high-degree bookkeeping nodes.Users can pan, zoom, and expand grouped nodes to inspect details interactively.
- TensorBoard: TensorBoard displays scalar, histogram-based, and image-based summaries, including loss, execution time, weight distributions, and learned filter visualizations.Users can examine how summary values change over relative wall time, absolute time, or graph-execution steps.
- EEG: EEG collects and visualizes fine-grained information about TensorFlow graph execution ordering and performance in both single-machine and distributed implementations.It is an internal tool and was not included in the initial November 2015 open-source release.
- EEG: EEG reconstructs distributed training steps with microsecond-level details and highlights communication, synchronization, DMA stalls, and thread-pool queueing delays.Traces combine Linux ftrace, lightweight thread tracing, and CUPTI sources; visualizations expose parallel dispatch and GPU-stream assignments.
10 Future Work
Future work will extend TensorFlow’s programming model and implementation while improving execution performance and automatically learned placement decisions. Planned directions include reusable cross-language functions, just-in-time compilation, and learned scheduling heuristics.
- TensorFlow developers will continue creating machine learning models and may extend the basic system as new needs emerge.The open source community may also propose new directions for the implementation.
- A proposed function mechanism would make entire TensorFlow subgraphs reusable components across different front-end languages.A function defined using the Python front end could be used as a building block from within the C++ frontend.
- A planned just-in-time compiler would optimize TensorFlow subgraphs using runtime profiling information about typical tensor sizes and shapes.The compiler is intended to understand operation semantics and perform optimizations such as loop fusion, blocking, and tiling for locality.
- Future work will improve placement and node scheduling algorithms that determine where nodes execute and when execution begins.TensorFlow currently uses heuristics, but the developers want the system to learn placement decisions, potentially with a deep neural network and reinforcement learning objective function.
11 Related Work
TensorFlow relates to neural-network, distributed machine-learning, image-processing, and cluster dataflow systems, while distinguishing itself through distributed execution and a flexible, general-purpose dataflow graph.
- Neural-network systems: Unlike Theano, Torch, Caffe, Chainer, and the Computational Network Toolkit, TensorFlow’s implementation maps computations across multiple machines rather than a single machine.TensorFlow also supports symbolic differentiation, like Theano and Chainer.
- Distributed machine-learning systems: Compared with DistBelief and Project Adam, TensorFlow’s general-purpose dataflow graph expresses a wider variety of machine-learning models and optimization algorithms.TensorFlow allows computations across many devices and machines and uses relatively high-level model descriptions.
- Distributed machine-learning systems: TensorFlow simplifies stateful distributed computation by representing parameters as variables and variable updates as additional graph nodes, rather than using separate parameter-server subsystems.This contrasts specifically with DistBelief’s separate parameter-server architecture.
- Image-processing systems: Unlike Halide, TensorFlow does not use higher-level operation semantics to generate optimized fused code, and TensorFlow executes computations in distributed settings rather than only on one machine.Halide uses operation semantics to account for parallelism and locality when combining operations.
- Distributed dataflow systems: TensorFlow uses a single optimized dataflow graph across devices, caches graph information to reduce coordination overhead, and supports hybrid iteration with concurrent replicas sharing variables.The system works best when cluster RAM can hold the computation’s working set.
12 Conclusions
The paper presents TensorFlow as a flexible data flow-based programming model with single-machine and distributed implementations, developed through extensive real-world use. It also reports open-sourcing TensorFlow and expresses hope for a vibrant shared community.
- TensorFlow is presented as a flexible data flow-based programming model with single-machine and distributed implementations.The paper describes both the programming model and implementations of it.
- The system grew from experience conducting research and deploying more than one hundred machine learning projects across Google products and services.These projects covered a wide range of Google products and services.
- The authors open sourced a version of TensorFlow and hope a vibrant shared community develops around its use.