Source-linked AI summary
Fathom: Reference Workloads for Modern Deep Learning Methods
Robert Adolf, Saketh Rama, Brandon Reagen, Gu-Yeon Wei, David Brooks
TL;DR
Deep-learning hardware research needs broader workload evidence because existing work concentrates on narrow techniques and application domains. The paper assembles eight Fathom workloads and analyzes them with TensorFlow-based application-level tools, finding that a small set of operation types usually accounts for most execution time while model-specific structure limits broad optimization benefits.
Problem
Hardware research has focused on a relatively narrow set of deep-learning techniques, motivating broader workload coverage for flexible solutions.
Method
The paper assembles eight state-of-the-art Fathom workloads and analyzes their execution behavior with high-level tools built around TensorFlow.
Results
A handful of heavy operation types, usually 5 to 15, account for upwards of 90% of program duration, but the dominant types differ across models.
Takeaways & Limitations
Performance is tied to application-level structure, so hardware support for convolution and matrix multiplication has limited benefits for models with non-convolutional or sparse components.
Abstract
from arXiv · showhide
Deep learning has been popularized by its recent successes on challenging artificial intelligence problems. One of the reasons for its dominance is also an ongoing challenge: the need for immense amounts of computational power. Hardware architects have responded by proposing a wide array of promising ideas, but to date, the majority of the work has focused on specific algorithms in somewhat narrow application domains. While their specificity does not diminish these approaches, there is a clear need for more flexible solutions. We believe the first step is to examine the characteristics of cutting edge models from across the deep learning community. Consequently, we have assembled Fathom: a collection of eight archetypal deep learning workloads for study. Each of these models comes from a seminal work in the deep learning community, ranging from the familiar deep convolutional neural network of Krizhevsky et al., to the more exotic memory networks from Facebook's AI research group. Fathom has been released online, and this paper focuses on understanding the fundamental performance characteristics of each model. We use a set of application-level modeling tools built around the TensorFlow deep learning framework in order to analyze the behavior of the Fathom workloads. We present a breakdown of where time is spent, the similarities between the performance profiles of our models, an analysis of behavior in inference and training, and the effects of parallelism on scaling.
I. INTRODUCTION
Fathom responds to a need for broader deep-learning workload coverage by assembling state-of-the-art models and analyzing their computational behavior for architecture research.
- Hardware research has increasingly targeted deep learning, but much prior work focuses on narrow algorithmic domains.The paper argues that broader workload understanding is needed for more versatile architectures.
- Fathom assembles workloads from across the deep learning community to represent diverse, state-of-the-art approaches.The models are drawn from top-tier research and include problems such as image classification and agent planning.
- The suite is intended to support research into architectures that serve a broader application space.
- The analysis uses a TensorFlow-based, high-level framework to study execution characteristics in the models themselves.
- The paper examines dominant operations, model similarity, training versus inference, and parallel scalability.
II. MOTIVATION: THE TIP OF THE ICEBERG
Deep learning hardware research has substantial opportunity, but its focus remains concentrated on a narrow subset of models and learning modalities.
- 10 exaflops is an estimated minimum for one modern deep-learning training run to converge.
- 10–100× performance or power improvements have been suggested for deep-learning hardware.The paper states that such gains could make previously intractable research overnight jobs and support constrained deployments.
- Architecture research disproportionately evaluates a narrow set of techniques, including the same CNN and mostly supervised or inference-only tasks.The survey found no recent hardware work supporting unsupervised or reinforcement learning.
- Fathom provides standard, verified reference workloads for exploring flexible architectures across deep-learning tasks.
- Unsupervised, reinforcement, and recurrent methods remain comparatively underserved by architecture research.
III. DESIGN AND IMPLEMENTATION CRITERIA
Fathom selects meaningful models for representativeness, diversity, and impact, while standardizing implementations and using TensorFlow-based tools that remain portable across frameworks.
- A. Choose meaningful models: Fathom selects models for representativeness, diversity, and impact because the workload set must remain both authoritative and manageable.
- A. Choose meaningful models: Fathom excludes input preprocessing and model-output post-processing from its workload scope.Existing benchmark suites are described as handling these techniques adequately.
- B. Implementation: Implementations use existing references when available, translate them when necessary, or recreate and validate them from original descriptions.
- B. Implementation: The workloads generally reuse original training and test data, substituting comparable datasets when proprietary data is unavailable.Deep Speech uses TIMIT instead of Baidu’s private utterance collection.
- C. Framework: TensorFlow is used for implementation and analysis because deep-learning frameworks share primitive-operation pipelines and the tools are designed to be framework-agnostic.
Sequence-to-Sequence Translation
The workloads include recurrent and memory-based approaches for sequence and reasoning tasks, illustrating Fathom’s coverage beyond conventional convolutional models.
- seq2seq is a recurrent neural network for machine translation using multilayer LSTMs and an attention-based context mechanism.
- The seq2seq model uses an encoder-decoder structure that maps an input sentence into a high-dimensional embedding before generating another language.
- Memory networks explicitly store and recall information by combining indirectly addressable memory with a neural network.
- End-to-end memory networks remove input type annotations and streamline training for bAbI natural-language reasoning.
Variational Autoencoder
The passage set introduces several influential deep learning models, including autoencoders, residual networks, VGG, AlexNet, and deep reinforcement learning. Their descriptions emphasize distinct architectures, training strategies, and application settings.
- Variational autoencoders learn compact embeddings to reconstruct inputs and can support dimensionality reduction, feature extraction, or data generation.Their embedding is assumed to capture a compact representation of realistic inputs.
- Residual networks use identity connections across convolutional layers to train networks exceeding 150 layers.The connections train layers on the difference between their input and output.
- VGG is a 19-layer convolutional network that uses more layers with smaller convolutional filters.This design improved accuracy and reduced the number of learnable parameters.
- AlexNet demonstrated that automatically trained neural networks could surpass hand-tuned image classifiers and highlighted GPU computational power.It also introduced dropout as a regularization mechanism.
- Deep reinforcement learning learned Atari-game actions from pixels and scores, using convolutional and dense layers with experience replay.Unlike supervised learning, it improves actions from in-game feedback rather than perfect demonstrations.
V. UNDERSTANDING THE PERFORMANCE CHARACTERISTICS OF DEEP LEARNING WORKLOADS
The paper analyzes Fathom workloads through TensorFlow operations, using operation-level measurements to connect model structure with execution behavior. This approach targets stable timing measurements while avoiding framework and hardware-profiling disconnects.
- TensorFlow provides direct access to primitive operations and model information, making runtime behavior assignable to layers and parameters.Low-level profiling provides detail but can lose the connection to the original model.
- Operations form coarse-grained dataflow-graph nodes implemented through Python construction functions and C++ computation functions.These functions may call lower-level libraries such as Eigen.
- Operation execution times are stationary and low-variance across program steps, supporting repeated sampling for performance measurement.Deep learning programs naturally separate into update-step or inference boundaries.
- Most experiments run on a CPU because unsupported GPU operations can split execution across the PCI bus and create severe performance artifacts.The experiments used a 4GHz Skylake i7-6700k or GeForce GTX 960 with TensorFlow v0.8, CUDA 7.0, and cuDNN 6.5-v2.
B. Operation type profiling
Operation-type profiling shows that execution time is highly concentrated, but the dominant operations vary by model. Comparing profiles also exposes architectural trends and deliberate design tradeoffs.
- Figure 3 excludes operations consuming 1% or less of execution time, so displayed row totals range from 90% to 100%.
- Convolutional networks are dominated by convolution, whereas fully connected networks depend heavily on matrix multiplication.
- AlexNet’s locally connected layers use 11% of runtime, compared with 7% for VGG’s fully connected layers and less than 1% for Residual’s classification layer.These models share the ILSVRC task and dataset, enabling a longitudinal comparison.
- Deep Speech’s structurally homogeneous recurrent model spends almost all significant computation on matrix multiplication and CTC loss operations.The breakdown supports the authors’ aim of making recurrent activations efficient.
- Figure 4 compares workload profiles by treating operation distributions as vectors, computing cosine-distance similarities, and applying agglomerative clustering.The tightly clustered lower group contains all convolutional networks.
C. Performance similarity
Fathom compares workloads through operation-level profiles and examines how training, inference, hardware, and parallelism shape performance. The profiles reveal both similarities among convolution-heavy models and limits to speedup from smaller operations.
- Operation-level similarity: Operation profiles are represented as vectors, compared with cosine distance, and clustered using agglomerative clustering with centroidal linkage.This provides a workload-level method for measuring similarity and organizing model relationships.
- Operation-level similarity: The three ImageNet networks cluster closely, while Deep Speech and seq2seq remain far apart despite both using recurrent networks.Their dissimilarity reflects different underlying operations, including CTC loss and stacked fully connected layers in Deep Speech.
- Training and inference: Training is more expensive than inference across all workloads, but the training-to-inference gap varies by model.Convolutional networks pay a somewhat higher training cost because backward convolution requires two reductions versus one in the forward phase.
- Parallelism: Parallelizing matrix multiplication and convolution can expose smaller data-dependent operations that limit application-level speedups.The figure frames this behavior as an application-level effect of Amdahl’s law.
- Training and inference: GPU performance is substantially higher, especially for workloads with skewed operation profiles, while CPU and GPU train-to-inference ratios remain strongly correlated.This indicates that workload-specific training and inference differences persist across hardware configurations.
E. Parallelism and operation balance
Parallelism changes which operations dominate execution rather than uniformly accelerating entire models. Smaller, data-dependent, or poorly parallelizable operations become important as major kernels speed up.
- Operation balance: Parallel hardware can make smaller operations visible because convolution and matrix multiplication no longer dominate execution time.The analysis varies thread-pool parallelism within operations and tracks absolute time by operation type.
- Operation balance: At eight threads, the deepq optimizer rises to around 7% of execution time after being hidden under single-threaded convolution and fully connected work.The optimizer contains many data-dependent operations whose relative share increases with parallelism.
- Operation balance: In seq2seq, the loss function becomes visible at higher thread counts after LSTM neurons and attention embeddings dominate at low parallelism.This is another example of bottleneck migration as major operations receive more parallel resources.
- Memory model: Memory-layer operations often parallelize poorly because they operate on small, skinny tensors with too few elements for thread-level parallelism.Elementwise multiplication and the optimizer are exceptions because they operate on wider tensors or many parameters simultaneously.
- Implications: Hardware support for convolution and matrix multiplication has limited benefits when models include non-convolutional layers, sophisticated losses or optimizers, or sparse storage.The resulting performance behavior is tied to application-level model structure.
VI. RELATED WORK
Prior benchmark suites and model zoos serve different purposes, while Fathom targets representative, modern deep learning workloads for quantitative architecture research. It packages eight models uniformly and analyzes their performance behavior.
- Benchmark suites: CortexSuite, BenchNN, and DjiNN and Tonic address different questions and do not claim to represent modern deep learning workloads broadly.Their included algorithms are either conventional, approximate-computing demonstrations, or selected for warehouse-scale scalability and total cost of ownership studies.
- Model zoos: Model zoos primarily provide living documentation, showcasing framework features and popular methods rather than serving as standardized architecture benchmarks.Their role includes demonstrating library usage and helping practitioners learn machine learning frameworks.
- Fathom: Fathom assembles eight modern deep learning models into one unified package with a consistent interface.The suite is intended to reduce workload-selection burden and support quantitative development of better deep learning hardware and systems.
- Fathom: Fathom provides workload insights about model similarity, cycle distribution, and how parallel execution shifts performance bottlenecks.These analyses are intended to guide researchers toward important architecture problems.