PaperlayerView exact revision on arXiv

Source-linked AI summary

Chainer: A Deep Learning Framework for Accelerating the Research Cycle

Seiya Tokui, Ryosuke Okuta, Takuya Akiba, Yusuke Niitani, Toru Ogawa, Shunta Saito, Shuji Suzuki, Kota Uenishi, Brian Vogel, Hiroyuki Yamazaki Vincent

arXiv:1908.00213v1cs.LGcs.CVcs.DCstat.ML

TL;DR

Under Define-and-Run, the neural network’s inner mechanism is inaccessible to users, making the computational graph essentially a black box for debugging and tuning. Chainer records computation during execution and provides GPU array operations through CuPy, along with distributed training and a computer-vision add-on. Chainer has been used successfully in applications including deep reinforcement learning, word2vec, recurrent language models, pose estimation, and variational auto-encoders.

  • Problem

    Under Define-and-Run, the neural network’s inner mechanism is inaccessible to users, making the computational graph essentially a black box for debugging and tuning.

    Source: §2 · p. 2

  • Method

    Chainer records computation during execution and provides GPU array operations through CuPy, along with distributed training and a computer-vision add-on.

    Source: §3.1 · p. 2, §5 · p. 4, §5.3 · p. 5

  • Results

    Chainer has been used successfully in applications including deep reinforcement learning, word2vec, recurrent language models, pose estimation, and variational auto-encoders.

    Source: §9 · p. 8

  • Takeaways & Limitations

    Chainer’s distributed training preserves Define-by-Run flexibility for dynamic neural networks, generative adversarial networks, and deep reinforcement learning.

    Source: §6 · p. 5, §6.3 · p. 6

    For all-reduce, model structures must be identical across workers within a single iteration, although they may change between iterations.

    Source: §6.3 · p. 6

Abstract

from arXiv · show

Software frameworks for neural networks play a key role in the development and application of deep learning methods. In this paper, we introduce the Chainer framework, which intends to provide a flexible, intuitive, and high performance means of implementing the full range of deep learning models needed by researchers and practitioners. Chainer provides acceleration using Graphics Processing Units with a familiar NumPy-like API through CuPy, supports general and dynamic models in Python through Define-by-Run, and also provides add-on packages for state-of-the-art computer vision models as well as distributed training.

1 INTRODUCTION

Deep learning’s expanding applications require frameworks that can support increasingly varied research models. Chainer is introduced as an open-source framework for implementing complex algorithms, training models, and tuning parameters.

1 INTRODUCTION

  • Deep learning’s move beyond pattern recognition into diverse domains and industries creates a need for frameworks to implement new research ideas.The passage also notes growing work on reinforcement learning and recurrent networks for variable-length data.Source: §1 · p. 1
  • Many existing frameworks prioritize CNNs for image processing and lack support for more general models and abstract data structures.Domain-specific model languages can also prevent developers from using standard debuggers, despite debugging’s importance in developing and tuning models.Source: §1 · p. 1
  • Chainer is an open-source framework intended to support implementing complex algorithms, training models, and tuning model parameters.The paper introduces it as providing simple and efficient support for these tasks.Source: §1 · p. 1

2 BACKGROUND AND MOTIVATION

Define-and-Run frameworks construct a model’s computational graph before executing it, which suits static models but complicates dynamic models and inspection. Chainer’s motivation is to address these constraints through a more flexible programming model.

2 BACKGROUND AND MOTIVATION

  • Define-and-Run constructs the computational graph first, then runs forward and backward calculations on training examples to minimize loss.Static models such as CNNs can benefit from having the full graph available for memory or runtime optimizations.Source: §2 · pp. 1–2
  • Supporting dynamic graphs is cumbersome when control flow must be encoded with graph operators instead of host-language syntax.TensorFlow is given as an example of this approach, using operators such as Switch and Merge.Source: §2 · p. 2
  • A whole-model computational graph can act as a black box, limiting users’ ability to inspect, debug, and tune models.Framework graph optimizations can compound the difficulty for profilers and debuggers to locate faults or identify improvements.Source: §2 · p. 2

3 DESIGN AND PROGRAMMING MODEL

Chainer’s programming model records computation during execution and uses object-oriented components to bind parameters and compose models. This design supports dynamic graphs while retaining modular model construction.

3.1 On Demand Graph Construction

  • Define-by-Run records operation history during forward computation, allowing the graph to follow host-language branches and loops and vary across invocations.The graph supports backtracking for gradient computation, while the program can use debuggers and profilers.Source: §3.1 · p. 2, §3 · p. 2

3.2 Object-Oriented Model Definition

  • Object-oriented model definition binds parameters to model fragments by defining parameterized components as classes and composing them into larger models.Parameters are initialized at object construction, and forward computation is written as a method.Source: §3.2 · pp. 2–3
  • This object-oriented style was first introduced by Chainer in 2015 and is now widely used in other Define-by-Run frameworks.A model’s fragment tree also allows parameters in selected subtrees to be collected by traversal.Source: §3.2 · p. 3

4 TECHNICAL FEATURES

Chainer’s technical features target memory-efficient backpropagation and support for differentiating through gradient computations. They manage graph references and retained data to reduce memory costs and avoid reference cycles.

4.1.1 Global Memory Usage Reduction.

  • Chainer’s memory-efficient backpropagation reduces peak memory by eliminating references to function nodes as graph backtracking processes them.This combines graph-release and backpropagation traversals, which both follow topological order.Source: §4.1.1 · p. 3, §4.1 · p. 3

4.1.2 Function-wise Local Memory Usage Reduction.

  • Operations declare which inputs or outputs they retain for backward computation, so unlisted arrays need not be saved.Backward implementations retrieve inputs only when needed.Source: §4.1.2 · p. 3
  • Separating user-facing Variable objects from graph VariableNode objects lets Chainer release array data when no user reference or retained operation requires it.A graph node holds an array-data reference only when an operation retains the variable as input or output.Source: §4.1.2 · p. 3

4.2 Double Backpropagation

  • Backpropagating through gradient computation, or double backpropagation, provides automatic differentiation for Hessian-vector products.Chainer implements it by making each operation’s backward computation differentiable, while addressing the risk of reference cycles and excess memory use.Source: §4.2 · p. 4

4.2.1 Interface to Access the Resulting Gradients.

  • The grad() interface returns gradients for specified inputs and outputs without adding references between graph objects, unlike Variable.backward().Variable.backward() stores gradients on variables and may create references that form cycles through the computational graph.Source: §4.2.1 · p. 4, §4.2 · p. 4

4.2.2 Output Retention for Double Backpropagation.

  • Differentiable backpropagation recreates a released output node during backward computation and restores its saved array data.This avoids retaining a reference to the output node that would create a cycle.Source: §4.2.2 · p. 4, §4.2 · p. 4

5 GPU SUPPORT BY CUPY

CuPy brings GPU computation to Python through a NumPy-compatible interface, with support for built-in operations and user-defined CUDA kernels.

5.1 CuPy Example

  • Figure 3 contrasts NumPy and CuPy usage; CuPy arrays store their contents in GPU memory, unlike NumPy arrays.The example replaces NumPy with CuPy to compute a Euclidean norm on the GPU rather than the CPU.Source: §5.1 · p. 4, Figure 3

5 GPU SUPPORT BY CUPY

  • CuPy accelerates NumPy-like computations on NVIDIA GPUs using CUDA libraries, including cuBLAS, cuDNN, and cuSPARSE.The paper frames GPU acceleration as important because neural-network computation requires substantial floating-point power.Source: §5 · p. 4

5.3 Custom CUDA Kernels

  • CuPy compiles and caches user-provided CUDA snippets; generic type parameters let generated kernels accept arrays of arbitrary types.Generic arguments can require matching array types while allowing different concrete types, such as integers or floats.Source: §5 · p. 4, §5.3 · p. 5

5.2 Supported Functionalities

  • CuPy implements many functions on cupy.ndarray objects and supports a subset of the NumPy API.Its interface is compatible with NumPy in most cases, including common array operations such as indexing, reshaping, and broadcasting.Source: §5.2 · p. 4, §5 · p. 4

5.2.1 Linear Algebra.

  • CuPy supports NumPy-compatible linear algebra functions, including decompositions and matrix solvers, backed by GPU-based cuSOLVER.Supported operations include eigen, Cholesky, QR, and singular value decompositions, as well as matrix inversion.Source: §5.2.1 · pp. 4–5

5.3 Custom CUDA Kernels

  • Figure 4 shows how users define an element-wise CUDA kernel by specifying input variables, output variables, and a code snippet.Element-wise kernels apply the same operation to each element; reduction kernels fold elements using a binary operator.Source: Figure 4 · p. 5, §5.3 · p. 5

5.2.2 Sparse Matrices.

  • CuPy supports sparse matrices through cuSPARSE, with coordinate, CSR, CSC, and diagonal storage formats.The sparse-matrix interfaces are the same as those in SciPy.Source: §5.2.2 · p. 5

5.2.3 Sorting.

  • CuPy provides NumPy-compatible sort, argsort, and lexsort functions backed by Thrust's GPU parallel algorithms.Source: §5.2.3 · p. 5

6 DISTRIBUTED PARALLEL TRAINING

Chainer’s distributed training uses synchronous data parallelism and minimal API additions to retain Define-by-Run flexibility. At 1024 GPUs, the reported scaling efficiency was 70% versus a single-GPU baseline and 80% versus a single-node baseline.

6.1.1 Data and Model Parallelism.

  • Data parallelism replicates models across workers and averages gradients from different minibatches, while model parallelism divides a model across workers and can complement data parallelism for huge models.Source: §6.1.1 · p. 5

6.2 Parallelism Design

  • Chainer adopts data parallelism with synchronous communication for deterministic behavior and convergence, using NCCL as its primary all-reduce library.The communicator abstraction can support any communication library with all-reduce; OpenMPI is also supported.Source: §6.2 · pp. 5–6

6.3 API Design

  • Distributed training adds a communicator, a gradient-exchanging optimizer, and dataset scattering while preserving Define-by-Run flexibility.Workers need identical model structures only within each iteration; structures may change between iterations.Source: §6.3 · p. 6

6.4 Evaluation

  • 70% versus single-GPU and 80% versus single-node (8-GPU) scaling efficiency at 1024 GPUs; mean 90-epoch training took 897.9 ± 3.3s.The mean was across five runs and included validation after each epoch.Source: Figure 6 · p. 6, §6.4 · p. 6

7 CHAINERCV

ChainerCV packages computer-vision models, pretrained weights, training scripts, and utilities behind a unified API. Its reimplementations perform close to the original reported scores, with training randomness noted as a possible source of differences.

7 CHAINERCV

  • ChainerCV addresses computer-vision needs with models, pretrained weights, training scripts, data loaders, and evaluation metrics across four task types.Source: §7 · p. 7, §7.1 · p. 7

7.1 Easy to use

  • A shared inference interface hides model-specific output and postprocessing differences, and automatically downloaded pretrained weights enable inference in two lines of Python.Customizable training scripts use Chainer’s training abstraction and can be extended to train on user datasets.Source: §7.1 · p. 7

7.2 Unified API

  • Consistent data representations and APIs let users compose models, loaders, metrics, and visualization utilities; the evaluation abstraction passes data from loaders through inference to metrics.Source: §7.2 · p. 7

7.3 Reproducibility

  • ChainerCV reimplementations perform close to original reported scores across supported models, though training randomness can contribute to score differences.Source: §7.3 · p. 7, Table 1 · p. 8

8 RELATED WORK

The related work situates Chainer among Define-by-Run frameworks, distributed-training systems, and computer-vision libraries, contrasting their implementation focus and support for reproducibility.

8 RELATED WORK

  • Chainer is optimized for Define-by-Run throughout its implementation and APIs, unlike frameworks where it is optional, resulting in a simpler codebase and lower costs for new contributors.Autograd had adopted the paradigm before Chainer, but was not intended as a deep-learning framework and lacked GPU acceleration for deep-model training.Source: §8 · p. 8
  • Asynchronous SGD can leave stale gradients and a parameter-server bottleneck; synchronous SGD is available in several frameworks, including Caffe2 and PyTorch.Caffe2 and PyTorch start all-reduce after each layer's backward computation completes to reduce synchronization's critical-path cost.Source: §8 · p. 8
  • DistBelief first integrated data and model parallelism, asynchronous SGD, and a master-worker heterogeneous model; TensorFlow, MXNet [9], and PaddlePaddle later offered these techniques.Source: §8 · p. 8
  • Research program codes often share verifiable results while giving less attention to readability and modularity; ChainerCV's closer library comparators include pytorch/vision and GluonCV.At the time of writing, pytorch/vision supported pretrained models only for image classification; GluonCV supported object detection and semantic and instance segmentation, but neither library made reproducibility its core goal.Source: §8 · p. 8

9 CONCLUSION

The conclusion presents Chainer as a framework used across several deep-learning applications. It highlights CuPy performance improvements and CPU/GPU-agnostic code, while anticipating further development through community collaboration.

9 CONCLUSION

  • Chainer has been used in deep reinforcement learning, distributed word2vec representations, recurrent language models, human pose estimation, and variational auto-encoders.Source: §9 · p. 8
  • CuPy performance improvements allow users to apply various deep-learning models with CPU/GPU-agnostic code.Source: §9 · p. 8
  • The authors anticipate that worldwide collaboration by Chainer developers and users on GitHub will make the framework more versatile and useful.Source: §9 · p. 8

A DETAILS OF EXPERIMENTAL SETUPS

This appendix provides detailed setups for experiments whose results appear in the main text.

A DETAILS OF EXPERIMENTAL SETUPS

A.1 Distributed Training

The distributed-training evaluation used specified software, a 128-node GPU cluster, and a 1024-worker setup with half-precision communication; ChainerMN was then a plugin library.

A.1 Distributed Training

  • The evaluation used development branches based on Chainer 3.0.0rc1 and ChainerMN 1.0.0, with NCCL 2.0.5 and OpenMPI 1.10.2.Source: §A.1 · p. 10
  • The in-house cluster had 128 nodes, each with two Intel Xeon E5-2667 processors, 256-GB memory, and eight NVIDIA Tesla P100 GPUs, interconnected by Mellanox Infiniband FDR.Source: §A.1 · p. 10
  • With 1024 workers, the per-worker minibatch was 32 and the total minibatch was 32k; computations generally used single precision and communication used half-precision floats.Source: §A.1 · p. 10
  • At evaluation time, ChainerMN was a Chainer plugin library; it has since been merged into Chainer's mainline.Source: §A.1 · p. 10
Loading 1908.00213v1…