PaperlayerView exact revision on arXiv

Source-linked AI summary

DyNet: The Dynamic Neural Network Toolkit

Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, Pengcheng Yin

arXiv:1701.03980v1stat.MLcs.CLcs.MS

TL;DR

Existing neural-network toolkits leave software-engineering challenges for naturally expressing and implementing dynamically structured architectures. DyNet unifies computation-graph declaration and execution through dynamic declaration, supported by an optimized backend to reduce graph-construction overhead. Case studies report efficiency comparable to static-declaration toolkits for standard architectures and simpler implementation for dynamic architectures.

  • Problem

    Existing static-declaration toolkits create software-engineering risks for dynamically structured neural networks and can make model implementation less natural.

  • Method

    DyNet uses dynamic declaration, unifying network-structure declaration and execution while minimizing computation-graph construction overhead with an optimized C++ backend.

  • Results

    DyNet achieves execution efficiency comparable to static-declaration toolkits for standard architectures and significantly simplifies implementation for dynamic architectures.

  • Takeaways & Limitations

    DyNet provides a toolkit for creating dynamic neural networks with conceptual simplicity and the ability to handle more complicated structures with minimal overhead.

  • Takeaways & Limitations

    Minibatching across more complex structures such as trees remains an open challenge for DyNet.

Abstract

from arXiv · show

We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet's dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet's speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released open-source under the Apache 2.0 license and available at http://github.com/clab/dynet.

1 Introduction

Deep learning shifts effort from feature engineering toward application-specific model engineering, making neural-network implementation and assessment a central challenge. DyNet addresses software-engineering risks in static declaration with dynamic declaration, while retaining comparable execution efficiency for standard architectures.

  • Motivation: Deep learning replaces application-specific feature engineering with application-specific model engineering, requiring ongoing development and evaluation of new model variants.This makes implementing models and assessing their performance part of routine deep-learning practice.
  • Existing tools: Existing toolkits simplify neural-network computation by providing primitives, parameter optimization, and automatic differentiation for task-specific predictions and losses.These capabilities reduce the engineering burden of implementing prediction and gradient-computation code.
  • Programming-model problem: Static declaration separates network declaration from execution, creating software-engineering risks for dynamically structured architectures such as variable-length sequences and recursive trees.The paper frames this separation as a barrier to expressing and implementing more complicated model structures naturally.
  • DyNet: DyNet is a toolkit based on dynamic declaration, a unified programming model in which declaration and execution are combined.It is presented as a proof of concept for reviving this alternative programming model.
  • Evaluation: DyNet achieves execution efficiency comparable to static-declaration toolkits for standard architectures while significantly simplifying implementations for dynamic architectures.The evaluation uses case studies in a single-machine environment.

2 Static Declaration vs. Dynamic Declaration

Static declaration separates computation-graph definition from execution, while dynamic declaration builds graphs procedurally during execution. Dynamic declaration is intended to simplify variable architectures, but requires low-overhead graph construction.

  • Static declaration: Static declaration first defines a computation graph, then repeatedly executes it on populated inputs to produce predictions or training gradients.
  • Static declaration: Static graphs can be optimized for repeated execution and scheduled across computational devices, but these benefits come with engineering trade-offs.
  • Challenges for static declaration: Variable-sized, variably structured, and dynamically branching inputs or outputs complicate fixed architectures and computation orders.
  • Challenges for static declaration: Expressing iteration, recursion, conditional execution, and complex inference in static graphs differs from imperative host-language code and can require considerable developer sophistication.
  • Dynamic declaration: Dynamic declaration unifies graph definition and execution, allowing each training example to use a different graph and host-language flow control.
  • Dynamic declaration: DyNet addresses dynamic declaration’s construction cost with an optimized C++ backend and graph design aimed at efficient graph construction and execution.

3 Coding Paradigm

DyNet programs build Expressions whose operations implicitly form a ComputationGraph, while persistent model parameters and trainers support repeated training examples. The coding paradigm also supports concise dynamic-structure implementations such as tree networks.

  • Coding paradigm overview: Expressions represent subcomputations, and operations combine them to implicitly build the computation graph in the background.
  • Coding paradigm overview: A DyNet model stores Parameters and LookupParameters, while a Trainer applies update rules such as stochastic gradient descent, AdaGrad, or Adam.
  • Training workflow: For each example, the program creates a graph, builds its expression, computes a forward result, back-propagates training loss, and updates model parameters.
  • Training workflow: Creating the graph inside the example loop permits flexible per-instance structures and native-language flow control, but makes fast graph construction necessary.
  • Dynamic graph examples: A tree-structured recursive network recursively encodes leaves, unary nodes, and binary nodes using expressions corresponding to each input tree.
  • Dynamic graph examples: The tree-network implementation uses 19 lines of code, while the benchmarked TreeLSTM implementation uses 39 lines of readable Python code.

4 Behind the Scenes

DyNet efficiently constructs a fresh computation graph for each training example or minibatch. Its lightweight graph representation, C++ backend, memory management, and tensor operations support rapid construction and execution.

  • DyNet creates new computation graphs efficiently for every training example or minibatch.Careful memory management stores forward and backward values so most time can be spent on computation.
  • A ComputationGraph is a directed acyclic graph of Node objects representing parameters, inputs, constants, and elementary-function results.Node shapes are inferred from their inputs, and incoming edges are stored as ordered references.
  • Forward functions compute each node’s result, while backward functions propagate loss derivatives and accumulate parameter gradients.Users implementing unsupported operations must provide corresponding forward and backward functions.
  • DyNet separates graph construction from execution: procedural operations create nodes, then forward and backward passes traverse the graph sequentially.This design contrasts with Chainer’s forward computation during graph construction and could permit graph optimization before execution.
  • The C++ backend, custom memory allocation, lightweight language wrappers, and Eigen operations target efficient graph building and tensor computation.The allocator advances pointers with simple integer arithmetic, while Cython minimizes Python wrapper overhead.

5 Higher Level Abstractions

DyNet provides higher-level Builders for recurrent, tree-structured, and large-vocabulary softmax models. These interfaces combine convenient abstractions with dynamic, sequence-oriented programming and efficient implementations.

  • DyNet Builders provide higher-level interfaces for recurrent networks, tree-structured networks, and large-vocabulary softmax functions.They are implemented on top of DyNet’s elementary differentiable operations.
  • Recurrent Neural Network Builders: RNNBuilders support Elman-style recurrent networks, LSTMs, and GRUs through a shared parent interface and concrete implementations.Canonical usage initializes the builder, starts a sequence, and adds inputs incrementally to obtain outputs.
  • Recurrent Neural Network Builders: Unlike static full-sequence APIs, DyNet’s canonical RNN interface allows inputs to be processed incrementally through procedural calls.Static APIs can share computations across inputs but require the full sequence to be homogeneous and known in advance.
  • Recurrent Neural Network Builders: DyNet’s sequence-processing API offers whole-sequence operations for users seeking fewer calls and efficiency gains while retaining the flexible canonical API.The paper reports significant improvements from the sequence-based interface, especially at smaller RNNLM settings.
  • Tree-structured Neural Network Builders: Tree-network Builders support recursive neural networks and tree-structured LSTMs with efficient, debugged implementations.The paper states that comparable implementations can also be written in a few dozen lines of Python.
  • Large-Vocabulary Softmax Builders: SoftmaxBuilder supports class-based and hierarchical softmax methods for probability distributions with large output spaces.Training can use neg_log_softmax(), while testing can sample outputs or compute the full distribution.

6 Efficiency Tools

DyNet includes sparse updates, minibatching, and multiprocessing to improve training efficiency. These tools reduce unnecessary parameter work, exploit vectorized hardware, and abstract parallel execution from the user.

  • DyNet improves computational efficiency through sparse updates, minibatching, and multiprocessing across CPUs.These features target different sources of training overhead.
  • Sparse Updates: Lookup parameters enable sparse updates because only vectors accessed by the current training instance receive non-zero gradients.Updating all parameters despite zero gradients can be wasteful when most parameters are untouched.
  • Sparse Updates: Sparse updates track accessed lookup vectors and update only parameters with non-zero gradients.The paper notes expected CPU speedups and describes caveats for GPU behavior.
  • Minibatching: Minibatching groups multiple examples for simultaneous processing, enabling matrix-matrix multiplication and parallel element-wise operations.These operations exploit vector-processing instructions on GPUs and CPUs.
  • Minibatching: DyNet treats minibatch size as a special dimension and performs broadcasting internally, reducing the need for users to manage batch dimensions.Users mainly provide multiple inputs and calculate losses using multiple labels.
  • Minibatching: Minibatching complex structures such as trees requires more complex algorithms and remains a future integration challenge.The paper identifies this as a scope boundary for the described minibatching support.
  • Multiprocessing: DyNet launches a pool of training processes and distributes data examples to workers after the user supplies a per-datum function.The toolkit handles internal data passing among workers.

7 Empirical Comparison

DyNet is evaluated against Theano, TensorFlow, and Chainer on four natural-language-processing tasks, with emphasis on speed, sparse updates, and implementation length. It generally outperforms the baselines on CPU and remains competitive on GPU, while dynamic interfaces simplify implementations.

  • Evaluation setup: DyNet is evaluated against Theano, TensorFlow, and Chainer on four natural language processing tasks using speed, accuracy, and implementation measures.The benchmarks include recurrent language modeling, bidirectional LSTM tagging, character-feature tagging, and TreeLSTM sentiment analysis.
  • Cross-toolkit speed: On CPU, DyNet is 1.66x to 3.20x faster than the fastest counterpart on RNNLM, with gains of 2.99x to 4.44x for BiLSTM tagging and 12.7x for TreeLSTM.The authors attribute these results to reducing graph-construction overhead and optimizing CPU and GPU speed.
  • Interface and sequence processing: Sequence-based computation substantially improves RNNLM speed, especially at smaller batch sizes, by sharing computations across time steps.The standard C++ and Python interfaces differ negligibly because Python wraps the core C++ implementation.
  • Cross-toolkit speed: On GPU, DyNet significantly outperforms other implementations for smaller-batch RNNLMs and all BiLSTM Tagger and TreeLSTM settings, while TensorFlow wins for RNNLM batches above 16.The reported GPU gap is smaller than on CPU, but DyNet remains competitive overall.
  • Sparse updates: Sparse updates produce speed gains ranging from 1.05x for RNNLM with minibatch size 16 to 20x for the BiLSTM tagger on CPU.The benefit is larger when sparse embeddings dominate computation and smaller when dense operations, such as a large softmax, remain costly.
  • Sparse updates: Sparse updates are only nominally faster on GPU in most cases and slower in some, making them more attractive for large embedding matrices with CPU training.The sparse-update comparisons are not directly comparable to the dense updates used in the other toolkits.
  • Implementation length: Dynamic-declaration implementations in Python are consistently shorter than Theano and TensorFlow implementations, indicating simpler implementations with fewer characters of code.The authors describe this conciseness as separate from dynamic graphs’ intuitive programming and debugging advantages.

8 Use Cases

DyNet has been used across a broad range of natural-language-processing projects, including parsing, machine translation, language modeling, tagging, and specialized linguistic tasks. Its use cases include both established architectures and newly developed models.

  • Syntactic Parsing: Parsing is DyNet’s most prominent use case, supporting stack LSTMs, bidirectional LSTM dependency-parsing features, recurrent neural network grammars, and hierarchical TreeLSTMs.These projects cover several tree- and sequence-structured parsing architectures.
  • Machine Translation: DyNet has supported machine-translation methods involving attention biases and character-based translation, as well as translation toolkits such as Lamtram and nmtkit.The cited applications include both individual methods and complete toolkit implementations.
  • Language and linguistic applications: DyNet has been used for hybrid neural/n-gram and generative syntactic language models, named entity recognition, part-of-speech tagging, and morphological inflection generation.Additional applications address coordination detection, semi-supervised preposition-sense disambiguation, and lexical semantic relations.

9 Conclusion

DyNet is a dynamic neural-network toolkit that simplifies complicated architectures while maintaining efficient execution. The conclusion identifies multi-device execution, automatic graph optimization, broader operation and optimizer support, and community contributions as ongoing directions.

  • DyNet supports complicated dynamic neural-network structures with conceptual simplicity and minimum overhead.
  • Multi-device Support: DyNet currently executes on a single GPU or CPU and does not support model parallelism.Future work targets executing a single computation graph across multiple devices.
  • On-the-fly Graph Optimization: Automatic on-the-fly graph optimization could combine similarly shaped operations while preserving DyNet’s intuitive interface.
  • Support for Operations and Optimizers: DyNet continues expanding its supported operations and optimizers to enable implementation of new methods from the literature.
  • DyNet is presented as a community effort that welcomes contributions to improve the toolkit.
Loading 1701.03980v1…