PaperlayerView exact revision on arXiv

Source-linked AI summary

Theano: A Python framework for fast computation of mathematical expressions

The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, Alexander Belopolsky, Yoshua Bengio, Arnaud Bergeron, James Bergstra, Valentin Bisson, Josh Bleecher Snyder, Nicolas Bouchard, Nicolas Boulanger-Lewandowski, Xavier Bouthillier, Alexandre de Brébisson, Olivier Breuleux, Pierre-Luc Carrier, Kyunghyun Cho, Jan Chorowski, Paul Christiano, Tim Cooijmans, Marc-Alexandre Côté, Myriam Côté, Aaron Courville, Yann N. Dauphin, Olivier Delalleau, Julien Demouth, Guillaume Desjardins, Sander Dieleman, Laurent Dinh, Mélanie Ducoffe, Vincent Dumoulin, Samira Ebrahimi Kahou, Dumitru Erhan, Ziye Fan, Orhan Firat, Mathieu Germain, Xavier Glorot, Ian Goodfellow, Matt Graham, Caglar Gulcehre, Philippe Hamel, Iban Harlouchet, Jean-Philippe Heng, Balázs Hidasi, Sina Honari, Arjun Jain, Sébastien Jean, Kai Jia, Mikhail Korobov, Vivek Kulkarni, Alex Lamb, Pascal Lamblin, Eric Larsen, César Laurent, Sean Lee, Simon Lefrancois, Simon Lemieux, Nicholas Léonard, Zhouhan Lin, Jesse A. Livezey, Cory Lorenz, Jeremiah Lowin, Qianli Ma, Pierre-Antoine Manzagol, Olivier Mastropietro, Robert T. McGibbon, Roland Memisevic, Bart van Merriënboer, Vincent Michalski, Mehdi Mirza, Alberto Orlandi, Christopher Pal, Razvan Pascanu, Mohammad Pezeshki, Colin Raffel, Daniel Renshaw, Matthew Rocklin, Adriana Romero, Markus Roth, Peter Sadowski, John Salvatier, François Savard, Jan Schlüter, John Schulman, Gabriel Schwartz, Iulian Vlad Serban, Dmitriy Serdyuk, Samira Shabanian, Étienne Simon, Sigurd Spieckermann, S. Ramana Subramanyam, Jakub Sygnowski, Jérémie Tanguay, Gijs van Tulder, Joseph Turian, Sebastian Urban, Pascal Vincent, Francesco Visin, Harm de Vries, David Warde-Farley, Dustin J. Webb, Matthew Willson, Kelvin Xu, Lijun Xue, Li Yao, Saizheng Zhang, Ying Zhang

arXiv:1605.02688v1cs.SCcs.LGcs.MS

TL;DR

The paper presents Theano as a system for representing mathematical expressions as computation graphs and compiling them into efficient executable functions. It describes symbolic differentiation, extensible operators, GPU support, and recent improvements, finding performance comparable to major alternatives while identifying Python’s GIL as a limitation.

  • Problem

    Theano addresses the need to express and execute mathematical models efficiently while supporting symbolic differentiation, graph optimization, and CPU/GPU computation.

  • Method

    The paper describes typed computation graphs, symbolic differentiation, extensible operators, GPU backends, graph optimizations, and runtime mechanisms for compiling and executing Theano functions.

  • Results

    Theano’s performance is comparable to Torch and TensorFlow, with convolutional-network performance slightly slower but comparable in forward and backward passes.

  • Takeaways & Limitations

    Theano pioneered combining high-level scripting with optimized kernels, symbolic graphs, symbolic differentiation, graph rewriting, and automatic kernel compilation.

  • Takeaways & Limitations

    Python’s GIL limits concurrent thread execution, and short functions may leave threads waiting for the lock rather than computing.

Abstract

from arXiv · show

Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements. Theano is being actively and continuously developed since 2008, multiple frameworks have been built on top of it and it has been used to produce many state-of-the-art machine learning models. The present article is structured as follows. Section I provides an overview of the Theano software and its community. Section II presents the principal features of Theano and how to use them, and compares them with other similar projects. Section III focuses on recently-introduced functionalities and improvements. Section IV compares the performance of Theano against Torch7 and TensorFlow on several machine learning models. Section V discusses current limitations of Theano and potential ways of improving it.

I. OVERVIEW

Theano combines Python’s accessible interface with an optimized CPU/GPU computation engine for symbolic mathematical expressions. Its NumPy-like API, extensibility, community, and downstream frameworks supported broad machine-learning use.

  • Theano compiles symbolic multi-dimensional-array expressions into optimized CPU or GPU computations, while automatically differentiating and optimizing the computation graph.Optimizations include pruning unnecessary variables, reusing partial results, in-place operations, and numerical-stability transformations.
  • Theano addresses Python’s slow and memory-intensive numerical execution by pairing Python’s flexibility with a fast computation engine.
  • Its NumPy-like API lets users transfer familiar array-oriented code while gaining automatic gradients, numerical improvements, and high-performance execution without changing user code.Custom graph expressions can also be written in Python, C++, or CUDA.
  • Theano is open-source, BSD-licensed, and supported by an active worldwide developer and user community.
  • Higher-level packages such as Pylearn2, Blocks, Lasagne, Keras, and PyMC3 build on Theano for machine learning and probabilistic programming.

A. Mathematical expressions

Theano represents mathematical expressions as typed directed acyclic graphs whose variables and operation applications support symbolic construction, differentiation, and graph reuse. Its differentiation system builds gradients symbolically through reverse-mode traversal and also supports forward-mode products.

  • 1. Graph structure: Theano graphs are bipartite directed acyclic graphs containing Variable nodes for data and Apply nodes for mathematical operations.Variables can feed multiple operations, while each computed variable has at most one producing operation, matching SSA form.
  • 1. Graph structure: Strongly typed variables represent dense CPU arrays, legacy or new GPU arrays, and sparse matrices, with types known during graph construction.
  • 1. Graph structure: Users construct graphs by creating typed symbolic inputs and applying Python functions, and can clone or reconnect graph parts through replacements.
  • 3. Symbolic differentiation: Reverse-mode differentiation traverses the graph backward, calling each Op’s grad method to build symbolic gradients that can be reused for learning rules or higher-order derivatives.
  • 3. Symbolic differentiation: Forward-mode differentiation is supported through the R operator, which traverses the graph from inputs to outputs to compute Jacobian-vector products.

4. Scan: Symbolic loops

Because Theano’s fixed acyclic graphs make data-dependent or unknown-length loops difficult to express, it provides Scan as a graph-level loop abstraction. Scan supports differentiation and integrates with compilation and optimization.

  • Fixed graph structure makes symbolic loops challenging, especially when iteration counts depend on data or sequence length.Explicit unrolling duplicates each iteration and cannot represent unknown-length or variable-count loops.
  • Scan represents an entire loop as one Apply node containing an isolated inner computation graph and managing communication with the outer graph.
  • Scan’s gradient is itself a Scan over reversed sequences, reproducing the gradient of an unrolled loop through back-propagation through time.
  • Theano compilation clones the relevant graph, applies optimizations, generates optimized C++ or CUDA code, and returns a callable function.
  • Graph rewrites can replace equivalent variables locally or globally, enabling canonicalization, numerical stabilization, operation fusion, GPU substitutions, and in-place computation.Examples include x*x ⇒ x^2, stable log1p transformations, and fused element-wise operations.

2. Shared variables

Shared variables provide persistent symbolic state across Theano functions, with explicit update rules that can modify stored arrays during execution. Compilation and runtime mechanisms support optimized CPU/GPU execution, caching, extensibility, and profiling.

  • Shared variables are persistent symbolic inputs whose values are available across Theano functions and are implicit inputs whenever referenced.
  • Update expressions act as implicit outputs and assign new shared-variable values after function execution, sometimes enabling in-place array updates.
  • Shared variables can be reassigned outside functions, may change shape, and are created on the GPU by default when GPU execution is enabled under the stated backend condition.
  • Theano avoids repeated code generation and compilation through a persistent on-disk cache, while Python and C++/CUDA implementations make new operations extensible.A Python runtime remains available for instrumentation, profiling, and debugging callbacks.
  • The runtime VM schedules required operations using variable storage, ordering constraints, computation functions, and execution state.The default C VM can directly execute C implementations of Ops, avoiding Python-call overhead for many small operations.

E. Related software

Theano is positioned as a symbolic computation core alongside higher-level machine-learning frameworks, differing from projects such as TensorFlow and Torch7 in execution scope and differentiation support. Recent improvements emphasize faster execution, broader GPU and multi-GPU support, graph optimization, and usability.

  • Positioning: Theano is a core symbolic computation system rather than a deep learning framework, although higher-level frameworks are built on top of it.It supports mathematical-model definition and automatic gradient computation, while packages such as Pylearn2, Blocks, Lasagne, and Keras provide higher-level interfaces.
  • Comparisons: TensorFlow shares Theano’s graph-compiling and symbolic-differentiation approach but focuses additionally on distributed multi-node computation.The passage contrasts TensorFlow’s graph rewriting with Theano-style mathematical simplification and kernel fusion.
  • Comparisons: Torch7 provides efficient C CPU and GPU kernels through Lua, while higher-level packages supply layer-based gradients and parameter updates.Autograd extends Torch with automatic differentiation by recording expression evaluation.
  • Comparisons: Other frameworks vary in flexibility and execution strategy, including MXNet’s symbolic layer gradients and distributed computation, Neon’s optimized GPU kernels, and Chainer’s dynamic graphs.These systems expose different trade-offs in symbolic representation, device dispatch, and graph construction.
  • Recent improvements: Recent Theano improvements target faster execution, broader GPU and multi-GPU support, faster optimization for large graphs, and better error reporting, visualization, and debugging.These changes are presented as improvements to both performance and ease of use.
  • Convolution interfaces: The growing number of convolution implementations creates a need for an interface that switches among methods with different speed, memory, and dependency trade-offs.Theano addresses this with abstract convolution Ops that can be replaced during compilation by optimized implementations.

2. Using cuDNN

Theano’s GPU improvements combine flexible convolution dispatch, pooled memory management, faster Scan compilation and execution, and a new libgpuarray backend. The backend adds data types, multiple-GPU model parallelism, asynchronous execution, and limited OpenCL support, while some kernel tuning and parallelism constraints remain.

  • Using cuDNN: Theano wraps cuDNN convolution and gradient implementations and lets users select algorithms globally or per graph node.Selection can be explicit or based on timing and shape-change policies.
  • GPU memory: CNMeM reduces GPU allocation overhead by maintaining large memory pools and returning reusable chunks instead of repeatedly synchronizing through cudaFree.Theano can reserve a configurable fraction of GPU memory for CNMeM.
  • Scan: Scan improvements reduce graph optimization and compilation time while also improving execution speed and stability.New optimizations move more computation outside loops, and the backend writes directly into output buffers rather than copying intermediate results.
  • Scan: Rewriting Scan’s gradient method improves scaling for many inputs and outputs, with nested-loop compilation sometimes falling from hours to minutes.The cleaner resulting graph requires less rewriting and can also execute faster.
  • New backend: The libgpuarray backend supports common data types including float16, views and strides, 64-bit indexing, and model parallelism across multiple GPUs.Multiple-GPU execution can nevertheless lose parallelism in some cases because of Python’s Global Interpreter Lock.
  • New backend: OpenCL support is basic and remains incomplete because many Theano GPU Ops do not yet support it.The paper identifies porting effort as a possible path to broader support.
  • New backend: The backend makes computation and data transfers asynchronous so CPU work, GPU work, and transfers can overlap when dependencies permit.It supports almost all operations of the old CUDA backend and cuDNN, but some kernels still need performance tuning, especially for int64 indexing.

6. Data parallelism with Platoon

Platoon addresses Python’s inability to use ordinary multi-threaded data parallelism by coordinating multiple worker processes that train synchronized model copies. It uses shared memory and asynchronous communication, with ASGD and EASGD currently available for parameter updates.

  • Data parallelism: Data parallelism splits input data across multiple copies of a model, requiring synchronization and aggregation of worker results.Model synchronization prevents copies from drifting too far apart during training.
  • Motivation: Python’s Global Interpreter Lock makes the usual multi-threaded approach to single-machine data parallelism unworkable.Platoon addresses the resulting need for multiple processes when training Theano models.
  • Platoon: Platoon uses a central controller and worker processes, each training a model copy on CPU or GPU, while shared memory reduces inter-process communication overhead.Worker communication with the controller is asynchronous, so workers do not wait for replies.
  • Optimization: Platoon currently supports Asynchronous SGD and Elastic Averaging SGD for updating central parameters.Additional synchronization rules can be added through new implementations.
  • Development workflow: The fast_compile optimizer now includes GPU-transfer optimizations, shortening graph optimization time at the cost of slightly slower execution and higher memory use.This trade-off is useful for development and prototyping.
  • Function reuse: Copied Theano functions reuse the original optimized graph, so similar functions can be created without recompiling.Copies can swap shared variables and update parameters, and may share intermediate storage to save memory and increase speed.

3. Save and reload optimized graphs

Theano supports saving optimized computation graphs for later restoration without repeating optimization, while adding visualization and diagnostic tools for understanding graph execution. These tools expose profiling and graph structure but test values are computed before runtime optimizations and can therefore produce misleading NaNs.

  • Save and reload: Optimized Theano graphs can be serialized and reloaded without being optimized again, supporting checkpointing and restoration of running experiments.Re-optimization can be forced when optional dependencies change, because removing or adding dependencies may make a function fail or become suboptimal.
  • Interactive visualization: The d3viz module exports interactive HTML visualizations of computation graphs instead of only text or static images.The graphs can be explored in a browser, with profiling colors, graph navigation, node and edge information, and nested-graph expansion.
  • Interactive visualization: Figure 1 uses redder nodes for longer computation times, blue arrows for returned views, and red arrows for destroyed inputs.These encodings connect profiling information with memory behavior in the computation graph.
  • Test values: Test values computed while building a graph help detect shape mismatches and unexpected intermediate values before execution.They are associated with inputs and propagated automatically to intermediate variables.
  • Test values: Because test values are computed only once during graph construction, they do not receive later numerical-stability optimizations and may produce NaNs absent from the optimized graph.This makes test-value diagnostics useful but not equivalent to runtime behavior.
  • Runtime diagnostics: NanGuardMode checks inputs and outputs of every Apply node during execution and raises an error when problematic values are detected.It targets symptoms such as NaNs, infinities, and very large values, which can have several underlying causes.

4. The PdbBreakPoint Op

The section introduces Theano’s benchmarking scope and reports comparable convolutional-network performance against Torch and TensorFlow, with a modest fast-compile trade-off.

  • 4. The PdbBreakPoint Op: PdbBreakPoint checks a symbolic condition during function execution and enters the Python debugger with monitored variable values when the condition is met.It is useful for failures that appear only after many training iterations.
  • Benchmark scope: The benchmarks compare Theano, Torch, and TensorFlow across convolutional, recurrent, and sequence-to-sequence models, including multi-GPU scaling with Platoon.Experiments used publicly available software where possible.
  • B. Convolutional networks: The convolutional-network benchmarks measure forward and backward processing time per minibatch across four ImageNet models.The models are AlexNet, OverFeat, VGG, and GoogLeNet.
  • B. Convolutional networks: Theano is slightly slower than Torch and TensorFlow on convolutional networks, but its forward and backward performance is comparable.Figure 2 reports milliseconds per batch, with lower values indicating better performance.

C. Recurrent neural networks: LSTM on Penn Treebank

The recurrent-network experiments evaluate LSTM variants on Penn Treebank and extend the analysis to sequence-to-sequence video captioning and multi-GPU training. Theano is competitive across these settings, while synchronization frequency strongly affects multi-GPU scaling.

  • C. Recurrent neural networks: LSTM on Penn Treebank: The Penn Treebank experiments compare Torch, TensorFlow, and Theano implementations on small, medium, and large LSTM models using training words per second.The models vary in layers, hidden units, and sequence length, with dropout and batch size 20.
  • C. Recurrent neural networks: LSTM on Penn Treebank: Theano ranks behind TensorFlow on the small LSTM but is slightly faster on the medium and large models.Torch is slower than Theano on all three models and slower than fast-compile Theano on the two larger models.
  • D. Sequence-to-sequence: Caption generation from video: The sequence-to-sequence experiment generates English descriptions from video-frame representations using an LSTM conditioned on a weighted sum of frame representations.Each frame is represented by a 1024-dimensional vector, and processing time is measured per minibatch.
  • D. Sequence-to-sequence: Caption generation from video: For video captioning, Theano has a small forward-pass advantage and a backward-pass disadvantage, with comparable total time overall.Theano is slightly faster on smaller batches, whereas TensorFlow is faster on larger ones.
  • E. Data parallelism for LSTM: Adding GPUs consistently increases LSTM processing speed, but synchronizing after every batch produces sub-linear scaling because of communication overhead.Synchronizing every 100 batches yields speed-ups close to the theoretical optimum: 2 for 2 GPUs and 3.9–4 for 4 GPUs.

V. LIMITATIONS AND CHALLENGES

Theano’s limitations arise from Python-related concurrency constraints, graph-optimization scaling, compilation overhead, and the substantial effort required for proposed architectural changes.

  • Python limitations: Python’s GIL limits concurrent execution because short functions repeatedly reacquire the lock when accessing Python and NumPy objects.Although compiled modules can execute quickly and potentially release the GIL during computation, Python-object reference management still requires reacquiring it.
  • Python limitations: Independent Python interpreters cannot run in different threads of the same process, unlike in Lua.
  • Potential architectural changes: Replacing Python objects with a C++-accessible array structure would require thread-safe memory management and rewriting existing C++ and CUDA Ops.The change could also make it harder to create new Ops by integrating existing Python code.
  • Optimization scalability: Graph optimization scales supra-linearly with graph size because repeated local-optimization passes become more numerous as graphs grow.A one-pass or two-pass redesign could improve scaling, but preserving stability optimizations would make it a large-scale project.
  • Compilation overhead: The same Op can generate many distinct C++ or CUDA modules, making compilation and filesystem loading costly.Passing compile-time properties dynamically at runtime could alleviate this overhead.

D. Loops and control-flow structures

Theano’s control-flow and scaling challenges motivate more flexible graph representations, memory-management strategies, and broader interoperability with other frameworks and research software.

  • D. Loops and control-flow structures: More flexible control flow is needed for attention mechanisms, nested or recursive loops, and shape changes across iterations.Theano currently expresses loops and conditionals through Scan and ifelse lazy Op.
  • D. Loops and control-flow structures: TensorFlow-style switch and merge nodes could support symbolic loops, but would require graph cycles, runtime recomputation, and rewritten Scan optimizations.
  • D. Loops and control-flow structures: Limited GPU memory can bottleneck training by restricting model size and forcing smaller batch sizes that under-use GPU processing power.Proposed responses include lower-precision storage, reordered execution, and moving intermediate values between memories.
  • D. Loops and control-flow structures: Theano and TensorFlow compile computation graphs before execution, whereas Torch executes expressions immediately like an interpreter.The paper suggests exploring JIT compilation to combine interpreter flexibility with graph compilation.
  • VI. CONCLUSION: Theano pioneered combining high-level scripting, optimized GPU kernels, symbolic graphs, and symbolic differentiation for gradient-based computation.Graph rewriting, optimization, and automatic kernel compilation are also becoming more widely used.
  • VI. CONCLUSION: Theano’s computation performance is on par with major research software such as Torch and TensorFlow while incorporating continued functionality and performance improvements.These improvements include cuDNN integration and data- and model-parallel distributed computation.
  • VI. CONCLUSION: Longer-term improvements could draw on machine-learning software, computer algebra systems, language design, and compiler design.
Loading 1605.02688v1…