Source-linked AI summary
Theano: new features and speed improvements
Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley, Yoshua Bengio
TL;DR
The paper addresses how to make Theano’s symbolic, graph-based machine-learning computations more capable and efficient. It adds symbolic-loop and differentiation features, improves runtime and parallel execution, and benchmarks performance against Torch7 and RNNLM. Theano is faster than competing software in most tested cases, while its flexible workflow introduces additional complexity.
Problem
The paper investigates how to extend Theano’s capabilities and improve efficiency while evaluating its performance relative to competing machine-learning software.
Method
The paper adds Scan and R-operator support, runtime optimizations, asynchronous GPU execution, CPU parallelism, and performance-focused input and iteration options, then benchmarks Theano against Torch7 and RNNLM.
Results
Theano is faster than competing software in most cases across the reported neural-network and recurrent-neural-network benchmarks, with performance depending on batch size, model size, and workload.
Takeaways & Limitations
The benchmarks expose relative strengths so users can choose the software that best suits their needs.
Takeaways & Limitations
Theano’s symbolic optimization and differentiation require a more complex workflow and make compilation or execution errors harder to interpret.
Abstract
from arXiv · showhide
Theano is a linear algebra compiler that optimizes a user's symbolically-specified mathematical computations to produce efficient low-level implementations. In this paper, we present new features and efficiency improvements to Theano, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
1 Introduction
Theano is a symbolic mathematical compiler for machine learning that automatically derives gradients and compiles efficient executable functions. The paper introduces new features and speed improvements, then benchmarks Theano against Torch7 and RNNLM.
- Theano symbolically defines mathematical functions, derives gradient expressions automatically, and compiles them into executable functions.
- The paper presents new Theano features and measures intended to improve implementation speed.
- The evaluation compares Theano with Torch7 on neural-network benchmarks and with RNNLM on recurrent-neural-network benchmarks.
2 Main features of Theano
Theano represents symbolic mathematical expressions as optimizable graphs, supports symbolic differentiation, and leverages NumPy, SciPy, CUDA, and community-oriented development practices. These capabilities support model prototyping and efficient numerical computation across dense, sparse, CPU, and GPU settings.
- Theano’s graph optimizations eliminate redundant computations, improve numerical stability, and accelerate array operations through loop fusion.
- Symbolic differentiation supports forward-mode R-operator computation, gradient backpropagation, and gradients through Scan-specified loops.
- Theano uses NumPy and SciPy implementations, while allowing optimized C++ and BLAS-based replacements for operations such as sparse-matrix computations.
- CUDA support provides GPU-resident dense arrays and generated implementations of mathematical operations, mostly for single-precision floating-point data.
- The project includes automated testing, stable releases, and an active user community supporting code quality and correctness.
3 New features in Theano
The paper adds symbolic-loop and forward-mode differentiation features while improving runtime execution, sparse operations, CPU parallelism, and asynchronous GPU computation. These changes expand recurrent-model support and reduce overhead in compiled computations.
- 3.1 Scan: Symbolic Loop in Theano: Scan abstracts an entire loop into one graph node linked to a symbolic graph for the loop’s internal computation.This avoids the limitations of Python loops and fully unrolled graphs, while supporting symbolic iteration counts or conditions.
- 3.1 Scan: Symbolic Loop in Theano: Scan supports efficient gradients, vector-Jacobian products, R-operator evaluation, and recurrent architectures without manually deriving gradients or optimizing implementations.
- 3.2 R-operator for Hessian-Free optimization: The R-operator evaluates directional derivatives, and composing TT.Lop(f, θ, TT.Rop(f, θ, γ)) reduces a recurrent Gauss-Newton computation from three passes to two.
- 3.3 Lazy Evaluation, CVM: The VM enables lazy evaluation by executing only branches needed for the output, such as the necessary branch of an if-then-else operation.
- 3.3 Lazy Evaluation, CVM: The CVM executes available C operations directly, avoiding Python interpreter returns and function-call overhead, especially for many operations on small operands.
- 3.5 Better support for sparse matrices: Theano improves sparse support for CSR and CSC matrices, including efficient operations needed to compute sparse derivatives.
- 3.6 Parallelism on CPU: OpenMP support enables parallel Theano operations, including a parallel implementation of 2-dimensional convolution.
- 3.7 Asynchronous function calls on GPU: The new GPU default defers waiting for kernel results until necessary, allowing CPU computation to run concurrently with GPU computation.The previous behavior remains available for profiling GPU-kernel execution time.
4 Benchmarks
The benchmarks compare Theano with Torch7 and RNNLM to expose their relative strengths across neural-network and recurrent-neural-network workloads. Results vary with batch size, model size, device, and computational intensity.
- Earlier studies reported conflicting comparisons: Theano initially outperformed Torch5, while Torch7 later outperformed Theano on the same benchmarks.
- The evaluation tests current Torch7 versions on neural-network tasks and compares Theano with RNNLM for recurrent-neural-network training.
4.1 Torch7
Torch7 and Theano are machine-learning computing frameworks designed to support rapid model development while retaining optimized execution. They share numerical libraries and CPU/GPU capabilities, but differ in language-level overhead, parallelism, and Theano’s graph-based workflow.
- Torch7 and Theano both ease implementation and testing of machine-learning models without abandoning manually optimized execution speed.
- Both frameworks use scripting languages, optimized scientific libraries, and C/C++ modules for performance-critical computation.
- Torch7 supports OpenMP multi-core CPU and CUDA GPU computation, while Theano’s non-BLAS CPU parallelization was only beginning to be explored.
- Lua’s lower interpreter and C-integration overhead can give Torch7 an advantage for simple functions and small data quantities.
- Theano’s graph optimization and symbolic differentiation are powerful, but require graph definition, compilation, and execution phases that make the workflow and errors more complex.
4.2 Benchmarking on Deep Learning Tasks
The deep-learning benchmarks compare Torch7 and Theano across neural-network architectures, execution devices, batch sizes, and Theano optimizations. Theano generally matches or exceeds Torch7, with performance depending on workload, device, and batch size.
- How to boost Theano’s performance: Theano’s default configuration includes the CVM runtime and asynchronous GPU operation execution.The benchmark’s left-most blue bar represents this default configuration.
- How to boost Theano’s performance: Disabling garbage collection increases memory usage but speeds execution, especially on GPUs where synchronization can undermine asynchronous kernel execution.
- How to boost Theano’s performance: Disabling input checks and conversions improves performance when inputs already have the expected type, but can raise execution-time exceptions for incompatible inputs.
- How to boost Theano’s performance: Calling f.fn(n_calls=N) bypasses the Python loop for repeated no-input function calls, returning only the final iteration’s outputs.This is useful when parameter updates matter more than intermediate function outputs, such as iterative training.
- Experimental Setup: Figure 1 measures examples per second for logistic regression, a one-hidden-layer network, and a three-layer deep network on CPU and GPU.CPU results include one-thread and four-thread OpenMP configurations; Torch7 uses standard Lua or LuaJIT, while Theano uses progressively optimized configurations.
- Results: With no mini-batches, Theano beats Torch7 on CPU models containing hidden layers, while Torch7 leads on CPU logistic regression and GPU batch-size-one cases.
- Results: With mini-batches of 10 or 60, Theano is faster than or equivalent to Torch7 across all three architectures.The difference disappears on the most computationally intensive tasks as language and framework overhead becomes negligible.
4.3 Benchmarking on Recurrent Neural Networks
The recurrent-network benchmark compares Theano with RNNLM using batch size one and reports sequence elements processed per second. RNNLM is faster for smaller models, while Theano catches up as model size increases.
- Benchmark Setup: Figure 2 benchmarks a simple recurrent network using sequence elements per second, with batch size one and matched input and output unit counts.
- Results: RNNLM is faster than Theano on smaller recurrent models, but Theano quickly catches up for larger sizes.The paper attributes this pattern mostly to Theano’s overhead, identified as a drawback of its recurrent-model flexibility.
5 Conclusion
The paper presents new Theano features and performance improvements, then evaluates them against competing software on multiple benchmarks. It concludes that Theano is faster than competing software in most tested cases and frames the benchmarks as guidance for tool selection and improvement.
- The paper presents recent Theano additions that make it more powerful for machine-learning software development and faster than competing software in most benchmark cases.
- The benchmarks expose relative strengths so users can choose the software that best suits their needs.