Source-linked AI summary
Blocks and Fuel: Frameworks for deep learning
Bart van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde-Farley, Jan Chorowski, Yoshua Bengio
TL;DR
Training complex neural networks on large datasets requires both manageable model construction and practical data processing. The paper introduces Blocks for Theano-based graph management and training utilities, and Fuel for standardized dataset access and preprocessing; together they support reproducible, resumable experiments.
Problem
Researchers need practical support for prototyping complex neural networks and processing large datasets during training.
Method
Blocks provides parametrized Theano operations, graph annotations, and training utilities, while Fuel provides standardized dataset interfaces, iteration, and on-the-fly preprocessing.
Results
Blocks experiments can be interrupted, serialized, and resumed mid-dataset pass without affecting final training results.
Takeaways & Limitations
The frameworks support manageable, reproducible neural-network experiments by integrating model construction, data handling, training, and checkpointing.
Abstract
from arXiv · showhide
We introduce two Python frameworks to train neural networks on large datasets: Blocks and Fuel. Blocks is based on Theano, a linear algebra compiler with CUDA-support. It facilitates the training of complex neural network models by providing parametrized Theano operations, attaching metadata to Theano's symbolic computational graph, and providing an extensive set of utilities to assist training the networks, e.g. training algorithms, logging, monitoring, visualization, and serialization. Fuel provides a standard format for machine learning datasets. It allows the user to easily iterate over large datasets, performing many types of pre-processing on the fly.
1. Introduction
Blocks and Fuel target researchers prototyping complex neural networks, with Blocks preserving Theano’s flexibility and Fuel addressing dataset processing and iteration.
- Blocks focuses on research and rapid prototyping of complex neural network models while maintaining Theano’s computational-graph flexibility.
- Fuel addresses dataset processing by providing tools to download datasets and iterate over and preprocess them efficiently.
- Both frameworks were developed with software-engineering practices including high test coverage, thorough documentation, and carefully considered APIs.
2. Blocks
Blocks organizes Theano-based neural-network development through reusable parametrized operations, graph annotations, and integrated training utilities.
- Bricks: Blocks provides reusable neural-network components called bricks, implemented as parametrized Theano operations.
- Bricks: Bricks use shared parameters to transform symbolic Theano variables and can contain nested bricks for hierarchical model configuration.
- Graph management: Blocks annotates graph variables with inputs, outputs, parameters, and model roles so complex Theano graphs remain manageable.
- Graph management: Graph queries let users apply model-agnostic transformations, including fine-grained regularization such as weight noise on selected LSTM weights.
- Training: The training system combines a Theano graph, training algorithm, and Fuel data stream through a main loop with monitoring, serialization, scheduling, plotting, and logging extensions.
3. Fuel
Fuel standardizes access to machine-learning datasets while supporting flexible iteration, chained preprocessing, large-scale storage, and dataset integration.
- Fuel: Fuel provides a common interface for published datasets and makes interfaces to new datasets easy to write.
- Iteration and preprocessing pipeline: It supports sequential or shuffled minibatches, in-memory or out-of-core datasets, and resampling methods such as cross-validation and bootstrapping.
- Iteration and preprocessing pipeline: Fuel performs on-the-fly preprocessing, including random image crops and text n-grams, with transformations that can be chained.
- Iteration and preprocessing pipeline: Fuel can run data operations in a separate process and transfer processed data to training through TCP sockets to sidestep Python’s GIL.
- Standardized data format: Built-in datasets are converted to annotated HDF5 files containing metadata about sources, splits, availability, and axis semantics.
- Standardized data format: Although HDF5 is the convention, Fuel’s dataset API also supports objects backed by other storage systems.
- Standardized data format: Fuel includes scripts for downloading raw data, converting it to HDF5, and recording module versions, interface versions, and conversion commands.
4. Serialization and checkpointing
Blocks supports transparent, reproducible checkpointing for long neural-network training runs, including interruption and resumption during a dataset pass.
- Blocks targets transparent experiment resumption across platforms while preserving reproducibility for training runs that may take days or weeks.
- Because Pickle cannot serialize many Fuel-dependent iterators, Blocks reimplemented itertools components to make them serializable.
- Experiments can be interrupted mid-pass, serialized, and resumed later without affecting final training results.
5. Documentation and community
Blocks and Fuel are supported by extensive documentation, active user communities, contributed examples, and public implementations of advanced neural network models.
- Both libraries provide API documentation and online tutorials for users.
- Two active mailing lists and a separate repository support users and contributions of non-trivial examples.
- Public examples include successfully implemented neural machine translation and Deep Recurrent Attentive Writer models.