PaperlayerView exact revision on arXiv

Source-linked AI summary

fastai: A Layered API for Deep Learning

Jeremy Howard, Sylvain Gugger

arXiv:2002.04688v2cs.LGcs.CVcs.NEstat.ML

TL;DR

fastai addresses the challenge of combining ease of use, productivity, flexibility, and configurability in deep learning software. It uses a layered API with high-level application components and lower-level composable foundations, and the authors report faster course development, positive early results, and benefits across research, practice, and teaching.

  • Problem

    Deep learning users need concise, productive interfaces without losing the flexibility to customize models, data processing, and underlying components.

  • Method

    fastai uses a layered architecture with reusable high-level application APIs, mid-level methods, low-level primitives, and shared abstractions such as type dispatch and separated data transforms.

  • Results

    The authors report rewriting a 14-hour, seven-module deep learning course more quickly and easily while replicating or improving its previous results.

  • Takeaways & Limitations

    The authors conclude that layered APIs can help researchers combine ideas, practitioners prototype and optimize without rewriting code, and students experiment with less boilerplate.

  • Takeaways & Limitations

    The paper covers fastai v2, which was in pre-release, and notes that further information about nbdev was planned for a future paper.

Abstract

from arXiv · show

fastai is a deep learning library which provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains, and provides researchers with low-level components that can be mixed and matched to build new approaches. It aims to do both things without substantial compromises in ease of use, flexibility, or performance. This is possible thanks to a carefully layered architecture, which expresses common underlying patterns of many deep learning and data processing techniques in terms of decoupled abstractions. These abstractions can be expressed concisely and clearly by leveraging the dynamism of the underlying Python language and the flexibility of the PyTorch library. fastai includes: a new type dispatch system for Python along with a semantic type hierarchy for tensors; a GPU-optimized computer vision library which can be extended in pure Python; an optimizer which refactors out the common functionality of modern optimizers into two basic pieces, allowing optimization algorithms to be implemented in 4-5 lines of code; a novel 2-way callback system that can access any part of the data, model, or optimizer and change it at any point during training; a new data block API; and much more. We have used this library to successfully create a complete deep learning course, which we were able to write more quickly than using previous approaches, and the code was more clear. The library is already in wide use in research, industry, and teaching. NB: This paper covers fastai v2, which is currently in pre-release at http://dev.fast.ai/

1. Introduction

fastai is designed to combine approachable, productive high-level APIs with deeply hackable, configurable lower-level components. Its layered architecture supports concise application development while preserving composability and direct access to underlying foundations.

  • Library and goals: fastai is an open-source deep learning library with documentation, tutorials, and support for vision, text, tabular, and time-series applications.It is available under the Apache 2 license and is also the subject of a fastai and PyTorch book.
  • Library and goals: Its two design goals are rapid productivity for practitioners and hackability for researchers, combining concise APIs with customizable models and sensible defaults.The high-level API is built on lower-level APIs that provide composable building blocks.
  • High-level API: Intelligent defaults integrate application information and best practices, including automatic loss selection and transfer-learning behaviors such as layer freezing and discriminative learning rates.These defaults reduce the information users must re-specify and make user code more readable.
  • Layered architecture: The mid-level API supplies application-specific deep learning and data-processing methods, while low-level APIs expose optimized primitives and foundations built on PyTorch and other libraries.The architecture is intended to allow users to customize components without hiding the underlying foundations.
  • Reported experience: Users can train a state-of-the-art vision model with four understandable lines of code and implement recent research papers in a couple of hours while matching reported performance.The authors also report training a ResNet-50 on ImageNet to accuracy in 18 minutes for their winning DawnBench entry.

2. Applications

fastai’s application APIs provide concise workflows for vision and text tasks, combining specialized data handling with automatic visualization and training defaults. The examples emphasize reusable abstractions, GPU-aware processing, and similar workflows across tasks.

  • Vision: A complete Oxford IIT Pets fine-tuning example uses a pretrained ImageNet model and reaches close to state-of-the-art accuracy after a couple of minutes on one GPU.The example creates DataLoaders, a Learner, and a four-epoch one-cycle training run.
  • Vision: DataLoaders combine training and validation data and can be created through application-specific subclasses or the flexible data block API.The ImageDataLoaders example uses a regular-expression labeller, with additional labellers available for common file and folder patterns.
  • Vision: fastai separates item-level CPU transforms from batch-level GPU transforms, while aug_transforms curates customizable augmentation best practices.This reduces the number of individual components users must learn to obtain common vision workflows.
  • Vision: The same concise vision workflow extends to segmentation, where the model and data definitions differ mainly in task-specific processing and visualization.The segmentation example uses a UNet learner, and the same display line can be reused for segmentation data.
  • Text: Text applications use similar training steps for language modeling and classification, with tokenized data displayed in DataFrames and specialized processing defaults.The processing pipeline handles capitalization and repeated-character patterns with special tokens, while remaining replaceable through the layered architecture.
  • Tabular and GPU processing: fastai integrates NVIDIA cuDF for end-to-end GPU-optimized data processing and model training.The paper describes this as the first deep learning framework to integrate with cuDF in this way.

3. High-level API design considerations

fastai’s high-level APIs provide consistent, expressive workflows across deep learning applications while retaining customization and compatibility with existing PyTorch code. They include shared training tools such as the 1cycle policy, learning-rate finder, and data block API.

  • Shared high-level components: A consistent visualization and model-output API supports different model and dataset types through fastai’s type dispatch system.The main methods are show_batch for input data and show_results for model results.
  • Shared high-level components: The same Learner functionality is available across applications, with 1cycle training combining learning-rate warm-up and annealing with opposite momentum changes.The 1cycle variant is recommended for training models obtained through the application APIs.
  • Shared high-level components: The learning-rate finder performs mock training to identify a useful learning rate more quickly than grid search or AutoML approaches described in the paper.It uses an exponentially growing learning rate over 100 iterations and selects the graph minimum divided by 10.
  • Data block API: The data block API systematically defines data preparation steps, including sourcing, splitting, labelling, processing, and optional batching.Its functional design addresses the practical importance of step order and provides mix-and-match data blocks.
  • Data block API: Data blocks support multiple application workflows, including image classification, segmentation, object detection, and language modeling.The examples use configurable blocks, getters, splitters, transforms, and batching to prepare datasets for modeling.
  • PyTorch interoperability: fastai can be added incrementally to pure PyTorch code, replacing an existing training loop while preserving the rest of the code.The resulting Learner provides callbacks, progress reporting, and integrated schedulers without other changes.
  • API design rationale: A consistent API helps users move between deep learning fields and reuse expertise, while the layered architecture permits customization of every part.This consistency also reduces what students need to learn across model types.

4. Mid-level APIs

fastai’s mid-level APIs address the gap between concise high-level applications and customizable low-level foundations. They provide reusable abstractions for training, callbacks, optimization, data loading, transforms, metrics, and external datasets.

  • Architecture: Mid-level APIs make it easier to build and customize high-level functionality without relying directly on increasingly complicated low-level APIs.These APIs are fully documented and available to users for creating applications or adapting existing ones.
  • Training abstractions: Learner and DataLoaders gather the information needed to choose appropriate training and data-processing behavior.Learner also supports transfer learning through parameter-group splitting, freezing, and differential learning rates.
  • Callbacks: Two-way callbacks can read and modify gradients, data, losses, control flow, and other training components at any point.They have supported mixup, GANs, mixed precision, hooks, and the learning-rate finder without changing the training loop.
  • Optimizer: The generic optimizer foundation separates statistics from parameter-update steppers, allowing new optimization techniques to be implemented without changing the foundation.The paper reports that every optimizer attempted in fastai used this foundation, while LAMB code closely follows the paper’s algorithm.
  • Metrics and datasets: The Metric abstraction computes metrics through reset, batch-level accumulation, and a final value property, reducing the need to store all predictions.fastai also curates external datasets and automatically downloads, extracts, and caches them when first used.
  • Data: fastai extends data loading through 15 customizable stages, transform pipelines, and inverse transforms for more flexible data-processing workflows.The stages span sample creation, item creation, and batch creation, while TfmdDL executes transform pipelines at customization points.

5. Low-level APIs

fastai’s low-level APIs provide layered abstractions for transforms, type dispatch, semantic tensors, GPU vision operations, and Python conveniences. These foundations support flexible, customizable data-processing and modeling workflows while building on PyTorch.

  • Core abstractions: The low-level stack supplies transform pipelines, type dispatch, semantic tensors, GPU-optimized vision operations, and convenience utilities.These abstractions form the foundation on which fastai’s higher-level APIs are built.
  • Transforms and pipelines: Transform objects pair callable encoding with user-defined decoding, allowing pipelines to invert transformations for understandable inference outputs.The combined encode/decode design also supports composition and serialization.
  • Transforms and pipelines: Transforms can coordinate tuple elements, vary behavior by data subset, and initialize state from input-dependent information.This supports cases such as segmentation augmentation, validation-time behavior, normalization statistics, and vocabularies.
  • Types and dispatch: fastai extends Python-style dispatch to two parameters and adds tensor subclasses whose semantic information is preserved through operations where possible.Two-parameter dispatch supports behavior customized to both model inputs and targets, while transforms help maintain subclass types.
  • Vision operations: Most computer-vision augmentation runs on GPU batches through PyTorch’s grid_sample, addressing CPU bottlenecks in modern training pipelines.The approach differs from the historical CPU augmentation and batching workflow.
  • Convenience functionality: fastai adds an L collection with a NumPy-like API, @delegates for exposing delegated arguments, and @patch for concise monkey-patching.These utilities improve customization and preserve useful signatures for IDE completion.

6. nbdev

nbdev is a notebook-centered programming environment that turns exploratory Jupyter development into distributable, tested Python packages. It adds tooling for source export, documentation, packaging, testing, continuous integration, and version control.

  • Motivation: nbdev addresses the gap between notebook exploration and software development by adding IDE-like programming features and distributable source generation.Jupyter supports exploration but lacks several facilities needed for complete software development.
  • Development workflow: nbdev automatically creates Python modules, including exported names through __all__, and synchronizes edits between notebooks and standard editors.This lets users work in either notebook or IDE-oriented environments.
  • Tooling: It generates searchable, hyperlinked documentation, uploads pip installers, runs notebook-defined tests in parallel, and supports continuous integration and version-control conflict handling.The documentation system is illustrated in Figure 12.
  • Scope: The paper defers fuller discussion of nbdev’s features, benefits, and history to a future paper.The current work introduces the environment and lists its principal development tools.

7. Related work

fastai’s design was influenced by earlier high-level deep-learning APIs, general machine-learning libraries, and especially PyTorch. It combines these influences while making distinct choices about layered composition and extensibility.

  • Deep-learning APIs: Earlier Python deep-learning APIs, including Calysto/conx, Lasagne, and Fuel/Blocks, form part of fastai’s historical context.The paper places fastai within a long development history of high-level deep-learning interfaces.
  • General machine learning: General machine-learning systems such as SPSS, SAS, S, R, SPLUS, and xlisp-stat influenced fastai’s data-processing and model-building direction.The paper contrasts different design choices in these ecosystems.
  • Composition abstractions: Scikit-learn, Torchvision, and pandas provide related function-composition abstractions for transforming data into model-ready formats.Scikit-learn also supports learning and prediction functionality.
  • PyTorch: PyTorch is fastai’s most important influence because its API is extensible and flexible and its implementation is efficient.fastai relies heavily on torch.Tensor and torch.nn while independently creating some overlapping higher-level functionality.

8. Results and conclusion

The authors report that fastai made course development and deep-learning implementation faster and easier while reproducing or improving prior results. They conclude that layered APIs benefit researchers, practitioners, and students, and can extend beyond Python and PyTorch.

  • Results: The authors rewrote a 14-hour, seven-module deep-learning course and report replicating or improving all results from previous versions more quickly and easily.They also report positive feedback from early adopters and adoption in the PyTorch ecosystem and Kaggle community.
  • Conclusion: Researchers can combine and restructure ideas, run experiments on strong baselines, and see links across different areas more easily with a layered API.This consequence is based on the authors’ experience with fastai.
  • Conclusion: Practitioners can build prototypes and optimize them on PyTorch foundations without rewriting code, while students can experiment without initial boilerplate overload.The paper attributes these benefits to the layered API approach.
  • Scope and future use: The core ideas are not limited to Python or PyTorch, as shown by the partial Swift port SwiftAI.The authors hope similar projects will be developed for additional languages and libraries.
Loading 2002.04688v2…