Source-linked AI summary
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander M. Rush
TL;DR
As Transformer models and pretraining spread across NLP, researchers and practitioners need infrastructure for training, adapting, distributing, and deploying them. This paper presents Transformers, an open-source library and model hub that provides unified access to Transformer architectures and pretrained models for varied NLP tasks and production use.
Problem
The growing use of Transformer architectures and pretrained models creates practical needs for training, analyzing, scaling, distributing, fine-tuning, deploying, and compressing them across platforms.
Method
Transformers combines extensible Transformer implementations, task-specific heads, and a centralized model hub under a unified API for research, fine-tuning, and deployment.
Results
The library supports adapting pretrained models across a wide variety of NLP tasks and deploying them through interoperable formats, including reported speed-ups with ONNX.
Takeaways & Limitations
Transformers provides shared open-source infrastructure for accessing, experimenting with, and deploying large-scale pretrained models in downstream NLP applications.
Abstract
from arXiv · showhide
Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{https://github.com/huggingface/transformers}.
1 Introduction
Transformers have become central to NLP because Transformer architectures and pretraining deliver strong, adaptable performance, while creating practical challenges for training, scaling, deployment, and model distribution. The Transformers library addresses these needs through extensible research- and production-oriented implementations and pretrained-model support.
- Motivation: Transformer architectures surpass convolutional and recurrent neural networks on natural language understanding and generation tasks while enabling parallel training and long-range sequence modeling.The architecture also scales with training data and model size.
- Motivation: Pretraining on generic corpora enables easy adaptation to specific tasks and improves accuracy across classification, language understanding, translation, coreference, inference, and summarization.The cited downstream applications include text classification, language understanding, machine translation, coreference resolution, commonsense inference, and summarization.
- Motivation: Widespread Transformer use creates practical requirements for training, analysis, scaling, augmentation, distribution, fine-tuning, deployment, and compression across platforms.The architecture also serves as a building block for sophisticated extensions and precise experiments.
- Library contribution: Transformers provides industrial-strength implementations of popular Transformer variants that are easy to read, extend, and deploy, alongside distribution and usage of diverse pretrained models.The library is designed to support both research and production through a unified foundation for Transformer-based architectures.
- Implementation: The project is maintained by Hugging Face engineers and researchers with support from over 400 external contributors, and is released under the Apache 2.0 license on GitHub.Detailed documentation and tutorials are available on Hugging Face’s website.
2 Related Work
Transformers builds on open-source NLP research tools, general-purpose NLP libraries, and framework-specific model hubs. Unlike broad model hubs, it is domain-specific and supports model analysis, usage, deployment, benchmarking, and replicability.
- Open-source research tools: Transformers draws structural inspiration from tensor2tensor and BERT, while its pretrained-model caching concept stems from AllenNLP.The library is also related to neural translation and language-modeling systems such as Fairseq.
- General-purpose NLP libraries: Earlier user-facing NLP libraries, including NLTK and Stanford CoreNLP, unified diverse approaches, while newer open-source libraries target machine-learning NLP tasks.Examples of newer libraries include Spacy, AllenNLP, flair, and related systems.
- Model hubs: Transformers is related to Torch Hub and TensorFlow Hub, which provide framework-specific model parameters for easy use.Unlike these hubs, Transformers is domain-specific and automatically supports model analysis, usage, deployment, benchmarking, and easy replicability.
3 Library Design
Transformers organizes NLP modeling around a standard data-to-prediction pipeline and three reusable components: tokenizer, transformer, and task-specific head. Its architecture-specific implementations, shared APIs, tokenizers, and interchangeable heads support diverse NLP applications and model adaptation.
- The library mirrors the standard NLP pipeline of processing data, applying a model, and making predictions, while focusing this report on core modeling specifications.
- Every model consists of a tokenizer, transformer, and head that respectively encode raw text, produce contextual embeddings, and generate task-specific predictions.Most user needs can be addressed with these three components.
- Transformers: Carefully tested Transformer variants share a multi-headed attention core but differ in positional representations, masking, padding, sequence-to-sequence design, and target applications.The library covers understanding, generation, conditional generation, fast inference, and multilingual use cases.
- Models use a hierarchy in which architecture-specific base classes implement computation graphs closely following original implementations, while Auto classes provide a unified API for rapidly switching models and frameworks.Auto classes instantiate models from user-specified pretrained-model configurations.
- Tokenizers and Heads: Tokenizer classes manage model-specific vocabularies and encoding or decoding, while interchangeable heads add task-specific output layers and optional loss functions for pretraining and fine-tuning.Pretrained heads can preserve the core Transformer parameters while enabling adaptation to other tasks.
4 Community Model Hub
The Model Hub supports community sharing and reuse of pretrained and fine-tuned Transformer models through a simple interface. It combines standardized model packaging and documentation with live inference and examples of research and deployment use.
- Community Model Hub: The Model Hub lets end-users access community-contributed pretrained and fine-tuned models for use with their own data.It contained 2,097 user models at the time described, while core models such as BERT and GPT-2 remained popular alongside specialized models.
- Community Model Hub: Users can upload models through a command-line interface as an archive containing a tokenizer, transformer, and task head.The bundle can contain a model trained through Transformers or converted from a checkpoint produced by another popular training tool.
- Community Model Hub: Each uploaded model receives a canonical name, landing page, and optional model card documenting architecture, training, datasets, citations, and known caveats.These pages support model download, caching, execution, and more transparent communication of model properties and biases.
- Community Model Hub: Transformer-specific metadata enables model pages to provide live inference and links to model-specific tools such as benchmarking and visualizations.Users can experiment with model outputs on real data directly from the model page.
- Community Case Studies: Community case studies show the hub supporting model architects, task trainers, and application users with distinct research and deployment goals.Examples include AllenAI model development, NYU’s Jiant research framework using the Transformers API, and Plot.ly’s search for a deployable summarization model.
5 Deployment
Transformers supports multiple production deployment strategies because deployment introduces needs distinct from training. It enables framework interoperability, production serving, ONNX conversion with nearly 4x speedups, and deployment on edge devices.
- Deployment goals: Transformers supports multiple production deployment strategies because production requirements differ substantially from training challenges.The library aims to make efficient production deployment easier for users with different needs.
- Framework interoperability: Models can move seamlessly between PyTorch and TensorFlow through standard serialization, supporting framework changes across training and serving.A model trained in one framework can be saved and reloaded in the other.
- Framework deployment: PyTorch models support TorchScript and TorchServing, while TensorFlow models can use serving options from its ecosystem.TorchScript enables more efficient Python execution or high-performance environments such as C++.
- ONNX deployment: Nearly 4x speedup was achieved when ONNX optimized BERT, RoBERTa, and GPT-2 models from Transformers.ONNX provides a standardized interoperable intermediate format for deployment and compilation.
- Edge deployment: Adapters convert models to CoreML weights for embedding in iOS applications, and similar methods can support Android edge devices.This enables on-the-edge machine learning on phones and home electronics.
6 Conclusion
Transformers is an open-source library and community that broadens access to large-scale pretrained models, supports experimentation, and enables downstream deployment with state-of-the-art performance. Its significant organic traction positions it to continue providing core infrastructure for NLP.
- 6 Conclusion: Transformers facilitates access to large-scale pretrained models for researchers and end-users.The library is intended to support building and experimenting on top of these models.
- 6 Conclusion: The library supports building, experimentation, and deployment in downstream tasks with state-of-the-art performance.These capabilities are presented as part of Transformers’ open-source design and community.
- 6 Conclusion: Transformers has gained significant organic traction since its release and is set up to continue providing core infrastructure.