Source-linked AI summary
Datasets: A Community Library for Natural Language Processing
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, Thomas Wolf
TL;DR
NLP’s expanding dataset ecosystem needs standardized interfaces, versioning, documentation, and scalable access. This paper presents Datasets, a community library that addresses these needs through distributed dataset builders, typed and cached representations, and hub-based documentation. The library supports hundreds of datasets and contributors, cross-dataset research, shared tasks, and large-scale processing.
Problem
NLP’s growing dataset ecosystem creates challenges in standardizing interfaces, versioning, documentation, and access across widely varying scales and construction procedures.
Method
Datasets uses community-contributed builders, typed tabular representations, Arrow-backed caching, memory mapping, streaming, and dataset-hub documentation.
Results
The library includes 681 datasets, involved 285 unique contributors and 485 hackathon commits, and supports cross-dataset shared tasks including access to 12 datasets through one line of code.
Takeaways & Limitations
Datasets standardizes processing, distribution, and documentation while supporting cross-dataset NLP, shared tasks, indexing, streaming, and datasets of varying size.
Takeaways & Limitations
Datasets removed by their authors are not centrally cached and become unavailable.
Abstract
from arXiv · showhide
The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.
1 Introduction
NLP’s growing dataset ecosystem creates challenges in standardizing interfaces, versions, documentation, and scale. Datasets addresses these challenges through a community library emphasizing ease of use, efficiency, and documentation.
- NLP datasets support evaluation, training, fine-tuning, pretraining, and language modeling across differing scales, structures, and annotation methods.
- The dataset ecosystem requires standardized interfaces, reproducible versions, scale-independent access, and attention to varied construction procedures.
- Ease-of-use and Standardization: Datasets provides one-line downloading, a standard tabular format, versioning, and citation for all datasets.
- Efficiency and Scale: Datasets is computation- and memory-efficient, supports tokenization and featurization, and can stream massive datasets through the same interface.
- Community and Documentation: The project is community-built, with hundreds of contributors, and documents datasets through tags and datasheets describing usage, types, and construction.
- Datasets is continually developed by Hugging Face and released under an Apache 2.0 license.
2 Related Work
Earlier projects grouped, distributed, or simplified access to NLP datasets, while Datasets distinguishes itself through a content-agnostic, framework-independent, community-oriented library for diverse resources.
- The Linguistic Data Consortium stores and serves language and speech datasets, while OntoNotes, Universal Dependencies, and OPUS organize task-, language-, or translation-specific resources.
- Datasets is content-agnostic and differs from related projects in its organization and access model.
- NLTK and spaCy provide convenient loading interfaces, while TensorFlow-Datasets and TorchText collect datasets in common cloud formats.
- Datasets is decoupled from modeling frameworks, offers a general-purpose tabular API, and provides specialized structures for language constructs.
- The project emphasizes community management, documentation through the dataset hub and data cards, and access to long-tail datasets across tasks and languages.
3 Library Tour and Design
Datasets retrieves raw data through community-contributed builders, represents it as typed tables, and uses caching, memory mapping, and parallel processing to support efficient access and transformation.
- In-Memory Access: Users can access slices directly, without loading the full dataset into memory.
- Dataset Retrieval and Building: Community-contributed builder modules process hosted raw data into a common dataset-interface representation.
- Data Point Representation: Each dataset is represented as a typed table with atomic, structured, categorical, sequence, translation, and higher-dimensional feature types.
- Datasets removed by their authors are not centrally cached and become unavailable.
- In-Memory Access: Apache Arrow provides on-disk caching and memory mapping, allowing large datasets to run on machines with relatively small device memory.
- User Processing: Dataset processing supports sorting, shuffling, splitting, filtering, arbitrary map functions, batching, multiprocessing, and automatic caching.
- Complete Flow: The complete flow downloads data, applies dataset-specific builder code, caches a typed table, and exposes a memory-mapped table for further cached processing.
4 Dataset Documentation and Search
The Dataset Hub helps practitioners find and assess datasets through structured metadata, data cards, and model links. Data cards also evolve as users identify artifacts, split issues, and social biases.
- The Dataset Hub supports navigation of the growing resource collection and promotes improved dataset documentation.
- Practitioners need to identify suitable datasets and understand how their properties align with a research or application goal.
- Each hub entry includes tags for languages, tasks, and licenses, a templated data card, and a list of trained models.
- Choosing a Dataset: Structured tags can surface datasets by use case, after which data cards provide information about language, licensing, personal identifying information, and other considerations.
- Using a Dataset: Data cards describe dataset splits, disk size, split differences, and fields that can guide model inputs and outputs.
- The Data Card as a Living Document: Data cards function as living documents that can incorporate later findings about annotation artifacts, split design, and social biases.
5 Dataset Usage and Use-Cases
Datasets is used across tasks, languages, and dataset scales, supporting cross-dataset benchmarking, reproducible shared tasks, and robustness evaluation. Its breadth includes 681 datasets and community contributions spanning many languages and formats.
- 681 datasets cover diverse task types, sizes, and languages, while popular downloads include English benchmarks such as GLUE and SQuAD.The library also includes datasets for different tasks and languages beyond its most-downloaded benchmarks.
- Case Study 1: N-task Pretraining Benchmarks: The library supports large-scale evaluation across nearly 100 NLP tasks through the EleutherAI LM Evaluation Harness.The harness uses Datasets to retrieve and cache datasets for testing large language models.
- Case Study 2: Reproducible Shared Tasks: Datasets provides a standardized method for reproducible shared tasks spanning multiple datasets.In the GEM workshop, users accessed 12 datasets through one line of code despite varied formats, languages, and dataset sizes.
- Case Study 3: Robustness Evaluation: The library supports robustness reports that evaluate NLP systems across subpopulations, transformations, evaluation sets, and adversarial attacks.These dimensions provide a more specific assessment than a single evaluation measure.
- Gym relies on a common data interface provided by Datasets.
6 Additional Functionality and Uses
Datasets adds functionality for processing very large corpora, computing metrics, building search indexes, and viewing datasets through a standardized interface. These features extend access across streaming, distributed evaluation, and interactive inspection.
- Streaming: Streaming processes datasets too large to fit on disk by buffering them on the fly and applying map to streamed batches.This capability supported distributed training of a very large open NLP model.
- Indexing: The library provides tools for building and using search indexes over arbitrary datasets.
- Data Viewer: The datasets viewer lets users switch datasets, subsets, and splits while inspecting rows, schemas, metadata, typed features, and multimodal data.Its standardized interface supports viewing almost 650 datasets on the web.
- Metrics can be applied to multi-task datasets and computed in a distributed manner, with some metrics implemented directly and others linked externally.
7 Conclusion
Hugging Face Datasets is an open-source, community-driven library that standardizes dataset processing, distribution, and documentation. It provides a common interface across dataset sizes and supports cross-dataset research with features including indexing and streaming.
- Hugging Face Datasets standardizes the processing, distribution, and documentation of NLP datasets through an open-source, community-driven library.
- The library uses the same interface for datasets of varying size and is designed to be easy to use and fast.
- At 650 datasets from over 250 contributors, Datasets facilitates standard dataset use, cross-dataset NLP, indexing, and streaming of large datasets.