Source-linked AI summary
Simplified Data Wrangling with ir_datasets
Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, Nazli Goharian
TL;DR
IR datasets can be difficult to find, manage, and process consistently. ir_datasets provides a lightweight tool and catalog for access to dataset content and operations. Its integrations with IR toolkits demonstrate its utility.
Problem
IR datasets are difficult to find and manage, and varied formats can hide subtle problems that affect proper use.
Source: §1 · p. 1Method
ir_datasets offers Python and command-line access to dataset entities, document lookup, and a catalog for discovering datasets and their characteristics.
Source: §1 · p. 2Results
Several IR toolkits added ir_datasets support, and the tool supports a wide variety of datasets and benchmarks.
Source: §1 · p. 2, §2.5 · p. 4Takeaways & Limitations
The authors hope ir_datasets reduces researcher burden and redundant toolkit copies while enabling new tools.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Managing the data for Information Retrieval (IR) experiments can be challenging. Dataset documentation is scattered across the Internet and once one obtains a copy of the data, there are numerous different data formats to work with. Even basic formats can have subtle dataset-specific nuances that need to be considered for proper use. To help mitigate these challenges, we introduce a new robust and lightweight tool (ir_datasets) for acquiring, managing, and performing typical operations over datasets used in IR. We primarily focus on textual datasets used for ad-hoc search. This tool provides both a Python and command line interface to numerous IR datasets and benchmarks. To our knowledge, this is the most extensive tool of its kind. Integrations with popular IR indexing and experimentation toolkits demonstrate the tool's utility. We also provide documentation of these datasets through the ir_datasets catalog: https://ir-datasets.com/. The catalog acts as a hub for information on datasets used in IR, providing core information about what data each benchmark provides as well as links to more detailed information. We welcome community contributions and intend to continue to maintain and grow this tool.
1 INTRODUCTION
IR datasets are difficult to find, manage, and process reliably, even when their formats appear simple. ir_datasets offers a lightweight shared tool for dataset operations and documentation, with integrations across IR toolkits.
1 INTRODUCTION
- IR datasets are difficult to find and manage, and even simple-looking formats can contain subtle problems, such as MS-MARCO’s double-encoding of special characters in roughly 20% of documents.The paper notes that dataset documentation is scattered and data formats vary.Source: §1 · p. 1
- Existing automatic acquisition tools have patchy dataset coverage, can be tightly coupled to their primary tool, and keep separate copies of data.Anserini and Pyserini use a hybrid approach, distributing queries and relevance judgments while relying mainly on manual document-processing instructions.Source: §1 · p. 1, Table 1 · p. 2
- The authors argue that automatic processing can ensure proper data handling, reduce redundant dataset copies, and ease experiments with alternative or custom datasets.These benefits are contrasted with manual instructions for downloading, extracting, and processing datasets.Source: §1 · p. 1
- Record-centric dataset libraries are ill-suited to relational IR data and document-by-ID lookups, while dataset schemata could support machine-readable documentation in the future.The paper cites HuggingFace Datasets and TensorFlow Datasets as examples of record-centric libraries.Source: §1 · p. 1
- ir_datasets provides Python and command-line access to dataset entities, fast document lookup, and a catalog, and several IR toolkits have added support.It supports discovery, acquisition, and management; users can iterate documents, queries, and relevance assessments for indexing, retrieval, and evaluation.Source: §1 · p. 2
2 IR_DATASETS
ir_datasets is designed as a lightweight, flexible layer for core dataset operations rather than indexing or model processing. Its hierarchical identifiers and Python API provide access to dataset entities and document lookups.
2 IR_DATASETS
- The design emphasizes core operations, low memory use, and “just enough” processing, leaving indexing and neural-network processing to other tools.The tool uses inexpensive data structures and iterators while retaining information that may be useful to users.Source: §2 · p. 2
2.1 Dataset Identifiers
- Hierarchical dataset IDs, usually corpus/benchmark, identify datasets across the Python API, command line, and online documentation.For example, cord19/trec-covid combines CORD-19 documents with TREC COVID queries and relevance judgments.Source: §2.1 · p. 2
2.2 Simple & Memorable Python API
- The Python API exposes documents, queries, and relevance assessments through entity iterators, with dataset-specific fields available through typed entity definitions.Iterators can support single-entity processing, batches, or loading all data into a list or Pandas dataframe.Source: §2.2 · p. 3
- Some datasets also expose scoreddocs for query-document scores and docpairs for contrasting document IDs used in training sequences.A scoreddocs tuple contains a query ID, document ID, and score; a docpairs entity contains a query and a pair of contrasting document IDs.Source: §2.2 · p. 3
2.3 Command Line Interface
The command-line interface supports basic dataset operations for non-Python tools and ad-hoc exploration. Its export and lookup commands correspond to key Python API operations.
2.3 Command Line Interface
- The CLI provides export and lookup operations, multiple output formats, and selectable output fields for tool integration or ad-hoc exploration.Export corresponds to dataset iterators, while lookup corresponds to the Python document-store lookup operation.Source: §2.3 · p. 3
2.4 Data Acquisition
ir_datasets automates downloads and validates their integrity when possible, while giving users instructions for datasets requiring manual acquisition. It supports a broad range of IR datasets and benchmarks.
2.4 Data Acquisition
- When possible, ir_datasets downloads content from original public sources, checks hashes, resumes interrupted downloads when supported, and notifies users before downloads requiring a usage agreement.The access and integrity of downloadable content are also checked periodically by continuous integration.Source: §2.4 · pp. 3–4
- Some collections require manual acquisition because of size, copyright, or licensing; users obtain them using instructions, after which ir_datasets handles remaining processing.The paper reports 12 document collections requiring manual acquisition.Source: §2.4 · p. 4
2.5 Supported datasets
- Supported datasets span popular benchmarks, large-scale shallow and click datasets, biomedical and multilingual collections, and ranking suites; the authors describe this as the largest collection and variety supported by any tool.The Python API also provides a mechanism for building a dataset object from files in simple formats.Source: §2.5 · p. 4
2.6 Document lookups
Document-ID lookup is a common IR need, from model processing to investigating failures. ir_datasets supports disk-based lookup across collection sizes, using different strategies for small and large datasets.
2.6 Document lookups
- The docs_store API retrieves one or multiple documents by ID from disk, providing an alternative to loading the entire collection into an in-memory hashmap.An in-memory hashmap may suit long-running processes when upfront cost is negligible and memory can hold the collection.Source: §2.6 · p. 4
- For collections up to a few million documents, ir_datasets builds a specialized on-disk lookup structure as needed, with compressed documents and a sorted ID-to-offset structure.The authors report lookup performance exceeding the compared indexes and databases; Table 3 benchmarks small datasets.Source: §2.6 · p. 4, Table 3 · p. 4
- For very large collections such as ClueWeb09 and ClueWeb12, copying all documents is impractical, and their directory structures help identify source files by document ID.ir_datasets uses periodic zlib decoding-state checkpoints to avoid reading an entire source file before reaching a requested document.Source: §2.6 · pp. 4–5
- For large collections, ir_datasets is slower than Pyserini on the first lookup, while caching greatly speeds subsequent fetches; ClueWeb12 overhead is 4.5GB versus Anserini's around 6TB.ClueWeb12 checkpoints speed lookups considerably with little storage overhead; Tweets2013-ia instead uses an ID-based lookup hierarchy because its source hierarchy is not conducive to lookups.Source: Table 4 · p. 4, §2.6 · p. 5
2.7 Fancy slicing
Fancy slicing lets users select document-collection ranges for processing, including partitions that can be assigned to separate workers. Its implementations seek to the appropriate source location to avoid unnecessary I/O and processing.
2.7 Fancy slicing
- Selecting document-collection ranges supports dividing work across GPUs or machines for parallel processing, including precomputing representations or first-stage retrieval.The passage describes this processing as embarrassingly parallel.Source: §2.7 · p. 5
- Unlike Python's islice, slicing objects from doc_iter avoids I/O and processing for skipped indices by seeking to the appropriate source location.The implementation varies by dataset while following typical Python slicing semantics.Source: §2.7 · p. 5
- Document iterators support slices for initial, final, indexed, strided, and fractional portions of a collection.Examples include every fifth document starting at index three and the first third of the collection.Source: §2.7 · p. 5
2.8 Documentation
The ir_datasets catalog provides dataset documentation, including cues about download availability and details for individual datasets. The package also uses automated tests to check its functionality and the continued availability and stability of downloadable content.
2.8 Documentation
- The catalog overview indicates which datasets are available for automatic download and which require obtaining data from a third party.Figure 2 uses a green checkbox for automatic downloads and a yellow triangle for third-party acquisition.Source: Figure 2 · p. 5
- Individual dataset pages include descriptions, relevant links, supported relations, citations, and code samples.The paper gives the TREC COVID dataset as an example.Source: §2.8 · p. 5, Figure 3 · p. 5
2.9 Automated Testing
- Automated unit and integration/regression tests check package behavior, guard against regressions, and check that downloadable content remains available and unchanged..Source: §2.9 · p. 5
3 INTEGRATION WITH OTHER TOOLS
ir_datasets integrates with IR toolkits while remaining independent of them. Its interfaces support dataset use across retrieval, ranking, and result-analysis workflows.
3 INTEGRATION WITH OTHER TOOLS
- Integrated tools use ir_datasets rather than making it depend on any of them.The CLI also supports tools such as PISA through document export.Source: §3 · p. 5
- Capreolus supports all ir_datasets datasets in its training and evaluation pipelines, beyond its built-in datasets.It provides components for document collections and query-and-qrels benchmarks.Source: §3 · p. 6
- PyTerrier automatically adds ir_datasets datasets, which users access by dataset ID with an irds: prefix.These datasets are then used seamlessly with the rest of PyTerrier.Source: §3 · p. 6
- OpenNIR uses ir_datasets for training, validation, and testing, indexing documents with Anserini and looking up text for training and scoring.Queries and relevance judgments feed into training and validation.Source: §3 · p. 6
- Anserini and Pyserini can use exported queries and qrels and efficiently receive documents for parallel indexing, though document-content support is otherwise sparse.Only a few collections have automatically downloadable indices; the rest rely on manual instructions.Source: §3 · p. 6
- DiffIR uses ir_datasets to display top-result text and highlight model-specific impactful spans.Source: §3 · p. 6
4 COMMUNITY CONTRIBUTIONS
The project invites community contributions through both a separate package and pull requests, with quality checks guiding development.
4 COMMUNITY CONTRIBUTIONS
- Contributors can extend ir_datasets through a separate package or submit pull requests to the main package.Source: §4 · p. 6
- Development should address ease of use, efficiency, data integrity, and documentation, with issues opened before implementation.Source: §4 · p. 6
- Typical TREC, TSV, and CSV formats are relatively straightforward to handle, while atypical formats likely need special processing.The project provides examples to guide contributors.Source: §4 · p. 6
5 FUTURE APPLICATIONS
The paper describes possible applications in private-data experiments and graphical dataset exploration. These are envisioned extensions of ir_datasets’ use.
5 FUTURE APPLICATIONS
- A private ir_datasets version could support shared tasks on clinical notes or proprietary collections inside secure environments, including ones without networking.Source: §5 · p. 6
- An ir_datasets-compatible API lets participants switch to a private dataset by changing its dataset ID, while supporting confidence in correct processing.Source: §5 · p. 6
- A graphical interface could reduce exploration friction by listing queries and linking to judged-document text.Source: §5 · p. 6
6 CONCLUSION
ir_datasets provides access to IR datasets through Python and command-line interfaces, and integrations demonstrate its utility. The conclusion presents it as a way to reduce researcher burden and redundant toolkit copies while supporting new tools.
6 CONCLUSION
- ir_datasets provides access to diverse search datasets and benchmarks, automatically downloading and verifying content when possible.Source: §6 · p. 6
- Its interfaces support iteration over documents, queries, and relevance judgments, plus document lookup by ID.Source: §6 · p. 6
- Integrations with IR experiment tools demonstrate the utility of these functions, while the catalog serves as a dataset-information hub with literature links and citations.The authors hope the tool reduces researcher burden and redundant dataset copies and enables new tools.Source: §6 · p. 6