Source-linked AI summary
Declarative Experimentation in Information Retrieval using PyTerrier
Craig Macdonald, Nicola Tonellotto
TL;DR
IR lacks the expressive, high-level formalism used by deep-learning frameworks for designing complex experiments. The paper introduces PyTerrier, a declarative Python framework that compiles and optimises retrieval pipelines for IR backends, and reports efficiency benefits on Anserini and Terrier pipelines.
Problem
IR lacks expressive high-level-language support for wide-ranging experiments, while reproducibility requires evaluating techniques within complete retrieval stacks.
Method
PyTerrier represents retrieval operations as transformers and composes them with operators into declarative pipelines compiled and optimised for IR platforms.
Results
95% lower mean response time on Robust ’04 short queries and 63% on ClueWeb09 were obtained by applying rank-cutoff optimisation to Anserini pipelines.
Takeaways & Limitations
PyTerrier supports end-to-end IR experimentation across Anserini and Terrier while improving efficiency through backend-aware optimisation.
Takeaways & Limitations
The current Experiment abstraction does not handle pipeline fitting or training, although variants for cross-validation and grid search are proposed.
Abstract
from arXiv · showhide
The advent of deep machine learning platforms such as Tensorflow and Pytorch, developed in expressive high-level languages such as Python, have allowed more expressive representations of deep neural network architectures. We argue that such a powerful formalism is missing in information retrieval (IR), and propose a framework called PyTerrier that allows advanced retrieval pipelines to be expressed, and evaluated, in a declarative manner close to their conceptual design. Like the aforementioned frameworks that compile deep learning experiments into primitive GPU operations, our framework targets IR platforms as backends in order to execute and evaluate retrieval pipelines. Further, we can automatically optimise the retrieval pipelines to increase their efficiency to suite a particular IR platform backend. Our experiments, conducted on TREC Robust and ClueWeb09 test collections, demonstrate the efficiency benefits of these optimisations for retrieval pipelines involving both the Anserini and Terrier IR platforms.
1 INTRODUCTION
The paper presents PyTerrier, a Python framework for expressing complex IR experiments as composable, declarative pipelines. It compiles and optimises these pipelines for IR backends, improving efficiency while supporting end-to-end evaluation and reproducibility.
- Motivation: IR experiments combine retrieval, fusion, feature extraction, learning-to-rank, and neural reranking, making end-to-end evaluation difficult.Such workflows typically require multiple configuration files and commands.
- Motivation: Reproducible end-to-end experiments are important because isolated benchmark datasets do not reveal how techniques interact within a complete search stack.Understanding required data and choices such as reranking depth reduces uncertainty about deployment in operational search engines.
- Framework: PyTerrier uses Python, transformers, and operators to specify advanced retrieval pipelines declaratively rather than procedurally.Transformers represent retrieval building blocks, while operators compose them into pipelines.
- Framework: PyTerrier pipelines form DAGs of IR operations that can be rewritten and compiled into optimised configurations for underlying search engines.The framework targets Anserini and Terrier and supports operations such as retrieval, reranking, result combination, and query rewriting.
- Contribution: The framework supports extensible IR activities and integration of deep learning techniques while retaining pipeline semantics.The implementation compiles pipelines on top of two Java-based IR platforms for fast retrieval operations.
2 RELATED WORK
Related work spans established IR platforms, data-processing systems, and deep-learning frameworks. PyTerrier adapts Python-based declarative dataflow ideas to IR while addressing platform coupling and development overhead.
- IR platforms: Established IR platforms include Lucene, Anserini, Terrier, Lemur/Indri, Galago, and PISA, but none of the discussed platforms is natively written in Python.Python bindings exist for some platforms, but the paper distinguishes bindings from native implementations.
- IR platforms: Existing IR platforms commonly mix experimental retrieval design with implementation and optimisation, which can limit reproducibility across systems.Different implementations of the same BM25 weighting model can produce different values for the same effectiveness metric.
- Data science: Python data-science toolkits support agile code adjustment and rerunning, often through notebook environments such as Jupyter or Google Colab.Pandas and scikit-learn exemplify structured data processing and machine-learning workflows in Python.
- Data science: Apache Spark compiles structured data-processing expressions into execution plans distributed across clusters, but its scale-oriented design can introduce interactive overhead.These properties motivated comparison with more notebook-oriented IR experimentation.
- Deep learning: TensorFlow and PyTorch represent computations as Python-defined dataflow DAGs that can be compiled into lower-level operations.PyTerrier adopts a similar domain-specific Python environment for IR experiments.
- Prior declarative IR work: Terrier-Spark expressed Terrier retrieval operations over Spark dataframes in Scala, but Spark overhead and Scala’s lower popularity hindered notebook-style agile development.PyTerrier instead uses Python and targets IR platforms directly.
3 DECLARATIVE RETRIEVAL OPERATIONS
PyTerrier represents retrieval experiments as composable transformers and operators over queries and retrieved documents, then evaluates pipelines through a common Experiment abstraction. This declarative representation supports extensible IR operations, multiple retrieval backends, and a focus on pipeline logic rather than execution order.
- Transformers: PyTerrier models IR operations as transformers that take query and/or retrieved-document lists and return transformed queries and/or results.Transformers can have partially specified inputs and outputs, allowing operations such as retrieval, rewriting, reranking, and feature extraction.
- Transformers: A Retrieve transformer maps queries to ranked results and can vary by weighting model, parameters, or retrieval system.The paper lists BM25, TF.IDF, language modelling, Indri, Terrier, and Anserini as possible instantiations.
- Transformers: Query expansion composes an initial retrieval, an Expand transformer over top-ranked documents, and a second retrieval of the reformulated queries.The Expand transformer calculates revised queries from the initial retrieved documents, while the second retrieval processes those queries.
- Transformers: Feature extraction supports multi-stage ranking pipelines by calculating additional query-dependent or query-independent features over an initial candidate set.Examples include PageRank, URL length, proximity, and field-based weighting features.
- Experiment abstraction: The Experiment function applies multiple retrieval pipelines to common queries and evaluates their outputs with shared effectiveness measures.It builds on pytrec_eval and turns pipeline execution and evaluation into a single abstraction.
- Experiment abstraction: The declarative environment lets researchers specify retrieval-pipeline stages conceptually while PyTerrier implements primitive operations and supports Anserini and Terrier backends.Pipeline composition separates logical experiment design from execution order and permits backend-specific implementation and optimisation.
4 IMPLEMENTING TRANSFORMERS AND OPERATORS
PyTerrier represents retrieval experiments as composable data-flow graphs that can be compiled and optimized for specific IR platforms. The section describes optimizations for dynamic pruning and feature computation, with empirical efficiency evaluation on two platforms.
- Pipeline compilation: PyTerrier models retrieval pipelines as data-flow graphs whose nodes are search operations and whose edges pass queries and documents.Transformers implement primitive operations such as search, rerank, and rewrite, while operators combine them.
- Pipeline compilation: Pipeline compilation uses pattern matching to identify pipeline expressions and combine transformers and operators for more efficient execution on a target platform.The compilation process targets specific IR software platforms and rewrites suitable pipeline patterns.
- Dynamic pruning optimisations: Dynamic pruning can accelerate rank-cutoff retrieval because reduced result counts raise scoring thresholds and allow MaxScore, WAND, or BlockMaxWAND to skip more documents.The optimization compiles a separate rank-cutoff operation into retrieval behavior that exploits the underlying pruning mechanism.
- Learning to rank optimisations: Learning-to-rank feature computation can incur repeated inverted-index scans for query-dependent features, creating substantial computational overhead during reranking.The described example computes TF.IDF and query-likelihood features, each of which can require additional posting-list access.
- Learning to rank optimisations: The paper considers doc vectors and fat postings architectures for computing query-dependent features more efficiently for documents entering the final retrieved set.The example feature pipeline requests BM25 retrieval with TFIDF and QL features.
- Evaluation: The next section evaluates the efficiency benefits of optimized pipelines over retrieval operations involving two underlying IR platforms.This establishes the transition from compilation and optimization mechanisms to empirical evaluation.
5 EXPERIMENTS
The experiments evaluate whether PyTerrier’s pipeline rewrites improve retrieval efficiency across Anserini and Terrier backends. Results show substantial gains from backend-specific optimisations, while performance depends on query length and execution strategy.
- Experimental aims: The experiments test whether rank-cutoff optimisation improves Anserini and whether fat postings improve Terrier for LTR pipelines.The evaluation uses TREC Robust ’04 and ClueWeb09 collections with Anserini and Terrier backends.
- Experimental setup: Table 3 reports mean response time in milliseconds before and after optimisation, with percentage improvement between original and optimised pipelines.
- RQ1: Anserini rank cutoffs: Anserini benefits from BlockMaxWAND for short Robust ’04 queries, whereas Terrier outperforms Anserini for longer TD and TDN queries.The passage attributes the weaker dynamic-pruning benefit for longer queries to the known efficiency characteristics of document-at-a-time pruning.
- RQ1: Anserini rank cutoffs: 95% and 63% are the maximum mean-response-time improvements from reducing Anserini’s requested rank cutoff from 1000 to 10 on Robust ’04 and ClueWeb09, respectively.The improvements are reported for short queries when comparing optimised and original Anserini pipelines.
- RQ2: Terrier LTR execution: Terrier executes the complex BM25-plus-feature pipeline faster than Anserini, while its optimised fat formulation uses one backend retrieval operation per query.Fat postings calculate multiple query-dependent features in a single pass, but their benefit decreases for longer queries because of memory pressure.
- Cross-backend execution: A single conceptual pipeline can run on multiple backends with different efficiencies, without requiring researchers to know each backend’s capabilities.
6 CONCLUSIONS AND OUTLOOK
The paper presents PyTerrier as a declarative framework for composing IR experiments from standard retrieval operations. It also automatically compiles and optimises equivalent pipelines using knowledge of backend capabilities.
- Contributions: PyTerrier provides a data model and framework for conducting IR experiments declaratively.
- Contributions: Its transformers represent standard retrieval operations, and operators combine those transformers into retrieval pipelines.
- Optimisation: The framework automatically compiles and optimises pipelines by encoding knowledge of underlying IR-system capabilities.
- Outlook: The authors propose future support for automatic parallelisation and incremental querying alongside distributing experiments as Jupyter notebooks.