Source-linked AI summary
PACRR: A Position-Aware Neural IR Model for Relevance Matching
Kai Hui, Andrew Yates, Klaus Berberich, Gerard de Melo
TL;DR
Neural IR had not fully exploited position-dependent information such as proximity and term dependencies beyond unigram matching. PACRR models these interactions with convolutional, pooling, and recurrent layers, and extensive TREC Web Track experiments show strong improvements across benchmarks. The paper also identifies limits in distinguishing documents with different relevance grades.
Problem
Deep neural IR models had successfully captured unigram matches, but how to use positional information such as proximity and term dependencies remained insufficiently explored.
Method
PACRR uses similarity matrices, convolutional layers, max pooling, and a recurrent layer to capture term dependencies, proximity, and query-document relevance signals.
Results
PACRR substantially outperforms four state-of-the-art neural IR models on TREC Web Track ad-hoc datasets and dramatically improves search results when reranking.
Takeaways & Limitations
PACRR demonstrates that preserving positional information can improve neural IR ranking and reranking performance on TREC Web Track data.
Takeaways & Limitations
The models still fall significantly short of human judges when distinguishing documents among different relevance grades.
Abstract
from arXiv · showhide
In order to adopt deep learning for information retrieval, models are needed that can capture all relevant information required to assess the relevance of a document to a given user query. While previous works have successfully captured unigram term matches, how to fully employ position-dependent information such as proximity and term dependencies has been insufficiently explored. In this work, we propose a novel neural IR model named PACRR aiming at better modeling position-dependent interactions between a query and a document. Extensive experiments on six years' TREC Web Track data confirm that the proposed model yields better results under multiple benchmarks.
1 Introduction
Neural IR models had captured unigram matches, but using positional information such as proximity and term dependencies remained difficult. PACRR addresses this gap by combining convolutional and recurrent components in a relevance-matching model.
- Relevance matching models directly assess document content with respect to a query, unlike dense semantic matching models that compare low-dimensional representations.
- Traditional IR suggests modeling unigram matches, positional information, term dependencies, and query coverage as relevance signals.
- MatchPyramid and local DUET incorporate convolutional positional signals but have difficulty significantly outperforming DRMM.
- PACRR combines convolutional kernels and recurrent layers to better account for position-dependent relevance factors.
- PACRR uses similarity matrices, convolutional and k-max-pooling layers, and a recurrent layer to capture n-gram, positional, and query-term interactions.
2 The PACRR Model
PACRR converts query-document pairs into similarity matrices and processes them with convolutional, max-pooling, and recurrent layers. This architecture preserves positional signals, captures local n-gram patterns and query coverage, and produces a global relevance score.
- PACRR converts each query-document pair into a similarity matrix and deep architecture that outputs a query-document relevance score.
- Relevance matching: Similarity matrices encode term-level interactions, while n-gram matching and query coverage appear as local high-similarity patterns across document terms and query rows.
- Relevance matching: PACRR standardizes variable-length matrices through query padding and document distillation using firstk or kwindow strategies.
- Convolutional relevance matching: Convolutional layers with 2×2 through lg×lg kernels capture bi-gram through lg-gram matching, while the original matrix represents unigram matching.
- Two max pooling layers: Two max-pooling layers retain the ns strongest signals for each query term, preserving query coverage and different matching patterns.
- Recurrent layer for global relevance: An LSTM combines normalized query-term IDF with salient signals across kernel sizes to produce the document relevance score.
- Training objective: PACRR trains on query, relevant-document, and non-relevant-document triples using a pairwise max-margin loss.
3 Evaluation
PACRR was evaluated against neural IR baselines across three TREC Web Track benchmarks using ERR@20, nDCG@20, and pairwise accuracy. It consistently improved QL and many submitted rankings, while remaining limited on some document-pair distinctions and navigational relevance judgments.
- Experimental Setup: The evaluation compared PACRR with DRMM, DUETL, MatchPyramid, and K-NRM across QL reranking, all-run reranking, and document-pair classification.The benchmarks used 2009–2014 TREC Web Track data, including 300 queries and more than 100k judgments.
- Experimental Setup: PACRR was selected using validation ERR@20 and evaluated with ERR@20 and nDCG@20 on held-out test years.Training used five years for each test year, reserving 50 queries for validation and using 200 for training.
- RERANKSIMPLE: 60% average improvements over QL were achieved by both PACRR variants when reranking Web Track 2012–14 results.Both variants significantly improved QL on all three years under both metrics; PACRR-firstk ranked in the top three across years and metrics.
- RERANKALL: At least 80% of submitted runs improved in nDCG@20 after PACRR reranking on individual years, with all 2010–12 runs improved by PACRR-firstk.Both variants improved existing rankings by at least 10% across different years, although neither improved every submitted run across all six years.
- Discussion: The evaluation could not consistently improve all submitted runs, and navigational-versus-relevant distinctions remained difficult.The authors suggest user-intent classification as a possible direction for improving treatment of navigational documents.
- Discussion: PACRR-firstk is suitable when documents fit in memory, whereas PACRR-kwindow offers comparable results for longer documents.The choice can also be treated as a hyperparameter selected using held-out validation data.
- PAIRACCURACY: PACRR-kwindow and PACRR-firstk reached 73.5% and 74.1% pairwise accuracy, compared with 67.4% for DRMM.The comparison merged navigational, highly relevant, and relevant labels into binary judgments.
4 Conclusion
PACRR preserves positional information to model term dependencies and proximity in neural IR. Its architecture combines convolutional and recurrent processing, and experiments show strong improvements on TREC Web Track datasets.
- PACRR preserves positional information to model term dependencies and proximity in neural IR.
- Multiple convolutional layers with different sizes capture term dependencies and proximity.
- A recurrent layer combines salient signals across query terms after two max-pooling layers.
- PACRR substantially outperforms four state-of-the-art neural IR models on TREC Web Track ad-hoc datasets.