PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval

Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Jingfang Xu, Xueqi Cheng

arXiv:1710.05649v2cs.IR

TL;DR

Existing deep IR models generate ranking scores without explicitly modeling relevance judgment. DeepRank simulates human judgment by detecting query-centric contexts, measuring local relevance with CNN or 2D-GRU, and aggregating signals with RNN and term gating. Experiments on LETOR4.0 and large-scale clickthrough data report significant improvements over existing deep IR and learning-to-rank baselines.

  • Problem

    Existing deep IR models generate ranking scores without an explicit model of the relevance-generation process, while handcrafted learning-to-rank features are time-consuming and incomplete.

  • Method

    DeepRank detects query-centric contexts, measures their local relevance with CNN or 2D-GRU, and aggregates them using an RNN and term-gating network.

  • Results

    DeepRank significantly outperforms existing deep IR models and pairwise and listwise learning-to-rank baselines on LETOR4.0 and large-scale clickthrough data.

  • Takeaways & Limitations

    Modeling detection, local relevance, and aggregation enables DeepRank to capture exact and semantic matching, proximity, query-term importance, and diverse relevance requirements.

  • Takeaways & Limitations

    The paper leaves the differences between DeepRank’s learned representations and effective learning-to-rank features for future investigation.

Abstract

from arXiv · show

This paper concerns a deep learning approach to relevance ranking in information retrieval (IR). Existing deep IR models such as DSSM and CDSSM directly apply neural networks to generate ranking scores, without explicit understandings of the relevance. According to the human judgement process, a relevance label is generated by the following three steps: 1) relevant locations are detected, 2) local relevances are determined, 3) local relevances are aggregated to output the relevance label. In this paper we propose a new deep learning architecture, namely DeepRank, to simulate the above human judgment process. Firstly, a detection strategy is designed to extract the relevant contexts. Then, a measure network is applied to determine the local relevances by utilizing a convolutional neural network (CNN) or two-dimensional gated recurrent units (2D-GRU). Finally, an aggregation network with sequential integration and term gating mechanism is used to produce a global relevance score. DeepRank well captures important IR characteristics, including exact/semantic matching signals, proximity heuristics, query term importance, and diverse relevance requirement. Experiments on both benchmark LETOR dataset and a large scale clickthrough data show that DeepRank can significantly outperform learning to ranking methods, and existing deep learning methods.

1 INTRODUCTION

DeepRank addresses limitations of handcrafted-feature learning to rank and earlier deep IR models by modeling relevance as detection, local measurement, and aggregation. Experiments on LETOR4.0 and clickthrough data report superior performance across the evaluated baselines.

  • Handcrafted features make learning-to-rank feature engineering time-consuming, incomplete, and over-specified.
  • Earlier deep IR models use automatically learned features from raw query and document text to address the feature-engineering problem.
  • DeepRank simulates human relevance judgment through relevant-location detection, local-relevance measurement, and global aggregation.
  • Its measure and aggregation components capture exact and semantic matching, passage retrieval, proximity, query-term importance, and diverse relevance requirements.
  • DeepRank significantly outperforms existing deep IR models and pairwise and listwise learning-to-rank baselines on LETOR4.0 and large-scale clickthrough data.

2 RELATED WORK

Related work includes feature-combining learning-to-rank methods and deep models that automatically learn ranking features or text-matching representations. These approaches differ in whether they model individual texts, local interactions, or ranked lists.

  • Learning-to-rank methods are categorized as pointwise, pairwise, or listwise according to their input and output spaces, hypotheses, and loss functions.
  • Pointwise methods score individual documents, pairwise methods learn preferences between document pairs, and listwise methods produce ranked lists.
  • DSSM maps queries and documents into a common semantic space and computes relevance using cosine similarity, while CDSSM uses CNNs to preserve local word order.
  • Deep text-matching models are commonly divided into representation-focused and interaction-focused approaches.
  • Interaction-focused models build local interactions between two texts and learn more complex interaction patterns for matching.

3 MOTIVATION

The paper derives DeepRank from a three-stage account of human relevance judgment: detecting relevant locations, measuring local relevance, and aggregating those local signals into a final label.

  • Human annotators scan document locations to find local relevance information, guided by the query-centric assumption that relevant information occurs around query terms.
  • The relevance label is generated by detecting relevant locations, measuring local relevance, and aggregating local relevances.
  • DeepRank is designed to model these three judgment stages as the problems of locating, measuring, and aggregating relevance.

4 DEEPRANK

DeepRank implements detection, local measurement, and aggregation as coordinated components that transform query-centered document contexts into a global ranking score.

  • DeepRank contains a Detection Strategy, Measure Network, and Aggregation Network corresponding to the three relevance-processing problems.
  • The detection step extracts query-centric contexts to represent locations where relevance occurs.
  • A CNN or 2D-GRU measures local relevance between the query and each query-centric context.
  • The aggregation step uses an RNN and term-gating network to combine local relevances into a global ranking score.
  • A query-centric context is a continuous document sequence centered on a position where a query term appears.

4.2 Measure Network

The measure network combines query and context representations with word-level interactions, then uses CNN or 2D-GRU architectures to determine local relevance while capturing proximity patterns.

  • Input Tensor: A word-level interaction matrix represents similarities between query words and context words.An indicator function captures exact matches, while cosine similarity captures semantic matching signals.
  • Input Tensor: The input tensor combines query representations, query-centric context representations, and word-level interaction signals.Each interaction element can encode exact or semantic matching, while the tensor extends those interactions with the corresponding word embeddings.
  • CNN: The one-layer CNN extracts matching patterns from the input tensor and max-pooling filters significant patterns for local relevance determination.Different kernels extract varied matching patterns before pooling selects salient ones for further relevance determination.
  • 2D-GRU: 2D-GRU recursively scans matrix or tensor positions, using top, left, diagonal, and current-position information to accumulate matching signals.Its hidden representation uses four gates, and the final output is taken from the bottom-right position or concatenated from bidirectional endpoints.
  • Proximity Heuristics: CNN and 2D-GRU capture proximity heuristics by modeling matching patterns associated with query terms occurring close together.The paper identifies proximity as an important retrieval signal and explains that both architectures can represent it through local pattern extraction or sequential accumulation.

4.3 Aggregation Network

The aggregation network integrates local relevances first by query term and then across terms, incorporating context positions, term importance, and diverse relevance requirements.

  • Aggregation Principles: Aggregation addresses query term importance and diverse matching requirements when converting local relevances into a global relevance score.The architecture treats some query terms as more important and considers how relevant contexts are distributed across positions.
  • Position Encoding: Each local relevance vector is augmented with a position indicator for its query-centric context.Position functions encode where the context occurs before query-term-level aggregation.
  • Query Term Level Aggregation: Local relevances centered on the same query term are grouped and integrated by an RNN using their position information.The resulting representation is a term-level global relevance representation, with GRU given as an example RNN.
  • Term Gating: A term gating network linearly combines query-term-level relevance representations using a weight parameter for each query term.This final aggregation accounts for the differing importance of query terms.

4.4 Model Training

DeepRank is trained end-to-end with stochastic gradient descent using a pairwise hinge loss for ranking, with regularization and early stopping used to address overfitting.

  • Optimization: DeepRank is trained end-to-end with stochastic gradient descent methods such as Adam.The implementation also uses L2 regularization and early stopping to deal with overfitting.
  • Ranking Loss: The pairwise hinge loss compares positive and negative documents for a query using their DeepRank relevance scores.The loss is L(q, d+, d−) = max(0, 1 − F(q, d+) + F(q, d−)).

5 EXPERIMENTS

The experiments evaluate DeepRank against learning-to-rank and deep-learning baselines on LETOR4.0 and ChineseClick, with additional analyses of data, tensor, and position-function choices. DeepRank significantly outperforms the baselines, while results show that dataset scale and architecture settings affect performance.

  • Experimental setup: Experiments compare DeepRank with learning-to-rank and deep-learning methods on LETOR4.0 and large-scale ChineseClick data.LETOR4.0 supports comparisons with both method families, while ChineseClick is used to compare deep models.
  • Datasets: LETOR4.0 evaluation uses MQ2007 and an expanded MQ2008 training set, totaling 69,623 and 84,834 query-document pairs, respectively.The MQ2008 training set is merged with MQ2007 training data because its original query count is too small for deep-model training.
  • Datasets: ChineseClick contains 12,520 queries, 115,562 documents, and 118,835 query-document pairs after preprocessing.The data comes from a commercial Chinese search engine and uses clicked documents as relevant examples.
  • Performance comparison: On MQ2007 and MQ2008, existing deep-learning models do not perform comparably with learning-to-rank methods, whereas deep models improve on larger ChineseClick data.The paper attributes the difference to the smaller LETOR4.0 datasets providing less data for optimizing deep models.
  • Performance comparison: On ChineseClick, DeepRank improves over BM25 by 21.0% in NDCG@1 and 11.5% in MAP, and over Match-SRNN by 11.0% and 4.3%, respectively.DeepRank is reported as the best-performing model among the compared deep-learning methods.
  • Model analysis: Using both word representations and word-level interactions in the input tensor improves performance, while reciprocal position weighting outperforms constant weighting.The position-function analysis also reports exponential performance comparable to reciprocal and linear performance slightly worse.

6 CONCLUSIONS AND FUTURE WORK

DeepRank is presented as a three-component deep learning architecture for relevance ranking that simulates human relevance judgment and outperforms learning-to-rank methods and existing deep IR models. Future work will compare its learned representations with effective hand-crafted learning-to-rank features.

  • 6 CONCLUSIONS AND FUTURE WORK: DeepRank combines detection, local relevance measurement, and aggregation to produce a global relevance score.Its components use query-centric contexts, CNN or 2D-GRU, and RNN with term gating.
  • 6 CONCLUSIONS AND FUTURE WORK: DeepRank captures exact and semantic matching, proximity heuristics, query term importance, and diverse relevance requirements.
  • 6 CONCLUSIONS AND FUTURE WORK: DeepRank significantly outperforms learning-to-rank methods and existing deep IR models on LETOR4.0 and large-scale clickthrough data.The paper identifies it as the first deep IR model to outperform existing learning-to-rank models.
  • 6 CONCLUSIONS AND FUTURE WORK: Future work will investigate differences between DeepRank’s automatically learned representations and effective learning-to-rank features.The authors state that this comparison may provide insights for designing more powerful deep IR architectures.

A DATA PREPROCESSING

The preprocessing pipeline normalizes and filters query and document text before modeling.

  • A DATA PREPROCESSING: Queries and documents are white-space tokenized, lower-cased, and stemmed with the Krovetz stemmer.Stopwords are removed using the INQUERY stop list.
  • A DATA PREPROCESSING: Document words occurring fewer than five times in the collection are removed.

B QUERY MATRIX & CONTEXT MATRIX

The query and context representations are expanded into a three-order tensor combining reduced word representations with word-level interactions.

  • B QUERY MATRIX & CONTEXT MATRIX: The query matrix and context matrix constructions are described as the basis for the model’s input representation.
  • B QUERY MATRIX & CONTEXT MATRIX: Each interaction element is extended to a three-dimensional vector incorporating query and query-centric context word representations.xi and yj denote the word embeddings of wi and vj, respectively.
  • B QUERY MATRIX & CONTEXT MATRIX: Linear transformations W Q and W D reduce word-embedding dimensions before constructing the input representation.The passage gives reduction from 50 dimensions to 2 dimensions as an example.
  • B QUERY MATRIX & CONTEXT MATRIX: The resulting three-order tensor combines a query matrix, a query-centric context matrix, and a word-level interaction matrix.

C CODE

Two released versions of DeepRank are available: the original TextNet implementation and a newer PyTorch implementation.

  • C CODE: The original DeepRank implementation is released as TextNet.
  • C CODE: A newer version of DeepRank is implemented in PyTorch.
Loading 1710.05649v2…