PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a Deep Listwise Context Model for Ranking Refinement

Qingyao Ai, Keping Bi, Jiafeng Guo, W. Bruce Croft

arXiv:1804.05936v2cs.IR

TL;DR

Global ranking functions can overlook query-specific feature distributions, making rankings suboptimal for individual queries. The paper proposes DLCM, which encodes top-result features with an RNN and re-ranks them using local context and an attention-based loss. Experiments report significant improvements over state-of-the-art learning-to-rank methods on benchmark corpora, especially for identifying the best document.

  • Problem

    Global ranking functions may be suboptimal because they ignore differences in relevant-document feature distributions across queries.

  • Method

    DLCM sequentially encodes top retrieved documents’ original feature vectors with an RNN, then uses the learned local context to re-rank results with an attention-based listwise loss.

  • Results

    The model significantly and consistently improves state-of-the-art learning-to-rank methods on benchmark corpora, particularly in finding the best document from initial ranked lists.

  • Takeaways & Limitations

    Local ranking context can improve existing learning-to-rank systems when refining ranked results, with particular value at high ranks.

  • Takeaways & Limitations

    Experiments could not use query or document text because the datasets lacked access, so raw-text relevance models and query-similarity baselines were excluded.

Abstract

from arXiv · show

Learning to rank has been intensively studied and widely applied in information retrieval. Typically, a global ranking function is learned from a set of labeled data, which can achieve good performance on average but may be suboptimal for individual queries by ignoring the fact that relevant documents for different queries may have different distributions in the feature space. Inspired by the idea of pseudo relevance feedback where top ranked documents, which we refer as the \textit{local ranking context}, can provide important information about the query's characteristics, we propose to use the inherent feature distributions of the top results to learn a Deep Listwise Context Model that helps us fine tune the initial ranked list. Specifically, we employ a recurrent neural network to sequentially encode the top results using their feature vectors, learn a local context model and use it to re-rank the top results. There are three merits with our model: (1) Our model can capture the local ranking context based on the complex interactions between top results using a deep neural network; (2) Our model can be built upon existing learning-to-rank methods by directly using their extracted feature vectors; (3) Our model is trained with an attention-based loss function, which is more effective and efficient than many existing listwise methods. Experimental results show that the proposed model can significantly improve the state-of-the-art learning to rank methods on benchmark retrieval corpora.

1 INTRODUCTION

Global learning-to-rank functions can miss query-specific relevance distributions, motivating local context from top results to refine initial rankings. DLCM directly encodes ranked-document features with an RNN and re-ranks results using an attention-based listwise approach.

  • Motivation: Global ranking functions may be suboptimal because relevant documents have different feature distributions across queries.For example, freshness matters more for political-news queries, while word matching may matter more for entertainment queries.
  • Motivation: Per-query ranking models are impractical because unseen queries lack labeled data and the query space is nearly infinite.Learning a local model from top retrieved documents offers a more practical compromise.
  • Motivation: Top-ranked documents provide local ranking context that can convey information about a query’s characteristics.Prior observations motivate using this context to refine learning-to-rank results.
  • Challenge: Extracting useful patterns from high-dimensional top-result feature vectors remains challenging, while prior methods often rely only on document text and require extra feature extraction.Those approaches can ignore ranking signals beyond textual representations.
  • Approach: DLCM sequentially encodes top-result feature vectors with an RNN, learns a local context embedding, and uses it to re-rank the initial results.The model is designed to build on existing learning-to-rank features and ranked lists without additional retrieval processing.
  • Results: Experiments on large-scale learning-to-rank corpora show significant and consistent improvements over state-of-the-art learning-to-rank algorithms.Analysis further reports particular strength in finding the best document among a group of results.

2 RELATED WORK

Related work spans general learning-to-rank methods, query-specific ranking, and top-result feature extraction. Existing approaches often require raw document text and overlook relationships encoded by other ranking features.

  • Learning-to-rank algorithms: Learning to rank applies machine learning to feature representations of query-document pairs across retrieval, question answering, and recommendation.Methods are commonly categorized as pointwise, pairwise, or listwise according to how training losses are computed.
  • Learning-to-rank algorithms: Deep learning-to-rank models generally learn a global relevance function from each query-document pair’s own information.They differ in loss functions and architectures but remain focused on global scoring.
  • Query-specific ranking: Query-specific ranking can use separate or similar-query models, but independent per-query training is infeasible because unseen queries lack labeled data.Prior work therefore constructs models from training queries or similar-query groupings.
  • Top-result context: Pseudo relevance feedback and related methods extract features from top retrieved documents to improve the initial ranking.These methods commonly model document similarity using word distributions.
  • Top-result context: Existing top-result methods require accessing raw document text after retrieval and ignore document relationships represented by non-term ranking features.This makes them expensive and limited when other ranking signals are important.

3 LEARNING TO RANK WITH THE LOCAL RANKING CONTEXT

The local-context formulation augments global learning-to-rank with query-specific information from the initially ranked documents. It seeks a context model and scoring function that jointly minimize ranking loss while operating directly on existing scalar features.

  • Problem formulation: Traditional learning-to-rank represents each query-document pair with x(q,d) and learns a global function f that outputs document ranking scores.The global function is obtained by minimizing a loss over labeled query-document pairs.
  • Problem formulation: For query q, the local context uses documents R_q sorted by the global score and their feature vectors X_q.This context summarizes the top retrieved list produced by the initial ranking function.
  • Context-aware ranking: The context-aware objective learns a local context model I(R_q,X_q) together with a scoring function ϕ that ranks documents using both features and context.The goal is to minimize the resulting loss function L.
  • Model requirements: A useful listwise context model must process scalar ranking features directly rather than requiring raw-document extraction or manually designed heuristics.This requirement supports reuse of ranking signals already produced by learning-to-rank systems.

4 DEEP LISTWISE CONTEXT MODEL

DLCM encodes the feature vectors of top-ranked documents with a GRU-based RNN to learn local ranking context, then re-ranks those documents using the encoded context and hidden outputs. Its Attention Rank loss focuses optimization on relative result importance and trains faster than ListMLE and SoftRank in the reported experiments.

  • Model Architecture: DLCM encodes top retrieved documents from a global ranking model with a GRU-based RNN and uses the resulting local context to re-rank them.The pipeline first generates a ranked list, encodes its feature vectors sequentially, and applies a local ranking function.
  • Context Encoding: The RNN processes documents from the lowest position to the highest, producing a final state that represents the encoded local ranking context.The recurrent structure combines current inputs with previous inputs and captures positional information.
  • Input Representation: The model directly uses existing query-document feature vectors without incorporating additional input features.These features include both document- and query-related signals.
  • Model Variants: Replacing the unidirectional RNN with a bidirectional RNN produced no improvement, while using only the reversed direction made performance significantly worse.The reported experiments indicate that reversed-direction information was not useful for retrieval.
  • Re-ranking: The local ranking function computes each document's score from its hidden output and the final encoded context representation.The score function is described as an attention-like function over the RNN outputs and context state.
  • Loss Function: Attention Rank compares model-derived attention with relevance-based attention and emphasizes the relative importance of results rather than directly predicting relevance labels.Its training was 2 and 20 times faster than DLCM trained with ListMLE and SoftRank, respectively, in the experiments.

5 EXPERIMENTAL SETUP

The experiments evaluate DLCM on three large benchmark learning-to-rank corpora using global retrieval baselines, re-ranking baselines, and a feature-concatenation neural baseline. Evaluation uses NDCG and ERR at ranks 1, 3, 5, and 10, with statistical significance tested at p ≤0.01.

  • Datasets: Experiments use Microsoft 30k, Microsoft 10k, and Yahoo! Webscope v2.0, which provide feature vectors and five-level relevance judgments.The Microsoft datasets use five-fold partitions with three training, one validation, and one testing fold.
  • Baselines: SVMrank and LambdaMART serve as global learning-to-rank baselines for initial retrieval.SVMrank is trained with pairwise losses, whereas LambdaMART is described as a state-of-the-art listwise-loss algorithm.
  • Baselines: The study compares DLCM trained with ListMLE, SoftRank, and Attention Rank against corresponding DNN re-ranking baselines.The DNN baselines use feed-forward networks with the same three loss functions.
  • Baselines: LIDNN provides an additional baseline by concatenating all ranked-list features and predicting document scores jointly with a feed-forward neural network.This tests DLCM against a direct feature-concatenation approach for modeling ranked-list context.
  • Scope: Text-based query and document models are excluded because the datasets do not provide text, preventing relevance modeling, document-relationship features, and query-similarity computation.The experimental goal is to improve learning-to-rank with local ranking context rather than construct new features or query-expansion models.
  • Evaluation and Training: Evaluation reports NDCG and ERR at ranks 1, 3, 5, and 10, with statistical differences assessed using the Fisher randomization test at p ≤0.01.The global ranking model is tuned on validation NDCG@10, and DLCM hyperparameters include list size n, abstraction dimension β, and hidden-unit number k.

6 RESULTS AND ANALYSIS

Across benchmark corpora, DLCM re-ranking produced stable improvements over global learning-to-rank baselines, while its gains were especially associated with exploiting local ranking context and identifying highly relevant documents. Analyses further examined baseline behavior, relevance-label promotions, query composition, and parameter sensitivity.

  • Overall performance: Re-ranking baselines could improve weak initial rankers but hurt strong ones, because they retained global assumptions and added no new information.SVMrank benefited from ListMLE, SoftRank, and AttRank, whereas LambdaMART performance decreased with these baselines.
  • Overall performance: DLCMs delivered stable and significant improvements across the evaluated global ranking algorithms.On Microsoft 30K, DLCM with SoftRank improved SVMrank by 40.9% on NDCG@1 and 80.6% on ERR@1; DLCM with AttRank improved LambdaMART by 1.3% and 4.7%, respectively.
  • Overall performance: DLCM with AttRank generally outperformed its ListMLE and SoftRank variants across several dataset–ranker combinations.The passage reports consistent gains over ListMLE and wins over SoftRank on Microsoft 30K LambdaMART, Microsoft 10K LambdaMART, and Yahoo! SVMrank; AttRank was also empirically faster.
  • Overall performance: DLCM gains were relatively small on Yahoo! LETOR set 1 because baseline rankings were nearly perfect and its features were individually high-quality ranking signals.LambdaMART reached 0.738 on NDCG@10 there, leaving less new information for DLCM to learn.
  • Pair-wise Ranking Analysis: DLCM most strongly promoted perfect documents, averaging 1.88 fewer negative ranking pairs per document, while excellent documents improved by 0.43 and lower labels changed little.The analysis concludes that DLCMs find perfect and excellent results effectively but do not discriminate fair results from irrelevant ones well.
  • Pair-wise Ranking Analysis: For queries with four perfect documents, NegPair reduction reached 2.64 (22.8%), compared with 0.99 (13.5%) for queries with one perfect document.NegPair reduction increased as the number of perfect results in the query increased, indicating greater usefulness when multiple perfect results were initially present.

7 CONCLUSION AND FUTURE WORK

The paper concludes that DLCM improves learning-to-rank systems by exploiting local ranking context, while identifying ranking diversity as an open direction for future work.

  • DLCM significantly improves baseline learning-to-rank methods on benchmark datasets and is particularly effective at finding the best documents in initial ranked lists.
  • The model uses an attention-based listwise loss, can be efficiently trained, and can be deployed over existing learning-to-rank models without additional feature extraction or retrieval processing.
  • The experimental results support the hypothesis that local ranking context from top retrieved documents is valuable for learning to rank.
  • Ranking diversity remains an unexamined concern because using local context may place similar documents in high positions; future work proposes decoding with ranking features and previous outputs.
  • Figure 4 examines DLCM performance on Microsoft 30k under different hyper-parameter settings.
Loading 1804.05936v2…