PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Ranking Models with Weak Supervision

Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, W. Bruce Croft

arXiv:1704.08803v2cs.IRcs.CLcs.LG

TL;DR

Neural ranking lacks abundant relevance labels, making it difficult to apply deep learning without supervision. The paper trains feed-forward neural rankers from BM25-generated weak labels and finds substantial MAP gains over BM25, while weakly supervised pre-training benefits supervised ranking with limited data.

  • Problem

    Neural ranking lacks readily available relevance-labeled data, and learning rankings from queries and documents without supervision is difficult.

  • Method

    The paper uses unsupervised IR models such as BM25 to generate weak labels and trains neural ranking models with varied objectives and input representations.

  • Results

    Over 13% and 35% MAP improvements over BM25 are reported on the Robust04 and ClueWeb collections, respectively.

  • Takeaways & Limitations

    Weakly supervised data can train neural rankers without supervised signals and can pre-train supervised neural ranking models when labeled data is limited.

  • Takeaways & Limitations

    Pair-wise inference requires averaging predictions against all candidate documents, giving O(n^2) time complexity and making the procedure impractical without approximations.

Abstract

from arXiv · show

Despite the impressive improvements achieved by unsupervised deep neural networks in computer vision and NLP tasks, such improvements have not yet been observed in ranking for information retrieval. The reason may be the complexity of the ranking problem, as it is not obvious how to learn from queries and documents when no supervised signal is available. Hence, in this paper, we propose to train a neural ranking model using weak supervision, where labels are obtained automatically without human annotators or any external resources (e.g., click data). To this aim, we use the output of an unsupervised ranking model, such as BM25, as a weak supervision signal. We further train a set of simple yet effective ranking models based on feed-forward neural networks. We study their effectiveness under various learning scenarios (point-wise and pair-wise models) and using different input representations (i.e., from encoding query-document pairs into dense/sparse vectors to using word embedding representation). We train our networks using tens of millions of training instances and evaluate it on two standard collections: a homogeneous news collection(Robust) and a heterogeneous large-scale web collection (ClueWeb). Our experiments indicate that employing proper objective functions and letting the networks to learn the input representation based on weakly supervised data leads to impressive performance, with over 13% and 35% MAP improvements over the BM25 model on the Robust and the ClueWeb collections. Our findings also suggest that supervised neural ranking models can greatly benefit from pre-training on large amounts of weakly labeled data that can be easily obtained from unsupervised IR models.

1 INTRODUCTION

The paper addresses the difficulty of training neural ranking models without relevance labels by using automatically generated weak supervision. It studies model objectives, input representations, and pre-training to improve ranking with limited supervision.

  • Ranking lacks the unsupervised deep-learning advances seen in other fields because learning relevance from queries and documents without supervision is difficult.
  • Weak supervision uses classic unsupervised IR models such as BM25 to automatically create noisy labels and billions of training instances at almost no cost.
  • The study compares point-wise and pair-wise ranking objectives with dense, sparse, and learned query/document embedding representations.
  • Ranking objectives target preferences rather than calibrated scores, while learned representations help networks generalize beyond imperfections in weak annotations.
  • Weakly labeled pre-training improves deep neural ranking performance when only a limited amount of supervised data is available.
  • The approach is presented as applicable beyond ranking, including filtering and classification, without requiring supervised data.

2 RELATED WORK

Prior neural ranking models generally depend on explicit relevance judgments or click logs, whereas this paper reuses traditional IR models to generate weak training data. The related work distinguishes representation-focused and interaction-focused neural architectures.

  • Neural ranking models include late combination architectures that independently represent queries and documents before computing similarity.
  • Early combination architectures instead feed query-document interactions directly into the network as input.
  • Existing neural ranking approaches were trained on explicit relevance judgments or clickthrough logs, but massive amounts of such data are not always available.
  • Weak supervision treats imperfect, cheaply obtained labels as training targets and can derive them from heuristics, related data, external knowledge, or partial annotation.
  • The paper uses traditional IR models as weak supervision signals to generate large training sets and train neural rankers that outperform baseline methods.

3 WEAK SUPERVISION FOR RANKING

The paper creates weakly labeled ranking data by applying an existing unsupervised retrieval model to unlabeled queries and documents, then trains neural ranking models on that signal.

  • Pseudo-labeling applies an unsupervised retrieval model such as BM25 to rank documents for many unlabeled training queries.
  • The resulting rankings or scores provide weak supervision for neural models that learn query-document representations and ad-hoc retrieval rankings.
  • The method can generate as much training data as needed from a large query set without relevance judgments.

4 NEURAL RANKING MODELS

The paper defines point-wise and pair-wise neural ranking models that learn from weakly supervised query-document scores. These models share a feed-forward architecture while differing in ranking objective, inference procedure, and input representation.

  • Ranking Models: Three ranking models are introduced: one point-wise score model and two pair-wise models, Rank and RankProb.The models are trained using weak supervision signals.
  • Score Model: The score model learns a retrieval score for each query-document pair from weak-supervisor relevance scores using mean squared error.Training instances contain a query, retrieved document, and weakly supervised relevance score.
  • Rank Model: The Rank model trains two parameter-sharing point-wise networks jointly with a pair-wise objective rather than learning calibrated scores.Its hinge loss penalizes examples violating the margin constraint, and ε=1 works well when outputs are compressed to [−1,1].
  • RankProb Model: RankProb predicts the probability that one document ranks above another during both training and inference.Its pair-wise training instances include a query, two documents, and their weak-supervisor scores.
  • Inference: Pair-wise predictions must be converted into scalar document scores at inference, requiring comparisons against all candidate documents with O(n^2) time complexity.The paper notes that approximations may reduce this inference cost for practical applications.
  • Neural Network Architecture: All models use a shared feed-forward network with an input mapping, fully connected hidden layers, ReLU activations, and a single continuous output.Dropout is used to reduce feature co-adaptation and overfitting; output activation varies by ranking architecture.
  • Input Representations: The input layer supports dense traditional-IR statistics, sparse term-frequency vectors, and embedding-based representations of queries and documents.Embedding representations learn term vectors, global term weights, and a compositional function during training; RankProb additionally represents a second document.

5 EXPERIMENTAL DESIGN

The experiments evaluate weakly supervised neural ranking on homogeneous news and heterogeneous web collections using human-labeled queries and standard retrieval metrics. Training uses real-user query logs, while BM25 candidate retrieval and neural reranking define the experimental pipeline.

  • Collections: The evaluation uses Robust04, a homogeneous collection with over 500k news articles, and ClueWeb09 Category B, a heterogeneous web collection with over 50 million English documents.Both are standard TREC collections.
  • Training Queries: Neural models are trained on unique AOL query strings sampled from a three-month period in 2006 after filtering navigational queries and non-alphanumeric characters.The queries come from real users of the AOL search engine.
  • Evaluation Queries: Evaluation uses 250 Robust04 queries from TREC Robust Track 2004 and 200 ClueWeb queries from TREC Web Tracks 2009–2012.Only topic titles are used as queries for ClueWeb.
  • Metrics: Retrieval effectiveness is measured with MAP@1000, P@20, and nDCG@20, with significance tested by two-tailed paired t-tests using p<0.05 and Bonferroni correction.The metrics assess ranked retrieval quality at the top 1000 or top 20 documents.
  • Implementation: The experiments use TensorFlow and Adam, tune hyperparameters on validation sets with batched GP bandits, and select batch sizes from [128,256,512].At inference, models rerank the top 2000 BM25-retrieved documents; BM25 uses Indri4 default parameters.

6 RESULTS AND DISCUSSION

Neural ranking models trained with BM25-based weak supervision outperform BM25 when they use ranking objectives and learned embedding representations. Analyses indicate that representation learning, rather than nonlinearity alone, drives the strongest gains and can also support supervised ranking with limited data.

  • Model comparisons: Over 13% and 35% MAP improvements over BM25 were achieved on Robust04 and ClueWeb, respectively.The improvements were reported for the best-performing neural ranking approach, with similar gains on other evaluation metrics.
  • Model comparisons: Ranking objectives outperform score prediction because they learn document preferences instead of reproducing imperfect weak-supervision scores.The paper contrasts rank and rankprob objectives with the score model in the weak-supervision setting.
  • Model comparisons: Embedding-vector representations outperform dense and sparse alternatives by letting networks learn query-document representations and interactions.The authors report this pattern across models and argue that engineered inputs can prevent the network from moving beyond the weak signal.
  • Model comparisons: Rankprob is more precise in MAP, whereas rank is faster at inference, with O(n) versus O(n^2) complexity.This presents a practical accuracy–inference-speed trade-off between the two ranking objectives.
  • Model behavior: Dense and sparse models overfit fully featurized inputs: training losses approach zero while validation losses do not, limiting generalization.The authors observed this behavior despite trying L2 regularization and dropout.
  • Model behavior: Models with similar input representations have similar query-level performance, while architecture alone does not determine performance proximity.Embedding-based models behave differently from other input representations, and the dense model is closest to BM25.

7 CONCLUSIONS

The paper uses traditional IR models such as BM25 to generate weakly labeled training data for neural ranking models, and identifies design ingredients supporting strong performance. Across Robust04 and ClueWeb, the best models outperform BM25, while weakly supervised pre-training helps when labeled data are limited.

  • The approach generates large training sets by using BM25 as a weak supervision signal for neural ranking models.The study varies ranking architectures, objectives, and input representations.
  • 13% and 35% MAP improvements over BM25 were achieved on Robust04 and ClueWeb, respectively.The models were trained with over six million queries and evaluated in ad-hoc retrieval.
  • Good weakly supervised performance depends on proper input representations, suitable objectives, and considerable training data.The authors emphasize learning features from raw data and supplying enough examples to learn effective embeddings.
  • Embedding representations may require many examples to learn retrieval-effective embeddings, which weak supervision can provide at low cost.
  • Future work includes more expressive architectures and combining multiple weak supervision signals such as query likelihood and PageRank.
Loading 1704.08803v2…