PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Neural Ad-hoc Ranking with Kernel Pooling

Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, Russell Power

arXiv:1706.06613v1cs.IRcs.CL

TL;DR

Ad-hoc ranking needs reliable soft matches because exact matching suffers from vocabulary mismatch, while generic distributed representations have had limited ranking success. K-NRM combines embedding-based word interactions, differentiable multi-level kernel pooling, and a learning-to-rank layer trained end-to-end. Across commercial query-log scenarios, it outperforms feature-based and neural baselines by as much as 65%, with analyses linking the gains to kernel-guided, relevance-focused embeddings.

  • Problem

    Generic distributed representations have had limited success for document ranking, while soft matches must be used carefully because exact matches are stronger relevance signals.

  • Method

    K-NRM uses embedding-based translation matrices, differentiable kernels for multi-level soft-TF features, and a learning-to-rank layer optimized end-to-end.

  • Results

    65%: K-NRM outperforms feature-based and neural ranking baselines across in-domain, cross-domain, and raw-click testing scenarios, with robust gains across query types.

  • Takeaways & Limitations

    Kernel-guided embedding learning produces relevance-focused, multi-level soft matches, and removing this learning substantially diminishes K-NRM’s advantage.

  • Takeaways & Limitations

    K-NRM’s large embedding parameter space may require substantial training data to avoid overfitting.

Abstract

from arXiv · show

This paper proposes K-NRM, a kernel based neural model for document ranking. Given a query and a set of documents, K-NRM uses a translation matrix that models word-level similarities via word embeddings, a new kernel-pooling technique that uses kernels to extract multi-level soft match features, and a learning-to-rank layer that combines those features into the final ranking score. The whole model is trained end-to-end. The ranking layer learns desired feature patterns from the pairwise ranking loss. The kernels transfer the feature patterns into soft-match targets at each similarity level and enforce them on the translation matrix. The word embeddings are tuned accordingly so that they can produce the desired soft matches. Experiments on a commercial search engine's query log demonstrate the improvements of K-NRM over prior feature-based and neural-based states-of-the-art, and explain the source of K-NRM's advantage: Its kernel-guided embedding encodes a similarity metric tailored for matching query words to document words, and provides effective multi-level soft matches.

1 INTRODUCTION

K-NRM addresses the difficulty of using soft matches reliably in document ranking by combining kernel pooling with end-to-end neural ranking. Experiments show robust gains over feature-based and neural ranking baselines, attributed to kernel-guided soft matching.

  • Motivation: Soft matches are weaker and less reliable than exact matches for ad-hoc ranking, making their effective use an open problem.The paper illustrates that embedding similarity can confuse related but query-inappropriate word pairs.
  • Approach: K-NRM represents query and document words with embeddings, constructs a translation model, and uses kernel pooling to softly count word pairs at multiple similarity levels.These soft-TF features are combined by a ranking layer, and all layers are differentiable for end-to-end optimization.
  • Approach: Kernel-guided embedding learning converts ranking loss into soft-TF pattern requirements and tunes embeddings toward a relevance-tailored similarity metric.The learned embeddings provide multi-level soft matches for query-document ranking.
  • Results: 65%: K-NRM outperforms feature-based and neural ranking states-of-the-art across in-domain, cross-domain, raw-click, head, torso, and tail scenarios.The experiments use a commercial search engine’s query log.

2 RELATED WORK

Prior retrieval models rely on exact matching, precomputed translation similarities, or learned representations of text and interactions. K-NRM combines customized embeddings with interaction-based kernel pooling to learn relevance-focused soft matches.

  • Traditional and translation models: Query-likelihood and BM25 models rely on exact query-document matches, while translation models use word-pair similarities to address vocabulary mismatch.Translation models can soft-match query words to document words using a precomputed translation matrix.
  • Embedding-based matching: Word embeddings provide a smooth, low-level approximation of word similarities and can improve translation models.Word2vec embeddings are described as reflecting word-pair PMI structure.
  • Embedding adaptation: Research has customized context-trained embeddings for search by using both embedding sides or adapting embeddings with pseudo-relevance feedback.These approaches question whether generic surrounding-context embeddings are suitable for ad-hoc ranking.
  • Neural ranking: Neural ranking models include representation-based approaches that rank from whole-text embeddings and interaction-based approaches that learn word-level matching patterns.Examples include DSSM, CDSSM, ARC-II, MatchPyramid, and DRMM.
  • K-NRM: K-NRM combines customized word embeddings with interaction-based neural ranking, using kernels to encode relevance preferences while producing soft matches.Its architecture includes embedding, translation, kernel-pooling, and learning-to-rank components.

3 KERNEL BASED NEURAL RANKING

K-NRM ranks query-document pairs from word-level embedding similarities by pooling multi-level soft matches into ranking features. Pairwise ranking loss backpropagates through the kernels to tune both ranking parameters and embeddings.

  • Translation Model: K-NRM constructs a translation matrix whose entries are embedding similarities between query words and document words.This uses embeddings rather than independently learning every word-pair similarity, reducing the parameterization from |V|^2 to |V| × L.
  • Kernel-Pooling: Kernels pool each query word’s document-word similarities into K-dimensional features, whose log-sums form the query-document ranking vector.The model uses differentiable kernels to summarize word-level interactions.
  • Kernel-Pooling: RBF kernels softly count matches around similarity levels defined by µ, with σ controlling the width of each soft-TF count.As σ approaches infinity, pooling becomes mean pooling; µ = 1 with σ approaching zero yields exact-match counting.
  • Learning to Rank: The ranking layer combines kernel features into the final score using learnable parameters and tanh activation, which is rank-equivalent to a typical linear learning-to-rank model.The model requires no manual ranking features.
  • Learning End-to-End: Pairwise ranking loss updates ranking parameters and propagates gradients through kernels, word similarities, and embeddings.The gradients adjust kernel scores and word-pair similarities so relevant documents receive higher scores than irrelevant documents.
  • Learning End-to-End: Kernel-guided updates align embeddings with multi-level soft-TF patterns that separate relevant from irrelevant documents during training.At test time, learned embeddings generate soft-TF features that the ranking layer converts into scores.

4 EXPERIMENTAL METHODOLOGY

The experiments evaluate K-NRM on click-derived relevance judgments across multiple testing scenarios, using commercial search logs and comparisons with traditional, feature-based, and neural ranking baselines. Evaluation emphasizes early ranking quality, while implementation choices include title-only neural inputs and a larger parameter space for K-NRM.

  • Data: The dataset contains 35 million Sogou search sessions and 96,229 distinct queries, with queries, displayed documents, clicks, and dwell times.Each query has an average of 12 displayed documents.
  • Data: Testing uses 1,000 head queries, while remaining queries train the neural models and tail-query performance assesses robustness.Head queries appeared more than 1,000 times in the query log.
  • Relevance Labels: Training relevance labels are generated from click data using DCTR relevance scores and preference pairs.DCTR computes query-document relevance from click-through rates.
  • Evaluation Scenarios: Testing-SAME uses DCTR, Testing-DIFF uses TACM with clicks and dwell times, and Testing-RAW uses the clicked document from single-click sessions as relevant.Testing-RAW covers single-click sessions, which represent 57% of testing sessions.
  • Metrics: Testing-SAME and Testing-DIFF use NDCG@1, NDCG@3, and NDCG@10, whereas Testing-RAW uses MRR; significance uses permutation tests with p < 0.05.The evaluation focuses on early ranking positions for commercial search.
  • Baselines and Implementation: Baselines include BM25, language models, RankSVM, coordinate ascent, DRMM, CDSSM, and Trans, while neural models and K-NRM use titles for training and testing.K-NRM has a larger parameter space because it learns an embedding for every Chinese word, which may require more data to avoid overfitting.

5 EVALUATION RESULTS

K-NRM consistently outperforms feature-based and neural baselines across label conditions, click-based evaluation, and query frequencies. Analyses attribute this advantage to kernel-guided multi-level soft matching and relevance-trained embeddings, while performance depends on training data coverage.

  • Ranking Accuracy: 83% and 28% are K-NRM’s margins over the next-best neural model on Testing-SAME NDCG@1 and NDCG@10, respectively.K-NRM also outperforms word-based baselines by over 65% on NDCG@1 and over 20% on NDCG@10.
  • Ranking Accuracy: More than 40% on NDCG@1 and 10% on NDCG@10 are K-NRM’s improvements over all baselines under Testing-DIFF.Testing-DIFF uses TACM-inferred relevance labels to assess robustness across differing click-model biases.
  • Ranking Accuracy: MRR rises from 0.2415 with Coor-Ascent to 0.3379 with K-NRM on Testing-RAW.The improvement corresponds to moving the clicked document from rank 4 to rank 3 and is largest at early ranking positions.
  • Source of Effectiveness: K-NRM’s advantage comes from kernel-guided embeddings that enforce different adjustments for query-document word pairs across similarity levels.Unlike click2vec and mean-pool, K-NRM propagates relevance signals through kernels that enforce multi-level soft matches.
  • Source of Effectiveness: Only about 10% of word pairs with similarities ≥0.5 are retained, while other pairs move toward white-noise kernels.The learned embeddings decouple broadly related terms and recover task-specific, irrelevant, synonym, and other relevance patterns.

6 CONCLUSION

K-NRM combines kernel-based multi-level soft matching with end-to-end ranking, and its learned embeddings encode relevance preferences for ad-hoc search. Experiments show robust gains over feature-based and neural ranking baselines, while analyses attribute the advantage to kernel-guided embedding learning.

  • K-NRM uses word-level interactions, kernel pooling for multi-level soft matches, and a learning-to-rank layer trained end-to-end.The kernels softly count matches at different similarity levels, producing ranking features from query-document word interactions.
  • K-NRM outperforms feature-based and neural ranking baselines by as much as 65% across in-domain, cross-domain, and raw-click scenarios.The gains are reported on a commercial search engine’s query log and are strongest at top ranking positions.
  • K-NRM’s advantage diminishes substantially without kernel-guided embedding learning, including variants using only exact matching, pretrained word2vec, or uni-level training.Some variants fail to outperform simple feature-based baselines.
  • Embedding analysis shows that more than 90% of word pairs mapped together by word2vec are decoupled for the stricter soft-match requirements of ad-hoc search.The model also maps search-task-related pairs to selected similarity levels and moves pairs between kernels according to their roles.
  • The authors present the findings as evidence that neural methods for information retrieval require task-specific understanding and tailoring.
Loading 1706.06613v1…