Source-linked AI summary
A Deep Look into Neural Ranking Models for Information Retrieval
Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, Xueqi Cheng
TL;DR
Neural ranking models still lack a sufficiently developed vision and clear design guidance. This survey reviews them through a unified formulation and multiple dimensions, while reporting that some models can outperform state-of-the-art learning-to-rank models using hand-crafted features.
Problem
Neural ranking models have not had a sufficiently developed vision, and their design principles remain poorly understood with few guidelines.
Method
The survey introduces a unified formulation and reviews existing neural ranking models across model architecture and model learning.
Results
Some neural ranking models learned from scratch can outperform state-of-the-art learning-to-rank models using tens of hand-crafted features.
Takeaways & Limitations
The survey organizes existing neural ranking models under a unified formulation to support understanding of their current status and future development.
Takeaways & Limitations
The literature still lacks a sufficiently developed vision and clear design guidelines for neural ranking models.
Abstract
from arXiv · showhide
Ranking models lie at the heart of research on information retrieval (IR). During the past decades, different techniques have been proposed for constructing ranking models, from traditional heuristic methods, probabilistic methods, to modern machine learning methods. Recently, with the advance of deep learning technology, we have witnessed a growing body of work in applying shallow or deep neural networks to the ranking problem in IR, referred to as neural ranking models in this paper. The power of neural ranking models lies in the ability to learn from the raw text inputs for the ranking problem to avoid many limitations of hand-crafted features. Neural networks have sufficient capacity to model complicated tasks, which is needed to handle the complexity of relevance estimation in ranking. Since there have been a large variety of neural ranking models proposed, we believe it is the right time to summarize the current status, learn from existing methodologies, and gain some insights for future development. In contrast to existing reviews, in this survey, we will take a deep look into the neural ranking models from different dimensions to analyze their underlying assumptions, major design principles, and learning strategies. We compare these models through benchmark tasks to obtain a comprehensive empirical understanding of the existing techniques. We will also discuss what is missing in the current literature and what are the promising and desired future directions.
1. Introduction
Neural ranking models address limitations of hand-crafted features and the difficulty of estimating relevance in complex retrieval tasks. This survey unifies and examines these models across design dimensions, learning strategies, benchmark tasks, and emerging research topics.
- Motivation: Information retrieval ranking remains difficult because existing learning-to-rank models rely on time-consuming, over-specific hand-crafted features and relevance is cognitively complex to estimate.Neural models are motivated by their ability to learn from raw inputs and model complicated tasks.
- Survey scope and approach: The survey focuses on textual retrieval and formulates existing neural ranking models under a unified framework for analysis across assumptions, architectures, and learning strategies.It compares representative models on benchmark tasks and discusses indexing, knowledge integration, visualized learning, contextual learning, and model explanation.
- Research development: Neural ranking research grew from early models such as DSSM and DeepMatch into broader applications spanning ad-hoc retrieval, community-based QA, and conversational search.Early work often targeted short-text ranking, while later research expanded to wider applications and practical effectiveness.
- Current progress: Neural ranking models learned from scratch can already outperform state-of-the-art learning-to-rank models using tens of hand-crafted features.The survey also reports that neural ranking models generate important features for modern search engines.
- Open challenges: Despite progress, neural ranking models have not matched breakthroughs in speech recognition or computer vision, and their design principles remain poorly understood.The authors also report that their special capabilities beyond traditional IR models have not yet been identified.
- Goals and organization: The paper aims to provide empirical understanding of existing techniques while identifying important and promising directions for future neural IR research.Its organization includes model reviews, empirical comparisons, trending topics, and a conclusion.
2. Major Applications of Neural Ranking Models
Neural ranking models are studied across ad-hoc retrieval, question answering, community question answering, and automatic conversation, each posing distinct relevance and vocabulary-mismatch challenges. The survey evaluates these applications using established and newer benchmark datasets while restricting some task coverage because of page limitations.
- The survey covers ad-hoc retrieval, question answering, community question answering, and automatic conversation as major textual IR applications.Product search and sponsored search are mentioned as additional applications but are excluded because of page limitations.
- Ad-hoc Retrieval: Ad-hoc retrieval matches short, potentially unclear queries with longer heterogeneous documents, creating vocabulary mismatch and diverse relevance patterns.Relevance is inherently vague and user dependent, making assessment challenging.
- Ad-hoc Retrieval: Ad-hoc retrieval evaluations use TREC collections, search logs, and the newer large-scale NTCIR WWW Task dataset.The listed resources include Robust, ClueWeb, GOV2, Microblog, AOL, and Bing Search collections or logs.
- Question Answering: QA has clearer relevance and less query-passage heterogeneity than ad-hoc retrieval, but vocabulary mismatch and difficult assessment remain.Ranking models must capture contextual matching, expected answer types, and other patterns in answer passages or sentences.
- Community Question Answering: CQA retrieves similar questions under an assumption that answers to similar questions can answer new questions, while short formulations preserve vocabulary mismatch.The survey focuses on retrieving similar questions rather than directly retrieving answers from the answer pool.
- Automatic Conversation: Automatic conversation is treated as an IR task for selecting social-chat responses from a repository, rather than as response generation.The survey excludes question answering and task completion from this formulation because QA is covered separately and task completion is usually not treated as IR.
3. A Unified Model Formulation
The survey formulates neural ranking as learning a relevance-scoring function over two texts, their separate representations, and their interaction. Unlike traditional learning-to-rank systems with manually defined features, neural models learn these components within network structures from raw texts or embeddings.
- Neural ranking models are primarily studied within a learning-to-rank framework that minimizes loss over labeled query-item examples.The framework assigns relevance labels to candidate documents, answers, or responses and learns a ranking function.
- The unified formulation scores two texts through representation functions ψ and φ, an interaction function η, and an evaluation function g.The representations extract features from each text, interaction extracts pair features, and evaluation computes the relevance score.
- The generalized inputs can be search queries, natural-language questions, or utterances paired with documents, answers, or responses.Labels represent ordered relevance grades from 1 through l.
- Traditional learning-to-rank systems usually use manually defined feature functions, whereas neural ranking models encode and learn representation, interaction, and evaluation functions in the network.Traditional evaluation functions may still be learned with models such as logistic regression or gradient-boosted decision trees.
- Neural ranking inputs may be raw texts or word embeddings, with embedding mapping treated as a basic component of the model.This contrasts with the traditional formulation, whose inputs are usually raw texts combined with hand-crafted features.
4. Model Architecture
Neural ranking architectures are organized by how they process the two inputs: symmetric or asymmetric structures, representation-focused or interaction-focused designs, and single- or multi-granularity approaches. These categories encode assumptions about input homogeneity, relevance representation, and fine-grained interactions.
- Representation-focused vs. Interaction-focused Architectures: Interaction-focused architectures model fine-grained relationships between the two inputs before producing the relevance score.Examples include Arc-II and MatchPyramid, which compute n-gram or word-level interactions and process them with convolutional and pooling layers.
- Symmetric vs. Asymmetric Architectures: Symmetric architectures use homogeneous processing for both inputs, whereas asymmetric architectures model heterogeneous inputs whose order affects the output.Siamese and symmetric interaction networks represent the symmetric family; asymmetric structures are mainly used when queries and documents differ substantially.
- Symmetric vs. Asymmetric Architectures: Asymmetric retrieval models handle query-document heterogeneity through query split, document split, or joint split strategies.Query split targets keyword queries, document split captures locally relevant passages, and joint split combines both assumptions.
- Representation-focused vs. Interaction-focused Architectures: Hybrid architectures combine representation-focused and interaction-focused sub-models to use both approaches in learning relevance features.The survey presents hybridization as a way to obtain the merits of both architectural categories.
- Representation-focused vs. Interaction-focused Architectures: Representation-focused architectures encode each input into high-level representations before applying a simple relevance function.Examples include DSSM, Arc-I, CNTN, CLSM, and LSTM-RNN, using fully connected, convolutional, or recurrent networks for encoding.
5. Model Learning
Neural ranking models use pointwise, pairwise, or listwise objectives, with training strategies also extending to multi-task and weakly supervised settings. The survey contrasts their computational properties, ranking alignment, and practical application boundaries.
- Pointwise Ranking Objective: Pointwise objectives independently predict relevance for each query-document pair, making them simple and scalable.They can use classification or regression losses, including cross entropy for binary or probabilistic labels.
- Pointwise Ranking Objective: Pointwise losses can produce practically meaningful outputs, such as click probabilities in sponsored search.Cross entropy with clickthrough-rate labels can directly predict the probability that a user clicks a search ad.
- Pointwise Ranking Objective: Pointwise objectives ignore document preference and order, so minimizing their loss does not guarantee the best ranking list.This is why they are generally considered less effective for ranking tasks.
- Pairwise Ranking Objective: Pairwise objectives optimize relative preferences between documents rather than their individual relevance labels.Pairwise cross entropy and hinge loss are common examples, with RankNet among the initial neural ranking studies.
- Pairwise Ranking Objective: Pairwise optimization can be effective when evaluation depends on ranking relevant documents, but preference accuracy does not always improve final list metrics.The mismatch arises because preferences may be predicted incorrectly and ranking metrics do not weight all document pairs equally.
- Listwise Ranking Objective: Listwise objectives optimize candidate-document lists directly and can more closely reflect final ranking performance than pairwise objectives.Their computational cost often restricts them to re-ranking small candidate sets, despite their generally greater effectiveness.
- Multi-task and Weakly Supervised Learning: Multi-task learning constructs shared representations or optimization constraints so information from one domain or objective can support another.Examples combine query classification with Web search ranking or jointly learn ranking functions from multiple search domains.
- Multi-task and Weakly Supervised Learning: Weak supervision enabled neural ranking models to achieve up to 35% improvement compared to BM25, used as the weak labeler.The cited result concerns training with weak supervision.
6. Model Comparison
The survey compares published neural-ranking results on ad-hoc retrieval and QA benchmarks under efforts to improve comparability. It finds architecture and dataset-size patterns that distinguish when neural models outperform traditional baselines.
- Evaluation setup: Published results are compared across ad-hoc retrieval and QA benchmarks, with traditional probabilistic, PRF, LTR, and non-neural methods included as baselines.The survey attempts to align experimental settings and favors comparisons reported within or across papers under comparable conditions.
- Ad-hoc retrieval: Traditional PRF and feature-engineered LTR models remain strong baselines that most raw-text neural ranking models struggle to beat.PRF and human-designed LTR features can also be integrated into neural ranking models to improve performance.
- Ad-hoc retrieval: Ad-hoc retrieval shows a shift from symmetric, representation-focused architectures toward asymmetric, interaction-focused designs, which may fit task heterogeneity better.The survey reports that asymmetric, interaction-focused, multi-granularity architectures generally perform better than symmetric, representation-focused, single-granularity architectures, with SNRM on Robust04 as an exception.
- Cross-task observations: Larger datasets with more queries and labels are more likely to yield larger neural-model improvements over non-neural methods.The survey reports that the best raw-text neural models can significantly outperform LTR models with human-designed features on sufficiently large datasets.
- Question answering: QA benchmarks favor symmetric architectures, with representation-focused models more common for short answer sentences and interaction-focused models for longer passages.Unlike ad-hoc retrieval, QA shows no clear winner between representation-focused and interaction-focused architectures.
- Question answering: QA results likewise show larger neural advantages on larger datasets, while performance generally increases over time alongside greater model capacity and mechanisms such as attention.Feature-engineering methods remain strong on small datasets such as TREC QA, whereas a neural/non-neural gap is reported on WikiQA and Yahoo! Answers.
7. Trending Topics
The survey identifies indexing, external knowledge, and visualized retrieval as emerging directions for neural ranking. These directions aim to extend neural models beyond small-set re-ranking while leaving efficiency, scalability, interpretability, and scope questions open.
- Indexing: from Re-ranking to Ranking: SNRM has shown superior performance against competitive baselines while operating as efficiently as term-matching models such as TF-IDF and BM25.
- Indexing: from Re-ranking to Ranking: Learned indexes and approximate-nearest-neighbor methods extend indexing research, but some neural retrieval architectures remain non-scalable to large collections.The survey also notes unresolved questions about interpreting representations, efficiency-effectiveness trade-offs, and compression of learned latent indexes.
- Indexing: from Re-ranking to Ranking: Learning to index seeks neural retrieval models that are both effective and efficient for searching large document collections.SNRM learns high-dimensional sparse query and document representations, optimizing relevance and sparsity so inverted indexes can support retrieval.
- Learning with external knowledge: External-knowledge approaches enrich neural ranking with structured sources such as knowledge bases and unstructured sources such as retrieved QA pairs, topics, or tags.Examples combine word- and entity-based representations, knowledge gates, pseudo-relevance feedback, and QA correspondence distillation.
- Learning with external knowledge: The survey identifies improving effectiveness with distilled external knowledge and understanding its role in ranking as continuing research needs.
- Learning with visualized technology: Visualized ranking represents matching between two inputs as an image, preserving layout information for neural relevance estimation from visual features.Reported studies show that visual features can improve learning-to-rank performance, but their added value beyond text-based methods remains under study.
8. Conclusion
The survey unifies neural ranking models across architecture and learning dimensions, compares them empirically on major retrieval tasks, and reviews promising research topics. It concludes that neural IR offers substantial opportunities while retaining important open questions.
- Contributions: It surveys published empirical results on ad-hoc retrieval and QA tasks to provide a comprehensive comparison of neural ranking methods.
- Contributions: The survey introduces a unified formulation for neural ranking models and reviews existing models through architecture and learning dimensions.The architecture analysis examines inputs, relevance features, and evaluation, while the learning analysis covers objectives and training strategies.
- Future directions: The paper discusses trending topics that are important or promising for future neural-ranking research.
- Conclusion: Neural ranking models form part of neural IR, a joint area of deep learning and information retrieval with opportunities for new research and applications.
- Conclusion: The authors expect significant breakthroughs in neural IR through continued community effort.