A Deep Look into Neural Ranking Models for Information Retrieval

• Neural ranking models can learn ranking features from raw text and model complex relevance judgments, addressing limitations of hand-crafted LTR features and shallow models.Existing hand-crafted features are time-consuming and over-specific, while relevance is difficult to estimate because judgments involve complicated human cognition.Source: §1 · p. 2
• The survey focuses on textual retrieval and takes a deep look at ranking models with deep neural networks rather than covering every aspect of neural IR.It formulates existing models under a unified framework and examines their assumptions, design principles, and learning strategies.Source: §1 · pp. 3–4
• Neural ranking research grew through models including DSSM, DeepMatch, ARC I and ARC II, and MatchPyramid, with much research focused on short-text ranking tasks.Source: §1 · p. 3
• Neural models learned from scratch can outperform state-of-the-art LTR models using tens of hand-crafted features, and already generate important features for modern search engines.DSSM has also been deployed widely in enterprise applications.Source: §1 · p. 4
• Despite this progress, neural ranking models lack comparable breakthroughs, clear design guidelines, and well-understood capabilities beyond traditional IR models.These limitations motivate summarizing the field’s current status and gaining insights for future development.Source: §1 · p. 4
• The paper compares representative models on benchmark tasks and discusses trending topics including indexing, knowledge integration, visualized learning, contextual learning, and model explanation.These discussions are intended to support better future neural ranking models, while some topics remain under-addressed and others are promising research directions.Source: §1 · pp. 4–5

2. Major Applications of Neural Ranking Models

• The survey covers ad-hoc retrieval, question answering, community question answering, and automatic conversation, while excluding other applications because of page limitations.Examples of excluded or unaddressed applications include product search [12] and sponsored search.Source: §2 · p. 5

2.1. Ad-hoc Retrieval

• Ad-hoc retrieval returns documents as a ranking list for continually submitted queries against a relatively static collection; queries are usually short, and query-document heterogeneity leads to vocabulary mismatch [41].Source: §2.1 · pp. 5–6

2.2. Question Answering

• The survey focuses on QA as answer passage/sentence retrieval, formulated as typical IR, with reduced heterogeneity compared with ad-hoc retrieval.QA sources can vary from structured data to unstructured sources.Source: §2.2 · pp. 6–7

2.3. Community Question Answering

• Community question answering retrieves similar questions from a question pool based on the assumption that answers to similar questions can answer users’ questions; direct answer retrieval is not discussed.Direct answer retrieval draws answers from an answer pool, resembles the QA task, and may use behavioral data such as upvotes/downvotes.Source: §2.3 · p. 8

2.4. Automatic Conversation

• Automatic conversation can rank repository responses or generate them, but this survey restricts it to IR-based social chat, including single-turn and multi-turn settings [71–75].In social chat, input utterances and responses are both short natural-language texts, producing homogeneity similar to community question answering.Source: §2.4 · p. 9
• Automatic conversation relevance emphasizes semantic correspondence or coherent structure, allowing responses from general acknowledgments to specific replies.For example, an utterance about developing myopia may receive either “Really?” or a specific comment about glasses.Source: §2.4 · p. 9
• Vocabulary mismatch is not the central challenge in automatic conversation; models must instead capture coherence and avoid general trivial responses.Source: §2.4 · p. 9
• Neural ranking models for automatic conversation have been evaluated on Ubuntu Dialog Corpus, Sina Weibo, MSDialog, and NTCIR STC campaign datasets [74, 75, 76, 77, 78, 79, 80, 81, 82, 83].The collections were gathered from social media sources including forums, Twitter, and Weibo.Source: §2.4 · p. 9

Unified formulation

• The generalized learning-to-rank task seeks an optimal ranking function by minimizing loss over labeled query–candidate pairs with ordered relevance grades.Queries may be search queries, questions, or utterances; candidates may be documents, answers, or responses.Source: §3 · p. 10
• The unified ranking function represents each input with ψ and φ, models pairwise interaction with η, and computes relevance through evaluation function g.The inputs are two texts, s and t, and g produces their relevance score from the extracted feature representations.Source: §3 · p. 10
• Traditional LTR uses manually defined ψ, φ, and η functions, while g can be learned with models such as logistic regression or gradient boosting decision trees.In neural ranking models, ψ, φ, η, and g are generally encoded in network structures and learned from data.Source: §3 · pp. 10–11
• Neural ranking models can take either raw texts or word embeddings as inputs, treating embedding mapping as a basic component.Traditional LTR approaches generally use raw texts as inputs.Source: §3 · p. 11

4.1. Symmetric vs. Asymmetric Architectures

• Neural ranking architectures divide into symmetric and asymmetric forms according to assumptions about the input texts s and t.Symmetric structures allow exchanging s and t without changing the output; asymmetric structures produce different outputs when their positions are exchanged.Source: §4.1 · pp. 11–12
• Symmetric architectures comprise siamese networks and symmetric interaction networks, with representative siamese models including DSSM and CLSM.Source: §4.1 · pp. 11–12
• Asymmetric architectures mainly serve ad-hoc retrieval because queries and documents are heterogeneous, and they use query split, document split, or joint split strategies.Query split matches query terms to documents, document split captures passage-level signals under the scope hypothesis [2], and joint split combines both assumptions.Source: §4.1 · pp. 12–13
• QA-oriented asymmetric models also use one-way attention, where question representations weight candidate answer words to enhance answer representations, as in IARNN and CompAgg.The strategy is illustrated as attention flowing from the question representation to the candidate answer sequence.Source: §4.1 · p. 14

4.2. Representation-focused vs. Interaction-focused Architectures

• Models are also categorized as representation-focused or interaction-focused according to assumptions about features extracted by representation and interaction functions.Some models adopt hybrid architectures to combine the merits of both categories.Source: §4.2 · pp. 14–15
• Representation-focused architectures assume relevance depends on compositional meaning, producing high-level input representations with complex η and scoring them using simple functions such as cosine or MLP.The supplied passage introduces this category but does not yet provide the corresponding interaction-focused details.Source: §4.2 · p. 15
• Representation-focused architectures evaluate relevance from high-level representations, fitting global-matching and short-text tasks such as CQA and AC while enabling offline representation precomputation.These models are efficient for online computation because representations can be calculated once after φ and ψ are learned.Source: §4.2 · p. 16
• Interaction-focused architectures directly learn relations between inputs, fitting general IR, exact or diverse matching requirements, and heterogeneous inputs such as ad-hoc retrieval and QA.They use non-parametric or parametric interaction functions, with parametric functions offering flexibility when sufficient training data offsets their larger cost.Source: §4.2 · p. 16
• Interaction-focused models are less efficient online because their interaction function cannot be precomputed before the input pair is observed; a telescope pipeline applies them later.Representation-focused models can serve early retrieval stages, while interaction-focused models rerank later candidates.Source: §4.2 · p. 17
• Hybrid architectures combine representation-focused and interaction-focused modeling through loose combined strategies, such as DUET, or compact coupled strategies, such as IARNN and CompAgg.Combined strategies merge submodel outputs, whereas coupled strategies use attention across inputs to learn representations compactly.Source: §4.2 · p. 18

4.3. Single-granularity vs. Multi-granularity Architecture

• Single-granularity architectures assume relevance can be estimated from final high-level features produced from single-form text inputs, treating φ, ψ, and η as black boxes.The inputs are represented only as word sets, sequences, or embeddings without additional language structures.Source: §4.3 · pp. 18–19
• Existing neural ranking models are divided into single-granularity and multi-granularity categories, with many established models—including DSSM, MatchPyramid, and DRMM—using the single-granularity assumption.The cited examples include both symmetric and asymmetric architectures.Source: §4.3 · p. 19
• Multi-granularity architectures combine features from different abstraction levels or language units because relevance estimation requires multiple granularities.They extend single-granularity architectures by incorporating network structures and inherent language structures.Source: §4.3 · pp. 19–20
• Vertical multi-granularity exploits the hierarchical nature of deep networks, enabling evaluation function g to leverage different-level abstraction of features for relevance estimation.For example, MACM builds a CNN over the interaction matrix from η and uses MLP.Source: §4.3 · p. 20
• Horizontal multi-granularity models assume language has intrinsic structures and improve relevance estimation by extending inputs from words to phrases/n-grams or sentences, rather than using simple words alone.Source: §4.3 · p. 20
• Multi-granularity models are expected to better fit fine-grained matching tasks such as ad-hoc retrieval and QA, but often require larger model complexity.Their enhanced capability comes from extracting and aggregating multi-granularity features.Source: §4.3 · p. 21

5.1. Learning objective

• Neural ranking objectives are broadly categorized as pointwise, pairwise, or listwise, with each group offering different loss functions and trade-offs across information-retrieval tasks.The survey introduces representative objectives in each category and discusses their advantages and disadvantages for neural ranking applications.Source: §5.1 · p. 21

5.1.1. Pointwise Ranking Objective

• Pointwise objectives independently predict each query-document pair’s relevance label, making training simple and scalable but ignoring document preferences and ranking order.Cross entropy can model probabilistic labels such as clickthrough rate, while pointwise losses are generally less effective for producing optimal ranking lists.Source: §5.1.1 · pp. 21–22

5.1.2. Pairwise Ranking Objective

• Pairwise ranking objectives optimize relative document preferences, with hinge loss used in neural ranking models such as DRMM and K-NRM, and pairwise cross entropy introduced in RankNet.Source: §5.1.2 · pp. 22–23

5.1.3. Listwise Ranking Objective

• Listwise objectives evaluate each query’s candidate document list jointly to reflect final ranking performance, but their computational cost often restricts them to small re-ranking sets.ListMLE models the likelihood of the optimal ranked list, while softmax-based losses are popular for neural ranking and useful with click data under unbiased learning.Source: §5.1.3 · pp. 23–25

5.1.4. Multi-task Learning Objective

• Multi-task learning jointly optimizes ranking and non-ranking objectives so information from one domain can support understanding in another.For example, Liu et al. jointly trained classification and Web-search losses through shared representations.Source: §5.1.4 · p. 25
• Multi-task learning commonly constructs shared representations effective for ranking across multiple tasks or domains and regularizes optimization so the final model is not designed specifically for one ranking objective.Cohen et al. introduced an adversarial learning framework.Source: §5.1.4 · p. 25

5.2. Training Strategies

• Supervised learning trains on labeled query-document pairs from experts, crowdsourcing, or user interactions, but limited annotations constrain neural models to smaller parameter spaces.This strategy assumes sufficient labeled training data and supports objectives such as pointwise and pairwise learning.Source: §5.2 · p. 26
• Up to 35% improvement over BM25 was observed when neural ranking models were trained with weak supervision using BM25 as the weak labeler.Weak supervision automatically generates query-document pseudo-labels and does not require labeled training data.Source: §5.2 · p. 26
• Semi-supervised learning leverages a small set of labeled query-document pairs and a large set of unlabeled data, including neural models with joint supervised and unsupervised loss functions.Examples include preference regularization, feature extraction using KernelPCA, and pseudo-label generation using labeled data.Source: §5.2 · p. 27

6. Model Comparison

• Published results are difficult to compare across papers because small preprocessing changes, such as tokenization and stemming, can produce significant differences.The survey therefore emphasizes results compared under shared settings or collected from studies evaluating models at a single site for fairness.Source: §6 · p. 27

6.1. Empirical Comparison on Ad-hoc Retrieval

• For ad-hoc retrieval, the published experimental results use benchmark datasets, including three representative choices: Robust04, a standard dataset with queries from TREC Robust Track 2004; Gov2MQ2007; and Sougou-Log.Source: §6.1 · pp. 27–28
• On ad-hoc retrieval, traditional probabilistic, PRF, and feature-based LTR models are strong baselines, but PRF and human-designed features can enhance neural models.Asymmetric, interaction-focused, multi-granularity architectures generally outperform symmetric, representation-focused, single-granularity designs; neural gains over non-neural models are larger with more queries and labels, with SNRM on Robust04 as a qualified exception.Source: §6.1 · p. 28

6.2. Empirical Comparison on QA

• On QA, symmetric architectures are widely adopted, representation-focused models are common for short answer retrieval, and interaction-focused models are common for longer passages, without a clear overall winner.Neural models show larger gains on bigger datasets, while feature-engineered methods remain strong on small TREC QA; performance generally increases over time with greater capacity and techniques such as attention.Source: §6.2 · p. 31
• Attention channels use matrices to encapsulate rich structural patterns.Source: §6.2 · p. 33
• aNMM uses attention to encode the importance of question terms.Source: §6.2 · p. 33
• The model aggregates interaction matching features.Source: §6.2 · p. 33

7.1. Indexing: from Re-ranking to Ranking

• SNRM learns high-dimensional sparse query and document representations, optimizing relevance and sparsity, and achieves superior performance while matching TF-IDF and BM25 efficiency.Multi-stage cascades propagate early retrieval errors, motivating standalone neural retrieval; SNRM uses sparse representations compatible with inverted indexes and minimizes L0 alongside relevance loss.Source: §7.1 · pp. 33–34
• Learning to index is promising for scalable neural retrieval, but representation interpretation, efficiency-effectiveness trade-offs, and learned-index compression remain open questions.Existing cascaded systems are inefficiently constrained to re-ranking small candidate sets, while some approximate-nearest-neighbor approaches are not scalable to large collections.Source: §7.1 · p. 34

7.2. Learning with External Knowledge

• External knowledge research augments text matching with structured knowledge bases, entities, concepts, topics, tags, retrieved results, and QA pairs to improve neural ranking.Models use entity-aware interactions, symbolic and distributional semantics, knowledge graphs, pseudo-relevance feedback, correspondence distillation, and knowledge gates; the role of distilled knowledge still needs study.Source: §7.2 · pp. 35–36

7.3. Learning with Visualized Technology

• Visualized IR constructs input matching as an image, enabling deep models to estimate relevance from visual features while preserving original layout information.This approach is presented as an alternative to textual IR and may expose useful features unavailable in traditional matching matrices.Source: §7.3 · p. 36
• Visual features learned from Web-page snapshots, screenshots, titles, snippets, and HTML can improve document or search-result relevance estimation and learning-to-rank performance.ViP combines CNN and LSTM representations of query-independent and query-dependent snapshots, while joint models learn visual patterns, textual semantics, and presentation structures; ViTOR results show significant LTR improvements from visual features.Source: §7.3 · p. 37
• Visualized technology is a novel direction that simulates human visual relevance judgment, but existing evidence covers only some tasks and leaves its broader benefits and applications unclear.Web pages provide rich layouts in which spatial proximity, font size, and colors may support relevance estimation, especially for ad-hoc retrieval; further research is needed beyond text-based methods.Source: §7.3 · p. 37

7.4. Learning with Context

• Because short queries underspecify information needs, neural ranking models can exploit short- and long-term history, situational context, and relevance feedback, although this integration remains relatively understudied.A deep-and-wide architecture preserves raw contextual features alongside abstract representations, and multi-task training incorporates short-term history into ranking and query suggestion.Source: §7.4 · p. 37
• Pseudo-relevance feedback extends neural ranking with learned representations: RFMF factors words, queries, and documents, while NPRF often yields significant improvements when added to models such as DRMM and KNRM.RFMF learns latent factors from the top retrieved document using non-negative matrix factorization; extending SNRM with pseudo-relevance feedback led to a similar conclusion.Source: §7.4 · p. 38

7.5. Neural Ranking Model Understanding

• Understanding what neural ranking models learn and what models suit each IR task remains a challenging and promising research direction.Existing work often depends on detailed knowledge of specific architectures, while broader questions concern model suitability, newly learned capabilities, and explanations for strong results.Source: §7.5 · pp. 39–40
• Neural-ranking explainability is largely unexplored: analyses find topical relevance in high-level layers, detailed text information in low-level layers, and task-dependent preferences for abstraction versus exact matching.Fine-grained tasks such as factoid QA generally require greater abstraction and semantic matching, whereas coarse-grained document retrieval often relies on exact query-document interaction.Source: §7.5 · pp. 39–40

Survey scope

• The survey introduced a unified formulation of neural ranking models and reviewed existing approaches across model architecture and model learning.Its purpose was to summarize research status, analyze methodologies, and gain insights for future development.Source: §8 · p. 40

Model architecture

• The architecture analysis examined input treatment, relevance features, evaluation, underlying assumptions, and major design principles.Source: §8 · p. 40

Learning and applications

• The survey reviewed learning objectives and training strategies, then compared published neural ranking results on ad-hoc retrieval and question answering.The empirical comparison was intended to clarify the current status of neural ranking models on major applications.Source: §8 · p. 40

Future directions

• The survey discussed important and promising trends, positioning neural ranking within neural IR and encouraging new research and applications.The authors expect community efforts to produce significant breakthroughs in the domain, as occurred in computer vision or NLP.Source: §8 · p. 40