PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Relevance Matching Model for Ad-hoc Retrieval

Jiafeng Guo, Yixing Fan, Qingyao Ai, W. Bruce Croft

arXiv:1711.08611v1cs.IR

TL;DR

Ad-hoc retrieval has produced few positive results with deep models partly because they do not adequately model relevance matching, which differs from semantic matching in NLP. The paper proposes DRMM, a query-term-level interaction architecture with histogram mapping, feed-forward matching, and term gating. On two benchmark datasets, DRMM significantly outperforms traditional retrieval models and state-of-the-art deep matching models.

  • Problem

    Deep models have shown few positive results on ad-hoc retrieval because important relevance-matching characteristics are not adequately addressed, unlike semantic matching in many NLP tasks.

  • Method

    DRMM uses query-term-level local interactions, matching histograms, a feed-forward matching network, and a term gating network to model relevance matching.

  • Results

    DRMM significantly outperforms traditional retrieval models and state-of-the-art deep matching models on two representative benchmark datasets.

  • Takeaways & Limitations

    Relevance matching requires model designs that explicitly address exact matching signals, query-term importance, and diverse matching requirements.

  • Takeaways & Limitations

    The paper defines “term” as an indexed search unit, such as a stemmed word or phrase, and assumes term vectors are learned beforehand from existing embedding models.

Abstract

from arXiv · show

In recent years, deep neural networks have led to exciting breakthroughs in speech recognition, computer vision, and natural language processing (NLP) tasks. However, there have been few positive results of deep models on ad-hoc retrieval tasks. This is partially due to the fact that many important characteristics of the ad-hoc retrieval task have not been well addressed in deep models yet. Typically, the ad-hoc retrieval task is formalized as a matching problem between two pieces of text in existing work using deep models, and treated equivalent to many NLP tasks such as paraphrase identification, question answering and automatic conversation. However, we argue that the ad-hoc retrieval task is mainly about relevance matching while most NLP matching tasks concern semantic matching, and there are some fundamental differences between these two matching tasks. Successful relevance matching requires proper handling of the exact matching signals, query term importance, and diverse matching requirements. In this paper, we propose a novel deep relevance matching model (DRMM) for ad-hoc retrieval. Specifically, our model employs a joint deep architecture at the query term level for relevance matching. By using matching histogram mapping, a feed forward matching network, and a term gating network, we can effectively deal with the three relevance matching factors mentioned above. Experimental results on two representative benchmark collections show that our model can significantly outperform some well-known retrieval models as well as state-of-the-art deep matching models.

1. INTRODUCTION

Deep models have produced few positive results for ad-hoc retrieval partly because they overlook relevance matching's distinctive requirements. The paper distinguishes semantic from relevance matching, proposes DRMM to address those requirements, and reports significant benchmark improvements over traditional and deep baselines.

  • Motivation: Deep models have achieved few positive results on ad-hoc retrieval despite successes in other machine-learning applications.The paper attributes this partly to important ad-hoc retrieval characteristics not being well addressed by deep models.
  • Motivation: Ad-hoc retrieval is often formalized as text matching alongside NLP tasks, although the paper argues that it primarily requires relevance matching rather than semantic matching.The distinction motivates different deep-model architectures.
  • Contributions: DRMM is a novel ad-hoc retrieval model that explicitly addresses the three key factors of relevance matching.Its design is introduced as an interaction-focused model operating at the query-term level.
  • Results: On two representative benchmark collections, DRMM significantly outperforms traditional retrieval models and state-of-the-art deep matching baselines across all evaluation metrics.The comparisons include well-known traditional models and deep models designed for general matching or ad-hoc retrieval.
  • Contributions: The paper identifies three differences between semantic and relevance matching that may require substantially different deep-model architectures.These differences are treated as a central contribution.

2. AD-HOC RETRIEVAL AS A MATCHING PROBLEM

Ad-hoc retrieval is commonly represented as scoring the match between a query and document, a formulation shared with several NLP matching tasks. Existing deep matching models divide into representation-focused and interaction-focused architectures, but have shown few positive ad-hoc retrieval results.

  • Matching formulation: Ad-hoc retrieval can be formalized as a matching problem that scores two texts from their representations.The two texts are typically a query and a document.
  • Matching formulation: The same general text-matching formulation is used for tasks including paraphrase identification, question answering, and automatic conversation.This shared formulation motivates comparing model architectures across tasks.
  • Model architectures: Representation-focused models build separate abstract representations before applying a relatively simple matching function.Examples include DSSM, C-DSSM, and ARC-I.
  • Model architectures: Interaction-focused models construct local interactions between texts and apply deep networks to learn hierarchical interaction patterns.Examples include DeepMatch, ARC-II, and MatchPyramid.
  • Evidence gap: Most existing deep matching models have been demonstrated mainly on NLP tasks, while ad-hoc retrieval has yielded few positive results.Even Web-search models such as DSSM and C-DSSM were evaluated on query–document-title pairs rather than typical full ad-hoc retrieval settings.

3. SEMANTIC MATCHING VS. RELEVANCE MATCHING

Semantic matching and relevance matching differ in their signals, term weighting, and scope of matching. Ad-hoc retrieval uses short keyword queries against documents of highly variable length, making exact matches, query-term importance, and diverse matching requirements central.

  • Task differences: Semantic matching typically identifies meaning and relations between homogeneous, sentence-like texts, whereas ad-hoc retrieval identifies document relevance to a short keyword query.Ad-hoc documents may range from tens of words to tens of thousands.
  • Semantic matching: Semantic matching emphasizes similarity signals and compositional meaning, often evaluating texts globally as wholes.These properties fit paraphrase identification, question answering, and related NLP tasks.
  • Relevance matching: Exact matching remains the most important ad-hoc retrieval signal because modern search engines rely on indexing and search paradigms.Semantic similarity signals address term mismatch but do not replace exact query-term matches.
  • Relevance matching: Ad-hoc retrieval must account for query-term importance because keyword queries can contain terms with different discriminative value.For example, in “bitcoin news,” “bitcoin” may be more important than “news.”
  • Relevance matching: Variable document length creates a diverse matching requirement because relevance may be global or scope-specific under different document-length hypotheses.The paper contrasts the Verbosity Hypothesis with the Scope Hypothesis.
  • Implications for models: Existing representation-focused models can lose detailed and exact matching signals, while interaction-focused models may fail to distinguish signal importance or handle diverse matching requirements.The paper therefore argues that a single architecture may not fit semantic and relevance matching equally well.

4. DEEP RELEVANCE MATCHING MODEL

DRMM is an interaction-focused model designed for ad-hoc relevance matching rather than semantic matching. It uses strength-focused histograms, a feed forward network, and query-term gating to model exact matches, hierarchical matching patterns, term importance, and diverse matching requirements.

  • Model Training: DRMM uses a joint query-term-level architecture and trains on query, relevant-document, and non-relevant-document triples with pairwise hinge loss.The loss encourages the relevant document’s predicted matching score to exceed the non-relevant document’s score by a margin.
  • Matching Histogram Mapping: DRMM builds local query-document term interactions and converts each query term’s variable-length interactions into a fixed-length matching histogram.The histogram groups cosine-similarity interactions by signal strength rather than position.
  • Matching Histogram Mapping: Matching histograms separate exact-match signals from similarity signals and avoid positional noise that may be irrelevant to ad-hoc retrieval.Exact matches receive a separate bin, while interactions are aggregated across positions.
  • Feed forward Matching Network: The feed forward matching network learns hierarchical matching patterns from interaction strengths and produces a relevance score for each query term.Unlike position-aware convolutional models, DRMM focuses on signal strengths rather than positional regularities.
  • Term Gating Network: A term gating network assigns an aggregation weight to each query term, explicitly modeling query-term importance in the final relevance score.The model considers term vectors and inverse document frequency as gating inputs, with softmax used as the gating function.

5. EXPERIMENTS

The experiments compare DRMM and established retrieval and deep matching models on Robust04 and ClueWeb-09-Cat-B, finding that DRMM achieves the strongest overall performance. Analyses show that relevance matching benefits from exact signals, query-term weighting, strength-based representations, and length-aware modeling.

  • Experimental Setup: Experiments use Robust04 and ClueWeb-09-Cat-B, comparing traditional retrieval, representation-focused, interaction-focused, and DRMM models.Five-fold cross-validation optimizes MAP for all retrieval models.
  • Retrieval Performance: Representation-focused and interaction-focused deep matching baselines cannot compete with traditional retrieval models on the benchmark collections.Representation-focused models perform significantly worse than traditional retrieval models, while interaction-focused baselines also fail to compete.
  • Retrieval Performance: Exact matching signals are important for relevance matching: MPIND and MPCOS substantially outperform MPDOT, whose exact-signal weighting is weaker.MPCOS consistently outperforms representation-focused models, while MPIND and MPCOS perform much better than MPDOT.
  • DRMM Analysis: LCH-based DRMM variants outperform NH-based variants, and IDF-based term gating works better than term-vector-based gating on both collections.The authors relate NH performance to lost document-length information and explain the weaker term-vector gating through insufficient term-importance information and a larger gating network.
  • Retrieval Performance: The best DRMM, DRMM_LCH×IDF, significantly outperforms all baselines; on ClueWeb-09-Cat-B titles, gains over BM25 are 11.9% MAP, 14.7% nDCG@20, and 12% P@20.On Robust04, its topic-description performance is comparable to its topic-title performance, unlike previous models.
  • Component Analysis: Removing term gating drops MAP by 6.8% on Robust04 and 3.5% on ClueWeb-09-Cat-B, while position-based dynamic pooling underperforms strength-based alternatives.K-max pooling works well on Robust04 but fails on ClueWeb-09-Cat-B, where larger document-length variation may bias it toward very long documents.

6. RELATED WORK

Related work applies deep matching to text pairs through representation-focused and interaction-focused architectures. The paper distinguishes these semantic-matching models from the relevance-matching problem of ad-hoc retrieval and introduces a model designed specifically for that setting.

  • Deep Matching Models: Existing deep matching models for text matching fall into representation-focused and interaction-focused architectural categories.The related work discusses models such as CNTN, ARC-I, DSSM, DeepMatchtree, Match-SRNN, and MatchPyramid.
  • Relevance Matching: Most reviewed deep matching models target semantic matching, which differs significantly from relevance matching in ad-hoc retrieval.The paper therefore introduces a model specifically designed for relevance matching.

7. CONCLUSIONS

The paper distinguishes relevance matching in ad-hoc retrieval from semantic matching in NLP and proposes DRMM to address relevance-specific factors. Experiments show strong benchmark performance, while future work calls for larger training data and deeper models.

  • DRMM explicitly addresses three relevance-matching factors through matching histogram mapping, a feed forward matching network, and a term gating network.
  • The model is motivated by significant differences between semantic matching in NLP tasks and relevance matching in ad-hoc retrieval.
  • Experiments on two representative benchmark datasets show that DRMM significantly outperforms traditional retrieval models and state-of-the-art deep matching models.
  • Future work proposes using larger training data, such as click-through logs, to train deeper DRMM and further explore its potential for ad-hoc retrieval.
  • Future work also considers phrase embeddings so local interactions can use semantic units larger than separate terms.
Loading 1711.08611v1…