PaperlayerView exact revision on arXiv

Source-linked AI summary

Relevance-based Word Embedding

Hamed Zamani, W. Bruce Croft

arXiv:1705.03556v2cs.IRcs.CLcs.LGcs.NE

TL;DR

Typical word embeddings optimize term proximity, which does not necessarily capture the relevance objective central to many IR tasks. The paper therefore learns relevance-based embeddings with two different objectives from retrieved documents and finds that they outperform proximity-based baselines across query expansion and query classification evaluations.

  • Problem

    Proximity-based objectives in word2vec and GloVe do not necessarily match information retrieval tasks' primary goal of modeling relevance.

  • Method

    The paper trains RLM to estimate a query-specific relevance distribution and RPE to classify terms as relevant or non-relevant, using top-ranked documents from millions of training queries.

  • Results

    Relevance-based embeddings outperform word2vec, GloVe, and other baselines in query expansion across four TREC collections and query classification on KDD Cup 2005.

  • Takeaways & Limitations

    Learning embeddings from query-document relevance information is effective for the evaluated information retrieval tasks.

Abstract

from arXiv · show

Learning a high-dimensional dense representation for vocabulary terms, also known as a word embedding, has recently attracted much attention in natural language processing and information retrieval tasks. The embedding vectors are typically learned based on term proximity in a large corpus. This means that the objective in well-known word embedding algorithms, e.g., word2vec, is to accurately predict adjacent word(s) for a given word or context. However, this objective is not necessarily equivalent to the goal of many information retrieval (IR) tasks. The primary objective in various IR tasks is to capture relevance instead of term proximity, syntactic, or even semantic similarity. This is the motivation for developing unsupervised relevance-based word embedding models that learn word representations based on query-document relevance information. In this paper, we propose two learning models with different objective functions; one learns a relevance distribution over the vocabulary set for each query, and the other classifies each term as belonging to the relevant or non-relevant class for each query. To train our models, we used over six million unique queries and the top ranked documents retrieved in response to each query, which are assumed to be relevant to the query. We extrinsically evaluate our learned word representation models using two IR tasks: query expansion and query classification. Both query expansion experiments on four TREC collections and query classification experiments on the KDD Cup 2005 dataset suggest that the relevance-based word embedding models significantly outperform state-of-the-art proximity-based embedding models, such as word2vec and GloVe.

W. Bruce Crof Center for Intelligent Information Retrieval College of Information and Computer Sciences

The passage identifies the University of Massachusetts Amherst.

  • The affiliation named is University of Massachusets Amherst.

1 INTRODUCTION

The paper addresses a mismatch between proximity-based word-embedding objectives and the relevance-centered goals of information retrieval. It proposes offline relevance-based embeddings and evaluates them across query expansion and query classification.

  • Word2vec and related embeddings predict adjacent words from context, targeting semantic or syntactic similarity rather than necessarily matching IR relevance objectives.
  • The paper reframes embedding learning around predicting words observed in documents relevant to a particular information need.
  • Offline training on top-ranked documents for millions of queries enables relevance-distribution prediction without retrieval runs at test time.
  • RLM models a relevance distribution over vocabulary terms, whereas RPE classifies each term as relevant or non-relevant for a query.
  • Across query expansion on four TREC collections and query classification on KDD Cup 2005, relevance-based embeddings outperform the baselines.

2 RELATED WORK

Prior work established dense word representations and their uses across NLP and IR, while also exposing limitations of proximity-based and query-time embedding approaches. This paper proposes two embedding models designed specifically for IR needs.

  • Word embeddings represent vocabulary terms as high-dimensional real-valued dense vectors, with word2vec and GloVe learning from contextual co-occurrence.
  • Embedding representations have supported applications including document distance, question retrieval, and other NLP and IR tasks.
  • Local embedding trains vectors on top-retrieved documents at query time, making it impractical in some applications and retaining a proximity-based objective.
  • The paper proposes two word-embedding models specifically designed for information retrieval needs.

3 RELEVANCE-BASED EMBEDDING

The paper replaces proximity-based word-embedding objectives with relevance-based learning from query–document information, using models that estimate relevance distributions or classify terms. The architecture maps queries into embeddings and learns word weights from pseudo-relevance feedback.

  • Motivation: Typical word embeddings capture term proximity, but this objective can conflict with IR relevance needs, such as avoiding “safe” when expanding “dangerous vehicles.”The proposed approach is motivated by matching learned representations to retrieval intent rather than contextual similarity alone.
  • 3.1 Neural Network Architecture: The model uses a feed-forward network with a single linear hidden layer to map sparse query vectors into query embeddings and estimate vocabulary-term weights.Query vectors are formed by averaging one-hot representations of their terms; WQ maps queries to embeddings, while Ww and bw support term-probability estimation.
  • 3.2 Modeling Relevance for Training: Training data pairs each query with a pseudo-relevance distribution estimated from its top 10 retrieved documents.The relevance model assigns zero probability to terms absent from those retrieved documents.
  • 3.3 Relevance Likelihood Maximization Model: Relevance likelihood maximization learns parameters that maximize the likelihood of generating relevance-model probabilities for the training queries.The model represents the predicted relevance distribution with softmax and uses hierarchical softmax to reduce probability-computation complexity from O(|V|) to O(log(|V|)).
  • 3.4 Relevance Posterior Estimation Model: Relevance posterior estimation instead casts relevance learning as binary classification of terms for each query.It uses a sigmoid classifier with negative sampling controlled by positive and negative-instance hyperparameters.
  • 3.4 Relevance Posterior Estimation Model: The posterior-estimation model does not directly provide the probability of each term given a query because its classification probabilities lack all components needed for Bayes-rule calculation.It can still support similarity between terms or queries, while query–term similarity requires an additional query-model estimation approach.

4 EXPERIMENTS

The experiments evaluate learned embeddings extrinsically on query expansion and query classification, comparing the proposed models with existing word-embedding algorithms. The stated comparison target is embedding methods rather than state-of-the-art task-specific systems.

  • Evaluation Design: The evaluation covers two IR tasks: query expansion and query classification.The experiments are designed to compare relevance-based embeddings with existing word-embedding algorithms.

4.1 Training

Training uses AOL query logs and retrieved documents to construct data for relevance-based embedding models. The retrieval setup supplies top-ranked documents for estimating relevance information.

  • Training Data: Millions of unique AOL queries from a three-month 2006 sample provide the training queries.The logs contain web-search queries submitted by real users to AOL.
  • Training Data: The top 10 documents retrieved for each query are used to estimate the training relevance distributions.The retrieval model is query likelihood with Dirichlet prior smoothing using µ = 1500.
  • Optimization: The models are implemented in TensorFlow and trained with stochastic gradient descent and back-propagation.Learning rate and batch size are tuned from the specified candidate sets on the training data.

4.2 Evaluation via Qery Expansion

The query-expansion evaluation uses four TREC collections and shows that relevance-based embeddings outperform proximity-based baselines, with RLM offering strong effectiveness and practical efficiency. Sensitivity analyses indicate that performance depends on expansion size, embedding dimensionality, and training-data volume.

  • Experimental Setup: The evaluation covers AP, Robust, GOV2, and ClueWeb, spanning homogeneous newswire and heterogeneous large-scale web collections.ClueWeb was spam-filtered; stopwords were removed and no stemming was performed.
  • Experimental Setup: Query expansion estimates a language model from word embeddings, then retrieves documents using KL-divergence with Dirichlet smoothing.The interpolation coefficient controls the weight of the original query model, and missing embedding terms are ignored.
  • Results and Discussion: Both relevance-based embedding models outperform all baselines across collections, while RLM consistently outperforms RPE.The comparison includes MLE, word2vec, and GloVe; improvements are often statistically significant.
  • Results and Discussion: RLM selects relevance-oriented expansion terms, whereas word2vec tends to add terms that are syntactically or semantically related to individual query terms.The two relevance-based models can select similar terms but assign them different weights because RPE classifies terms rather than learning a relevance distribution.
  • Results and Discussion: RLM performs close to RM3 on Robust and slightly better on GOV2 without pseudo-relevant documents, while requiring only one retrieval run.RM3 requires two retrieval runs and top retrieved documents, creating an efficiency distinction for real-world applications.
  • Results and Discussion: Performance is best with 10 expansion words in newswire collections, 15 in GOV2, stabilizes above 300 dimensions, and becomes more stable after four million training queries.Increasing training queries from one million to four million significantly improves performance.

4.3 Evaluation via Qery Classification

The paper evaluates relevance-based embeddings for supervised multi-label query classification on the KDD Cup 2005 dataset, comparing them with word2vec and GloVe. The models outperform these baselines, while performance depends on embedding dimensionality, training-data volume, and whether ranking or term weighting matters.

  • Dataset and setup: The KDD Cup 2005 dataset contains 800 user queries assigned up to five labels from 67 predefined categories by three human editors.
  • Dataset and setup: Query classification uses a simple kNN approach that ranks categories by similarity between each query embedding and each category centroid.
  • Dataset and setup: Performance is evaluated with precision and F1-measure using five-fold cross-validation, with statistical significance tested by a two-tailed paired t-test.
  • Results: Relevance-based embedding models significantly outperform word2vec and GloVe on both evaluation metrics.
  • Results: RPE performs better than RLM in query classification because correct category ordering matters, whereas similarity-value magnitudes do not.
  • Sensitivity analyses: Performance generally improves with embedding dimensionality and stabilizes above 400 dimensions, while at least 4 million training queries are needed for accurate embeddings.

5 CONCLUSIONS AND FUTURE WORK

The paper concludes that relevance-based word embeddings better align representation learning with information retrieval objectives than proximity-based embeddings. It evaluates two relevance-oriented models across query expansion and query classification, and identifies broader evaluation and improved training signals as future work.

  • Conclusions: The paper proposes relevance-based embeddings instead of learning word vectors solely from term proximity.
  • Conclusions: RLM estimates a relevance-distribution probability for each word and query, whereas RPE classifies each term as relevant or non-relevant.
  • Conclusions: Across query expansion on four TREC collections and query classification, relevance-based embeddings outperform state-of-the-art word embedding algorithms.
  • Conclusions: The models select expansion terms related to the whole query, unlike typical embeddings that select terms related to individual query terms.
  • Future work: Future work includes evaluating additional IR tasks and training with clicked documents instead of or alongside top retrieved documents.
Loading 1705.03556v2…