PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Word Embeddings for Automatic Query Expansion

Dwaipayan Roy, Debjyoti Paul, Mandar Mitra, Utpal Garain

arXiv:1606.07608v1cs.IR

TL;DR

The paper studies applying word embeddings to query expansion for standard ad hoc retrieval and compares embedding-based query expansion with RM3. The paper tests word-embedding query expansion variants that select related terms through nearest-neighbor methods, including an incremental approach. Embedding-based query expansion generally improves retrieval over original queries, but all tested methods are significantly inferior to RM3.

  • Problem

    The paper studies applying word embeddings to query expansion for standard ad hoc retrieval and compares embedding-based query expansion with RM3.

    Source: §1 · p. 1

  • Method

    The paper tests word-embedding query expansion variants that select related terms through nearest-neighbor methods, including an incremental approach.

    Source: §2 · p. 2, §2.3 · p. 2

  • Results

    Embedding-based query expansion generally improves retrieval over original queries, but all tested methods are significantly inferior to RM3.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors identify combining embeddings with co-occurrence-based techniques such as RM3 as future work.

    Source: §5 · p. 3

    The study uses embeddings only to select similar words and trains word2vec over the full vocabulary, leaving exhaustive embedding use and local training for future work.

    Source: §5 · p. 4

Abstract

from arXiv · show

In this paper a framework for Automatic Query Expansion (AQE) is proposed using distributed neural language model word2vec. Using semantic and contextual relation in a distributed and unsupervised framework, word2vec learns a low dimensional embedding for each vocabulary entry. Using such a framework, we devise a query expansion technique, where related terms to a query are obtained by K-nearest neighbor approach. We explore the performance of the AQE methods, with and without feedback query expansion, and a variant of simple K-nearest neighbor in the proposed framework. Experiments on standard TREC ad-hoc data (Disk 4, 5 with query sets 301-450, 601-700) and web data (WT10G data with query set 451-550) shows significant improvement over standard term-overlapping based retrieval methods. However the proposed method fails to achieve comparable performance with statistical co-occurrence based feedback method such as RM3. We have also found that the word2vec based query expansion methods perform similarly with and without any feedback information.

1. INTRODUCTION

The introduction motivates word embeddings for query expansion and asks whether nearest-neighbor expansion improves ad hoc retrieval and how it compares with RM3. It previews gains over unexpanded queries but lower performance than RM3.

1. INTRODUCTION

  • A word embedding maps each collection word or phrase to a lower-dimensional vector whose distance is expected to indicate semantic relatedness.The paper motivates embeddings because query expansion seeks terms semantically related to a user query.Source: §1 · p. 1
  • Nearest neighbors of individual query terms provide expansion candidates, while terms close to the query as a whole are generally preferable.The introduction notes that vector similarity from word2vec offers a way to find semantically related words.Source: §1 · p. 1
  • The work studies embedding-based query expansion for standard ad hoc retrieval, a setting where this use of embeddings had not been explored.Source: §1 · p. 1
  • The paper asks whether nearest neighbors of query terms improve retrieval, which queries benefit, and how embedding-based expansion compares with RM3.Source: §1 · p. 1
  • Embedding-based query expansion generally improves retrieval over original queries, but all tested methods are significantly inferior to RM3.The introduction reports significant gains against unexpanded queries and contrasts them with RM3.Source: §1 · p. 1

2. WORD EMBEDDING BASED QUERY EXPANSION

The section presents three word2vec-based query-expansion methods and extends queries with adjacent-word bigrams. It also describes how candidate terms are selected and weighted for retrieval.

2.1 Pre-retrieval kNN based approach

  • Candidate expansion terms are ranked by their mean cosine similarity to all query terms, and the top K are selected.For each query term, the method first collects its K nearest embedding-space neighbours.Source: §2.1 · p. 2

2.2 Post-retrieval kNN based approach

  • Restricting candidates to terms in pseudo-relevant documents narrows the search domain while leaving the remaining expansion procedure unchanged.The documents are those retrieved at top ranks for the initial query.Source: §2.2 · p. 2

2.3 Pre-retrieval incremental kNN based approach

  • The incremental method repeatedly prunes and reorders neighbours to favor terms similar both to the query term and to one another.The authors use five iterations; they note that l ≥ 10 may cause query drift, while l ≤ 2 behaves similarly to the basic pre-retrieval model.Source: §2.3 · p. 2

2.4 Extended Query Term Set

  • Adding adjacent-word bigrams to the query term set represents compositions of query words using summed word embeddings.The extended set Q′ is used in place of Q when finding neighbours and scoring candidates.Source: §2.4 · p. 2

2.5 Retrieval

  • Expansion-term weights combine query-term likelihood with normalized similarity scores for the selected candidates.The retrieval experiments use a language model with Jelinek–Mercer smoothing.Source: §2.5 · p. 2

3. EVALUATION

Across the evaluated TREC and WT10G collections, embedding-based query expansion generally improves on the language-model baseline, while RM3 performs significantly better. The evaluation also compares query-level effects and the use of query composition.

3. EVALUATION

  • The evaluation uses TREC Disk 4 and 5 with TREC 6, 7, 8, and Robust query sets, plus the WT10G web collection.The method was implemented with Lucene and compared using its language model with linear smoothing.Source: §3 · pp. 2–3, Table 1 · p. 3

3.1 Experimental Setup

  • Experiments use title-only queries, 200-dimensional Word2Vec embeddings, and parameter-grid searches to select each method’s best-performing settings.The embedding setup uses a five-word window, continuous bag-of-words training, and negative sampling; the language-model smoothing parameter is set to 0.6.Source: §3.1 · p. 3

3.2 Results

  • Composition positively affects MAP for pre-retrieval kNN, so it is used for similarity computation in the remaining approaches.Table 2 compares nearest-neighbor selection using raw query terms with selection using composed queries.Source: §3.2 · p. 3, Table 2 · p. 3
  • Embedding-based QE methods almost always outperform the LM baseline, while RM3 is significantly superior across all query sets.The incremental variant appears most consistent in improving over the baseline.Source: §3.2 · p. 3
  • The incremental method improves AP for most queries and hurts it for only a few, although RM3 wins more often than it loses against incremental QE.Question 2—characterizing which queries benefit—remains for further investigation.Source: §3.2 · p. 3

4. DISCUSSION

The discussion attributes gains over the text-similarity baseline to word2vec’s semantic and contextual information. It also explains why the embedding variants perform similarly and identifies a runtime cost for incremental kNN.

4. DISCUSSION

  • Embedding-based QE selects terms by similarity in an abstract embedding space, but does not capture other useful features such as frequent co-occurrence with query terms.The authors report RM3 significantly outperforming the proposed methods on TREC ad-hoc and web datasets, and attribute the difference to co-occurrence statistics being more powerful than abstract-space similarity.Source: §4 · p. 3
  • Incremental kNN takes more time than pre- or post-retrieval kNN because it uses an iterative pruning step.Source: §4 · p. 3

5. CONCLUSION AND FUTURE WORK

The conclusion reports that word-embedding query expansion improves on unexpanded retrieval but remains inferior to RM3. It proposes combining embeddings with co-occurrence methods and exploring local training.

5. CONCLUSION AND FUTURE WORK

  • Word-embedding QE performs better than the unexpanded baseline but remains significantly inferior to feedback methods such as RM3.The proposed future direction is to combine embeddings with co-occurrence-based methods such as RM3.Source: §5 · p. 3
  • The experiments use embeddings only to select similar words, leaving other aspects of their representations for future study.The paper also proposes training word2vec locally from pseudo-relevance documents as a possible way to reduce the generalization effect of whole-vocabulary training.Source: §5 · p. 4

TREC8: Baseline vs. Incremental

The supplied figure caption describes per-query AP differences, and the comparison labels name several method pairs.

TREC8: Baseline vs. Incremental

Loading 1606.07608v1…