PaperlayerView exact revision on arXiv

Source-linked AI summary

Query Expansion with Locally-Trained Word Embeddings

Fernando Diaz, Bhaskar Mitra, Nick Craswell

arXiv:1605.07891v2cs.IRcs.CL

TL;DR

The paper asks whether global word embeddings adequately represent topic-specific language for query expansion in ad hoc retrieval. It trains embeddings on query-related documents and finds that local embeddings outperform global ones across retrieval conditions, while remaining less efficient and not yet fully explained.

  • Problem

    Global embeddings use fixed or corpus-dominant representations that may miss topic-specific language, raising whether local representations better support query expansion.

  • Method

    The paper retrieves query-related documents with a language-modeling approach, trains local word embeddings from them, and uses embedding similarity for query expansion.

  • Results

    Local embeddings significantly outperform global embeddings for query expansion across all tested conditions.

  • Takeaways & Limitations

    Smaller, high-quality topical data can provide better performance than vast global training data, and local representations may benefit other tasks with topical locality.

  • Takeaways & Limitations

    Local embeddings are less efficient than global embeddings, and the supporting experiments do not yet provide a solid explanation of the results.

Abstract

from arXiv · show

Continuous space word embeddings have received a great deal of attention in the natural language processing and machine learning communities for their ability to model term similarity and other relationships. We study the use of term relatedness in the context of query expansion for ad hoc information retrieval. We demonstrate that word embeddings such as word2vec and GloVe, when trained globally, underperform corpus and query specific embeddings for retrieval tasks. These results suggest that other tasks benefiting from global embeddings may also benefit from local embeddings.

1. INTRODUCTION

Global word embeddings provide fixed representations that can miss topic-specific language, motivating locally trained embeddings for query expansion. The paper proposes learning embeddings from topically constrained corpora to capture query-relevant lexical nuances.

  • 1. INTRODUCTION: Global embeddings assign each word a fixed representation regardless of discourse context, which can miss topic-specific meanings.Their representations may capture only coarse patterns from topics dominant in the training corpus.
  • 1. INTRODUCTION: Topic-specific representations are motivated for query expansion because retrieval provides an extrinsic way to evaluate lexical similarity.Prior work had shown that global embedding similarity could outperform classic pseudo-relevance feedback.
  • 1. INTRODUCTION: The paper learns embeddings from topically constrained corpora rather than large unconstrained corpora, retraining them on documents related to each query.These local embeddings are intended to capture topic-specific language more effectively than global embeddings.
  • 1. INTRODUCTION: The study tests whether the global-versus-local performance pattern observed in other retrieval methods also holds for word embeddings.The cited prior evidence concerns query expansion, latent semantic analysis, cluster-based retrieval, and term clustering.

2. MOTIVATION

Word2vec emphasizes frequent corpus word-context pairs, creating a mismatch when topic-specific language differs from the corpus distribution. The paper motivates local training by showing how topic weighting and divergence identify important terms that global training may underemphasize.

  • 2. MOTIVATION: Word2vec trains a neural network to predict a word from a small set of context words using an instance loss over observed word-context pairs.The model uses term and context embeddings and randomly sampled negative contexts.
  • 2. MOTIVATION: Global training estimates its objective from corpus word-context distribution pc, even when analysis is restricted to a specialized topic.Topic-specific distributions pt can differ substantially from the whole-corpus distribution.
  • 2. MOTIVATION: Topic-frequent contexts receive importance weight ω = pt(w,c)/pc(w,c), amplifying their loss relative to corpus training.Specialized contexts can be frequent in the topic but underemphasized when training according to the corpus objective.
  • 2. MOTIVATION: A TREC topic analysis found a non-trivial number of terms with much larger topic-to-corpus importance weights.The analysis used documents related to one retrieval topic and displayed the weight histogram in Figure 1.
  • 2. MOTIVATION: Pointwise KL divergence highlights terms that are both relatively more probable in the topic and substantial in absolute topic probability.Such terms tend to be good query-expansion candidates, but large losses on them may produce poor topic-relevant embeddings.
  • 2. MOTIVATION: Heuristic downsampling of frequent terms does not prevent global word2vec techniques from producing inferior performance for specific topics.

3. LOCAL WORD EMBEDDINGS

The paper constructs query-specific document distributions from retrieval scores and samples from them to train local word embeddings. This adapts embedding training to the topical language associated with each query.

  • Query-specific document sampling: Users typically provide keywords rather than topic-specific documents, so the method retrieves documents to approximate the query’s topical set.The approach uses information retrieval techniques to generate this query-specific document set.
  • Query-specific document sampling: Documents are represented as maximum-likelihood language models, and their similarity to the query is measured with KL divergence.Lower KL divergence corresponds to greater similarity between document and query language models.
  • Query-specific document sampling: The document scores are transformed with a softmax into a multinomial p(d) over the corpus, weighting documents relevant to the query topic.The resulting distribution is acknowledged to be imprecise, but highly scored nonrelevant documents may still use language consistent with known relevant documents.
  • Local embedding training: A local word embedding is trained by sampling documents from p(d) instead of uniformly from the corpus.The paper uses word2vec for this sampling-based local training, while noting that arbitrary document-sampling optimization methods could be used.

4. QUERY EXPANSION WITH WORD EMBEDDINGS

The query-expansion method derives expansion weights from term embeddings, forms an expansion language model, and interpolates it with the original query model. It compares globally trained embeddings with locally trained embeddings sampled from query-relevant documents.

  • Embedding-based expansion: The expansion language model pq+ is formed by normalizing the weights associated with selected expansion terms.This creates a language-model representation of the expansion terms.
  • Expanded retrieval: The expansion model is interpolated with the original query model and then used to rank documents.The resulting ranking is called the expanded query score of a document.
  • Embedding-based expansion: Expansion terms receive weights from UUTq, where U is the term embedding matrix and q is the query term vector.The method selects the top k terms, normalizes their weights, and computes the expansion language model pq+.
  • Global versus local embeddings: The comparison uses a global embedding trained by uniformly sampling corpus documents and a local embedding trained by sampling documents from p(d).The local model is the paper’s proposed alternative.

5. METHODS

The evaluation uses three retrieval collections, multiple global and local embedding sources, standard ranking metrics, and cross-validation of retrieval and embedding parameters. Local embeddings are trained with query-weighted document sampling under a fixed experimental protocol.

  • Datasets: The experiments use trec12, robust, and web retrieval datasets drawn from TREC newswire and the ClueWeb 2009 Category B corpus.The web collection retains documents with a Waterloo spam rank above 70.
  • Embedding sources: Global baselines include four GloVe embeddings, one Google News word2vec embedding, and corpus-trained embeddings for trec12 and robust.The web experiments use GloVe trained on Common Crawl instead of a global embedding trained on the full web collection.
  • Embedding sources: Local word2vec embeddings are trained from documents retrieved in the target corpus or from Gigaword and a December 2014 Wikipedia snapshot.The auxiliary sources are intended to provide larger or higher-fidelity local training collections.
  • Evaluation: Performance is evaluated with NDCG@10 and interpolated precision at standard recall points.NDCG@10 emphasizes performance at high ranks, while interpolated precision describes the entire ranked list.
  • Evaluation: All retrieval experiments use 10-fold cross-validation over expansion terms, interpolation weight, and local-training learning rate.The cross-validated ranges are k ∈[5 −500], λ ∈[0, 1], and α ∈{0.1, 0.01, 0.001}.
  • Implementation: Local models sample 1000 documents from p(d) with replacement, train for 80 iterations, and use a fixed embedding dimension of 400.Expanded queries rescore the top 1000 documents, with candidate expansion terms restricted to the initial retrieval vocabulary.

6. RESULTS

Local embeddings significantly outperform global embeddings for query expansion across conditions, although the proposed explanation based on topical divergence receives only weak empirical support.

  • Retrieval results: Embedding-based query expansion outperforms the query likelihood baseline across all tested conditions.For trec12, an embedding trained on the target corpus significantly outperforms other global embeddings despite using substantially less training data.
  • Retrieval results: Local embeddings significantly outperform global embeddings for query expansion across all conditions.For news collections, Gigaword retrievals help local training, whereas Wikipedia retrievals tend to underperform; the web collection benefits more from Wikipedia.
  • Retrieval results: Local methods almost always strictly dominate the baseline and global expansion across all recall levels in interpolated precision-recall curves.Global methods improve NDCG@10, but those gains are not reflected in the precision-recall curves.
  • Analysis: The hypothesized relationship between local KL divergence and local-over-global improvement is positive across collections but low in rank correlation.Table 3 measures Kendall’s τ and Spearman’s ρ between NDCG@10 improvement and local KL divergence.
  • Qualitative analysis: Qualitative analysis visualizes high-frequency relevant terms in global and local embedding spaces for the query “ocean remote”.

7. DISCUSSION

The discussion argues that local training can outperform global representations because task-relevant data may be more valuable than vast but mismatched data, while efficiency and explanatory questions remain open.

  • Implications: Local embeddings can provide better performance when training data is smaller but more appropriate to the task.The discussion contrasts globally estimated representations with query-time use of topical information.
  • Evaluation: Figure 4 compares query likelihood with the best global and local query-expansion methods using interpolated precision-recall curves.
  • Implications: Local contextualization can use queries or other context such as sentences, paragraphs, or documents to re-weight the corpus.
  • Limitations: Local embedding training is quite inefficient compared with global embeddings.The authors suggest offline computation at an intermediate granularity could improve efficiency without training a query-specific model each time.
  • Limitations: The supporting experiments do not yet provide a solid explanation for why local embeddings outperform global embeddings.The authors call for further theoretical and empirical analysis.

8. RELATED WORK

Related work adapts language and word representations to topical context, contrasting corpus-level global embeddings with topic-specific alternatives. Prior IR approaches include matrix-based and co-occurrence-based embeddings, while local latent semantic analysis restricts modeling to query-relevant documents.

  • Topical adaptation: Topic-specific word embeddings had been proposed alongside multiple embeddings per word, but this paper identifies training topic-specific embeddings as unexplored.The related work positions the paper as extending topical adaptation into word-embedding training.
  • Topical adaptation: Topic-dependent language-model adaptation trains models from topic-specific documents gathered before or during discourse, sometimes combining local and global models.This paper instead trains embeddings from documents retrieved for the current query and focuses on direct comparison rather than interpolation.
  • Word embeddings for IR: Figure 5 represents candidate expansion terms as points, coloring them by frequency in relevant documents and marking the query in blue.Contours encode distance from the query in the two-dimensional projection.
  • Word embeddings for IR: IR word representations include term-document approaches such as LSA, PLSA, and LDA, and term-cooccurrence approaches such as word2vec.The cited literature distinguishes these families by their underlying data sources.
  • Word embeddings for IR: Term-document models vary by task and can perform poorly for retrieval unless combined with lexical features.This motivates evaluating representation methods with retrieval-specific metrics rather than relying only on general similarity tasks.
  • Local latent semantic analysis: Restricting latent semantic analysis to documents relevant to a query significantly improved over corpus-level analysis for routing tasks.This earlier result provides precedent for specializing representations to a query or topic.

9. CONCLUSION

The paper presents query expansion with word embeddings as a simple, effective method and emphasizes query-specific local training. Its results support considering local embeddings as potentially superior representations in research that currently uses global vectors.

  • Conclusion: The paper demonstrates a simple and effective method for query expansion with word embeddings.The conclusion highlights query-specific local training as the central value of the approach.
  • Conclusion: Query-specific local training is presented as a potentially superior representation to global embedding vectors.The conclusion broadens this consideration to other research adopting global embeddings.
  • Conclusion: The conclusion argues that embedding methods should be adapted to the topic rather than applied uniformly across corpora.Its cooking metaphor emphasizes choosing representations suited to the task and topical context.
Loading 1605.07891v2…