PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Vector Spaces for Unsupervised Information Retrieval

Christophe Van Gysel, Maarten de Rijke, Evangelos Kanoulas

arXiv:1708.02702v4cs.IRcs.CL

TL;DR

Vocabulary mismatch makes retrieval difficult when queries and relevant documents use different terms. NVSM learns word and document representations from scratch without relevance signals, and the authors report stronger ranking than existing latent vector-space approaches. Adding NVSM to lexical and vector-space rankers supplies a complementary signal.

  • Problem

    Vocabulary mismatch is a critical search challenge because queries and documents may use different terms to describe the same concepts.

    Source: §1 · p. 1

  • Method

    NVSM learns low-dimensional word and document representations without relevance signals by training word sequences from a document to predict that document.

    Source: §1 · p. 2

  • Results

    NVSM performs better than existing latent vector-space and bag-of-words approaches, and adds a complementary signal to lexical language models.

    Source: §7 · p. 21

  • Takeaways & Limitations

    An unsupervised ensemble of NVSM models trained with different hyperparameters performs better than a single cross-validated model.

    Source: §7 · p. 21

    Latent document vector spaces have space complexity tied to the number of documents, limiting their scalability to large collections.

    Source: §7 · p. 21

Abstract

from arXiv · show

We propose the Neural Vector Space Model (NVSM), a method that learns representations of documents in an unsupervised manner for news article retrieval. In the NVSM paradigm, we learn low-dimensional representations of words and documents from scratch using gradient descent and rank documents according to their similarity with query representations that are composed from word representations. We show that NVSM performs better at document ranking than existing latent semantic vector space methods. The addition of NVSM to a mixture of lexical language models and a state-of-the-art baseline vector space model yields a statistically significant increase in retrieval effectiveness. Consequently, NVSM adds a complementary relevance signal. Next to semantic matching, we find that NVSM performs well in cases where lexical matching is needed. NVSM learns a notion of term specificity directly from the document collection without feature engineering. We also show that NVSM learns regularities related to Luhn significance. Finally, we give advice on how to deploy NVSM in situations where model selection (e.g., cross-validation) is infeasible. We find that an unsupervised ensemble of multiple models trained with different hyperparameter values performs better than a single cross-validated model. Therefore, NVSM can safely be used for ranking documents without supervised relevance judgments.

1 INTRODUCTION

Vocabulary mismatch leaves relevant documents undiscovered when queries and documents use different terms. NVSM learns word and document representations from scratch without relevance labels, using a document-prediction objective.

1 INTRODUCTION

  • Vocabulary mismatch is a critical search challenge, especially when relevant documents contain none of the query terms and remain undiscovered.The paper motivates semantic matching methods that can operate early in retrieval.Source: §1 · pp. 1–2
  • Latent semantic models can lack term specificity, while prediction-based models learned from scratch have received little attention in unsupervised document retrieval.The authors argue that retrieval needs specialized solutions and an appropriate optimization objective.Source: §1 · p. 2
  • NVSM learns low-dimensional word and document representations without relevance signals by training word sequences from a document to predict that document.The model is optimized from scratch using gradient descent.Source: §1 · p. 2
  • NVSM is designed to capture semantic matching, cluster documents with similar language, and neglect words associated with many documents.Because latent document spaces require one vector per document, the paper evaluates collections of roughly 200,000 to 2 million articles.Source: §1 · p. 2
  • The paper compares lexical and semantic retrieval models, analyzes NVSM’s internals, and presents a fast C++/CUDA implementation and hyperparameter advice.Its comparisons include document-retrieval test collections and latent vector-space models.Source: §1 · p. 3

2 RELATED WORK

The related work ranges from latent semantic models and methods that reuse pretrained representations to neural systems that learn representations from scratch. The paper positions NVSM as a shallow, unsupervised retrieval model trained directly on the collection.

2 RELATED WORK

  • Retrieval models represent queries and documents in finite-dimensional spaces, distinguishing term-based bag-of-words vectors from lower-dimensional latent representations.The section surveys representation learning for retrieval, emphasizing neural methods.Source: §2 · p. 3, §2.1 · p. 3

2.1.1 Latent semantic models.

  • LSI and pLSI were introduced to mitigate query-document mismatch, while LDA is a topic model that generalizes to unseen documents.Source: §2.1.1 · p. 3

2.1.2 Combining pre-trained word or document representations.

  • Methods using pretrained representations derive document vectors from word vectors for reranking or combine representation similarities; examples include embedding aggregation, translation models, and BoWE.The paper notes that nearest-neighbor search is a practical way to compare latent vector-space models.Source: §2.1.2 · p. 4

2.1.3 Learning from scratch.

  • Prior work learning representations from scratch includes semantic hashing, doc2vec, and neural language models for entity ranking; the paper contrasts its vector-space retrieval with reranking-only use of doc2vec.Source: §2.1.3 · p. 4

2.2 Neural IR and language modeling

  • Neural IR also includes supervised ranking and click-based matching, while neural language-model work spans NLP and speech; NVSM instead learns shallow representations directly from the collection.The authors attribute improvements over LSE to increased regularization, faster training through reduced internal covariate shift, and term specificity in learned word representations.Source: §2.2 · pp. 4–5

3 LEARNING SEMANTIC SPACES

NVSM learns word and document representations from sampled n-gram/document pairs, then ranks documents by similarity to query representations. Its objective adds mechanisms for semantic matching and term specificity, while implementation is constrained by memory use.

3.1 The Neural Vector Space Model

  • NVSM extends LSE with learned word and document representations, a transformation between their spaces, and nearest-neighbor ranking by query-document cosine similarity.Queries are composed from word representations and projected into document feature space.Source: §3.1 · p. 6

3.2 The objective and its optimization

  • Training makes n-gram representations predictive of their source documents by bringing paired representations close and separating them from other documents.Mini-batches contain sampled n-gram/document pairs; n-grams are composed by averaging constituent word representations.Source: §3.2 · pp. 6, 8
  • The objective adds word competition and standardization so discriminative words contribute more, frequent-word regularities are separated from concept matching, and negative-sample count does not bias positive instances.The learned word-vector norms provide a notion of term specificity.Source: §3.2 · pp. 7, 9

3.3 Implementation

  • Training is memory-bound because parameters and optimizer state must remain on the GPU; the authors implemented the procedure in C++/CUDA to use sparsity and reduce memory use.For 1M documents and a 64K-term vocabulary, the paper estimates a 3.30GB resident-memory lower-bound example.Source: §3.3 · pp. 9–10

4 EXPERIMENTAL SETUP

The experiments compare NVSM with latent and lexical retrieval methods on TREC newswire article collections. They assess retrieval effectiveness, per-query differences, combinations of lexical and latent signals, and learned term-frequency regularities.

4.1 Research questions

  • The experiments ask how NVSM compares with latent and lexical rankers, whether its per-query gains are broad, whether combining it with other rankers helps, and what collection regularities it learns.The questions also probe the relation between term specificity and document length statistics.Source: §4.1 · p. 10

4.2 Benchmark datasets & experiments

  • Evaluation uses six news-article collections and TREC topics, with separate validation and test queries except for the New York collection, whose settings are selected using Robust04 validation.The authors focus on article retrieval to isolate query-document matching from factors such as freshness and link or click graphs.Source: §4.2 · pp. 10–11

4.3.1 Models compared.

  • Comparisons include latent vector-space methods and lexical query-likelihood models; nearest-neighbor retrieval is used so documents without query terms can be surfaced beyond an initial lexical candidate pool.Latent methods include doc2vec, word2vec, LSI, LDA, LSE, and NVSM.Source: §4.3.1 · pp. 11–13

4.3.2 Combinations of QLM and latent features.

  • Ranker combinations are selected by cross-validation over linear feature weights, using QLM log-probabilities and cosine similarities for NVSM and word2vec with self-information weighting.The selected weights maximize MAP on fold training data before scoring the fold test set.Source: §4.3.2 · p. 13

4.4 Evaluation measures and statistical significance

  • Retrieval is evaluated with MAP@1000, NDCG@100, and P@10, with paired t-tests for differences and permutation tests for correlations.Term-specificity analyses compare word-vector norms across frequency groups using Welch’s t-test.Source: §4.4 · pp. 13–14

5 RESULTS

Across retrieval benchmarks, NVSM outperforms latent rankers, while query-level and combination analyses show complementary behavior with lexical models. Its term representations also reflect Luhn-style specificity.

5.1 Performance of NVSM

  • NVSM outperforms every latent ranker on all benchmarks and significantly exceeds the self-information-weighted word2vec method on MAP@1000 except NY.Against lexical QLM, NVSM performs better on AP88-89 and WSJ.Source: §5.1 · p. 15

5.3 Semantic vs. lexical matching

  • Adding NVSM to QLM+w2v-si improves MAP@1000 on all benchmarks, significantly on five of six.QLM+NVSM outperforms QLM+w2v-si on four of six benchmarks; on FT, the relative performance increase nearly doubles despite no significance.Source: Table 3 · p. 16, §5.3 · pp. 17–18

5.2 Query-level analysis

  • NVSM beats w2v-si on 40 to 55% of queries, while w2v-si beats NVSM on 20 to 35%.QLM and NVSM also make very different errors.Source: Figure 1 · p. 15, §5.2 · p. 17

5.3 Semantic vs. lexical matching

  • NVSM performs well when lexical matching is needed as well as when semantic matching is needed, giving it a hybrid matching signal.The analysis relates query-term overlap with relevant documents to per-query MAP@1000 differences between NVSM and other methods.Source: Table 4 · p. 17, §5.3 · p. 18

5.4 NVSM and Luhn significance

  • NVSM term representations give medium-frequency terms greater L2-norms than infrequent or frequent terms, matching Luhn significance patterns.The relationship is statistically significant (p < 0.01) and learned from an unsupervised objective without relevance information; LSE instead shows term norms growing linearly with collection frequency.Source: §5.4 · p. 18

6 UNSUPERVISED DEPLOYMENT

The section proposes unsupervised model selection by combining NVSMs trained with different n-gram widths. This ensemble outperforms a single cross-validated model, while training convergence informs the chosen iteration count.

6 UNSUPERVISED DEPLOYMENT

LSE NVSM LSE NVSM

  • With document dimensionality kd = 256, MAP@1000 converges as training progresses across n-gram widths, so the authors train for 15 iterations and use the last model.Source: LSE NVSM LSE NVSM · p. 19, Figure 3 · p. 20
  • The ensemble sums standardized scores from NVSMs trained with different n-gram widths, estimating score statistics from each model’s top-1,000 results.Source: LSE NVSM LSE NVSM · p. 20
  • The unsupervised ensemble outperforms the single cross-validated model on every benchmark, significantly in MAP@1000 except NY.Source: LSE NVSM LSE NVSM · p. 20, Table 5 · p. 21

7 CONCLUSIONS

The conclusions present NVSM as an unsupervised latent retrieval model that supports both lexical and semantic matching. They also identify limits around short documents, scaling, and incremental updates.

  • NVSM learns document representations without supervision and performs better than existing latent vector space and bag-of-words approaches.Source: §7 · p. 21
  • NVSM supplies a complementary relevance signal and supports lexical matching as well as semantic matching; additional objectives may improve retrieval performance.Source: §7 · p. 21
  • The word-sequence objective is less applicable to very short documents, and scaling latent document vectors and distributed training remain challenges.The paper reports extending LSE to spaces of about 500k documents, while leaving distributed learning for future work.Source: §7 · p. 21
  • Representations for newly added documents have no theoretically sound incremental method, and fixed-parameter training offers no guarantee of quality, especially with topic drift.Source: §7 · pp. 21–22
  • Future work includes greater model expressiveness, deeper analysis of learned representations and component combinations, and training scalability.Source: §7 · p. 22

Related papers

Loading 1708.02702v4…