Source-linked AI summary
End-to-End Retrieval in Continuous Space
Daniel Gillick, Alessandro Presta, Gaurav Singh Tomar
TL;DR
The paper addresses end-to-end retrieval beyond systems that use embeddings only to rerank discrete-search candidates. It trains dual encoders for continuous retrieval and reports that the trained models improve on BM25 across both similar-question evaluations.
Problem
Embedding-based IR models have typically reranked candidates from discrete retrieval, while end-to-end retrieval evaluation must account for candidate-selection bias and random-pair robustness.
Source: §1 · p. 1, §3.1 · p. 3Method
The paper uses simple dual encoders with ANN search for continuous retrieval and adapts pairwise datasets into retrieval evaluations.
Source: §1 · p. 1Results
In-domain and multitask dual encoders improve on BM25 on both the Quora and AskUbuntu evaluations.
Source: §4 · p. 4Takeaways & Limitations
The results support end-to-end retrieval as a viable alternative to discrete retrieval.
Source: §5 · p. 5The AskUbuntu experiments use question titles only, leaving modeling of question bodies to future work.
Source: §3.3 · p. 3
Abstract
from arXiv · showhide
Most text-based information retrieval (IR) systems index objects by words or phrases. These discrete systems have been augmented by models that use embeddings to measure similarity in continuous space. But continuous-space models are typically used just to re-rank the top candidates. We consider the problem of end-to-end continuous retrieval, where standard approximate nearest neighbor (ANN) search replaces the usual discrete inverted index, and rely entirely on distances between learned embeddings. By training simple models specifically for retrieval, with an appropriate model architecture, we improve on a discrete baseline by 8% and 26% (MAP) on two similar-question retrieval tasks. We also discuss the problem of evaluation for retrieval systems, and show how to modify existing pairwise similarity datasets for this purpose.
1 Introduction
Discrete retrieval relies on words that can poorly represent conceptual meaning, while continuous retrieval uses learned dense representations to compare items. The paper studies end-to-end retrieval with ANN search, addressing a gap left by systems that use embeddings only to rerank discrete-search candidates.
1 Introduction
- LSI represented text as dense, low-dimensional vectors but did not address efficient retrieval from the learned space.It applied Singular Value Decomposition to word cooccurrences.Source: §1 · p. 1
- Continuous retrieval can improve recall beyond exact word choice, provide more granular similarity scores, learn query-candidate relationships, and enable cross-modal retrieval.Source: §1 · p. 1
- Prior embedding-based IR models rerank discrete-search candidates, while retrieval models must encode queries and candidates independently for fast lookup.Unlike rerankers focused on proposed candidates, retrieval models must also perform well on random pairs.Source: §1 · p. 1
- End-to-end continuous retrieval uses standard ANN search and simple retrieval-trained models to outperform discrete retrieval systems.The paper also evaluates modified pairwise datasets for retrieval.Source: §1 · p. 1
2 Dual Encoders
The paper frames dual encoders as a modular way to represent paired items in a shared space and retrieve neighbors using their similarity. It describes in-batch negative-sampling losses and a training setup for similar-question retrieval.
2 Dual Encoders
- Dual encoders independently map paired items into a shared space, where a similarity function scores their encodings.The encoders may be identical or different; retrieval uses cosine similarity to support nearest-neighbor search.Source: §2 · p. 2
- A simple similar-question model averages word embeddings, so its question encoder parameters are the learned embeddings.At retrieval time, candidate encodings are indexed offline and queries are encoded for cosine-distance neighbor lookup.Source: §2 · p. 2
2.1 Loss functions
- When sentence-pair negatives are not enumerable, negative sampling approximates them; in-batch strategies use other batch items as negatives.In-batch sampled softmax applies a row-wise softmax with the diagonal pair as positive and off-diagonal pairs as negatives.Source: §2.1 · p. 2
2.2 Training
- Training uses mini-batch gradient descent with momentum, a fixed 0.01 learning rate, and in-batch precision@1 for tuning.Unless noted otherwise, the batch size is 1000, embeddings are 300-dimensional, and early stopping is used to avoid overfitting.Source: §2.2 · p. 3
3 Experimental Setup
The experimental setup adapts pairwise datasets into incomplete retrieval tasks and evaluates models using MAP@100. It also describes the limits of pairwise evaluation, the ANN-search approach, and the baseline systems.
3.1 Evaluating end-to-end retrieval
- Pairwise and reranking test sets can be biased toward candidates chosen by existing systems and may not distinguish performance on random pairs.Retrieval evaluation is also incomplete because only a small subset of relevant candidates can be labeled, so unlabeled candidates are treated as irrelevant.Source: §3.1 · p. 3
3.2 Approximate nearest neighbor search
- Approximate nearest-neighbor search was nearly as accurate as exhaustive search, with MAP@100 declining no more than 0.4% as candidate sets grew from 20k to 1M.The experiments used exhaustive search to avoid confounding effects from ANN algorithm choice.Source: §3.2 · p. 3
3.3 Constructing retrieval tasks
- The retrieval-task construction turns positive pairs into a graph, takes its transitive closure, and uses graph neighbors as known relevant results.Candidates are all items in the test set; the AskUbuntu evaluation uses question titles, leaving question-body modeling to future work.Source: §3.3 · p. 3
3.4 Baselines
- Baselines include identity retrieval, TFIDF, BM25, averaged pretrained word embeddings with IDF variants, and Skip-Thought embeddings.The embedding baselines include standard word2vec, Paralex-trained word2vec, and GloVe.Source: §3.4 · p. 4
4 Analysis of Results
Across Quora and AskUbuntu, the multi-task dual encoder gave the best overall results; Quora experiments also favored in-batch sampled softmax and larger batches.
4 Analysis of Results
- The multi-task dual encoder combining data from all three tasks produced the best overall MAP@100 results on Quora and AskUbuntu.The Paralex-only dual encoder was trained to convergence before adding the multi-task loss.Source: §4 · p. 4
- 88.3 vs. 90.4 MAP@100: Quora-only standard cross-entropy underperformed negative-sampling strategies, while in-batch sampled softmax gave the best results and converged faster than in-batch cross-entropy.In-batch triplet loss was fairly similar to in-batch sampled softmax.Source: §4 · p. 4
- 82.8, 87.9, 89.2, and 90.4 MAP@100 on Quora corresponded, respectively, to batch sizes 2, 10, 100, and 1000.Retrieval results improved as batch size increased.Source: §4 · p. 4
5 Conclusion
The paper separates pairwise scoring from retrieval and introduces dual-encoder training with negative sampling that improves standard retrieval baselines. It presents end-to-end retrieval as a viable alternative while identifying scale, post-retrieval scoring, and modality extensions as future work.
5 Conclusion
- The paper distinguishes pairwise scoring, including reranking, from retrieval, and presents a general dual-encoder abstraction plus a negative-sampling setting that improves standard retrieval baselines.The abstraction supports training arbitrary complex distance functions.Source: §5 · p. 5
- The results begin to show end-to-end retrieval as a viable alternative to discrete retrieval.Source: §5 · p. 5
- Future work targets larger tasks with more retrieval candidates, post-retrieval scoring or reranking, and multimodal use of the dual-encoder framework.Source: §5 · p. 5