Source-linked AI summary
Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, Daxin Jiang
TL;DR
DSI indexes document text but retrieves from shorter queries, creating a data-distribution mismatch that is more pronounced in cross-lingual retrieval. DSI-QG indexes generated and cross-encoder-selected queries instead, and significantly outperforms the original DSI and other popular baselines on monolingual and cross-lingual passage retrieval.
Problem
DSI indexes document text but retrieves docids from shorter queries, and this mismatch is more pronounced when documents and queries use different languages.
Source: §1 · p. 1Method
DSI-QG generates potentially relevant queries for each document and uses a cross-encoder ranker to select queries for indexing.
Source: §1 · p. 2, §3.1 · p. 3Results
DSI-QG significantly outperforms the original DSI and other popular baselines on both monolingual and cross-lingual passage retrieval tasks.
Source: §7 · p. 10Takeaways & Limitations
The authors conclude that DSI-QG addresses the indexing–retrieval mismatch in monolingual and cross-lingual passage retrieval.
Source: §7 · p. 10The query-generation model and cross-encoder ranker require substantial additional computation during offline indexing, but do not affect online query latency.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
The Differentiable Search Index (DSI) is an emerging paradigm for information retrieval. Unlike traditional retrieval architectures where index and retrieval are two different and separate components, DSI uses a single transformer model to perform both indexing and retrieval. In this paper, we identify and tackle an important issue of current DSI models: the data distribution mismatch that occurs between the DSI indexing and retrieval processes. Specifically, we argue that, at indexing, current DSI methods learn to build connections between the text of long documents and the identifier of the documents, but then retrieval of document identifiers is based on queries that are commonly much shorter than the indexed documents. This problem is further exacerbated when using DSI for cross-lingual retrieval, where document text and query text are in different languages. To address this fundamental problem of current DSI models, we propose a simple yet effective indexing framework for DSI, called DSI-QG. When indexing, DSI-QG represents documents with a number of potentially relevant queries generated by a query generation model and re-ranked and filtered by a cross-encoder ranker. The presence of these queries at indexing allows the DSI models to connect a document identifier to a set of queries, hence mitigating data distribution mismatches present between the indexing and the retrieval phases. Empirical results on popular mono-lingual and cross-lingual passage retrieval datasets show that DSI-QG significantly outperforms the original DSI model.
1 INTRODUCTION
DSI unifies indexing and retrieval in one Transformer, but its document-text indexing inputs differ from shorter queries, especially in cross-lingual retrieval. DSI-QG indexes generated, relevant queries selected by a cross-encoder to address this mismatch.
1 INTRODUCTION
- DSI learns document-text-to-docid connections during indexing, then generates ranked docids from query text with the same Transformer.Traditional IR pipelines instead decouple indexing and retrieval.Source: §1 · p. 1
- Indexing uses long document text while retrieval uses shorter queries, creating an input mismatch that is more pronounced in cross-lingual retrieval.In that setting, document and query languages differ.Source: §1 · pp. 1–2
- Earlier DSI work found higher effectiveness after truncating long documents, while labeled-query fine-tuning leaves many documents without query-based indexing signals.For most documents without labeled relevant queries, DSI builds connections using the original document text alone.Source: §1 · p. 2
- DSI-QG generates potentially relevant queries for each document, then uses a cross-encoder to rank them and pass only the top-m to DSI indexing.For cross-lingual retrieval, the query generation model can generate queries in different languages.Source: §1 · p. 2
- Experiments on monolingual and cross-lingual retrieval datasets show DSI-QG improves DSI effectiveness by a large margin.The framework uses query generation and a cross-encoder ranker to represent documents for indexing.Source: §1 · p. 2
2 PRELIMINARIES
The original DSI trains a T5 model to map document text to docids and retrieves docids by generating them from queries. Its indexing inputs differ from shorter queries, and supervised query signals may be unavailable for many documents.
2 PRELIMINARIES
- DSI fine-tunes a T5 sequence-to-sequence model to generate each document’s identifier from its text during indexing.Training uses teacher forcing and cross-entropy loss.Source: §2 · p. 2
- Atomic Docids use one T5-vocabulary token per identifier, but their embedding-layer size limits large-corpus use, so the experiments exclude them.Source: §2 · p. 3
- The experiments use arbitrary String Docids, which avoid corpus-size limits; semantic clustering of docids is left for future work.Source: §2 · p. 3
- At retrieval, DSI autoregressively generates docid strings from queries and uses beam search to rank them by generation probability.The retrieval input is usually much shorter than the document text used for indexing.Source: §2 · p. 3
- Because human-judged relevant queries may be absent or scarce for many documents, supervised DSI fine-tuning leaves the mismatch largely unresolved, especially in large collections.Source: §2 · p. 3
3 THE DSI-QG FRAMEWORK
DSI-QG indexes documents using generated relevant queries, with a cross-encoder selecting queries for DSI; for cross-lingual retrieval, it generates queries in target languages.
3 THE DSI-QG FRAMEWORK
- A query-generation model proposes candidate queries, and a cross-encoder selects the top-n to represent each document for downstream DSI indexing.The framework combines these components before passing selected queries to DSI.Source: §3 · p. 3
3.1 DSI with query generation
- DSI-QG replaces each document with generated queries that share its docid, training DSI to associate those queries with that identifier.Each generated query is used separately; DSI receives short queries during both indexing and retrieval, addressing the input mismatch.Source: §3.1 · p. 3
- The T5 query-generation model learns from document text to labeled relevant queries; random top-k sampling favors diverse outputs, with k=10 and n queries generated per document.The paper identifies query-generation quality and the number of generated queries as factors in DSI-QG effectiveness.Source: §3.1 · pp. 3–4
3.2 DSI-QG with cross-lingual query generation
- For cross-lingual retrieval, a multilingual T5 model generates target-language queries from English documents, linking their identifiers to queries in other languages.The prompt specifies the target query language and document text.Source: §3.2 · p. 4
3.3 Ranking generated queries with a cross-encoder ranker
- A monoBERT cross-encoder ranks generated queries by relevance and retains the top-m; it uses query-document inputs and supervised contrastive training with annotated positives and BM25 hard negatives.Cross-encoder token interactions support relevance modeling; filtering removes the remaining n−m generated queries.Source: §3.3 · p. 4
- The large query-generation and ranking transformers add substantial offline indexing computation, but do not affect online query latency.Reducing the indexing computation is left for future work.Source: §3.3 · p. 4
4 EXPERIMENTAL SETTINGS
The experiments use NQ 320k for monolingual retrieval and XOR QA 100k for cross-lingual retrieval, compare DSI-QG with retrieval baselines, and evaluate with ranking metrics.
4.1 Datasets
- XOR QA 100k pairs queries in seven languages with English documents; 93k Wikipedia documents are added to form a more challenging 100k-document collection.Its training and dev sets contain around 15k and 2k gold document-query pairs, respectively.Source: §4.1 · p. 4
4.2 Baselines
- The comparison includes sparse BM25, BM25 with T5-generated query augmentation, SEAL, mDPR, and the original DSI, with some baselines limited to particular tasks.SEAL results are reported only for NQ 320k; mDPR runs only on XOR QA 100k.Source: §4.2 · pp. 4–5
4.3 Evaluation Measures
- Evaluation uses Hits@1 and Hits@10 on both datasets, plus nDCG@10 on XOR QA 100k.nDCG@10 is omitted for NQ 320k because it is unavailable for some baselines, preventing comparisons.Source: §4.3 · p. 5
4.4 Implementation Details
- DSI-QG uses query-generation, cross-encoder-ranking, and DSI Transformer models, with multilingual T5 for XOR QA and standard T5 initialization for NQ’s DSI model.The generated-query input length is capped at 32 tokens; models are implemented with Hugging Face Transformers and trained on eight Tesla A100 GPUs.Source: §4.4 · p. 5
5 RESULTS
DSI-QG improves on original DSI across mono-lingual and cross-lingual retrieval, while experiments show that ranking generated queries and selecting an appropriate number can improve effectiveness. Its query-based indexing also brings indexing input lengths closer to retrieval-query lengths.
(c) nDCG@10
- 132% and 46% are the Hits@1 and Hits@10 gains for T5-base DSI-QG over original DSI; T5-large gains are 83% and 32%.Model size has relatively little impact on DSI-QG effectiveness, and the smaller model requires less GPU memory and offers faster retrieval inference.Source: (c) nDCG@10 · p. 6
5.1 Effectiveness on Mono-lingual Retrieval
- DSI-QG outperforms all considered baselines on both Hits measures, while original DSI generally performs worse than other baselines, except for T5-large on the reported comparisons.BM25 with docT5query achieves the best Hits@1 among the baselines considered.Source: §5.1 · p. 5, (c) nDCG@10 · p. 6
5.2 Effectiveness on Cross-lingual Retrieval
- Across languages, DSI-QG achieves the highest Hits values except for Finnish Hit@10 and Arab and Russian nDCG@10, while DSI-base often retrieves no relevant document at Hit@1.The authors attribute the cross-lingual difficulty as likely reflecting a language gap exacerbating the data-distribution mismatch.Source: §5.2 · p. 7
5.3 Impact of Cross-encoder Ranker and Query Selection
- Ranking generated queries before indexing yields higher Hit@1 convergence than random selection for both model sizes, with particular benefit to the base model.The figure compares ranked and randomly selected query sets on XOR QA 100k.Source: §5.3.1 · p. 7, Figure 2 · p. 7
5.3.1 Impact of Cross-encoder Ranker.
- Hit@1 effectiveness rises sharply from m=1 to m=5; m=5 already exceeds original DSI, while m≥50 adds no significant gains over m=5–20.Here m is the number of generated queries used to represent each document; at m=100 the ranker has no effect on the selected set.Source: §5.3.1 · p. 7, Figure 3 · p. 7
5.4 Qualitative Analysis of Generated Queries and Ranking
- Generated-query quality, estimated by mDPR retrieval effectiveness, generally declines as the cross-encoder-assigned rank moves later.Using mDPR provides an estimate of query quality decoupled from DSI-QG training.Source: §5.4 · p. 8, Figure 4 · p. 8
5.5 Analysis of Length of DSI-QG Input
- DSI-QG indexing inputs are shorter and more similar in token length to retrieval queries, whereas original DSI inputs differ greatly between indexing and retrieval.Lengths are measured using the T5 tokenizer.Source: §5.5 · p. 8
6 RELATED WORK
Related work applies autoregressive generation and generated-query expansion to retrieval. DSI-QG augments document representations at indexing and extends query-generation-based retrieval to cross-lingual settings.
6.1 Retrieval via autoregressive generation
- DSI-QG augments document representations at indexing to bridge the indexing–retrieval gap, unlike autoregressive DSI and SEAL retrieval schemes.The related-work discussion also distinguishes DSI-QG from Wang et al., which considers English document retrieval only.Source: §6.1 · p. 9
6.2 Generate-then-rank
- DSI-QG uses a cross-encoder to rank generated queries and select high-quality queries for representing documents, following a generate-then-rank design.The passage motivates ranking by noting that generation is usually harder than classification for deep learning models.Source: §6.2 · p. 9
6.3 Query generation for information retrieval
- Prior query-generation methods expand documents for BM25 or use generated queries instead of original text, and DSI-QG’s query generation is akin to docT5query.The passage notes that query-only representations have also achieved better retrieval effectiveness than original document text.Source: §6.3 · p. 9
- TILDE generates independent unigram query tokens rather than query sequences, unlike the multi-token queries used at retrieval in DSI-QG’s datasets.The reported average query lengths are 12.1 and 14.8 tokens in the two datasets.Source: §6.3 · p. 9
- Unlike prior query-generation work focused on monolingual retrieval, this work also explores query generation for cross-lingual information retrieval.The passage describes Wang et al. as considering monolingual augmentation for English document retrieval.Source: §6.3 · p. 9, §6.1 · p. 9
7 CONCLUSION
The paper identifies indexing–retrieval data distribution mismatch as a problem for DSI, especially in cross-lingual passage retrieval. It proposes DSI-QG, which uses selected generated queries as document representations, and reports that it significantly outperforms original DSI and other popular baselines on both monolingual and cross-lingual passage retrieval tasks.
7 CONCLUSION
- Indexing–retrieval data distribution mismatch negatively affects DSI on monolingual passage retrieval and is more severe in cross-lingual retrieval, where DSI becomes impractical.Source: §7 · p. 10
- DSI-QG significantly outperforms original DSI and other popular baselines on monolingual and cross-lingual passage retrieval tasks.The framework uses a query generation model and cross-encoder ranker to select relevant queries as document representations.Source: §7 · p. 10
- DSI-QG addresses the mismatch by using a cross-encoder ranker to select generated relevant queries to represent documents at indexing.Source: §7 · p. 10