PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense X Retrieval: What Retrieval Granularity Should We Use?

Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, Dong Yu

arXiv:2312.06648v3cs.CLcs.AIcs.IR

TL;DR

Dense retrieval depends on how the corpus is segmented and indexed, but the effects of that inference-time choice are underexplored. The paper compares passage, sentence, and proposition units, finding propositions perform better across retrieval and downstream QA; it also reports shorter prompts can match performance, with study scope limited to English Wikipedia and evaluated retrievers.

  • Problem

    The paper investigates how retrieval-corpus granularity affects passage retrieval and whether fine-grained units can replace passages in downstream QA.

    Source: §1 · p. 2

  • Method

    The authors propose propositions as atomic, self-contained factoid units and compare passage-, sentence-, and proposition-indexed Wikipedia for dense retrieval.

    Source: §1 · p. 2, §4.3 · p. 4

  • Results

    Proposition retrieval outperforms passage- and sentence-level retrieval on passage retrieval and downstream QA across five open-domain QA datasets; average passage-based Recall@20 gains are +10.1 for unsupervised and +2.7 for supervised retrievers.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Finer-grained retrieved units increase question-relevant information density, allowing prompts to achieve the same performance with shorter inputs and faster inference.

    Source: §1 · p. 2

    The study is limited to English Wikipedia and evaluates six popular dense retriever types, most of which use bi- or dual-encoder architectures.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Dense retrieval has become a prominent method to obtain relevant context or world knowledge in open-domain NLP tasks. When we use a learned dense retriever on a retrieval corpus at inference time, an often-overlooked design choice is the retrieval unit in which the corpus is indexed, e.g. document, passage, or sentence. We discover that the retrieval unit choice significantly impacts the performance of both retrieval and downstream tasks. Distinct from the typical approach of using passages or sentences, we introduce a novel retrieval unit, proposition, for dense retrieval. Propositions are defined as atomic expressions within text, each encapsulating a distinct factoid and presented in a concise, self-contained natural language format. We conduct an empirical comparison of different retrieval granularity. Our experiments reveal that indexing a corpus by fine-grained units such as propositions significantly outperforms passage-level units in retrieval tasks. Moreover, constructing prompts with fine-grained retrieved units for retrieval-augmented language models improves the performance of downstream QA tasks given a specific computation budget.

1 Introduction

The introduction frames retrieval granularity as an inference-time choice and proposes propositions as an alternative to passages or sentences. It reports stronger retrieval and downstream QA performance with propositions, and says finer-grained prompts can achieve the same performance with shorter inputs.

1 Introduction

  • Proposition retrieval yields the best passage-retrieval and downstream open-domain QA performance; average passage-based Recall@20 gains are +10.1 for unsupervised and +2.7 for supervised retrievers.The reported best-performance comparison is shown in Figure 1.Source: Figure 1 · p. 1, §1 · p. 2
  • The study asks how corpus indexing granularity affects passage retrieval and whether fine-grained units can replace passages in downstream QA.Source: §1 · pp. 1–2
  • Passages can contain extraneous details, while sentences may remain complex or lack context needed to judge query-document relevance.For example, a sentence referring to “the tower” may omit the coreference “Pisa Tower.”Source: §1 · p. 2
  • The authors propose propositions—atomic expressions that each encapsulate a distinct factoid in concise, self-contained natural language—as a retrieval unit.The example describes the Tower of Pisa’s current leaning angle in a self-contained way that precisely responds to the question.Source: §1 · p. 2
  • The authors introduce FACTOIDWIKI, an English Wikipedia dump segmented into propositions, and compare passage, sentence, and proposition retrieval across five open-domain QA datasets.The experiments compare four dual-encoder retrievers.Source: §1 · pp. 2–3
  • Finer-grained retrieved units increase question-relevant information density, allowing prompts to match performance with shorter inputs and faster inference.Source: §1 · p. 2

2 Proposition as a Retrieval Unit

The paper treats propositions as atomic units of meaning for dense retrieval. Each should preserve the text’s semantics while remaining minimal and independently interpretable.

2 Proposition as a Retrieval Unit

  • Each proposition corresponds to a distinct piece of meaning, and together the propositions should represent the semantics of the entire text.Source: §2 · p. 3
  • A proposition should be minimal and not further divisible into separate propositions.Source: §2 · p. 3
  • A proposition should be contextualized and self-contained, including necessary context such as coreference to make its meaning independently interpretable.For example, the text resolves “the tower” to “the Leaning Tower of Pisa.”Source: §2 · p. 3
  • The Tower of Pisa example divides a passage into three propositions about its angle before restoration, its current angle, and its horizontal displacement.Source: §2 · p. 3

3 FACTOIDWIKI: Proposition-Level Index and Retrieval for Wikipedia

The section describes how Wikipedia is segmented and indexed at passage, sentence, and proposition granularity, with propositions generated by a finetuned model. A manual sample review assesses the generated propositions’ quality.

3 FACTOIDWIKI: Proposition-Level Index and Retrieval for Wikipedia

  • The 2021-10-13 English Wikipedia dump is segmented into passages, sentences, and propositions for comparison.The resulting processed corpus is called FACTOIDWIKI.Source: §3 · p. 3
  • A finetuned Flan-T5-large Propositionizer generates proposition lists from passages, trained on 42k GPT-4-generated passage-to-proposition examples.GPT-4 first creates a seed set from prompted passages; the model is then finetuned on that set.Source: §3 · p. 3
  • On a random sample of 50 passages, almost all generated propositions were faithful, while a small portion were not stand-alone.The error analysis also tracked unsupported propositions and propositions that could be split further.Source: §3 · p. 4

4 Experimental Settings

The experiments compare passage, sentence, and proposition indexing on five open-domain QA datasets using four dense retrievers. They evaluate passage retrieval and downstream QA, including language-model QA under a fixed retrieved-token budget.

4 Experimental Settings

  • Experiments compare passage retrieval and downstream QA when dense retrievers use Wikipedia indexed at different granularities.The evaluation uses FACTOIDWIKI.Source: §4 · p. 4

4.1 Open-Domain QA Datasets

  • The five datasets are Natural Questions, TriviaQA, Web Questions, SQuAD, and Entity Questions.The retrieval source is Wikipedia.Source: §4.1 · p. 4

4.2 Dense Retrieval Models

  • The four retrievers are SimCSE and Contriever, which are unsupervised, and DPR and GTR, which use human-labeled query-passage supervision.DPR is trained on NQ, TQA, WebQ, and SQuAD query-passage pairs; GTR is fine-tuned on MS MARCO and NQ.Source: §4.2 · p. 4

4.3 Passage Retrieval Evaluation

  • Passage Recall@k measures the share of questions whose answer occurs among the top-k retrieved passages, including when retrieval scores come from sentences or propositions.For sentence- and proposition-level indexing, a passage score is the maximum similarity of its units to the query; a FiD reader also evaluates exact-match QA.Source: §4.3 · p. 4

4.4 Open-domain QA Evaluation on Retrieval-Augmented Language Models

  • For retrieval-augmented language-model QA, the evaluation feeds only the top 100 or 500 retrieved tokens to LLaMA-2-7B and scores exact match.Four-shot demonstrations are used to align outputs with dataset formats.Source: §4.4 · pp. 4–5

5 How Does Granularity Influence Passage Retrieval?

Across five datasets, proposition-level indexing generally improves passage retrieval and downstream QA, with a particularly clear retrieval advantage on tasks unseen during retriever training. The reported gains are associated especially with queries about less common entities.

5 How Does Granularity Influence Passage Retrieval?

  • Proposition-level indexing gives on-par or superior passage retrieval despite all tested dense retrievers having been trained on passage-level documents.Across five datasets, proposition retrieval consistently outperforms sentence and passage retrieval on average for the tested retrievers.Source: §5 · p. 5, §5.1 · p. 5

5.1 Passage Retrieval Performance

  • Proposition retrieval improves average Recall@5 for unsupervised retrievers, while supervised retrievers show smaller average gains and comparable results on NQ, TQA, and WebQ.DPR’s proposition and sentence retrieval are slightly worse than passage retrieval on three of its four training datasets, with SQuAD the exception.Source: §5.1 · p. 5
  • On datasets unseen during retriever training, proposition retrieval has a clear advantage; DPR gains 25% relative Recall@5 on EntityQuestions.GTR’s proposition retrieval on SQuAD improves Recall@5 by 16% relative.Source: §5.1 · pp. 5–6

5.2 Retrieval on Finer-grained Index ⇒ Better Cross-Task Generalization

  • The proposition-index advantage over passages is larger for questions about less common entities and decreases as entity frequency rises.Entity frequency is estimated from occurrences in the top 1,000 BM25-retrieved passages; about 25% of test entities occur no more than three times.Source: §5.2 · p. 6, Figure 3 · p. 6

5.3 Higher Passage Recall ⇒Higher Downstream QA Accuracy

  • Proposition-level retrieval achieves the highest average exact match across all four retrievers, and usually the highest score across retrieval tasks and datasets.Downstream QA trends are highly consistent with passage-retrieval recall.Source: §5.3 · p. 7

6 How Does Granularity Influence Retrieval-Augmented LMs?

Under a fixed retrieved-token budget, proposition prompts generally yield higher QA performance than passage prompts. The paper attributes this pattern to a greater density of question-related information in finer-grained units.

6.1 Open-domain QA Performance

  • Proposition prompts improve EM@500 over passage prompts by +4.1 for SimCSE, +3.2 for Con-Retriever, +2.7 for DPR, and +2.8 for GTR.Across retrievers, propositions also yield higher average EM@l than other granularities.Source: §6.1 · p. 7
  • With the LLaMA-2-7B evaluation, using sentences and propositions rather than passages also improves performance on TQA and WebQ, though those datasets are not very sensitive to retrieval type.The reported highest improvements over the closed-book setting are +4.9 on TQA and +3.2 on WebQ, achieved by GTR with propositions.Source: §6.1 · p. 7

6.2 Finer-grained Granularity ⇒Higher Density of Question-Related Information

  • Finer-grained units are described as denser in query-relevant information, making the correct answer more likely to appear among the top-l retrieved words.The evaluated range is l = 0 to 500 words across five datasets.Source: §6.2 · p. 7
  • For a fixed word budget, proposition retrieval has a higher gold-answer recall than sentence or passage retrieval; the largest gain over passages occurs within 100–200 words.Recall across the three granularities converges as the word count increases.Source: §6.2 · p. 7

7 Related Work

Prior work addresses dense-retriever generalization through model and training changes, while this paper focuses on corpus indexing granularity at inference. It also situates proposition retrieval among proposition extraction and context-compression approaches.

7 Related Work

  • Dual-encoder retrievers can generalize poorly to new tasks with scarce training data, motivating augmentation, continued pretraining, task-aware training, and hybrid or mixed retrieval strategies.The related work also cites multi-vector retrieval as a way to increase model expressivity and retrieval granularity.Source: §7 · pp. 7–8
  • For GTR, finer-grained retrieval has higher gold-answer recall than coarser units across all tested limits on the number of retrieved-text words.The figure caption states the comparison but supplies no numeric values.Source: Figure 4 · p. 8
  • Unlike multi-vector methods that encode units into multiple vectors, this work changes corpus indexing granularity without updating the dense retriever or its parameters.The authors describe indexing granularity as an orthogonal inference-time strategy for improving generalization.Source: §7 · p. 8
  • Proposition extraction has roots in summarization evaluation and structured representations, with recent work extracting free-text propositions using few-shot LLM prompting or fine-tuned compact models.Generative summaries are described as complementary: they address document-level understanding, while propositions target queries about long-tailed entities.Source: §7 · p. 8
  • With limited context windows, filtering or compressing retrieved documents is one prior route to improve reader context; this work instead changes retrieval granularity to increase information density at fixed context length.Table 5 evaluates LLaMA-2-7B QA with four-shot prompts using passage, sentence, or proposition contexts capped at 100 or 500 tokens.Source: Table 5 · p. 8, §7 · p. 9

8 Conclusion

The paper examines corpus-index and prompt granularity for retrieval and downstream QA. Across five open-domain QA datasets, proposition retrieval outperforms sentence- and passage-level retrieval, with analyses pointing to cross-task generalization and denser question-related prompt information.

8 Conclusion

  • Across five open-domain QA datasets, proposition retrieval outperforms passage- and sentence-level retrieval on passage retrieval and downstream QA.The analyses associate finer-grained indexing with better cross-task generalization and greater question-related information density in prompts.Source: §8 · p. 9

Limitations

The study’s conclusions are bounded by its retrieval corpus, evaluated retriever types, and language. The authors identify broader corpora, model families, and languages as outside the current scope.

Limitations

  • The study uses Wikipedia as its retrieval corpus because most open-domain QA datasets adopt Wikipedia.The limitation concerns the corpus scope of the granularity study.Source: Limitations · p. 9
  • The evaluation covers six popular dense-retriever types, most of which use bi- or dual-encoder architectures; broader retriever coverage is left for future work.The study is also limited to English Wikipedia, with other languages left for future exploration.Source: Limitations · p. 9

Ethical Considerations

The appendices describe the study’s ethics statement, corpus processing, propositionizer training and quality assessment, indexing, model choices, and additional analyses.

Ethical Considerations

  • The authors state that the article follows the ACL Code of Ethics and report no obvious risks related to harmful effects, environmental impact, fairness, or privacy.Source: Ethical Considerations · p. 9

A Retrieval Corpus Processing

  • The October 2021 English Wikipedia dump was segmented into 100-word passages, sentences, and propositions, yielding 41 million passages, 114 million sentences, and 257 million propositions.A passage contains 6.3 propositions on average; a sentence contains 2.3.Source: A Retrieval Corpus Processing · p. 13

B Training the Propositionizer

  • The Propositionizer was fine-tuned on 42,857 GPT-4-generated proposition pairs; it scored 0.822 F1, and larger models with more training data performed better.The study compared Flan-T5 base- and large-size models across training-data amounts; its proposition-set F1 uses BertScore with roberta-large similarity.Source: B Training the Propositionizer · pp. 13–14, Figure 5 · p. 14

D Offline Indexing

  • The FACTOIDWIKI index is about 768GB, but the authors report that sharding, approximate search, GPU parallelism, and distributed indexes can make proposition retrieval viable.The corpus contains six times as many propositions as passages; the authors describe sub-linear search relative to the vector count with efficient indexing.Source: D Offline Indexing · p. 14

E Retriever Models and QA Models

F Additional Results

  • Entity frequency is estimated by occurrence in BM25’s top-1,000 paragraphs, so ambiguous surface forms can make frequent-looking entities hard to answer and lower recall.The authors note that this proxy yields some differences from results using hyperlink counts.Source: F Additional Results · p. 15
  • Across six dense retrievers, proposition recall consistently exceeds sentence and passage recall as context length varies, which the authors attribute to greater question-related density in proposition units.Source: F Additional Results · p. 15

G Error Case Study

The case study traces retrieval errors to entity and reference ambiguity in passages and missing antecedent context in sentences. It contrasts these with proposition retrieval’s fact-and-context packaging while noting a multi-hop boundary.

G Error Case Study

  • Passage retrieval is challenged by entity or reference ambiguity, including confusion between “Super Bowl 50” and “Super Bowl 5” and an irrelevant “atomic number” mention.The examples show top-1 retrieved units that fail to provide the correct answer.Source: G Error Case Study · p. 15, Table 6 · p. 16, Table 7 · p. 16
  • Sentence retrieval can also fail when the correct sentence uses “it” to refer to the pericardial sac.The case study identifies missing context as an additional sentence-level challenge.Source: G Error Case Study · p. 15
  • Propositions package one fact with necessary context, but may not link separate facts needed for multi-hop reasoning over long-range text.The actor-and-character example has no single proposition entailing both the question and answer.Source: G Error Case Study · p. 15
  • In Tables 6 and 7, gray proposition context is illustrative and is not provided to retrievers or downstream QA models.The tables mark correct answers with underlining.Source: Table 6 · p. 16, Table 7 · p. 16
  • Finer-grained retrieval has higher recall across all numbers of words considered.Figure 7 reports recall of the gold answer in retrieved text limited to the first k words.Source: Figure 7 · p. 17

Related papers

Loading 2312.06648v3…