InPars: Data Augmentation for Information Retrieval using Large Language Models

• Reranking 1,000 candidate documents with a billion-parameter model can cost approximately 15 USD per query, making direct large-model retrieval prohibitively expensive.Source: §1 · p. 1
• Dense retrieval lowers inference cost through precomputed document vectors, but billion-parameter models remain impractical because every collection document still requires an inference pass.Source: §1 · pp. 1–2
• Domain-specific IR training data is scarce, while general-purpose labeled datasets do not always generalize effectively to out-of-domain collections.Source: §1 · p. 2
• InPars uses large language models to generate labeled data in a few-shot manner, then finetunes retrieval models to rerank first-stage search results instead of using the large models directly.Source: §1 · p. 2
• In unsupervised settings, InPars largely outperforms recently proposed methods, while combining synthetic and supervised finetuning achieves state-of-the-art results on two of three transfer-learning datasets.Source: §1 · p. 2

• Data augmentation uses generated examples to improve data-driven models in low-resource settings, and large language models can produce synthetic data of reasonable quality for transfer learning.Source: §2 · p. 2
• Unsupervised dense retrievers can approach BM25 on some datasets by extracting text segments likely to be relevant and training on them as positive pairs.Source: §2 · p. 2
• Supervised sequence-to-sequence models have been used to generate synthetic question-text pairs from different collections for dense-retriever transfer learning.Source: §2 · p. 2
• InPars instead relies on simple prompts and only a few supervised examples to generate questions from large language models, following a zero-shot synthetic-generation paradigm.Source: §2 · p. 2

3 Our Method: InPars

• Given a document and a prefix of example question-document pairs, InPars uses a language model to generate a relevant question and treats the resulting pair as a positive training example.Source: §3 · pp. 2–3
• Thousands of synthetic examples can be generated from randomly sampled documents using only a small fixed set of manually annotated examples; experiments use three examples.Source: §3 · p. 3
• InPars selects the top K generated pairs by autoregressive token probability before finetuning, and this filtering step largely improves IR metrics.Source: §3 · p. 3
• Few-shot generation allows InPars to adapt retrievers to different collections or IR tasks where labeled data is expensive and often limited.Source: §3 · p. 3
• InPars requires neither target-corpus pretraining nor modifications to the loss function, making it suitable for non-neural retrieval algorithms as well.Source: §3 · p. 3

4.1 Datasets

• The evaluation covers MS MARCO, TREC-DL, Robust04, Natural Questions, and TREC-COVID, spanning passage ranking, news, Wikipedia, and COVID-19 scientific-literature collections.Source: §4.1 · p. 3

4.2 Training Data Generation

• 100,000 documents are sampled and one question is generated per document with GPT-3 Curie, retaining the top K=10,000 query-document pairs as positives.Documents shorter than 300 characters are discarded, and greedy decoding is used.Source: §4.2 · pp. 3, 5
• The GBQ prompt uses MS MARCO examples of bad and manually created good questions to encourage more context-aware queries than the Vanilla prompt.Vanilla uses three document-question examples, while GBQ retains only the generated good question for each document.Source: §4.2 · p. 4
• Negative examples are formed by randomly selecting one BM25-retrieved document from the top 1000 candidates for each generated query.Source: §4.2 · p. 5

4.3 Retrieval Methods

• Retrieval uses BM25 to obtain 1000 candidates, followed by a monoT5 neural reranker trained with balanced positive and negative examples.The monoT5 base and 3B models are trained for 156 steps, and results are averaged over three seeds.Source: §4.3 · p. 5
• Models are trained per collection, with in-domain synthetic data expected to help while MS MARCO documents still provide reasonable results.For Robust04, documents are split into overlapping passages and scored with a MaxP-style aggregation because full documents are too long for the Transformer.Source: §4.3 · p. 5
• The OpenAI Search API reranks only 100 BM25 candidates because reranking more documents is expensive, although Curie improved TREC-DL 2020 nDCG@10 from 0.5422 to 0.5602 with 1000 candidates.The API cost is approximately 5.85 USD per 1000 query-document pairs using Davinci.Source: §4.3 · pp. 5–6

Main results

• InPars unsupervised models outperform equivalent-size OpenAI Search API models, Contriever, and cpt-text, with monoT5-3B surpassing the larger Curie and Davinci models.Source: §5 · p. 6, Table 2 · p. 7
• Combining supervised MS MARCO training with in-domain synthetic data achieves state-of-the-art results on Robust04 and TREC-COVID and surpasses MS MARCO-only zero-shot transfer.The gain is especially noteworthy on TREC-COVID, whose subject was relatively recent and unknown to many pretrained language models.Source: §5 · pp. 6–7
• Further finetuning on 10,000 synthetic examples slightly decreases effectiveness on MS MARCO and TREC-DL after training on 530,000 manually annotated examples.The authors hypothesize that the synthetic examples differ from the original MS MARCO distribution.Source: §5 · pp. 6–7

6.1 Prompt Selection and Source Corpus

• GBQ questions were more descriptive and document-specific, while Vanilla questions were more generic and answerable by broader sets of documents.Generating 100,000 Curie questions cost 100-300 USD regardless of prompt type.Source: §6.1 · p. 7
• Vanilla prompts achieved the highest scores on MS MARCO and TREC-DL 2020, whereas GBQ prompts with in-domain documents worked best on Robust04, NQ, and TREC-COVID.The 3B models benefited most from in-domain examples, while monoT5 220M had no clear winner across prompt and source-corpus choices.Source: §6.1 · pp. 7–8

6.2 Model Size Impact on IR Metrics

• Larger GPT-3 models steadily increased MS MARCO MRR@10, although the gains were very slow.The authors hypothesize that larger models generate more specific questions, reducing false-negative examples caused by questions relevant to many passages.Source: §6.2 · p. 8, Figure 3 · p. 8
• OpenAI Search results showed the same general relation between GPT-3 size and IR effectiveness, although Curie outperformed Davinci in most experiments.Davinci rivaled Curie as the best-performing model in those results.Source: §6.2 · p. 8

6.3 Filtering by the Most Likely Questions

• Using only the top K = 10, 000 synthetic pairs improved MS MARCO MRR@10 by 4 points over finetuning on all 100,000 examples.The filtering approach was applied to all other datasets to avoid relying on test sets and preserve the fully unsupervised setup.Source: §6.3 · p. 8

6.4 Was GPT-3 Trained on Supervised IR Data?

• Only 5,285 of 93,200 unique Davinci questions matched the MS MARCO question set, or less than 5.7%.Among questions generated from 6,396 MS MARCO-relevant documents, 725 matched MS MARCO ground truth, or less than 12%; the authors treat these low overlaps as evidence against MS MARCO finetuning or memorization.Source: §6.4 · p. 8

7 Conclusion and Future Work

• InPars generates synthetic IR training data with large language models in a few-shot manner, using their learned information more efficiently and effectively.Source: §7 · p. 8
• Future work includes finetuning dense retrievers, using bad questions as negatives, scaling datasets to millions of examples, and improving pair selection.Source: §7 · p. 8

• Table 3 provides Robust04 examples generated with Vanilla and GBQ prompts.Source: Table 3 · p. 13
• Table 4 provides TREC-COVID examples generated with generated good and bad questions.Source: Table 4 · p. 14
• Table 5 provides Natural Questions examples generated with generated good and bad questions.Source: Table 5 · p. 15