Source-linked AI summary
InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval
Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, Rodrigo Nogueira
TL;DR
Information retrieval methods need synthetic training data when high-quality in-domain supervision is scarce; InPars [1] and Promptagator [2] use LLMs to generate queries. InPars-v2 uses GPT-J to generate synthetic queries and monoT5-3B to select the highest-scoring query-document pairs for training. A BM25 pipeline followed by a monoT5 reranker finetuned on InPars-v2 data achieves a new state of the art on the average of BEIR datasets.
Abstract
from arXiv · showhide
Recently, InPars introduced a method to efficiently use large language models (LLMs) in information retrieval tasks: via few-shot examples, an LLM is induced to generate relevant queries for documents. These synthetic query-document pairs can then be used to train a retriever. However, InPars and, more recently, Promptagator, rely on proprietary LLMs such as GPT-3 and FLAN to generate such datasets. In this work we introduce InPars-v2, a dataset generator that uses open-source LLMs and existing powerful rerankers to select synthetic query-document pairs for training. A simple BM25 retrieval pipeline followed by a monoT5 reranker finetuned on InPars-v2 data achieves new state-of-the-art results on the BEIR benchmark. To allow researchers to further improve our method, we open source the code, synthetic data, and finetuned models: https://github.com/zetaalphavector/inPars/tree/master/tpu
1 Introduction and Background
InPars-v2 addresses scarce in-domain retrieval data by extending LLM-based synthetic query generation with reranker filtering and an open-source query generator.
- Data augmentation addresses the scarcity of high-quality in-domain training data in practical AI applications.Source: §1 · p. 1
- InPars [1] and Promptagator [2] generate alternative document queries with LLM prompts for synthetic retrieval training data.Source: §1 · p. 1
- InPars-v2 selects synthetic query-document pairs with a reranker and improves retrieval effectiveness on BEIR.Source: §1 · pp. 1–2
- Unlike InPars-v1 [1], InPars-v2 uses an open-source query generator and releases code and data for TPU reproduction.Source: §1 · p. 2
2 Methodology
InPars-v2 generates synthetic queries with GPT-J, filters them using monoT5 relevance scores, constructs positive and negative pairs, and evaluates finetuned rerankers after BM25 retrieval.
Synthetic query generation
- GPT-J generates one query per sampled BEIR document using three MS MARCO examples and the InPars-v1 “gbq” prompt.The method samples up to 100k documents per corpus and uses greedy decoding.Source: §2 · p. 2
Pair filtering
- monoT5-3B scores 100k synthetic query-document pairs, retaining the top 10k as positive training pairs instead of using InPars-v1 log probabilities.Scoring takes approximately 1.5 hours on a TPU v3-8.Source: §2 · p. 2
Training data construction
- Negative pairs are formed by randomly sampling a document from the top 1000 BM25 results for each synthetic query.The resulting training set contains 10k positive and 10k negative pairs.Source: §2 · p. 2
Reranker training
- Each dataset-specific monoT5-3B reranker is finetuned first on MS MARCO and then for one epoch on its synthetic dataset.The procedure produces 18 rerankers, one for each BEIR dataset.Source: §2 · p. 2
Evaluation pipeline
- Evaluation retrieves 1,000 documents with BM25 and then reranks them using the corresponding finetuned monoT5-3B model.Source: §2 · p. 2
3 Results
InPars-v2 achieves a new average state of the art on BEIR, while Promptagator and RankT5 remain stronger on argument-retrieval datasets such as Touche and ArguAna.
- The InPars-v2 pipeline achieves a new state of the art on the average of all BEIR datasets.Table 1 compares BM25, monoT5-3B, InPars-v1, InPars-v2, Promptagator, and RankT5 using nDCG@10.Source: §3 · p. 2, Table 1 · p. 3
- InPars-v2 is substantially better than InPars-v1 on TREC-News, Climate-FEVER, Robust, and Touche.Source: §3 · p. 2
- Promptagator and RankT5 outperform BM25 on Touche and ArguAna, where monoT5 and InPars-v2 do not surpass BM25.These datasets focus on argument retrieval and benefit from custom prompts.Source: §3 · p. 2
- Promptagator uses smaller 110M-parameter T5 models and no supervised MS MARCO data for retrieval and reranking.Source: §3 · p. 2
- The paper releases its code, models, and data, while noting that fuller BEIR results were planned for an upcoming paper.Source: §3 · pp. 2–3
4 Conclusion
InPars-v2 improves InPars with a publicly available language model and better query-document pair selection, achieving effectiveness on par with the state of the art on BEIR.