Source-linked AI summary
Synthetic Test Collections for Retrieval Evaluation
Hossein A. Rahmani, Nick Craswell, Emine Yilmaz, Bhaskar Mitra, Daniel Campos
TL;DR
Building retrieval test collections is difficult because real query logs can be inaccessible and creating queries and relevance judgments is costly. This paper uses LLMs to generate synthetic queries and judgments, finding that fully synthetic collections can yield evaluation results similar to traditional collections.
Problem
Real query logs can be inaccessible, and manually creating queries and relevance judgments is costly, limiting the size of publicly available test collections.
Source: §1 · p. 1Method
The paper investigates fully synthetic test collections by generating queries and relevance judgments with LLMs.
Source: §1 · p. 1, §2.2 · p. 3Results
Fully synthetic test collections can produce retrieval-evaluation results similar to those from traditional test collections.
Source: §4 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Test collections play a vital role in evaluation of information retrieval (IR) systems. Obtaining a diverse set of user queries for test collection construction can be challenging, and acquiring relevance judgments, which indicate the appropriateness of retrieved documents to a query, is often costly and resource-intensive. Generating synthetic datasets using Large Language Models (LLMs) has recently gained significant attention in various applications. In IR, while previous work exploited the capabilities of LLMs to generate synthetic queries or documents to augment training data and improve the performance of ranking models, using LLMs for constructing synthetic test collections is relatively unexplored. Previous studies demonstrate that LLMs have the potential to generate synthetic relevance judgments for use in the evaluation of IR systems. In this paper, we comprehensively investigate whether it is possible to use LLMs to construct fully synthetic test collections by generating not only synthetic judgments but also synthetic queries. In particular, we analyse whether it is possible to construct reliable synthetic test collections and the potential risks of bias such test collections may exhibit towards LLM-based models. Our experiments indicate that using LLMs it is possible to construct synthetic test collections that can reliably be used for retrieval evaluation.
1 INTRODUCTION
Test collections are costly to build because real query logs are hard to access, manual query creation is labor-intensive, and relevance judgments require substantial human effort. The paper investigates whether LLMs can generate both queries and judgments for reliable synthetic collections, reporting evaluation results similar to traditional collections.
1 INTRODUCTION
- Real query logs are difficult to access, while manually creating queries and obtaining relevance judgments require substantial effort; these costs limit collection size and may reduce reliability.Manual queries also lack a guarantee of representing real information needs.Source: §1 · p. 1
- Prior IR studies used LLMs to generate synthetic training data for retriever augmentation, relevance labels [11] [18], and query variants; no prior study had thoroughly explored fully synthetic test collections.Source: §1 · p. 1
- LLM-generated queries and relevance judgments can form synthetic test collections with evaluation results similar to those from traditional test collections.The approach aims to avoid reliance on real query logs, manual query creation, and manual relevance judgments.Source: §1 · p. 1
2 SYNTHETIC TEST COLLECTION
The section tests synthetic queries and relevance judgments as building blocks for retrieval test collections. Fully synthetic collections preserve system ordering similarly to real queries with human judgments.
2.1 Synthetic Query Generation
- For passage retrieval, the authors generated queries from sampled passages using T5 and GPT-4, then had experts screen candidates for reasonableness.The passages were filtered before query generation, and one query was generated per passage with each method.Source: §2.1 · p. 2
- Real queries averaged 120.1 relevant documents per query, versus 71.76 for T5 and 77.87 for GPT-4 queries.T5-query pools also tended to contain more non-relevant documents than the other query types.Source: §2.1 · p. 2, Table 2 · p. 2
- Synthetic queries with human judgments yielded similar system rankings to real queries, with Kendall’s τ=0.8151.Source: §2.1 · p. 3
2.2 Synthetic Relevance Judgment Generation
- Assuming only the seed passages are relevant produced low system-ordering agreement with human-judged real queries (Kendall’s τ=0.157).This sparse-judgment result contrasts with the stronger agreement from synthetic queries judged by humans.Source: §2.2 · p. 3, Figure 2 · p. 3
- GPT-4/manual judgment agreement was Cohen’s κ=0.24 on real queries and κ=0.26 on synthetic queries.The reported agreement is on the full four-point relevance scale.Source: §2.2 · p. 3, Table 3 · p. 3
- On synthetic queries, GPT-4 consistently underestimated “Perfectly relevant” and “Highly relevant” labels, matching human “Perfectly relevant” judgments in 28% of cases.The authors suggest human assessors use distinctions not fully captured by an LLM.Source: §2.2 · p. 3
- Kendall’s τ=0.8568 for system ordering on fully synthetic collections versus real queries with human judgments.The collection combines synthetic queries and synthetic judgments.Source: Figure 2 · p. 3, §2.2 · p. 4
3 ANALYSIS OF BIAS IN SYSTEM EVALUATION
The analysis finds little clear evidence of systematic favoritism toward systems using the same language model as the synthetic-query generator. However, systems of all types perform higher on synthetic than real queries, suggesting the synthetic collections may be easier and overestimate performance.
3 ANALYSIS OF BIAS IN SYSTEM EVALUATION
- The collection using T5- and GPT-4-generated queries showed little to no bias toward LLM-based systems overall.Source: §3 · p. 4
- The bias analysis grouped TREC DL 2023 runs into GPT, T5, GPT+T5, and other system categories.Source: §3 · p. 4
- All system types consistently perform higher on synthetic collections than on real queries, suggesting synthetic collections are easier and overestimate performance.Source: §3 · p. 4
- The experiments found no clear systematic favoritism toward GPT-4 or T5 systems evaluated with queries generated by the same model.Source: §3 · p. 4
4 CONCLUSION AND FUTURE WORK
The paper concludes that fully synthetic test collections can produce evaluation results similar to those from traditional collections. Its preliminary bias analysis is encouraging, but further study is needed to understand possible biases and mitigation strategies.
4 CONCLUSION AND FUTURE WORK
- Fully synthetic collections containing generated queries and judgments can produce evaluation results similar to traditional test collections.Source: §4 · p. 4
- The preliminary analysis found little to no bias toward systems based on the particular LLM used to generate a synthetic collection.Source: §4 · p. 4
- Further research is needed to deepen understanding of biases in fully synthetic collections and develop mitigation strategies.Source: §4 · p. 4