PaperlayerView exact revision on arXiv

Source-linked AI summary

BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives

Xiaoyue Wang, Jianyou Wang, Weili Cao, Kaicheng Wang, Ramamohan Paturi, Leon Bergen

arXiv:2402.14151v2cs.IRcs.AIcs.CLcs.LG

TL;DR

Similarity-based retrieval does not capture multi-faceted user objectives, motivating an evaluation benchmark for complex-objective IR. BIRCO assembles five diverse tasks, and LLM ranking methods perform better than embeddings and small language models but none performs strongly across all tasks.

  • Problem

    Traditional similarity-focused retrieval does not capture multi-faceted objectives such as finding papers that refute a claim.

    Source: §1 · p. 1

  • Method

    BIRCO benchmarks complex-objective retrieval across five diverse tasks and restricts LLM searches to smaller candidate pools for evaluation.

    Source: §3.8 · p. 5, §6 · p. 8

  • Results

    LLM ranking methods outperform embedding methods and small language models on BIRCO, but none achieves strong results across all tasks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The benchmark findings leave room for stronger retrieval performance across complex-objective tasks.

    Source: §6 · p. 8

    LLMs are computationally expensive, can only be effectively used for passage reranking, and may remain prohibitively costly even with limited candidate pools.

    Source: §7 · p. 9

Abstract

from arXiv · show

We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.

1 Introduction

Complex user objectives can require retrieval beyond semantic similarity, such as finding papers that refute a claim. BIRCO benchmarks such retrieval for LLM-based systems while aiming to limit evaluation cost and prevent answers without corpus inspection.

1 Introduction

  • Similarity-driven search can miss objectives such as finding papers that refute a scientific claim, because it may also retrieve papers supporting that claim.Source: §1 · p. 1
  • BIRCO evaluates information retrieval systems on five open-source datasets with paragraph-length queries and multi-faceted objectives.The datasets form a challenging test bed for methods addressing complex user search needs.Source: §1 · p. 1
  • LLMs may answer retrieval queries from pretraining without examining documents, while evaluating thousands of documents per query can be prohibitively expensive.Source: §1 · p. 1
  • BIRCO combines complex objectives and aggressive filtering to require corpus inspection, while its 50–100 documents per query keep LLM evaluation low-cost.Source: §1 · p. 1

2 Related Work

Prior IR benchmarks mostly use sentence-level queries focused on semantic similarity, while several newer datasets pose more complex, multi-dimensional retrieval tasks. The related work also identifies a lack of documented IR benchmarks designed specifically for LLM-based systems.

IR Benchmarks

  • Most IR benchmarks use sentence-level queries focused on semantic similarity, with ArguAna identified as an exception for counterargument retrieval.Source: §2 · pp. 1–2

Complex Query IR Tasks

  • Several newer datasets pose complex retrieval tasks with paragraph-length queries and passages that must match along multiple dimensions.Figure 1 provides an overview of these datasets.Source: §2 · p. 2, Figure 1 · p. 2

LLM-based IR systems

  • LLM-based reranking methods use relevance logits, pairwise or list-wise comparisons, or four-way labels, and have primarily been evaluated on sentence-level queries.Source: §2 · p. 2

Benchmarks for LLMs

  • The related work reports no released or documented IR benchmark designed to evaluate LLM-based IR systems.Source: §2 · p. 2

3 A Benchmark for LLM-based IR Systems

BIRCO is a compact benchmark for evaluating LLM-based retrieval on complex, multi-faceted objectives. Its design addresses contamination and evaluation cost while testing systems across diverse, difficult tasks.

3 A Benchmark for LLM-based IR Systems

  • BIRCO targets LLM-based IR with longer, multi-faceted queries, complex objectives, reduced contamination, increased difficulty, and smaller candidate pools.These features distinguish it from traditional IR benchmarks.Source: §3 · p. 2

3.3.1 Measuring contamination

  • On 10 BEIR datasets, GPT-4 answered many queries without candidate documents; answers were compared with ground truth and binned into three categories.The authors use this procedure to assess whether retrieval benchmarks measure query-document comparison rather than answering from prior knowledge.Source: §3.3.1 · pp. 3–4
  • Annotators exactly agreed on 91% of labels, with Spearman correlation 0.81 and macro F1-score 0.76.Agreement was measured on a separate set of 100 queries from the 10 BEIR datasets.Source: §3.3.1 · p. 4

3.3.2 Reducing contamination

  • BIRCO filters queries GPT-4 could potentially answer without candidate passages, removing 7% from Clinical-Trial, 8% from RELIC, and 56% from WhatsThatBook.No queries were removed from DORIS-MAE or ArguAna; the passage attributes WhatsThatBook’s higher filtering rate to GPT-4’s extensive knowledge of literary novels.Source: §3.3.2 · p. 4

3.4 Query Complexity

  • Queries average between 2 and 11 facets, requiring systems to assess documents across multiple dimensions of relevance.The facets are identified by prompting GPT-4 to break queries into distinct components.Source: §3.4 · p. 5

3.5 Task Complexity

  • BIRCO has low query–relevant-passage lexical overlap and lower reported IR performance than other benchmarks; ArguAna and Clinical-Trial include 22% and 24% hard negatives, respectively.The passage cautions that metrics across datasets are difficult to compare directly because they depend on relevant-passage counts and relevance scales.Source: §3.5 · p. 5, §3.7 · p. 5

3.1 Dataset Descriptions

  • Five datasets span debate, computer science, biomedicine, and literature, with low word overlap across datasets; RELIC and WhatsThatBook are most similar.The datasets cover varied retrieval tasks, including counterargument retrieval, patient-to-trial matching, and matching literary analyses to quotations.Source: §3.1 · pp. 2–3, §3.6 · p. 5

3.2 BIRCO Evaluation Methodology

  • To control evaluation cost, each query receives 50 BM25 and ada-002 candidates, while dataset splits avoid query or passage overlap.The ground-truth passage is inserted when necessary.Source: §3.2 · p. 3, §3.8 · p. 5

4 A Framework for LLM-based Retrieval

The paper presents a modular framework for evaluating LLM-based retrieval choices. It compares ranking and scoring, reasoning and task decomposition, and prompts that do or do not specify task objectives.

4 A Framework for LLM-based Retrieval

  • The framework is modular, allowing investigation of how different factors affect LLM retrieval performance.The paper refers readers to Appendix G for an illustration.Source: §4 · pp. 5–6

4.1 Ranking vs. Scoring

  • LLMs can rank passages comparatively with RankGPT4’s sliding-window filtering or score passages individually on a 0–10 scale.Direct scoring can produce ties; E5-Large-v2 is the default tie-breaker.Source: §4.1 · p. 6

4.2 Chain-of-Thought Reasoning

  • Reason+O has the LLM generate and follow decision criteria before scoring, whereas Score+O directly scores the query-passage pair without reasoning.The methods are compared to investigate natural-language reasoning in complex-query retrieval.Source: §4.2 · p. 6

4.3 Task Decomposition

  • Subtask+O independently scores subtasks defined by LLM-generated criteria, then averages those scores to reduce relevance-evaluation complexity.The subtasks are derived from the decision criteria.Source: §4.3 · p. 6

4.4 Task Objective Awareness

  • Objective-aware prompts include the task description, while the simpler baseline asks for semantically similar passages without that objective information.Prompts that include task objectives are suffixed with “O.”Source: §4.4 · p. 6

5 LLM Benchmark Results

GPT-4 achieves the strongest results, but small embedding and fine-tuned language-model baselines remain strong relative to other LLMs. Across retrieval strategies, task decomposition and chain-of-thought reasoning do not yield consistent gains; detailed task objectives help GPT-4 on selected datasets.

5 LLM Benchmark Results

  • GPT-4 achieves the strongest results across retrieval strategies, and only GPT-4-based systems consistently outperform RankLLaMA and MonoT5.Small embedding models and fine-tuned language models provide strong baselines relative to all LLMs except GPT-4.Source: Table 5 · p. 7, §5 · p. 8
  • Comparative ranking versus direct scoring has no general performance effect, except that Llama 70B performs better with direct scoring.Source: §5 · p. 8
  • Chain-of-thought helps some task-model combinations, but its effects are inconsistent and slightly decrease retrieval performance in aggregate.Source: §5 · p. 8
  • Task decomposition does not consistently improve performance; GPT-4 with decomposition has the strongest average, largely driven by ArguAna.The model decomposes queries into subtasks and evaluates each separately.Source: §5 · p. 8
  • Detailed task-objective descriptions do not help smaller LLMs, but improve GPT-4 with comparative ranking on ArguAna and RELIC.Without objectives, GPT-4 direct scoring performed poorly on development sets, with qualitative evidence pointing to difficulty determining relevance; these datasets have non-standard objectives.Source: §5 · p. 8

5.1 Error Analysis

  • GPT-4 ranks hard negatives higher than other passages on average, indicating that these similar but non-relevant documents increase retrieval difficulty.Table 6 reports average reciprocal rank for hard negatives.Source: §5.1 · p. 8, Table 6 · p. 8

6 Conclusion

BIRCO benchmarks five diverse, complex-objective retrieval tasks and is more challenging than previous IR benchmarks. LLM ranking methods outperform embedding and small language-model methods, but no approach performs strongly across all tasks.

6 Conclusion

  • BIRCO comprises five diverse retrieval tasks with complex objectives and is significantly more challenging than previous IR benchmarks.Source: §6 · p. 8
  • Embedding methods and small language models perform weakly on BIRCO, while LLM ranking methods perform better but none are strong across all tasks.Source: §6 · p. 8
  • Clear task-objective instructions improve performance on certain tasks, comparative ranking has no demonstrated general advantage over direct scoring, and query decomposition helps on only one dataset.The authors also found no evidence that chain-of-thought reasoning improves retrieval performance.Source: §6 · pp. 8–9
  • Strong BIRCO performance requires models that understand multifaceted user intents, while GPT-4-based methods still fall short across tasks and are costly beyond the smallest IR tasks.Source: §6 · p. 9

7 Limitations

BIRCO covers only a limited sample of retrieval needs, and LLM inference can be costly beyond small IR tasks. Better prompts may also improve performance despite the authors’ development-set optimization.

7 Limitations

  • BIRCO covers only a limited sample of the many types of user retrieval needs.Source: §7 · p. 9
  • LLM inference is prohibitively expensive for all but the smallest IR tasks, and LLMs are effectively limited to passage reranking in a multi-stage process.Depending on the use case, inference may remain prohibitively expensive even with a limited candidate pool.Source: §7 · p. 9
  • Better prompts may improve LLM performance, although prompts were optimized on the development sets.Source: §7 · p. 9

8 Ethical Considerations

The paper reports no ethical concerns and details its evaluation cost, access methods, and model baselines. The baselines span embedding, fine-tuned retrieval, and generative models.

8 Ethical Considerations

  • No ethical concerns are reported for this work.Source: §8 · p. 9

A Experiment Details

  • GPT4-based systems take under 12 hours and cost about $500–$1,000 to evaluate, while other LLM systems take under 3 hours and cost under $50.Source: A Experiment Details · p. 13

A.1 Baselines

  • The baselines include embedding models, fine-tuned retrieval models, and generative models such as Llama2 chat and StripedHyena-Nous-7B chat.The listed systems also include monoT5-3B, TART-1.5B, and RankLLaMA-7B.Source: §A.1 · p. 13

A.2.1 Analysis for Non-LLM models

Among non-LLM models, performance varies by model family and synthetic-data training, while GPT4-based systems outperform all embedding models. The accompanying task-objective prompts were developed using small dataset-specific development sets.

A.2.1 Analysis for Non-LLM models

  • All embedding models perform significantly worse than GPT4-based models; Promptagator appears comparable to RankLLaMA on three datasets.TART and MonoT5 are comparable to the best pretrained embedding models on average; TART+O drops significantly relative to TART.Source: §A.2.1 · p. 13

B Task Objectives for BIRCO

B.1 Task Objective for DORIS-MAE

  • DORIS-MAE queries are paragraph-spanning research questions, and candidate abstracts are retrieved for how well they meet the stated needs.Source: §B.1 · p. 13

B.2 Task Objective for ArguAna

These passages specify the objectives for four BIRCO tasks and describe how candidate pools, test sets, and development triplets were constructed. Candidate-pool size and query selection affect task difficulty.

B.2 Task Objective for ArguAna

  • ArguAna retrieves an argument that directly opposes the query’s stance on the same topic and point.The task excludes citations from consideration and seeks the debate topic, query stance, and directly opposing passage.Source: §B.2 · pp. 13, 15

B.3 Task Objective for WhatsThatBook

  • WhatsThatBook retrieves the book description that best matches the remembered details, including details expressed informally or implicitly.Source: §B.3 · p. 15

B.4 Task Objective for RELIC

  • RELIC retrieves a passage that directly supports a claim around a masked quotation and fits naturally without repeating the query.Source: §B.4 · p. 15

B.5 Task Objective for Clinical-Trial

  • Clinical-Trial matches patient case reports to trials for recruiting eligible patients.The task’s stated motivation is that recruiting trial candidates is time-consuming and resource-intensive.Source: §B.5 · pp. 15–16

C The Effects of Candidate Pool Construction

  • For ArguAna, WhatsThatBook, and RELIC, the test candidate pools contain 50 items; the test sets also use selected decontaminated queries or original dataset splits.The remaining nine Clinical-Trial queries are used for development.Source: C The Effects of Candidate Pool… · p. 16
  • E5 performance is higher on RELIC and WhatsThatBook test sets because their candidate pools are smaller; ArguAna’s filtered test queries are longer and potentially harder.Clinical-Trial’s test set is competitive with its complete set, which includes nine additional queries, four potentially contaminated.Source: C The Effects of Candidate Pool… · p. 16

D Development Set

  • Development sets use non-overlapping query-passage triplets, with fewer than 100 triplets per dataset used in practice for prompt design.Triplets pair a query with a relevant and a less relevant passage; Clinical-Trial uses its three-level relevance ordering, while DORIS-MAE uses score-separated pairs.Source: D Development Set · pp. 16–17

E Analysis of the Comparison between Datasets

The comparison analysis qualifies benchmark metrics and reports lower average E5-v2 performance on BIRCO than BEIR, with RELIC particularly challenging. The following passages describe facet measurement, scoring approaches, and error patterns.

E Analysis of the Comparison between Datasets

  • SimLM reaches 81.9% R@20 on NQ versus 49.6% on BIRCO, which the authors treat as evidence BIRCO is harder for that model.They note that BIRCO’s maximum possible R@20 is about 94% and caution that metrics are influenced by relevant-passage counts and relevance scales.Source: E Analysis of the Comparison between… · p. 17
  • 38.6% average nDCG on BIRCO is significantly below BEIR’s 50.0% average nDCG for E5-v2.The paper says BIRCO’s average is slightly elevated by its smaller candidate pool and inclusion of DORIS-MAE and Clinical-Trial.Source: E Analysis of the Comparison between… · p. 17
  • 11.2% nDCG@10 makes RELIC E5-v2’s most challenging task despite its smaller candidate pool.The passage presents same-R/D comparisons as a way to compare dataset nDCG@10.Source: E Analysis of the Comparison between… · p. 17

F Definition of Facets in BIRCO Queries

G Scoring-based Framework Illustration

  • Each query uses one shared set of decision criteria across its candidate passages; passage-conditioned criteria are not investigated.The passage describes the alternative as potentially more specific but more variable.Source: G Scoring-based Framework Illustration · p. 18

H Error Analysis for Models

  • In Reason+O, low-quality decision criteria typically accompany low performance on a query.Source: H Error Analysis for Models · p. 18
  • Subtask+OGPT performs better or comparably to Reason+OGPT; the reported ArguAna improvement is significant.The passage associates decomposition with greater stability and, for certain complex tasks, more accurate scoring and annotation.Source: H Error Analysis for Models · p. 18

H.1 Comparison vs Scoring

  • Comparison-based and scoring-based approaches both improve over previous models; comparison lacks interpretable reasoning, while scoring decomposed subtasks offers greater stability and natural-language reasoning.The comparison approach is comparable on DORIS-MAE and WhatsThatBook and slightly better on RELIC.Source: §H.1 · p. 18

H.2 Features that impact Task Difficulty

The appendix examines query-feature associations with retrieval difficulty, describes GPT-4 contamination filtering, and provides dataset licenses, query–passage examples, and prompting templates.

H.2 Features that impact Task Difficulty

  • Task difficulty was each query’s average nDCG@10 across Score+O, Reason+O, and Subtask+O; tested features included query length, facet count, lexical overlap, and candidate-pool size.Source: §H.2 · p. 18
  • <15% of variance was explained; relevant-passage lexical overlap correlated positively with nDCG for ArguAna (p<0.005) and WhatsThatBook (p<0.0005), while all-passage overlap correlated negatively for RELIC (p<0.0005).Source: §H.2 · p. 18

I Decontamination Filtering Procedure for BIRCO

  • Decontamination used human semantic checks for DORIS-MAE and ArguAna, title-guess and corpus filtering for WhatsThatBook, treatment-similarity checks for Clinical-Trial, and 5-gram filtering for RELIC.WhatsThatBook filtering removed 38 queries, 4,817 descriptions, and 238 additional queries; Clinical-Trial removed 4 queries, and RELIC removed 311.Source: I Decontamination Filtering Procedure… · pp. 18–19

J Dataset License

  • The five datasets use CC-BY-NC 4.0 for DORIS-MAE, CC-BY-SA 4.0 for ArguAna and Clinical-Trial, ODC-BY for WhatsThatBook, and MIT for RELIC.Source: J Dataset License · p. 19

K Example (Query, Passage) Pairs from the Dataset

L All Prompting Strategies

L.5 Prompts for Decontamination

  • Separate templates are provided for decontamination in BIRCO and BEIR.Source: §L.5 · p. 19

Related papers

Loading 2402.14151v2…