PaperlayerView exact revision on arXiv

Source-linked AI summary

BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, Tao Yu

arXiv:2407.12883v4cs.CLcs.AIcs.IR

TL;DR

Existing retrieval benchmarks often support surface-level matching, leaving complex queries that require reasoning to identify relevant documents underrepresented. BRIGHT addresses this gap with a diverse benchmark of naturally occurring and curated queries, and its evaluation finds poor existing retrieval performance while showing benefits from explicit query reasoning and retrieved-document augmentation for question answering.

  • Problem

    Existing retrieval benchmarks often use queries where keyword or semantic matching is sufficient, providing limited coverage of real-world queries requiring intensive reasoning to identify relevant documents.

  • Method

    BRIGHT constructs a text-retrieval benchmark from diverse naturally occurring and curated data, including StackExchange, coding, and theorem-based query-document tasks.

  • Results

    Existing retrieval models perform poorly on BRIGHT, while LLM-generated reasoning steps improve retrieval by up to 12.2 points and retrieved documents improve downstream question answering.

  • Takeaways & Limitations

    BRIGHT provides a realistic evaluation setting for advancing reasoning-intensive retrieval and improving downstream reasoning-task performance.

  • Takeaways & Limitations

    Relevance judgments are subjective, although the benchmark retains only cases on which multiple annotators, including domain experts, agree.

Abstract

from arXiv · show

Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.

1 INTRODUCTION

BRIGHT targets retrieval settings where identifying relevant documents requires intensive reasoning beyond keyword or semantic matching. Across diverse, naturally occurring data, existing models perform poorly, while explicit query reasoning improves retrieval and downstream question answering.

  • Motivation: BRIGHT introduces level 3 retrieval, where query-document relevance requires intensive reasoning rather than straightforward keyword or semantic matching.The benchmark targets complex real-world cases such as matching economic theories or locating syntax documentation from error messages.
  • Benchmark: BRIGHT contains 12 datasets spanning diverse advanced domains and naturally occurring or curated human data.Its sources include StackExchange, blogs, syntax documentation, and STEM problem-solutions.
  • Results: Up to 12.2 points of average retrieval improvement result from generating explicit chain-of-thought reasoning about queries before retrieval.Reasoning-enhanced queries provide one effective strategy for improving performance on BRIGHT.
  • Results: Further training on benchmark documents produces no substantial performance gains, indicating robustness against potential large-scale pre-training data leakage.The authors position this finding as support for future research on reasoning-intensive retrieval.

2 RELATED WORK

Prior retrieval benchmarks cover information-seeking, instruction-following, multi-hop, long-context, and diverse-domain settings, but generally rely on query-document semantic overlap. Related reasoning benchmarks instead evaluate model reasoning in mathematics and coding rather than reasoning-intensive retrieval.

  • Benchmarking retrieval: Existing retrieval datasets commonly target question answering, claim verification, entity retrieval, instruction following, multi-hop reasoning, or long-context retrieval.Comprehensive benchmarks such as BEIR evaluate diverse domains and tasks.
  • Benchmarking retrieval: BEIR and related benchmarks generally use relevant documents with high semantic overlap with the query.BIRCO addresses multifaceted objectives but is limited to LLM reranking and approximately 100 candidates per query.
  • Dense retrieval models and retrieval augmented generation: Dense retrieval systems encode text with rich representations and are trained on unsupervised, supervised, and LLM-generated data.The paper benchmarks sparse and dense, small and large, and open-source and proprietary models.
  • Benchmarking reasoning: Mathematics and coding benchmarks such as GSM8K, MATH, HumanEval, and MBPP primarily evaluate reasoning abilities rather than retrieval.These benchmarks provide context for distinguishing reasoning evaluation from reasoning-intensive document retrieval.

3 CONSTRUCTING BRIGHT

BRIGHT constructs a reasoning-intensive retrieval benchmark from diverse real-world and curated data, with relevance grounded in concepts needed to solve queries rather than surface similarity. Its datasets cover StackExchange web-page retrieval, coding documentation and algorithmic similarity, and theorem-based problem retrieval.

  • 3.1 TASK FORMULATION: BRIGHT defines reasoning-intensive retrieval as matching queries with documents connected through reasoning traces such as principles, algorithms, or theorems.These traces may involve identifying intent, modeling the problem, and drawing sub-conclusions, and are often absent from the query itself.
  • 3 CONSTRUCTING BRIGHT: The benchmark uses 12 datasets spanning seven StackExchange domains, coding tasks, and theorem-based mathematics and physics problems.StackExchange domains include Sustainable Living, Economics, Psychology, Robotics, Earth Science, Biology, and Stack Overflow.
  • 3.2 STACKEXCHANGE: RETRIEVING WEB PAGES THAT HELP ANSWER QUESTIONS: StackExchange queries combine post titles and content, while positive passages come from linked web pages that help address the question and are retained only after relevance review.Selected posts require an accepted or highly voted answer containing URLs; linked pages are split into passages and irrelevant pages are discarded.
  • 3.2 STACKEXCHANGE: RETRIEVING WEB PAGES THAT HELP ANSWER QUESTIONS: StackExchange negatives include semantically similar Google results and other passages that do not satisfy the query’s specific requirements.Annotators search using post titles or summarized keywords, collect up to five negative web pages per query, and use all non-positive corpus passages as negatives.
  • 3.3 CODING: RETRIEVING DOCUMENTATION OR SIMILAR SOLVED PROBLEMS: Coding datasets ground relevance in syntax usage or algorithmic design, using a Pony manual for documentation retrieval and LeetCode problems with solutions for algorithmic similarity.Pony queries are code-generation problems paired with the full language manual, while LeetCode positives are annotated similar problems and the corpus also includes CodeSearchNet Python code.
  • 3.4 THEOREM-BASED QUESTIONS: RETRIEVE SOLVED PROBLEMS USING THE SAME THEOREMS OR RELEVANT THEOREM STATEMENTS: Theorem-based datasets treat solved problems or theorem documents as positive when they share the theorem or problem-solving skill required by the query’s solution.The corpus includes solved STEM problems and theorem statements from collections such as ProofWiki.

4 EXPERIMENTS

Experiments show that existing retrieval models struggle on BRIGHT, while reasoning-enhanced queries improve retrieval and stronger retrieved evidence improves downstream question answering.

  • Retrieval-augmented question answering: Stronger retrieval generally produces better question-answering results, although QA outcomes may not fully capture retrieval quality.The generator may fail to integrate retrieved documents, and the evaluator may miss similarities or differences between open-ended answers.
  • Main retrieval results: 24.3 nDCG@10 is the best performance achieved by existing retrieval models on BRIGHT, indicating that the benchmark remains highly challenging.BM25 matches models below 1B parameters but substantially trails larger models, while even stronger models remain unable to solve reasoning-intensive retrieval.
  • Reasoning-enhanced retrieval: LLM-generated reasoning steps significantly improve retrieval compared with original queries, especially for Llama3-70B and GPT-4.GritLM-generated steps improve BM25 but are less effective for other models; the best score remains below 30.
  • Reasoning-enhanced retrieval: BM25 achieves the strongest leaderboard performance when GPT-4 reasoning steps are used as new queries.The comparison uses average nDCG@10 on BRIGHT across original queries and reasoning-step queries.
  • Retrieval-augmented question answering: 1.9 points is the gain achieved by the top-performing retriever, Qwen, in end-to-end question answering, compared with the relevant baseline.Oracle documents yield a 4.1-point gain, showing additional room for retrieval improvements.

5 ANALYSIS

Further analyses test reranking, pre-training leakage, and long-context retrieval, finding persistent difficulty and limited benefit from in-domain exposure.

  • Reranking: Reranking experiments compare a MiniLM cross-encoder and LLM rerankers over the top 10 or 100 retrieved documents against unre-ranked retrieval.The cross-encoder scores query-document pairs, while LLM rerankers are also evaluated on the retrieved candidates.
  • Robustness against data leakage from pretraining: BRIGHT remains robust to data leakage when GritLM is continued-training on StackExchange content without query-document mappings.The fine-tuned model shows only a slight performance decrease, suggesting that conventional training procedures have limited impact.
  • Long-context retrieval: 27.8 recall@1 is the highest score reported for long-context retrieval across the evaluated StackExchange and Pony datasets.The setting uses complete web pages, reduced retrieval pools, and recall@1 because nDCG@10 is more susceptible to randomness in this configuration.

6 CONCLUSION

BRIGHT is introduced as a realistic benchmark for retrieval scenarios where identifying relevant documents requires intensive reasoning beyond lexical and semantic similarity.

  • Conclusion: BRIGHT contains realistic retrieval scenarios requiring intensive reasoning to identify relevant documents.The benchmark is built from existing online document structures with substantial human curation and correctness verification.
  • Conclusion: Existing retrieval models perform extremely poorly on BRIGHT, with a maximum nDCG@10 score of only 24.3.LLM-generated reasoning steps improve retrieval, but the best model still scores below 30.
  • Conclusion: Strong retrieval results can significantly improve downstream performance on reasoning tasks.The conclusion identifies downstream task performance as a practical application of reasoning-intensive retrieval.

CODE OF ETHICS AND ETHICS STATEMENT

The authors state that BRIGHT uses public data for academic research, excludes harmful or private information, and is intended for broad research use.

  • Ethics and data use: BRIGHT’s sources are public and used solely for academic research rather than commercial purposes.The authors describe this use as compliant with the copyright rights granted by the sources.
  • Ethics and data use: The datasets are stated to exclude harmful information, including racial discrimination, violence, and private data.The authors frame the dataset as intended to support societal and human welfare.
  • Ethics and data use: The authors state that researchers are free to use the dataset for research purposes.

REPRODUCIBILITY

The reproducibility materials specify model instructions, benchmark configurations, implementation details, and computational requirements for rerunning BRIGHT experiments.

  • Reproducibility: The authors provide code and data to facilitate reproduction of the experiments.The cited passage directs readers to the project website for implementation materials.
  • Model configurations: Table 7 documents the 13 benchmarked models, including parameter counts, architectures, context lengths, and instruction usage.The table distinguishes sparse and dense models and identifies encoder or decoder architectures when known.
  • Instructions: Benchmarking instructions are separately specified for StackExchange, LeetCode, Pony, and mathematics datasets.Tables 8–11 cover StackExchange, LeetCode, Pony, and AoPS/TheoremQA instructions.
  • Compute: The evaluation uses NVIDIA V100, A100, or H100 GPUs, while BM25 runs in under one hour on CPU-only machines.Open-source dense models below 1B parameters require about 8 hours on one H100 GPU, while larger models can require up to 36 hours.

A.3 CONTINUAL TRAINING SETUP

The continual-training setup evaluates whether in-domain StackExchange training improves BRIGHT retrieval while preserving language-generation ability, alongside corpus-construction and annotation choices.

  • Continual training: GritLM training uses contrastive and language-modeling objectives to retain retrieval and generation capabilities.The contrastive loss targets retrieval capability, while the language-modeling loss preserves language generation ability.
  • Corpus processing: Passages are split with simple separators such as double newlines or markdown headings to simulate automatic processing of long documents.The authors note that this heuristic may not be optimal for every document.
  • Annotation: StackExchange positives are manually verified with reasoning traces, while annotators avoid false negatives by selecting clearly differentiated posts.The annotation process also uses domain-specific distinctions among previously annotated examples.
  • STEM corpus: The STEM corpus combines problem-and-solution pairs from GSM8K, GSM8K-RFT, MATH, AQuA-RAT, TheoremQA, and CAMEL-Math.These documents support the TheoremQA Questions and AoPS retrieval datasets.
  • Corpus limitations: Exhaustively annotating every problem-solution pair for every test query was rejected because the required annotation cost was too high.The alternative would have produced harder negatives and additional positives.

B.4 THEOREMQA: ANNOTATING RELEVANT THEOREMS

TheoremQA annotation links rephrased questions to theorem documents through candidate retrieval and model-assisted relevance judgments, while the broader benchmark construction uses curated math and coding resources.

  • Theorem documents: TheoremQA-Theorems maps test queries to useful theorem proofs and definitions sourced from ProofWiki.ProofWiki is described as a community-driven collection containing more than 20K formal definitions and theorem proofs.
  • Candidate construction: Candidate theorem documents are gathered by theorem-name matching and BM25 retrieval of the top k = 10 documents.Documents with more than 100 theorem-name matches are discarded as overly common.
  • Relevance annotation: GPT-4 labels candidate theorem documents as positive or negative according to whether their theorems are used in problem solutions.Manual annotation of 50 instances yielded Cohen’s κ = 0.62 with model judgments, and only queries with at least one positive document are retained.
  • AoPS construction: AoPS Wiki supplies annotated links between mathematical topics or techniques and competition problems.The collected material includes problems from competitions such as AMC, AIME, and IMO, alongside explanatory mathematics articles.
  • Coding datasets: LeetCode positives come from its Similar Questions field, whose linked problems overlap in data structures, algorithms, or logical reasoning.The retrieval pool combines LeetCode and CodeSearchNet examples containing problem descriptions and solutions.
  • False-negative control: LeetCode questions are filtered by topic overlap, excluding corpus questions when at least half of the topics overlap with the test query.The threshold is expressed as |T(Q)∩T(Q′)| / |T(Q)| ≥ 0.5.

C DATA EXAMPLES

BRIGHT’s examples span several reasoning types and domains, and its annotation process identifies relevant passages, filters negatives, and documents how the datasets were collected.

  • Reasoning categorizations: BRIGHT categorizes retrieval requirements into deductive, analogical, causal, and analytical reasoning.The categories distinguish applying general principles, matching underlying logic, tracing causes, and building reasoning chains from critical concepts.
  • Reasoning categorizations: These reasoning categories help users interpret dataset-specific improvements and support more systematic development of retrieval models.The stated purpose is to understand achievements when performance improves on particular datasets.
  • StackExchange: Annotators retain StackExchange posts whose answers are accepted or highly voted and contain external URL links.The selection requires acceptance by the user or more than five votes, plus one or more links.
  • Relevance annotation: A linked document is considered relevant when it provides critical information, such as concepts, details, theorems, lemmas, or code needed to address the post.Annotators record why the document and post are relevant and what reasoning connects them.
  • Negative construction: Negative passages are selected to be topically related but unhelpful for the query, while other negatives are checked for different subtopics.This prevents retrieval based purely on lexical or semantic similarity.
  • Passage processing: Web pages are split into small passages using newline symbols, markdown headings, or fixed-length tokens.Positive passages are further refined according to the annotation criteria.
  • AoPS: AoPS annotators find theorem or technique pages linked to at least two competition problems, record both links, and manually check scraped content.The search is organized across algebra, geometry, calculus, probability, number theory, and other categories.

F.2 THEOREMQA

TheoremQA annotators validate rewritten questions by checking solvability and consistency with the original reasoning and answer, then revise or discard invalid items.

  • Annotators first determine whether each rewritten question is solvable.
  • For solvable questions, annotators check consistency with the original question and solution, including identical reasoning steps and final answers.
  • Consistent questions are marked valid and may receive minor edits for grammar or fluency.
  • Unsolvable or inconsistent questions are corrected using the original question and solution when possible; otherwise, they are discarded.

G LIMITATIONS AND FUTURE WORK

The paper acknowledges subjectivity in relevance judgments and limits its scope beyond retrieval, while identifying multimodal settings and other embedding tasks as future directions.

  • Limitations: Relevance judgments are subjective, so human retrieval results may not exactly match the annotations.The authors retain only queries on which multiple annotators, including domain experts, agree.
  • Future Work: The benchmark does not address multimodal settings, which the authors identify as an avenue for future exploration.
  • Future Work: The paper notes that reasoning may also be required for embedding tasks beyond retrieval, such as clustering.
  • Scope: BRIGHT is designed for realistic retrieval scenarios rather than abstract reasoning tests that retrieve answers instead of documents.
Loading 2407.12883v4…