BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

1 INTRODUCTION

• Unlike fact-based benchmarks where relevance often follows keyword or semantic overlap, BRIGHT requires reasoning to identify deeper connections between queries and relevant documents.Such connections can involve analogous underlying principles, as in finding a story explained by the same economic theory or syntax documentation matching an error message.Source: §1 · pp. 1–2, Figure 1 · p. 2
• BRIGHT comprises 12 datasets: seven StackExchange sets use unanimously validated answer-cited documents, while five coding and math sets link queries to shared algorithms or theorems.Its queries and corpora span diverse domains and include web data such as blogs, syntax documentation, and STEM problem-solutions.Source: §1 · p. 2, Figure 1 · p. 2
• SFR-Embedding-Mistral scores 59.0 on the MTEB retrieval subset, BEIR, but only 18.3 nDCG@10 on BRIGHT.The paper reports the 59.0 score for the MTEB retrieval subset and the 18.3 nDCG@10 score for BRIGHT.Source: §1 · p. 2
• Generating query chain-of-thought before retrieval yields average improvements of up to 12.2 points; Qwen-retrieved documents improve QA over closed-book, while oracle documents improve it more.The larger gain from oracle documents highlights the potential benefits of improving retriever models.Source: §1 · p. 2
• Further training on benchmark documents produces no substantial performance gains, demonstrating BRIGHT’s robustness to potential data leakage during large-scale pre-training.The authors hope the findings inspire research to advance reasoning-intensive retrieval.Source: §1 · pp. 2–3

Benchmarking retrieval

• Unlike prior benchmarks such as BEIR (Thakur et al., 2021), BIRCO (Wang et al., 2024), and RAR-b, BRIGHT pairs realistic user queries with natural documents requiring deep reasoning.BIRCO is limited to LLM reranking with about 100 candidate documents per query; RAR-b adapts reasoning datasets to retrieve answers, whereas BRIGHT's answers are unlikely to appear as document substrings.Source: §2 · p. 3

Dense retrieval models and retrieval augmented generation

• We benchmark sparse and dense, small and large, and open-source and proprietary retrievers; initial analyses show stronger retrievers improve generation on reasoning-intensive tasks.Retrieval-augmented generation uses retrieved documents to help produce coherent answers, though the work focuses mainly on retrieval.Source: §2 · p. 3

Benchmarking reasoning

• Math and coding reasoning benchmarks typically use question-answer pairs from textbooks, online resources, competitions, or experts, whereas BRIGHT adds annotations to high-quality dataset queries for realistic retrieval.Examples include GSM8K, MATH (Hendrycks et al., 2021), HumanEval (Chen et al., 2021), and MBPP (Austin et al., 2021).Source: §2 · p. 3

3.1 TASK FORMULATION

• Relevant documents are connected to queries through reasoning traces such as underlying principles, algorithms, or theorems, which are typically absent from the query itself.The retrieval task is to find relevant documents from a corpus given a query.Source: §3.1 · p. 4

3.2 STACKEXCHANGE: RETRIEVING WEB PAGES THAT HELP ANSWER QUESTIONS

• StackExchange queries combine post titles and content from seven knowledge-intensive domains; relevant passages come from linked pages in accepted or highly voted answers.Annotators assess whether linked pages help reason through the query; only posts with an accepted answer or one receiving more than five votes and at least one URL are selected.Source: §3.2 · pp. 4–5
• StackExchange negatives include topically related Google results that do not answer the query, and all nonpositive passages in each dataset’s unified corpus.Annotators search using post titles or summarized keywords and collect up to five negative web pages per query.Source: §3.2 · p. 5

3.3 CODING: RETRIEVING DOCUMENTATION OR SIMILAR SOLVED PROBLEMS

• Coding retrieval covers Pony documentation for required syntax and LeetCode problems sharing algorithmic design, with real-world problems chosen to demand reasoning.Pony instructions are queries and annotated syntax documentation is positive (Su et al., 2024); LeetCode queries retrieve similar solved problems, with a corpus combining LeetCode and CodeSearchNet (Husain et al., 2019).Source: §3.3 · p. 6

3.4 THEOREM-BASED QUESTIONS: RETRIEVE SOLVED PROBLEMS USING THE SAME THEOREMS OR RELEVANT THEOREM STATEMENTS

• Theorem-based datasets retrieve solved problems sharing a theorem or problem-solving skill, and also pair TheoremQA queries with relevant ProofWiki theorem statements.TheoremQA questions are rephrased into applied scenarios while preserving the required theorem and are human-reviewed (Chen et al., 2023b); AoPS problems remain unrephrased and are labeled by shared problem-solving skill. ProofWiki theorem matches are aligned by title and verified with GPT-4.Source: §3.4 · pp. 6–7
• 24.3 nDCG@10 was the best retrieval model’s average across BRIGHT’s 12 datasets.Table 2 reports this average as evidence that reasoning-intensive retrieval challenges current retrievers.Source: Table 2 · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation covers 13 retrieval models, from lexical BM25 and small and large open-source dense models to proprietary systems, with nDCG@10 as the main metric.The model set includes strong BEIR and MTEB performers; BM25 is described as strong on BEIR and comparable there to larger trained dense retrievers.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

• 24.3 nDCG@10 is the best model score reported on BRIGHT, and BM25 matches sub-1B models but trails larger models, showing existing retrieval remains challenging.The paper concludes that keyword matching is insufficient for reasoning-intensive retrieval and that new methods are needed.Source: §4.2 · p. 8
• Llama-3-70B and GPT-4 reasoning-step queries significantly improve retrieval over original queries; GritLM helps BM25 but is less effective for other models, and the best score remains below 30.BM25 achieves the best leaderboard performance when using GPT-4-generated reasoning steps as queries.Source: §4.2 · p. 8, Figure 3 · p. 9
• 1.9 points is the QA gain with top-performing retriever Qwen, while oracle documents boost performance by 4.1 points, leaving room for retrieval improvement.Stronger retrievers generally produce better QA performance; QA scores may not fully reflect retrieval quality because of generator and evaluator limitations.Source: §4.2 · pp. 8–9, Table 4 · p. 8

5.1 RERANKING WITH LLMS ENHANCES RETRIEVAL PERFORMANCE

• LLM reranking generally improves retrieval, with Gemini-1.0 adding 1.4 and GPT-4 adding 3.1 to BM25-based scores, while MiniLM12 hurts performance.LLM gains grow with stronger models and higher k, but final performance remains highly dependent on the underlying retrieval system; cross-encoder performance worsens as more documents are reranked.Source: §5.1 · p. 9

5.2 ROBUSTNESS AGAINST DATA LEAKAGE FROM PRETRAINING

• StackExchange pretraining without query-document mappings produces only a slight performance decrease, showing BRIGHT’s robustness to exposure to its retrieval documents.GritLM was continued-training on StackExchange content from the retrieval pool using language-modeling loss and contrastive learning on question-answer pairs; Table 5 reports no average improvement after in-domain knowledge learning.Source: §5.2 · p. 10, Table 5 · p. 10

5.3 LONG-CONTEXT RETRIEVAL WITH A REDUCED SEARCH SPACE IS CHALLENGING

• 27.8 recall@1 is the highest reported score across eight StackExchange and Pony datasets, indicating long-context retrieval remains challenging despite reduced search pools.The evaluation uses complete, unsplit web pages and recall@1 because nDCG@10 is more susceptible to randomness in the smaller pools, and processing long documents is challenging.Source: §5.3 · p. 10, Table 6 · p. 10

6 CONCLUSION

• Existing retrieval models reach a maximum nDCG@10 of only 24.3 on BRIGHT; adding LLM-generated reasoning improves performance, but the best model remains below 30.The conclusion also reports that strong retrieval results can significantly improve downstream performance on reasoning tasks.Source: §6 · p. 10
• BRIGHT is introduced as the first retrieval benchmark for realistic scenarios that require intensive reasoning to identify relevant documents.Its construction uses existing online document structures and substantial human effort to curate the benchmark and verify its correctness.Source: §6 · p. 10
• Strong retrieval results on BRIGHT can significantly improve downstream performance on reasoning tasks, demonstrating a practical application of reasoning-intensive retrieval.Source: §6 · p. 10

• The datasets come from public sources and are used solely for academic, noncommercial research in compliance with source-granted copyright rights.Source: CODE OF ETHICS AND ETHICS STATEMENT · p. 11
• The authors state that the datasets contain no harmful information, such as racial discrimination or violence, and no private data.Source: CODE OF ETHICS AND ETHICS STATEMENT · p. 11
• Researchers may use the dataset for research, and the authors state that their data and experiments meet standards of authenticity and accuracy.Source: CODE OF ETHICS AND ETHICS STATEMENT · p. 11

REPRODUCIBILITY

• The authors provide the code and data at the BRIGHT project site to facilitate reproducing the experiments.The passages direct readers to the project site for these materials.Source: REPRODUCIBILITY · p. 11

A.1 MODELS AND INSTRUCTIONS

• Most model parameters follow official tutorials; Inst-L and Inst-XL instead use a 2048-token maximum context length, which the authors found significantly improved performance.This is the stated exception to the default parameter-setting procedure.Source: §A.1 · p. 17
• Google embedding models distinguish queries from documents using RETRIEVAL_QUERY and RETRIEVAL_DOCUMENT task values, while Voyage uses query and document input_type values.The instructions used for BGE, Inst-L, Inst-XL, E5, GritLM, Qwen, and SFR are specified by dataset in Tables 8–11.Source: §A.1 · p. 17
• Table 7 documents all 13 benchmarked models, including their parameters, architecture, context limits, instruction use, and checkpoint or implementation version.The table distinguishes sparse and dense models, and classifies dense models as encoders or decoders when known.Source: Table 7 · p. 17
• Benchmark instruction specifications are provided for StackExchange, LeetCode, Pony, and Math datasets, with StackExchange domain names represented by {domain}.The Math datasets are AoPS and TheoremQA.Source: Table 8 · p. 18, Table 9 · p. 18, Table 10 · p. 18

A.2 COMPUTING RESOURCES

• Experiments ran on V100, A100, or H100 GPUs; evaluations used FlashAttention for dense models, while BM25 ran on CPU-only machines.BM25 took less than 1 hour; open-source dense models under 1B parameters took about 8 hours, and those over 1B took up to 36 hours on one H100; proprietary-model runs finished within 2 days, depending on API bandwidth.Source: §A.2 · p. 18

A.3 CONTINUAL TRAINING SETUP

• Across 10 training epochs, continual training of GritLM on StackExchange data showed no significant performance improvement on BRIGHT’s StackExchange datasets.Training combined contrastive and language-modeling losses; the benchmarked checkpoint was taken from each epoch.Source: Table 12 · p. 19, §A.3 · p. 19

B.1 STACKEXCHANGE

• StackExchange documents were split using simple separators, simulating automatic processing of long documents without human or expert intervention.Separators included two newline symbols and “#” in Markdown files, without assumptions about file structure.Source: §B.1 · p. 19
• Annotators manually verified StackExchange query-document relevance with reasoning traces and selected distinct posts to avoid false negatives among examples in each domain.They chose posts whose entities and meanings differed from previously annotated examples, ensuring positive documents were not shared across example pairs.Source: §B.1 · p. 19

B.2 STEM QUESTION AND SOLUTION CORPUS FOR THEOREMQA AND AOPS

• The STEM corpus combines problem-solution pairs from six existing datasets, with metadata-based category exclusions for TheoremQA-Q and AoPS queries to reduce false negatives.No GSM8K, GSM8K-RFT, or AQuA-RAT pairs were excluded because they were considered relatively elementary compared with the test queries.Source: §B.2 · p. 20
• Annotating every candidate problem-solution pair for relevance was rejected as too costly, despite potentially producing harder negatives and additional positives.Instead, the authors manually selected dataset labels to exclude for each test query.Source: §B.2 · p. 20

B.3 THEOREMQA: REPHRASING QUESTIONS INTO SPECIFIC SCENARIOS

• TheoremQA questions were rewritten as real-world scenarios because explicitly named theorems created keyword overlap that could make retrieval easier through keyword matching.TheoremQA questions were paired with MathInstruct examples annotated with reasoning steps and final answers.Source: §B.3 · p. 20
• GPT-4 rewrites were manually checked for solvability and consistency with the original reasoning and answer, yielding 206 questions from the original 800.Authors edited some rewrites for fluency and coherence, and discarded those that were unsolvable or inconsistent.Source: §B.3 · p. 20

B.4 THEOREMQA: ANNOTATING RELEVANT THEOREMS

• TheoremQA test queries are paired with ProofWiki theorem proofs and definitions, with original theorem names mapped to consistent corpus documents.ProofWiki is provided by MathPile (Wang et al., 2023b).Source: §B.4 · p. 20
• TheoremQA candidate documents come from theorem-name substring matches, unless more than 100 are found, and BM25 retrieval of the top 10 using the theorem name and definition.The candidate set is constructed from ProofWiki documents.Source: §B.4 · pp. 20–21
• GPT-4 labels each TheoremQA candidate as positive or negative based on whether its theorem is used in the solution; queries are retained only if they have a positive document.Manual labels for 50 instances had Cohen’s κ = 0.62 agreement with model judgments.Source: §B.4 · p. 21

B.5 AOPS: CONNECTING AOPS PROBLEMS TO THE MATH DATASET

• AoPS Wiki topic articles link math techniques to problems, providing skill annotations missing from MATH (Hendrycks et al., 2021); collected problems are matched against MATH and merged when identical.Nonmatching problems are added as new corpus instances.Source: §B.5 · p. 21

B.6 LEETCODE

• LeetCode retrieval uses problem descriptions and solutions from LeetCode and CodeSearchNet, with similar-question links serving as gold pairs based on shared solution methods.The website does not state how similar-question links are populated; qualitative analysis found overlap in data structures, algorithms, or reasoning.Source: §B.6 · p. 21
• GPT-4 classifies LeetCode questions grounded in real-world concepts to reduce false positives; 80 manual checks yielded Cohen’s κ = 0.73, and 142 of 291 model-positive questions were sampled for testing.The authors used six-shot demonstrations and validated the model’s judgments with their annotations.Source: §B.6 · pp. 21–22
• To reduce LeetCode false negatives, corpus questions with at least half their topics overlapping a test question are excluded, and the remaining corpus uses Python functions from CodeSearchNet (Husain et al., 2019).LeetCode topic tags include algorithms and data structures such as stacks and breadth-first search.Source: §B.6 · p. 22

C DATA EXAMPLES; D REASONING CATEGORIZATIONS

• The appendix illustrates four reasoning types across 12 datasets and provides additional BRIGHT data examples in Tables 20–31.Source: D REASONING CATEGORIZATIONS · p. 22, C DATA EXAMPLES · p. 22

D REASONING CATEGORIZATIONS

• BRIGHT’s categories are deductive application of principles, analogical transfer of underlying logic, causal diagnosis, and analytical use of concepts to complete reasoning chains.Source: D REASONING CATEGORIZATIONS · p. 22

E DOWNSTREAM PERFORMANCE

• Oracle documents consistently improved GPT-4o-2024-08-06 performance on all three theorem-based tasks, with statistically significant gains on TheoremQA Questions and TheoremQA Theorems.The evaluation compared closed-book, random-document, and oracle-document contexts, averaging results across five random seeds.Source: E DOWNSTREAM PERFORMANCE · p. 23

F ANNOTATOR INSTRUCTIONS; F.1 STACKEXCHANGE

• For StackExchange, annotators screened answered, sufficiently voted posts and linked sources, checked that sources materially helped solve the query, and sought hard negatives unrelated to the problem.They recorded why each linked document was relevant and the reasoning needed to connect it to the post.Source: §F.1 · p. 23

F.3 AOPS

• AoPS annotators linked Wiki theorems or techniques to at least two competition problems; scraped statements and solutions were manually quality-checked.Categories were assigned to annotators to avoid overlap.Source: §F.3 · p. 24

F.4 LEETCODE

• LeetCode annotators classified whether a problem relied on real-world concepts, distinguishing absent, commonplace-in-coding, and less-common concepts.The last category includes examples such as building height, planting trees, and games.Source: §F.4 · p. 24

F.5 LLM USAGE

• Annotators used LLMs for clarification and StackExchange negative-search keywords, verified information with trustworthy sources, and excluded LLM outputs from BRIGHT.Tools included ChatGPT, GPT-4, and Claude-3.Source: §F.5 · p. 24

F.6 SENSITIVE INFORMATION

• All BRIGHT data were manually collected, verified, and reviewed to remove personally identifiable or offensive content.Source: §F.6 · p. 24

• Annotators first determine whether each rewritten question is solvable.Source: §F.2 · p. 23
• For a solvable rewrite, they check whether the original question and solution support the same reasoning steps and final answer.Source: §F.2 · p. 23
• Consistent rewrites are marked valid and may receive minor edits; inconsistent or unsolvable ones are corrected if possible, otherwise discarded.Source: §F.2 · p. 24

G LIMITATIONS AND FUTURE WORK

• BRIGHT’s relevance judgments are subjective, but retaining queries with unanimous agreement from multiple annotators, including domain experts, is intended to make them reliable and consistent.The authors note that accepted or highly voted StackExchange answers do not guarantee universal agreement about document relevance.Source: G LIMITATIONS AND FUTURE WORK · p. 25
• The authors identify reasoning-intensive clustering and multimodal settings as directions for future exploration that this paper does not address.Source: G LIMITATIONS AND FUTURE WORK · p. 25

H POTENTIAL SOCIAL IMPACTS

• The paper anticipates that BRIGHT could improve search and information access, support retrieval research and collaboration, and help develop more effective search engines.Source: H POTENTIAL SOCIAL IMPACTS · p. 25

• Dataset Construction:

• BRIGHT was purpose-built for realistic retrieval scenarios, whereas RAR-b adapts existing multiple-choice benchmarks into retrieval using original questions and their answer options.Source: • Dataset Construction · p. 25
• BRIGHT uses documents longer than 100 tokens to reflect practical information seeking, unlike RAR-b’s often-under-10-word options and more conceptual focus.Source: • Dataset Construction · p. 25

I COMPARISON TO RAR-B

• Both BRIGHT and RAR-b assess reasoning-intensive retrieval, but the authors say their approaches and objectives differ and plan clearer comparisons of RAR-b in a revision.Source: • Dataset Construction · p. 25, I COMPARISON TO RAR-B · p. 25