PaperlayerView exact revision on arXiv

Source-linked AI summary

RAR-b: Reasoning as Retrieval Benchmark

Chenghao Xiao, G Thomas Hudson, Noura Al Moubayed

arXiv:2404.06347v2cs.CLcs.IR

TL;DR

RAR-b asks whether dense retrievers can encode reasoning abilities important to retrieval-augmented generation, where retrieval gains on reasoning-intensive tasks can be inconsistent. It reframes reasoning as retrieval and evaluates systems across architectures and training recipes, reporting behavioral discrepancies between retrievers and LLMs and identifying directions for future research.

  • Problem

    Reasoning-intensive RAG has shown inconsistent gains or performance degradation, while dense models need evaluation beyond factual recall and semantic matching.

    Source: §1 · p. 2

  • Method

    RAR-b reframes reasoning tasks as retrieval and evaluates open-source and proprietary retrievers and rerankers across architectures and training recipes.

    Source: §1 · p. 2

  • Results

    The evaluation reveals behavioral discrepancies between retrievers and LLMs and insights from comparing systems with different architectures and training recipes.

    Source: §1 · p. 2, §5 · p. 15

  • Takeaways & Limitations

    The authors present RAR-b as a benchmark for tracking retrieval models’ reasoning abilities and suggest that models excelling on it are well-suited to enhance LLMs in RAG systems.

    Source: §5 · p. 15

    The authors note that generative rerankers are not yet trivial to train or cheap at inference, and identify effective decoder-reranker training and resource-efficient inference as research directions.

    Source: §3.2 · p. 10

Abstract

from arXiv · show

Semantic textual similartiy (STS) and information retrieval tasks (IR) tasks have been the two major avenues to record the progress of embedding models in the past few years. Under the emerging Retrieval-augmented Generation (RAG) paradigm, we envision the need to evaluate next-level language understanding abilities of embedding models, and take a conscious look at the reasoning abilities stored in them. Addressing this, we pose the question: Can retrievers solve reasoning problems? By transforming reasoning tasks into retrieval tasks, we find that without specifically trained for reasoning-level language understanding, current state-of-the-art retriever models may still be far from being competent for playing the role of assisting LLMs, especially in reasoning-intensive tasks. Moreover, albeit trained to be aware of instructions, instruction-aware IR models are often better off without instructions in inference time for reasoning tasks, posing an overlooked retriever-LLM behavioral gap for the research community to align. However, recent decoder-based embedding models show great promise in narrowing the gap, highlighting the pathway for embedding models to achieve reasoning-level language understanding. We also show that, although current off-the-shelf re-ranker models fail on these tasks, injecting reasoning abilities into them through fine-tuning still appears easier than doing so to bi-encoders, and we are able to achieve state-of-the-art performance across all tasks by fine-tuning a reranking model. We release Reasoning as Retrieval Benchmark (RAR-b), a holistic suite of tasks and settings to evaluate the reasoning abilities stored in retriever models. RAR-b is available at https://github.com/gowitheflow-1998/RAR-b.

1 INTRODUCTION

RAR-b asks whether dense retrievers encode reasoning abilities, extending evaluation beyond factual recall and semantic matching. It reframes reasoning tasks as retrieval and evaluates current systems to characterize their reasoning behavior and identify promising directions.

1 INTRODUCTION

  • Reasoning-intensive RAG remains unreliable: retrieval can yield inconsistent gains or even degrade LLM performance.This motivates evaluating retrievers’ advanced language-understanding abilities beyond factual recall or semantic matching.Source: §1 · p. 2
  • RAR-b reframes reasoning tasks as retrieval to assess reasoning representations in dense models.The benchmark examines whether dense representations encode reasoning expressions and challenges the status quo of reasoning-oriented RAG.Source: §1 · p. 2
  • The benchmark builds on sentence and document models’ established semantic-similarity and topical-alignment abilities to call for evaluating reasoning abilities.The paper asks whether dense representations encode reasoning needed for complex language understanding.Source: §1 · p. 2
  • The paper evaluates state-of-the-art open-source and proprietary retrievers and rerankers, including unsupervised, supervised, and instruction-aware systems.The evaluation is intended to provide an in-depth understanding of current retrieval models’ reasoning behaviors.Source: §1 · p. 2
  • RAR-b compares systems with different architectures and training recipes to identify behavioral insights and possible pathways toward reasoning-level language understanding.The authors frame these insights as directions for future research.Source: §1 · p. 2

2 RAR-B

RAR-b evaluates retrievers in a standalone setting by recasting reasoning problems as retrieval, using answer similarity to a query as the test. It combines tasks, datasets, models, and retrieval settings to examine reasoning representations across a broad range of capabilities.

2 RAR-B

  • RAR-b evaluates retrievers alone on reasoning tasks recast as retrieval, using their performance as a proxy for their upper-bound assistance to LLMs in standard RAG.The benchmark integrates 12 tasks derived from 17 datasets and tests both multiple-choice and full-dataset retrieval settings.Source: §2 · p. 2

2.1 PROBLEM FORMULATION

  • A retriever’s ideal outcome is to encode the ground-truth answer as more similar to the reasoning query than other corpus candidates, though current sentence encoders may not achieve this.Full-dataset retrieval uses nDCG@n and recall@n; multiple-choice retrieval uses accuracy.Source: §2.1 · p. 3

2.2 DATASETS

  • The datasets cover commonsense, temporal, spatial, numerical, and symbolic reasoning, with code retrieval included as a proxy for symbolic-language understanding.The suite includes commonsense datasets such as αNLI and HellaSwag, TempReason and SpartQA, pooled MATH and GSM8K answers, and HumanEvalPack and MBPP; C-STS is excluded from full retrieval because of sparse annotation concerns.Source: §2.2 · pp. 3–4, §2.4 · p. 6
  • The benchmark groups tasks into levels, from more IR-like commonsense tasks to less familiar formats and temporal, numerical, spatial, and symbolic reasoning.The authors also motivate level-2 evaluation with numerical, temporal, and spatial query examples they say current state-of-the-art representation models do not handle natively.Source: §2.2 · pp. 4–5

2.3 MODELS

  • The evaluation covers unsupervised, supervised, and instruction-aware bi-encoders, alongside representative cross-encoder rerankers.Bi-encoders include open-source and proprietary models; the listed rerankers include BGE-reranker-large, TART-Full, and Cohere rerank-english-v2.0.Source: §2.3 · pp. 5–6

2.4 REASONING AS RETRIEVAL

  • Tasks and candidate answers share a representation space, so finding an answer becomes similarity matching; the benchmark tests both multiple-choice and full-dataset retrieval.Full retrieval uses a pooled set of unique candidates, while multiple-choice retrieval uses the choices available to each question; the two settings model different retrieval roles in RAG.Source: §2.4 · p. 6

3 RESULTS

Fine-tuning a Cohere reranker achieves state-of-the-art performance when reranking the top 100 documents, while the retrieval and multiple-choice results show that reasoning remains difficult for current retrievers. Newer and decoder-based models show promise, but instructions can distract models and performance on full retrieval can overstate in-depth understanding.

3.1 FULL-DATASET RETRIEVAL SETTING

  • Newer models tend to outperform older ones on RAR-b, with decoder-based embedding models identified as promising for narrowing the retriever–LLM behavioral gap.Figure 1 represents relative performance using geometric means across tasks, centered by subtracting the mean across models.Source: §3.1 · pp. 6–7, Figure 1 · p. 7
  • Instruction prompts generally distract retrievers, including instruction-aware models; the effect is more pronounced for encoder models and smaller decoder variants.For models that degrade, the effect is more severe in full-dataset than multiple-choice retrieval, which the authors attribute intuitively to the larger candidate pool.Source: §3.1 · p. 8
  • Full-dataset nDCG@10 is often decent, but retrieving all correct candidates for downstream LLM decisions remains beyond current performance.The authors identify recall@n ≈1 as the ideal for ensuring correct answers are available to later LLM decisions.Source: §3.1 · p. 9

3.2 DOES RERANKER HELP?

  • Fine-tuning Cohere’s reranker achieves state-of-the-art performance by reranking the top 100 documents, subject to the first-round retrieval ceiling.Figure 3 reports average without-instruction nDCG@10 across the evaluated tasks.Source: §3.2 · p. 10, Figure 3 · p. 10

3.3 MULTIPLE-CHOICE RETRIEVAL SETTING

  • Models remain near chance on Winogrande and CSTS in multiple-choice retrieval, exposing how full-setting Winogrande scores can reflect answer-word matching rather than fine-grained understanding.Contriever’s full-setting Winogrande nDCG@10 is 0.471 without instruction, yet its MCR performance is at chance level.Source: §3.3 · pp. 10–11
  • Instructor performs better on the first six tasks but remains at chance on both constructed CSTS settings; the authors hypothesize an encoder generalization gap, not yet systematically investigated.The passage notes that similarly instruction-tuned decoder models, including E5-Mistral, GritLM, and OpenAI models, perform better on CSTS.Source: §3.3 · p. 11
  • Contriever is a strong baseline across datasets and robust to instructions; the authors speculate that reasoning-dataset question–answer formats resemble its span-pair training data.The proposed explanation is that question–answer pairs in datasets such as HellaSwag resemble consecutive document spans.Source: §3.3 · p. 12

4 BEHAVIORAL ANALYSIS

The behavioral analysis probes retrievers across temporal-reasoning and code-retrieval settings, distinguishing knowledge and comprehension from robustness to distractors. Its code experiments expose cross-language hard-negative distraction, an entity-matching shortcut, and weaker performance on mixed natural-language/code data.

4.1 BEHAVIORAL ANALYSIS THROUGH THE LENS OF TEMPORAL REASONING

  • TempReason-L1 uses Pure time-arithmetic questions, while L2 and L3 add Pure, Fact, and Context+Fact settings for time-situated question answering.The seven sub-settings support analysis of parameterized knowledge, reading comprehension, and robustness to irrelevant context.Source: §4.1 · p. 12
  • Pure tests stored knowledge without references; Fact supplies answer-bearing facts in the query, while Context+Fact adds context and is excluded from average TR scores.Context+Fact is used for analysis because differing maximum sequence lengths can make comparisons unfair; E5-Mistral and GritLM use 8192 tokens in that setting.Source: §4.1 · pp. 12–13

4.2 BEHAVIORAL ANALYSIS THROUGH THE LENS OF CODE RETRIEVAL

  • A pooled multilingual setting underperforms the average of individual-language settings, indicating that same-content code in another language can outrank the groundtruth despite language instructions.The authors interpret the gap as a failure to perceive programming languages through instructions.Source: §4.2 · pp. 13–14
  • In HumanEvalPack, Instruct nDCG@10 rises from 0.374 to 0.915 for Instructor-XL, but shared function declarations enable entity matching, so the authors retain the Original format.The Instruct setting retrieves complete code from a self-contained instruction; the repeated function declaration can let retrieval rely solely on matching that declaration.Source: §4.2 · p. 14
  • On TinyCode’s mixture of natural language and code, models generally perform less well, although controlled experiments are difficult to construct.The evaluation samples 15k queries and 150k documents, including the groundtruth.Source: §4.2 · pp. 14–15

5 CONCLUSION

The paper argues that retrievers need evaluation beyond semantic similarity and topical matching. RAR-b casts reasoning as retrieval, reports behavioral discrepancies between retrievers and LLMs, and positions strong RAR-b representations as suited to enhance LLMs in RAG systems.

5 CONCLUSION

  • RAR-b casts reasoning as retrieval and reveals retriever–LLM behavioral discrepancies; representations that excel on the benchmark are positioned as suited to enhance LLMs in RAG systems.The paper presents RAR-b as a benchmark for tracking progress in retrievers’ next-level language understanding.Source: §5 · p. 15

A DATASET FORMAT

The dataset format lists Dataset, Instruction, Query, and Doc fields, and includes a math-retrieval instruction example.

A DATASET FORMAT

B POTENTIAL SPARSE ANNOTATION PROBLEM OF C-STS

C-STS’s paired-condition annotations constrain how the dataset can be converted into retrieval settings.

B POTENTIAL SPARSE ANNOTATION PROBLEM OF C-STS

C ILLUSTRATION OF THE ENTITY MATCHING SHORTCUT

HumanEvalPack’s Instruct setting performs highly largely because the query and groundtruth share a function declaration that retrievers can match by overlap.

C ILLUSTRATION OF THE ENTITY MATCHING SHORTCUT

  • HumanEvalPack’s Instruct setting performs highly largely because its query and groundtruth share a function declaration, enabling retrieval through overlap.The paper identifies this entity-matching shortcut as the main reason for the high performance.Source: C ILLUSTRATION OF THE ENTITY MATCHING… · pp. 22–23
  • The example query asks for a Python function to detect whether two numbers are closer than a given threshold.The prompt includes sample lists and expected Boolean outputs.Source: C ILLUSTRATION OF THE ENTITY MATCHING… · p. 22
  • The example groundtruth loops over number pairs and compares their absolute distance with the threshold.It skips matching an element with itself before checking the distance.Source: C ILLUSTRATION OF THE ENTITY MATCHING… · p. 23

D FULL SETTING RECALL@10

Table 7 reports recall@10 for Full-dataset Retrieval, a measure of whether the groundtruth answer is retrieved among the top ten documents.

D FULL SETTING RECALL@10

  • Recall@10 measures whether the groundtruth answer is retrieved among the top ten documents.The paper describes it as a straightforward measure of average empirical retrieval probability.Source: D FULL SETTING RECALL@10 · p. 23
  • For this setting, recall@10 gives the average empirical probability that the groundtruth appears in the top ten retrieved documents.No recall@10 values are supplied in these passages.Source: D FULL SETTING RECALL@10 · p. 23
  • Table 7 presents recall@10 performance for Full-dataset Retrieval.The passage identifies this as the table’s scope.Source: Table 7 · p. 23, D FULL SETTING RECALL@10 · p. 23
Loading 2404.06347v2…