ACARE: Agent-Context-Aware Reranker for Iterative Search

1 INTRODUCTION

• Conventional rerankers score documents against the agent’s sub-query alone, which may omit the broader task and retrieve documents that do not answer the user’s question.Search agents iteratively issue queries, inspect returned documents, and decide what to search for next.Source: §1 · p. 1
• A second-stage reranker can use the original question and available reasoning without requiring local access to the document corpus.This supports lexical and dense retrieval as well as search APIs that expose document text.Source: §1 · p. 1
• ACARe is a 4B decoder-only LLM fine-tuned from Qwen3-Reranker-4B to score candidate documents using the sub-query and agent context.It is trained with original and re-mined DR-Synth labels and new agent trajectories, with independent context dropout for missing fields.Source: §1 · p. 1
• Across three benchmarks, four first-stage retrievers plus API search, and two harnesses, ACARe improves dataset-averaged nDCG@10 by 0.9–2.0 percentage points over the base reranker and uses fewer searches for all five main agents.On ClimbMix, it raises answer accuracy by 8.2–9.1 percentage points over no reranking and matches or exceeds the base reranker’s mean accuracy.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

• Context-aware first-stage retrievers use agent context but require local access to the document corpus, whereas ACARe reranks returned documents.AgentIR embeds the reasoning trace with the sub-query, and ITER also incorporates the original question and preceding sub-queries.Source: §2 · p. 2
• Session and conversational search condition ranking on prior search activity or dialogue, while ACARe uses the original question and reasoning trace to interpret an agent-generated sub-query.ACARe’s checkpoint also handles the absence of either context field.Source: §2 · p. 2
• ReasonIR, RaDeR, and ReasonEmbed train dense retrievers for one-shot neural retrieval rather than interpreting sub-queries within an evolving agent trajectory.Their focus differs from ACARe’s trajectory-context reranking.Source: §2 · p. 2
• Agent-feedback systems train retrieval tools from browsing behavior, downstream outcomes, or agent–retriever interaction, whereas ACARe uses context at ranking time.ACARe’s single checkpoint supports four context modes.Source: §2 · p. 2
• ACARe independently scores documents and supports four context modes, unlike generative listwise approaches that can incur autoregressive decoding and sequential sliding-window costs.Concurrent RubricRanker trains a generative listwise selector with search rubrics.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• The agent’s search trajectory contains the original question, reasoning, actions, observations, and generated sub-queries, with retrieval and reranking supplying documents to the agent.The final output can be a ranked document list or an evidence-grounded answer.Source: §3.1 · p. 3

3.2 AGENT CONTEXT

• ACARe addresses the mismatch between a sub-query and the original question by including the original question and the reasoning trace preceding the current search when available.It does not make an additional model call to elicit a trace, though added context increases reranker input length.Source: §3.1 · p. 3, §3.2 · p. 3
• Four inference modes use the sub-query alone, with the trace, with the original question, or with both; the model processes one candidate document alongside the query and retained context.Agents without exposed reasoning can use the question-only context mode, while deployments without either field can use the sub-query-only mode.Source: §3.2 · p. 3

3.3 MULTI-TASK TRAINING OVER CONTEXT MODES

• ACARe is trained as one checkpoint for all four context modes, independently dropping the reasoning trace and original question with probability 0.5.When both fields are available, each mode has probability 0.25; missing source fields can change the effective distribution.Source: §3.3 · p. 4
• Training compares each sub-query’s positive document with retrieved negatives using localized contrastive estimation over group scores, while inference converts yes/no logit differences into relevance scores.Training inputs are capped at 5,120 tokens and inference inputs at 8,192 tokens.Source: §3.3 · p. 4

3.4 TRAINING DATA

• The training labels retain the original DR-Synth set, add re-mined candidates for its sub-queries, and include a new rollout using a second retriever.The pool contains 16,601 instances, compared with 5,238 in DR-Synth; re-mining changes documents but preserves sub-queries and traces.Source: §3.4 · p. 4

4 EXPERIMENTAL SETUP

• Retrieval Bench evaluates iterative retrieval on BCP, BRIGHT, and TRQA, while Piika evaluates end-to-end answer generation on BrowseComp-PlusCM and BCP.Retrieval metrics include nDCG@10 and Recall@10; iterative retrieval also reports search calls, LLM turns, and agent API cost per query.Source: §4 · p. 4
• Cost calculations exclude retrieval and reranker serving costs, which are assumed negligible compared with agent LLM API costs for the locally hosted 4B–8B models.Retrieval Bench normally retrieves 100 candidates and returns ten unseen documents after reranking when enabled.Source: §4 · p. 5
• One-shot comparisons include no reranking, Jina-Reranker-v3, Qwen3-Reranker-4B, and ACARe on the same candidates, with ACARe using the context-free mode.The first-stage retrievers include BM25, AgentIR-4B, Qwen3-Embedding-4B, and Nemotron-3-Embed-8B.Source: §4 · p. 5
• The main iterative retrieval experiments use five LLM agents; open-weight agents use question, trace, and sub-query context, while proprietary agents use question and sub-query context.Each configuration is evaluated five times, except Piika studies, which use six runs; quality metrics are averaged equally across the three benchmarks.Source: §4 · p. 5

5.1 ONE-SHOT RETRIEVAL

• 9.8–12.8 pp dataset-averaged nDCG@10 gains over no reranking and 1.7–4.5 pp over Qwen3-Reranker-4B; ACARe leads nDCG@10 and Recall@10 across all retriever–dataset combinations.Both rerankers use the same context-free prompt, so these gains reflect fine-tuning rather than additional inference context.Source: §5.1 · p. 5, Table 1 · p. 6

5.2 AGENTIC RETRIEVAL

• In Figure 1, ACARe has higher mean nDCG@10 than the base reranker in all 15 agent–dataset combinations and fewer searches in 14.Eight combinations retain significant gains in at least one quality metric after correction.Source: Figure 1 · p. 6, §5.2 · p. 6
• Across five agents, ACARe gains 0.9–2.0 pp each in dataset-averaged nDCG@10 and Recall@10 over Qwen3-Reranker-4B, with 0.3–1.0 fewer searches per query.The agentic evaluation uses BM25 on BCP, Nemotron-3-Embed-8B on BRIGHT, and AgentIR-4B on TRQA—the retrievers with the smallest one-shot nDCG@10 improvements over no reranking.Source: §5.1 · p. 5, §5.2 · p. 6, Table 2 · p. 7
• On TRQA, the base reranker lowers mean nDCG@10 for four agents, whereas ACARe improves all five; ACARe’s nDCG@10 gains are significant for all five.ACARe also improves Recall@10 significantly for four agents; searches fall for four agents, while Terra’s small increase is nonsignificant.Source: §5.2 · p. 7
• Both rerankers reduce mean agent API cost versus no reranking; ACARe saves an additional 12% over Qwen3-Reranker-4B for Nemotron-3 Ultra.That additional reduction is statistically significant; differences between rerankers for the other four agents are not statistically significant across five runs.Source: §5.2 · p. 7

6.1 LATENCY AND RERANKING DEPTH

• At reranking depths 50 and 100, ACARe reduces mean end-to-end latency by 7–8% versus Qwen3 while improving nDCG@10, though both rerankers remain slower than no reranking.The latency reduction at depth 50 and quality gains at depth 100 are statistically significant.Source: §6.1 · p. 7, Figure 2 · p. 8

6.2 EFFECT OF INFERENCE CONTEXT

• On BCP with Nemotron-3 Ultra and BM25, ACARe M3 reaches 72.6 nDCG@10; ACARe M0 scores 70.8 versus 69.7 for Qwen3-Reranker-4B, and all five metrics improve significantly over Qwen3-Reranker-4B and no reranking.Compared with M0, M3 reduces searches from 12.7 to 11.6 per query and has the highest mean Recall@10 and lowest mean API cost among the four modes.Source: §6.2 · pp. 7–8

6.3 SMALL VS. LARGE AGENTS

• ACARe closes 73%, 74%, and 51% of the nDCG@10 gap for Luna, Flash-Lite, and Haiku, versus 46%, 46%, and 26% with Qwen3.When both agents use ACARe, the smaller agents incur 8%, 28%, and 38% of their larger counterparts’ API cost, with nDCG@10 scores 4.6, 13.4, and 15.5 pp lower, respectively.Source: Figure 3 · p. 8, §6.3 · p. 8

6.4 TRANSFER TO END-TO-END ANSWER GENERATION

• ACARe significantly improves accuracy and precision over no reranking for both agents, with the fewest mean searches across six runs.With Luna, ACARe matches Qwen3’s mean accuracy with significantly higher precision.Source: §6.4 · p. 9, Table 4 · p. 9
• With identical M3 inputs, ACARe reaches 57.6% mean accuracy versus 55.9% for Qwen3, but final gains are not significant after correction.ACARe leads on all four metric means; its 1.7 pp accuracy and 1.0 pp precision gains are not significant after correction.Source: Table 5 · p. 9, §6.4 · p. 9

7 CONCLUSION

• ACARe is a pointwise LLM reranker trained to use agent context for search-agent sub-queries, improving mean retrieval quality across three benchmarks and five LLM families while reducing search effort.The conclusion describes it as the first such reranker to the authors’ knowledge.Source: §7 · p. 9
• On TRQA, ACARe improves context-aware AgentIR’s results for all five agents, while Qwen3 lowers mean nDCG@10 for four.Source: §7 · p. 9
• On BCP, smaller agents close 51–74% of the nDCG@10 gap to larger agents without reranking, versus 26–46% with Qwen3.Source: §7 · p. 9
• ACARe’s benefits extend to answer generation through a BM25 API over 553 million documents.Source: §7 · p. 9
• Independent, parallelizable scoring makes deeper reranking practical to explore with this lightweight reranker.Source: §7 · p. 10

8 LIMITATIONS AND FUTURE WORK

• Training retains the original DR-Synth seed questions, agent LLM, and oracle reranker; varying these components could broaden supervision.The data expansion adds an agent rollout with a second retriever.Source: §8 · p. 10
• Evaluation covers two existing deep-search datasets with fixed corpora, BCP and TRQA; additional datasets and live-web search APIs remain future work.Source: §8 · p. 10
• Cost analysis covers agent API usage and assumes agent LLMs are substantially larger and costlier than retrieval tools; total serving-cost measurement remains needed.Further reducing agent size toward the 4B reranker could make reranking costs non-negligible.Source: §8 · p. 10

• ACARe may inherit biases and other limitations from its Qwen3-Reranker-4B base model.Source: ETHICS STATEMENT · p. 10
• Relevance fine-tuning does not ensure that highly ranked documents are accurate, fair, or safe.Source: ETHICS STATEMENT · p. 10
• Downstream applications should account for these risks when using retrieved evidence.Source: ETHICS STATEMENT · p. 10

• Claude and OpenAI Codex assisted with drafting, literature review, code development, and analysis checks.Source: AI-USE STATEMENT · p. 10
• LLMs generated synthetic training trajectories, provided relevance supervision, and judged answers.Source: AI-USE STATEMENT · p. 10
• The authors report reviewing and vetting all manuscript sections, claims, citations, and analyses through multiple revision rounds.Source: AI-USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The reproducibility materials document data construction, model training, evaluation configurations, cost accounting, aggregation, variability estimates, and statistical tests.Source: REPRODUCIBILITY STATEMENT · p. 10
• The authors state that the trained checkpoint, training data, and model-training and evaluation code will be released soon.Source: REPRODUCIBILITY STATEMENT · p. 11

A TRAINING DETAILS

• The training procedure includes a question-level validation split for development, followed by one full-pool training run with the recipe fixed.Hyperparameters and the data mixture were selected using validation LCE loss, positive-first accuracy, and MRR.Source: A TRAINING DETAILS · p. 14
• The new rollout produced 276 correct trajectories out of 500 (55.2%), compared with 249 (49.8%) in the original DR-Synth rollout.The difference is observed between the rollouts; repeated-rollout variability was not estimated, and only successful trajectories contribute training instances.Source: A TRAINING DETAILS · p. 15
• The training pool contains 16,601 instances over 54,838 distinct passages; 15.7% of re-mined instances have malformed traces but remain usable without a trace.Source: A TRAINING DETAILS · p. 15

B EXPERIMENTAL SETUP DETAILS

• Evaluation samples are reproducible using seed 42, with the selection seed and selected query list planned for release alongside the evaluation code.Source: B EXPERIMENTAL SETUP DETAILS · p. 18

C PER-DATASET ONE-SHOT RESULTS

• One-shot results are reported by dataset, while agentic results are broken down by benchmark and first-stage retriever.The cited table descriptions identify BRIGHT and TRQA one-shot results and agentic results for BCP, BRIGHT, and TRQA.Source: C PER-DATASET ONE-SHOT RESULTS · p. 22, Table 10 · p. 23, Table 11 · p. 24

D PER-BENCHMARK AGENTIC RESULTS

• The benchmark breakdown reports that Qwen3-Reranker-4B lowers mean TRQA nDCG@10 for four agents, while ACARe improves it for all five.Source: D PER-BENCHMARK AGENTIC RESULTS · p. 24, Table 13 · p. 25, Table 14 · p. 25

E COST ANALYSIS

• Reported API costs cover agent LLM charges, excluding retrieval and reranker serving costs; these exclusions may matter more for cheaper agents.Source: E COST ANALYSIS · p. 26
• Cost variation was not separated into document timing and provider-cache effects, which can change repeated input charges even at similar token volumes.Source: E COST ANALYSIS · p. 26
• ACARe uses significantly fewer search calls than Qwen3-Reranker-4B for every agent, but significantly lowers API cost only for Nemotron-3 Ultra, by approximately 12%.Sonnet’s 3% mean saving does not remain significant after correction; the other three agents’ cost differences are also nonsignificant.Source: E COST ANALYSIS · p. 26
• ACARe has higher mean API cost than Qwen3-Reranker-4B in seven agent-benchmark cases, so search-effort gains do not imply uniform cost improvements.Only GPT-5.6 Terra’s two increases remain significant after correction.Source: E COST ANALYSIS · p. 26

F LATENCY AND RERANKING DEPTH

• At depth 50, ACARe lowers mean end-to-end latency from 90.8 to 84.1 s (p = 0.025); at depth 100, it lowers latency from 99.5 to 91.6 s (p = 0.110).Only the depth-50 latency reduction is significant under the study’s correction.Source: F LATENCY AND RERANKING DEPTH · p. 27, Table 15 · p. 27
• At depth 100, ACARe raises nDCG@10 from 69.7 to 72.6 (p = 0.025) and Recall@10 from 72.2 to 75.2 (p = 0.038).Both depth-100 quality gains are significant under the study’s correction.Source: F LATENCY AND RERANKING DEPTH · p. 27, Table 15 · p. 27

G SMALL VS. LARGE AGENTS

• 73%, 74%, and 51% of the GPT, Gemini, and Claude retrieval-quality gaps were closed by ACARe for smaller agents, versus 46%, 46%, and 26% with Qwen3-Reranker-4B.The comparisons are between deployed model tiers, not isolated parameter-count changes; Gemini and Claude pairs also differ in version or generation.Source: G SMALL VS. LARGE AGENTS · p. 27, Table 16 · p. 28
• With ACARe on both agents, smaller agents scored 4.6, 13.4, and 15.5 pp lower at 8.1%, 27.6%, and 37.5% of larger agents’ API costs.The larger agent with ACARe remained the most effective configuration in each family; these comparisons are descriptive.Source: G SMALL VS. LARGE AGENTS · p. 28

H RETRIEVER HEADROOM ON BRIGHT

• With Qwen3-Embedding-4B on BRIGHT, ACARe’s advantages over Qwen3-Reranker-4B rose to 0.9 nDCG@10 pp and 1.1 Recall@10 pp, versus 0.2 nDCG@10 pp and tied rounded recall with Nemotron-3-Embed-8B.ACARe’s mean nDCG@10 changed from 42.7 to 42.5 across retrievers, while the authors caution that the comparison does not isolate training-data effects or establish a significant interaction.Source: H RETRIEVER HEADROOM ON BRIGHT · p. 29, Table 17 · p. 29

I AGGREGATION AND STATISTICAL TESTS

• Welch tests use independent run-level values and BY correction within comparison families, but five or six runs make test assumptions difficult to assess and do not establish generalization to other questions.Nonsignificance indicates insufficient evidence of a difference, not equality.Source: I AGGREGATION AND STATISTICAL TESTS · pp. 29–30
• All five agents retained significant aggregate gains in both quality metrics, fewer searches, and fewer LLM turns; aggregate cost was significantly lower only for Nemotron-3 Ultra.In benchmark breakdowns, significant nDCG@10 gains remained for Ultra and Flash on BCP and all five agents on TRQA.Source: I AGGREGATION AND STATISTICAL TESTS · p. 30

J.1 TRANSFER TO CLIMBMIX

• The ClimbMix transfer study evaluated Luna with ACARe M2 and Lightning with M3; 19 Lightning executions without final answers counted as incorrect while retaining retrieval and efficiency measurements.The study used BM25 over ClimbMix’s 553-million-document collection and six runs per agent and retrieval configuration.Source: §J.1 · p. 32

J.2 BASE-RERANKER CONTEXT CONTROL ON BCP

• On BCP, ACARe M3 had the best means across accuracy, recall, precision, and searches, but its differences from Qwen3 M3 were not significant after correction.Accuracy was 57.6% versus 55.9%, recall 65.1% versus 63.9%, precision 21.2% versus 20.1%, and searches 8.4 versus 8.7; all 200 questions remained in each metric.Source: §J.2 · p. 33, Table 20 · p. 33

J.3 SHARED AGENT AND JUDGE PROMPTS

• All retrieval configurations shared Piika’s two-tool workflow, and the answer judge used a reference-based prompt with GPT-5.5 at low reasoning effort.The prompts substitute the question, final agent response, and reference answer into braced placeholders.Source: Figure 11 · p. 34, Figure 12 · p. 35