Source-linked AI summary
Critic-R: Improving Agentic Search using Instruction-tuned Retrievers with Natural Language Introspective Feedback
Md Zarif Ul Alam, Alireza Salemi, Hamed Zamani
TL;DR
Agentic search often relies on frozen retrievers or expensive co-training and gold-passage supervision, leaving retrieval failures insufficiently addressed. Critic-R closes the reasoning–retrieval feedback loop with inference-time query refinement and trajectory-supervised retriever training, achieving substantial gains across multi-hop QA benchmarks. The authors report improvements for both individual mechanisms and their combination.
Problem
Existing agentic search methods often treat retrieval as frozen, while joint optimization can require retriever or reasoning-model training and gold-passage supervision.
Method
Critic-R uses a critic conditioned on the agent’s introspective trace to refine queries at inference time and train a retriever from successful and failed refinement trajectories.
Results
10.9% relative improvement overall is achieved by combining Critic-R-Zero with Critic-Embed, with additional gains reported for each component across multi-hop QA benchmarks.
Takeaways & Limitations
Explicitly modeling and optimizing retrieval quality within the agentic loop improves downstream task accuracy and provides a path toward more robust agentic search.
Takeaways & Limitations
The critic depends on reliable introspective feedback and has primarily been evaluated with multi-hop QA over a static Wikipedia corpus, not dynamic or private document environments.
Abstract
from arXiv · showhide
Agentic search systems iteratively interact with retrieval models to answer complex queries. Despite substantial progress, optimizing retrievers for agentic search remains challenging, often requiring heavy co-training or gold-standard annotations that limit real-world applicability. We propose Critic-R, a framework that explicitly closes the feedback loop between the reasoning agent and the retrieval model during both inference and training. Critic-R introduces a critic model that evaluates the agent's introspective reasoning trace after consuming retrieved evidence to determine whether the retrieved context sufficiently supports the next reasoning step. Critic-R has two complementary mechanisms: Critic-R-Zero, an inference-time query refinement loop that iteratively rewrites queries and retrieval instructions, and Critic-Embed, an optimization approach for retrieval models that leverages successful and failed refinement trajectories as automatic supervision without requiring manual relevance annotation. We evaluate Critic-R on HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle. Results show that Critic-R significantly improves both retrieval quality and downstream answer accuracy.
1 Introduction
Critic-R treats retrieval quality as a bottleneck in agentic search and closes the feedback loop between reasoning and retrieval during inference and training. Its refinement and retriever-optimization mechanisms improve retrieval and downstream answering across multi-hop QA benchmarks.
- Agentic search often treats retrieval as a frozen component, even though sub-optimal retrieval can bottleneck performance and existing joint-training methods may require unavailable training access or gold passages.
- Critic-R uses a separate critic to assess whether retrieved context supports the reasoning agent’s next step before subsequent retrieval or reasoning.
- Critic-R-Zero iteratively rewrites retrieval queries and instructions when evidence is insufficient, without modifying or retraining the reasoning agent.
- Critic-Embed uses successful retrievals as positives and rejected documents as hard intra-trajectory negatives, avoiding manual relevance annotations.
- 12.4% overall relative improvement is reported for Critic-R-Zero across challenging multi-hop question-answering benchmarks.
- 7.5% overall relative improvement is achieved by Critic-Embed over off-the-shelf retrievers and prior co-trained baselines.
- 10.9% relative improvement overall is achieved when the trained retriever and inference-time refinement loop are combined into Critic-R.
2 Related Work
Related work addresses agentic search through reasoning-agent optimization, retrieval optimization, or additional inference-time computation. Critic-R instead targets retrieval failures with an external refinement loop and uses its trajectories for retriever training.
- Retrieval-Augmented Generation and Agentic Search: RAG and agentic search evolved from single-query retrieval toward interleaving reasoning with retrieval for multi-hop questions whose information needs emerge during reasoning.
- Retrieval-Augmented Generation and Agentic Search: Critic-R is orthogonal to Search-R1’s agent-training objective because it inspects and repairs individual retrieval calls at inference time.
- Retrieval optimization for Agents: Prior retrieval-optimization methods use generator likelihood, task metrics, or LLM-judged passage utility as training signals.
- Retrieval optimization for Agents: Agentic-R jointly models retrieval relevance and answer correctness, while CoSearch jointly trains a generative reranker and reasoning agent using a composite reward.
- Retrieval optimization for Agents: Critic-R-Zero differs by requiring no gradient updates: an external critic evaluates retrieved context using agent feedback and rewrites search instructions and queries when needed.
- Inference-Time Scaling for Reasoning: Inference-time scaling can allocate extra computation through longer reasoning, self-consistency, or process supervision; Critic-R-Zero directs that investment specifically toward recovering from bad retrievals.
3 The Critic-R Framework
The Critic-R framework combines Critic-R-Zero’s speculative retrieval-refinement loop with Critic-Embed’s trajectory-based contrastive retriever training. A critic uses the reasoner’s introspective feedback to accept useful evidence, reject failed retrievals, and guide further searches.
- Critic-R-Zero: The framework uses a frozen ReAct reasoning agent that alternates reasoning and actions, with search actions initiating retrieval and final answers terminating the trajectory.
- Critic-R-Zero: An instruction-aware retriever returns top-k documents for each query, but initial single-turn retrieval frequently fails to satisfy the reasoner’s information need.
- Critic-R-Zero: Critic-R-Zero speculatively presents retrieved documents to the reasoner, then has a separate critic judge whether they resolve the current information need.
- Critic-R-Zero: Accepted documents enter the positive set, while documents from unsuccessful refinement attempts enter the candidate negative set.
- Critic-R-Zero: When evidence is rejected, the critic uses the reasoner’s dissatisfaction trace to generate a refined query and retrieval instruction for another attempt, up to a refinement limit.
- Critic-Embed: Critic-Embed fine-tunes the retriever from successful and failed refinement trajectories, retaining only trajectories whose final predictions are correct.
- Critic-Embed: The contrastive objective pairs each query with a positive document and contrasts it against in-batch negatives and intra-trajectory hard negatives using cosine similarity and a temperature.
- Unified Critic-R: The complete Critic-R system combines a stronger trained retriever with inference-time introspection to reduce initial search errors while recovering from complex retrieval failures.
4 Experiments
The experiments test Critic-R-Zero's inference-time refinement, Critic-Embed's transferable supervision, and their evaluation across multi-hop QA settings. Results compare critic scales, retrieval depths, retriever baselines, and refinement trajectories under controlled protocols.
- Retriever comparison: Critic-R-Zero trajectories provide transferable supervision: accepted documents become positives and rejected documents become intra-trajectory hard negatives for retriever training.The training data includes roughly 11K natural contrastive pairs and 67K positive-only samples, and Critic-Embed is trained with an InfoNCE objective.
- Datasets: The evaluation covers HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle, with additional Critic-R-Zero experiments on NQ, TriviaQA, and PopQA.The multi-hop datasets require synthesizing information across multiple documents, while the additional datasets test single-hop retrieval settings.
- Research questions and setup: Critic-R-Zero evaluates inference-time refinement by varying reasoner and critic scale while holding the frozen retriever, retrieval depth, and refinement budget fixed.The main protocol uses Search-R1, frozen Stella-400M retrieval at top k=1, and K=2 refinement attempts.
- Critic-R-Zero results: Any critic strictly improves over the no-critic ablation in every reasoner, dataset, and metric cell, including the difficult Bamboogle and MuSiQue datasets.The reported comparison attributes the lift to the critic's verdict and instruction rewrite rather than extra forward passes alone.
- Critic-R-Zero results: Larger critics are not uniformly better: scaling from 32B to 72B yields diminishing returns and occasional degradation, while 32B is optimal for some harder settings.For the 7B reasoner, the 32B critic averages 0.3293 / 0.4176 EM/F1 versus 0.3192 / 0.4108 for the 72B critic.
- Retriever comparison: Critic-Embed is the best-performing retriever in every tested setting, retaining the lead at top k=1, 3, and 5 under the same Search-R1 agent.At top k=1 on Bamboogle, Critic-Embed reaches 0.4480 / 0.5872 EM/F1, compared with 0.3520 / 0.4963 for Stella-400M and 0.4240 / 0.5260 for Agentic-R.
RQ3: Does combining the inference-time loop and the trained retriever yield further gains?
Combining Critic-R-Zero's inference-time refinement loop with the Critic-Embed retriever produces the strongest overall configuration, although their relative advantage varies by dataset.
- The four-way comparison uses one Search-R1 (14B) reasoner, top k = 1, and, when active, a Qwen2.5-72B critic with K =2 refinement attempts.
- 0.3957 EM / 0.4959 F1 is the best multi-hop average for the combined Critic-R system.It exceeds both Critic-Embed alone and Critic-R-Zero alone under the same Search-R1 (14B) reasoner and top k = 1 setting.
- Critic-Embed alone improves the static Stella-400M baseline from 0.3472 / 0.4470 to 0.3794 / 0.4806 EM/F1 without test-time refinement.
- Critic-R-Zero on the frozen backbone reaches 0.3903 / 0.4855 EM/F1, modestly ahead of Critic-Embed alone on this configuration.
- The per-dataset comparison is mixed: Critic-R leads on Bamboogle and 2Wiki, while Critic-R-Zero leads on HotpotQA and Musique.
5 Conclusion
The conclusion presents Critic-R as a framework that treats retrieval as a bottleneck in agentic search and combines inference-time refinement with retriever optimization. Across multi-hop QA benchmarks, the combined system improves downstream task accuracy.
- Critic-R addresses retrieval as a critical bottleneck by evaluating retrieved evidence against the reasoning agent's introspective trace.
- Critic-R-Zero iteratively refines queries and retrieval instructions, while Critic-Embed fine-tunes the retriever on contrastive trajectories without manual relevance annotation.
- Across challenging multi-hop QA benchmarks, the combined Critic-R system achieves substantial improvements in downstream task accuracy.
Limitations
Critic-R depends on reliable introspective feedback from the reasoning agent and has been evaluated mainly on static Wikipedia-based knowledge-intensive QA. Its behavior in dynamic, distribution-shifting environments remains unevaluated.
- Critic-R's critic depends heavily on the reasoning agent accurately identifying missing information or evaluating retrieved documents.
- Weaker or smaller language models may produce inaccurate introspective traces that degrade the critic's verification signal.
- Experiments focus on multi-hop and general knowledge-intensive QA with a static Wikipedia corpus, not real-time web search or private enterprise documents.
- The efficacy of Critic-R in dynamic environments with greater corpus noise and distribution shifts has not yet been evaluated.
A General-Domain QA Results
On general-domain QA, Critic-R-Zero follows the same pattern as on multi-hop benchmarks: critics improve over no-critic settings across reasoner scales, with the largest critic typically strongest on average.
- Any critic reliably improves over the no-critic ablation across all reasoner scales on NQ, TriviaQA, and PopQA.
- The Qwen2.5-72B critic typically delivers the strongest average performance among the evaluated critics.
B Implementation Details
Critic-R uses a ReAct reasoning agent whose introspective trace helps a separate critic judge whether retrieved documents support the current sub-query. When evidence is insufficient, the critic diagnoses the gap and refines the retrieval request.
- Reasoning Agent: The reasoning agent emits a thinking trace followed by either a search sub-query or a final answer, while retrieved documents are inserted into its context.The agent is explicitly instructed to verbalize missing or misaligned evidence in its subsequent thinking trace.
- Critic Model: The critic evaluates the original question, current sub-query, retrieved documents, and introspective trace before issuing a binary satisfaction verdict and diagnostic reason.The verdict uses yes/no output, with the reason identifying missing evidence when applicable.
- Query Refinement: After a negative verdict, the critic receives the failed sub-query, instruction, and diagnostic reason to produce refined retrieval instructions and a refined query.Separating judgment from refinement avoids rewriting adequate retrieval results and focuses refinement on the diagnosed evidence gap.
C Dataset Statistics
The experiments report evaluation-set sizes for seven QA datasets spanning multi-hop and general-domain question answering. The dataset-size table covers the benchmark splits used in the study.
- Evaluation Sets: Table 6 reports evaluation-set sizes for seven QA datasets used in the experiments.When an official test split is unavailable, the study uses the dev split; otherwise, it uses the test split.
- Dataset Groups: The first four datasets are multi-hop QA benchmarks, while the final three are predominantly single-hop, general-domain QA benchmarks.The multi-hop suite comprises HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle; the general-domain suite comprises NQ, TriviaQA, and PopQA.
D Critic-Embed Training Details
Critic-R’s implementation uses structured prompts for the reasoning agent and critic, with separate satisfaction-judgment and query-refinement modes. The retrieval experiments include a general-QA scaling configuration for Critic-R-Zero.
- Critic-Embed Training: Critic-Embed uses a Stella-400M embedding model fine-tuned with InfoNCE for five epochs using a temperature of 0.02.Training uses an effective batch size of 128, learning rate 2×10^-5, weight decay 0.01, warmup, gradient clipping, and FP16.
- General-QA Scaling: Table 5 evaluates Critic-R-Zero with a frozen Stella-400M retriever, top-k=1, and K=2 refinement attempts while varying critic size.Bold entries mark the best EM/F1 for each reasoner, dataset, and metric combination.
- Prompting: The reasoning agent follows a ReAct prompt template, while the critic uses distinct prompts for satisfaction judgment and query refinement.Structured tags mark model outputs and runtime-substituted variables in these prompts.
- Satisfaction Judgment: The satisfaction prompt asks the critic to determine whether retrieved documents support the current sub-query using the global query, local query, documents, and reasoning-model feedback.The prompt assigns evaluation as the critic’s only task.
- Query Refinement: The query-refinement prompt is invoked when satisfaction judgment returns no and asks the critic to rewrite the failed sub-query and retrieval instruction.This prompt operationalizes the next retrieval attempt after an unsatisfactory result.