PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah

arXiv:2605.15184v1cs.CL

TL;DR

The paper addresses the limited evidence on how retrieval strategy interacts with agent architecture, tool-result presentation, and increasing irrelevant context. It evaluates grep and vector retrieval across custom and provider-native harnesses in two LongMemEval experiments, finding that retrieval ordering and performance depend on the harness and scaling condition, with conclusions bounded to long-memory conversational QA.

  • Problem

    Existing research has not systematically compared retrieval strategy with agent architecture, tool-calling style, result presentation, and increasing irrelevant surrounding text in agentic search.

  • Method

    The study runs two experiments comparing grep and vector retrieval across Chronos and provider-native CLI harnesses, inline or file-based results, and progressively larger distractor contexts on LongMemEval.

  • Results

    Across the experiment 1 comparisons, grep generally achieves higher accuracy than vector retrieval, while retrieval ordering under scaling varies by harness and backbone.

  • Takeaways & Limitations

    Retrieval performance in agentic search is not stable across harnesses or tool-calling stacks, so static retriever comparisons can understate agent-scaffolding variance.

  • Takeaways & Limitations

    The conclusions are tied to long-memory conversational QA, where answers often contain literal spans, and may differ for paraphrased, visual-heavy, or code-semantic evidence.

Abstract

from arXiv · show

Recent advances in Large Language Model (LLM) agents have enabled complex agentic workflows where models autonomously retrieve information, call tools, and reason over large corpora to complete tasks on behalf of users. Despite the growing adoption of retrieval-augmented generation (RAG) in agentic search systems, existing literature lacks a systematic comparison of how retrieval strategy choice interacts with agent architecture and tool-calling paradigm. Important practical dimensions, including how tool outputs are presented to the model and how performance changes when searches must cope with more irrelevant surrounding text, remain under-explored in agent loops. This paper reports an empirical study organized into two experiments. Experiment 1 compares grep and vector retrieval on a 116-question sample from LongMemEval, using a custom agent harness (Chronos) and provider-native CLI harnesses (Claude Code, Codex, and Gemini CLI), for both inline tool results and file-based tool results that the model reads separately. Experiment 2 compares grep-only and vector-only retrieval while progressively mixing in additional unrelated conversation history, so that each query is embedded in more distracting material alongside the passages that matter. Across Chronos and the provider CLIs, grep generally yields higher accuracy than vector retrieval in our comparisons in experiment 1; at the same time, overall scores still depend strongly on which harness and tool-calling style is used, even when the underlying conversation data are the same.

1 Introduction

Agentic retrieval research has not systematically examined how retrieval strategy interacts with agent architecture and tool-calling design. This study focuses on retrieval, harness, presentation, noise, and cross-harness effects in end-to-end workflows.

  • Existing evaluations largely isolate lexical and dense retrieval from the agent architectures and iterative tool-mediated loops used in modern agentic systems.Standalone pipelines typically concatenate retrieved documents into a prompt rather than modeling agent-directed search and refinement.
  • Retrieval, harness, and presentation: The study examines how lexical versus dense retrieval combines with agent orchestration and inline versus file-based tool-result presentation.
  • Noise and scale: It also characterizes end-to-end behavior as irrelevant surrounding content grows relative to task-relevant material.
  • Heterogeneity across agent stacks: The comparison spans custom and provider-native harnesses while holding the underlying conversation corpus fixed.

2 Overview of Retrieval in Agentic Systems

Agentic retrieval is iterative and agent-directed, combining a retrieval strategy with a harness that manages the tool-calling loop. Retrieval outputs may be delivered inline or through files, creating different context-management demands.

  • Agentic retrieval lets the model choose searches, query counts, and whether results require refinement rather than concatenating fixed top-k results into a prompt.
  • Retrieval strategies: Lexical retrieval matches exact terms or patterns, while dense retrieval uses embedding-space similarity and hybrid methods combine lexical and semantic signals.
  • Agent harnesses: The agent harness constructs prompts, dispatches tools, receives results, and controls whether the loop continues or terminates.
  • Agent harnesses: Custom harnesses provide fine-grained control over prompts, tools, context construction, result formatting, and termination, but require substantial engineering effort.
  • Agent harnesses: Provider-native CLI harnesses expose shell utilities such as grep and cat while keeping context construction and iteration control largely opaque to users.
  • Tool-calling architectures: Inline tool results simplify access but consume context-window space, whereas file-based results defer context consumption until the agent explicitly reads them.

3 Methodology

The study evaluates retrieval strategies on LongMemEval conversation questions using per-question files containing raw dialogue and structured temporal events. Grep and vector tools provide contrasting retrieval mechanisms over the same prepared data.

  • The evaluation uses a 116-question representative LongMemEval subset covering long conversations across multiple sessions.
  • Questions include oracle sessions with answer-bearing information and distractor sessions irrelevant to the query.
  • The benchmark spans knowledge updates, multi-session aggregation, assistant and user recall, preferences, and temporal reasoning.
  • Per-question files serialize dialogue turns with structured temporal events extracted through the Chronos preprocessing pipeline.
  • Grep applies regular-expression matching over raw text in local files without embeddings, vector indexes, or external services.
  • Vector search embeds turns and events, retrieves approximate nearest neighbors, reranks them, and returns an agent-selected top-k set.

3.3 Agent Harnesses

The study compares a custom Chronos harness with provider-native CLI agents under dynamically configured search environments. These harnesses differ in tool access, prompting, and execution context.

  • Chronos: Chronos uses LangChain with grep and vector search over conversation turns and temporal events, plus grep access in programmatic mode.
  • Chronos: Chronos dynamically conditions system instructions, search hints, and tool guidance on the detected question category.
  • Provider-native CLI agents: Three provider-native CLI agents receive the question and generated search strategy, then invoke bash-callable grep and vector wrappers through absolute paths.

3.4 Tool-Calling Architectures

The study compares standard inline tool delivery with programmatic file-based delivery, which shifts how agents access retrieved results and manage context. It also evaluates five LLMs across multiple capability levels using an auxiliary LLM grader.

  • Standard tool calling: Standard mode injects search results directly into the agent’s working context through stdout or tool response messages.Results compete with the system prompt and conversation history for context-window space.
  • Programmatic tool calling: Programmatic mode writes search results to a file that the agent must read or search, decoupling result size from context pressure.This requires the agent to explicitly select which retrieved results to consume.
  • Models: The evaluation covers Claude Opus 4.6, Claude Haiku 4.5, GPT-5.4, Gemini 3.1 Pro, and Gemini 3.1 Flash-Lite.
  • Evaluation: An auxiliary GPT-4o grader evaluates each hypothesis against the question and reference answer using category-conditioned binary judgments.

4.1 Experiment 1: Retrieval Mode, Harness, and Tool Calling Method

Experiment 1 tests how retrieval mode, agent harness, and tool-output delivery jointly affect long-memory QA accuracy on the 116-question LongMemEval-S subset. Inline grep generally outperforms inline vector retrieval, but harness and delivery choices substantially reshape the comparison.

  • Experimental design: The factorial experiment varies grep-only versus vector-only retrieval, Chronos versus provider-native CLI harnesses, and inline versus programmatic tool delivery.Each table row fixes a harness–model pair while varying retrieval mode and tool-calling method.
  • Results: Inline grep exceeds inline vector for every harness–model pair, with the largest margin at 86.2% versus 62.9% for Chronos with Gemini 3.1 Flash-Lite.The narrowest reported margin is 76.7% versus 75.0% for Claude Code with Claude Opus 4.6.
  • Results: The same Claude Opus 4.6 backbone reaches 93.1% under Chronos but 76.7% under Claude Code, showing that harness changes can rival retriever changes.The harness shapes prompting, tool descriptions, hit rendering, query scheduling, and stopping behavior.
  • Results: Programmatic vector retrieval exceeds programmatic grep on five of ten harness–model pairs, so delivery changes can reorder the retrieval comparison.File-based routing introduces a read–integrate–retry stage whose success depends on the agent’s tool-use competence.
  • Discussion: Lexical retrieval may benefit LongMemEval because many answers depend on recovering literal dates, counts, preferences, and spans.This interpretation is presented as a possible explanation for inline grep’s strength, not as a causal attribution established by trace-level analysis.
  • Discussion: Benchmarking only BM25 versus approximate nearest-neighbor retrieval in a static pipeline understates variance introduced by agent scaffolding.
  • Discussion: Programmatic routing trades context bandwidth for compositional tool competence, with gains realized only when agents reliably complete the read–integrate–retry loop.
  • Discussion: Claude Haiku 4.5 on Claude Code shows especially large inline grep–vector gaps, although the proposed explanation remains a hypothesis without trace-level causal attribution.

4.2 Experiment 2: Context Scaling with Increasing Noise

Experiment 2 tests how grep-only and vector-only retrieval behave as irrelevant conversation sessions accumulate around oracle evidence. The results show that retrieval ordering and scaling behavior depend on the harness, model, and sampled distractors rather than following a single monotonic noise trend.

  • Experimental setup: The experiment varies session limits from s5 through s30 and full, with oracle sessions retained and distractors added from the same per-question bundle.Full denotes the complete haystack of 39–66 sessions per item.
  • Grep scaling: Grep accuracy is non-monotonic: Chronos Opus peaks at 90.5% at s20, falls to 85.3% at s30, and reaches 89.7% at full.Claude Code Opus similarly peaks at 95.7% at s20 before reaching 94.0% at full.
  • Harness interactions: Retriever ordering depends on the harness: Claude Code favors grep, Gemini CLI Pro favors vector, and Chronos shows crossings as session limits increase.For example, Chronos Opus is vector-ahead at s5–s20 and full but grep-ahead at the penultimate limit.
  • Scaling behavior: Vector retrieval is often stronger at low session counts, but grep can close or overtake later depending on the backbone and harness.The scaling study therefore treats retrieval behavior as an interaction among distractor sampling, tool transcripts, and the agent’s stopping policy.
  • Interpretation limits: The observed curves should be read as stochastic samples because distractors are redrawn at each session limit, so mid-grid peaks do not establish absolute difficulty ordering.Paired grep–vector comparisons remain matched within each sampled configuration.
  • Interpretation limits: Incomplete Codex scaling rows prevent a vendor-complete comparison of how CLI grep and CLI vector retrieval age under matched distraction caps.The strongest conclusion is conditional: grep–vector ordering is not preserved under scaling, and its shape depends on the observed stack.

5 Limitations

The conclusions are scoped to long-memory conversational question answering, where questions rely on multi-session chat, explicit time expressions, and personal or user facts.

  • The findings are tied to long-memory conversational QA over multi-session chat, explicit time expressions, and personal or user facts.

6 Conclusion

The paper studies retrieval strategy alongside agent architecture, tool-output presentation, and increasing irrelevant context. Across its comparisons, grep generally outperforms vector retrieval, while harness and tool-calling choices strongly affect scores.

  • The study addresses how retrieval strategy interacts with agent architecture and tool-calling paradigms in agentic workflows.
  • Experiment 1 compares grep and vector retrieval across Chronos, Claude Code, Codex, and Gemini CLI with inline and file-based results.The evaluation uses a 116-question LongMemEval sample.
  • Experiment 2 progressively mixes unrelated conversation history with relevant passages to test retrieval under increasing distraction.
  • Across the evaluated harnesses, grep generally achieves higher accuracy than vector retrieval, including for every evaluated inline harness-model pair.
  • Overall scores depend strongly on harness and tool-calling style, with file-based delivery and provider CLI shells able to erase or invert grep’s advantage.

A Per-Category Accuracy

Table 4 reports per-category accuracy for grep-only retrieval in Chronos at the full haystack across inference models.

  • Table 4 reports per-category accuracy (%) on the 116-question subset for Chronos using grep-only retrieval at the full haystack.
  • The table presents results for each inference model, with GPT-4o serving as the grader.
Loading 2605.15184v1…