PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests

Jingjie Ning, João Coelho, Yibo Kong, Yunfan Long, Bruno Martins, João Magalhães, Jamie Callan, Chenyan Xiong

arXiv:2601.17617v3cs.IRcs.CL

TL;DR

Agentic search lacks empirical evidence about session dynamics and how retrieved evidence appears in later queries. This paper analyzes 14.44M DeepResearchGym requests, annotates intents and reformulations, and introduces CTAR to measure lexical traceability. It finds short, intent-dependent sessions with substantial cross-step evidence traceability, while noting that API logs do not reveal success or client-side control.

  • Problem

    The field lacks empirical understanding of how agentic search sessions unfold, vary by intent, and reflect retrieved evidence in later queries.

  • Method

    The paper analyzes sessionized DeepResearchGym logs, applies intent and trajectory annotations, and introduces CTAR to measure lexical traceability to retrieved evidence.

  • Results

    Agentic search sessions are generally short, behavior varies by intent, and newly introduced query terms are often lexically traceable to retrieved evidence across steps.

  • Takeaways & Limitations

    The findings provide candidate signals for repetition-aware stopping, intent-adaptive retrieval budgeting, and cross-step context tracking.

  • Takeaways & Limitations

    The observational API-level logs lack success labels and client-side prompts, memory, and control policies, so the patterns do not establish effects on answer quality.

Abstract

from arXiv · show

LLM-powered search agents are increasingly being used for multi-step information seeking tasks, yet the IR community lacks empirical understanding of how agentic search sessions unfold and how retrieved evidence is reflected in later queries. This paper presents a large-scale log analysis of agentic search based on 14.44M search requests (3.97M sessions) collected from DeepResearchGym, i.e., an open-source search API accessed by external agentic clients. We sessionize the logs, assign session-level intents and step-wise query-reformulation labels using LLM-based annotation, and propose Context-driven Term Adoption Rate (CTAR) to quantify whether newly introduced query terms are lexically traceable to previously retrieved evidence. Our analyses reveal distinctive behavioral patterns. First, over 90\% of multi-turn sessions contain at most ten steps, and 89\% of inter-step intervals fall under one minute. Second, behavior varies by intent. Fact-seeking sessions exhibit high repetition that increases over time, while sessions requiring reasoning sustain broader exploration. Third, query reformulations are often traceable to retrieved evidence across steps. On average, 54\% of newly introduced query terms appear in the accumulated evidence context, with additional traceability to earlier steps beyond the most recent retrieval. These findings provide candidate signals for repetition-aware stopping, intent-adaptive retrieval budgeting, and explicit cross-step context tracking. We released the anonymized logs, making them available at a public HuggingFace~\chref{https://huggingface.co/datasets/cx-cmu/deepresearchgym-agentic-search-logs}{repository}.

1 Introduction

Agentic search requires empirical analysis of how sessions unfold, how behavior varies by intent, and how later queries reflect retrieved evidence. This paper addresses that gap with large-scale logs, intent and trajectory annotations, and evidence-traceability analysis.

  • Benchmark scores do not reveal how agents’ queries evolve across steps or how retrieved context is reflected in later queries.
  • The study analyzes session-level intent and step-wise query reformulation to characterize what agents pursue and how they search.
  • 14.44M search requests become 3.97M sessions from DeepResearchGym, providing an at-scale view of autonomous agents using a shared retrieval backend.
  • Over 90% of multi-turn sessions contain at most ten steps, while 89% of inter-step intervals fall under one minute.
  • Fact-seeking sessions show increasing repetition, whereas reasoning sessions sustain broader exploration across steps.
  • CTAR measures evidence-conditioned query evolution and supports candidate signals for repetition-aware stopping, intent-adaptive budgeting, and cross-step context tracking.

2 Related Work

Prior work established benchmarks, infrastructures, and large-scale interaction-log analysis, but benchmark scores provide limited visibility into real agentic search behavior. This paper positions session-level log analysis as a way to study query evolution and context use in practice.

  • Human search-log research studies within-session learning, query reformulation, and implicit feedback such as clicks and dwell time.
  • Large-scale LLM interaction datasets characterize real-world conversations and usage patterns across systems and evaluation platforms.
  • Agentic-search research includes tool-use benchmarks, planning systems, behavior taxonomies, and reproducible infrastructures such as DeepResearchGym.
  • Benchmark scores alone provide limited visibility into how agents search in practice, motivating behavioral measurement from real session traces.

3 Data and Log Processing

The study uses DeepResearchGym’s shared retrieval infrastructure to transform timestamped API logs into sessions and assess their scale, diversity, benchmark concentration, and reproducibility. The resulting stream is broad and not dominated by a small set of repeated or benchmark-derived queries.

  • 3.1 DRGym Log Overview: DeepResearchGym is a model-agnostic retrieval backend whose logs expose queries, parameters, timestamps, and returned evidence but not client-side prompts or policies.
  • 3.1 DRGym Log Overview: The backend performs dense retrieval over ClueWeb22-A-EN and FineWeb using a static web snapshot for consistent experiments.
  • 3.1 DRGym Log Overview: 14.44 million requests produce 3.97M sessions from 558 anonymized client IPs across 25 countries, with weekly traffic peaking at 2.49 million requests.
  • 3.1 DRGym Log Overview: Mean pairwise query cosine similarity is 0.12, close to the random-vector baseline of approximately 0, indicating semantic diversity.
  • 3.1 DRGym Log Overview: 53.89% of distinct queries occur at most three times, including 38.38% singleton queries; the top-100 queries account for 1.51% of requests.
  • 3.1 DRGym Log Overview: Benchmark-similar queries constitute less than 0.4% of the sample across four benchmarks, suggesting the logs reflect diverse open-ended usage.
  • 3.3 Data Release: The cleaned, anonymized logs are publicly released with direct identifiers removed and the fields needed to reproduce the analyses retained.
  • 3.2 Preprocessing and Sessionization: Sessionization removes malformed, internal, and burst-outlier entries, then combines semantic continuity with a temporal constraint to assign requests to sessions.

4 Methodology

The paper models agentic search at session and trajectory levels, using LLM-based labels and CTAR to measure how later query terms relate lexically to retrieved evidence.

  • Measurement framework: Session-level intent captures the information need, while trajectory-level reformulation captures how adjacent queries change during search.The framework distinguishes Declarative, Procedural, and Reasoning intents and labels adjacent transitions such as narrowing, broadening, exploration, and repetition.
  • Data and annotation: 14.44M search requests are organized into timestamped sessions and analyzed globally, by intent, and across adjacent query pairs.The logs come from DeepResearchGym, and the annotation pipeline focuses on multi-turn sessions with two to ten queries.
  • Evidence traceability: CTAR measures the fraction of newly introduced query terms that can be lexically traced to retrieved evidence.It uses exact-match tracing over extracted unique, lowercased, non-stopword tokens rather than semantic similarity.
  • Evidence traceability: CTAR_last traces new terms to the immediately preceding evidence, whereas CTAR_agg traces them to evidence from any prior step.Comparing these contexts measures additional lexical overlap from earlier evidence without assuming that the agent causally used or retained it.

5 Aggregate Session Statistics

Aggregate statistics show that agentic sessions are usually short, fast, and governed by largely fixed retrieval depth, while multi-turn analyses describe only sessions continuing beyond one request.

  • Session structure: 47.77% of sessions contain one query, while 90% of multi-turn sessions have length ≤10.The analysis excludes single-query sessions from trajectory metrics, so its findings may over-represent complex, uncertain, or iterative information needs.
  • Temporal dynamics: Human logs report 1.7 queries per session and 77.6% single-query sessions, but the paper cautions that these measures do not directly establish task success or difficulty.The comparison is used only to characterize the faster, more extended pacing of agentic search.
  • Temporal dynamics: 89.21% of inter-step intervals are under one minute, including 56.12% within 0–10 seconds.Step latency includes inference, batching, network delay, and scheduling, so it is treated as a pacing signal rather than direct deliberation time.
  • Retrieval depth: Only 1.35% of sessions vary retrieval depth across steps, with depth concentrated at K∈{1, 5, 10}.Because the backend supports 1 ≤ K ≤ 100, the pattern suggests retrieval count is often treated as hard-coded rather than adapted within a session.

6 Intent-Conditioned Session Behavior

Search behavior differs by intent: Declarative sessions repeat and incur high interaction costs, Procedural sessions pair deeper retrieval with stable progress, and Reasoning sessions explore through larger query drift.

  • Intent distribution: 88.64% of multi-turn sessions are Declarative, followed by Reasoning at 7.41% and Procedural at 3.96%.The taxonomy treats these categories as fact-seeking, complex synthesis, and method-execution tasks, respectively.
  • Declarative: Declarative sessions use the shallowest retrieval yet incur the highest interaction costs.The pattern is consistent with more iterations when per-step retrieval is shallow, although the logs do not determine whether extra steps improve verification.
  • Procedural: Procedural sessions pair deeper retrieval with more semantically stable progression and longer queries than Declarative sessions.This pattern is consistent with broader evidence coverage co-occurring with fewer refinement steps.
  • Reasoning: Reasoning sessions match Declarative sessions in turn count but show the largest semantic drift, longest queries, and moderate retrieval depth.Their distinguishing signal is within-session reformulation rather than duration or retrieval depth.

7 Trajectory Moves and Topologies

Agentic search trajectories combine refinement, exploration, and repetition, with move frequencies and stability patterns varying by intent and evolving across session steps. Query reformulations also show substantial lexical traceability to retrieved evidence, especially for specialization and exploration.

  • Trajectory Moves and Topologies: Agents classify adjacent query pairs as Specialization, Generalization, Exploration, or Repetition, using similarity and result-overlap measures to characterize trajectory stability.Specialization narrows constraints, Generalization relaxes them, Exploration pivots within a topic, and Repetition produces identical or near-duplicate reformulations.
  • Trajectory Moves and Topologies: Under 11% of moves are Generalization, while Exploration accounts for roughly 36–48%, indicating a strong preference for local refinement and facet pivots over broadening.Exploration also causes greater evidence turnover and slower transitions than incremental refinement.
  • Intent Differences: Declarative sessions show about one-third Repetition, Reasoning sessions nearly 48% Exploration, and these intent differences correspond to retry-like versus broader sub-question search.Procedural sessions occupy a different mixture of refinement, pivoting, and repetition.
  • Stability as a Diagnostic: Repetition preserves retrieved results at approximately 78% Result Overlap, whereas Exploration yields approximately 7%, making sustained stability a diagnostic of retry-like behavior.Specialization and Generalization fall between these extremes.
  • Pacing Implications: Exploration has a 14.0s median inter-request interval versus 6.0s for Repetition, linking broader pivots with higher processing cost.This pacing difference makes the choice between local edits and alternative moves consequential for retrieval strategy.
  • Trends over Steps: Across steps, Declarative sessions increasingly retry, Procedural sessions emphasize refinement, and Reasoning sessions sustain pivoting with low retrying.Exploration and Repetition often form multi-step runs, while nearly half of Generalization moves are followed by Specialization.
  • Context-Driven Term Adoption Rate: Mean CTAR is 54.35% overall, with aggregated context adding 5.81 percentage points over last-step evidence and CTAR reaching 78.35% for Specialization versus 20.92% for Repetition.Exploration also has high aggregated CTAR at 69.59%; CTAR is lexical and does not establish causal evidence use.
  • Context-Driven Term Adoption Rate: CTAR is strongest for recent evidence but remains non-trivially traceable to earlier steps, supporting lightweight auditing of cross-step evidence consistency.The measure counts lexical overlap, so semantic paraphrases and abstractions are excluded.

8 Discussion, Implications, and Limitations

The discussion frames trajectory and CTAR patterns as observational diagnostic signals rather than evidence of downstream success, while identifying candidate interventions for repetition, retrieval budgeting, and context management.

  • Scope and Limitations: The analysis is observational and lacks downstream success labels, so its patterns should be treated as diagnostics and hypotheses rather than evidence that behaviors improve or harm answer quality.Client-side prompts, memory, and control policies are also unobserved.
  • Repetition as a Candidate Stall Signal: Declarative repetition reaches 42.68% by Step 9, making sustained lexical overlap a candidate signal for testing broader queries, alternative moves, or human review.The logs cannot determine whether repetition reflects stalled search, verification, cautious checking, or client-side constraints.
  • Intent-Adaptive Resource Allocation: 91.64% of requests use K∈{1, 5, 10}, suggesting largely static retrieval depth despite intent-dependent usage differences and motivating intent-aware budgeting tests.Procedural sessions use deeper retrieval than Declarative sessions in the observed logs.
  • Evidence Grounding as an Audit Signal: Aggregated context raises CTAR by 5.81 pp over last-step context, but the Specialization–Repetition contrast reflects co-occurrence rather than causal evidence that low CTAR causes retries.CTAR can support lightweight auditing for future context-management modules.

9 Conclusions

The paper concludes that agentic search exhibits measurable intent-conditioned trajectories and lexical evidence traceability in API logs. These patterns offer diagnostic structure for future control and evaluation, but not evidence of downstream success.

  • Conclusions: The 14.44M-request analysis shows that multi-step agentic search has measurable intent-conditioned reformulation patterns even from API-level traces.The study converts raw DRGym logs into sessions and analyzes session structure, query transitions, and evidence traceability.
  • Conclusions: Declarative sessions are more retry-like, while Procedural and Reasoning sessions mix refinement, exploration, and repetition differently, with high-stability runs especially visible in Declarative tasks.These patterns help distinguish local edits, facet pivots, and near-duplicate returns within sessions.
  • Conclusions: CTAR finds many new query terms lexically traceable to returned evidence, strongest for recent retrieval and weaker but non-trivial for earlier steps.It is a lightweight audit signal and does not establish causal evidence use or capture semantic paraphrases.
  • Implications and Future Work: The findings motivate future controller tests for high-stability loops, broader queries, facet pivots, adaptive retrieval budgeting, and context management.The paper also releases the dataset and analysis protocol to support reproducible evaluation.

A Log Sessionization Procedure

The sessionization pipeline trains a semantic continuity model and then assigns incoming queries online to per-IP sessions using continuity scores and a temporal cutoff.

  • Sessionization Pipeline: The pipeline trains a semantic continuity model from approximately 200K same-IP, nearest-in-time query pairs labeled by an LLM judge.Qwen3-Embedding-0.6B embeddings form fixed dense pair features for the model.
  • Sessionization Pipeline: Queries are processed chronologically per IP and assigned to the highest-scoring active session when its continuity score is at least 0.5; otherwise a new session starts.The procedure compares each query with the most recent query in each active session.
  • Sessionization Pipeline: A gap exceeding 10 minutes triggers a new session even when the continuity score would otherwise support assignment.This temporal hard cutoff supplements the semantic continuity decision.

B Auxiliary Metric Definitions

This section defines the auxiliary metrics used in the analyses and provides their formal notation and formulas for reproducibility.

  • The auxiliary metrics are formally defined and summarized in Table 7 for reproducible analysis.
  • For each session, queries are represented as an ordered sequence, with dense embeddings and cosine similarity used in the notation.
  • The notation also defines normalized query-token sets and retrieved-evidence sets at the logged retrieval depth.

C Representative Query Examples

This section provides representative examples and the prompts used to label session intents and step-wise query reformulations in agentic search trajectories.

  • Table 8 presents representative queries for the paper’s intent categories, while Table 9 illustrates step-wise transitions between queries.
  • Intent Categories: Query pairs are judged as belonging to the same research task when they naturally support one user question, even if they cover different aspects or detail levels.
  • Intent Categories: Session intent is classified as Declarative, Procedural, or Reasoning based on the information need expressed across session queries.
  • Trajectory Categories: Step-wise reformulations are labeled Specialization, Generalization, Exploration, or Repetition according to how the current query changes relative to the previous query.
  • The paper identifies itself as Agentic Search in the Wild and positions the examples within its SIGIR ’26 publication context.
Loading 2601.17617v3…