Source-linked AI summary
InteractComp: Evaluating Search Agents With Ambiguous Queries
Mingyi Deng, Lijun Huang, Yani Fan, Fanqi Kong, Jiayi Zhang, Fashen Ren, Jinyi Bai, Fuzhen Yang, Dayi Miao, Zhaoyang Yu, Yifan Wu, Yanfei Zhang, Fengwei Teng, Yingjia Wan, Song Hu, Yude Li, Xin Jin, Conghao Hu, Haoyu Li, Qirui Fu, Tai Zhong, Xinyu Wang, Xiangru Tang, Nan Tang, Chenglin Wu, Yuyu Luo
TL;DR
Search-agent benchmarks often assume queries are complete, leaving ambiguity resolution during search under-tested. INTERACTCOMP uses target-distractor questions to evaluate clarification, and across 17 models the best interaction accuracy is far below performance with complete context.
Problem
Many search benchmarks assume complete, unambiguous queries, and existing benchmarks do not assess whether agents recognize ambiguity and gather disambiguating information during search.
Source: §1 · p. 1Method
INTERACTCOMP uses 210 expert-curated target-distractor questions across 9 domains, designed to be verifiable with context but ambiguous without interaction.
Source: §1 · p. 2Results
Across 17 models, the best achieves 13.73% accuracy with interaction versus 71.50% with complete disambiguating context.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Language agents have demonstrated remarkable potential in web search and information retrieval. However, many search-agent benchmarks assume that user queries are complete and unambiguous. This assumption leaves under-tested a practical failure mode: agents may face ambiguous requests where the intended target cannot be identified without clarification. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates controlled ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp.
1. Introduction
INTERACTCOMP addresses the gap between search agents’ progress on complete queries and their limited ability to recognize ambiguity and clarify intent during search. Across 17 models, performance remains low with interaction available despite much stronger results with complete context.
1. Introduction
- Over 15 months, INTERACTCOMP performance stagnated around 6–14% while BrowseComp performance improved seven-fold.The authors describe this divergence as a blind spot in agent development.Source: Figure 1 · p. 1, §1 · p. 3
- Existing search benchmarks provide resources upfront, while general interaction benchmarks lack grounding in verifiable search tasks.The paper identifies assessing ambiguity recognition and information-gathering interaction during search as the unmet need.Source: §1 · p. 2
- INTERACTCOMP tests whether agents recognize ambiguity and interact to resolve it, using shared attributes of a target and popular distractor to make search alone insufficient.Its 210 expert-curated questions span 9 domains and have verifiable answers.Source: §1 · p. 2
- 13.73% was the best accuracy in the full interaction setting, compared with 71.50% for the best model given complete disambiguating context.The paper attributes this gap to systematic overconfidence rather than search and reasoning deficits.Source: §1 · p. 2
- Increasing interaction opportunities from 5 to 20 rounds raised accuracy only from 14% to 20%, whereas forced interaction raised it from 14% to 40%.The authors interpret the forced-interaction gains as evidence that capability is latent but current strategies fail to engage it.Source: §1 · pp. 2–3
- The benchmark is presented as a source of clean reward signals for future training approaches.The authors also position it as a foundation for addressing interaction failures in search agents.Source: §1 · p. 3
2. Related Work
Prior work studies web-scale search, tool use, and user interaction, but INTERACTCOMP focuses on resolving ambiguity within a verifiable search-agent workflow.
2. Related Work
- Web-scale and tool-augmented benchmarks assess information gathering, reasoning, and tool use, while search-agent designs use reinforcement learning, data synthesis, or workflow engineering.Examples include BrowseComp (Wei et al., 2025), GAIA (Mialon et al., 2023), and Search-R1 (Jin et al., 2025).Source: §2 · p. 3
- Other benchmarks study ambiguous questions, clarification, collaborative reasoning, or user-centric tasks with underspecified goals.The passage reports that agents uncover fewer than 30% of user preferences through active questioning in UserBench.Source: §2 · p. 3
- INTERACTCOMP embeds ambiguity resolution in a search loop where agents choose whether to search, ask, or answer, with short verifiable final answers.It differs from standalone QA or clarification tasks by requiring clarification before effective retrieval and providing natural reward signals from search outcomes.Source: §2 · p. 3
3. The INTERACTCOMP Benchmark
INTERACTCOMP evaluates whether search agents can resolve ambiguous queries through interaction, using questions that are easy to verify once distinguishing context is obtained. Its construction pairs a target with a popular distractor and validates that the ambiguity requires interaction.
3. The INTERACTCOMP Benchmark
- INTERACTCOMP evaluates whether agents can recognize ambiguity and gather context through interaction during search, following the principle “Easy to verify, Interact to disambiguate.”Instances are human-constructed with search tools and language-model assistance.Source: §3 · p. 3
3.1. Task Overview
- Agents receive an ambiguous question and can search, ask a clarifying question, or answer; the simulated responder is limited to “yes,” “no,” or “I don’t know.”The binary protocol tests whether agents identify useful questions rather than simply requesting more information.Source: §3.1 · p. 4
3.2. Data construction and Verification
- The target-distractor method forms questions from attributes shared by a lesser-known target and popular alternative, while keeping target-distinctive attributes as hidden context.This makes multiple interpretations plausible while allowing the context to disambiguate the verifiable target answer.Source: §3.2 · p. 4, §3 · p. 4
3.2.1. CONSTRUCTION PROCESS
- Construction selects entities, separates shared from distinctive attributes, writes questions from shared attributes, and reserves distinctive attributes as context.Question-context pairs are intended to uniquely identify the target while questions alone remain ambiguous.Source: §3.2.1 · p. 4
3.2.2. VERIFICATION PROCESS
- Two-stage verification checks completeness and interaction necessity, including cross-paraphrase stability, manual search review, and model trials without interaction.Questions answered by at least two of three tested models without interaction are revised to strengthen ambiguity.Source: §3.2.2 · pp. 4–5
3.3. Data Statistics
- The dataset spans 9 topic domains, with Science & Engineering the largest at 21.3%; questions are mostly 40–80 words and contexts typically 40 to over 200 words.It contains 139 English samples (66.19%) and 71 Chinese samples (33.81%).Source: §3.3 · p. 5, Figure 3 · p. 5
4. Experiments
Across controlled interaction settings, the experiments find that models perform poorly when they must identify ambiguity and seek clarification themselves, despite stronger accuracy with complete context. Analyses of scaling, response formats, forced interaction, failure types, and model development probe where this gap persists.
4.1. Experimental Setup
- A ReAct framework compares answer-only, search, interaction, and combined configurations to isolate retrieval and clarification; main results use the combined mode with a 10-round limit.The evaluation measures interaction behavior, accuracy, calibration error, and computational cost.Source: §4.1 · pp. 5–6
4.2. Main Results
- GPT-5 reaches only 13.73% accuracy across 17 models, with systems showing either excessive conservatism or ineffective over-questioning on ambiguous queries.GPT-4o-mini asks questions in 73.95% of available rounds but reaches 7.14% accuracy; its calibration error is 37.44 CE, compared with Doubao-1.6's 84.35 CE.Source: §4.2 · p. 6, Table 1 · p. 7
4.3. Ablation Analysis
- Search-only accuracy (6.74-9.52%) trails with-context performance (40.93-71.50%), while clarification can elicit discriminative attributes missing from the initial query.Ask-only performance therefore does not show that the task is solvable from internal knowledge; the attributes needed for disambiguation are absent from the query and model knowledge.Source: §4.3 · pp. 6–7, Table 2 · p. 7
4.4. Scaling Analysis
- GPT-5 accuracy rises from 14.00% to 20.00% as interactions increase, but quadrupling round limits from 5 to 20 raises its interactions only from 1.14 to 1.90.Claude-Sonnet-4 uses only 0.78 interactions per instance even under the larger limit.Source: Table 3 · p. 8, §4.4 · p. 8
4.5. Natural Language Interaction Analysis
- GPT-5 reaches 25.71% accuracy with free-form responses, still 42% below its 67.88% with-context accuracy.Richer responses improve performance but do not close the gap to the with-context ceiling.Source: Table 4 · p. 8, §4.5 · p. 8
4.6. Forced Interaction Analysis
- GPT-5 doubles accuracy from 20% to 40% when compelled to ask 8 questions, while Claude-Sonnet-4 shows only modest gains.The results indicate that effective information acquisition varies across models; additional questions alone do not ensure useful information.Source: Figure 4 · p. 8, §4.6 · p. 8
4.7. Error Attribution Analysis
- Failures include strategic overconfidence and tactical identification errors: GPT-5 has 70.5% overconfidence errors, while Claude-Sonnet-4 has 87.8% identification errors.The analysis attributes the remaining gap below the with-context ceiling to failures to elicit discriminative attributes, so asking more is not sufficient when questions miss those attributes.Source: §4.7 · p. 8
4.8. Longitudinal Study
- Over 15 months, BrowseComp performance rises seven-fold (10% to 70%) while recent models remain at 6-14% INTERACTCOMP accuracy.The passage describes interaction performance as stagnant despite substantial improvement on the search benchmark.Source: §4.8 · p. 9
5. Conclusion
InteractComp evaluates whether search agents recognize ambiguous queries and resolve them through interaction. Across 17 models, results show a large gap between performance with complete context and interaction-enabled performance, alongside stagnating interaction capability.
5. Conclusion
- InteractComp tests whether agents detect ambiguity and ask for discriminative information when queries remain underdetermined without clarification.Its instances are designed to be easy to verify once the right context is obtained.Source: §5 · p. 9
- 13.73% accuracy with interaction available contrasts with 68–72% accuracy with complete context across 17 models.Forced-interaction settings roughly double accuracy, while natural-language interaction raises GPT-5 to 25.71%.Source: §5 · p. 9
- Over 15 months, BrowseComp performance improved seven-fold while InteractComp performance remained largely stagnant.The authors identify systematic overconfidence as the dominant bottleneck across the 17 models.Source: §5 · p. 9
- Grounded, verifiable search outcomes make InteractComp a potential source of clean reward signals for training uncertainty-aware interactive agents.The authors identify reinforcement learning as a practical training use for the benchmark.Source: §5 · p. 9
Impact Statement
InteractComp provides a controlled setting for evaluating multi-turn, user-centric agents facing ambiguous information needs. The authors connect this evaluation to real-world assistant applications while noting considerations raised by interactive questioning.
Impact Statement
- Interactive clarification and user feedback are modeled to evaluate agents under ambiguous information needs.The stated capabilities include reasoning about uncertainty, asking targeted questions, and adapting to partial or constrained user responses.Source: Impact Statement · p. 10
- The authors identify information retrieval, decision support, and task-oriented assistance as relevant applications where initial intent is often underspecified.Source: Impact Statement · p. 10
- Interactive questioning raises considerations around user privacy, interaction fatigue, and overoptimization toward elicitation strategies.The paper frames balancing information gain, user burden, and task accuracy as a topic for study.Source: Impact Statement · p. 10
A. Benchmark and Implementation Details
The benchmark instance format includes an ambiguous query, simulated-user context, and a concise answer.
A. Benchmark and Implementation Details
- Each task instance includes an ambiguous query.Source: Table A1 · p. 13
- Each task instance includes the simulated user's context.Source: Table A1 · p. 13
- The instance also includes a concise answer.Source: §A.1 · p. 13
A.2. Agent Implementation
The implementation evaluates agents with different combinations of answering, search, and interaction, using bounded action rounds and a simulated responder. Search uses a fixed retrieval setup, while agent prompts specify action selection and evidence-gathering behavior.
A.2.1. AGENT CONFIGURATIONS
- Four configurations compare answer-only, answer-plus-search, answer-plus-interaction, and full search-plus-clarification agents.The implementation uses ReAct and records complete search, interaction, and answer trajectories; Table A2 summarizes configuration actions, response types, and round limits.Source: §A.2.1 · p. 13, Table A2 · p. 14
A.2.2. SEARCH IMPLEMENTATION
- Search uses the Google Serper API with a 30-second timeout and retrieves the top five organic results.Source: §A.2.2 · p. 13
A.2.3. ACTION SPACE DESIGN
- Agents have at most 10 rounds, with one action per round, and forced-interaction experiments require minimum interaction thresholds before answering.The prompts direct agents to gather evidence for missing information and allow closed-ended yes/no questions, with responses including “I don’t know.”Source: §A.2.3 · pp. 13–14
A.3. Responder Simulation
- A GPT-4o responder evaluates agent questions against context and returns “yes,” “no,” or “I don’t know.”The constrained response format prioritizes diagnostic clarity over conversational realism.Source: §A.3 · p. 15
A.4. Data Construction Pipeline
The appendix documents the evaluation protocol and calibration metric, then tests whether key findings hold across responder choices, human validation, commercial systems, and independent runs.
A.4. Data Construction Pipeline
- Table A3 provides a step-by-step example of the data construction pipeline.Source: Table A3 · p. 16
A.5. Evaluation Protocol
- Repeated sampling across identical context–question pairs for k = 3 trials indicates stable behavior despite stochasticity.Source: §A.5 · p. 16
A.6. Calibration Error (C.E.).
- Calibration Error uses five equal-width confidence bins and the absolute gap between each bin’s empirical accuracy and mean confidence.The passage also defines each bin’s sample count and the total number of valid samples.Source: §A.6 · p. 17
B.1. Responder Sensitivity
- All models remain far below the with-context ceiling, so the main conclusion is not solely driven by GPT-4o as responder.The comparison uses a 50-question subset and contrasts default GPT-4o with a matched responder in which the answering model also serves as responder.Source: §B.1 · p. 17
B.2. Human Validation of the Simulated Responder
- Krippendorff’s α = 0.707, while Cohen’s κ is 0.756 with annotator 1 and 0.693 with annotator 2, indicating acceptable agreement in this controlled setting.The authors characterize simulated feedback as an approximation, not a replacement for real user interaction.Source: §B.2 · p. 17
B.3. Commercial DeepResearch Systems
- Doubao Deep Search scores 10.0% accuracy and OpenAI Deep Research 6.67% on the full benchmark, remaining in the same low-accuracy band as ReAct-based agents.The authors suggest richer agentic scaffolding alone does not eliminate the ambiguity-resolution gap.Source: §B.3 · p. 17
B.4. Ranking Consistency Across Runs
- Rankings show statistically significant consistency across runs, but the main conclusion rests on the gap between full interaction performance and the with-context ceiling.The analysis compares three main runs; exact point-wise rankings are not central to the conclusion.Source: §B.4 · p. 17, Table B2 · p. 18
B.5. Interaction Utility Analysis
On a 50-question subset, interaction utility measures show that asking more questions does not necessarily improve disambiguation.
B.5. Interaction Utility Analysis
- The analysis measures average belief change per ask, post-ask correctness gain, and usefulness rate on a 50-question subset.Source: §B.5 · p. 18, Table B3 · p. 18
- The results motivate rewarding useful clarification rather than raw question frequency in future training.Source: §B.5 · p. 18
- GPT-4o-mini asks 2.4× more questions than GPT-5 yet achieves only about one-fifth of GPT-5’s post-ask correctness gain.The table covers average belief change per ask, post-ask correctness gain, and usefulness rate.Source: Table B3 · p. 18, §B.5 · p. 18
C.1. Rounds Constraints Results
The section presents performance metrics across varying interaction levels and a failed Sports-domain example in which clarification and search miss the discriminative clues.
C.1. Rounds Constraints Results
- Table C1 compares accuracy, expected calibration error, and average interaction rounds across varying interaction levels.Source: Table C1 · p. 19
C.2. Case Study On Failed Case.
- In the Sports case, the agent asked four clarification questions and made three searches but focused on hockey-like attributes rather than clues unique to Hornussen.Source: §C.2 · p. 20, Table C2 · p. 20
- The case ends with the model incorrectly identifying Ice Hockey instead of Hornussen.Source: Table C2 · p. 20, §C.2 · p. 20