PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild

Xiaohongshu Inc

arXiv:2605.27882v2cs.CLcs.AI

TL;DR

Existing search benchmarks do not reflect how users collaboratively refine vague information needs through multi-turn interaction. VibeSearchBench evaluates this setting with persona-grounded, schema-free knowledge graphs and graph matching, finding that all seven frontier models remain substantially inadequate, with the best average F1 reaching only 30.30.

  • Problem

    Existing benchmarks rely on over-specified queries, single-turn interactions, and fixed schemas rather than iterative refinement of vague user needs.

  • Method

    VibeSearchBench uses 200 bilingual tasks across 20 domains, pairing user personas with schema-free knowledge graphs and evaluating seven models through graph matching.

  • Results

    30.30 average F1 is the best result, while all models score below 33, indicating substantial inadequacy for VibeSearch.

  • Takeaways & Limitations

    VibeSearch remains challenging because agents struggle with context overflow, inefficient intent elicitation, and structurally flat knowledge-graph outputs.

  • Takeaways & Limitations

    Models extract factual relations effectively but fail to organize information hierarchically, producing predominantly flat, instance-level triples.

Abstract

from arXiv · show

LLM-based agents score well on search benchmarks, yet real users consistently find results unsatisfying, revealing a persistent evaluation-experience gap. We attribute this gap to existing benchmarks' reliance on over-specified queries, single-turn interactions, and fixed-schema evaluation, none of which reflect real search behavior where users and agents collaboratively refine vague intent through multi-turn dialogue. We term this paradigm VibeSearch and introduce VibeSearchBench, a benchmark comprising 200 manually curated bilingual (Chinese and English) tasks across 20 domains, split into VibeSearch-Pro (professional) and VibeSearch-Daily (daily-life) subsets. Each task pairs a user persona with a schema-free ground-truth knowledge graph, and is evaluated through a progressive-disclosure user simulator and a graph-matching evaluation framework. We benchmark seven frontier models under both the ReAct framework and the OpenClaw agent harness. Results show that all models remain substantially inadequate for VibeSearch (best F1: 30.30), highlighting the need for fundamental advances in long-context reasoning, proactive intent elicitation, and structured knowledge construction.

1 Introduction

VibeSearchBench addresses the evaluation–experience gap by modeling search as a long-horizon, multi-turn process in which agents proactively elicit and refine users’ vague information needs. It introduces 200 bilingual tasks across 20 domains and two scenario subsets to evaluate this paradigm.

  • Motivation: Existing benchmarks over-specify queries and restrict interaction to single turns, preventing agents from actively eliciting and refining user intent.Users typically cannot fully articulate their needs upfront, whereas realistic search iteratively combines vague queries, partial results, clarification, and emerging preferences.
  • VibeSearch paradigm: VibeSearch requires bidirectional convergence: agents should interleave partial results with follow-up questions rather than clarify first and search later.This process co-evolves vague user needs and concrete solutions through sustained user-agent interaction.
  • Benchmark: VibeSearchBench evaluates long-horizon proactive search with 200 manually curated tasks spanning 20 domains, split evenly between VibeSearch-Pro and VibeSearch-Daily and between Chinese and English.The benchmark is designed around distinct tasks and scenario diversity across professional and daily-life settings.
  • Benchmark design: Each benchmark task uses a persona with trigger-conditioned progressive disclosure, while agents search autonomously, return partial results, and interact with users to unlock subsequent information.The benchmark overview depicts this multi-turn process together with schema-free knowledge-graph output evaluated against a ground-truth graph.

2 Related Work

Prior search benchmarks mainly evaluate fully specified, single-turn tasks along depth or breadth, while agent-harness benchmarks increasingly assess general agentic capabilities but devote only a fraction of tasks to search and research scenarios.

  • Benchmarking Search: Existing search benchmarks largely use fully specified, single-turn tasks organized around complementary depth and breadth axes.BrowseComp [3] and DeepSearchQA emphasize persistent multi-hop browsing for hard-to-find facts, whereas WideSearch [4] evaluates aggregating parallel sources into pre-specified tables.
  • Benchmarking Search: Depth-oriented benchmarks require persistent multi-hop browsing to retrieve hard-to-find facts, as illustrated by BrowseComp [3] and DeepSearchQA.
  • Benchmarking Agent Harness in the wild: Agent-harness benchmarks cover general agentic capabilities, but most allocate only a fraction of tasks to search- and research-oriented scenarios.Examples include Claw-Eval [16], ClawBench [17], WildClawBench, QwenClawBench, PinchBench, and Claw-Mark [21].

3 VibeSearchBench

VibeSearchBench models search as multi-turn interaction between a user persona with vague, staged needs and an agent, evaluated by matching a predicted knowledge graph against a schema-free ground truth. The benchmark contains 200 bilingual tasks across 20 domains, with substantial graph-based information requirements.

  • Task Formalization: Each task pairs a user persona and vague initial query with staged information needs, while a schema-free directed graph represents arbitrary target information as entity-relation triples.The persona includes user background and preferences, and each disclosure stage specifies a trigger condition and newly revealed requirement.
  • Interactive Protocol: The search process is multi-turn: the agent searches using dialogue history and tools, and the simulator reveals successive requirements only when the agent satisfies explicit trigger conditions.The interaction continues until all stages are addressed or the available budget is exhausted.
  • Data Construction and Quality Control: Experts create distinct-domain scenarios, simulate progressive need refinement, construct consistent ground-truth graphs, and apply independent dual review for scenario, interaction, disclosure, factual, and persona-graph quality.The review process checks both the realism and coherence of the multi-turn search process and the correctness of its structured knowledge.
  • Evaluation Framework: After interaction, evaluation compares the predicted and ground-truth graphs using triplet-level precision, recall, and F1 with information-entailment matching.The LLM judge accommodates aliases, relation synonyms, subsumption, collective coverage, and compositional derivation rather than relying only on exact matches.
  • Benchmark Statistics: 200 tasks span 20 domains, split evenly between VibeSearch-Pro and VibeSearch-Daily and between Chinese and English, with ground-truth graphs averaging 212.43 nodes and 298.32 triples.VibeSearch-Pro graphs average 373.56 triples versus 223.07 for VibeSearch-Daily graphs.

4 Experiments

Experiments evaluate seven frontier LLMs under ReAct and OpenClaw on VibeSearchBench, finding that current models remain substantially inadequate despite differences in framework, interaction behavior, and resource use. Claude Opus 4.6 achieves the best reported OpenClaw average F1 of 30.30, while proactiveness and context management help explain performance differences.

  • Overall: 30.30 average F1 is Claude Opus 4.6’s best OpenClaw result, while all models score below 33, demonstrating substantial inadequacy on VibeSearch.Claude Opus 4.6 and DeepSeek-V4-Pro form the top tier (F1 ≥27), followed by Kimi K2.6, while GPT-5.4 and Qwen3.5-397B-A17B score 20–23; OpenClaw usually slightly outperforms ReAct.
  • Precision vs. Recall: Most models have higher Recall than Precision, favoring broad coverage at the cost of irrelevant triples; Gemini-3.1 Pro is the exception with P=34.61 and R=20.63.Claude reaches P=24.88 and R=36.34, while its Daily Recall is 39.20 and Precision is 21.60; Gemini leaves nearly 84% of Pro ground-truth triples unrecovered.
  • Pro vs. Daily: Pro F1 consistently exceeds Daily F1 for most models because professional information is more concentrated and structured, whereas Daily information is more scattered and needs are harder to anticipate.Claude scores 29.79 vs. 25.95 and DeepSeek 28.70 vs. 25.37; Gemini-3.1 Pro reverses this pattern with Daily 24.66 vs. Pro 22.41.
  • Proactiveness: Claude’s ReAct #Asst/#User ratio of 8.26 and highest F1 link stronger proactiveness with better VibeSearch performance.Claude performs 7–8 tool calls per user reply on average, whereas Gemini-3.1 Pro has the lowest ratio, 2.84, and passively waits for user-driven exploration.
  • Interaction Efficiency: GPT-5.4’s high resource consumption does not improve performance: its context compression count is 1.27 versus <0.7 for other models, causing information loss and degradation.Despite 99.6 assistant turns and an OpenClaw #Asst/#User ratio of 4.34, GPT-5.4’s verbose output triggers frequent context overflow and redundant subsequent searches.
  • Framework Effects on Interaction Patterns: Framework changes alter efficiency differently: Claude improves from 27.87 to 30.30 F1 as assistant turns decrease, while Seed2.0 Pro improves from 23.22 to 25.23 as assistant turns increase.Claude’s assistant turns fall from 109.8 to 93.6 under OpenClaw, whereas Seed2.0 Pro’s rise from 73.0 to 84.8 expands exploration space.

5 Analysis

Failure analysis shows that context management, intent elicitation, and knowledge-graph organization remain major bottlenecks in ReAct trajectories. OpenClaw ablations likewise show that sub-agents and memory mechanisms do not significantly improve VibeSearch performance, while the evaluation framework agrees with human experts above 98.5%.

  • Information Retrieval and Context Management Failures: Compressed trajectories lose 8–12 F1 points on average, while context overflow makes agents forget disclosed requirements and misalign downstream outputs.F1 falls from 0.26 to 0.16 on average; GPT-5.4 drops from 0.25 without compression to 0.12 with at least two compressions.
  • Multi-Turn Interaction and Intent Elicitation Failures: Models rarely reach [DONE], and trajectories exceeding 15 user turns average 0.18 F1 versus 0.23 for trajectories with at most 10 turns.Longer trajectories reflect both harder tasks with scattered information and wasted turns caused by misaligned questions.
  • Knowledge Graph Construction and Output Failures: Even the best model achieves 100% coverage for factual relations but 0% for organizational and hierarchical relations, yielding flat knowledge-graph outputs.This indicates that models extract facts effectively but fail to organize them hierarchically.
  • OpenClaw Mechanism Ablation: Sub-agents and memory mechanisms fail to significantly improve F1, despite increasing workload or context pressure and providing little effective cross-task transfer.Sub-agent delegation increases assistant turns by 64%–129% and tools by 82%–129%, while local-memory F1 remains within ±0.5 and life-long-memory differences stay below 1.0.
  • Evaluation Reliability: The evaluation framework achieves above 98.5% overall agreement with human experts across three LLM judges, supporting its use as a substitute for human annotation.Kimi achieves the highest agreement at 98.92%.

6 Conclusion

VibeSearchBench evaluates long-horizon proactive search through multi-turn collaborative intent refinement and schema-free information-graph construction. Across seven frontier models tested with ReAct and OpenClaw, the best model achieves only 30.30 F1, with context overflow, inefficient intent elicitation, and structurally flat outputs identified as key bottlenecks.

  • Conclusion: 30.30 F1 is the best result among seven frontier models evaluated under both ReAct and OpenClaw.
  • Conclusion: VibeSearchBench evaluates agents on long-horizon proactive search requiring multi-turn collaborative refinement of vague user intent and schema-free information graphs.
  • Conclusion: Context overflow, inefficient intent elicitation, and structurally flat knowledge-graph outputs are identified as key bottlenecks.

Contribution

The contribution section lists the paper’s contributors, their organizational affiliations, and designated core contributors and project lead.

  • Contribution: The author list designates several contributors as core contributors and Xiang Cheng as project lead.The symbols indicate † for core contributor and ‡ for project lead.
  • Contribution: The paper lists contributors from the General Post-training Team at Xiaohongshu Dots Studio and UniPat AI.The affiliations identify Xiaohongshu Dots Studio as affiliation 1 and UniPat AI as affiliation 2.

A Evaluation Details … A.4 Implementation

The framework evaluates predicted knowledge graphs by judging ground-truth triple coverage, evidence-supported precision, and their harmonic-mean F1. It improves efficiency through batched parallel judging with detailed prompts, warnings, and examples.

  • A Evaluation Details: The appendix formalizes and implements the graph-based evaluation framework introduced in Section 3.4.
  • A.1 Triplet Recall: Triplet recall counts a ground-truth triple as covered when predictions directly express, subsume, collectively cover, or compositionally derive its information.
  • A.1 Triplet Recall: Triplet recall is the fraction of ground-truth triples covered by the predicted graph.
  • A.2 Triplet Precision: Triplet precision is the fraction of predicted triples that support coverage of at least one ground-truth triple.During recall evaluation, the LLM judge records which predicted triples provide supporting evidence for each covered ground-truth triple.
  • A.3 Triplet F1: Triplet-level F1 is computed as the harmonic mean of triplet precision and triplet recall.
  • A.4 Implementation: Evaluation efficiency is improved by partitioning ground-truth triples into parallel batches and using prompts with criteria, error warnings, and worked examples.The specific LLM-judge prompt is provided in the appendix.

B Tool Specifications

The agent uses four function-calling tools for web search, webpage access, academic literature retrieval, and code execution. Search discovers sources, Visit improves information coverage, Scholar Search supports professional tasks, and Python handles computation.

  • Tool Specifications: Four function-calling tools cover web search, webpage content access, academic literature retrieval, and sandboxed code execution, with free invocation at each reasoning step.The tools are Search, Visit, Scholar Search, and Python.
  • Tool Specifications: Search returns the top N results with titles, URLs, and snippets, serving as the most frequently used tool for discovering sources and initial clues.The agent supplies a query string and receives ranked search results.
  • Tool Specifications: Visit accepts URLs and a goal description, returning tailored webpage summaries that provide more complete information than search snippets and correlate strongly with information coverage.The tool extracts detailed content focused on the agent’s specified goal.
  • Tool Specifications: Scholar Search retrieves academic papers from Google Scholar and is primarily used for domain-specific information in the VibeSearch-Pro subset.Results include titles, links, publication dates, sources, and snippets.
  • Tool Specifications: Python executes code in a sandbox and returns standard output and error for data processing, numerical calculations, structured-data parsing, and output formatting.It is primarily used for computation and data-handling tasks.

C Task Examples … D.1 Full Performance Results

The task examples show how VibeSearch combines progressively disclosed user intent with schema-free knowledge graphs whose structures differ by domain. Full evaluation results additionally show that run-to-run randomness materially affects reported performance, making multi-run evaluation important for estimating capability ceilings.

  • C Task Examples: Across the examples, each task pairs a persona-driven interaction sequence with a representative knowledge-graph substructure, illustrating how VibeSearch task complexity varies by domain.The Pro graph is deep and hierarchical, whereas the Daily graph is flat and layered around successive filters.
  • C.1 VibeSearch-Pro Example (Mathematics / History of Analysis): The Pro example centers on a self-learner seeking the historical development, attribution, proof gaps, and institutional relationships behind real and complex analysis.The persona’s questions move from Newton–Leibniz and early calculus through rigorization, theorem attribution, and mathematician networks.
  • C.1.1 User Persona: The Pro mathematics-history example uses a persona with 11 strictly ordered disclosure stages that progressively expand questions from calculus evolution to mathematician relationships.The simulator discloses one stage at a time, persists on unmet triggers, and withholds proactively triggered information until the assistant asks appropriately.
  • C.1.2 Ground-Truth Knowledge Graph: The Pro ground-truth graph contains 260 nodes, 349 triples, and 112 relation types across 5 thematic dimensions and 23 subtopics.Its schema-free, semantically rich relations connect abstract dimensions and subtopics to concrete historical entities, matching exploratory information needs.
  • C.2 VibeSearch-Daily Example (Entertainment / Game Selection): The Daily example centers on a selective game buyer whose criteria are progressively narrowed toward recent, award-recognized, buy-to-play games with specific technical and art-team attributes.The task ends by requiring a complete comparison of every game that satisfies all disclosed requirements.
  • C.2.1 User Persona: The Daily game-selection example uses a 10-stage progressive filtering pipeline that withholds additional preferences unless the assistant elicits them or satisfies prior stages.Later stages filter by release year, awards, price, hardware, scale, business model, engine, art departments, and recognition before requesting a final comparison.
  • C.2.2 Ground-Truth Knowledge Graph: The Daily ground-truth graph contains 108 nodes, 229 triples, and 14 relation types arranged in 8 layers corresponding to the filtering pipeline.Its structured, uniform relations capture both entities surviving each requirement and the attributes needed for criteria-based decisions.
  • D.1 Full Performance Results: Best-run F1 exceeds average F1 by approximately 4–6 points across all models, showing that single-run randomness materially affects performance across both frameworks and subsets.Table 12 reports average and best-run Precision, Recall, and F1 for all models under both frameworks on both subsets; best-run reporting better reflects each model’s capability ceiling.

D.2 Tool Usage and Token Consumption … E.4 Knowledge Graph Structural Alignment

The analyses show that search depth, context management, interaction quality, and structural knowledge construction jointly limit agent performance. Models often retrieve explicit facts but fail to sustain efficient, targeted dialogue and reconstruct complex organizational relations.

  • D.2 Tool Usage and Token Consumption: Scholar-search calls are strongly domain-dependent: Claude Opus 4.6 averages 23.13 on Pro versus 0.41 on Daily, while GPT-5.4 averages 27.23 versus 0.60.The Pro subset’s academic domains drive substantially greater scholar-search use than Daily tasks.
  • D.2 Tool Usage and Token Consumption: Tool usage varies by model: Gemini-3.1 Pro makes 0.05 Pro and 0.46 Daily visits, whereas Claude Opus 4.6 and GPT-5.4 make 30–56 visits per task.Python use is also sparse and model-dependent, ranging from Claude’s 11.67 Daily calls to below 0.5 calls for Seed2.0 Pro and Kimi K2.6.
  • E.1 Context Compression and Retrieval Depth: Each context compression reduces F1 by approximately 6 points, while two or more compressions use 3.2× the tokens of uncompressed trajectories yet achieve less than half the F1.This degradation under ReAct explains GPT-5.4’s paradoxical position as the highest-resource yet lowest-overall-F1 model in the cited analysis.
  • E.1 Context Compression and Retrieval Depth: On VibeSearch-Daily, Gemini-3.1 Pro trajectories visiting at least one page reach Recall 0.34 versus 0.22 for snippet-only trajectories, a 55% relative improvement.Gemini averages only 1.1 Pro visits and 1.9 Daily visits, indicating insufficient retrieval depth despite avoiding compression.
  • E.2 Progressive Disclosure Stage Completion: Under ReAct, no trajectory virtually reaches [DONE]; runs terminate through agent answers or max_rounds exhaustion, leaving some latent user needs unaddressed.The appendix provides quantitative evidence supporting the paper’s error analysis.
  • E.2 Progressive Disclosure Stage Completion: More dialogue turns correlate negatively with Triplet F1 across models, with low-turn versus high-turn trajectories differing by 1–9 F1 points.The pattern reflects both intrinsically harder, more scattered tasks and agents’ inefficient satisfaction of trigger conditions.
  • E.3 Interaction Strategy and Intent Elicitation: Dismissive rates rise from 0.6–1.8% on Pro to 2.1–7.9% on Daily, while redirect rates remain 5–6% on Pro and 3–6% on Daily across models.Passive interaction leaves agents unable to formulate targeted follow-ups, whereas excessive proactiveness generates irrelevant questions; both reflect shallow coverage and poor interaction quality.
  • E.4 Knowledge Graph Structural Alignment: Models achieve 0% coverage for organizational or hierarchical relations but 100% for factual relations, extracting explicit assertions while failing to reconstruct organizational scaffolding.The contrast is reported for Kimi K2.6 on the first 100 VibeSearch-Pro tasks and reflects a broader structural-alignment failure.

E.5 Invalid Prediction Type Analysis · F Annotation details · G Prompt Details

Invalid predictions mainly arise from irrelevant metadata, subjective judgments, and malformed final knowledge-graph outputs, while annotation used over 60 experts and the appendix specifies simulator and extraction prompts.

  • E.5 Invalid Prediction Type Analysis: Bibliographic metadata predictions have invalidity ≥98%, primarily driving Claude Opus 4.6’s low Pro Precision of 0.137.Models mechanically extract pages, volume, DOI, and citation information outside users’ needs.
  • E.5 Invalid Prediction Type Analysis: Subjective assessment predictions have invalidity ≥95% because models inject evaluative judgments instead of extracting factual information.Affected relations include significance, structural innovation, and core_contribution.
  • E.5 Invalid Prediction Type Analysis: Seed2.0 Pro accounts for 28 zero-F1 trajectories, with malformed JSON, empty outputs, and schema misalignment producing catastrophic failures.One example generated 237 triples for a single task while receiving F1 = 0.
  • E.5 Invalid Prediction Type Analysis: Table 18 reports the most frequently invalidated prediction relation types on the first 100 VibeSearch-Pro tasks.Invalidity rate is the fraction judged not to support any ground-truth triple.
  • E.5 Invalid Prediction Type Analysis: These failures show that final knowledge-graph construction remains fragile under pressure from long interaction histories.Malformed JSON includes missing colons and non-standard key names.
  • F Annotation details: More than 60 experts annotated the task, with approximately $300 paid per quality-inspected task and total dataset annotation cost around $60,000.Payment was based on the number of tasks completed and passing quality inspection.
  • G Prompt Details: The appendix provides complete prompts for the user simulator and triple extraction module.Table 19 gives the simulator system prompt with task-specific persona and query placeholders, while Table 20 gives the post-interaction extraction prompt for structured knowledge graphs.
Loading 2605.27882v2…