Source-linked AI summary
GISA: A Benchmark for General Information-Seeking Assistant
Yutao Zhu, Xingshuo Zhang, Maosen Zhang, Jiajie Jin, Liancheng Zhang, Xiaoshuai Song, Kangzhi Zhao, Wencong Zeng, Ruiming Tang, Han Li, Ji-Rong Wen, Zhicheng Dou
TL;DR
Existing agentic-search benchmarks can produce unnatural, static tasks and often separate deep search from broad aggregation. GISA introduces 373 human-crafted queries with structured, dynamic evaluation and complete human trajectories; the best-performing model reaches only 19.30% overall exact match, showing substantial remaining difficulty.
Problem
Existing benchmarks often reverse-engineer queries from answers, emphasize either deep search or aggregation, and rely on static answer sets vulnerable to memorization.
Method
GISA constructs human-crafted queries with four structured answer formats, live updates, and complete human-annotated search trajectories.
Results
19.30% overall exact match is achieved by the best-performing model evaluated on GISA.
Takeaways & Limitations
GISA provides a deterministic, process-supervised benchmark for studying general information-seeking agents across deep reasoning and broad aggregation.
Takeaways & Limitations
GISA’s 373-query scale may not support large-scale supervised fine-tuning, and the benchmark currently excludes multimodal web content.
Abstract
from arXiv · showhide
The advancement of large language models (LLMs) has significantly accelerated the development of search agents capable of autonomously gathering information through multi-turn web interactions. Various benchmarks have been proposed to evaluate such agents. However, existing benchmarks often construct queries backward from answers, producing unnatural tasks misaligned with real-world needs. Moreover, these benchmarks tend to focus on either locating specific information or aggregating information from multiple sources, while relying on static answer sets prone to data contamination. To bridge these gaps, we introduce GISA, a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries that reflect authentic information-seeking scenarios. GISA features four structured answer formats (item, set, list, and table), enabling deterministic evaluation. It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization. Notably, GISA provides complete human search trajectories for every query, offering gold-standard references for process-level supervision and imitation learning. Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30\% exact match score, with performance notably degrading on tasks requiring complex planning and comprehensive information gathering. These findings highlight substantial room for future improvement.
1 Introduction
GISA addresses limitations in existing agentic-search benchmarks by using human-crafted, structured, dynamic tasks that combine deep investigation with broad information aggregation. It also provides human search trajectories for process-level supervision, while evaluations show current systems remain far from reliable performance.
- Existing benchmarks often reverse-engineer queries from answers, producing tasks that can diverge from authentic information needs and natural forward search.
- GISA contains four structured answer formats—items, sets, lists, and tables—that support deterministic evaluation with strict matching metrics.
- GISA unifies vertical investigation and horizontal summarization, evaluating deep reasoning and broad information aggregation within complex, long-horizon tasks.
- GISA separates stable and live queries, periodically updating live answers to reduce memorization and data contamination.
- Complete human search trajectories accompany every query, supporting process reward modeling, imitation learning, and verification of realistic task solvability.
- 19.30% overall exact match is achieved by the best-performing model, while commercial deep research systems struggle to outperform LLM-based agents.
2 Related Work
Related work develops increasingly capable search agents and benchmarks them on web retrieval, multi-hop reasoning, and cross-page information integration. However, several existing benchmarks use reverse-engineered tasks that may not reflect real user queries.
- LLM-driven search agents now support autonomous planning and dynamic decision-making beyond early retrieval-augmented generation.
- Current agent architectures span breadth-oriented parallel search, depth-oriented serial reasoning, knowledge graphs, Monte Carlo Tree Search, and multi-agent collaboration.
- Benchmarks such as BrowseComp, InfoDeepSeek, and xbench-DeepSearch evaluate deep search through multi-hop reasoning and cross-page information integration.
- Many existing benchmark tasks are reverse-engineered from answers and may therefore deviate from real user queries.
3 Benchmark
The benchmark section introduces GISA’s design and construction process before presenting its evaluation protocol.
- GISA’s design and construction process is detailed before the evaluation protocol.
- The section organizes the benchmark discussion around construction and evaluation.
- The stated workflow moves from benchmark construction to evaluation protocol.
3.1 Construction
GISA uses a human-centered, multi-stage construction workflow to create realistic queries, structured answers, and verified search trajectories. The process spans brainstorming, refinement, annotation, and quality checking, with feasibility, temporal, and formatting constraints.
- Construction: GISA’s construction pipeline has four stages: brainstorming, query refinement, human annotation, and quality checking.
- Brainstorming: Annotators browse domain-specific sources and record questions that arise spontaneously, producing seed questions across ten topic domains.
- Query Refinement: Queries are assigned item, set, list, or table answer formats to enable deterministic evaluation and combine deep and wide search.
- Query Refinement: A feasibility check rejects questions whose answers are readily available in pre-aggregated web pages and favors tasks requiring both broad aggregation and individual verification.
- Query Refinement: Queries are classified as stable or live, with live ground truths scheduled for monthly updates.
- Human Annotation: Human annotation records Google searches, result pages, clicks, and timestamps while prohibiting other search engines and LLMs.
3.2 Evaluation
GISA evaluates structured answers with exact match as a strict universal metric and answer-type-specific metrics for sets, lists, and tables. Its pipeline also normalizes outputs and ground truth to reduce formatting effects.
- The evaluation pipeline extracts TSV content from Markdown code blocks, falls back to raw responses when needed, and normalizes predictions and ground truth.Normalization standardizes headers and cleans numeric values to reduce formatting discrepancies.
- Exact match assigns 1 only to outputs completely identical to the ground truth and 0 otherwise.
- Set-type answers are evaluated with F1 to measure overlap between predicted and ground-truth items through precision and recall.
- List-type answers receive separate F1 and order scores because both item content and sequence matter, while duplicates limit RBO's suitability.The order score is computed as 2M/T using matching elements and the total number of elements in both lists.
- Table-type answers are assessed with row-level F1 for complete rows and item-level F1 for individual cell values.
4 Experiments
Experiments compare ReAct-based LLM agents and commercial search systems on GISA, revealing low overall accuracy, broad failure modes, and several performance trade-offs. Further analyses examine search behavior, contamination resistance, inference-time scaling, error patterns, and cost.
- Baseline Methods & Settings: Agents receive identical task descriptions and constraints, while ReAct-based systems additionally use Search and Browse tools.The evaluation includes diverse LLMs and commercial deep-search and deep-research systems.
- Main Results: 19.30% overall EM is achieved by the best-performing model, Claude 4.5 Sonnet (thinking), leaving substantial room for improvement.Manual inspection identifies limited problem decomposition, insufficient self-correction, and inadequate web traversal as recurring failure modes.
- Main Results: Performance degrades as information breadth increases, with agents struggling especially on table-type tasks and on organizing and formatting lists and tables.Item-type questions generally perform better, but information quantity does not necessarily determine question difficulty.
- Main Results: Moderate tool usage accompanies Claude 4.5 Sonnet’s stronger performance, whereas substantially more calls by DeepSeek-V3.2 and GLM-4.7 coincide with lower scores.The authors suggest irrelevant retrieved content and longer contexts may harm reasoning quality.
- Main Results: Thinking mode raises Claude 4.5 Sonnet’s overall EM from 16.36% to 19.30% and DeepSeek-V3.2’s from 11.53% to 14.47%, but consumes significantly more tokens.Commercial systems do not outperform LLM-based ReAct agents; Google Search AI Mode is relatively faster but less accurate.
- Further Analysis: Best@k increases from 8.90% to 22.22% at k=16 for Qwen3-Max, while Majority@k consistently trails, exposing answer-selection challenges.The scaling experiment uses 40 random queries and k independent runs per query.
5 Limitations
GISA has a limited scale, text-only scope, and a capped experimental interaction budget. These constraints restrict training scale, multimodal evaluation, and information collection in some cases.
- GISA contains 373 queries, which may be sufficient for evaluation but insufficient for large-scale supervised fine-tuning.The annotation process requires substantial human involvement at every stage.
- The benchmark evaluates only text-based information seeking and excludes images, videos, and other multimodal web content.Future benchmarks could assess agents’ visual information-processing abilities.
- Experiments cap each query at 30 tool invocations, and a small number of cases failed to collect adequate information because of this limit.The authors suggest more flexible resource allocation strategies as future work.
6 Ethical Considerations
GISA’s construction follows privacy and consent safeguards while attempting to reduce bias through diversified query domains. The authors nevertheless acknowledge that bias may remain.
- The annotation browser extension records only task-relevant interactions within designated search sessions and collects no personally identifiable information, cookies, or extra browsing history.All annotators participated voluntarily with informed consent.
- The authors diversified query domains to mitigate potential biases, while acknowledging that the benchmark may still contain bias.
7 Conclusion
The paper introduces GISA as a benchmark combining structured, contamination-resistant evaluation with human search trajectories for general information-seeking agents. Experiments show that current systems remain far from human performance, leaving room to improve planning and tool use.
- GISA comprises 373 human-crafted queries with item, set, list, and table answer formats for evaluating general information-seeking agents.
- GISA unifies deep reasoning and broad information aggregation, supports deterministic evaluation, includes live answers to resist contamination, and provides human trajectories for process-level supervision.
- 18.23% overall exact match was achieved by the best model in evaluations of mainstream LLMs and commercial search products.The results indicate substantial room for improvement in problem decomposition, adaptive search planning, and efficient tool utilization.
A Data Format
GISA stores questions, answers, and human search trajectories as separate structured data components. These formats encode question metadata, answer schemas, interaction records, and supporting figure and table examples.
- Data Format: GISA is organized into questions, answers, and human trajectories.
- Questions: Each question JSON records an identifier, encrypted text, answer type, stable-or-live status, and topic category.Encryption is used to reduce contamination from model pre-training.
- Answers: Each answer is stored in a question-specific CSV supporting item, set, list, and predefined-schema table formats.Item answers occupy the first cell; set and list answers use one column, while tables use multiple columns.
- Behavior Comparison: Table 5 compares search behaviors across different models and human annotators.
- Human Trajectories: Each human-trajectory JSON contains annotator queries, search-engine results, and click actions recording page navigation.Figure 7 illustrates an example trajectory JSON template.
B Implementation Details
The evaluation uses ReAct-based agents equipped with Search and Browse tools, while commercial systems follow a similar prompt structure and produce TSV-formatted answers.
- Agent and Tool Configuration: ReAct-based agents use Search and Browse tools with a maximum of 30 tool invocations per session.The system prompt and tool definitions are provided in Figures 8 and 10.
- Agent and Tool Configuration: The Search Tool uses Google Search through the Serper API to retrieve the top-10 search results for an input query.
- Agent and Tool Configuration: The Browse Tool uses the Jina API to extract URL content and an identical non-thinking backbone model to summarize it for consistency and lower cost.
- Commercial Systems: Commercial systems use a similar prompt structure and return answers in TSV format enclosed within answer tags.Their prompt instructs the system to determine whether the answer is an item, list, or table.
C Human-Model Search Behavior Analysis
The analysis compares model and human search behavior using query and URL overlap alongside query-reformulation diversity, revealing moderate alignment and differences in search strategy.
- Behavioral Metrics: Search Similarity measures the overlap between model-generated and human queries using averaged maximum Jaccard similarity across questions.Queries are lowercased and tokenized before comparing their term sets.
- Behavioral Metrics: Search Diversity measures consecutive-query diversity through average adjacent-query Jaccard similarity, with lower values indicating more diverse reformulations.
- Behavioral Metrics: Browsing Similarity measures overlap between model-visited and human-visited URLs using Jaccard similarity of the two URL sets.
- Observed Results: 24.82%–38.97% search similarity scores indicate moderate alignment between model and human query formulations.GPT-5.2 records 24.82%, while DeepSeek-V3.2 thinking records 38.97%.
- Observed Results: 21.73% search diversity for Claude 4.5 Sonnet contrasts with 43.05% for Qwen3-235B-A22B, indicating focused versus exploratory query patterns.