Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents

1 Introduction

• Earlier retrieval and agent approaches do not fully address complex web-search questions, and irrelevant retrieved text can harm response quality.The related-work discussion describes query-based retrieval and more structured planning or information aggregation approaches.Source: §1 · p. 1
• Chinese web-search research lacks a suitable evaluation dataset and metrics that capture semantic quality, while existing agents can require costly fine-tuning or closed-source models.The paper also notes a lack of comprehensive studies of open- and closed-source LLMs in web search.Source: §1 · p. 2
• Level-Navi Agent is a general-purpose, training-free framework in which a Planner decomposes questions and a Searcher gathers information at different levels.The framework iterates until it has enough information to answer the initial question and can use open-source LLMs.Source: §1 · p. 2
• Web24 provides a Chinese-internet dataset with detailed question and source classifications, alongside four metrics for evaluating LLMs executing the agent.The authors state that the benchmark presents the performance of different LLMs for AI web search.Source: §1 · p. 2
• Models may avoid web searches despite not knowing an answer, and low task fidelity can produce non-compliant, poor-quality responses.The paper characterizes the first behavior as overconfidence and the second as failure to fully understand instructions.Source: §1 · pp. 2–3

2 Level-Navi Agent

• Level-Navi Agent uses a Planner and a Level-Info Agent to iteratively decompose a user question, gather information, and return an answer.The section introduces the overall framework and illustrates its pipeline and an example query.Source: §2 · p. 3, Figure 2 · p. 3

2.1 Planning Agent

• The Planning Agent selects the next set of parallel sub-questions after considering the information gathered so far, repeating until it judges the evidence sufficient.Rather than produce a complete plan at once, it adjusts the plan in response to search feedback.Source: §2.1 · pp. 3–4

2.2 Level-Info Agent

• The Level-Info Agent checks model knowledge, search-result summaries, and opened pages in sequence, stopping when information is sufficient and reducing site-reading and search-API costs.In the illustrated example, the Planner first requests information about nominated games, then searches for each game's release date in parallel.Source: §2.2 · pp. 4, 6, Figure 3 · p. 5

3.1 Data Composition

• Web24 labels each pair by source, domain, and type; most questions come from pre-December 2024 Chinese news reports with verifiable URLs, while knowledge items are checked for search necessity.Source: §3.1 · p. 6, Figure 4 · p. 6
• Its five everyday domains and four question types are designed to represent daily information needs and are distributed in Table 1.Source: §3.1 · pp. 6–7, Table 1 · p. 6

3.2 Evaluation Metrics

• Four metrics combine answer correctness, semantic similarity, response relevance, and average Level-Info Agent invocation count, with fewer invocations scoring better.Correctness uses an LLM evaluator, while semantic similarity compares embedding vectors. Relevance compares inferred questions with the original question; fewer invocations are associated with faster processing and lower search-API costs.Source: §3.2 · pp. 7–8
• The total score weights correctness at 60, semantic similarity and relevance at 15 each, and the exponentially decayed searcher-count score at 10.Source: §3.2 · p. 8

4.1 Experimental Results of Our Agent on the Web24 Dataset

• In Web Search Agent experiments under the Level-Navi Agent framework and benchmark testing, Qwen2.5-72B and Deepseek-V2.5 perform best among the LLM results presented in Tables 2 and 3.Source: §4.1 · p. 8
• Within the Qwen series, roughly fivefold scaling from 3B to 14B improved scores by about 6 points, versus about 3 points from 14B to 72B.The authors note diminishing marginal returns from parameter scaling and recommend focusing on higher-quality information sources.Source: §4.1 · p. 8
• Three-shot prompting significantly improved pass rates over zero-shot prompting, while some Qwen models above 7B had pass rates close to 1 even zero-shot.Pass rate reflects whether tasks were completed, including errors such as incorrect output formatting or function calls.Source: §4.1 · pp. 8–9
• Llama-series results were less satisfactory in scores and pass rates, and the authors report that Gemma and Mistral struggled to follow Chinese instructions or answer in Chinese.The authors suggest multilingual optimization as important for Chinese-context performance.Source: §4.1 · pp. 9–10

4.2 Comparison with Other Products

• Level-Navi Agent reached the same level as commercial products on correctness, relevance, and semantic similarity, while Kimi scored slightly higher without a significant gap.The comparison used 100 randomly selected Web24 examples; the authors also describe the agent as flexible and cost-effective because users can switch models.Source: §4.2 · pp. 10–11, Figure 5 · p. 10

4.3 Analysis of Metrics

• Traditional token-based metrics can misrepresent open-ended web-search answers: longer relevant responses may lower F1 while increasing recall, whereas the proposed metrics align with observed model differences.The authors state that human or LLM evaluation may better assess answers containing rich relevant material and varied semantic expressions.Source: §4.3 · p. 11, Table 4 · p. 11

4.4.1 Overconfidence Phenomenon in Web Search Function Usage.

• Qwen2.5-7B invoked search functions in about 90% of Agent invocations, while GLM-4-9B’s function-call ratio fell from 30% to a single-digit percentage.The authors call refraining from search despite not knowing an answer “Overconfidence” and recommend balancing positive and negative training examples.Source: Figure 6 · p. 12, §4.4.1 · p. 12

4.4.2 Low Task Fidelity in Conducting Chinese Tasks.

• Llama 3.1 responses sometimes mixed instructions with answers, indicating low Task Fidelity; few-shot prompting improved this issue only for Llama3.1-70B.The paper uses Task Fidelity for whether an agent comprehends and follows instructions; the authors recommend improving smaller models’ multilingual ability.Source: §4.4.2 · pp. 12–13, Table 5 · p. 13

• Level-Navi Agent combines multiple agents with hierarchical reasoning and searching to address web-search challenges.Source: §5 · p. 13
• The paper evaluates LLM performance on Chinese Internet web-search tasks using new evaluation metrics and methods.Source: §5 · p. 13
• Data-driven error analysis identifies LLM limitations in web search and offers recommendations for improvement.Source: §5 · p. 13