PaperlayerView exact revision on arXiv

Source-linked AI summary

RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intent

Xiao Chen, Yicheng Zhao, Yingying Wu, Zhendong Chu, Changyi Ma, Qingsong Wen, Xuan Song

arXiv:2609.30717v1cs.IR

TL;DR

Existing benchmarks often assume explicit user intent or isolated tool calls, leaving realistic recommendation orchestration under fuzzy instructions underexplored. RecToolBench builds an MCP benchmark and evaluates tool-using agents, finding persistent weaknesses in grounding and multi-step orchestration.

  • Problem

    Existing benchmarks often use explicit instructions, simplified settings, or isolated calls, leaving fuzzy-intent recommendation tool orchestration underexplored.

    Source: §1 · pp. 1–2

  • Method

    RecToolBench constructs executable fuzzy recommendation tasks with a synthesize–fuzzify–judge pipeline and evaluates trajectories using rule-based checks and rubric-based LLM judging.

    Source: §1 · pp. 2–3

  • Results

    Across task levels, models can execute tools correctly yet remain weak in parameter grounding, dependency tracking, evidence integration, and grounded recommendations.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    Tool orchestration under fuzzy user intent remains a key bottleneck for reliable agentic recommender systems.

    Source: §6 · p. 9

    Evaluation uses a controlled MCP environment and covers only a subset of recommendation scenarios, tool ecosystems, user behaviors, and agent designs.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent advances in agentic recommender systems are shifting recommender systems from passive filtering engines to instruction-following agents that use external tools to resolve user intent. However, existing benchmarks often assume explicit user intent, simplified tool environments, or isolated function calls, leaving realistic tool orchestration for recommendation underexplored. To bridge this gap, we propose RecToolBench, a Model Context Protocol (MCP)-based benchmark for evaluating tool-using recommender agents under fuzzy user instructions. RecToolBench contains more than 1,200 executable tasks across three recommendation domains, 13 MCP servers, and 32 tools, spanning single-tool calls, parallel tool calls, sequential tool chains, and hybrid tool orchestration. We construct RecToolBench with a scalable synthesize--fuzzify--judge pipeline that generates executable fuzzy recommendation tasks, and evaluates agent trajectories using rule-based execution checks and rubric-based LLM evaluation. Experiments on representative LLMs show that syntactically valid tool calls do not guarantee successful recommendations. Models struggle with semantic parameter grounding, multi-step evidence integration, and grounded final recommendations, especially as orchestration complexity increases. Our results identify tool orchestration under fuzzy user intent as a major bottleneck for agentic recommender systems. Our data and code are available at https://github.com/ShawnChenn/RecToolBench.

1 Introduction

Existing benchmarks often miss fuzzy recommendation instructions and compositional tool use. RecToolBench evaluates these workflows and finds that model weaknesses deepen as orchestration becomes more complex.

1 Introduction

  • Existing benchmarks often assume explicit instructions or isolated calls, while fuzzy recommendation requests require intent inference, coordinated tools, and grounded recommendations.Syntactically valid calls can still use parameters misaligned with fuzzy intent, making execution-only metrics over-optimistic.Source: §1 · pp. 1–2
  • RecToolBench is an MCP-based benchmark with over 1,200 tasks across three domains, 13 servers, 32 tools, and four levels of orchestration complexity.Its levels range from single-tool calls to hybrid dependencies and orchestration.Source: §1 · p. 2, Figure 1 · p. 2
  • Failures range from tool and parameter selection errors in SLMs to grounding and dependency-aware orchestration weaknesses, with L4 exposing gaps between execution scores and grounded recommendations.Models can achieve strong rule-based scores while Hit Ratios and grounding scores remain much lower in hybrid tasks.Source: §1 · pp. 2–3
  • A scalable synthesize–fuzzify–judge pipeline generates executable fuzzy tasks grounded in tool schemas and evaluates trajectories with execution checks and rubric-based LLM judging.The benchmark covers diverse workflows and compositional dependencies.Source: §1 · pp. 2–3

2 Benchmark Construction

The benchmark formalizes fuzzy recommendation tasks as tool-using interactions with candidate items and ground-truth outcomes. It builds executable tasks from real-world recommendation contexts through an ambiguity-controlled synthesis pipeline.

2.1 Problem Formulation

  • Each task is represented by an underspecified instruction, available tools, candidate items, and ground truth for the target item and expected tool calls.Agents plan, invoke tools, and produce a final recommendation based on the instruction, tools, and prior observations.Source: §2.1 · p. 3

2.2 Task Synthesis

  • RecToolBench provides 13 domain-specific MCP servers with 32 executable tools spanning six recommendation capabilities.The capabilities include retrieval, review and sentiment analysis, rating-based filtering, personalized recommendation, geospatial reasoning, and web search.Source: §2.2 · p. 3
  • AmbiFlow identifies feasible workflows from tool schemas and user-item contexts, generates instructions and ground-truth tools, fuzzifies instructions, then filters tasks for quality.Fuzzification uses five ambiguity types while preserving underlying intent; human validation assesses realism, executability, and dependency correctness.Source: §2.2 · pp. 3–4
  • The benchmark contains 1,288 tasks: 300 L1 tasks and 988 compositional L2–L4 tasks, drawn from Amazon Electronics, Yelp 2019, and Foursquare interactions.The task counts are 675 Amazon, 465 Yelp, and 148 Foursquare.Source: §2.2 · pp. 3–4

3 Evaluation Protocol and Metrics

The evaluation protocol uses multi-round planning and tool execution while preserving compact planning context and a full trajectory. Metrics combine execution checks, comparisons with annotated workflows, and LLM-judge assessments of parameter and trajectory quality.

3.1 Evaluation Protocol

  • Each rollout maintains compact planning state Sp, an external Candidate Bus B, and full trajectory H; the planner uses the instruction, tool schemas, history, and candidate summary to plan or stop.The Candidate Bus preserves reusable intermediate items without adding full payloads to the planner context.Source: §3.1 · p. 4
  • Parallel or serial tool execution is followed by item-ID extraction and updates to candidate-bus status and agent state.These intermediate updates support subsequent planning.Source: §3.1 · p. 4
  • When the planning state exceeds the token budget or the response is empty, it is compressed while preserving tool names, round order, key parameters, and critical findings.This keeps long execution traces available for evaluation while maintaining compact planning context.Source: §3.1 · p. 4

3.2 Evaluation Metrics

  • Evaluation combines rule-based robustness checks with LLM-judge strategic-quality rubrics; PA covers L1-L2, trace dimensions assess L3-L4, and reference workflows are supplied only for L1-L3.L4 has no reference workflow because hybrid tasks can admit multiple valid parallel and sequential combinations.Source: §3.2 · pp. 4–6
  • Rule-based checks assess tool-name validity, schema compliance, and execution success, which requires a call to finish without error and return a non-empty result.Source: §3.2 · pp. 4–5
  • Table 2 presents representative Amazon Electronics task examples by difficulty, with L2-L4 examples drawn from relatively high-ambiguity queries.Source: Table 2 · p. 5
  • Tool Selection Accuracy or F1 compares predicted tools with gold annotations, while Parallel Ratio, Efficiency, and Hit Ratio measure orchestration and outcomes.Source: §3.2 · p. 5

4 Experiment Results

Across L1–L4, strong schema-level tool execution does not ensure successful recommendation: parameter grounding, dependency-aware orchestration, and evidence integration remain difficult, especially in hybrid tasks.

4 Experiment Results

  • The experiment evaluated nine representative models at difficulty levels L1–L4, averaging results within each level to isolate performance as orchestration complexity increased.LLM-as-a-Judge scores were averaged over five shuffled evaluations per task.Source: §4 · p. 6

4.1.1 Atomic Tool Using Abilities (L1)

  • In L1, proprietary models reached 100% Tool Selection Accuracy, while the best SLM, Qwen-2.5-7B-it, scored 58% Tool Selection Accuracy and 45% Parameter Accuracy.Format-valid calls could still be semantically wrong: Gemma-3-4B had 99% Tool Name Valid Rate but 49% Tool Selection Accuracy, and Qwen-2.5-3B had 95% Execution Success Rate but 10% Parameter Accuracy.Source: §4.1.1 · p. 6

4.1.2 Parallel Tool-Use Abilities (L2)

  • Across L2 models, rule-based Parameter Accuracy remained below 24%, and reference-free Parameter Appropriateness largely agreed with that ordering (Spearman ρ = 0.88).Some SLMs made as many or more calls than proprietary models, but repetitive or incorrectly parameterized calls often returned empty results.Source: §4.1.2 · pp. 6–7, Table 3 · p. 7

4.1.3 Sequential Tool-Use Abilities (L3)

  • GPT-5.4 and Claude-Sonnet-4.5 achieved 96.67% and 95.00% Hit Ratios in sequential L3, while SLMs remained weaker in dependency awareness and information grounding.Qwen-2.5-7B had the highest SLM Tool Selection F1 at 60.18%, but only 52.76% Hit Ratio.Source: §4.1.3 · pp. 7–8, Table 4 · p. 7

4.1.4 Hybrid Tool-Use Abilities (L4)

  • 54.05% and 44.59% Hit Ratio were the best proprietary-model results in hybrid L4, while SLMs reached 20%–28%.High schema compliance, execution success, and tool-selection scores still did not ensure hybrid-task success.Source: §4.1.4 · p. 8, Table 5 · p. 8

4.2 Round and Tool Calls Across Models

  • Across call and round patterns, proprietary models tracked the expected execution budget more closely, while Llama models repeated calls and SLMs often serialized independent L3 steps.Efficient tool use therefore involves redundancy control, dependency-aware parallelization, and reliable termination, not simply invoking tools.Source: Table 6 · p. 8, §4.2 · p. 9

4.3 Error analysis

  • GPT-5.4’s dominant L4 failure modes were dependency reasoning failures (54.1%) and target retrieval misses (53.6%).Even strong proprietary models struggled to maintain cross-step dependencies and propagate intermediate evidence into final recommendations.Source: §4.3 · p. 9

5 Related Work

Prior recommender-agent systems use heterogeneous tools and evaluation protocols, while existing recommendation and general tool-use benchmarks leave fuzzy intent grounding and broader recommendation workflows undercovered.

5.1 Tool-Augmented Recommender Systems

  • Tool-augmented recommender systems range from LLM-coordinated recommendation tools to surrogate-user tool chains, production multi-agent frameworks, and safety-intermediary agents.Their heterogeneous tool sets, task formulations, and evaluation protocols motivate a unified recommendation benchmark.Source: §5.1 · p. 9

5.2 Related Benchmarks

  • Existing recommendation benchmarks use static QA, narrow interactive shopping tasks, or limited e-commerce tool calling, while general tool and MCP benchmarks omit recommendation-specific fuzzy intent grounding and tool compositions.RecToolBench contrasts with these lines by providing recommendation-native tasks and explicit difficulty levels.Source: §5.2 · p. 9

6 Conclusion

RecToolBench benchmarks tool orchestration for fuzzy recommendation instructions across difficulty levels and workflow patterns. Its results identify grounding and coordination failures as continuing barriers to recommendation success.

6 Conclusion

  • RecToolBench evaluates tool-using agents under fuzzy recommendation instructions using executable tasks with graded difficulty and orchestration patterns from single-tool use to hybrid workflows.The benchmark is MCP-based.Source: §6 · p. 9
  • Current models often execute tools correctly but remain weak in parameter grounding, dependency tracking, evidence integration, and grounded recommendations.Source: §6 · p. 9
  • The authors identify tool orchestration under fuzzy recommendation instructions as a key bottleneck for reliable agentic recommender systems.Source: §6 · p. 9

Limitations

The paper identifies limits in its evaluation setting and benchmark coverage. The appendix also documents the MCP servers and tools used for the three dataset-based tasks.

Limitations

  • Evaluation is limited to a controlled MCP environment, a subset of recommendation scenarios, tool ecosystems, and user behaviors, and selected agent designs.The authors cite cost constraints for the limited model coverage.Source: Limitations · p. 10

A Details of Used MCP Servers

B Benchmark Details

This section details task synthesis, fuzzification, and quality checks, then defines the rule-based metrics used for Level 1 single-tool evaluation.

B.1 Synthesis Pipeline Details

  • Tasks receive low, moderate, or high ambiguity across five categories, sampled at 40%/40%/20%.The categories are underspecification, soft contradiction, vague quantifiers, preference uncertainty, and implicit constraints.Source: §B.1 · p. 16
  • Filtering checks scale with task level, from schema executability and parameter validity to dependency-chain and hybrid-workflow structure.Level 2 checks parallel independence, Level 3 checks dependency chains, and Level 4 checks hybrid parallel-serial structure and fan-in dependencies.Source: §B.1 · p. 16
  • Two experienced annotators inspect query naturalness, recoverable goals, workflow executability, and dependency annotations, with Cohen’s κ of 0.78.Disagreements are resolved through discussion, and low-quality instances are removed or regenerated.Source: §B.1 · p. 16

B.2 Prompts for Task Synthesis

  • The synthesis prompts cover user-query instruction generation, fuzzy matching, and data-quality filtering.The supplied description introduces these prompt categories without detailing their contents.Source: §B.2 · p. 16

C.1 L1 Metrics

  • Level 1 assesses schema understanding with five rule-based metrics, including tool-name validity, schema compliance, execution success, tool selection accuracy, and parameter appropriateness.An independent LLM judge scores parameter appropriateness under fuzzy intent without access to the generated reference.Source: §C.1 · p. 16

C.2 L2 Metrics

Level 2 evaluates parallel multi-tool tasks through schema, tool-use, planning, and completion metrics; Level 3 adds trajectory-based evaluation and judge-scored grounding and dependency awareness.

C.2 L2 Metrics

  • Level 2 evaluates schema adherence, tool selection and parameter passing, planning efficiency, and final recommendation quality in tasks intended to finish in one round.Its measures include Schema Compliance, Execution Success, Tool Selection F1, Parameter Accuracy, Parallel Ratio, Efficiency Score, and Hit Ratio.Source: §C.2 · pp. 16–17
  • Table 10 gives examples of tasks requiring parallel tool invocation or multi-turn sequential tool use.Source: Table 10 · p. 17
  • Efficiency Score reaches 1.0 when the agent invokes exactly the expected tools without redundancy, and decreases with redundant calls.The metric compares expected tools with the unique tools actually invoked across rounds.Source: §C.2 · p. 17

C.3 L3 Metrics

  • Level 3 scores multi-turn trajectories with schema, tool-use, sequential-efficiency, and end-to-end Hit Ratio measures.Sequential efficiency penalizes duplicate invocations with identical tools and arguments.Source: §C.3 · p. 17
  • A rubric-based LLM judge evaluates Information Grounding—whether the final answer derives from tool execution results rather than hallucinations—and Dependency Awareness—whether the agent follows dependencies using previous-step outputs—on 1–10 scales, normalized to [0, 1].Source: §C.3 · p. 18

C.4 L4 Metrics

The section specifies Level 4 hybrid-workflow metrics, guidance for generating fuzzy prompts, and criteria for filtering task quality; it also points to detailed Level 1 and Level 2 results tables.

C.4 L4 Metrics

  • Level 4 combines rule-based schema, execution, tool-selection, and Hit Ratio measures with rubric-based evaluation of planning, efficiency, parameter appropriateness, and grounding.Because hybrid workflows may not have a unique reference parameterization, parameter accuracy is evaluated semantically by an LLM judge rather than exact match.Source: §C.4 · p. 18

Fuzzy Prompt

  • Fuzzy prompts turn sequential tasks into conversational requests while preserving inferable dependencies and avoiding disclosure of the target item.Prompt instructions also preserve task values and candidate IDs, while applying specified ambiguity types and levels.Source: Fuzzy Prompt · p. 20

Data Quality Filter

  • The data-quality filter assesses solvability, recommendation utility, and consistency between a fuzzy description and its dependency analysis.Its criteria include tool and data availability, realistic preferences and multi-signal reasoning, and recoverability of the intended workflow.Source: Data Quality Filter · pp. 21–22

D.1 Level 1

  • Tables 11–13 report detailed Level 1 results for each dataset, with the highest and second-highest scores highlighted.Source: §D.1 · p. 23

D.2 Level 2

  • Tables 15–17 report detailed Level 2 results for each dataset, with the highest and second-highest scores highlighted.Source: §D.2 · p. 23

D.3 Error Analysis

The L4 error analysis finds that dependency reasoning failures and target retrieval misses are the most common annotated failure modes in GPT-5.4 trajectories.

D.3 Error Analysis

  • 54.1% of L4 trajectories show Dependency Reasoning Failure, while 53.6% show Target Retrieval Miss.Dependency failures involve mishandled tool dependencies; retrieval misses occur when the target item is not found despite a plausible trajectory.Source: §D.3 · p. 23
  • 18.5% of trajectories have no clear annotated failure, while 7.7% end prematurely before sufficient evidence or the required tool sequence.The no-clear-failure category includes trajectories that may be correct or fail for reasons outside the taxonomy.Source: §D.3 · p. 23
  • Unproductive Exploration accounts for 0.9% of trajectories and involves extra steps that do not improve the final decision.Such exploration may increase latency or distract from the original goal.Source: §D.3 · p. 23

D.4 Robustness of LLM-as-a-judge evaluation

The evaluation compares LLM-judge scores with ratings from three independent reviewers; prompt shuffling and score averaging raise the reported human agreement score.

D.4 Robustness of LLM-as-a-judge evaluation

  • Three independent reviewers score each judge pipeline’s output to assess alignment with human preferences.Source: §D.4 · p. 23
  • Reviewer scores use a 3-point scale: 0 denotes disagreement, 1 partial agreement, and 2 full agreement.Source: §D.4 · p. 23
  • With prompt shuffling and score averaging, the human agreement score rises from 1.37 to 1.52 out of 2.Source: §D.4 · p. 23

Related papers

Loading 2609.30717v1…