PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Seer: Synthesizing Scenarios from Specification Understanding

Harish Karumuri, Mahesh Vemula, David Lopes Pegna

arXiv:2608.26133v1cs.CL

TL;DR

Realistic, current evaluation scenarios for tool-using agents are costly to curate and quickly become outdated as APIs evolve. Agent Seer generates complete MCP-based evaluation harnesses without live tools or manual annotation, achieving mean tool-calling correctness of 0.911, mean coherence of 0.855, and complete coverage on small and medium specifications.

  • Problem

    Realistic tool-agent evaluation remains laborious to curate, while fixed benchmarks can become outdated as tool APIs evolve.

  • Method

    Agent Seer converts MCP tool specifications alone into complete harnesses containing graded scenarios, mock outputs, expected tool sequences, and multi-turn dialogues.

  • Results

    Mean tool-calling correctness is 0.911 and mean coherence is 0.855 across 337 scenarios on seven MCPs, with complete coverage on small and medium specifications.

  • Takeaways & Limitations

    Parameter schema complexity most strongly correlates with quality variation, while argument value accuracy is the dominant remaining sub-failure.

  • Takeaways & Limitations

    The framework relies on LLM-generated ground truth and is best understood as a proxy for identifying broad capability gaps and relative performance differences.

Abstract

from arXiv · show

Evaluating AI agents that use external tools requires realistic test scenarios that capture how practitioners compose tools and iterate across conversation turns. Constructing such scenarios by hand demands deep domain expertise, does not scale across tool ecosystems, and produces static benchmarks that cannot track evolving APIs. We observe that tool specifications -- function names, natural-language descriptions, and typed parameter schemas -- already encode sufficient semantic information to synthesize realistic evaluation scenarios without manual curation or live tool execution. Agent Seer builds off this latent information: from a single Model Context Protocol (MCP) specification, with no examples, no live tool access, and no domain-specific tuning. This pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded multi-turn dialogues that exhibit strong tool-calling correctness and conversational coherence. Evaluation quality is measured by applying this pipeline on seven MCP specifications spanning diverse domains and tool-suite sizes and measuring the tool-calling correctness and conversational coherence. The pipeline achieves strong quality across all domains, with complete tool coverage on small and medium specifications. Two findings emerge within this analysis: parameter schema complexity is the strongest correlate of quality variation -- tool-suite size plays a smaller, orthogonal role -- and argument value accuracy is the dominant failure mode among imperfect scenarios, a sub-dimension invisible to coarse-grained name-match metrics.

1 Introduction

Agent evaluation is hindered by costly manual curation, static benchmarks that lag evolving APIs, and difficulty generating tool-output-grounded multi-turn interactions. Agent Seer addresses this cold-start problem by converting MCP tool specifications alone into self-contained evaluation harnesses through structured scenario, mock-output, and dialogue generation.

  • Evaluation challenges: Manual benchmarks require connecting user intent to plausible parameterized tool calls and cross-turn tool chains, making coverage across combinatorial tool pathways impractical to scale by hand.Hand-authored benchmarks include Froger et al. (2026), Liu et al. (2023a), and Yao et al. (2025).
  • Evaluation challenges: Fixed benchmarks become stale as tool APIs evolve, while multi-turn evaluation must generate follow-ups reacting to specific tool outputs rather than repeating generic instructions.The latter is particularly difficult without access to real tool responses.
  • Agent Seer: Tool specifications encode enough semantic information for an LLM to infer workflows, populate realistic parameters, synthesize tool responses, and construct grounded multi-turn dialogues without live execution or manual annotation.This shifts the bottleneck from human curation to structured extraction, especially for new, private, rapidly evolving, and long-tail enterprise APIs.
  • Agent Seer: Agent Seer implements four validated stages—semantic interpretation, scenario generation, mock output synthesis, and multi-turn expansion—to produce self-contained harnesses with expected tool sequences, calibrated mock outputs, data-grounded dialogues, and held-out oracles.The harness artifacts are decoupled from execution backends and can be consumed by MCP-compatible evaluation frameworks.

2 Background and Related Work

Tool-calling evaluation data is difficult to obtain for new, private, or rapidly evolving tool suites because realistic multi-turn scenarios require domain expertise. Agent Seer addresses this cold-start gap by synthesizing execution-free evaluation harnesses from tool specifications through structured, validated stages and synthetic tool responses.

  • Motivation: Agent Seer addresses the absence of representative scenarios, tool sequences, and outputs for new or evolving tool suites without requiring live execution.Hand curation requires identifying important user goals and modeling tool composition, partial results, and cross-turn iteration.
  • Related Work: Prior evaluation spans single-function prediction, multi-step use over curated APIs, multi-turn simulated-user assessment, and broad benchmarks including GAIA, AgentBench, and WorkArena.These approaches provide increasingly rich evaluation environments but do not by themselves resolve the cold-start problem for unavailable or changing tool suites.
  • Related Work: Synthetic-data methods divide into live-execution trajectory generation, simulated tool environments, and specification-only generation.Live-execution examples include APIGen, TOUCAN, GEM, and Agent World Model; simulated examples include Simia, Gecko, LOGIGEN, τ-bench, τ 2-bench, and ToolSandbox; specification-only examples include DiGiT-TC and FuncBenchGen.
  • Related Work: Tool-use evaluation commonly relies on exact match, execution success rate, or single-call binary AST matching, while newer methods decompose traces or analyze argument errors.FuncBenchGen reports systematic propagation of stale or incorrect arguments, and Agent GPA evaluates Goal-Plan-Action stages in live environments.
  • Related Work: Unlike live-execution and simulated-environment approaches, Agent Seer uses only tool specifications while synthesizing plausible responses without a runtime environment.Its distinction is the execution-independent setup, not prompt-driven generation itself.
  • Contribution: Agent Seer’s contribution is a four-stage pipeline with validated structured outputs at each boundary, producing harnesses decoupled from any execution backend.This structure distinguishes the method from other prompt-driven generation systems.

3 Pipeline Stages

Agent Seer transforms MCP specifications into a self-contained evaluation harness through four schema-validated stages: semantic enrichment, scenario generation, synthetic outputs, and multi-turn expansion. The resulting dialogues preserve tool dependencies, ground follow-ups in mock data, and enable evaluation without live tool access.

  • Four schema-validated stages catch malformed artifacts at boundaries while transforming MCP specifications into a self-contained evaluation harness.A downstream framework supplies prompts and mock tool responses, then scores emitted calls against the scenario workflow as a held-out oracle.
  • Semantic enrichment converts each raw tool specification into functional descriptions, parameter roles, primary use cases, and organizational context.Structured prompts connect terse API documentation to richer scenarios.
  • Scenario generation creates simple single-domain workflows and complex multi-domain workflows, each specifying ordered calls, parameter values, novelty, and follow-up questions.Quick explanations and novelty reasons make tool-selection and workflow-composition reasoning explicit during generation.
  • Synthetic output generation can operate from specifications alone or use example traces, matching available structure and recording each mock output’s grounding tier.This makes the method a spectrum from fully unsupervised generation to trace-informed fidelity.
  • Multi-turn expansion splits scenarios at phase boundaries to preserve BFCL v3 multi-step and multihop dependencies, grounding follow-ups in concrete synthetic-output values.Single-turn expansions are discarded, while successful expansions produce data-grounded conversational dialogues.

4 Generation Quality Assessment

Generation quality is assessed with an unsupervised LLM-as-judge framework measuring tool-calling correctness and conversational coherence. Tool-calling correctness is decomposed into four normalized dimensions, while coherence aggregates five normalized sub-aspects.

  • Evaluation framework: The framework evaluates tool-calling correctness (TC) and conversational coherence (Coh) using LLM-as-judge scoring, exclusively through the unsupervised path.Generated scenarios serve as both outputs and oracle data; oracle-grounded evaluation is supported when reference data exists.
  • Tool-calling correctness: Tool-calling correctness averages necessity, tool selection, tool ordering, and tool argument correctness after independent 0–10 scoring and normalization to 0–1.Necessity concerns whether a tool was warranted; selection covers correctness, specificity, and completeness; ordering covers sequence logic, dependencies, and efficiency, and is excluded when one tool is called.
  • Tool-calling correctness: Tool argument correctness averages completeness, name, value, type, format, and relevancy, with cascading penalties causing critical parameter errors to collapse the argument mean.A wrong parameter name or missing required parameter zeros value, type, and format; a wrong value cascades to type, format, and relevancy.
  • Conversational coherence: Coherence averages logical flow, completeness, conciseness, topic relevance, and context retention, while turn and conversation scores use arithmetic means.Coherence sub-aspects are scored on a 1–3 scale and normalized to 0–1; per-MCP rankings and schema-complexity correlation remain robust to alternative aggregation schemes.

5 Experimental Evaluation

Across 337 scenarios on seven MCPs, Agent Seer achieves mean tool-calling quality of 0.911 and mean coherence of 0.855. Quality is broadly consistent, with complete tool coverage for six specifications and parameter schema complexity emerging as the main source of variation and failure.

  • Robustness and coherence: Out-of-family judging preserves the tool-calling results, while coherence remains only weakly correlated with tool-calling at record level (r = +0.23).The cross-family comparison finds ∆µTC≈0 and preserves MCP ranking with ρ = 0.86; Slack has the highest coherence at 0.938 and Git the lowest at 0.757.
  • Quality correlates: Parameter schema complexity correlates negatively with tool-calling quality, whereas tool count correlates only modestly and positively.At tool level, parameter count and optional-parameter fraction correlate with tool calling at r = −0.29 and −0.30, both p < 0.001.
  • Complexity breakdown: Complex scenarios reduce tool calling by −7.3pp and coherence by −5.3pp relative to simple scenarios.The largest tool-calling degradation occurs in Elasticsearch (−12.2pp), followed by Git (−11.1pp), while Selenium is stable.
  • Tool-calling failures: Argument correctness is the dominant tool-calling failure: only 42% of records are perfect, with value accuracy accounting for 223 failures.The pipeline generally gets names and types right but struggles with parameter values and completeness, including omitted Redis expiration parameters that coarse name-match metrics miss.
  • Overall quality: 0.911 mean tool-calling and 0.855 mean coherence are achieved across 337 scenarios spanning seven MCPs.The corresponding medians are 0.979 and 0.933, respectively.

6 Conclusion

Agent Seer converts MCP tool specifications into reusable evaluation harnesses without live tool execution or manual annotation. Across seven diverse specifications, it achieves full tool coverage on small and medium MCPs while identifying diagnostic quality patterns, with the methodology intended to address the cold-start evaluation gap.

  • Agent Seer converts MCP specifications into complete harnesses containing graded scenarios, mock tool outputs, and multi-turn dialogues without live execution or manual annotation.The harnesses are reusable artifacts for live tool environments or simulated agents.
  • Across n = 7 structurally diverse specifications, the pipeline achieves full tool coverage on small and medium MCPs and identifies parameter schema complexity as the strongest correlate of quality variation.Argument value accuracy is also surfaced as a consistent diagnostic pattern among imperfect scenarios.
  • The findings are observations within the experiment rather than universal claims, while the durable contribution is a methodology that closes the cold-start evaluation gap for MCP-compatible tool suites.The reusable harnesses can feed live tool environments or simulated agents (Li et al., 2025).

Limitations · A Extended Experimental Results

The evaluation is limited by LLM-generated ground truth, independently generated sequential-call mocks, and restricted experimental scope. Coverage also declines for large tool specifications, while complexity stratification and LLM-based judging require additional safeguards.

  • Limitations: LLM-generated ground truth introduces systematic bias, so the framework should be treated as a proxy for broad capability gaps and relative performance differences.It does not establish direct ground-truth validity.
  • Limitations: Independent mock-output generation can break referential integrity because IDs or values may not align across dependent sequential calls.A shared workflow state dictionary is proposed to address this limitation.
  • Limitations: At 64 tools, Illustrator coverage falls to 56%, motivating coverage-aware iterative generation for larger specifications.The proposed direction includes iterative generation with targeted tool sampling.
  • Limitations: Cross-domain workflows and extensions to OpenAPI, gRPC, and function-calling schemas are outside the evaluation scope despite being structurally straightforward.The existing name-plus-typed-parameter scaffolding supports these as directions for future work.
  • Limitations: Prompt framing rather than structural enforcement determines simple-versus-complex stratification, making post-generation structural filters a needed improvement.Structural complexity metrics could improve the reliability of this distinction.
  • Limitations: Seven MCP specifications, one generation model, and 54 multi-turn records cannot establish universal claims or provide strong statistical power.Expansion is skewed toward complex scenarios: 30.8% versus 2.8% for simple scenarios, because meaningful follow-up turns require sufficient workflow substance.
  • Limitations: LLM-as-judge circularity may reward scenarios that look good to an LLM rather than scenarios with actual quality.The framework addresses this partly through a capability gap between the Gemini 2.5 Flash judge and weaker Gemini 2.5 Flash Lite generator.

A.1 Evaluation Scale … A.5 Workflow Composition and Category Diversity

The evaluation covers 337 generated scenarios and 391 turn-level records across seven MCP specifications, with quality varying by scenario complexity and coherence dimension. Workflows range from atomic tool use to long sequential compositions, while all 871 mock calls lack specification-provided example outputs.

  • A.1 Evaluation Scale: 337 generated scenarios yielded 391 evaluation records because multi-turn scenarios contribute one record per turn.Table 6 breaks down the evaluation scale by MCP specification; records therefore exceed scenarios.
  • A.2 Complexity Breakdown: Tool-calling quality degrades most under complexity for Elasticsearch (−12.2pp) and Git (−11.1pp), whereas Selenium is slightly higher on complex scenarios.The ∆ column measures the change from simple to complex scenarios, with negative values indicating degradation.
  • A.3 Coherence Sub-Dimensions: Conciseness averages 2.90 (std 0.33) on the raw 1–3 coherence scale, while completeness is the primary bottleneck at 2.04.The completeness deficit reflects responses that address requests but omit contextual detail practitioners would expect.
  • A.4 Grounding Tiers: Grounding tiers indicate whether mock outputs use concrete examples, transferable examples from similar tools, or only parameter schemas.All seven MCP specifications lack example outputs, so the 871 mock calls are assigned under that no-example condition.
  • A.5 Workflow Composition and Category Diversity: Selenium produces the longest workflows, averaging 9.71 calls with a maximum of 17, reflecting sequential browser automation tasks.Illustrator workflows are also comparatively long, averaging 3.33 calls and reaching 12.
  • A.5 Workflow Composition and Category Diversity: Redis and Slack favor atomic single-tool scenarios, averaging 1.27 and 1.39 calls, respectively.These workflow lengths contrast with the longer compositions observed for Selenium and Illustrator.
  • A.5 Workflow Composition and Category Diversity: The 337 scenarios span 108 unique categories, or 3.1 scenarios per category, with category count scaling by scenario count rather than tool count.This indicates category diversity tracks the number of generated scenarios more closely than MCP tool-suite size.

A.6 Failure Details · A.7 Per-MCP Dimension Scores

Failures concentrate in argument values, especially for Git’s complex-schema tools, while per-MCP scores show Selenium highest overall and Git weakest on arguments. The analysis also categorizes coherence issues from evaluator notes.

  • A.6 Failure Details: Git has the highest failure rate at 7%, followed by Filesystem at 5% and Redis at 1%, while four MCPs have zero failures.
  • A.6 Failure Details: Git’s failures are driven by argument correctness for tools with complex parameter schemas.
  • A.6 Failure Details: 223 argument failures involve value accuracy, exceeding relevancy (44), format compliance (35), type compliance (31), completeness (16), and name accuracy (11).
  • A.6 Failure Details: Table 10 groups coherence issues into a taxonomy derived from evaluator notes.
  • A.7 Per-MCP Dimension Scores: Selenium’s overall mean TC score is 0.989, the highest of any MCP despite its longest workflows.
  • A.7 Per-MCP Dimension Scores: Git’s argument dimension score is 0.780, the weakest, consistent with its high parameter density.

A.8 Out-of-Family Judge Replication

Out-of-family judging replicates the tool-calling ranking and value-accuracy failure pattern, while coherence scores and rankings vary substantially by judge. Tool-calling is unbiased across judges, but Qwen 3.5 rates coherence more strictly, so absolute coherence levels remain judge-dependent.

  • Judge replication: Tool-calling means and rankings are robust across judges, with overlapping per-MCP CIs and MCP-level correlation ρ=0.86; Redis remains the top MCP.Table 12 reports 95% percentile bootstrap CIs with B=2,000; tool-calling CIs overlap for every MCP.
  • Failure-mode taxonomy: Both judges identify value_accuracy as the dominant argument failure mode by a 4–5× margin, with failure counts agreeing within 5%.This replicates the central qualitative finding that value-accuracy errors dominate the argument sub-dimensions.
  • Limitations: Without a third judge or human anchor, neither judge can be designated correct for coherence, so absolute coherence levels are reported as judge-dependent.The systematic coherence shift reflects stricter out-of-family scoring rather than a contradiction in the underlying data.
  • Visual diagnostics: Visual agreement is tight for tool-calling but offset for coherence: the coherence Bland–Altman limits shift approximately −0.14 below zero with wider spread.Tool-calling differences are centered near zero, whereas coherence differences show a visible offset below y=x.
  • Judge replication: Coherence is judge-dependent: Qwen 3.5 scores systematically lower, with ρCoh=0.46 and different worst MCPs—Git under Gemini versus Illustrator under Qwen 3.5.Coherence CIs are disjoint for all seven MCPs, and the Qwen 3.5−Gemini mean difference is ∆µCoh≈−0.16 with 95% CI [−0.173, −0.138].

B Aggregation Sensitivity … G.1 Overall Score Distributions

The paper combines and evaluates tool-calling and coherence scores, documents a four-stage prompt pipeline with structured outputs, and analyzes score distributions across 391 records. It also tests aggregation sensitivity and situates its contributions against prior literature across data-generation and metric-decomposition gaps.

  • B Aggregation Sensitivity: Per-MCP rankings remain highly stable under arithmetic, harmonic, and minimum aggregation despite corpus-wide mean shifts to 0.911, 0.851, and 0.781.Spearman ρ is 0.964 for arithmetic versus harmonic, 0.964 for arithmetic versus minimum, and 0.929 for harmonic versus minimum; only Filesystem and Git swap near the bottom.
  • C Novelty Summary: The contribution is positioned against prior literature through two gaps: data generation and metric decomposition.Table 17 organizes these comparisons into rows 1–3 and rows 4–6, respectively.
  • D Pipeline Prompts: The pipeline interprets each MCP tool specification into a structured five-field semantic explanation covering functionality, parameters, use cases, and enterprise context.The interpreter returns JSON constrained to the tool name, what it does, what it needs, why it is used, and enterprise context.
  • D Pipeline Prompts: Scenario-generation prompts produce simple or complex enterprise workflows with exact function calls, realistic schema-conforming parameters, follow-ups, and broad tool coverage.A coverage suffix requires every available tool to appear in at least one workflow, with follow-up scenarios added for uncovered tools.
  • D Pipeline Prompts: Synthetic outputs replace placeholders with concrete data, follow reference schemas or examples, assign confidence levels, and make expected responses reference the full workflow.Confidence is high for concrete function-specific examples, medium for analogous examples, and low when no example output is provided.
  • E Evaluation Prompts: Evaluation uses unsupervised LLM judges for tool-calling correctness and conversational coherence, with normalized dimension scores aggregated by arithmetic mean.Tool-calling assesses four dimensions on 0–10 subscales with cascading argument penalties, while coherence assesses five dimensions on 1–3 scales.
  • F Structured Output Schemas: Pydantic models constrain every pipeline stage to produce well-formed JSON.The structured schemas provide output constraints throughout the pipeline.
  • G.1 Overall Score Distributions: Across 391 evaluation records, tool-calling scores concentrate in the upper range, with 31.7% perfect scores and 2.3% below 0.5, while coherence is left-skewed with a mode near 0.9.Figure 3 displays both overall score distributions.

G.2 Per-Domain Analysis

Figures 4–10 characterize each MCP specification through scenario categories, workflow lengths, tool co-occurrence, and sequential adjacency graphs. Workflow means range from 1.3 calls for Redis to 9.7 for Selenium, with maxima of 12 for Illustrator and 17 for Selenium.

  • Per-Domain Analysis: Across all seven specifications, Figures 4–10 report scenario categories, workflow-length distributions, tool co-occurrence, and sequential adjacency patterns.Co-occurrence graphs are undirected, whereas adjacency graphs represent directed tool-transition patterns within ordered workflows.
  • Per-Domain Analysis: 3.3 calls is Illustrator’s mean workflow length, with a maximum of 12 calls.Illustrator’s figure also presents its scenario categories, co-occurrence graph, and sequential adjacency graph.
  • Per-Domain Analysis: Slack, Filesystem, and Git have mean workflow lengths of 1.4, 2.0, and 2.2 calls, respectively.Their figures likewise show scenario categories, tool co-occurrence, and sequential adjacency.
  • Per-Domain Analysis: 9.7 calls is Selenium’s mean workflow length, compared with 1.3 for Redis and 1.9 for Elasticsearch.Selenium workflows reach a maximum of 17 calls, while Elasticsearch averages 1.9 calls.
Loading 2608.26133v1…