Agent Seer: Synthesizing Scenarios from Specification Understanding

1 Introduction

• Manual curation cannot practically scale realistic scenario coverage across the combinatorial space of tool pathways.Realistic scenarios must connect user intent to specific tool calls, supply plausible parameter values, and capture tool chaining across turns.Source: §1 · p. 1
• Fixed benchmarks become outdated as APIs evolve, while multi-turn evaluation requires follow-ups grounded in specific tool outputs rather than generic instructions.Generating such output-responsive interactions is particularly difficult without access to real tool responses.Source: §1 · p. 1
• Tool specifications encode semantics for inferring plausible workflows, realistic parameters, synthetic responses, and multi-turn dialogues, shifting the bottleneck from human curation to structured extraction.This addresses cold-start evaluation: producing realistic evaluation data for tool suites without existing data, particularly new, private, or rapidly evolving APIs.Source: §1 · pp. 1–2
• Agent Seer transforms MCP specifications into evaluation harnesses through four stages: semantic interpretation, scenario generation, mock output synthesis, and multi-turn expansion.Validated stage boundaries catch schema violations. Artifacts include held-out oracles and mock outputs, enabling any MCP-compatible framework to execute and grade agents.Source: §1 · p. 2
• Seven MCP specifications show generation quality variation correlates more strongly with parameter schema complexity than with tool-suite size.The specifications span diverse enterprise domains, with complete tool coverage achievable on small and medium specifications.Source: §1 · p. 2
• Argument sub-dimension cascading is the dominant generation failure mechanism, invisible to coarse-grained name-match metrics.Domain-specific failure signatures characterize where and why the pipeline falls short.Source: §1 · p. 2

2 Background and Related Work

• Tool-agent evaluation faces a cold-start gap when scenarios, expected tool sequences, and representative outputs are unavailable or require deep domain expertise to curate.New, private, or rapidly evolving tool suites may lack usage data; available data can omit niche cases and realistic multi-turn tool composition.Source: §2 · p. 2

2.1 Agent Benchmarks

• Agent benchmarks have expanded from single-function prediction to multi-turn evaluation, but remain static and require manual curation or live tool access.Examples include general evaluation environments such as GAIA (Froger et al., 2026) and MCP-centric suites such as MCPVerse (Lei et al., 2025).Source: §2.1 · p. 2

2.2 Synthetic Data Generation

• Synthetic-data approaches generate training trajectories through live tool execution or replace tools with simulated environments for training and evaluation.APIGen (Liu et al., 2024; Prabhakar et al., 2025) uses execution-based verification. Simulated evaluation environments include τ-bench, τ 2-bench (Barres et al., 2025), and ToolSandbox (Lu et al., 2025).Source: §2.2 · pp. 2–3
• Spec-only generation produces synthetic data directly from specifications, including fine-tuning requests and contamination-free benchmarks.DiGiT-TC (Crouse et al., 2026) back-translates tool-call sequences into user requests; FuncBenchGen (Maekawa et al., 2026) models call dependencies as DAGs to define benchmarks.Source: §2.2 · p. 3

2.3 Evaluation Methodology

• Agent Seer evaluates tool-calling correctness and conversational coherence per turn through structured multidimensional LLM judging, including per-argument decomposition.FuncBenchGen (Maekawa et al., 2026) identifies stale or incorrect arguments propagating across syntactically valid chained calls—a failure that coarse-grained name matching misses.Source: §2.3 · p. 3

2.4 Positioning

• Agent Seer combines specification-only input, LLM-synthesized tool responses, and validated outputs across four stages to produce harnesses independent of execution backends.Its claimed novelty is the pipeline structure, not the prompt-driven generation mechanism shared with earlier systems.Source: §2.4 · p. 3

3.1 Tool Interpretation

• Tool interpretation enriches raw MCP specifications with functional descriptions, required parameters and their semantic roles, primary use cases, and organizational context.A structured LLM prompt requests these four semantic fields for each tool, connecting terse API documentation to richer scenarios.Source: §3.1 · p. 3

3.2 Scenario Generation

• Scenario generation uses prompting to distinguish simple, single-domain operational workflows from complex, novel workflows combining tools across domains.Each scenario includes a title, user-facing instruction, ordered tool calls with parameter values, novelty explanation, and natural follow-up question. Per-call quick_explanation and scenario-level novelty_reason fields require reasoning about tool selection, workflow composition, and evaluation value.Source: §3.2 · p. 3

3.3 Mock Output Generation

• Mock output generation synthesizes a response for every function call, using optional examples when available and tool descriptions otherwise.Provided examples guide output structure, allowing available traces to improve fidelity without making them mandatory. Each output records a high, medium, or low grounding tier reflecting reference-material availability.Source: §3.3 · p. 3

3.4 Multi-Turn Scenario Expansion

• Multi-turn expansion splits scenarios at natural phase boundaries, preserves tool-call dependencies, and grounds follow-ups in concrete synthetic output values.Splits preserve the multi-step dependencies and multihop synthesis patterns formalized by BFCL v3. The prompt prescribes no turn count, but single-turn expansions are discarded as insufficiently substantive.Source: §3.4 · pp. 3–4

3.5 Harness Artifact Format

• The pipeline produces a self-contained evaluation harness that tests agents on previously unseen tool suites without live access.A downstream framework presents prompts, supplies mock tool responses, and scores emitted calls against the generated scenario workflow as a held-out oracle.Source: §3.5 · p. 4

4 Generation Quality Assessment

• Generation quality evaluation combines tool-calling correctness and conversational coherence using LLM-as-judge scoring (Liu et al., 2023b).Although the framework supports oracle-grounded evaluation when reference data exists, this study exclusively uses unsupervised evaluation because generated scenarios serve as both output and oracle.Source: §4 · p. 4

4.1 Tool-Calling Scoring

• Tool-calling evaluation replaces coarse pass/fail scoring used in prior work (Qin et al., 2024) with four independently scored dimensions, normalized from 0–10 to 0–1.Arithmetic means aggregate dimensions within turns and turns within conversations. Per-MCP rankings and the schema-complexity correlation remain robust under harmonic-mean and minimum aggregation.Source: §4.1 · p. 4
• Tool usage scoring measures necessity (Huang et al., 2023), while tool selection scoring averages correctness, specificity, and completeness.Overuse is recorded diagnostically but excluded from the aggregate because necessity dominates appropriateness.Source: §4.1 · p. 4
• Tool ordering scoring averages sequence logic, dependency handling, and execution efficiency, excluding single-tool turns where ordering is not applicable.For single-tool turns, the ordering dimension is omitted from the turn-level aggregate.Source: §4.1 · p. 4
• Argument correctness averages six sub-dimensions, with cascading penalties ensuring that a single critical error collapses the argument mean.The sub-dimensions are completeness, name, value, type, format, and relevancy. Prompt instructions zero value, type, and format for wrong names or missing required parameters; wrong values also zero type, format, and relevancy.Source: §4.1 · p. 4

4.2 Coherence Scoring

• Conversational coherence averages five aspects: logical flow, completeness, conciseness, topic relevance, and context retention.Each aspect is scored on a 1–3 scale, normalized to 0–1, and aggregated using an arithmetic mean.Source: §4.2 · p. 4

5.1 Experimental Setup

• 337 scenarios across seven MCP specifications produced 391 evaluation records, with multi-turn expansion succeeding for 54 scenarios (16.0%).Gemini 2.5 Flash Lite generated schema-constrained outputs at temperature 0.7 with up to three validation retries; expansion favored complex scenarios (30.8%) over simple ones (2.8%). Gemini 2.5 Flash judged at temperature 0, averaging four tool-calling dimensions with cascading argument penalties and five coherence dimensions normalized from a 1–3 scale.Source: §5.1 · p. 5

5.2 Quality Results

• Across 337 scenarios on seven MCPs, the pipeline achieves a mean unsupervised tool-calling score of 0.911 and mean coherence of 0.855, with all seven MCPs exceeding 0.85 overall.The respective 95% confidence intervals are [0.897, 0.925] and [0.838, 0.872]; all seven MCPs exceed 0.91 on simple scenarios. Out-of-family re-scoring preserves tool-calling results (approximately zero mean shift, paired r=0.79, MCP ranking ρ=0.86), but coherence shifts approximately −0.16 with only partial MCP ranking preservation (ρ=0.46).Source: Table 3 · p. 5, §5.2 · p. 5, §5.6 · p. 7
• 7.3pp lower tool-calling scores distinguished complex from simple scenarios, while tool-calling and coherence remained largely independent diagnostic signals.Complex scenarios also lost 5.3pp in coherence, with non-overlapping simple/complex confidence intervals for both metrics; tool-calling degradation varied from −12.2pp for Elasticsearch to stability for Selenium (0.932→0.935). Tool-calling/coherence correlations were weak (record-level r = +0.23; tool-level r = +0.16); missing information or shallow responses dominated coherence issues (373 records), and 23% of records combined correct tools with poor coherence at a 0.75 threshold across all MCPs.Source: §5.2 · p. 6, §5.3 · p. 7

5.3 Failure Mode Analysis

• 57% of records received partial argument scores, with value accuracy dominating argument failures (223 records versus 44 for relevancy).Usage was near-perfect, selection was correct on 77% of records, and ordering failed on 10% of multi-tool scenarios; only 42% achieved perfect argument scores. Redis illustrated a failure invisible to name matching: correct tool names and key/value arguments accompanied omitted expiration parameters implied by the scenario.Source: Table 4 · p. 6, §5.3 · pp. 6–7
• 0.336% of tool invocations were hallucinated (3 of 893), all in Git, while complex Git failures reflected parameter overload.Three simple Git scenarios substituted real CLI commands absent from the specification, accounting for half of Git’s six records with TC<0.5. Git averaged 11.2 parameters per tool, 95% optional, with differing semantics for shared parameter names; its mean argument score was 0.780, falling to 0.726 on complex scenarios as workflows compounded per-call errors.Source: §5.3 · p. 7

5.5 Coverage and Diversity

• 14–56-tool specifications achieved complete tool coverage, whereas Illustrator’s 64-tool specification reached 56%.Tool-usage Gini coefficients ranged from 0.146 for Redis to 0.340 for Git, except Selenium’s 0.676, reflecting workflows averaging 9.7 calls concentrated on core tools. The corpus contained 781 unique tool co-occurrence pairs across 138 multi-tool scenarios and 108 unique scenario categories.Source: Table 5 · p. 7, §5.5 · p. 7

5.4 Domain Cluster Analysis

• 0.935 TC for Selenium exceeded Illustrator’s 0.898 despite their similarly large tool suites, attributed to Selenium’s lower parameter density.The structurally similar Redis–Elasticsearch pair differed by only ∆0.036 TC. Filesystem and Git differed by ∆0.019 TC despite parameter densities of 1.8 versus 11.2, because selection and ordering compensated for Git’s lower argument correctness (0.780 versus Filesystem’s 0.894).Source: §5.4 · p. 7

5.6 Summary of Findings

• Parameter schema complexity was the strongest correlate of quality variation, while tool-suite size played a smaller, orthogonal role.Across seven MCPs, average parameters per tool and optional-parameter fraction correlated negatively with tool-calling (r = −0.60 and −0.66), while tool count correlated positively (r = +0.40). Across 222 tools, the schema correlations remained negative for tool-calling (r = −0.29 and −0.30) and coherence (r = −0.41 and −0.34), all p < 0.001.Source: §5.2 · pp. 5–6, §5.6 · p. 8

6 Conclusion

• Seven structurally diverse specifications were evaluated, with Agent Seer achieving full tool coverage on small and medium MCPs.Full coverage is not claimed for all specification sizes.Source: §6 · p. 8
• Parameter schema complexity is the strongest correlate of quality variation in this sample, while argument value accuracy is the dominant remaining sub-failure.These findings are observations from n = 7 specifications, not universal claims.Source: §6 · p. 8
• Agent Seer’s four-stage pipeline converts MCP tool specifications into graded scenarios, mock tool outputs, and multi-turn dialogues without live tool execution or manual annotation.The authors identify the methodology as the durable contribution.Source: §6 · p. 8
• Reusable harnesses feed live tool environments or simulated agents (Li et al., 2025), closing the cold-start evaluation gap for MCP-compatible tool suites.Source: §6 · p. 8

Limitations

• LLM-generated ground truth introduces systematic generator biases, making the framework a proxy for identifying broad capability gaps and relative performance differences.The paper identifies ground truth reliability as its most significant limitation.Source: Limitations · p. 8
• Independently generated mock outputs can leave IDs or values inconsistent across dependent calls.A shared state dictionary across each workflow would address cross-call referential integrity.Source: Limitations · p. 8
• 56% coverage at 64 tools in Illustrator exposes a coverage ceiling for larger specifications.The proposed remedy is iterative generation with coverage-aware tool sampling.Source: Limitations · p. 8
• Workflows spanning multiple MCP specifications and extensions to OpenAPI, gRPC, or function-calling schemas remain outside the evaluation.The paper describes these extensions as structurally straightforward because the formats share function names and typed-parameter scaffolding, but leaves their evaluation to future work.Source: Limitations · p. 8
• Simple-versus-complex stratification relies on prompt framing rather than structural enforcement.Post-generation filters based on structural complexity metrics could improve stratification.Source: Limitations · p. 8
• Seven MCP specifications, one generation model, and limited multi-turn records (n = 54) constrain universal claims and statistical power.Generation used Gemini 2.5 Flash Lite. Expansion favored complex scenarios (30.8%) over simple scenarios (2.8%), reflecting the need for sufficient workflow substance to support meaningful follow-ups.Source: Limitations · p. 8

A Extended Experimental Results

• r≈0.79 tool-calling agreement (n=384), no mean shift, and preserved MCP ranking (ρ=0.86) support replication with Qwen3.5-122B-A10B-FP8.Re-scoring covered the full 391-record corpus. Argument value-accuracy failures dominated under both judges: Gemini 240 records and Qwen 3.5 252, a 4–5× margin over the next sub-dimension.Source: Limitations · p. 9

Limitations

• ∆µ≈−0.16 for coherence and moderate record-level correlation (r=0.42) show that absolute coherence scores and MCP-level rank preservation remain judge-dependent.Using a stronger judge than generator and out-of-family replication addresses, but does not eliminate, LLM-as-judge circularity. Systematic validation against human quality judgments remains future work.Source: Limitations · p. 9

A.1 Evaluation Scale

• 337 generated scenarios yield 391 evaluation records because multi-turn scenarios contribute one record per turn.Records comprise a single-turn entry plus one multi-turn entry for each additional turn.Source: §A.1 · p. 11, Table 6 · p. 11

A.2 Complexity Breakdown

• −12.2pp for Elasticsearch and −11.1pp for Git are the largest tool-calling score declines from simple to complex scenarios.Selenium remains stable, scoring slightly higher on complex scenarios.Source: Table 7 · p. 11, §A.2 · p. 11

A.3 Coherence Sub-Dimensions

• 2.04 mean completeness is the main coherence bottleneck, whereas conciseness is near-ceiling at mean 2.90 (standard deviation 0.33) on the 1–3 scale.Responses address user requests but omit contextual detail that practitioners would expect, making completeness the most actionable improvement target.Source: §A.3 · p. 11, Table 8 · p. 11

A.4 Grounding Tiers

• 871 mock calls receive low-grounding annotations because all seven MCP specifications lack example outputs.High grounding uses concrete specification examples, medium uses transferable examples from similar tools, and low uses parameter schemas alone. Annotations persist in harness artifacts for downstream filtering when richer specifications are used.Source: §A.4 · p. 11, Table 9 · p. 12

A.5 Workflow Composition and Category Diversity

• 9.71 calls on average (maximum 17) make Selenium workflows the longest; Illustrator averages 3.33 calls (maximum 12).Selenium’s length reflects sequential browser automation tasks; Redis averages 1.27 calls and Slack 1.39, tending toward atomic single-tool scenarios.Source: §A.5 · p. 12
• 108 unique categories span 337 scenarios (3.1 scenarios per category), with category count scaling with scenario count rather than tool count.Source: §A.5 · p. 12

A.6 Failure Details

• 7% of Git records score below 0.5, versus Filesystem’s 5% and Redis’s 1%; four MCPs have zero failures.Source: §A.6 · p. 12
• Git’s failures are driven by argument correctness for tools with complex parameter schemas.Source: §A.6 · p. 12
• 223 records exhibit value-accuracy failures, making value accuracy the dominant argument-failure sub-dimension.Source: §A.6 · p. 12
• 44 records exhibit relevancy failures, followed by format compliance (35), type compliance (31), completeness (16), and name accuracy (11).Source: §A.6 · p. 12

A.7 Per-MCP Dimension Scores

• 0.989 is Selenium’s ordering score, the highest among MCPs despite its longest workflows.Source: Table 11 · p. 12
• 0.780 is Git’s arguments score, the weakest among MCPs, consistent with its high parameter density.Source: Table 11 · p. 12

Replication setup

• 391 existing evaluation transcripts were re-scored with Qwen3.5-122B-A10B-FP8, changing only the evaluator while preserving the generation pipeline and rubric.Joining by (MCP, conversation_id) yielded 384 paired tool-calling scores and 380 paired coherence scores for comparison.Source: §A.8 · p. 12

Per-MCP agreement

• Across all seven MCPs, tool-calling confidence intervals overlap between judges, whereas coherence intervals are disjoint and Qwen 3.5 scores coherence systematically lower.Per-MCP means were evaluated with 95% percentile bootstrap confidence intervals using 2,000 resamples.Source: §A.8 · p. 12, Table 12 · p. 13

Ranking preservation

• ρ=0.86 for tool-calling and ρ=0.46 for coherence quantify MCP-level Spearman rank correlations between judges across seven MCPs.Redis remains the top tool-calling MCP, but the worst-coherence MCP changes from Git under Gemini to Illustrator under Qwen 3.5.Source: §A.8 · pp. 13–14

Failure-mode taxonomy replication

• 4–5× more failures occur in value_accuracy than the next-highest argument sub-dimension under both judges, with counts agreeing within 5%.Failures are records with sub-dimension scores below 1.0, replicating the central qualitative failure-mode finding.Source: Table 14 · p. 13, §A.8 · p. 13

Systematic shift

• Approximately zero mean tool-calling shift contrasts with a −0.16 mean coherence shift (Qwen 3.5 − Gemini), making absolute coherence levels judge-dependent.The 95% CIs are [−0.009, +0.008] for tool-calling and [−0.173, −0.138] for coherence; visual diagnostics show centered tool-calling differences but downward-shifted, more dispersed coherence differences. Without a third judge or human anchor, neither coherence judge can be designated correct.Source: §A.8 · p. 13, Figure 1 · p. 13

Schema complexity profile

• r=−0.60 for average parameters per tool and r=−0.66 for optional ratio associate schema complexity with lower per-MCP mean tool-calling scores.Tool count has a smaller positive correlation, r=+0.40. Tool-level analysis (n=222) confirms negative correlations with mean unsupervised tool-calling scores for average parameters (r=−0.29) and optional ratio (r=−0.30), both with p<0.001.Source: Table 15 · p. 14

C Novelty Summary

• The novelty summary organizes comparisons with prior literature around two gaps: data generation and metric decomposition.Table 17 assigns rows 1–3 to data generation and rows 4–6 to metric decomposition.Source: C Novelty Summary · p. 14, Table 17 · p. 15

B Aggregation Sensitivity

• 0.964 Spearman correlations between arithmetic aggregation and both alternatives show highly stable per-MCP rankings despite substantial shifts in corpus-wide means.Arithmetic, harmonic, and minimum aggregation yield means of 0.911, 0.851, and 0.781 over 385 records, excluding six coherence-only records; the top three remain identical, with only Filesystem and Git swapping. Schema-complexity correlations remain negative under every rule and localize most strongly to arguments (r = −0.86), while argument correctness remains the dominant failure mode: 60.5% of 448 turns score below 1.0.Source: Table 16 · p. 14, B Aggregation Sensitivity · p. 14

D Pipeline Prompts / D.1 Stage 1: Tool Interpretation / F Structured Output Schemas

• Tool interpretation records names, functionality, parameter requirements, use cases, and enterprise tags; Pydantic schemas enforce well-formed JSON throughout the pipeline.These fields structure the semantic explanation produced for each tool.Source: §D.1 · p. 14, F Structured Output Schemas · p. 16

D.2 Stage 2: Scenario Generation

• Scenario generation creates simple and complex workflows with schema-valid calls, logical progression, context management, and coverage prompts targeting every tool.Complex prompts emphasize creative, multistep tool combinations. Follow-up prompts request additional scenarios for tools left uncovered, allowing combinations with previously covered tools.Source: §D.2 · pp. 14–15

D.3 Stage 3: Mock Output Generation

• Mock outputs use concrete, realistic data and reference examples or output schemas, with expected responses covering the full workflow.Confidence is high when an example exists for that function, medium when only similar functions have examples, and low when no example exists for that function.Source: §D.3 · pp. 15–16

D.4 Stage 4: Multi-Turn Expansion

• Multi-turn expansion uses mock results to choose conversational breakpoints, batches related operations, and preserves every original tool call exactly once.Turns report concrete results and retain existing function names and parameter structures. Without mock outputs, standard mode removes data-driven principles and mock-output fields but preserves phase boundaries, batching, self-contained turns, and function-reuse constraints.Source: §D.4 · p. 16

E Evaluation Prompts / E.1 Tool-Calling Evaluation / E.2 Coherence Evaluation

• Reference-free LLM judges assess four tool-calling dimensions and five coherence dimensions, normalizing scores to 0–1 and combining them through arithmetic means.Tool-calling sub-dimensions use a 0–10 scale, while coherence dimensions use 1–3; usage takes its necessity score directly. Prompt-enforced cascading penalties propagate argument errors to dependent sub-dimensions, while single-call ordering is excluded and values from prior chained outputs are not penalized.Source: E Evaluation Prompts · p. 16, §E.1 · p. 16, §E.2 · p. 16

G Score Distributions and Per-Domain Analysis / G.1 Overall Score Distributions

• 31.7% of tool-calling scores are perfect, while 2.3% fall below 0.5 across the reported evaluation distribution.The distributions cover 391 evaluation records; coherence scores are left-skewed with a mode near 0.9.Source: §G.1 · pp. 16–17

• Each specification’s analysis distinguishes simple versus complex scenarios, workflow-length distributions, undirected tool co-occurrence, and directed transitions between tools in ordered workflows.Co-occurrence indicates tools appearing in the same scenario; sequential adjacency captures tool A → tool B transitions.Source: §G.2 · p. 17
• 3.3 calls is Illustrator’s mean workflow length, with a maximum of 12 calls across its 64-tool specification.Source: Figure 5 · p. 20
• 1.3, 1.4, 1.9, 2.0, and 2.2 calls are mean workflow lengths for Redis, Slack, Elasticsearch, Filesystem, and Git, respectively.These specifications contain 47, 16, 20, 14, and 33 tools, respectively.Source: Figure 4 · p. 19, Figure 6 · p. 21, Figure 7 · p. 22
• 9.7 calls is Selenium’s mean workflow length, with a maximum of 17 calls across its 56-tool specification.Source: Figure 10 · p. 25