PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis

Yongjian Tang, Costel Moraru, Doruk Tuncel, Hanya Elhashemy, Steffen Klepke, Thomas Runkler

arXiv:2610.13273v1cs.SEcs.AI

TL;DR

API specifications support integration and maintenance, but existing approaches provide limited support for their post-generation validation and market-driven evolution. Agent4Spec uses specialized agents for validation, market analysis, and revision. The strongest frontier models show strong results across these tasks, including 100% syntax-error detection and over 95% correct implementation of proposed corrections.

  • Problem

    Existing approaches mainly address initial specification generation or document-level maintenance, providing limited support for validating full OAS files, implementing repairs safely, and deciding whether APIs should evolve with market expectations.

    Source: §I · p. 1

  • Method

    Agent4Spec uses an orchestrator and specialized agents for OAS syntax validation, competitive market analysis, endpoint suggestions, and optional human-approved specification revision.

    Source: §I · p. 1

  • Results

    The strongest frontier models achieved 100% syntax-error detection, over 95% correct implementation of proposed corrections, and up to 1.82 human-approved endpoint suggestions out of three.

    Source: §I · p. 1

  • Takeaways & Limitations

    Full-specification revision is more difficult than error diagnosis, since successful file generation does not always imply a correct repair.

    Source: §VII · p. 8

    Compact models show practical limitations in tool use, full-file revision, and market-oriented analysis.

    Source: §I · p. 1

Abstract

from arXiv · show

API specifications serve as critical documentation for modern software development, directly impacting integration efficiency and client code generation. While recent advancements have leveraged Large Language Models (LLMs) to generate initial specifications from source code, existing approaches treat this process as a one-time event - overlooking the needs for continuous validation, long-term maintenance, and market-driven evolution of these artifacts. To address this issue, we present Agent4Spec, an autonomous multi-agent framework featuring two distinct pipelines: (1) a validation pipeline that detects and self-corrects OpenAPI syntax errors, and (2) a market analysis pipeline that conducts competitive research, identifies functional gaps against industry leaders, and automatically generates missing API-spec functionalities. We evaluate Agent4Spec on two curated datasets of API specifications spanning both general-purpose and industrial scenarios. Among 9 SOTA LLMs, the top-performer achieves 100% syntax error detection rates and over 95% implementation success for proposed corrections, while securing over 50% adoption rates for suggested endpoint improvements, representing a practical tool for API development.

I. INTRODUCTION

Agent4Spec addresses the post-generation work of validating and evolving machine-readable API specifications, which prior approaches largely leave limited or separate from initial generation. It combines specialized agents for validation, market analysis, and revision, and evaluates the framework across general-purpose and industrial APIs.

I. INTRODUCTION

  • Agent4Spec targets post-generation validation and enhancement, addressing limited support for validating full OAS files, safely implementing repairs, and deciding whether APIs should evolve with market expectations.Its orchestrator routes requests to validation or market analysis, with optional human review before market-driven changes are implemented.Source: §I · p. 1
  • The evaluation uses Guru-Spec and Industry-Spec, curated datasets covering general-purpose and industrial API scenarios, and examines nine LLMs across frontier and compact open-weight tiers.The tiers allow the study to examine effectiveness and resource trade-offs.Source: §I · p. 1
  • 100% syntax-error detection, over 95% correct implementation of proposed corrections, and up to 1.82 human-approved endpoint suggestions out of three were achieved by the strongest frontier models.The evaluation also exposes compact-model limitations in tool use, full-file revision, and market-oriented analysis.Source: §I · p. 1

A. API Specification Generation

  • Source-code and documentation-based systems mainly generate initial specifications, whereas Agent4Spec focuses on validating, repairing, and potentially enhancing existing OAS files.The cited generation systems include Respector, LRASGen, OOPS [9], SpeCrawler [6], and OASBuilder.Source: A. API Specification Generation · pp. 1–2

B. API Documentation Understanding and Maintenance

  • Prior API-documentation work addresses understanding, retrieval-augmented code generation, or documentation updates, while Agent4Spec works on machine-readable OAS artifacts and full-specification revision.The cited work includes Li et al., Chen et al. [3], and Lee et al..Source: B. API Documentation Understanding and… · p. 2

C. Agentic Workflows and Evaluation

  • Agentic software-development systems use task decomposition, tools, and iterative feedback, but Agent4Spec applies this approach to OAS maintenance with developer control before endpoint changes.The compared systems mainly address source code, repositories, or natural-language development tasks.Source: C. Agentic Workflows and Evaluation · p. 2

III. AGENT4SPEC

Agent4Spec routes requests through either a syntax-validation pipeline or a sequential market-analysis pipeline, then can revise the specification. Its agents use tools and market evidence to produce error repairs or prioritized endpoint suggestions.

III. AGENT4SPEC

  • The orchestrator directs each request to either validation or market analysis before revision, with optional developer review of suggestions.Only one pathway executes before the revision agent is called.Source: §III · p. 2

A. Validation Pipeline

  • The validation agent reads the API YAML file, runs OpenAPI validation to detect syntax errors, and proposes fixes from the resulting error message; a revision agent can implement them.The validation agent uses file-reading and OpenAPI-specification validation tools.Source: A. Validation Pipeline · pp. 2–3

B. Market Analysis Pipeline

  • Four sequential agents identify the API domain, research market trends and competitors, assess prioritized functional gaps, and suggest concrete endpoints with schemas and migration guidance.The market summary includes insights on market leaders, emerging trends, and pricing models where available.Source: B. Market Analysis Pipeline · p. 3

C. Specification Revision Agent

Agent4Spec’s revision agent applies proposed corrections or endpoint suggestions while preserving the specification’s structure, with iterative re-validation for large files. The evaluation spans two API datasets, nine models, and quantitative and human-judged market-analysis metrics.

C. Specification Revision Agent

  • Revised specifications are repeatedly sent through validation for up to n rounds, until all errors are eliminated.The revision agent applies validation reports or market-oriented endpoint suggestions while preserving the original content structure.Source: C. Specification Revision Agent · p. 3

D. Human-in-the-Loop

  • Human developers can review and approve pipeline suggestions before the revision agent applies them.The finalized report after human approval is represented as R′ = H(R).Source: D. Human-in-the-Loop · p. 3

A. Datasets

  • Guru-Spec contains 300 public API specifications, while Industry-Spec contains 200 industrial APIs covering enterprise and industrial domains.The datasets span broad web services and industrial scenarios.Source: A. Datasets · pp. 3–4

B. Models

  • The evaluation covers nine models: five proprietary frontier models and four open-source alternatives, with compact models deployed through Ollama.All models use the minimum temperature setting.Source: B. Models · p. 4

C. Metrics

  • Syntax validation uses quantitative detection, proposal, correctness, implementation, and save-rate metrics, while open-ended market analysis is assessed by LLM judges and developers on a 1–5 scale.Three developers independently assess market-analysis summaries, and their scores are averaged.Source: C. Metrics · p. 4

V. EXPERIMENTS – SYNTAX VALIDATION PIPELINE

The validation-pipeline evaluation manually introduces syntax errors into collected API specifications, covering several error types that test structural and specification-level checking.

V. EXPERIMENTS – SYNTAX VALIDATION PIPELINE

  • Each collected API specification was manually seeded with syntax errors, including invalid data types, missing parameters, and inconsistent naming.The examples include indentation, missing required response-description, and type-mismatch errors.Source: §V · pp. 4–5
  • An indentation mismatch can alter YAML hierarchy and cause structural validation failures.The example identifies an indentation error in an OpenAPI YAML schema.Source: §V · p. 5
  • The test cases also include a missing mandatory response-description field and conflicts between property definitions and their values.Source: §V · p. 5

A. Syntax Error Detection

Agent4Spec is evaluated on syntax-error detection and full-file revision, then on market-analysis stages from domain identification through functional-gap analysis. Frontier models lead overall, while compact models show limitations in tool use and market-oriented analysis.

A. Syntax Error Detection

  • 100.0% syntax-error detection on both datasets was reached by Claude-Sonnet-4.6 and Gemini-3.1-Pro, and Agent4Spec outperformed the single-prompt baseline across three validation metrics.The system proposed repairs for every detected error; frontier solution correctness was 91.5–99.5%, while LLaMA-3.1-8B and Gemma-4-4B had detection and correct-solution rates no higher than 8.0%.Source: A. Syntax Error Detection · p. 5, Table IV · p. 6

B. Syntax Error Revision

  • Frontier models saved 100.0% of revised specifications on both datasets, but correct implementation was lower and compact models struggled with full-file revision.Qwen-3.5-9B achieved 10.3% correct implementation on Guru-Spec and 5.5% on Industry-Spec; iterative re-validation is needed to verify revised specifications.Source: B. Syntax Error Revision · p. 5, Table V · p. 6

VI. EXPERIMENTS – MARKET ANALYSIS PIPELINE

  • Market analysis is evaluated with LLM-as-a-judge and human ratings because the task has no standard answers.Source: §VI · p. 5

A. Domain Identification

  • Human domain-identification scores ranged from 2.76 to 3.82, with Claude-Sonnet-4.6 highest at 3.82; both evaluation types favored larger, more recent models.LLM-judge ratings were more conservative and less aligned with the human ranking.Source: A. Domain Identification · p. 5, Figure 2 · p. 6

B. Online Search

  • All three LLM judges ranked GPT-5.2 highest for online search, while human evaluators rated Claude-Sonnet-4.6 highest at 3.34.Compact models tended to underuse search and rely on prior knowledge, leaving reports without timely information.Source: B. Online Search · p. 6, Figure 3 · p. 6

C. Gap Analysis

  • Human evaluators rated Claude-Sonnet-4.6 highest in gap analysis at 4.74, while compact models were weaker overall and LLaMA-3.1-8B received 1.82 from humans.Evaluator rankings varied, and the passage connects compact models’ search-tool underuse and reliance on prior knowledge with a weaker factual basis for gap analysis.Source: C. Gap Analysis · pp. 6–7, Figure 4 · p. 7

D. Endpoint Suggestion

Endpoint suggestions distinguish model performance, but human approval remains below two of three suggestions on average. Market analysis also consumes more tokens and time than syntax validation, while the conclusion emphasizes model capability, verification, and human oversight.

D. Endpoint Suggestion

  • 4.56 was GPT-4.1's human evaluator score for endpoint suggestions, ahead of Claude-Sonnet-4.6 (4.44) and Gemini-3.1-Pro (3.94).Figure 5 presents the endpoint-suggestion evaluation.Source: Figure 5 · p. 7, D. Endpoint Suggestion · p. 7
  • Qwen-3.5-9B had a human score of 1.82 despite its competitive online-search score; qualitative inspection found GPT-5.2 and Qwen-3.5-9B often prioritized market-insights reports over endpoint improvements, leading to biased, lower-scoring results.Source: D. Endpoint Suggestion · p. 7

E. Human-in-the-Loop

  • 1.82 out of 3 was GPT-5.2's highest human-approved endpoint-suggestion count, and even the best model averaged fewer than two approvals.Figure 6 reports average approved suggestions out of three; Claude-Sonnet-4.6, Gemini-3.1-Pro, and GPT-4.1 received 1.79, 1.73, and 1.71, respectively.Source: Figure 6 · p. 7, E. Human-in-the-Loop · p. 7

F. Resource Consumption

  • 162–1,402 output tokens and 1.8–37.1 seconds were required for validation, compared with 720–5,467 tokens and 16.1–182.2 seconds for online search.Figure 7 reports token usage and inference latency per specification; local compact models were especially latency-bound during online search.Source: F. Resource Consumption · pp. 7–8, Figure 7 · p. 8

VII. CONCLUSIONS AND FUTURE WORK

  • The conclusion reports complete syntax-error detection by Claude-Sonnet-4.6 and Gemini-3.1-Pro on both datasets, while noting that successful file generation does not guarantee a correct repair.It also concludes that reliable deployment depends on model capability, tool use, verification, and human oversight.Source: §VII · p. 8
  • The authors call for experiments on more emerging models to generalize observed accuracy, token-cost, and latency trade-offs.They also identify specialized prompting, supervision, and fine-tuning as possible directions for improving tool-use reliability and local deployment practicality.Source: §VII · p. 8
  • Because no public OpenAPI repair benchmark is available, evaluation uses manually injected common errors rather than paired faulty and corrected real-world specifications.The authors propose building a benchmark from real-world version histories and issue reports.Source: §VII · p. 8

APPENDIX A PROMPTS FOR AGENTS

The appendix prompts define an orchestrator that lets developers choose validation, market comparison, and revision paths, with specialized agents handling syntax checks and repairs. They also specify how outputs, constraints, and missing inputs are managed.

A. Orchestrator Agent

  • The orchestrator coordinates specialized subagents and offers developers a menu to run validation, market comparison, revision, or sequences of these tasks.The prompt says to respect the user's choices rather than require the full sequence.Source: A. Orchestrator Agent · p. 10
  • The prompts instruct the orchestrator to ask whether developers want to assess an API specification and to offer next steps after returning a revised file.They also call for concise, actionable summaries after agent calls.Source: A. Orchestrator Agent · pp. 10–12
  • The orchestrator summarizes validation findings and asks whether the user wants market comparison or specification revision next.The prompt specifies a concise summary of error counts, critical issues, and suggested fixes.Source: A. Orchestrator Agent · p. 11
  • For revision, the orchestrator passes the original specification and selected validation or market outputs to one specification-generation agent, which returns a revised specification.The prompt permits revision using either input source or both, and recommends validation fixes first when conflicts arise.Source: A. Orchestrator Agent · pp. 10–11
  • The orchestrator can reuse validation findings, market suggestions, and constraints across steps, while allowing users to skip, rerun, or reorder tasks.It is instructed to handle unreadable paths, agent errors, and missing prerequisites by requesting inputs or offering next steps.Source: A. Orchestrator Agent · p. 11

B. Validation Agent

  • The syntax-validation agent reads the specification, uses an OpenAPI validation tool to detect errors, proposes fixes, and can generate a corrected specification.Its report includes file type, detected OAS version, overall status, issue locations, reasons, and fixes.Source: B. Validation Agent · p. 12

C. Domain Identification Agent

The prompts define a sequential market-analysis workflow, from identifying an API domain through research and gap analysis to endpoint design. A unified revision agent applies validation corrections to the specification.

C. Domain Identification Agent

  • The domain-identification agent reads an API specification, identifies its domain, and provides three concise bullets on developments for future market research.Its output includes the identified domain and domain information suitable for subsequent research.Source: C. Domain Identification Agent · p. 12

D. Online Search Agent

  • The online-search agent uses two queries and fetched search results to produce a 150–250-word market summary, without assuming knowledge beyond those results.The requested insights cover leaders, competitors, trends, standards, pricing, developer experience, and adoption; quantified statistics are included when available.Source: D. Online Search Agent · p. 13

E. Gap Analysis Agent

  • The gap-analysis agent compares market leaders with the specification, maps paths and design patterns, and prioritizes gaps with rationale, dependencies, and constraints.It covers feature coverage, performance, security, pricing, developer experience, and ecosystem, and is instructed not to propose endpoints.Source: E. Gap Analysis Agent · pp. 13–14

F. Endpoint Suggestion Agent

  • The endpoint-design agent turns high- and medium-priority gaps into concrete, pattern-aligned endpoint changes and mini-specifications, preferring additive, backward-compatible changes.Its design protocol includes request and response details, errors, performance, security, and async or webhook patterns when needed.Source: F. Endpoint Suggestion Agent · pp. 14–15

G. Revision Agent

  • The revision agent applies only corrections explicitly marked in the validation report, then saves a complete corrected specification in the original folder under a new filename.The prompt directs it to load the original specification and validation report before saving the revised file.Source: G. Revision Agent · p. 15

APPENDIX B PROMPT FOR LLM-AS-A-JUDGE

The appendix instructs LLM judges to rate market-analysis outputs on a five-point scale, balance alignment with the specification against intrinsic quality, and provide explicit reasons. It gives stage-specific criteria for domain identification, market research, and gap analysis.

APPENDIX B PROMPT FOR LLM-AS-A-JUDGE

  • The judge assigns a 1–5 rating with an explicit reason for outputs across four sequential market-analysis stages.The prompt describes ratings as balancing evaluation criteria rather than requiring a score alone.Source: Appendix B · p. 17

A. Evaluation on Domain Identification

B. Evaluation on Market Research

  • Market-summary ratings balance alignment with the specification against accuracy, sourcing, coverage, structure, clarity, relevance, and currency.The prompt asks the judge to assess competitors, pricing, adoption, and trends among the potential coverage areas.Source: B. Evaluation on Market Research · pp. 17–18

C. Evaluation on Gap Analysis

  • Gap-analysis ratings balance alignment with the specification and market analysis against coverage, mapping accuracy, prioritization, constraints, and actionability.The rubric evaluates whether the output covers required categories and follows the task's structural and constraint requirements.Source: C. Evaluation on Gap Analysis · p. 18
  • An excellent gap analysis precisely maps the specification, grounds category-by-category market gaps, and justifies priorities with dependencies and constraints.The rubric also requires concise bullets, labeled assumptions, and design notes aligned with naming, versioning, async or webhook needs, and compliance.Source: C. Evaluation on Gap Analysis · p. 19

D. Evaluation on Endpoint Suggestions

The endpoint-suggestion evaluator balances alignment with the source specification and market inputs against the plan’s intrinsic quality. The appendix case study then walks through a user-initiated validation workflow, from detecting a misspelled OpenAPI field to proposing its correction.

D. Evaluation on Endpoint Suggestions

  • The evaluator gives a 1–5 rating that balances endpoint-plan alignment with market or gap inputs and intrinsic qualities such as testability, security, and backward compatibility.Its rubric ranges from major misalignment and missing mini-specifications to strong coverage of prioritized gaps and clear, actionable specifications.Source: D. Evaluation on Endpoint Suggestions · pp. 19–20

APPENDIX C CASE STUDY

  • The case study records a step-by-step Claude-Sonnet-4.6 execution, including agent actions, tool calls, and runtime responses.A video of the process with human interaction is provided in supplementary materials.Source: Appendix C · p. 21

OrchestratorAgent

Human User (Event 3)

  • The syntax check detects that “descriptio” in the info block does not match the expected “description” key and presents a corrected YAML snippet.The detected specification version is OpenAPI 3.0.1.Source: Human User (Event 3) · p. 21, Summary · p. 21

Summary

  • The summary reports an invalid specification and describes the typo correction, then asks whether to forward it for generation of a complete validated specification.Source: Summary · p. 21

Human User (Event 4)

The case study shows the system saving a corrected specification, revalidating it, and then conducting a market comparison. The comparison identifies prioritized gaps in the NORM service and proposes competitive improvements.

Human User (Event 4)

Human User (Event 6)

System / Tool Usage

  • The market-research stage fetches competitor and standards pages, including Siemens, ETSI, EAN Search, and GS1 resources.Source: System / Tool Usage · p. 23

GapIdentifierAgent

  • The existing service has one validation operation, while market leaders offer lookup, search, similar-product search, and bulk export; emerging identifiers are also identified as a medium-priority coverage gap.Source: GapIdentifierAgent · p. 23
  • The comparison flags limited security controls and developer support, including no SDKs, health endpoint, or 4xx/5xx schemas, and notes single-region deployment and no webhook integration.Source: GapIdentifierAgent · p. 23
  • The gap report also identifies a potential code-generation risk from an implicit oneOf discriminator and a misspelled description field that may affect documentation generators.Source: GapIdentifierAgent · pp. 23–24
  • The gap analysis prioritizes batch or asynchronous processing, error schemas, and developer experience as high-priority areas, with emerging identifiers, security granularity, and failover at medium priority.Source: GapIdentifierAgent · p. 24

EndpointPlannerAgent

The NORM endpoint plan proposes additive routes and API changes to address prioritized gaps. It specifies asynchronous bulk validation and job polling, alongside modifications to the existing validation endpoint and options for continuing the session.

Endpoint Plan — NORM Service v3.5.0 Suggested Endpoints

  • High-priority additions include asynchronous batch validation, job-status polling, a health check, and a supported-systems listing; the plan also adds error schemas to the existing validation endpoint.Source: Endpoint Plan — NORM Service v3.5.0… · p. 24

Endpoint Mini-Specs 1. POST /norm/v3/validate/batch — Async Bulk Validation

2. GET /norm/v3/jobs/{jobId} — Poll Async Job Status

  • The job-status endpoint uses a job ID with offset and limit parameters and returns status, progress, and paginated results, with ETag caching.Source: §2 · p. 24

3. Modifications to Existing POST /norm/v3/validate

  • Changes to the existing validation endpoint include strict error-response schemas, an explicit discriminator, a typo fix, and removal of an empty schema object.Source: §3 · p. 24
  • The plan assumes Siemens Entra ID can use custom OAuth2 scopes and that seven-day job-result retention fits internal transient-data limits.Source: §3 · p. 24

What would you like to do next?

Related papers

Loading 2610.13273v1…