PaperlayerView exact revision on arXiv

Source-linked AI summary

Specification Portability Across LLM Development Agents: Cross-Agent Compatibility in Specification-Driven Software Migration

Oleg Grynets, Oleksii Ilchuk, Dariia Zatulna, Vasyl Lyashkevych

arXiv:2608.21208v1cs.SEcs.AIcs.LO

TL;DR

The paper asks whether specifications used in LLM-based specification-driven development remain effective when transferred between heterogeneous agents. It evaluates this question through Oracle-to-PostgreSQL migration experiments using multiple agents, metrics, and ingestion strategies. Results show agent-dependent portability effects, with rewriting improving Gemini in the tested configuration and retrieval appearing on both Gemini and Copilot Pareto frontiers.

  • Problem

    Evidence is limited on whether a complete specification preserves implementation effectiveness when transferred from its originating LLM development agent to another heterogeneous agent.

  • Method

    The study combines a specification-first Oracle-to-PostgreSQL migration pipeline with cross-agent experiments comparing native and foreign specifications, adaptation strategies, and multiple quality metrics.

  • Results

    Cross-agent transfer produced strongly agent-dependent results: Kiro-origin specifications degraded Gemini when consumed directly, rewriting improved Gemini in the tested configuration, and retrieval was the only common strategy on Gemini’s and Copilot’s reported Pareto frontiers.

  • Takeaways & Limitations

    Specifications in heterogeneous SDD workflows should not automatically be treated as agent-neutral artifacts; portability, agent-specific interpretation, and retrieval-based access require explicit consideration.

Abstract

from arXiv · show

This paper investigates cross-agent specification portability using Oracle-to-PostgreSQL migration as a controlled software transformation task. The study combines two experimental stages. First, a specification-first migration pipeline was evaluated on 1,006 PL/SQL files, of which 623 were successfully regenerated and 380 generated scripts executed successfully in PostgreSQL 16. Second, cross-agent experiments were conducted on a dataset of 1,802 Oracle scripts with corresponding PostgreSQL implementations using Amazon Kiro, Google Gemini, and GitHub Copilot, with Claude Code and Cursor included in the initial single-agent evaluation. Native and foreign specifications were assessed using Token F1, exact match, SQL syntax validity, AST exact match, AST mean similarity, and immediate runnability. The results show that specification size alone does not predict implementation quality and that cross-agent transfer can produce substantial agent-dependent degradation. The strongest replicated case occurred when Gemini directly consumed a Kiro-origin specification, producing a Token F1 of 0.035, SQL syntax validity of 2.33%, and AST mean similarity of 0.015. Rewriting substantially improved Gemini in the tested configuration, compression did not provide a universal benefit, and retrieval-augmented ingestion was the only common strategy represented on the per-agent Pareto frontiers of both Gemini and Copilot. The findings suggest that specifications in heterogeneous SDD workflows should not automatically be treated as agent-neutral artifacts and motivate explicit consideration of specification portability, agent-specific interpretation, and retrieval-based access in multi-agent software engineering.

I. INTRODUCTION

LLM-based specification-driven development introduces intermediate specifications into increasingly agentic and multi-agent software workflows. This paper studies whether such specifications remain effective across heterogeneous agents, using Oracle-to-PostgreSQL migration and adaptation experiments.

  • Multi-agent development makes intermediate-artifact interoperability consequential because artifacts created by one role can become inputs to another.
  • The research asks whether specification origin, size, representation, and agent-specific interpretation affect implementation quality and whether adaptation strategies reduce incompatibility.
  • Oracle-to-PostgreSQL migration is used as a controlled software-transformation task for studying cross-agent specification transfer.The study combines a specification-first migration pipeline with cross-agent experiments.
  • 1,006 PL/SQL files were evaluated in the first study, with 623 successfully regenerated and 380 regenerated files executing successfully in PostgreSQL 16.
  • The cross-agent study compares native and foreign specifications across lexical, syntactic, structural, and operational metrics.Measures include Token F1, exact match, SQL syntax validity, AST exact match, AST mean similarity, and immediate runnability.
  • The study’s contribution is empirical: it evaluates portability only in the reported agents, specifications, metrics, and experimental configurations.

A. Specification-Driven and Specification-Based Software

Specification-driven development treats specifications as explicit artifacts that constrain or guide implementation, while specification-based migration uses an intermediate textual representation between source and target code. The reviewed literature motivates retrieval and multi-agent workflows but leaves cross-agent specification effectiveness insufficiently measured.

  • Specification-driven development integrates explicit specifications with implementation and verification, while contemporary LLM agents can consume them directly during code generation.
  • Specification-Based Code–Text–Code transforms source code into a neutral textual description before generating the target implementation.
  • The intermediate specification can capture behavior, identifiers, control conditions, side effects, data dependencies, and domain-specific intent.
  • Oracle-to-PostgreSQL migration is suited to this approach because the systems share SQL foundations but differ in dialect-specific and procedural constructs.
  • Repository-level research shows that retrieval can improve access to distributed context, but irrelevant retrieved information can harm robustness and efficiency.
  • Existing multi-agent studies emphasize roles, communication, orchestration, and collective quality rather than whether specifications remain equally effective across implementation agents.

E. Requirements Ambiguity, Intent, and Intermediate

Requirements clarification and intermediate representations can reduce ambiguity and preserve transformation-relevant information, but equivalent functionality may still be organized differently for different agents. The paper therefore examines representation adaptation, context reduction, and retrieval without treating specification size as sufficient evidence of quality.

  • Requirements-oriented methods use tests or clarification questions to distinguish alternative behaviors arising from ambiguous intent.
  • Specifications may describe the same target functionality while organizing constraints, dependencies, examples, and implementation guidance differently.
  • 1,597 lines for Kiro versus 193 lines for Gemini did not establish that the larger specification produced universally stronger implementation results.
  • Rewriting a Kiro-origin specification substantially changed Gemini’s measured outcome, whereas the same strategy did not provide a universal improvement for Copilot.
  • Aggressive context reduction can remove semantically important information, making token optimization a quality-cost trade-off rather than simple shortening.
  • The study evaluates whether SDD specifications can be considered agent-neutral, restricting conclusions to the observed experimental setting and tested ingestion strategies.

IV. CROSS-AGENT SPECIFICATION PORTABILITY

The paper defines cross-agent portability by comparing quality when a specification produced under one agent environment is consumed by another. It treats compatibility as metric-specific and directional rather than as a single symmetric property.

  • Specification portability is the capability of a specification produced under one development-agent environment to preserve implementation effectiveness when consumed by another.
  • The evaluated metrics include Token F1, SQL Syntax Validity, AST Mean Similarity, and Immediate Runnability.
  • The compatibility framework distinguishes approximate portability, compatibility loss, and cases where a foreign representation benefits the target agent.
  • Cross-agent compatibility is evaluated separately for each quality metric because implementation quality is multidimensional.
  • Cross-agent compatibility should be regarded as directional rather than symmetric.

V. METHODOLOGY

The migration pipeline preprocesses and decomposes source SQL, then uses an agent loop to generate specifications and derive PostgreSQL implementations with iterative tool-mediated refinement.

  • Pipeline structure: Source SQL is preprocessed and decomposed into database objects before specification generation and PostgreSQL implementation.Object-level decomposition separates deterministic preparation from LLM-mediated generation.
  • Pipeline structure: Object-level decomposition reduces irrelevant input and allows incorrect target objects to be regenerated independently.A failed conversion therefore requires regeneration of only the affected object rather than the complete source file.
  • Agent loop: The agent loop coordinates the application layer, controller, LLM, and MCP-based tools during specification and code generation.The sequence controls iterative tool invocation and termination.
  • Agent loop: The loop repeatedly supplies messages and tool schemas, executes requested tools, and returns results to the agent context for refinement.Execution ends on an explicit finishing operation, no further tool calls, or the iteration limit.
  • Agent loop: Average consumption was approximately 2,000 tokens per iteration across approximately three iterations, ranging from approximately 900 to 3,000 tokens by object size.The reported range distinguishes smaller and larger migration objects.

B. Stage 2: Cross-Agent Experiment

Stage 2 evaluates how specifications transfer between agents by comparing native and foreign configurations on a shared Oracle-to-PostgreSQL migration task.

  • Experimental design: The second stage changes the experimental unit from migration objects to the relationship between specification origin and implementation agent.This isolates cross-agent transfer as the comparison of interest.
  • Experimental design: Native workflows kept specification generation, review, code generation, and testing within the same agent environment.The same environment produced and subsequently consumed the specification.
  • Experimental design: Cross-agent experiments retained the specification artifact while changing the implementation agent.Native and foreign specifications were compared after rerunning native configurations for Kiro, Gemini, and Copilot.
  • Experimental scope: The systematic cross-agent stage concentrated primarily on Kiro, Gemini, and Copilot because of pricing constraints.Five environments were initially investigated, but the systematic stage was narrower.
  • Task and dataset: Agents received the same functional request for a deterministic Oracle-to-PostgreSQL migration utility.The requested system covered datatype conversion, unsupported-construct preservation, reports, optional database access, structural parsing, and testing.
  • Task and dataset: The evaluation dataset contained 1,802 Oracle scripts with corresponding PostgreSQL scripts, varying in length and complexity.This dataset supplied paired source and target implementations for the cross-agent evaluation.

VI. EVALUATION METRICS

The evaluation distinguishes lexical, syntactic, structural, and operational aspects of generated migration quality, while the initial pipeline reports regeneration and execution outcomes.

  • Metrics: Token F1 measures token-level similarity, while Exact Match measures the percentage of outputs identical to expected target scripts.These metrics capture lexical agreement at different strictness levels.
  • Metrics: SQL Syntax Validity measures the percentage of outputs recognized as syntactically valid SQL using sqlglot.It evaluates syntactic validity rather than exact correspondence to the target.
  • Metrics: Immediate Runnability measures scripts executable without additional manual modifications in the experimental database environment.The original study interprets this value as a lower bound because testing used an empty live database with generated stubs.
  • Metrics: AST Exact Match measures exact parsed-tree agreement, whereas AST Mean Similarity measures average structural similarity between generated and expected ASTs.Together they assess structural correspondence beyond lexical overlap.
  • Metrics: The metric set intentionally distinguishes lexical, syntactic, structural, and operational quality.The reported measures are complementary rather than a single aggregate score.
  • Initial pipeline outcomes: Approximately 85% of tables achieved regeneration success, while queries performed worst and schema-dependent constructs remained substantially more difficult.The initial pipeline established feasibility but showed variation across database-object categories.

B. Specification Size Does Not Explain Quality

Specification size alone does not explain implementation quality, while cross-agent transfer produces asymmetric, agent-dependent outcomes that can be reproduced in the Gemini–Kiro case.

  • Specification size: Kiro produced approximately 1,597 specification lines and Gemini approximately 193, yet the larger specification did not lead to the strongest implementation across evaluated metrics.The result provides evidence against a simplistic proportional relation between specification size and quality.
  • Specification size: Specification quality is interpreted through organization, determinacy, agent interpretability, and testing rather than size alone.These factors are presented as a more appropriate interpretation of specification effectiveness.
  • Cross-agent behavior: Native and foreign specifications separate whether a specification was generated within the target agent environment or another agent environment.The complete evaluated combinations are summarized in the cross-agent transfer table.
  • Cross-agent behavior: Cross-agent transfer was asymmetric and agent-dependent: Gemini degraded most with a Kiro-origin specification, while other combinations retained or improved individual metrics.The comparison concerns the measured native and foreign configurations.
  • Replicated Gemini–Kiro result: Token F1 was 0.035, SQL Syntax Validity was 2.33%, and AST Mean Similarity was 0.015 when Gemini directly consumed a Kiro-origin specification in the replicated run.The repeated experiment confirmed the same qualitative degradation pattern after the first run was treated as anomalous.
  • Cross-agent behavior: Copilot behaved differently, remaining relatively stable in some conditions and benefiting from a foreign specification in others.The results therefore do not support a universal transfer effect across agents.

D. Specification Adaptation Strategies

The study evaluates rewriting, compression, and retrieval-based ingestion as ways to adapt foreign specifications, finding agent-dependent effects and a shared Pareto advantage for retrieval-based access.

  • Rewriting: Rewriting the foreign specification substantially improved Gemini’s results relative to direct Kiro-specification ingestion.The knowledge source remained Kiro’s specification while its representation was transformed before implementation.
  • Compression: Compression reduced specification length, but the experiment found no universal improvement in specification incompatibility.This supports the observation that specification length is not an adequate proxy for operational quality.
  • Retrieval-Augmented Ingestion: Retrieval-based ingestion gave agents selective access to specification content rather than direct access to the complete specification.The strategy used retrieval tools for selective specification access.
  • Retrieval-Augmented Ingestion: RAG was the only evaluated ingestion strategy represented on both Gemini’s and Copilot’s per-agent Pareto frontiers.It did not outperform every alternative on every metric, but provided the most consistent tested cross-agent trade-off.
  • Pareto Analysis: Gemini degraded substantially under direct foreign-specification ingestion, whereas Copilot remained comparatively stable in several tested conditions.Consequently, the global Pareto frontier was dominated by Copilot configurations.
  • Pareto Analysis: Per-agent Pareto analysis showed that the optimal ingestion strategy differs between Gemini and Copilot.Separate frontiers were evaluated because a global frontier can be dominated by one agent’s generally stronger configurations.

E. Proposed Specification Portability Model

The proposed model treats specifications as structured artifacts whose effectiveness depends on more than their information content. It distinguishes knowledge, representation, dependency structure, and granularity, then models agent-specific transformation and implementation.

  • Specification Model: The portability model represents a specification as S = ⟨K, R, D, G⟩: knowledge, representation, dependency structure, and granularity.These four properties are proposed as distinct dimensions of specification quality.
  • Agent-Specific Transformation: The effective specification received by an implementation agent is produced by transforming the original specification using the specification and agent context.The transformation is represented as S′ = T(S, I_s, A_i).
  • Operational Effectiveness: Information content and operational effectiveness are not identical in specification-driven development.Specifications expressing approximately the same functional intention may trigger substantially different downstream behavior across heterogeneous agents.
  • Operational Effectiveness: Multi-agent workflows can preserve, reformulate, omit, duplicate, ambiguously interpret, or reprioritize operational knowledge at each transition.The model distinguishes knowledge preservation from operational interpretation preservation.
  • Operational Effectiveness: Specification portability should be evaluated through resulting implementation quality as well as information preservation.The same specification may be interpreted differently depending on the implementation agent and ingestion strategy.

VIII. DISCUSSION

The discussion finds that specification size does not predict implementation quality and that cross-agent effectiveness depends on representation, origin, agent, and ingestion strategy. It therefore supports agent-specific adaptation and structured or retrieval-based specification views rather than assuming agent neutrality.

  • Empirical Findings: Specification size alone did not explain implementation quality, while direct transfer produced strongly agent-dependent results.Gemini’s degradation with a Kiro-origin specification was reproduced in a second run.
  • Empirical Findings: Rewriting substantially improved Gemini in the tested configuration, whereas compression did not provide a universal benefit.The effects of adaptation strategies differed across agents.
  • Interpretation: The findings do not support treating foreign specifications as inherently worse; effectiveness depends on representation, origin, and implementation agent.The paper frames cross-agent degradation as an interaction effect rather than a universal property of foreign specifications.
  • Implications: A specification successful with one agent should not automatically be treated as equally suitable for another agent.Agent substitution may require specification adaptation or an alternative ingestion strategy.
  • Implications: The paper proposes preserving a common specification while providing agents with adapted or selectively retrieved views.This approach separates semantic content from its agent-specific presentation.
  • Implications: Retrieval-based access appears to be the most robust common option when agent compatibility is unknown, but universal optimality is not established.This conclusion is limited to the evaluated strategies and agents.
  • Initial Migration Study: The initial migration pipeline regenerated 623 of 1,006 PL/SQL files, with 380 regenerated scripts executing successfully in PostgreSQL 16.This established the practical feasibility of the specification-first migration approach before the cross-agent experiments.
  • Empirical Findings: RAG was the only common ingestion strategy on the per-agent Pareto frontiers for Gemini and Copilot.It did not outperform all alternatives on every metric.
Loading 2608.21208v1…