PaperlayerView exact revision on arXiv

Source-linked AI summary

Architecture as Capability Equalizer for Coding Agents

Arquimedes Canedo

arXiv:2608.21747v1cs.SEcs.AIcs.CL

TL;DR

As coding agents generate entire modules from high-level descriptions, how developers should communicate architectural intent remains open. A controlled comparison finds that specification format matters more for non-frontier than frontier models.

  • Problem

    As agents generate entire modules from high-level descriptions, how developers should communicate architectural intent remains an open question.

    Source: §I · p. 1

  • Method

    A controlled comparison tested five architecture-specification formats across six models in 90 multi-turn agent trials.

    Source: §X · p. 13

  • Results

    Specification format showed a strong format × model interaction: quality spreads were 0.17–0.92 for frontier models and 0.83–2.42 points for non-frontier models.

    Source: §X · p. 13

  • Takeaways & Limitations

    Structured architecture specifications function as a capability equalizer, with value inversely proportional to model strength.

    Source: §X · p. 13

    With n = 3 per cell, individual cell comparisons are underpowered, so the format × model interaction is exploratory rather than statistically tested.

    Source: J. Limitations · p. 12

Abstract

from arXiv · show

LLM-based coding agents generate complete software systems from high-level descriptions, yet little is known about how the format of architecture specifications affects the quality of generated code or whether this effect depends on model capability. We present a controlled experiment comparing five informationally equivalent specification formats (informal prose, Mermaid diagrams with constraints and ADRs, OpenAPI, C4/Structurizr DSL, and TypeScript interface contracts with ArchUnit-style rules) across six models from three vendor families (Anthropic Claude, OpenAI GPT, Google Gemini). Across 90 multi-turn agent trials, specification format shows a strong format x model interaction. On the strongest models (Sonnet 4.6, GPT-5), format barely matters (quality spread 0.17-0.92). On weaker models, format produces spreads of 0.83-2.42 points, with code-proximate formats (OpenAPI, TypeScript contracts) recovering most of the capability gap. Mid-tier models can consume more tokens than frontier models for worse output when they enter compilation debugging loops that stronger models avoid. Self-validation rates collapse from 100% (Sonnet) to 0% (Gemini Flash) across the capability spectrum. TypeScript contracts triple API route coverage for the weakest model (33% to 100%). Structured architecture specifications serve as a capability equalizer, with value inversely proportional to model strength and the largest returns for cost-optimized deployments.

I. INTRODUCTION … B. Structured Prompting

Architecture specification format matters most for weaker coding agents: code-proximate formats can recover capability gaps, while format has little effect on frontier models. The paper situates this comparison within prior work on structured prompting and specification-driven code generation.

I. INTRODUCTION

  • The evaluation combines static constraint checking with LLM judging, and the paper releases 93 trial transcripts and experiment artifacts.The study assesses architectural adherence, constraint compliance, and completeness; automated checking covers about 80% of architecture, while judge constraint scores correlate r = 0.21 with automated compliance.Source: §I · pp. 1–2
  • On frontier models, format spreads are 0.17–0.92 quality points, versus 0.83–2.42 on non-frontier models, where code-proximate formats recover over half the capability gap.The study compares five specification formats across three quality dimensions.Source: §I · p. 1
  • Haiku uses 735K tokens for score 6.50, while Sonnet uses 640K tokens for score 8.42, revealing an inverted cost-efficiency valley.Mid-tier models may consume more tokens than frontier models while producing worse output.Source: §I · p. 1
  • Demo run rate falls from 100% for frontier models to 0% for the smallest model; three failure modes interact differently with specification format.The failure modes are compilation death spiral, premature termination, and perfectionist iteration.Source: §I · p. 1
  • Gemini Flash route coverage rises from 33% under prose to 100% under TypeScript interface contracts, a 3× improvement from format alone.Coverage was measured by automated route analysis.Source: §I · p. 1

A. Specification-Driven LLM Code Generation

  • Prior work finds OpenAPI gains in single-turn functional generation [1], performance loss from denser constraints [2], and mixed effects from executable specifications and structured processes [3] [22] [23].This study varies specification format while holding content constant, extending the comparison to architecture-level specifications in multi-turn agent generation.Source: A. Specification-Driven LLM Code… · p. 2

B. Structured Prompting

  • Chain-of-thought prompting and structured task decomposition improve reasoning or multi-file generation, but prior studies address task-level rather than architecture-level prompting.Source: B. Structured Prompting · p. 2

C. Architecture Conformance … A. System Under Test

The paper tests how architecture-specification formats affect code conformance and whether format effects vary with model capability. Its multi-turn experiment evaluates generated systems with automated checks and LLM judging.

C. Architecture Conformance

  • The study extends architecture-governance work by testing whether specification format affects conformance rates and interacts with model capability.Prior work proposed post-generation checks comparing dependency graphs with declared constraints.Source: C. Architecture Conformance · p. 2

D. Diagrams-as-Code and API Specifications

  • The experiment is the first to compare Mermaid, C4/Structurizr DSL, and TypeScript interface contracts as inputs for system-level code generation.The passage notes that OpenAPI supplies formal, machine-readable API contracts and that text-based architecture formats are widely used for version-controllable documentation.Source: D. Diagrams-as-Code and API… · p. 2

E. Multi-Turn Agents and Evaluation

  • The evaluation combines automated static constraint checking with LLM-as-judge scoring in a multi-turn agent harness.The cited closest precedent judged architectural quality for patches to existing code, whereas this study evaluates greenfield generation.Source: E. Multi-Turn Agents and Evaluation · p. 2

F. Cross-Model Capability Effects

  • Specification-format effects were 10× larger on Claude Haiku 4.5 than Claude Sonnet 4.6 in the cross-model experiment.Prior benchmarks showed large code-generation performance gaps between model tiers, but had not measured whether specification format moderates that gap.Source: F. Cross-Model Capability Effects · p. 2

III. APPROACH

  • The experiment describes one reference system in five informationally equivalent formats and gives them to six LLMs from three vendor families.Generated code is evaluated through three independent channels.Source: §III · p. 2

INPUTS (5 formats) PROCESSING OUTPUT EVALUATION

  • Each multi-turn session yields a TypeScript codebase assessed by an LLM judge, automated verification, and process metrics.The judge scores four dimensions on a 1–10 scale; verification covers route coverage and constraint compliance, while process metrics include token cost, compilation, and demo execution.Source: Figure 1 · p. 3

A. System Under Test

  • The system under test is a self-contained, in-memory Task Management API whose services must communicate through an event bus rather than direct calls.Its architectural complexity comes from inter-service communication patterns and data-ownership rules.Source: A. System Under Test · p. 3

B. What We Mean by “Architecture” … 6) Format Comparison:

The paper defines architecture as a seven-part system specification and compares five formats that encode the same architecture, varying only its representation. The formats range from prose to machine-parseable API and code contracts.

B. What We Mean by “Architecture”

  • Architecture comprised seven elements: components, communication patterns, data ownership, behavioral constraints, API routes, design rationale, and file structure.The specified system had an API Router, five domain services, and an Event Bus; services communicated through events and exclusively owned their in-memory stores.Source: B. What We Mean by “Architecture” · p. 3

C. Specification Formats

1) Prose (Control):

  • The prose control described the system in flowing natural-language paragraphs, resembling a senior developer’s architecture design document.Source: 1) Prose (Control) · p. 3

2) Mermaid + Constraints + ADRs:

3) OpenAPI + Mermaid + Constraints:

  • The OpenAPI format extended Mermaid, constraints, and ADRs with an OpenAPI 3.0 YAML contract for routes, schemas, status codes, and data models.The passage characterizes this as a formal, machine-checkable API contract.Source: 3) OpenAPI + Mermaid + Constraints · p. 3

4) C4/Structurizr DSL:

  • The C4 format used System Context-to-Container-to-Component decomposition in Structurizr DSL, supplemented by diagrams, component tables, constraints, and ADRs.Source: 4) C4/Structurizr DSL · p. 4

5) TypeScript Contracts + Architecture Rules:

  • The TypeScript format supplied exact model and service interfaces, event payload types, and pseudo-code ArchUnit-style architecture rules.An example rule prohibited service files from importing other service files.Source: 5) TypeScript Contracts + Architecture… · p. 4

6) Format Comparison:

  • Table I compares how each format expresses the constraint against direct service-to-service calls, illustrating a progression toward machine-parseable specificity.Source: 6) Format Comparison · p. 4

D. Experimental Harness … B. Key Observations

Across Sonnet 4.6 trials, specification formats produced similar overall scores, while differing in constraint compliance and token cost. The experiment used a shared multi-turn harness and consistent evaluation across six models.

D. Experimental Harness

  • The TypeScript harness gave each independent agent trial five tools for iteratively writing files, compiling, fixing errors, running demos, and signaling completion.The tools were write_file, read_file, list_files, run_command, and done.Source: D. Experimental Harness · p. 4

E. Evaluation

  • A blind LLM judge scored each trial from 1–10 on architectural adherence, completeness, code quality, and constraint compliance.The judge was blind to specification type.Source: E. Evaluation · p. 4

IV. EXPERIMENTAL SETUP

  • 90 trials covered 30 format-by-model cells across six models from Anthropic, OpenAI, and Google, with the same implementation prompt and Claude Sonnet 4.6 judge across rounds.Each trial began in a clean directory; the shared judge was used to ensure score differences reflected code quality rather than judge capability.Source: §IV · p. 4

A. Sonnet 4.6 Results (Frontier Model)

B. Key Observations

  • 0.17 points separated Sonnet’s format means, from 8.33 (C4) to 8.50 (Mermaid+constraints), with no format dominating across dimensions.The best and worst trials both came from structured formats: Mermaid-1 and C4-1 scored 9.50, while Mermaid-3 scored 7.25.Source: B. Key Observations · p. 4
  • OpenAPI, prose, and TypeScript contracts had zero automated constraint violations across trials, while Mermaid and C4 did not achieve the best automated compliance.OpenAPI achieved zero constraint violations; automated checks included static import analysis, structural completeness, and TypeScript compilation.Source: B. Key Observations · p. 4, E. Evaluation · p. 4
  • 433K tokens made prose 2.4× less costly than Mermaid+constraints (1,060K), while their overall scores were comparable (8.42 vs. 8.50).Prose had the tightest distribution, with a range of 0.25; structured formats raised the quality ceiling while introducing volatility.Source: B. Key Observations · p. 4

C. Qualitative Analysis · D. Haiku 4.5 Results (Smaller Model) · E. Cross-Model Comparison

Specification format interacts with model capability: score spreads are small on frontier models and larger on non-frontier models. Qualitative trials show how structured formats guide architectural choices.

C. Qualitative Analysis

  • Across formats, resolving comment-notification enrichment without Comment-to-Task calls ranged from ad hoc prose workarounds to structured routing guided by each format’s specification.Mermaid trials used router-passed enrichment data, while OpenAPI, C4, and TypeScript contracts clarified orchestration, layer responsibility, and accepted service-method data.Source: C. Qualitative Analysis · p. 4
  • Structured conditions more consistently placed services in the specified src/services/ subdirectory, while prose trials varied between flat and nested structures.The interface signatures in TypeScript contracts also made explicit what data each service method accepts.Source: C. Qualitative Analysis · p. 4
  • Both 9.50-scoring trials, Mermaid-1 and C4-1, shared custom error classes and clean event-bus isolation, and the judge noted methodical specification-following.These trials also consumed the most tokens in their conditions.Source: C. Qualitative Analysis · pp. 4–5

D. Haiku 4.5 Results (Smaller Model)

  • Haiku 4.5 was tested as a smaller, faster, cheaper model using Sonnet 4.6 as the consistent judge, with complete trial results presented in Table IV.The supplied passage gives the study setup and table scope but no Haiku score values.Source: D. Haiku 4.5 Results (Smaller Model) · p. 5, Table IV · p. 7

E. Cross-Model Comparison

  • Mermaid+constraints leads on Sonnet, TypeScript contracts on GPT-5, OpenAPI on Haiku and Gemini Pro, and C4 on Gemini Flash; all structured formats outperform prose on smaller models.No single structured format dominates across all models.Source: E. Cross-Model Comparison · p. 5
  • 0.17 points on Sonnet and 0.92 on GPT-5, versus 0.83–2.42 on non-frontier models, show that format matters more for non-frontier models.Gemini Pro’s 2.42-point spread is driven by one anomalous condition, TS Contracts: 4.50.Source: E. Cross-Model Comparison · p. 5, Table V · p. 7

VI. AGENT PROCESS ANALYSIS … C. Debugging Intensity

Process analysis tracks how agents validate and debug their code across model tiers and specification formats. The results show differences in self-validation, compilation effort, and file rewriting.

VI. AGENT PROCESS ANALYSIS

  • The harness records every file write, compilation attempt, demo execution, and error encountered during each trial.The section analyzes how agents build software under different specifications and model tiers, process data rarely reported in code-generation evaluations.Source: §VI · p. 7

A. Self-Validation Behavior

  • Sonnet ran end-to-end validation in every trial, while Gemini Flash never ran the demo; demo run rates declined monotonically across the capability spectrum.Agents that skip self-validation may produce code that compiles but fails at runtime.Source: Table VI · p. 7, A. Self-Validation Behavior · pp. 7–8

B. Compilation Effort

  • Haiku attempted TSC 8.6 times per trial versus Sonnet’s 5.1, yet had a higher pass rate (61% vs. 53%); GPT-5-mini had 4.7 attempts and a 17% pass rate.The paper reports that GPT-5-mini often introduced new errors while fixing old ones.Source: B. Compilation Effort · p. 8
  • On Haiku, structured specifications reduced TSC failures from 3.33 (prose mean) to 2.33 (Mermaid+constraints).The authors attribute the improvement to explicit component boundaries helping the agent produce correct code on the first attempt.Source: B. Compilation Effort · p. 8

C. Debugging Intensity

  • Gemini Flash’s rewrite rate was 33% with prose versus 13% with OpenAPI, which provided a clearer implementation target.Rewrite rate is the fraction of file writes that overwrite a previously written file, used as a proxy for debugging intensity.Source: Table VII · p. 8, C. Debugging Intensity · p. 8
  • Sonnet with Mermaid+constraints had a 36% rewrite rate despite final quality of 8.50.These trials involved 40+ turns of iteration toward closer specification adherence, with files rewritten to better match the specification.Source: Table VII · p. 8, C. Debugging Intensity · p. 8

D. Code Volume and Completeness · E. Automated Architecture Verification · F. Resource Efficiency

TypeScript contracts bring the weakest model to perfect automated architecture compliance and full route coverage, while code volume alone does not determine completeness. Resource efficiency also varies: fewer tokens can yield strong score-per-token ratios, while debugging loops can make a cheaper model more costly in practice.

D. Code Volume and Completeness

  • Generated code ranged from 590 lines/9.7 files (Gemini Flash) and 642/12.1 (GPT-5-mini) to 1,350/15.7 (Haiku) and 1,504/14.3 (Sonnet), yet completeness was 7.0 vs. 9.0.GPT-5-mini and Gemini Flash produced less than half of Sonnet’s code, but the judge prioritized architectural correctness over implementation depth.Source: D. Code Volume and Completeness · p. 8

E. Automated Architecture Verification

  • TypeScript contracts achieved 100% API route coverage for Gemini Flash, up from one-third under prose, Mermaid, and OpenAPI; GPT-5-mini reached 100% with structured formats versus 85% on prose.Sonnet, Haiku, and GPT-5 achieved 100% route coverage regardless of format.Source: E. Automated Architecture Verification · p. 8, Table VIII · p. 9
  • TypeScript contracts achieved 100% compliance across all six models, while the weakest model scored below 71% on prose despite receiving the same architectural information.Every other format had at least one model below 97% compliance.Source: E. Automated Architecture Verification · p. 8, Table VIII · p. 9
  • TypeScript interfaces let models implement API surfaces and internal service boundaries directly, avoiding the translation required by diagrams, C4 models, and OpenAPI.For models with limited architectural reasoning capacity, removing that translation step can distinguish a complete implementation from an incomplete one.Source: E. Automated Architecture Verification · p. 9
  • GPT-5 achieved 100% compliance on every format, extracting from prose the architectural structure that the weakest Google model could extract only from typed interfaces.The passage describes structured specifications as compensating for the capability gap between these extremes.Source: E. Automated Architecture Verification · p. 9

F. Resource Efficiency

  • GPT-5-mini and Gemini Flash had the highest score-per-token ratios while using 3–4× fewer tokens, but Haiku used 735K tokens versus Sonnet’s 640K and scored 1.9 points lower.Haiku’s extended compilation-debugging loops consumed more tokens than Sonnet’s, so the cheaper model was not always cheaper to complete a task.Source: F. Resource Efficiency · p. 9

G. Failure Mode Taxonomy … B. Structured Specs and the Capability Gap

Structured architecture specifications narrowed quality gaps for weaker models, while architecture guidance itself improved Sonnet 4.6’s results over requirements alone. Specification format also shaped distinct model failure modes and their costs.

G. Failure Mode Taxonomy

  • Haiku averaged 8.6 TSC attempts per trial in compilation death spirals, Gemini Flash stopped after 12.5 turns without validation, and Sonnet used 38+ turns and 1,000K+ tokens.These modes were compilation death spiral, premature termination, and perfectionist iteration, respectively.Source: G. Failure Mode Taxonomy · p. 9
  • Structured specifications can reduce compilation death spirals on weaker models but trigger costly perfectionist iteration on strong models without proportional quality gain.The compilation loop can consume turns without converging; Sonnet reaches compilable code early but continues rewriting to match the specification.Source: G. Failure Mode Taxonomy · p. 9

A. Does Architecture Guidance Matter at All?

  • Without architectural guidance, Sonnet achieved 100% route coverage and a working demo but used a flat handler-and-shared-store pattern and produced no event bus in three trials.The trials omitted architectural guidance such as components, event bus, service boundaries, data ownership rules, and file structure.Source: A. Does Architecture Guidance Matter… · pp. 9–10
  • 3.34 points separated Sonnet 4.6’s overall judge score with architecture guidance from requirements alone (5.08 → 8.42).The steepest drops without guidance were architectural adherence (−4.67) and constraint compliance (−5.00).Source: A. Does Architecture Guidance Matter… · p. 9, Table X · p. 10

B. Structured Specs and the Capability Gap

  • 33% route coverage under prose rose to 100% under TypeScript contracts for Gemini Flash, a 3× improvement from format alone.The comparison used the same model and information.Source: B. Structured Specs and the Capability… · p. 10
  • 0.17 for Sonnet and 0.92 for GPT-5 were the format spreads, versus 0.83–2.42 across four non-frontier models; Fig. 3 shows the gap narrowing from 2.8 to 1.7 points.The Anthropic comparison was 0.17 for Sonnet versus 1.67 for Haiku, a 10× difference; OpenAI’s spreads were 0.92 and 0.83 across tiers.Source: Figure 3 · p. 10, B. Structured Specs and the Capability… · p. 10

C. No Universal Best Format … 2) Inter-Judge Agreement:

No single structured specification format is best across models, and the evaluation finds both capability-linked differences in agent behavior and meaningful variation between judges. The findings identify model-specific format winners, a mid-capability cost inversion, declining self-validation, and moderate inter-judge agreement.

C. No Universal Best Format

  • The best format varies by model: Mermaid leads Sonnet (8.50), OpenAPI Haiku (7.25) and Gemini Pro (6.92), TypeScript contracts GPT-5 (7.50), and C4 Gemini Flash (6.83).Claude models respond well to visual component diagrams, GPT-5 to typed interface contracts, and Gemini models to hierarchical decomposition and formal API schemas; teams should test formats with their specific model.Source: C. No Universal Best Format · p. 10

D. Inverted Cost Efficiency

  • Haiku 4.5 consumes 735K tokens per trial, 15% more than Sonnet’s 640K, while scoring 1.9 points lower.Haiku makes 8.6 compilation attempts per trial versus Sonnet’s 5.1, spending tokens on repeated attempts and extended debugging loops.Source: D. Inverted Cost Efficiency · p. 10
  • GPT-5-mini (225K, score 6.72) and Gemini Flash (223K, score 6.35) are fast and cheap but reach lower quality through premature termination, unlike the debugging-loop cost inversion.The inversion forms a capability–cost “valley” among models capable of iterative repair but not quick success; Dente et al. [2] observed constraints most degraded mid-capability models that attempted but failed to satisfy them.Source: D. Inverted Cost Efficiency · p. 10

E. Self-Validation as a Capability Indicator

  • Demo-run rates decline from Sonnet’s 100% to Gemini Flash’s 0%, with Haiku at 80%, GPT-5 at 53%, GPT-5-mini at 40%, and Gemini Pro at 20%.The passage describes demo run rate as an objective proxy that correlates strongly with final quality and needs no judge, subjective scoring, or ground truth.Source: E. Self-Validation as a Capability… · pp. 10–11
  • If an agent does not run its own tests, its generated code should not be trusted without external validation.Source: E. Self-Validation as a Capability… · p. 11

1) Judge–Automation Correlation:

  • GPT-5 aligns more strongly than Sonnet on completeness (r = 0.85 vs. 0.63) and constraint violations (r = −0.48 vs. −0.33), while both correlate weakly with architecture compliance.The study compared judge scores with automated metrics across Sonnet’s 90 trials and GPT-5’s 28 parseable re-judged trials, using GPT-5 as a cross-vendor second judge.Source: 1) Judge–Automation Correlation · p. 11, Table XI · p. 11, F. Can We Trust the Judge? · p. 11
  • In 11 of 90 Sonnet-judged trials, automated checks found zero violations while the judge scored constraint compliance ≤5.The judge identified semantic violations invisible to import analysis, including router-mediated cross-service orchestration and constructor-injected dependencies bypassing the event bus; one score was a perfect 10 despite a detected cross-service import.Source: 1) Judge–Automation Correlation · p. 11

2) Inter-Judge Agreement:

  • Sonnet and GPT-5 agree moderately on relative rankings (r = 0.47–0.60), but GPT-5 scores +0.42 points higher on average and is most lenient on constraint compliance (+1.27).GPT-5 re-judged 30 trials, five per model and one per specification format; Table XII reports the inter-judge comparison.Source: 2) Inter-Judge Agreement · p. 11, Table XII · p. 11

3) Same-Family Bias: … J. Limitations

Cross-judge checks find leniency rather than same-family preference, while code-proximate specifications and tier-specific guidance offer practical value. The findings are qualified by limits in sample size, model and system coverage, and specification authorship.

3) Same-Family Bias:

  • GPT-5 was uniformly more lenient than Sonnet across vendor families, with bias from +0.67 on Anthropic code to +1.08 on Google code, rather than showing same-family preference.Both judges’ relative rankings were consistent (r = 0.47–0.60), supporting the format × model interaction and indicating primary results were not inflated for Sonnet-generated code.Source: Table XIII · p. 11, 3) Same-Family Bias · p. 11

G. Process Metrics as First-Class Evaluation

  • Process measures—TSC pass rate, demo run rate, rewrite rate, and turn count—reveal deployment-relevant differences that outcome scores miss, including Sonnet’s iterative validation versus Gemini Flash’s rapid stopping.The paper argues that models can reach similar scores through radically different processes, making demo run rate useful as a deployment-readiness signal.Source: G. Process Metrics as First-Class… · p. 11, §I · p. 12

H. Specification as Implementation Interface

  • TypeScript contracts achieved 100% route coverage across all six models, suggesting compilable interface specifications can remove translation between a human-readable description and implementation structure.This extends Wang et al.’s CodeSpec [3] finding on executable checkers to human-authored interface contracts; the authors argue specifications should be compilable, not merely parseable.Source: H. Specification as Implementation… · pp. 11–12

I. Practical Recommendations

  • Specification recommendations vary by tier: prose for frontier models, OpenAPI or TypeScript contracts for mid-tier models, and TypeScript contracts for Gemini Flash.Structured formats yield marginal gains at significant token overhead for frontier models; mid-tier models show 0.83–2.42 point spreads, while TypeScript contracts alone achieve 100% route coverage for the small-model case.Source: Figure 4 · p. 12, §I · p. 12

J. Limitations

  • Across the 5 × 6 design, n = 3 per cell leaves individual comparisons underpowered; the format × model interaction is exploratory, while automated route-coverage and compliance measurements are deterministic.The reported 33% →100% route-coverage improvement requires no statistical test; n ≥5 per cell would enable formal hypothesis testing.Source: J. Limitations · p. 12
  • The evaluation covers six proprietary models and one moderately complex system, so open-source models and larger systems may show different or stronger format effects.The authors specifically note that lower-tier open-source models may exhibit different interaction patterns and that larger systems may amplify effects even for frontier models.Source: J. Limitations · p. 12
  • Limitations include using Sonnet 4.6 as the sole judge, testing only five formats, and having the same authors write every specification.Claude self-preference magnitude is unknown; systems-engineering and infrastructure formats were untested, and the prose or structured specifications may have been unusually clear or well-organized.Source: J. Limitations · p. 12

VIII. THREATS TO VALIDITY · IX. FUTURE WORK · X. CONCLUSION

The paper concludes that specification format has capability-dependent effects and functions as a capability equalizer; it also identifies validity limits and future tests.

VIII. THREATS TO VALIDITY

  • Specifications came from the same authors, and sequential model rounds could introduce temporal confounds; each trial nevertheless started with a clean context.The authors reviewed all five specifications for information equivalence to mitigate potential author bias.Source: §VIII · p. 12
  • External validity is limited to one ∼7-component system; LLM-as-judge may diverge from expert assessment and introduce same-family bias when judging Sonnet code.Generalization to other domains and larger systems requires study; recent work reports strong judge–expert correlation.Source: §VIII · p. 12

IX. FUTURE WORK

  • Future work will test additional model tiers to locate the format-effect threshold and hybrid specifications combining formats’ dimension- and model-specific strengths.Examples include GPT-4o versus GPT-4o-mini, Gemini Pro versus Flash, and OpenAPI plus TypeScript interfaces and constraints.Source: §IX · p. 12
  • Future studies will evaluate systems with 10+ services and whether a second agent can extend code while preserving architectural constraints across formats.The authors hypothesize that the format × model interaction strengthens as system complexity grows.Source: §IX · pp. 12–13
  • Future work will investigate why structured formats increase variance, whether self-verification reduces it, and whether real-time constraint checking reduces model-capability dependency.The proposed enforcement approach integrates architectural constraint checking into the agent’s tool loop.Source: §IX · p. 12

X. CONCLUSION

  • Quality spreads were 0.17–0.92 for frontier models and 0.83–2.42 for non-frontier models; TypeScript contracts raised the weakest model’s API route coverage from 33% to 100%.The comparison covered five formats, six models from three vendor families, and 90 multi-turn agent trials.Source: §X · p. 13
  • Mid-tier models can consume more tokens than frontier models in compilation-debugging loops; self-validation rates fell from 100% to 0% across the capability spectrum.Compilation death spirals and premature termination interacted differently with specification format.Source: §X · p. 13
  • Structured specifications act as a capability equalizer rather than improving frontier models’ best achievable quality; weaker and cost-optimized deployments need proportionally greater specification investment.The paper frames specification format as architectural governance, echoing Konrad et al..Source: §X · p. 13

Related papers

Loading 2608.21747v1…