PaperlayerView exact revision on arXiv

Source-linked AI summary

Architecture as Capability Equalizer for Coding Agents

Arquimedes Canedo

arXiv:2608.21747v1cs.SEcs.AIcs.CL

TL;DR

It remains unclear how architecture specification formats affect code generated by LLM-based coding agents and whether those effects depend on model capability. A controlled comparison across five formats and six models finds a strong format × model interaction, with code-proximate formats most robust as model capability declines.

  • Problem

    How developers should communicate architectural intent to LLM-based coding agents remains open, while current guidance is typically informal prose.

  • Method

    The study compares five architecture specification formats across six models from three vendor families in 90 multi-turn trials and three quality dimensions.

  • Results

    A strong format × model interaction appears: quality spreads are 0.17–0.92 for frontier models versus 0.83–2.42 points for non-frontier models.

  • Takeaways & Limitations

    Structured architecture specifications function as a capability equalizer, with value inversely proportional to model strength and code-proximate formats most robust to degradation.

  • Takeaways & Limitations

    With n = 3 per cell, individual comparisons are underpowered, so the format × model interaction is exploratory rather than statistically tested.

Abstract

from arXiv · show

LLM-based coding agents generate complete software systems from high-level descriptions, yet little is known about how the format of architecture specifications affects the quality of generated code or whether this effect depends on model capability. We present a controlled experiment comparing five informationally equivalent specification formats (informal prose, Mermaid diagrams with constraints and ADRs, OpenAPI, C4/Structurizr DSL, and TypeScript interface contracts with ArchUnit-style rules) across six models from three vendor families (Anthropic Claude, OpenAI GPT, Google Gemini). Across 90 multi-turn agent trials, specification format shows a strong format x model interaction. On the strongest models (Sonnet 4.6, GPT-5), format barely matters (quality spread 0.17-0.92). On weaker models, format produces spreads of 0.83-2.42 points, with code-proximate formats (OpenAPI, TypeScript contracts) recovering most of the capability gap. Mid-tier models can consume more tokens than frontier models for worse output when they enter compilation debugging loops that stronger models avoid. Self-validation rates collapse from 100% (Sonnet) to 0% (Gemini Flash) across the capability spectrum. TypeScript contracts triple API route coverage for the weakest model (33% to 100%). Structured architecture specifications serve as a capability equalizer, with value inversely proportional to model strength and the largest returns for cost-optimized deployments.

I. INTRODUCTION … C. Architecture Conformance

The paper investigates how architecture-specification format affects coding-agent outcomes, testing five formats across models and quality dimensions. It finds that structured, code-proximate formats particularly benefit weaker models, while format matters little for frontier models.

  • I. INTRODUCTION: Five architecture-specification formats are compared across architectural adherence, constraint compliance, and completeness.The formats separate structural description, behavioral constraints, and design rationale to test whether structured specifications improve generated-code quality.
  • I. INTRODUCTION: 0.17–0.92 quality spread occurs on frontier models, versus 0.83–2.42 points on non-frontier models, with code-proximate formats recovering over half the capability gap.This establishes a strong format × model interaction: format matters far more for non-frontier models.
  • I. INTRODUCTION: 735K tokens and a 6.50 score characterize Haiku, compared with 640K tokens and an 8.42 score for Sonnet.Mid-tier models capable of iterative repair can consume more tokens than frontier models for worse output, creating a valley in the capability–cost curve.
  • I. INTRODUCTION: Self-validation drops from 100% for frontier models to 0% for the smallest model.The paper identifies compilation death spirals, premature termination, and perfectionist iteration as distinct failure modes interacting differently with specification format.
  • I. INTRODUCTION: 33% of specified API routes are implemented by Gemini Flash under prose, versus 100% under TypeScript interface contracts.The format-only change produces a 3× improvement measured by automated route coverage analysis.
  • A. Specification-Driven LLM Code Generation: BaxBench reports +5.8% to +9.6% pass@1 gains for OpenAPI over prose, whereas this study examines architecture-level specifications with multi-turn agents and architectural-quality metrics.The study extends API-level findings while differing in generation setting and evaluation focus.
  • A. Specification-Driven LLM Code Generation: 71.8% pass rate versus 43.8% is reported for executable architecture specifications versus textual specifications in CodeSpec.CodeSpec provides iterative checker feedback, while this study uses human-authored declarative documents as static context.
  • B. Structured Prompting: Chain-of-thought prompting and structured task decomposition establish benefits for reasoning and multi-file code generation, but prior studies focus on task-level rather than architecture-level prompting.Architecture conformance work further motivates automated dependency-graph checks and contract-driven governance as frameworks for evaluating architectural guidance.

D. Diagrams-as-Code and API Specifications … A. System Under Test

The paper introduces a cross-model experiment comparing five architecture-specification formats for code-generation agents, using multi-turn sessions and three independent evaluation channels. It also defines a self-contained Task Management API whose architectural complexity tests whether specification formats affect generated systems.

  • D. Diagrams-as-Code and API Specifications: The study is the first to compare Mermaid diagrams, C4/Structurizr DSL, and TypeScript interface contracts as architecture-specification inputs for code-generation agents.It positions these formats alongside established text-based architecture documentation and OpenAPI contracts.
  • E. Multi-Turn Agents and Evaluation: SWE-agent showed that tool use improves agent capability over single-turn generation, while SWE-bench measures functional correctness rather than architectural quality.LLM-as-judge has been reported at 85% agreement with human experts, although same-model self-preference bias is documented.
  • F. Cross-Model Capability Effects: Frontier models resolve 30–50% of SWE-bench issues, whereas smaller models resolve under 5%, motivating tests of whether structured specifications disproportionately help weaker models.The experiment directly compares Claude Sonnet 4.6 with Claude Haiku 4.5 for this interaction.
  • III. APPROACH: The experimental pipeline gives six LLMs from three vendor families five informationally equivalent architecture formats and evaluates their generated code through three independent channels.The channels are an LLM judge, automated verification, and process metrics, with a separate no-architecture baseline.
  • A. System Under Test: The reference system is a Task Management API designed with sufficient architectural complexity to expose differences between specification formats.It comprises seven components.
  • A. System Under Test: The system includes an API Router, User, Project, Task, Comment, and Notification services, plus an in-memory Event Bus.The Task Service includes status transitions from todo to in-progress to done, while notifications are event-driven.
  • A. System Under Test: The API uses exclusively in-memory storage with no external dependencies, making it fully self-contained.Its complexity comes from inter-service communication patterns, data ownership rules, and requiring services to communicate through the Event Bus rather than direct calls.

B. What We Mean by “Architecture” … E. Evaluation

The study defines architecture as a complete, system-level specification and tests whether representation format affects implementation quality when the encoded architecture remains unchanged. It compares five formats using independent multi-turn coding-agent trials evaluated through automated checks and blind judging.

  • B. What We Mean by “Architecture”: Architecture comprises seven elements: components, communication patterns, data ownership, behavioral constraints, API surface, design rationale, and an additional seventh element not enumerated in the supplied passage.The supplied passages explicitly list components, communication patterns, data ownership, six hard rules, 25 routes, and three ADRs, while stating that architecture has seven elements.
  • B. What We Mean by “Architecture”; C. Specification Formats: All five specifications encode the same components, operations, constraints, communication patterns, routes, and rationale; only their representation differs.The experiment therefore isolates representation format as the variable affecting how agents implement the architecture.
  • 3) OpenAPI + Mermaid + Constraints:: OpenAPI extends Mermaid and constraint documentation with a complete OpenAPI 3.0 contract covering routes, schemas, status codes, and data models.The contract is described as formal and machine-checkable.
  • 4) C4/Structurizr DSL:: C4/Structurizr DSL expresses hierarchical System Context, Container, and Component decomposition alongside diagrams, detail tables, constraints, and ADRs.Its supplemental materials preserve the same architectural constraints and rationale.
  • 5) TypeScript Contracts + Architecture Rules:; 6) Format Comparison:: TypeScript contracts provide exact model and service interfaces, event payload types, and ArchUnit-style pseudo-code rules governing imports between service files.The example rule forbids files matching services/*-service.ts from importing other service files.
  • D. Experimental Harness; E. Evaluation: Each trial uses an independent TypeScript-based agent session with five file-and-command tools, iterating through writing, compilation, debugging, demos, and completion signaling.Automated evaluation checks static import violations, structural completeness, and TypeScript compilation; blind judges score adherence, completeness, quality, and compliance on a 1–10 scale.

IV. EXPERIMENTAL SETUP … C. Qualitative Analysis

A controlled 90-trial experiment tested five equivalent architecture formats across six models, with consistent prompts, evaluation, and trial conditions. Results show minimal format differences for Sonnet, while qualitative outcomes varied in how agents resolved cross-service constraints and followed architectural structure.

  • IV. EXPERIMENTAL SETUP: 90 trials covered 30 cells spanning five formats and six models from Anthropic, OpenAI, and Google.The models included Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5, GPT-5-mini, Gemini 2.5 Pro, and Gemini 2.5 Flash.
  • IV. EXPERIMENTAL SETUP: Each trial allowed 16,384 output tokens per turn and 50 turns, began from a clean directory, and used identical prompts and Claude Sonnet 4.6 judging.The shared judge was intended to ensure score differences reflected code quality rather than judge capability.
  • A. Sonnet 4.6 Results (Frontier Model): 8.33–8.50 Sonnet means produced only a 0.17-point spread, with Mermaid+constraints highest and no format dominating every dimension.Table II contains all 15 Sonnet trials, while Table III aggregates conditions.
  • V. RESULTS: Structured formats produced both the experiment’s best and worst trials—9.50 for Mermaid-1 and C4-1 versus 7.25 for Mermaid-3—while prose ranged only 0.25.This indicates a higher quality ceiling alongside greater volatility for structured formats.
  • B. Key Observations: OpenAPI had zero automated constraint violations, matching prose and TypeScript contracts, whereas Mermaid and C4 did not achieve the best automated compliance.Explicit constraint lists alone did not guarantee the strongest automated compliance.
  • B. Key Observations: 433K tokens made prose 2.4× cheaper than Mermaid+constraints at 1,060K while yielding comparable overall scores of 8.42 versus 8.50.Prose was identified as the most cost-efficient format in the Sonnet comparison.
  • C. Qualitative Analysis: Cross-service comment enrichment exposed format-specific reasoning: prose used ad hoc workarounds, while Mermaid, OpenAPI, C4, and TypeScript contracts supplied increasingly explicit orchestration guidance.The shared issue required notification to obtain a task’s assignee ID even though Comment Service could not call Task Service directly.
  • C. Qualitative Analysis: The two 9.50-scoring trials combined custom error classes and clean event-bus isolation, while all structured conditions more consistently followed the specified src/services/ folder structure.The judge described the strongest agents as following the specification methodically; prose varied between flat and nested structures.

RULE 1: NO_CROSS_SERVICE_IMPORTS · D. Haiku 4.5 Results (Smaller Model)

The rule forbids service files from importing other service files. Haiku 4.5 was evaluated as a smaller, faster, cheaper model using Sonnet 4.6 for consistent scoring, with results presented in Table IV.

  • RULE 1: NO_CROSS_SERVICE_IMPORTS: Service files matching "services/*-service.ts" MUST NOT import from other "services/*-service.ts" files.
  • RULE 1: NO_CROSS_SERVICE_IMPORTS: The Haiku trials consumed the most tokens in their conditions.This indicated that thorough specification adherence required more iteration.
  • D. Haiku 4.5 Results (Smaller Model): The experiment was repeated with Claude Haiku 4.5 to test whether specification format interacts with model capability.
  • D. Haiku 4.5 Results (Smaller Model): Claude Haiku 4.5 was characterized as a smaller, faster, cheaper model.
  • D. Haiku 4.5 Results (Smaller Model): Haiku results used Sonnet 4.6 as the judge for consistent scoring.
  • D. Haiku 4.5 Results (Smaller Model): Table IV presents the complete Haiku results.

E. Cross-Model Comparison · VI. AGENT PROCESS ANALYSIS · A. Self-Validation Behavior

Specification format interacts strongly with model capability: format has little effect on frontier models but substantially affects non-frontier models, where structured formats outperform prose. Process data further show that self-validation declines across the capability spectrum, with some agents stopping without runtime verification.

  • E. Cross-Model Comparison: Format interacts strongly with model capability across the evaluated models.The comparison covers six models from three vendor families and 90 trials.
  • E. Cross-Model Comparison: 0.17 points is Sonnet’s format spread, versus 0.83 for GPT-5-mini and 2.42 for Gemini Pro.The smaller-model spreads are up to 14× larger than Sonnet’s.
  • E. Cross-Model Comparison: Mermaid+constraints leads on Sonnet, TypeScript contracts on GPT-5, OpenAPI on Haiku and Gemini Pro, and C4 on Gemini Flash.No single structured format dominates across all models, while all structured formats outperform prose on smaller models.
  • E. Cross-Model Comparison: 0.17 and 0.92 are the format spreads for Sonnet and GPT-5, while non-frontier models range from 0.83 to 2.42.Gemini Pro’s average score is 6.07, and its large spread is driven by TS Contracts scoring 4.50 in one anomalous condition.
  • VI. AGENT PROCESS ANALYSIS: Every file write, compilation attempt, demo execution, and error was captured to analyze how agents build software under different specifications and model tiers.The process analysis targets data rarely reported in code-generation evaluations.
  • A. Self-Validation Behavior: Self-validation was measured through TypeScript compilation attempts and end-to-end demo executions.The harness logs tsc --noEmit and npx tsx src/demo.ts, with results summarized by model in Table VI.
  • A. Self-Validation Behavior: 100% of Sonnet trials validated output end-to-end, while Gemini Flash’s demo run rate was 0% across trials.Demo run rates decline monotonically across the capability spectrum; agents that skip self-validation can ship code that compiles but fails at runtime.

B. Compilation Effort … F. Resource Efficiency

Compilation effort, debugging intensity, code volume, and automated verification vary substantially by model and specification format. Code-proximate TypeScript contracts most strongly compensate for weaker models, while token efficiency depends on avoiding prolonged debugging loops rather than simply minimizing output.

  • B. Compilation Effort: 8.6 TSC attempts per trial versus 5.1 for Sonnet, Haiku achieved a higher pass rate of 61% versus 53%.GPT-5-mini had the worst compilation efficiency: 4.7 attempts with a 17% pass rate, often introducing errors while fixing others.
  • B. Compilation Effort: 3.33 TSC failures under prose fell to 2.33 with Mermaid+constraints on Haiku, as explicit component boundaries improved first-attempt correctness.
  • C. Debugging Intensity: 33% of Gemini Flash’s prose file writes were rewrites, versus 13% with OpenAPI, while Sonnet reached 36% with Mermaid+constraints despite 8.50 final quality.Sonnet’s high rewrite rate reflected trials exceeding 40 turns while iterating toward closer specification adherence.
  • D. Code Volume and Completeness: 642 lines from GPT-5-mini and 590 from Gemini Flash were less than half Sonnet’s 1,504 lines, yet completeness was only 1–2 points lower.The judge prioritized architectural correctness over implementation depth, favoring smaller well-structured systems over verbose but flawed ones.
  • E. Automated Architecture Verification: 100% route coverage with TypeScript contracts versus one-third under prose, Mermaid, and OpenAPI gave Gemini Flash a 3× format-only improvement.Sonnet, Haiku, and GPT-5 achieved 100% regardless of format, while GPT-5-mini fell to 85% on prose.
  • E. Automated Architecture Verification: 100% weighted compliance across all six models and three vendor families was achieved only by TypeScript contracts.Typed interfaces are code-proximate: they specify both the API surface and internal service boundaries, unlike formats requiring representational translation.
  • E. Automated Architecture Verification: 100% on every format was achieved by GPT-5, whereas the weakest Google model required typed interfaces to extract equivalent architectural structure.Structured specifications therefore compensated for the capability gap between the frontier OpenAI model and weakest Google model.
  • F. Resource Efficiency: 735K tokens versus Sonnet’s 640K left Haiku scoring 1.9 points lower because it entered extended compilation-debugging loops Sonnet avoided.GPT-5-mini and Gemini Flash had the highest score-per-token ratios because they consumed 3–4× fewer tokens, showing that cheaper models are not always cheaper to completion.

G. Failure Mode Taxonomy … B. Structured Specs and the Capability Gap

The study identifies distinct agent failure modes, confirms that architecture guidance substantially improves quality, and shows that structured specifications narrow weaker models’ capability gap. Their largest practical effect is on route coverage, where TypeScript contracts raise Gemini Flash from 33% to 100%.

  • G. Failure Mode Taxonomy: Three failure modes emerge: compilation death spirals, premature termination, and perfectionist iteration.Haiku averages 8.6 TSC attempts per trial, while Gemini Flash stops after 12.5 turns without compiling or testing.
  • A. Does Architecture Guidance Matter at All?: Architecture guidance raises the overall judge score by 3.34 points, from 5.08 to 8.42.The requirements-only prompt removes component, event-bus, service-boundary, data-ownership, and file-structure guidance while retaining identical features and tooling.
  • A. Does Architecture Guidance Matter at All?: −4.67 architectural adherence and −5.00 constraint compliance are the steepest quality drops without architecture guidance.The agent still achieves 100% route coverage and a working demo, but defaults to flat handlers and shared storage.
  • A. Does Architecture Guidance Matter at All?: None of three requirements-only trials produced an event bus, despite satisfying every functional requirement.This demonstrates that functional completeness does not imply architectural invention when service isolation and event-driven communication are unspecified.
  • B. Structured Specs and the Capability Gap: Frontier models show format spreads of 0.17 and 0.92, versus 0.83–2.42 for four non-frontier models.Anthropic shows a 10× difference between Sonnet’s 0.17 and Haiku’s 1.67, while OpenAI’s spreads are roughly equal across tiers.
  • B. Structured Specs and the Capability Gap: Structured formats lift weaker models toward frontier performance, narrowing the capability gap.Under prose, quality drops steeply from frontier to weaker models; structured specifications reduce that decline.
  • B. Structured Specs and the Capability Gap: 33% to 100%: Gemini Flash’s specified API route coverage triples under TypeScript contracts rather than prose.The comparison holds model and information constant, making format alone responsible for the improvement.

C. No Universal Best Format … 2) Inter-Judge Agreement:

Format effectiveness varies by model, while mid-tier models can incur higher completion costs through debugging loops. Self-validation declines with capability, and judge scores show moderate ranking agreement but systematic calibration differences.

  • C. No Universal Best Format: The best format varies by model, with Mermaid+constraints leading Sonnet, OpenAPI leading Haiku and Gemini Pro, TypeScript contracts leading GPT-5, and C4 leading Gemini Flash.No single structured format dominates across all models.
  • C. No Universal Best Format: Model-specific strengths explain format differences: Claude favors Mermaid, GPT-5 typed contracts, and Gemini hierarchical decomposition or formal API schemas.Teams should test each model against multiple formats rather than assume one universal structured format.
  • D. Inverted Cost Efficiency: 735K tokens versus 640K: Haiku 4.5 consumes 15% more than Sonnet while scoring 1.9 points lower.Haiku averages 8.6 compilation attempts per trial versus Sonnet’s 5.1, producing extended debugging loops.
  • D. Inverted Cost Efficiency: 225K tokens and score 6.72 for GPT-5-mini versus 223K and 6.35 for Gemini Flash show that the cost inversion is not universal.These models terminate prematurely rather than entering extended debugging; the inversion affects models capable of iterative repair but unable to succeed quickly.
  • E. Self-Validation as a Capability Indicator: 100% to 0%: demo run rates decline monotonically from Sonnet to Gemini Flash, with Haiku at 80%, GPT-5 at 53%, GPT-5-mini at 40%, and Gemini Pro at 20%.Demo execution is an objective behavioral proxy requiring no judge, subjective scoring, or ground truth, and it correlates strongly with final quality.
  • E. Self-Validation as a Capability Indicator: If an agent does not run its own tests, its generated code should not be trusted without external validation.This guardrail follows from self-validation’s relationship to model capability and final quality.
  • 1) Judge–Automation Correlation:: GPT-5 correlates more strongly than Sonnet with completeness (r = 0.85 vs. 0.63) and constraint violations (r = −0.48 vs. −0.33), while both correlate weakly with architecture compliance.This indicates that architectural quality is evaluated on a different axis from import-level static analysis.
  • 2) Inter-Judge Agreement:: r = 0.47–0.60: the two judges agree moderately on relative rankings, but GPT-5 scores 0.42 points higher on average and +1.27 on constraint compliance.GPT-5 is uniformly more lenient than Sonnet, indicating absolute-calibration differences.

3) Same-Family Bias: … J. Limitations

Cross-judge validation finds leniency rather than same-family preference and preserves the format × model interaction, while practical guidance favors code-proximate specifications for weaker models. The study’s limitations include underpowered cell comparisons, proprietary models, one system, judge-family concerns, incomplete format coverage, and author-controlled specification quality.

  • 3) Same-Family Bias:: GPT-5 was uniformly more lenient than Sonnet across vendor families, with bias ranging from +0.67 on Anthropic code to +1.08 on Google code.The observed pattern is leniency bias, not same-family preference.
  • 3) Same-Family Bias:: Relative rankings were consistent across judges (r = 0.47–0.60), supporting the format × model interaction and reducing concern that Sonnet’s judgments inflated Sonnet-generated code.This cross-judge validation strengthens the primary findings.
  • G. Process Metrics as First-Class Evaluation: Process metrics such as TSC pass rate, demo run rate, rewrite rate, and turn count reveal deployment-relevant differences that outcome scores alone can miss.Models can achieve the same score through radically different processes.
  • H. Specification as Implementation Interface: 100% route coverage across all six models was achieved only by TypeScript contracts, which directly encode implementation structure rather than requiring translation from description to code.The result motivates compilable rather than merely parseable architecture specifications.
  • I. Practical Recommendations: Frontier models need no structured specification, mid-tier models benefit from OpenAPI or TypeScript contracts, and small models require TypeScript contracts for complete route coverage.The recommended strategy depends on model tier.
  • I. Practical Recommendations: All models should receive numbered constraints, while demo run rate should be monitored as a deployment-readiness signal.Constraint compliance is the dimension most sensitive to format and model capability.
  • J. Limitations: n = 3 per cell in the 5 × 6 design leaves individual cell comparisons underpowered, so the format × model interaction is exploratory rather than statistically tested.The strongest evidence comes from deterministic census measurements such as route coverage and compliance scores.
  • J. Limitations: The study used six proprietary models, one moderately complex system, five engineering-oriented formats, and author-written specifications, while Sonnet judged every round.Open-source models, larger systems, additional formats, and different specification authors may produce different interaction patterns; Sonnet’s same-family bias magnitude is unknown.

VIII. THREATS TO VALIDITY · IX. FUTURE WORK · X. CONCLUSION

The study finds that architecture format interacts strongly with model capability, while its conclusions are bounded by threats to validity and several proposed avenues for broader evaluation. Structured specifications chiefly equalize weaker agents rather than improving frontier-model ceilings.

  • VIII. THREATS TO VALIDITY: All specifications were authored by the same researchers, model rounds were sequential, and each trial began from a clean context without shared state.Information-equivalence review mitigated author bias, but temporal effects such as API load or model updates could confound comparisons.
  • VIII. THREATS TO VALIDITY: The experiment covered one system, six models from three families, and five formats, while its ∼7-component complexity may understate larger-scale advantages.Generalization to data pipelines and UI applications requires further study.
  • VIII. THREATS TO VALIDITY: LLM-as-judge evaluation may diverge from human experts, and Sonnet judging Sonnet-generated code may introduce same-family bias.Recent work nevertheless reports strong correlation between LLM and expert assessment.
  • IX. FUTURE WORK: Future work should test additional model tiers and hybrid specifications combining OpenAPI, TypeScript interfaces, and constraint lists.These experiments would map capability thresholds and assess dimension-specific and model-specific strengths.
  • IX. FUTURE WORK: Larger systems with 10+ services should be evaluated because the format × model interaction may strengthen with system complexity.The current study’s moderate complexity limits conclusions about scaling.
  • IX. FUTURE WORK: Future studies should examine variance reduction, automated constraint enforcement, and incremental development across formats.Proposed tests include selfverification, real-time architectural feedback, and preserving constraints when a second agent extends existing code.
  • X. CONCLUSION: 90 multi-turn agent trials found a strong format × model interaction: frontier-model spreads were 0.17–0.92, versus 0.83–2.42 points for non-frontier models.OpenAPI and TypeScript contracts were the most robust to model degradation.
  • X. CONCLUSION: Mid-tier models sometimes used more tokens for worse output in compilation debugging loops, while self-validation collapsed from 100% to 0%; structured specifications equalized capability.Their value was inversely proportional to model strength, and they primarily improved weaker-agent reliability rather than frontier quality ceilings.
Loading 2608.21747v1…