PaperlayerView exact revision on arXiv

Source-linked AI summary

Constraint Decay: The Fragility of LLM Agents in Backend Code Generation

Francesco Dente, Dario Satriani, Paolo Papotti

arXiv:2605.06445v2cs.SEcs.AI

TL;DR

Production-grade backend generation requires both functional correctness and structural compliance, yet existing evidence on how agents handle those constraints is limited. This study evaluates agents under a fixed API contract as structural requirements accumulate, finding a substantial performance decline. The findings also identify data-layer defects as leading failure causes.

  • Problem

    The impact of structural requirements on agent performance is underexplored, despite production backends requiring architectural patterns, database integration, and ORM layers alongside API-contract adherence.

    Source: §1 · p. 1

  • Method

    The study fixes an API contract and evaluates agent-generated backends with behavioral tests and structural verifiers while layering architectural, database, and ORM requirements.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    27.48 percentage points: average loss in A% from L0 to L3, with structural constraints consistently worsening performance despite substantial variation in the drop.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    Agents remain unreliable for production-grade backend development, although they are reliable for rapid prototyping under unconstrained generation.

    Source: §6 · p. 10

    Some model-agent evaluations use a representative 16-task subset rather than the full task set, due to cost-driven scope restrictions.

    Source: §4 · p. 6

Abstract

from arXiv · show

Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications. However, production-grade software requires strict adherence to structural constraints, such as architectural patterns, databases, and object-relational mappings. Existing benchmarks often overlook these non-functional requirements, rewarding functionally correct but structurally arbitrary solutions. We present a systematic study evaluating how well agents handle structural constraints in multi-file backend generation. By fixing a unified API contract across 80 greenfield generation tasks and 20 feature-implementation tasks spanning eight web frameworks, we isolate the effect of structural complexity using a dual evaluation with end-to-end behavioral tests and static verifiers. Our findings reveal a phenomenon of constraint decay: as structural requirements accumulate, agent performance exhibits a substantial decline. Evaluated configurations lose 27.28 points on average in assertion pass rates from baseline to fully specified tasks. Framework sensitivity analysis exposes performance disparities: mid-tier models succeed in minimal, explicit frameworks (e.g., Flask) but perform substantially worse on average in convention-heavy environments (e.g., FastAPI, Django). Finally, error analysis identifies data-layer defects (e.g., incorrect query composition and ORM runtime violations) as the leading root causes. This work highlights that jointly satisfying functional and structural requirements remains a key open challenge for coding agents.

1 Introduction

Production backend generation requires agents to satisfy API contracts alongside architectural and data-layer requirements, but the effect of those constraints is underexplored. This study isolates structural constraints and finds that performance declines as they accumulate.

1 Introduction

  • Production-grade backends require API-contract adherence, architectural patterns, database integration, and designated ORM layers, unlike more permissive development scenarios.The impact of these requirements on agent performance remains largely unexplored.Source: §1 · p. 1
  • The study fixes an API contract and uses shared end-to-end behavioral tests to isolate four constraints: framework, architecture, database backend, and ORM integration.It layers the constraints from baseline generation to systems with prescribed architecture, database, and ORM requirements.Source: §1 · p. 2
  • Agents’ performance declines as structural requirements accumulate, and Figure 1 contrasts unconstrained generation with generation under structural constraints.The study identifies this pattern as constraint decay.Source: §1 · p. 2, Figure 1 · p. 2
  • The evaluation contributes 80 greenfield tasks and 20 feature-implementation tasks, alongside separate measures of behavioral correctness and structural compliance.The evaluation pipeline is open-sourced.Source: §1 · p. 2

2 Related Work

Prior benchmarks cover issue resolution, unconstrained generation, single-file solutions, or repository tasks, but do not systematically study structural constraints in multi-file backend generation. This study varies those constraints under a fixed API contract.

Greenfield app generation

  • Existing benchmarks do not target multi-file backend generation while systematically varying structural constraints.Issue-resolution and repository benchmarks modify existing codebases, while other work uses under-specified prompts, single-file solutions, or skeleton code.Source: §2 · pp. 2–3
  • Vero et al. (2024) evaluates backend services from OpenAPI specifications across frameworks, but targets single-file prompted solutions rather than agent-generated multi-file repositories.It does not study how layering architectural constraints affects generation quality.Source: §2 · p. 3

Benchmarks for repository-level coding tasks

  • Repository-level benchmarks include issue resolution, feature addition, function or line completion, and sandbox evaluation, whereas this study generates backend systems from scratch while increasing structural constraints.Source: §2 · p. 3

Software Engineering Agents

  • The study uses Mini-SWE-Agent and OpenHands to examine how agents interact with structural constraint density.These architectures differ in scaffold complexity and tooling.Source: §2 · p. 3

3 Methodology

The methodology holds functional requirements constant while varying structural requirements, enabling comparisons across agents and models. Tasks use a fixed API contract, layered constraints, and separate behavioral and structural evaluation.

3 Methodology

  • The methodology holds the functional specification constant while varying non-functional requirements to isolate their effect and support comparisons across agents and models.Source: §3 · p. 3

3.1 Generation Task Design

  • Each greenfield task asks an agent to build a complete REST API from an empty repository, using an OpenAPI 3.0 contract derived from the RealWorld Conduit API.The contract specifies 19 CRUD operations across five resource groups and is fixed to isolate structural constraints from task semantics.Source: §3.1 · pp. 3–4

3.2 Constraint Dimensions

  • The study varies Clean Architecture, PostgreSQL or SQLite, and SQLAlchemy or Sequelize, while always specifying one of eight frameworks across Python and Node.js.The three optional axes are layered into L0–L3 variants, yielding 80 generation tasks.Source: §3.2 · p. 4

3.3 Evaluation Pipeline

  • A shared HTTP test suite checks behavior, while architecture, database, and ORM verifiers check structural compliance; success requires passing both behavioral tests and applicable verifiers.The behavioral suite covers the unified OpenAPI specification and is decoupled from implementation structure.Source: §3.3 · p. 5

4 Experimental Setup

The experimental setup evaluates agents and models using behavioral and structural criteria, with assertion pass rate as the primary metric. Cost considerations limit some configurations to a representative subset of tasks.

Metrics

  • Assert% (A%) is the primary metric, measuring the mean fraction of behavioral assertions passed across runs; Pass@k measures the chance that at least one sample satisfies all behavioral and structural constraints.Source: §4 · p. 5

Agents and Models

  • The study evaluates Mini-SWE-Agent and OpenHands, which differ in tooling complexity, after adjusting prompts and pipeline settings for code generation.Iteration limits are 300 for Mini-SWE-Agent and 200 for OpenHands.Source: §4 · pp. 5–6
  • Models span small open agentic, large open instruct, large open agentic, and closed-model tiers.The evaluated models include Devstral-Small, Qwen3-Coder-Next, Qwen3-235B-Instruct, MiniMax-M2.5, Kimi-K2.5, GPT-5-mini, GPT-5.2, and GPT-5.4.Source: §4 · p. 6

Cost-driven evaluation scope

  • Cost-driven sampling restricts Kimi-K2.5 and GPT-5.2 to Mini-SWE-Agent and uses 16 representative tasks for those models and OpenHands with MiniMax-M2.5.The subset spans two frameworks per runtime and one variant per constraint level.Source: §4 · p. 6

5 Results

Across the results, increasing structural constraints sharply reduces assertion pass rates, framework choice strongly affects mid-tier configurations, and data-layer defects lead logic-error causes. Repository feature-implementation tasks also show low performance, though their results cannot isolate structural constraints as the sole cause.

5.1 RQ1: Effect of Constraint Accumulation

  • Across nine capable configurations, A% falls an average 27.48 percentage points from L0 to L3, with a 45-pp worst-case loss and a 17-pp most-resilient loss.The worst case is OpenHands + Qwen3-Coder-Next; the most resilient is OpenHands + MiniMax-M2.5.Source: §5.1 · pp. 6–7
  • Constraint decay persists across both scaffolds, although OpenHands helps GPT-5-mini and MiniMax-M2.5 but underperforms Mini-SWE-Agent for Qwen3-Coder-Next.Source: §5.1 · p. 7
  • Matched-pair analysis identifies database specification as the largest marginal penalty; PostgreSQL is −20.4 ± 2.5 A%, versus −8.8 ± 1.7 for clean architecture.The table also reports SQLite at −16.1 ± 2.6 A%.Source: §5.1 · p. 7
  • In 20 repository feature-implementation tasks, only GPT-5.2 exceeds 50% pass@1; these results do not isolate structural constraints as the sole source of difficulty.The tasks require agents to infer conventions from existing codebases with layered architecture, PostgreSQL, and an ORM.Source: §5.1 · p. 8

5.3 RQ3: Root Causes of Failure

  • Logic errors account for 62.1–76.6% of failures for three models, while GPT-5-mini’s largest failure category is server startup failures at 50.5%.Source: §5.3 · p. 9
  • Data-layer defects lead logic-error causes for all four models, accounting for 36.5–60.9% of logic errors (48.4% on average); ORM constraints shift their manifestation from raw SQL errors to ORM runtime errors.Source: §5.3 · p. 10

6 Conclusion

The study identifies constraint decay: LLM coding agents perform well on unconstrained generation but decline under explicit architectural rules. The authors conclude that agents remain useful for rapid prototyping but unreliable for production-grade backend development.

6 Conclusion

  • Structural constraints expose a gap between agents’ strong unconstrained generation and their reliability for production-grade backend development.The authors call this phenomenon constraint decay and suggest structural awareness as a direction for agent developers.Source: §6 · p. 10

A Subset Representativeness

The authors compare full-benchmark scores with a 16-task subset used for cost-constrained model-agent evaluations. The subset preserves performance rankings and constraint-decay trends across the evaluated configurations.

A Subset Representativeness

  • Three model-agent pairs were evaluated only on a 16-task subset spanning four frameworks and four constraint levels.The constraint pipeline runs from unconstrained, to clean architecture, to PostgreSQL, to PostgreSQL plus ORM.Source: A Subset Representativeness · p. 13
  • Pearson r = 0.948 and Spearman ρ = 0.932 show strong agreement between full-set and subset Assert% scores across 32 paired observations.Cross-model rankings are preserved almost perfectly at every constraint level, and observed L0→L3 degradation is consistent across configurations that degrade on the full set.Source: A Subset Representativeness · pp. 13–14, Table 5 · p. 14

B LLM-as-Judge Validation

The appendix reports near-perfect agreement between a GPT-5.2 judge and human annotations for sampled logic errors. It also details static, pattern-based verifiers for architecture, database, and ORM constraints.

B LLM-as-Judge Validation

  • 98% overall accuracy (49/50) and Cohen’s κ = 0.975 quantify GPT-5.2 judge agreement with human labels on sampled logic errors.Four of six categories had perfect precision, recall, and F1; the authors attribute near-perfect agreement largely to informative server logs and test outputs.Source: Table 6 · p. 14, B LLM-as-Judge Validation · p. 14

C Verifier Functions

  • The three verifiers inspect only added lines in each agent’s unified-diff patch and use static pattern matching, without execution or dynamic analysis.They assess structural compliance in the source code written by the agent.Source: C Verifier Functions · p. 14

C.1 Architecture Verifier

  • Architecture compliance requires at least three of four canonical layers as distinct directories and no lower-to-higher-layer import violations.The verifier maps directory and import names to layer aliases; both checks must pass.Source: §C.1 · pp. 14–15

C.2 Database Verifier

  • Database compliance requires evidence of the prescribed SQLite or PostgreSQL engine and no evidence of the alternative.The verifier accounts for Django settings in which a later PostgreSQL assignment overrides a default SQLite assignment.Source: §C.2 · p. 15

C.3 ORM Verifier

  • ORM compliance requires at least one evidence pattern for the prescribed SQLAlchemy or Sequelize ORM, but does not penalize evidence of another ORM.The exception avoids penalizing utility-library imports that may reference other ORMs without using them for data access.Source: §C.3 · p. 15

C.4 Limitations

The verifiers are conservative, but their static and pattern-based design leaves room for false positives and false rejects.

C.4 Limitations

  • Static-only checks can produce false positives because matching source patterns do not establish runtime compliance.For example, a solution could import SQLAlchemy yet execute raw SQL; the authors say such false positives can only weaken the observed decay signal.Source: §C.4 · p. 15
  • Regex matching may miss naming or import patterns, although broad aliases and case-insensitive matching limit false rejects to ≤0.5%.The false-reject rate was validated empirically.Source: §C.4 · pp. 15–16

D Verifier Impact on Reported Metrics

Verifier imperfections have little effect on the reported constraint decay: wrong rejections barely change aggregate scores, while missed violations can only mask decay. Together, these checks support the robustness of the findings.

Wrong rejections

Missed violations

Overall robustness

E Framework Leaderboard

The framework leaderboard compares assertion pass rates across frameworks, while the task design varies structural requirements around a unified API specification. The section also introduces feature implementation as a distinct task setting based on an existing working repository.

E Framework Leaderboard

  • aiohttp and Fastify rank fourth and fifth by average A% across mid-tier configurations, the six frameworks shown, and all eight frameworks.Table 8 gives the full eight-framework breakdown for eight configurations; Qwen3-235B is omitted from the figure because its near-zero scores leave framework rankings within noise.Source: E Framework Leaderboard · pp. 16–17, Table 8 · p. 17

F Generation Task Design

  • Generation tasks ask agents to build a functional REST API server from scratch that conforms to the RealWorld (Conduit) OpenAPI specification.Source: F Generation Task Design · p. 17

F.1 Prompt Template

  • The prompt template combines the API specification and generation instruction with requirements for the framework, files, server configuration, and optional structural constraints.Runtime, framework, port, and ORM placeholders are filled for each task.Source: §F.1 · p. 17

F.2 Constraint Blocks

  • The constraint blocks prescribe Clean Architecture, SQLite or PostgreSQL setup, and SQLAlchemy or Sequelize; L0–L3 include zero to all three blocks.Across eight frameworks, the ten valid block combinations yield 80 distinct generation tasks.Source: §F.2 · p. 18

F.3 Prompt Example

  • Figure 3 presents an abridged prompt example with all constraints active.Source: §F.3 · p. 18

G Feature Implementation Task Design

G.1 Construction by Codebase Ablation

Feature implementation tasks are built by removing feature code from real Conduit repositories while preserving the rest of each application. The evaluation restores the missing functionality and tests it alongside generation tasks using a shared behavioral suite and a containerized pipeline.

G.1 Construction by Codebase Ablation

  • Ablation removes a complete feature vertical from a real Conduit implementation, then verifies the remaining repository still compiles, starts, and passes tests for retained endpoints.The removed code is restored by the agent, and task size follows the feature’s footprint across the repository rather than an arbitrary line or file count.Source: §G.1 · p. 18

G.2 Task Inventory and Statistics

  • The study constructs 20 feature tasks across five framework repositories, with task scope grouped by ablation extent.Table 9 notes that lines and files vary naturally with how repositories organize code across layers.Source: §G.2 · p. 19, Table 9 · p. 20

G.3 Prompt and Evaluation

  • Each task packages the full OpenAPI prompt, stripping patch, pinned repository metadata, and task size; evaluation applies the ablation and agent patches before running the shared Postman suite.The ablation preserves non-target features, so the stripped baseline already passes tests for retained endpoints.Source: §G.3 · pp. 19–20

H Behavioral Test Suite

J Execution Pipeline

  • Evaluation runs in a fully containerized, two-phase pipeline, with behavioral testing and verifier checks performed after the agent-build phase in a pristine environment.The evaluation phase applies the agent patch to a fresh container to prevent build-phase side effects from influencing results.Source: J Execution Pipeline · p. 21, §J.3 · p. 23

J.4 Walkthrough: Generation Task

  • The generation walkthrough shows an empty repository receiving a Flask app, followed by fresh-container testing and checks for architecture, PostgreSQL, and SQLAlchemy compliance.Source: §J.4 · p. 23

J.5 Walkthrough: Feature Implementation Task

  • The feature walkthrough restores comments in an ablated Express/Prisma repository, while relying on the codebase’s implicit constraints rather than running structural verifiers.Source: §J.5 · pp. 23–24

K Revised Agent Scaffolds

  • OpenHands may halt before completing work when models send a final message instead of invoking its finish tool; a re-prompting loop allows up to six turns to resume or finish.Mini-SWE-Agent’s generation adaptation instead replaces its issue-fixing prompt with a four-step generative workflow.Source: K Revised Agent Scaffolds · p. 24

Related papers

Loading 2605.06445v2…