PaperlayerView exact revision on arXiv

Source-linked AI summary

Built to Remember, Shipped to Forget: Operational Contracts, Failures, and Repairs in Containerized LLM Agent Deployments

Meriem Mastouri, Emna Ksontini, Amine Barrak

arXiv:2610.34620v1cs.SE

TL;DR

Agent frameworks and container platforms document conflicting lifecycle expectations, leaving the obligations of their combination unstated. This study derives 14 operational contracts and tests them across deployed systems, induced events, and developer repairs; contract violations are common, and most predicted failures are observed at runtime.

  • Problem

    Framework and platform documentation specifies each layer’s behavior but leaves unstated the deployment obligations created when their lifecycle requirements are combined.

    Source: §I · p. 1, §III · p. 3

  • Method

    The study derives 14 operational contracts from first-party documentation and evaluates them with deployed systems, induced platform events, and mined developer repairs.

    Source: §I · pp. 1–2, §IX · p. 11

  • Results

    75.5% of 433 systems have at least one applicable contract, and 14 of 16 predicted violations manifest as runtime failures.

    Source: §I · p. 2, §IX · p. 11

  • Takeaways & Limitations

    The contract catalogue provides deployment requirements for practitioners and specifications for automated checks, while the repair taxonomy suggests stronger framework and platform defaults.

    Source: §IX · p. 11

    Repository-based RQ2 violation rates are lower bounds because a contract is marked broken only on positive evidence and otherwise is not decidable.

    Source: §VII · p. 10

Abstract

from arXiv · show

LLM agent systems reach production as container images run by platforms such as Docker, Compose, and Kubernetes. Framework and platform document opposite expectations: to the platform a container is disposable, while the framework persists conversations, pauses runs for human input, and resumes interrupted work from saved state; each side specifies only its own behavior, so the obligations their composition creates are stated nowhere. We present the first empirical study of this boundary, stating these obligations as operational contracts: guarantees a deployment must provide to preserve an agent system's required behavior when a deployment-platform event occurs. From the first-party documentation of LangGraph and the three platforms we derive 14 contracts and evaluate them with three independent sources of evidence: 433 deployed open-source agent systems, seven events induced on 12 of them, and 122 developer repairs mined from 39,261 commits, pull requests, and issues. At least one contract applies to 75.5% of the systems, yet its guarantee is routinely missing: of systems that must keep state beyond the process, 37.4% lose it to a restart and 41.5% to a container replacement, and every assessable deployment stops the framework's production server mid-drain. Induced events turn 14 of 16 predicted violations into runtime failures and all seven predicted holds pass. The failures are born in deployment but repaired in code: developer repairs independently cover twelve of the fourteen contracts, 78% change application code, and only two in three fully restore the violated guarantee. The 14-contract catalogue, detection rules, fault-injection harness, and 35-pattern repair taxonomy provide a basis for checking and strengthening agent deployments.

I. INTRODUCTION

Container platforms treat containers as disposable, while agent frameworks depend on persistent state and resumable work; neither layer states the obligations their composition creates. This study derives operational contracts and tests their prevalence, runtime effects, and repairs.

I. INTRODUCTION

  • The platform and framework document opposing lifecycle expectations, but neither specifies what a deployment must guarantee when those expectations are combined.Platforms may stop or replace containers, while frameworks preserve conversations and resume interrupted work; resumption can repeat completed external actions.Source: §I · p. 1
  • The study derives 14 operational contracts from first-party documentation and evaluates them using 433 deployed systems, induced platform events, and mined developer repairs.The runtime evaluation induces seven events on 12 systems; the repair study identifies 122 repairs from 39,261 commits, pull requests, and issues.Source: §I · pp. 1–2
  • 75.5% of 433 systems have at least one applicable contract, and violation rates range from 11% to 100%.Under fault injection, 14 of 16 predicted violations manifest as failures, while all seven predicted holds pass.Source: §I · p. 2
  • Developer repairs cover twelve contracts through 35 recurring fix patterns, and the taxonomy points to application code as a place where deployment guarantees are implemented.The catalogue offers deployment requirements for practitioners and specifications for automated checks; the taxonomy suggests stronger framework and platform defaults.Source: §I · p. 2

II. STUDY DESIGN

The study combines evidence from documentation, repository histories, and deployed agent systems. Its corpus is selected from LangGraph repositories that ship container deployments and execute a repository-defined agent graph.

II. STUDY DESIGN

  • The study design uses documentation for RQ1, repository histories for RQ4, and deployed agent systems for RQ2 and RQ3.The latter two questions share a corpus and evidence collected from each system.Source: §II · p. 2

A. Corpus extraction

  • The corpus starts with GitHub repositories mentioning LangGraph and narrows to non-fork Python repositories that use LangGraph and ship a Dockerfile.The search yields 225,367 candidates; a threshold of at least 10 stars leaves approximately 4,345 repositories before further filtering.Source: A. Corpus extraction · p. 2
  • Repositories are pinned to commits no later than the 6 July 2026 snapshot cutoff, then manually checked to ensure the deployment runs a single agent graph defined in the repository.The preceding filters identify 574 repositories for this verification.Source: A. Corpus extraction · p. 2

B. Annotation procedure

The annotation procedure combines mechanically extracted deployment facts with codebook-based labels, establishes labeling reliability, and screens repositories to identify deployed LangGraph systems and their primary graphs.

B. Annotation procedure

  • Deployment facts are extracted from source and deployment files, while codebook labels capture interpreted code properties and distinguish absent, inapplicable, and uncertain values.Some labels cover the corpus; targeted readings apply only to systems covered by particular contracts.Source: B. Annotation procedure · p. 2
  • Two authors independently labeled applicable fields in a 148-repository pilot, adjudicated disagreements, revised ambiguous rules, then split the remaining 426 repositories.Both authors independently coded a random 10% of the remaining set for corpus-wide labels; after pilot agreement, the first author completed targeted readings.Source: B. Annotation procedure · p. 2
  • 433 deployed agent systems remained after excluding repositories where containers did not execute the repository’s graph or that deployed collections of unrelated graphs.The screening began with 574 repositories; it also identified each in-scope repository’s primary deployed graph as the unit of analysis.Source: B. Annotation procedure · p. 2
  • Pilot agreement was κ = .62 and AC1 = .91 for scope (137/148 repositories), and κ = .61 and AC1 = .94 for collection status (140/148).Unless otherwise stated, reported κ values refer to the pilot.Source: B. Annotation procedure · p. 2

III. RQ1: WHAT MUST CONTAINERIZED DEPLOYMENTS GUARANTEE FOR AGENT SYSTEMS?

Operational contracts make explicit the deployment obligations created when agent-framework requirements meet platform events. The study derives and classifies 14 contracts, covering state, running work, side effects, services, streams, and external resources.

III. RQ1: WHAT MUST CONTAINERIZED DEPLOYMENTS GUARANTEE FOR AGENT SYSTEMS?

  • The gap is not simply state persistence: interrupted work may remain unfinished even when state survives, and resumption may repeat side effects.Neither framework nor platform states the obligation that arises when their documented behaviors are combined.Source: §III · p. 3
  • Each operational contract connects a framework requirement, a platform event that can violate it, and a deployment obligation that prevents the violation.Contracts are derived through source selection, candidate extraction, human validation and consolidation, and classification.Source: §III · p. 3
  • The contract derivation uses first-party documentation for LangGraph and for Docker, Docker Compose, and Kubernetes.Candidate pairs are validated against their supporting quotations, and candidates expressing the same obligation under the same event are consolidated.Source: §III · pp. 3–4
  • The resulting 14 contracts cover failures involving state loss or inconsistency, interrupted work, repeated external actions, unreleased resources, and broken client streams.Examples include a stop terminating work before draining completes and container replacement interrupting a client stream.Source: §III · p. 5
  • Only four contracts protect saved state; the other ten address obligations such as resuming interrupted work, harmless re-execution, worker availability, stream recovery, and resource release.Four contracts are high severity: OC1–OC3 for state loss and OC7 for repeated external actions.Source: §III · p. 5
  • More than half of the contracts are statically checkable, including all four saved-state contracts and three of the four high-severity contracts.OC7 requires runtime evidence about whether repeated side effects are harmless; OC4 is statically checkable only for LangGraph’s production server with its configured drain window.Source: §III · p. 5

IV. RQ2: DO DEPLOYED AGENT SYSTEMS MEET THE CONTRACTS?

The study assesses which operational contracts apply to deployed agent systems and whether repository evidence shows them holding. Contract violations are common across durable state, interrupted work, and shutdown obligations.

Applicability and verdicts

  • Repository verdicts classify contracts as not applicable, holding, broken, or not decidable; runtime evidence is required for OC7, OC8, OC11, and OC14.Detection rules combine mechanically extracted facts with author-assigned labels; OC4 is decidable only for LangGraph production-server deployments.Source: §IV · p. 5
  • 75.5% of the 433 systems have at least one applicable operational contract.The state contracts apply to 61.2% of systems; 19.9% run background work and 60.3% stream output.Source: §IV · pp. 5–6

Saved state

  • 37.4% of systems requiring durable state lose it on restart, most often because they lack a persistent store or fail to route state to one.Among 265 systems, 166 (62.6%) hold OC1; 77 of the 99 violations ship no persistent store, while 22 do not reach the store they ship.Source: §IV · p. 6
  • Only 50.6% of systems needing durable state have no further repository-establishable broken contract after the cumulative checks.State survives restart for 62.6% and also survives container replacement for 58.5%; the 50.6% figure is an upper bound because 85 of those systems have an applicable contract not decidable from repository evidence.Source: §IV · p. 6, Figure 3 · p. 6

Interrupted work

  • 88.5% of systems running background work provide no repository-evident mechanism to resume an interrupted run.Of 61 applicable systems, six use a queue that redelivers unacknowledged work and one resumes unfinished runs at start-up; 28 of the 54 without resumption mechanisms hold OC1.Source: §IV · p. 6

Pauses and shutdown

  • 41.1% of decidable systems that pause for human input break OC1, so the paused run does not outlive the process.Among eight assessable production-server deployments, all retain Docker’s 10-second stop grace period, 170 seconds shorter than LangGraph’s 180-second drain window.Source: §IV · p. 6

V. RQ3: DO THE VIOLATIONS BECOME OBSERVABLE FAILURES UNDER ORDINARY CONTAINER OPERATIONS?

The authors induce ordinary container-platform events on selected deployments and compare observed behavior with contract-based predictions. Most predicted violations become runtime failures, while every scorable prediction that a contract holds is confirmed.

Event selection and execution

  • The experiment covers seven Docker or Compose events on 12 selected systems, retaining shipped configurations except for model-routing changes.Events include restart, replacement, kill, stop with Docker’s default 10-second grace period, forced recreation, server-process crash, and cold start.Source: §V · p. 7

System selection

  • The team selects contracts with inducible, observable events, then uses contract coverage to choose runnable systems from an initial pool of 433.The final 12 systems collectively cover the required verdicts; each contract is exercised on only one to five systems.Source: §V · p. 7

Runtime checks

  • Checks compare preserved conversations, running-work progress, resumption, signal delivery, readiness, server restart, repeated actions, and stream recovery against prior predictions.Events are repeated three times; outcomes are reported only with at least two valid repetitions that agree.Source: §V · pp. 7–8

Runtime outcomes

  • 14 of 16 predicted violations (87.5%) manifest as runtime failures, and all seven scorable predictions that a contract holds pass.One additional hold prediction is inconclusive; the experiment confirms that satisfying one contract does not ensure another will hold.Source: §V · p. 8
  • Interrupted runs may retain completed progress yet remain unfinished when no mechanism resumes them; re-execution can repeat model calls or external actions.The deep-research agent’s interrupted stream also cannot resume or retrieve remaining output, leaving the client without the rest of the answer.Source: §V · p. 8
  • Two repository-predicted violations do not become runtime failures because Panaversity’s pulled image supplies database retries and persistent conversation storage.Without the image safeguard, the predicted replacement failure occurs in SRE-Agent.Source: §V · p. 8

VI. RQ4: HOW DO DEVELOPERS REPAIR CONTRACT VIOLATIONS?

The authors mine repository histories for developer repairs associated with deployment-platform failures, then classify the fixes and whether they meet the full contract. Repairs cover most contracts, usually change application code, and often leave deployment-dependent obligations only partly satisfied.

Mining and validation

  • The search returns 39,261 artifacts, screens 7,268 from 270 repositories, and yields 122 validated repairs from 63 repositories.Two authors independently validate 130 of 396 screened candidates before applying the resulting rules to the remainder.Source: §VI · pp. 8–9

Repair coverage and patterns

  • Developers’ 122 repairs address 12 of 14 operational contracts and recur as 35 distinct fix patterns.111 repairs address contract violations; the other 11 concern failures common to containerized services, and no repair addresses OC12 or OC13.Source: §VI · p. 9
  • Repairs most often address running work (62 of 111), followed by saved state (39) and clients or external resources (10).Resuming interrupted runs is the most common contract repair, with 23 repairs in 15 repositories.Source: §VI · p. 9

Where fixes are made

  • Among 122 repairs, 95 (78%) changed code only, 4 (3%) changed both code and deployment files, and 22 (18%) changed only deployment files; one had no inspectable diff.Of 22 deployment-only repairs, 16 (73%) used one of four fixes: mounting the database on a volume, running the application as PID 1, waiting for the database before setup, or sizing the memory limit.Source: §VI · p. 9

Repair completeness

  • Only 66% of repairs fully satisfy the deployment obligation; 34% do so only partially.Saved-state repairs are complete in 32 of 39 cases, while only 2 of 13 OC4 repairs provide enough time to drain work before termination.Source: §VI · p. 9
  • Completeness depends on deployment-platform support: 11 of 13 OC4 repairs initiate draining but retain the 10-second stop grace period.For OC8, 11 of 23 repairs are partial, including eight that report interrupted runs rather than resume them.Source: §VI · p. 9

VII. THREATS TO VALIDITY

The study qualifies its findings by limiting contract assessment to documented and repository-visible behavior and by bounding reported proportions to a defined deployment population.

VII. THREATS TO VALIDITY

  • Because contracts are documentation-bounded and RQ2 relies on repository evidence, behavior hidden inside deployed images can escape assessment; undecidable cases make violation rates lower bounds.RQ3 uses pre-event checks, validated events, and three repetitions to mitigate measurement error.Source: §VII · pp. 9–10
  • Manual labels and repair coding involve judgment; the authors mitigate this through independent coding, adjudication, codebook refinement, and reported agreement.Authors validate retained LLM-extracted candidates and estimate screening misses from a random sample of rejected artifacts.Source: §VII · p. 10
  • Reported proportions characterize LangGraph with Docker, Compose, and Kubernetes in public Python repositories with at least 10 stars, not other frameworks or platforms.RQ3 selects 12 systems to validate runtime manifestation rather than prevalence; RQ4 is limited by keyword retrieval and public repair histories.Source: §VII · p. 10

A. LLM-based agents and their failures.

Prior work examines challenges in agent development and failures within agent systems, while this study focuses on failures at the framework–deployment boundary.

A. LLM-based agents and their failures.

  • Studies of LLM application and agent development report recurring developer challenges, including deployment [6], [24].Other work measured agents in production and examined developer practices, adoption, configuration, and observability, [59]–[63].Source: A. LLM-based agents and their failures · p. 10
  • Agent-failure research examines failures in execution traces, agent bugs, error propagation, and accountability in role-specialized pipelines [22], [23],,.Comparable failure studies also preceded agent research in deep-learning software,.Source: A. LLM-based agents and their failures · p. 10
  • This study instead derives contracts from framework and platform documentation and tests whether ordinary deployment events violate requirements of otherwise correctly executing agents.The examined events include restart, replacement, and shutdown.Source: A. LLM-based agents and their failures · p. 10

B. Container deployment and operational correctness.

Prior container-deployment research assesses artifacts against practices that apply independently of the application; operational contracts target obligations created by the agent–environment interaction.

B. Container deployment and operational correctness.

  • Container-deployment studies characterize Docker usage, refactorings, technical debt, and misconfigurations in Dockerfiles, Compose files, and Kubernetes manifests.These studies assess artifacts against practices independent of the application they deploy.Source: B. Container deployment and… · p. 10
  • The closest cited study examined 1,993 Dockerfiles from machine-learning projects, characterizing their roles and build inefficiencies [4].Source: B. Container deployment and… · p. 10
  • Operational contracts instead capture obligations arising from the interaction between agent code and its deployment environment.Source: B. Container deployment and… · p. 10

IX. CONCLUSION

The conclusion frames agent–platform obligations as explicit operational contracts and reports that they are widespread, often violated, and frequently repaired in application code.

IX. CONCLUSION

  • The study introduces 14 operational contracts to make deployment obligations explicit for LangGraph with Docker, Compose, and Kubernetes.The obligations arise because platforms routinely restart, replace, and terminate containers while agent systems carry saved state and running work across those events.Source: §IX · pp. 10–11
  • 75.5% of 433 systems have at least one applicable contract; among systems needing durable state, 37.4% lose it on restart and 41.5% on container replacement.Source: §IX · p. 11
  • Induced events produced runtime failures for 14 of 16 predicted violations, while all seven scorable predictions that a contract holds were confirmed.Source: §IX · p. 11
  • Developer repairs cover twelve of fourteen contracts through 35 fix patterns; 78% modify application code, and only two thirds fully satisfy the deployment obligation.Source: §IX · p. 11
Loading 2610.34620v1…