PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Let the Model Write the YAML: Deterministic, Minimal-Diff GitOps Remediation from LLM-Proposed Field Changes

Pruthvi Davineni

arXiv:2609.00227v1cs.SEcs.AIcs.DC

TL;DR

In GitOps, model-authored files and diffs are an unsafe way to apply infrastructure fixes, even when tolerant patching accepts most diffs. The paper instead pairs a model-proposed structured field-change intent with deterministic span-editing. On real manifests, this approach was correct and consistent across runs; its guarantee covers faithful application, not whether the proposed fix is right.

  • Problem

    GitOps remediation needs reviewable manifest edits, but existing tooling either changes the live cluster imperatively or relies on model-authored files or diffs that are unreliable.

    Source: §2 · p. 2

  • Method

    The model emits a structured field-change intent, and deterministic code locates the target scalar’s YAML span and replaces it in raw text.

    Source: §4.1 · p. 2, §4.3 · p. 2

  • Results

    Span-editing was 100% correct and identical across both models and all 415 runs, while GNU patch silently misapplied 14–20% of applied diffs.

    Source: §1 · p. 1

  • Takeaways & Limitations

    For unattended GitOps edits, deterministic span-editing provides correctness, determinism, and minimality when faithfully applying a known change is the goal.

    Source: §7 · p. 6

    The guarantee concerns faithful application, not whether the model chose the right intent; human pull-request review addresses that question.

    Source: §7 · p. 6

Abstract

from arXiv · show

LLM agents increasingly diagnose incidents and propose remediations. In a GitOps workflow, applying a fix means editing a version-controlled config file, and the obvious implementation, having the model author the edited file or a diff, is what practitioners reach for first. Evaluating that choice on real Kubernetes manifests, we find no text-generation strategy is safe for unattended automation. Unified diffs are unsafe: under strict patching almost none apply, but that is an artifact, since a tolerant tool (GNU patch) applies 96%, yet silently misapplies about 1 in 7 (14-20%) with no error signal. Full-file rewrite is capability-dependent: a small model corrupts the file, while a frontier model is usually correct but non-deterministic (it silently drops a field or edits a neighbor on some runs) and must regenerate the whole file, costing O(file size) per edit. We present an alternative that separates the semantic decision (which resource, field, and value) from the syntactic act of editing the file. The agent emits only a structured field-change intent; a deterministic pipeline indexes manifests by (kind, name), locates the target scalar's exact character span via the YAML parser's node position marks, and replaces only that span in the raw text. Because the file is never re-serialized, the diff is minimal by construction, formatting and comments are preserved, and the edit is correct and deterministic independent of the model, at O(1) generation cost. The contribution is the pairing of an LLM-proposed intent with a deterministic, fail-closed application contract for GitOps. We implement it in KubeAstra (Apache-2.0) and release the benchmark. Our claim is scoped to faithful application of a known change; whether the change is right is left to human PR review.

1. Introduction

The paper argues that model-authored files and diffs are unsafe for unattended GitOps remediation, and proposes separating the model’s semantic intent from deterministic file editing. Its benchmark finds deterministic span-editing avoids the silent errors and variability of text-generation strategies, while the guarantee is limited to faithfully applying a known change.

1. Introduction

  • The evaluation isolates faithful application of a known, correct semantic change from the separate question of whether the proposed fix is right.The paper evaluates real manifests using text-generation strategies and a deterministic approach across a small and a frontier model.Source: §1 · p. 1
  • Span-editing was 100% correct, with zero collateral changes and identical results across both models and all 415 runs; tolerant GNU patch silently misapplied 14–20% of applied diffs.Strict application accepted few diffs, but GNU patch applied 96%, showing that rejection rates alone understate the risk.Source: §1 · p. 1

Contributions:

  • The proposed design pairs structured intent with deterministic application, replacing only the target scalar while preserving comments, quoting, and key order.The paper identifies the pairing as its design principle and node-mark span-editing as its concrete mechanism.Source: Contributions · p. 2

2. Background & Motivation

  • YAML re-serialization can reflow, re-quote, reorder, and strip comments, so round-tripping through an object serializer loses file fidelity.In GitOps, applying a fix means editing a manifest and opening a pull request; the remediation step moves an agent from advising to editing production source.Source: §1 · p. 1, §2 · p. 2
  • Existing tooling either changes the live cluster outside GitOps or lets the model author files or diffs, leaving no reviewable, minimal, guaranteed-faithful GitOps change.GitOps makes the repository the source of truth, with changes flowing through reviewed pull requests.Source: §1 · p. 1, §2 · p. 2

3. Problem Statement & Design Goals

The design treats the model as an untrusted source of file bytes: it supplies a structured field-change intent, while deterministic code locates and edits the target scalar. The contract preserves unrelated bytes and refuses ambiguous or absent targets, with a Kustomize fallback for a generated resource lacking a checked-in source line.

3. Problem Statement & Design Goals

  • The system targets scalar changes in Deployment- or StatefulSet-style resources in plain-YAML and Kustomize repositories, with minimal-diff, fidelity, determinism, fail-closed, and reviewability goals.The agent is not trusted to write file bytes or merge; a human reviews and merges the pull request.Source: §3 · p. 2

4.1 The field-change intent (the interface — C1):

  • The model emits FieldChange with resource identity, field_path, new value, and reason; named list segments identify containers and environment variables by name, not index.The model’s contribution is choosing which scalar and what value.Source: §4.1 · p. 2

4.2 Repository indexing without git or disk:

  • Repository indexing fetches and extracts a tarball in memory, parses YAML documents, and indexes each document by (kind, name), with namespace used to disambiguate.Unparseable files are skipped; the approach uses no git binary, disk token, or manifest cache.Source: §4.2 · p. 2

4.3 Node-mark span location (the mechanism — C2):

  • The locator walks parser-marked YAML nodes along field_path, resolves named-list items by name, and returns the terminal scalar’s character span.Zero matches or an absent field cause refusal; multiple matches prefer the environment overlay or otherwise refuse and list candidates.Source: §4.3 · p. 2

4.4 Span replacement (why the guarantees hold — C1):

  • Replacing only the located span in raw text preserves bytes outside it, making the edit deterministic and minimal while preserving comments, quoting, and key order.For in-scope scalar fields, the span lies within one line, so the diff touches exactly that line; the guarantee does not establish that the intent is semantically correct.Source: §4.4 · pp. 2–3, Figure 1 · p. 3

4.5 Kustomize patch-append fallback:

  • For a generated resource with no checked-in source line in a known Kustomize overlay, the fallback appends a strategic-merge patch and kustomization entry; preview and open separate review from pushing a PR.The protocol has a no-merge invariant: a CI test enforces that the human always merges.Source: §4.5 · p. 3, §4.6 · p. 3

5. Implementation

The section implements a deterministic span-edit pipeline and evaluates it against model-authored rewrites and diffs on real manifests. Results distinguish reliable, model-independent editing from the capability, determinism, cost, and safety limitations of text-generation strategies.

5. Implementation

  • KubeAstra’s GitOps component spans about 520 lines of Python, with roughly 150 lines in the safety-critical locator and span editor, and 36 dedicated tests.It uses no new native dependencies and runs in server and desktop modes.Source: §5 · p. 3

6.2 Corpus:

  • Evaluation covers 83 field-change tasks from two Apache-2.0 sources; about 80 edit a single multi-resource file, so reported rates are edit-site rates within a small file set, not population estimates.The study assesses exact-change correctness, collateral changes or fabrication, and preservation of comments and formatting with minimal diffs.Source: §6.2 · p. 3, §6.1 · p. 3

6.4 Results:

  • Span-edit was correct across all five seeds for all 83 tasks, while frontier full-file rewriting was flaky on 6/83 tasks and its quality varied by model capability.The full-file correctness rate changed from 0.024 to 0.976; span-edit remained 1.000 regardless of model.Source: §6.4 · p. 4, §6.7 · p. 5

6.5 A note on methodology (max tokens):

  • An early 4,000-token cap truncated the frontier model’s approximately 9.3K-token full-file output; reported B1 results use a cap above the largest file.Diff-based systems were unaffected because their outputs were small.Source: §6.5 · p. 4

6.6 Cost (RQ, generation tokens per edit):

  • Span-editing has fixed, small generation cost, while the most reliable text strategy regenerates the whole file and therefore costs O(file size) per edit.The reported $34.69 Sonnet run is secondary to the scaling contrast; application itself does not invoke the model.Source: §6.6 · p. 4, §6.8 · p. 6

6.7 Strict vs. lenient diff application — the diff story revised:

  • 14.0% of diffs applied by patch --fuzz were wrong, rising to 20.2% when whitespace was ignored; stricter appliers avoided some misapplications by refusing more diffs.Content-based hunk matching raised correctness from 2.7% to 67.5% with zero misapplication, while GNU patch applied 96.4%.Source: §6.7 · pp. 4–5

6.4 Results:

  • 415/415 span-edit runs were correct, versus 405/415 for frontier full-file rewrite; strict diffs rarely applied, and small-model rewrites reflowed files.Across field types, SUT remained at 1.00; span-edit correctness is independent of model capability.Source: §6.4 · p. 4, Table 1 · p. 5, Figure 2 · p. 5

6.8 Fail-closed refusal (G5), measured:

  • The locator refused absent, ambiguous, and unresolvable targets with 1.00 precision, but a YAML alias resolved to its anchor and produced a dangling alias.Alias handling remains a known gap; detecting anchor/alias nodes and refusing is future work.Source: §6.8 · p. 5

7. Discussion & Limitations

The paper argues that deterministic span editing is preferable to text generation for unattended GitOps changes, while limiting its guarantees to faithfully applying an intent rather than deciding whether the fix is right. Its v1 scope and evaluation are bounded by supported edit types, repository formats, and a small, concentrated corpus.

7. Discussion & Limitations

  • Diff authoring is unreliable at every tier, while full-file authoring is non-deterministic and unbounded-cost; deterministic span editing targets automation’s needs.The guarantees concern faithful application, not whether the proposed change is correct; human PR review remains responsible for that question.Source: §7 · p. 6
  • V1 handles scalar changes to existing fields; structural edits such as adding a container or block remain future work.Its stated scope covers Deployment/StatefulSet-style resources in plain-YAML and Kustomize repositories.Source: §7 · p. 6
  • Helm/Argo value indirection is out of scope: when a live Deployment is generated from a separate values file, v1 refuses and a resolver remains future work.Source: §7 · p. 6
  • The application path is GitHub-only, and the locator skips templated files because it requires a parseable file.Source: §7 · p. 6
  • About 80 of 83 tasks come from one multi-resource file, so reported rates are edit-site rates in a small file set, not population estimates.The single-file-dependent flaky-6 determinism result is indicative pending confirmation on the planned multirepo corpus.Source: §7 · p. 6

Threats to validity (the ones a reviewer will raise first):

The threats discussion measures the diff-applier spectrum, audits the correctness oracle, and identifies limits in refusal testing and model coverage. It also reports an alias-handling gap in the fail-closed contract.

Threats to validity (the ones a reviewer will raise first):

  • GNU patch applied 96.4% of Sonnet diffs, but silently misapplied 14–20% of accepted diffs; strict application yielded 2.7% correct and offset-tolerance 67.5%.The authors exclude git apply because it is stricter for the headerless, no-context hunks emitted by the models.Source: Threats to validity (the ones a… · p. 6
  • On six adversarial categories, fail-closed precision was 1.00 and recall 0.889; the known leak is YAML aliases, where the locator edits the anchor line instead of refusing.The main 83-task corpus never needed refusal, so broad-corpus refusal coverage remains future work.Source: Threats to validity (the ones a… · pp. 6–7
  • The oracle matched independent adjudication on all 30 audited cases, including every misapplied case underlying the 14–20% finding.The sample comprised 10 SUT edits, 10 GNU patch --fuzz outputs called correct, and 10 called misapplied.Source: Threats to validity (the ones a… · p. 7
  • Capability dependence is inferred from only two models, and the six flaky tasks have not yet been characterized.A third model would be needed to establish monotonicity; the flaky tasks also require further investigation.Source: Threats to validity (the ones a… · p. 7

8. Related Work

The paper situates its work among LLM agents, structured editing, automated repair, cloud operations, and GitOps foundations. Its distinction is to apply an LLM-specified change faithfully rather than generate a patch or stop at mitigation recommendations.

8. Related Work

  • ReAct-style agents [1] and SWE-bench [2] exemplify LLM reasoning-and-acting and code-editing evaluation; this work removes the model from byte-level editing for scalar config changes.Source: §8 · p. 7
  • Coccinelle, tree-sitter, and Semgrep provide structured editing precedents; this work adds an LLM-intent and fail-closed GitOps contract with minimal-diff and no-fabrication guarantees.Source: §8 · p. 7
  • Unlike automated program repair, which generates candidate patches and validates them against tests, this approach applies a specified change and leaves correctness of intent to the model.Source: §8 · p. 7
  • Cloud-operations systems use LLMs for root-cause analysis and mitigation recommendations [5] [6] [9], but stop short of guaranteed-faithful GitOps edits; this work addresses that application step.Source: §8 · p. 7
  • OpenGitOps and Argo CD and Flux establish Git as the source of truth and the PR as the change unit targeted here.Source: §8 · p. 7

Back matter

The authors release the benchmark and span-edit implementation under Apache-2.0. They state an impact framing of human review and no auto-merge to reduce autonomous production-change risk.

Back matter

  • The benchmark, including corpus, oracle, harness, and result data, and the KubeAstra span-edit implementation are released under Apache-2.0.Source: Back matter · p. 7
  • The stated impact approach retains human review and no auto-merge to reduce the risk of autonomous production change.Source: Back matter · p. 7

Appendix A: Baseline prompts

The baselines receive each change as natural language with a worked example, making the comparison fair rather than a straw-man test. The appendix specifies prompts for full-file and unified-diff editing, including a retry variant.

Appendix A: Baseline prompts

  • All baselines receive the requested change as natural language plus a worked one-shot example.The change description is generated from the structured intent.Source: Appendix A · p. 7
  • The full-file prompt instructs the model to change only the named field while preserving comments, formatting, quoting style, and key order.It also prohibits touching any other field.Source: Appendix A · p. 7
  • B1 asks the model to return the complete edited manifest, whereas B2 asks for a unified diff that applies cleanly.The manifest is placed at the end of the system message for B2; the shared prefix supports prompt caching across tasks and seeds.Source: Appendix A · p. 8
  • B3 retries a rejected diff by appending the apply or parse error and requesting a corrected diff, up to two times.The retry prompt asks for a clean diff.Source: Appendix A · p. 8

Appendix B: Locator pseudocode

The locator uses YAML parser node positions to find a target scalar’s exact character span, returning a refusal when the requested path cannot be located safely. It handles mapping keys and uniquely named list items while failing closed on invalid targets.

Appendix B: Locator pseudocode

  • The locator composes YAML documents into node trees with position marks, then returns the target scalar’s [start, end) character span or None.The pipeline applies the change by replacing only that raw-text span.Source: Appendix B · p. 8
  • For mapping nodes it follows the matching key; for sequence nodes it finds the item by its name field rather than by index.This supports named lists such as containers and environment variables.Source: Appendix B · p. 8
  • The locator refuses out-of-range documents, missing keys or named items, paths descending into scalars, and targets that are maps or lists rather than scalars.Duplicate (kind, name) ambiguity is caught upstream in the index.Source: Appendix B · p. 8

Appendix C: Oracle (correctness checker)

The oracle checks whether an output parses, changes the intended target to the intended value and type, and leaves all other leaves unchanged. It separately measures comment and formatting preservation as fidelity signals rather than correctness criteria.

Appendix C: Oracle (correctness checker)

  • An output is correct only if it parses as YAML, sets the intended target value, and adds, removes, or changes no other leaf.The target document is located by (kind, name) at the same index in both files.Source: Appendix C · p. 8
  • The oracle compares values in their intended parsed types, so a string/integer mismatch counts as a real difference.Lists of uniquely named mappings are flattened by name, not index.Source: Appendix C · p. 8
  • Comment preservation and format preservation are secondary fidelity signals and do not affect correctness.A reflowed but structurally correct output passes correctness while failing fidelity; injected collateral and dropped-field outputs are flagged.Source: Appendix C · p. 8
Loading 2609.00227v1…