PaperlayerView exact revision on arXiv

Source-linked AI summary

A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

Hasan Najib Mahmud, Shreya Gupta, Isha Chaudhary, Nathaniel Enis, Ravi Mangal, Gagandeep Singh, Corina Pasareanu

arXiv:2608.18389v1cs.AI

TL;DR

The paper asks whether repository-level code agents remain reliable when code is rewritten into semantically equivalent forms. It evaluates agents across randomized semantics-preserving variants and finds small but configuration-dependent degradation, revealing a jagged robustness frontier across models and scaffolds.

  • Problem

    Evidence is limited on whether repository-level coding agents remain reliable under superficial, semantics-preserving code changes, despite deployment and benchmark use.

  • Method

    The study evaluates two agentic scaffolds backed by four models on 54 SWE-bench instances using randomized repository transformations and paired unperturbed-versus-perturbed runs.

  • Results

    Resolve rates decline in most configurations, with up to 6.7 percentage points degradation and statistically significant effects in 6 of 16 configurations.

  • Takeaways & Limitations

    Robustness rankings vary across scaffolds and benchmarks, while the simpler mini-SWE agent is consistently more robust to perturbation.

  • Takeaways & Limitations

    Semantics-preservation is operationalized as matching per-test outcomes across the project’s test suite.

Abstract

from arXiv · show

AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds - Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode - revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases.

Introduction

The paper examines whether repository-level code agents remain reliable when code is changed through semantics-preserving transformations, addressing concerns that benchmark performance may overstate deployment reliability. It evaluates paired unperturbed and perturbed repositories across agent scaffolds, models, and SWE-bench datasets, finding generally small but configuration-dependent robustness losses and increased effort.

  • Motivation: AI coding tools are mainstream, but benchmark scores may miss susceptibility to superficial code perturbations caused by shortcut learning.This motivates testing whether agent behavior changes when surrounding code is altered without changing semantics.
  • Method: The study compares unperturbed repositories with semantically equivalent variants generated by a randomized sampler of local semantics-preserving transformations.The transformations mimic routine refactorings or stress-test agents.
  • Findings: 6 of 16 configurations show statistically significant resolve-rate degradations, with drops reaching up to 6.7 percentage points in the most affected configurations.The paired methodology is designed to separate perturbation effects from the intrinsic stochasticity of LLM-based agents.
  • Evaluation: Two agentic scaffolds, four frontier models, and 54 instances from SWE-bench Verified and SWE-bench Pro define the evaluation setting.The scaffolds are mini-SWE agent and OpenCode; the models are Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B.
  • Findings: Perturbations increase agent effort, raising step counts by up to 9.9% and token cost by up to 22.9% even when resolve rate changes little.These effects show that robustness involves more than whether the issue is ultimately resolved.

Semantics-Preserving Transformations

The study defines semantics-preserving transformations as rewrites that retain observable behavior and operationalizes them through per-test equivalence. It implements a catalog of 14 such transformations, validates them empirically, and applies sequences of transformations to whole repositories for agent evaluation.

  • Definition: A semantics-preserving transformation produces the same return value or exception and externally observable effects as the original program on every input.If one program halts, the other must also halt.
  • Definition: The evaluation operationalizes semantic preservation as identical per-test outcomes between the original and transformed programs across a project’s test suite.
  • Transformation catalog: The implemented catalog contains 14 SPTs, combining routine refactoring-like rewrites with behaviorally inert or unnatural code injections.Examples include reordering commutative operands, swapping if/else branches, splitting string literals, and injecting dead code or methods.
  • Transformation catalog: Each SPT pairs a structural pattern for identifying candidate sites with a rewrite rule; Dead String Assignment and Dead Method Injection additionally require a target keyword.The former injects an unread string assignment, while the latter appends a dead method named after the supplied keyword.
  • Validation: Validation uses differential testing on test suites from SymPy, sqlfluff, and xarray, with each transformation applied in isolation.The projects span symbolic mathematics, SQL linting, and labeled N-dimensional arrays, respectively.
  • Repository perturbation: Because agent repair tasks require repository-level perturbations, the method applies finite sequences of SPTs to source code rather than transforming only individual file locations.

Sampling Semantics-Preserving Variants

The sampler generates semantically equivalent repository variants by randomly selecting transformations, files, and candidate sites while always including files modified by the gold patch. Five hyperparameters control variant count, transformation count, keyword targeting, rewrite fraction, and file selection limits.

  • Variant generation: The sampler applies randomly selected semantics-preserving transformations at randomly chosen sites in randomly chosen files, producing independent variant task instances.Variants are repositories semantically equivalent to the seed and are consumed by the agent evaluation loop.
  • File and site selection: Gold-patch files are always included, ensuring every variant targets the solution-relevant region rather than selecting it by chance.Other non-test files are included probabilistically for each transformation.
  • File and site selection: Within each included file, the sampler uniformly selects a fraction ϕ of candidate sites for rewriting and binds keyword-bound transformations to up to Nk extracted targets.Keyword targets are extracted from the issue description with a single LLM call and can be cached.
  • Variant diversity: Resampling pfile independently for each transformation yields populations ranging from a single localized edit to pervasive repository-wide perturbation.This variation arises because different transformations can reach different repository portions.
  • Sampler controls: Five hyperparameters govern sampling: N variants, Nt transformations, Nk keyword targets, ϕ rewritten sites, and Nf files per keyword-bound transformation.Nf is a practical necessity because agents frequently interact with keyword-bound transformations across many files.

Experiments

The experiments test how semantics-preserving transformations affect agent outcomes, effort, and failure modes across two scaffolds, four models, and two repository-repair benchmarks. Perturbations usually reduce resolve rates, more broadly increase effort, and produce distinct localization, editing, and validation failures.

  • Outcome robustness: 13 of 16 configurations show positive mean degradation, while 6 of 16 exclude zero in their 95% intervals.No scaffold-model pair is significant on both benchmarks, indicating configuration-dependent effects.
  • Outcome robustness: 1.34 versus 1.88 points is mini-SWE agent’s degradation against OpenCode on SWE-bench Verified; 1.88 versus 3.65 points on SWE-bench Pro.Mini-SWE agent remains more robust despite reversing the absolute resolve-rate advantage across benchmarks.
  • Effort: 22.9% is the largest SWE-bench Verified cost increase, compared with a maximum step overhead of 9.9%.Cost rose in all eight Verified configurations, and cost per step increased in twelve of sixteen configurations overall.
  • Effort: −19.8% is Opus’s mean step change under mini-SWE agent on Verified, while output tokens per step more than double and cost rises +30.4%.This trajectory is uniform across instances: fewer, denser actions replace longer trajectories.
  • Failure patterns: Agents exhibit four observed failure patterns: detecting and reverting transformations, degraded code localization, editing perturbed code, and corrupted patch validation.Examples include noisy dead strings misleading grep, unnecessary simplification of transformed code, and misattribution of a test failure to a split string literal.

Related Work

Prior work shows that semantics-preserving surface perturbations can undermine code models, general LLMs, and structured problem solving. This work extends that literature from fixed-input, single-turn tasks to repository-level, multi-turn coding agents and highlights scaffold-dependent robustness rankings.

  • Robustness of code language models: Code models are brittle to identifier renaming, dead-code insertion, operator substitutions, and other semantics-preserving perturbations.Prior studies examine attacks, robustness constraints, and black-box settings for pretrained code models.
  • Robustness of code language models: Contrastive learning on equivalent programs and naturalizing transformations as pretraining objectives improve code-model robustness.
  • Robustness of code language models: Existing code benchmarks and attack frameworks perturb docstrings, signatures, syntax, or prompt-context refactorings to evaluate short-generation, classification, and code-completion behavior.Minor refactorings in prompt context have been shown to significantly alter Copilot’s completions.
  • From fixed inputs to repository-level agents: Prior work evaluates single-turn inferences on fixed-length inputs, whereas this study perturbs entire codebases used by multi-turn agents that localize faults, navigate repositories, and synthesize patches.The perturbations are applied randomly over the codebase.
  • Adversarial robustness of LLMs: General LLMs and structured reasoning systems also shift under semantics-preserving changes, including up to 76-point accuracy swings, up to 10% performance drops, and altered model rankings.Reported variations include prompt formatting, character-, word-, and sentence-level attacks, changed names, numbers, phrasing, and reordered options.
  • Scaffold-dependent robustness: In the code domain, robustness rankings can change when the same model is embedded in a different agent architecture, adding scaffold interaction as a dimension absent from prior work.Related certification work has studied agentic robustness for simple, non-code tool selection.

Conclusion

The work contributes a library and randomized sampler for generating semantically equivalent repository variants, alongside a methodology for isolating perturbation effects from agent stochasticity. It also presents itself as, to the authors’ knowledge, the first systematic [claim incomplete in supplied passage].

  • The work contributes local, semantics-preserving transformations for code repositories and a randomized sampler that generates semantically equivalent variants without agent feedback.The sampler draws repository variants without using feedback from the agent.
  • The experimental methodology isolates perturbation effects from the intrinsic stochasticity of LLM-based agents through paired seed-and-variant runs.The methodology balances statistical validity against the cost of agentic runs.
  • The authors describe the work as, to their knowledge, the first systematic [claim incomplete in supplied passage].

Generative AI Usage Statement

Generative AI tools supported manuscript preparation by improving writing clarity, assisting with experimental Python scripts, and developing arguments in Section E. The authors reviewed everything carefully and remain fully accountable.

  • Generative AI tools were used to improve writing clarity, assist with Python scripts for experiments, and develop arguments in Section E.
  • The authors carefully reviewed all AI-assisted work and remain fully accountable for the manuscript.

A Semantics-preserving Transformation Implementation Details · Rewrites · Inert Insertions

The sampler implements semantics-preserving transformations as guarded rewrites and inert insertions, with applicability conditions designed to remain safe across diverse repositories. Rewrites alter existing structure, while inert code creates behaviorally neutral but sometimes lexically salient decoys.

  • A Semantics-preserving Transformation Implementation Details: Applicability conditions must exclude every unsafe variant because repository-level name resolution, types, introspection, and class namespaces can expose transformations.Builtins may be shadowed, operations may be type-dependent, and programs may observe locals, frames, metaclasses, or attributes.
  • A Semantics-preserving Transformation Implementation Details: The transformations are organized into rewrites of existing structure and insertions of inert code.
  • Rewrites: Rewrites preserve control flow through conditional inversion, conjunction splitting, comparison swapping, guarded negation, and one-iteration while-loop unrolling.Each rewrite restricts candidate forms to avoid semantic differences, including chained comparisons, literal conditions, top-level negations, and loops with else clauses.
  • Rewrites: For-loop rewriting uses explicit iter/next handling, randomized collision-resistant names, and a sentinel to emulate for ... else behavior.The transformation also tracks breaks belonging to the rewritten loop so the original loop contract is retained.
  • Rewrites: Other rewrites permute commutative operands only after type inference, rename narrowly selected local variables, and split eligible string literals into concatenations.These restrictions avoid noncommutative Python behavior, binding changes, aliasing hazards, and escape or annotation semantics.
  • Inert Insertions: Dead assignments and methods act as lexically salient edit decoys, but new bindings and methods are skipped when introspection or observable class namespaces could reveal them.Dead methods remain callable but have unreachable bodies and return None; namespace-sensitive contexts include locals(), eval/exec, and frame inspection.
  • Inert Insertions: Inert insertions include tautological guards, exception re-raising wrappers, unreachable assignments, unread keyword-string assignments, and dead methods.The first wrappers target 5-to-100-line blocks, while dead-code placement preserves leading docstrings.

B Sampler Procedure Details

Algorithm 2 specifies the sub-procedure invoked by the sampler in Algorithm 1.

  • Algorithm 2 describes the sub-procedure called by the sampler described in Algorithm 1.

C Experimental Procedure Details … Computational Infrastructure

The study uses stratified, solvable task sampling and paired baseline-versus-perturbed agent evaluations, executed in isolated containers on a high-end server. Its infrastructure combines local and Bedrock model serving with fixed temperature and internally consistent token-cost accounting.

  • C Experimental Procedure Details: Claude Opus 4.5-resolved SWE-bench Verified instances were stratified by repository–difficulty combinations, with one instance sampled per stratum.This ensured balanced coverage while selecting tasks solvable under standard, unperturbed conditions.
  • Sampling Procedure: 26 passing SWE-bench Pro instances were sampled across repositories containing at least one valid Python file suitable for SPT application.Because official instance-wise results were unavailable, equal numbers were initially sampled from each repository and evaluated with Claude Opus 4.5.
  • Full Experimental Procedure: Each instance followed four phases: seed retrieval, N-run unperturbed baseline evaluation, variant generation, and one-run-per-variant perturbed evaluation.Every perturbed run used a freshly provisioned environment.
  • Computational Infrastructure: Experiments ran on one server with 128 cores, 256 threads, 768 GB RAM, six 96-GB GPUs, and a 7 TB NVMe drive.The system used Ubuntu 24.04.4 LTS, and each agent run executed in its own Docker 29.5.2 container.
  • Computational Infrastructure: Qwen 3.6-27B was served locally with vLLM v0.20.1 on two GPUs, while Claude Opus 4.5, Kimi K2.5, and Minimax M2.5 used Amazon Bedrock.All four models used temperature 1.0; experiments used Python 3.10 and specified versions of LibCST, Astroid, and NLTK.
  • Computational Infrastructure: SPTs used LibCST v1.8.5, type inference used Astroid v3.3.0, and WordNet lookups used NLTK v3.6.0.The evaluated scaffolds included mini-SWE agent v1.17.5 and OpenCode.
  • Computational Infrastructure: δcost provides an internally consistent within-instance comparison because identical pricing schedules were applied to unperturbed and perturbed runs.Bedrock models used provider per-token prices, while locally served Qwen 3.6-27B used OpenRouter rates for the same model.

Randomness and Reproducibility … Selection of hyperparameter values

The study is stochastic in both variant generation and model execution, so reproducing its reported effects requires fresh sampling under fixed hyperparameters. Because each configuration is costly, the authors fix sampler hyperparameters rather than search their space, while releasing the implementation and analysis materials publicly.

  • Randomness and Reproducibility: The sampler randomly selects transformations, file-inclusion probability, file sites, and target keywords, while temperature-1.0 model sampling makes runs stochastic even on fixed repositories.A random seed would determine the 20 variants but not the agent’s outcomes on them.
  • Randomness and Reproducibility: The experiment runs baseline evaluations before generating variants and evaluates agent-produced patches through the stated robustness-experiment pipeline.The procedure provisions an unperturbed environment for repeated baseline runs, then generates variants for evaluation.
  • Randomness and Reproducibility: Reproducing the results requires drawing 20 fresh variants per instance at the same hyperparameters and recovering the reported effects.The bootstrap statistical protocol is the exception because it is seeded.
  • Code and Instance Lists: The authors publicly release code and data containing the experiment implementation, SPTs, variant sampler, evaluation harness, analysis scripts, dashboard, prompts, and instance lists.The repository is available at the URL specified in the paper.
  • Agent Scaffold Comparison: Table 2 compares the mini-SWE agent and OpenCode scaffolds.The supplied passage identifies the comparison but does not report its numerical results.
  • Selection of hyperparameter values: 2,160 agent runs are required for a single sampler configuration, calculated as 54 × 40, making even a coarse hyperparameter grid roughly an order-of-magnitude cost increase.The study therefore does not search the sampler’s hyperparameter space and instead fixes each parameter.
  • Selection of hyperparameter values: Nt = 3 of 14 transformations makes each variant a composition of several semantics-preserving transformations.The passage also specifies ϕ = 0.7, which leaves a minority of candidate sites in a file untouched.

D Additional Experiment Results … Variance, and why K = 1 is the efficient allocation

Across additional experiments, perturbations usually reduce resolve rates and increase effort, while degradation concentrates in a small number of instances. The estimator is unbiased for any repetition count, but under a fixed run budget, one run per distinct variant minimizes variance.

  • D Additional Experiment Results: 13 of 16 scaffold–model–benchmark combinations resolve fewer issues after perturbation; the other three change by less than one point.Capability rankings remain largely stable despite the resolve-rate declines.
  • Effort Overhead over All Runs: 25.4% is the largest cost increase on SWE-bench Verified, while SWE-bench Pro cost rises by up to 14.7%.Verified cost rises in all eight configurations, whereas Pro cost rises in seven of eight.
  • Per-Instance Degradation Plots: 7 of 224 Verified cells and 7 of 208 Pro cells have intervals excluding zero; point estimates are positive in 65 versus 35 negative Verified cells.With 20 runs per condition, even perfect resolution in both conditions has a ±16-point Newcombe interval.
  • Per-Instance Degradation Plots: 50 points is the largest reported Pro-instance loss, while a seaborn instance improves by 55 points under OpenCode with MiniMax.The largest losses concentrate in a small tail, and the single seaborn instance should not be treated as a repository-level effect.
  • E Unbiasedness of the Degradation Estimator: N = 20 variants are sampled, and one agent run per variant is sufficient because per-instance degradation remains unbiased for any number of runs per variant.The estimator targets mean resolve probability across semantically equivalent variants while separating variant difficulty from agent randomness.
  • The estimator: At K = 1, the per-variant rate is an ordinary binomial proportion, so perturbed resolve rates use the standard binomial estimator.Each run records a resolve/no-resolve outcome, and the estimator averages the resulting per-variant rates.
  • Why it is unbiased for any K, including K = 1: K = 1 is exactly as unbiased as larger K because repetition only reduces variance around an already correct per-variant center.This follows by conditioning on each variant, averaging over variants, and applying linearity of expectation.
  • Variance, and why K = 1 is the efficient allocation: K = 1 minimizes estimator variance whenever σ2 > 0, allocating every run to a new variant rather than repeating variants.At K = 1, the estimator is a standard binomial proportion with no variant-level clustering correction.

F Qualitative Analysis Details … Additional Qualitative Insights

The qualitative analysis combines targeted instance selection with LLM-assisted trajectory review and human evaluation. It finds variable SPT detection and response, prompt-dependent jaggedness, and uneven validation robustness across agents and repositories.

  • Instance Selection Procedure: 12/20 instances were manually selected for notable variance, degradation, step count, file coverage, cost, or mutant failures, while 8/20 came from agent-failed mutants.The manual criteria included high variance (3), high degradation (3), large step-count increase (1), file coverage (2), cost (1), and 1/20 mutant failures (2).
  • Instance Selection Procedure: An LLM prompt reviewed 115 failing trajectories using SPT logs, condensed trajectories, touched-file intersections, and SPT-marked snippets before shortlisting 8 trajectories for human evaluation.The prompt predicted possible SPT impact from applied transformations, overlapping files, and snippets from agent thoughts or tool outputs.
  • AI-Assisted Trajectory Analysis: The same prompt extracted trajectory events from raw snippets and classified them as Localization, Debugging, Planning, Patching, Validation, Recovery, or General.These categories describe repository exploration, output-based reasoning, patch planning, implementation, testing, backtracking, and additional events.
  • Additional Qualitative Insights: SPT detection varies substantially, including repeated failures to acknowledge transformations even when agents repeatedly access mutated code.Double negation can be missed, while even previously detected transformations such as if-true wrappers may be ignored, as in Opus on Psf-Requests with mini-SWE agent.
  • Additional Qualitative Insights: Agents respond inconsistently after detecting SPTs: Qwen on Matplotlib with OpenCode claimed repeated repository resets without performing them, while MiniMax avoided transformed core files.MiniMax instead sought information elsewhere in the repository, increasing localization time and file coverage.
  • Additional Qualitative Insights: Issue-description quality and issue nature may affect jaggedness, as conflicting NodeBB instructions about breaking and not modifying unit tests produced high variance across agents.The analysis compared Qwen, Kimi, and Opus on the same NodeBB issue across model and scaffold pairs.
  • Additional Qualitative Insights: Validation robustness differs across repositories: Kimi used a three-level Astropy test suite, whereas Pytest and Qutebrowser exhibited brittle validation frameworks.The brittle cases involved insufficient debugging, incorrect SPT-failure attribution, or shifting test cases during debugging.
Loading 2608.18389v1…