Source-linked AI summary
If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs
Xietao Wang-Lin, Anton Isopoussu, Louis Mahon
TL;DR
The paper asks what happens when LLMs blindly and repeatedly inspect and repair code without history or objective stopping feedback. It analyzes these dynamics across repair settings and probes the model’s internal bug representation. LLMs over-detect bugs, damage correct programs more than they repair incorrect ones, and can cycle through repeated edits; steering the bug-related direction changes editing propensity.
Problem
Little is known about long-term code correctness and termination when LLMs iteratively bug-fix code without objective task-performance feedback.
Method
The study analyzes blind iterative bug-fixing with whole-file edits and search/replace blocks, then uses linear probes and steering interventions to study code-bugginess representations.
Results
LLMs are overeager to find pseudo-bugs, causing correct programs to fail while only a minority of incorrect programs reach a passing state; search/replace blocks also induce more mistakes and cycling changes.
Takeaways & Limitations
A latent buggy-code direction coincides with bug-detection decisions and can control editing propensity, informing sensitivity control in autonomous agents.
Takeaways & Limitations
The report studies Gemini 2.5 Flash-Lite and Qwen 2.5-7B-Instruct in blind, goal-free settings, leaving larger models and exact-goal or multi-file environments for future work.
Abstract
from arXiv · showhide
Large language models (LLMs) have become ubiquitous in software development, with LLM-based automated program repair tools increasingly used during code review. In this report, we explore the iterative blind use of LLMs as bug-fixers. Across multiple models and repair environments, we find that LLMs consistently claim to detect bugs in entirely bug-free programs while the rate of repair of buggy programs is less than that of the damage to correct programs. We also explore the long-term dynamics of this iterative process, and find that this frequently reaches a pseudo-bug-fixing cycle where the same changes are added and removed again ad infinitum. Lastly, via mechanistic probing, we unveil the existence of a steering vector which controls the editing propensity, suggesting that LLMs have an internal representation of ``buggy code", and that this representation is what is falsely activated to induce pseudo-bug fixing. These results provide insight towards the dynamics of fully autonomous bug-fixing systems, as well as stopping conditions under ambiguous goals.
1 Introduction
The paper studies blind iterative LLM bug-fixing, asking how code evolves and when agents stop without objective task feedback. It measures repair and damage dynamics, pseudo-bug cycles, and the internal representation associated with detecting bugs.
- The study focuses on history-free iterations, where each agent sees only the current code state rather than previous states.This setting can arise from blind approval, context-window limits, or delegated multi-agent workflows.
- The report quantifies how iterative bug-fixing improves buggy code while damaging correct code through pseudo-bug removal.
- Under standard settings, the damage rate can become significantly higher than the repair rate.
- Pseudo-bug-fixing cycles occur repeatedly, with the same changes potentially added and removed over time.
- Mechanistic probing identifies a “buggy code” representation activated when models detect real or pseudo-bugs, but not when they judge code bug-free.
2 Related work
Prior work frames iterative LLM generation and coding as dynamical processes whose outputs become later inputs. This report extends those perspectives to blind bug-fixing and probes the latent representation controlling code-bugginess judgments.
- Prior dynamical-systems research studies recursive training, looped transformers, and iterative generation with cycles and fixed points.
- Related coding work repeatedly prompts LLMs to improve code, including readability refinement and self-refinement applications.
- Long-horizon coding studies report that correctness and conciseness degrade as LLMs iteratively extend previous code.
- This report uses linear probes to obtain a steering vector for “code bugginess” that controls the iterative bug-fixing loop.
3 Iterative bug-fixing
The paper models iterative bug-fixing as a discrete dynamical system over code states, comparing whole-file edits with atomic search/replace changes. Each iteration applies the model’s proposed edit, or an identity update when an atomic change is invalid.
- The study compares whole-file editing with search/replace blocks, where the model emits one atomic change applied programmatically.Search/replace blocks are used instead of line-number-based formats, and the model is prompted to fix one important bug at a time.
- Iterative bug-fixing is represented as a fixer function that maps code to revised code, with repeated applications forming a discrete dynamical system.
- With search/replace blocks, valid edits update the code while invalid blocks produce an identity change, preserving the current code state.
- Stochastic decoding turns the fixer into a Markov transition kernel over code states.
- The experiments use temperature 0.7, top-p 0.9, and up to 100 iterative turns for stochastic generation.
4 Experimental Evaluation
Across competitive-programming submissions, the experiments compare repair and damage rates and track how iterative fixing evolves under SRB and whole-file edits. Damage is at least as large as improvement, while SRBs produce more frequent and longer cycles than whole-file edits.
- Experimental Setup: The evaluation uses 20 randomly selected problems and 40 randomly selected C++ submissions per problem from CodeContests+, classified by hidden-test performance.Correct submissions pass all hidden tests within the time limit; partial credit is not used.
- Repair Rate and Damage Rate: Damage is at least as large as improvement, with SRBs showing substantially higher damage than repair and whole-file edits showing similar rates.This pattern remains qualitatively similar for τ = 0 and τ = 0.7.
- Attractor Types: The pass-rate trajectories compare correct and incorrect starting submissions across turns for SRB and whole-file editing configurations.Figure 2 uses Gemini 2.5 Flash-Lite at τ = 0, while Figure 3 compares Gemini 2.5 Flash-Lite with Qwen2.5-7B-Instruct at τ = 0.7.
- Attractor Types: Fixed points can arise when the fixer outputs no change, produces an identity SRB, fails to locate a search block, or degenerates into repeated comments.For whole-file edits, fixed outcomes consist of previously seen programs or degenerate comments.
- Attractor Types: SRBs induce significantly more cycles, more degeneration, longer runs to convergence, and longer cycles than whole-file edits.The authors attribute these differences partly to SRBs restricting the model to one bug fix at a time.
- Attractor Types: Cycles can alternate between correct and incorrect states, including length-2 loops that repeatedly switch between the two.Thus, convergence to a cycle does not imply that all recurring states share the same correctness status.
5 Bug steering vector
The study constructs a steering vector for bug presence and finds that it captures a latent “buggy code” direction that can control editing propensity. Steering can suppress edits and preserve correct submissions or increase edits, though positive steering also induces hallucinated fixes.
- 5 Bug steering vector: The vector is constructed by contrasting high-confidence bug-present and bug-free examples using final-token activations.A separate classification prompt assigns positive and negative examples according to whether the model predicts a bug with top-p greater than 0.9.
- 5.1 Results: The steering vector achieves moderate early-layer AUC and approaches 1.0 around layer 19, while its cosine similarity with the readout direction remains near random.This suggests the vector is not simply reading the model’s explicit Y/N verdict from late-layer activations.
- 5.1 Results: Negative steering completely stops the repair loop and preserves correct submissions, but it does not repair incorrect submissions.Positive steering produces slightly more rounds because increased bug reports can cause search blocks to be absent from the code.
- 5.1 Results: Positive steering increases editing propensity, but the resulting additional edits can make the model hallucinate code to fix.The steering experiments therefore connect the latent direction to both repair and damage behavior.
6 Discussion
The discussion concludes that blind iterative bug-fixing makes LLMs overeager to find pseudo-bugs, damaging correct programs while repairing only a minority of incorrect ones. It also identifies steering-based control of this behavior, while limiting the evidence to two models and blind, goal-free settings.
- 6 Discussion: LLMs are overeager to find pseudo-bugs, causing correct programs to fail while only a minority of incorrect programs reach a passing state.The discussion frames this as the central outcome of iterative bug-fixing under information constraints.
- 6 Discussion: Search/replace blocks induce more mistakes, cycling changes, and failure modes than whole-file editing.Reported failure modes include degenerations and attempts to search for code absent from the source file.
- 6 Discussion: Linear probing identifies a latent “buggy code” direction that coincides with the model’s decision that code contains bugs to fix.Inference-time steering changes editing propensity: positive steering increases both damage and repair rates, while negative steering does the opposite.
- 6 Discussion: The study focuses on Gemini 2.5 Flash-Lite and Qwen 2.5-7B-Instruct in a blind environment without goals.The authors propose extending the analysis to larger models, exact-goal environments, and multi-file settings.
B.1 Compute
The experiments used substantial external API compute for Gemini and a single local GPU for Qwen, with permissive output parsing and fixed judging limits. The appendix also documents selected behaviors from the bug-fixing loop.
- B.1 Compute: Gemini experiments generated around 2B tokens, including failed experiments, although identical repeated results cannot be guaranteed.The study checked that determinism was preserved during iterations.
- B.1 Compute: Qwen-2.5-7B-Instruct experiments ran locally on a single NVIDIA A10G with 24 GB of memory.
- B.1 Compute: Outputs and search/replace blocks were parsed permissively because LLMs were inconsistent in following formatting instructions.Parsing accepted Markdown markers, XML-style tags, raw output, and whitespace-equivalent matches when needed.
- B.1 Compute: Judging used four times the dataset time limit, a 10-second default when unspecified, a 2 GB memory cap, and gnu++17 with gnu++14 fallback.
- B.1 Compute: The appendix selects examples illustrating behaviors observed during the bug-fixing loop.
C.1 Degenerations
The degeneration examples show whole-file and search/replace outputs that repeatedly rewrite code and reasoning, producing malformed or redundant repair behavior. The examples include altered loop logic, repeated bug claims, and edits to generated-number construction.
- C.1 Degenerations: The Gemini whole-file example proposes broad loop and interval changes while acknowledging that the replacement logic is likely not what was intended.The output changes iteration bounds, range checks, and interval calculations together.
- C.1 Degenerations: One generated construction inserts values into a set through a loop and then builds a sentinel-bounded vector from that set.The example includes repeated accumulation of x and explicit sentinel values at both ends of V.
- C.1 Degenerations: The degeneration excerpts include the input variables and a computed quantity named all alongside the altered repair logic.
- C.1 Degenerations: The reasoning degeneration repeatedly claims a critical or significant bug and proposes alternative loop conditions without establishing a stable correction.Several excerpts repeat the same bug diagnosis, including duplicated claims about a while-loop condition.
C.2 Identity replace
The identity-replacement example shows Gemini identifying a premature loop termination and proposing a corresponding edit. The model’s diagnosis targets the break statement after printing the year.
- C.2 Identity replace: Gemini identifies the break statement after printf as causing premature loop termination after the first iteration.The proposed repair targets the break placement inside the while loop.
C.3 Cycles
The cycle example alternates between two competing fixes for the domain-length calculation. The first iteration adds one to the length, while the second removes it based on a different interpretation of the loop boundary.
- C.3 Cycles: The example is identified as a length-2 cycle under Gemini 2.5 Flash-Lite with SRB.The displayed iterations show the alternating domain-length diagnoses.
- C.3 Cycles: The second iteration reverses the fix, claiming l - k + 1 is incorrect when a non-alphabetic character follows the domain.This contradicting diagnosis supplies the second state in the length-2 cycle.
C.4 Activation steering
Activation steering changes how readily the model edits code: positive steering increases edits, while negative steering suppresses them. The examples show both false bug detection on correct programs and contrasting handling of an actual typo.
- C.4 Activation steering: For a genuine output typo, the model identifies Word as incorrect and proposes replacing it with World.The corresponding repair changes only the printed string.
- C.4 Activation steering: The model edits a correct Hello World program by replacing bits/stdc++.h with iostream after incorrectly labeling the namespace as a bug.Other runs instead describe the same program as correct and produce no edit.
- C.4 Activation steering: Under steering, positive settings produce slightly longer transients, whereas negative settings make most runs collapse within the first few rounds.Figure 9 reports the associated transient- and cycle-length distributions for Qwen2.5.
- C.4 Activation steering: Layer 22 is used as the main steering layer because earlier layers appear to encode different issue types, while layer 22 has a more balanced view.The report hypothesizes that layers 15, 16, and 19 emphasize complexity, formatting, and combined bug types, respectively.
- C.4 Activation steering: An ablation compares SRBs with a whole-file edit that uses the same reasoning trace before producing the corrected code.This tests whether cycling and longer transients arise from SRBs rather than the reasoning trace alone.