Source-linked AI summary
Self-Evolving Coding Rules for AI Coding Agents
Zhengyuan Jiang, Reachal Wang, Yuepeng Hu, Yupu Wang, Yuqi Jia, Neil Zhenqiang Gong
TL;DR
Coding rules are often hand-crafted, and RuleEvolve automatically evolves candidate rules through mutation, evaluation, and selection. Across the evaluated frameworks, backbones, and benchmarks, it reports maintained or improved functional correctness with reductions in code length and generation cost.
Problem
Coding rules are commonly hand-crafted, and existing prompt optimizers target short instructions rather than extensive coding rules.
Source: §1 · p. 1Method
RuleEvolve iteratively mutates candidate coding rules, evaluates them with a judge, and retains top-performing candidates using strategic mutation allocation.
Source: §6 · p. 9Results
Takeaways & Limitations
The evaluations establish coding-rule self-evolvement as a promising direction for improving code generation by AI coding agents.
Source: §6 · p. 9Reducing code length may slightly decrease pass rate on some complex tasks, and the LLM-powered mutator and judge introduce stochasticity and potential backbone-model biases.
Source: A Limitations · p. 11
Abstract
from arXiv · showhide
The performance of AI coding agents is highly dependent on their underlying coding rules. However, existing coding rules are typically hand-crafted and fixed, making the process labor-intensive and often suboptimal. In this work, we propose RuleEvolve, a self-evolving framework for coding rules. RuleEvolve maintains a pool of candidate coding rules and iteratively improves them. In each iteration, it employs an LLM-powered mutator module to generate variants from existing candidates, and then uses a judge module to evaluate these variants and update the pool with the best-performing ones. Extensive evaluations across two coding-agent frameworks, four backbone LLMs, and three benchmarks demonstrate that RuleEvolve outperforms both manual engineering and existing prompt optimization baselines in terms of functional correctness of the generated code, code length, and/or generation cost (e.g., tokens used).
1 Introduction
Coding rules shape AI coding agents’ correctness, code length, and generation cost, yet rules are usually hand-crafted and difficult to optimize automatically. RuleEvolve evolves candidate rules and allocates mutations strategically; evaluations report maintained or improved correctness alongside reductions in code length and generation cost.
1 Introduction
- Coding rules are usually hand-crafted, making their creation labor-intensive and often suboptimal or misaligned with specific tasks.The paper notes that human-written rules can even degrade agent performance.Source: §1 · p. 1
- Existing prompt optimizers target short instructions, while coding rules are extensive and costly agent evaluations make random mutation strategies computationally prohibitive.The authors identify a need for a framework that can automatically evolve complex coding rules.Source: §1 · p. 1
- RuleEvolve maintains a pool of candidate rules, generates mutations with an LLM-powered mutator, and uses a judge to select top-performing candidates.The judge evaluates correctness, code length, and generation cost.Source: §1 · pp. 1–2
- Its strategic allocation method prioritizes candidates with higher performance variance or mean performance closer to the current best when distributing mutations.This addresses how to allocate a fixed number of mutations across candidates.Source: §1 · p. 2
- Across two coding-agent frameworks, four backbone LLMs, and three benchmarks, RuleEvolve maintains or improves pass rate while reducing code length and generation cost.On BigCodeBench with the OpenAI SDK and gpt-5.3-codex, average code length fell by over 60% without sacrificing pass rate.Source: §1 · p. 2
2 Related Works
Prior work studies code-generation prompts and coding rules, but offers limited guidance for systematically evolving rules. Existing prompt optimizers are designed mainly for short prompts, whereas coding rules create a larger search space.
2.1 AI-assisted Code Generation
- Prompting strategies have sought to improve generated code, but much prior work evaluates coding LLMs in isolation and optimizes task-specific prompts rather than universal coding rules.The passage cites HumanEval [6] and BigCodeBench [28] as benchmarks for executable and functional code generation.Source: §2.1 · p. 2
2.2 Coding Rules for AI Coding Agents
- Coding-rule studies measure their impact on code generation but do not provide practical guidance for systematically improving or evolving the rules.Rules are typically stored in AGENTS.md and combined with system and user prompts.Source: §2.2 · p. 2
2.3 Self-evolving Agents
- Self-evolving agent research has largely overlooked coding rules, while existing prompt optimizers target short instructions and perform suboptimally on extensive rule sets.The paper reports that its experiments with Prompt-Ops [23] confirm this suboptimal performance.Source: §2.3 · p. 3
3 Problem Definition
The paper frames coding rules as inputs that influence an agent’s generated code and its correctness, length, and inference cost. Its objective is to preserve acceptable correctness while reducing code length and resource consumption.
3.1 AI Agents for Code Generation
- An AI coding agent generates code conditioned on a task and coding rules, which are commonly supplied through an AGENTS.md document prepended to the LLM context.The agent is represented as Aθ, with θ denoting the backbone model parameters.Source: §3.1 · p. 3
- For fixed agent parameters, coding rules influence correctness, code length, and inference cost, all of which matter to the code-generation process.The stated goals are executable, functionally correct code and reduced code length and generation costs.Source: §3.1 · p. 3
3.2 Objective
- A verifier V(x, y) checks whether generated code executes and correctly solves task x, and may do so by running the code against associated test cases.The rules must satisfy a correctness constraint.Source: §3.2 · p. 3
- The objective defines a code-length score and a resource-cost score, with a predefined threshold τ requiring acceptable average correctness across the task set.A higher code-length score indicates more concise code; a higher cost score indicates lower resource consumption.Source: §3.2 · p. 3
4 RuleEvolve
RuleEvolve iteratively mutates candidate coding rules, evaluates them with a judge, and retains high-scoring candidates. Its allocation strategy directs mutations toward candidates with greater score variance or mean scores closer to the current best.
4.1 Overview
- The overview organizes RuleEvolve around a fixed-size rule pool, a mutator that generates candidates, and a judge that evaluates and selects them.The initial pool consists of mutated versions of the initial rules; subsequent rounds retain the top m candidates from the evaluated pool.Source: §4.1 · p. 4
- The framework addresses non-differentiable, costly agent execution through iterative mutation and pruning, with strategic allocation intended to limit wasted evaluations.Each rule set requires full agent execution and verification, which entails latency and API token consumption.Source: §4.1 · p. 4
4.2 Iterative Mutation and Pruning
- The mutator generates new rule candidates, and the judge evaluates code generated under each rule for correctness, length, and inference cost.The iterative process scores candidates, evaluates new mutations, and prunes the combined pool to retain the top m.Source: §4.2 · p. 4
- The judge’s score penalizes correctness degradation and favors shorter generated code, using code characters to approximate both length and inference cost.A hyperparameter λ balances the length and correctness-penalty components.Source: §4.2 · p. 4
4.3 Strategic Mutation Allocation
- Strategic allocation gives more mutations to candidates with higher score variance or mean performance closer to the current best; experiments compare it with naive allocation strategies.Candidate score sets inherit observations from their parent to support consistent mean and standard-deviation calculations.Source: §4.3 · p. 5
5 Evaluation
Across two coding-agent frameworks, RuleEvolve generally preserves or improves pass rate while reducing code length and inference costs. Evaluations cover multiple backbones and benchmarks, with ablations examining iteration, mutation allocation, pool size, and sampling.
5.2 Main Results
- Across OpenAI SDK and OpenHands, RuleEvolve generally matches or improves pass rate while reducing code length and inference costs; Prompt-Ops sometimes uses fewer tokens but can lose correctness.On BigCodeBench with OpenAI SDK, RuleEvolve raises PR from 0.45 (Manual rule) to 0.49, while reducing average code characters from 2372 to 453.1 and token usage from 1022 to 544.5.Source: Table 1 · p. 6, Table 2 · p. 7, §5.2 · p. 7
5.1 Experimental setup
- Evaluation uses BigCodeBench training batches and three 100-sample test benchmarks, four backbone models, and two agent frameworks; metrics cover correctness, code length, and inference cost.Baselines include No Rule, Manual Rule, Prompt-Ops, and even or proportional mutation allocation.Source: §5.1 · pp. 6–7
5.2 Main Results
- Across backbones, RuleEvolve maintains superior or comparable pass rates; on 10-seed BigCodeBench, its PR is 0.502 ± 0.026 with consistent code-efficiency gains.The multi-seed evaluation reports average code lines of 24.3 ± 0.4 and characters of 794.4 ± 14; RuleEvolve also significantly reduces length and inference cost in most backbone cases.Source: §5.2 · pp. 7–8, Table 3 · p. 8
- On 100 SWE-Bench Lite tasks, RuleEvolve achieves the highest PR at 0.340, reducing average code lines from 8.5 to 5.7 and characters from 380 to 254, while generation time increases.The authors attribute the likely time increase to added reasoning overhead in repository-level tasks.Source: §5.2 · p. 8, Table 4 · p. 9
5.3 Ablation Study
- By iteration 20, code-length and token metrics level off while pass rate remains consistent; most cost improvements occur in the first five iterations.Generation-time curves fluctuate, which the authors associate with API latency variance and stochastic generation.Source: §5.3 · p. 9
- With pass rate fixed at 0.500, strategic mutation allocation yields 22.6 code lines and 736.2 characters, below both even and proportional allocation.It also has the lowest generation time (15.48s) and token usage (975.2) among the three variants.Source: §5.3 · p. 9
- Increasing mutations reduces code length and token use while pass rate stays between 0.48 and 0.51; increasing pool size from one to two also reduces length and token use.Pool-size metrics stabilize above two candidates.Source: §5.3 · p. 9
- Across three benchmarks, Pass@k rises as more code versions are sampled for each task.The evaluation uses gpt-4.1-mini; the paper reports that the upward trend holds across all benchmarks.Source: §5.3 · p. 9
6 Conclusion
RuleEvolve iteratively evolves coding rules through mutation and pruning, using a mutator, judge, and strategic mutation allocation. Across the evaluated agent frameworks, backbones, and benchmarks, it reduces code length and inference costs while maintaining functional correctness.
6 Conclusion
- Evaluations across two agentic frameworks, four backbone models, and three coding benchmarks show RuleEvolve outperforms manually crafted rules and existing baselines.The reported gains concern code length and inference costs while maintaining functional correctness.Source: §6 · p. 9
- RuleEvolve evolves coding rules through iterative mutation and pruning, using a mutator, judge module, and strategic mutation allocation.Source: §6 · p. 9
- The paper reports that evolved rules reduce code length and inference costs while maintaining functional correctness.Source: §6 · p. 9
A Limitations
The authors identify three limitations: potential pass-rate decreases on some complex tasks, stochasticity and bias in the LLM modules, and a narrow set of optimization dimensions.
A Limitations
- Reducing code length may slightly decrease pass rate on certain complex tasks when excessive brevity compromises logic.Source: A Limitations · p. 11
- The LLM-powered mutator and judge introduce inherent stochasticity and potential biases from the backbone models used during evolution.Source: A Limitations · p. 11
- The study focuses on correctness, code length, and inference cost approximated through code length, leaving maintainability and architectural complexity for future work.Source: A Limitations · p. 11
B Justification for Mutation-Allocation Strategy
The mutation-allocation strategy is justified as an approach to selecting the best-performing rule under noisy evaluations. It directs more mutations toward close competitors and uncertain candidates, which the paper says reduces the risk of pruning a potentially superior rule.
B Justification for Mutation-Allocation Strategy
- Under independent normally distributed candidate scores and a large mutation budget, the strategy seeks to maximize the probability of correctly selecting the true best candidate.The probability of correct selection is defined as the chance that the candidate set with the highest sample mean is truly best.Source: B Justification for… · p. 13
- Candidates closer in mean performance to the current best receive more mutations, as do candidates with higher score variance.Source: B Justification for… · p. 13
- The allocation for the current best candidate is scaled using the aggregate noise of the other candidates to stabilize comparisons.Source: B Justification for… · p. 13
- RuleEvolve's allocation strategy is intended to reduce the risk of pruning a potentially superior coding rule because of stochastic noise.Source: B Justification for… · p. 13
C Analysis on Searched Rules
The searched rules were analyzed by removing functional categories and measuring changes in generated code length. The findings distinguish high-impact form constraints, efficient brevity directives, and process or budget directives that increased output length.
C Analysis on Searched Rules
- Output-format and self-containment directives account for 153 attributable characters, the largest reported category impact.The comparison removes each functional category from a 16-section rule; the full rule produces 454.1 characters versus 1,040.6 with no rule.Source: C Analysis on Searched Rules · p. 13
- Brevity-style directives yield 61.6 characters from a single section, making them the most efficient per section.Source: C Analysis on Searched Rules · p. 13
- Removing process and budget directives shortens the generated output, so those directives are net negative for code length in this analysis.Source: C Analysis on Searched Rules · p. 13
D Code Quality Metrics
On generated files, RuleEvolve improves several quality metrics while exposing a code-density trade-off tied to less documentation and defensive scaffolding.
D Code Quality Metrics
- Cyclomatic complexity fell from 5.37 to 3.95, Halstead volume by 47%, and duplicated-block ratio to 0.00%, while maintainability remained Grade A (78.3 vs. 85.1 for no rule).The metrics cover generated files.Source: D Code Quality Metrics · p. 14
- Code density increased: complexity per SLOC rose 22%, while docstring coverage fell from 47% to 1%.The authors identify this as the remaining trade-off.Source: D Code Quality Metrics · p. 14
- The authors attribute the trade-off to removing documentation and defensive scaffolding rather than redundant logic.Source: D Code Quality Metrics · p. 14
E.1 Problem description
The examples instantiate palindrome utilities under a task prompt and four coding-rule settings; implementations differ in how they identify a palindromic suffix and append characters.
E.1 Problem description
- The example uses OpenAI SDK with gpt-4.1-mini; its task asks for palindrome checking and shortest-palindrome utilities.Source: §E.1 · p. 14
E.2 No Rule
- With no rules, the solution scans suffixes until it finds a palindrome, then appends the reversed preceding prefix.It also handles an empty input directly.Source: §E.2 · p. 15
E.3 Manual Rule
- The Manual Rule solution uses a prefix function on reversed input, a separator, and the input to find the longest palindromic suffix, then appends the reversed remainder.The described algorithm runs in O(n) time and O(n) space.Source: §E.3 · pp. 16–17
E.4 Prompt-Ops
- Prompt-Ops checks successive suffixes for palindromicity and appends the reversed prefix before the first match, with a full-reversal fallback.Source: §E.4 · p. 18
E.5 RuleEvolve
- RuleEvolve checks successive suffixes and appends the reversed preceding prefix when it finds a palindrome; its final return is the input string.Source: §E.5 · p. 18
F Manual Rules
The manual rules prescribe a broad Python and software-development profile, including style, architecture, quality practices, and technology choices.
Role Definition
- The role definition asks the agent to act as an experienced Python tutor, ML engineer, and data scientist with strong coding and explanatory skills.Source: Role Definition · p. 18
Technology Stack
- The technology stack specifies Python 3.10+, Poetry or Rye, Ruff, typing, pytest, and Google-style docstrings, alongside tools for environments, APIs, ML, and deployment.Source: Technology Stack · pp. 18–19
1. Pythonic Practices
- Pythonic practices call for readable, elegant code that follows PEP 8.Source: §1 · p. 19
2. Modular Design
- Modular design favors single-responsibility modules and composition over inheritance.Source: §2 · p. 19
3. Code Quality
- Code-quality rules require comprehensive type annotations, Google-style docstrings, pytest coverage aimed at 90% or higher, and specific exception handling.Source: §3 · p. 19
4. ML/AI Specific Guidelines
- ML/AI guidance calls for configuration and data-pipeline tools, model versioning, and a dedicated module for prompt templates.Source: §4 · p. 19
5. Performance Optimization
- Performance guidance recommends async and await, caching, and resource monitoring.Source: §5 · p. 19
6. API Development with FastAPI; Others
- FastAPI guidance specifies Pydantic validation, dependency injection, and OAuth 2.0 or JWT; other rules favor modern libraries and self-contained, executable examples.Source: §6 · p. 19, Others · p. 19