PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Transformation Rule Synthesis using LLMs: Potential and Limits

Axel Allain, Aymeric Blot, Djamel Eddine Khelladi, Mathieu Acher

arXiv:2609.03592v1cs.SE

TL;DR

LLMs offer explainability and determinism challenges, alongside rising costs for repetitive large-codebase changes. This study evaluates rule synthesis across DSLs, models, tasks, and benchmarks, finding strong potential with frontier models while exposing limits on complex transformations and reproducibility.

  • Problem

    LLM-based code modification can be costly, error-prone, difficult to explain, and nondeterministic, while systematic evidence on generating reusable transformation rules is limited.

  • Method

    The study evaluates three LLMs across Comby, GritQL, and Ast-Grep using six benchmarks covering four software-evolution tasks.

  • Results

    GPT-5.4 shows strong rule-synthesis performance, while effectiveness varies with transformation complexity, edit structure, and DSL expressiveness.

  • Takeaways & Limitations

    Rule synthesis is better suited to semi-automated workflows in which generated rules are reviewed, refined, and applied at scale.

  • Takeaways & Limitations

    Performance may degrade on larger codebases, deeply nested structures, global-context transformations, and complex non-local edits.

Abstract

from arXiv · show

Due to their black-box nature, LLMs suffer from limited explain- ability and a lack of determinism. Their usage cost can also rise, particularly with repetitive tasks on large codebases. To mitigate this, we conduct a novel empirical study targeting three domain- specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep. We evaluate three LLMs (GPT-5.4, GPT-oss-120B, and Llama3.1-8B) on six diverse datasets covering four software- evolution tasks: API misuse correction, program repair, API migra- tion, and language version migration. Our results provide evidence that transformation rule synthesis moves beyond proof-of-concept with strong frontier models. GPT-5.4 achieves consistently high rule applicability rates and produces transformations closest to the ground truth across most benchmarks. Smaller and open-weight GPT-oss-120B and Llama3.1-8B models remain effective for simpler, localized changes but struggle with complex migration scenarios. We also observe non-negligible generalizability through the usage of meta-variables and through a high reuse score in the first quartile of many datasets. Finally, when compared to the anti-unification algorithm, LLMs outperform it in correctness, but underperform in rule applicability. Overall, our results show great potential for LLMs to generate sound, correct, generalizable, and reusable rules.

1 Introduction

The paper studies whether LLMs can generate explainable, deterministic, and reusable code transformation rules from observed code changes. It evaluates this capability across multiple models, DSLs, software-evolution tasks, and benchmarks.

  • 1 Introduction: LLMs are evaluated for generating DSL-based transformation rules by inferring rewrite patterns from original and evolved code pairs.The study also examines how documentation, rule examples, and limited retrieval affect rule generation.
  • 1 Introduction: The evaluation covers three LLMs, three transformation DSLs, four software-engineering tasks, and six dedicated benchmarks.The tasks include program repair, API misuse correction, API migration, and language version migration.
  • 1 Introduction: Strong frontier models provide evidence that transformation-rule synthesis has moved beyond proof-of-concept, although effectiveness depends on task and edit structure.The fit between the edit structure and the rule abstraction also affects performance.
  • 1 Introduction: The study systematically analyzes rule soundness, correctness, and generalization across DSLs, software-evolution tasks, and datasets.It also compares DSLs, evaluates retrieval-augmented generation, and analyzes model success and failure modes.

2 Background

Code search and transformation tools reduce manual effort in large-scale edits, but textual tools require expertise and may miss program structure. Transformation DSLs address this by expressing reusable syntactic or AST-level rewrites with metavariables.

  • 2 Background: Search and transformation tools help developers locate and automate repetitive large-scale code modifications.Examples include grep and ripgrep for search, and sed and sd for textual edits.
  • 2 Background: Textual tools and regular expressions require substantial expertise and may produce unintended transformations because they cannot capture program syntax.Their purely textual operation limits their ability to represent syntactic structure.
  • 2 Background: Transformation DSLs make rules easier to write and reuse by matching program elements at syntactic or AST level.They typically use matching patterns and rewriting patterns enriched with metavariables.

3 Motivating Example

Large codebases make repeated manual or LLM-mediated edits costly and error-prone, motivating reusable transformation rules. The paper asks whether LLMs can infer generalized rules from example changes across several DSLs.

  • 3 Motivating Example: Repeatedly prompting LLMs for individual changes can increase token consumption, latency, and computational cost on large codebases.Manual modifications are also described as tedious and error-prone.
  • 3 Motivating Example: A reusable transformation rule can apply one refactoring automatically across all matching occurrences in a codebase.The example replaces explicit null checks with a concise Optional API.
  • 3 Motivating Example: The example expresses the refactoring as a Comby rule with matching and rewriting patterns containing metavariables.The rule captures the variable and function structure while rewriting the null-check form.
  • 3 Motivating Example: The study investigates whether LLMs can reliably synthesize correct and reusable rules from before-and-after code examples across Comby, Ast-Grep, and GritQL.The paper identifies this as an area lacking systematic evaluation.

4 Research Questions

The research questions examine whether LLMs can generate sound rules cost-effectively and how complete those rules are in coverage, structural richness, and generalization.

  • 4 Research Questions: The first research question assesses whether LLMs generate sound transformation rules from code diffs while maintaining reasonable computational and financial cost.It evaluates validity and token expense during rule generation.
  • 4 Research Questions: The second research question examines generated-rule completeness through coverage, structural richness, and generalization capacity.The supplied passage introduces this question but does not provide its full wording.

5 Methodology

The methodology uses three LLMs and three transformation DSLs to generate rules from code changes, then evaluates them across diverse datasets and quality metrics.

  • Method Overview: LLMs generate transformation rules from code diffs to replay software evolutions, which are then evaluated with several quality metrics.
  • Models: The study compares GPT-5.4, GPT-oss-120B, and Llama3.1-8B across different architectures, parameter scales, and deployment constraints.Temperature is set to 0 to reduce output variability and improve reproducibility, without guaranteeing fully deterministic outputs.
  • DSL Tools: The evaluated DSLs are Comby, Ast-Grep, and GritQL, which differ in syntax and operate with varying levels of concrete or abstract structural matching.Comby uses template variables, Ast-Grep matches AST nodes, and GritQL provides expressive AST-based patterns with SQL-like syntax.
  • Datasets: The evaluation covers six datasets grouped across API misuse correction, program repair, API migration, and language version migration tasks.Datasets were selected for open access, diverse origins and languages, available code pairs, varied transformation complexity, and human-written edits.
  • Evaluation Metrics: Rule quality is assessed through soundness, applicability, failures, token generation, completeness, rule count, rule size, and metavariable usage metrics.Rule Applicability Rate measures the percentage of generated rules that are syntactically valid and executable by their DSL engine.

6 Results

GPT-5.4 provides the strongest overall rule soundness and transformation correctness, while smaller models are more effective on simpler edits. Rule construction reveals a trade-off between explicit coverage and abstraction, and anti-unification trades correctness for applicability or over-generalization.

  • Soundness: GPT-5.4 achieves the highest Rule Applicability (RA), ranging from 68.2% to 100%, and the fewest failures across most DSLs.GritQL is the most reliable DSL, while GPT-oss-120B performs strongly with Comby without RAG at 93.6%–100% RA.
  • Soundness: GPT-5.4 significantly outperforms GPT-oss-120B in 7 of 24 configurations, while GPT-oss-120B wins in 4 and both outperform Llama3.1-8B in at least 16.GPT-5.4’s advantages are concentrated mostly on Ast-Grep; GPT-oss-120B’s wins occur on Comby without RAG.
  • Rule quality: GPT-5.4 produces larger, more explicit rules, whereas GPT-oss-120B generates more compact, generalized rules with greater metavariable use.GPT-5.4 typically produces 1.1–3.7 rules with 3%–13% MTR, while GPT-oss-120B typically produces 1.6–4.7 rules with 7%–20% MTR.
  • Rule quality: Anti-unification reaches a maximum MTR of 62 versus 20 for LLM-generated rules, producing overly generic and unreadable transformations.Its broad metavariable replacement explains the high abstraction but can make rules match nearly any code fragment.
  • Generalization and reuse: Reuse is concentrated among a smaller set of highly repetitive transformations, with many generic rules lowering AM/EM despite producing numerous matches.The reuse plots show high reuse in the first quartile and extreme outliers corresponding to recurrent transformations.
  • Accuracy: GPT-5.4 achieves the best syntactic and semantic alignment, reaching up to 98.1% TD and strong EM/AM results on complex benchmarks.It reaches up to 56.0% AM on Defects4J and 54.1% EM/AM on BugsInPy, while GPT-oss-120B remains competitive on simpler Comby settings.

7 Recommendations

The study recommends matching models and workflows to transformation complexity, DSL constraints, and task structure. It favors semi-automated, decomposed workflows and identifies hybrid model use, improved tooling, and task-suitable benchmarks as practical directions.

  • Recommendations for Practitioners: Transformation-rule generation is better suited to semi-automated workflows than fully autonomous refactoring.Generated rules can be reviewed, refined, and applied at scale.
  • Recommendations for Practitioners: Complex refactorings should be decomposed into smaller, composable subrules to improve reliability and interpretability.Examples alone may not recover the intended abstraction level; explicit user intent or additional specifications can support generalization.
  • Recommendations for Practitioners: Frontier models handle complex transformations better, whereas smaller models support simple one-line rules and local deployment, motivating hybrid workflows.Local deployment is especially relevant when privacy constraints are critical.
  • For Researchers: Researchers should study task amenability, controllable generalization, and decomposition strategies for complex transformations.Inferred rules can also support software-evolution analysis, repository mining, and benchmark augmentation.
  • Benchmark Design: Future benchmarks should account explicitly for task suitability because large, heterogeneous diffs may not share structures that yield concise reusable rules.Defects4J, BugsInPy, and PyMigBench are identified as especially challenging examples.
  • Transformation-Language Designers and Tool Builders: DSL and tool designers should support rule-scope control, decomposition, LLM-friendly formats, and better error recovery.Many failures arise from over- or under-generalization and syntactic configuration errors such as malformed YAML or invalid multi-node patterns.
  • Recommendations for Practitioners: Model and workflow choices should align with the target DSL and software-evolution task.The study reports that adding context alone is insufficient when the conditioning signal is poorly matched.

8 Related Work

Prior work generates transformation or detection rules from code changes, recurring patterns, or synthetic examples. This study differs by evaluating LLM-based synthesis on real-world changes across multiple DSLs rather than relying on a single DSL or aggregated synthetic examples.

  • Rule Generation from Code Changes: Prior research has generated transformation rules from code edit examples and recurring security or optimization patterns.Many approaches infer rules from input-output code pairs.
  • Rule Generation from Code Changes: Ketkar et al. use Comby and anti-unification to generalize type-change edits by replacing differences with metavariable placeholders.PyEvolve extends this approach with graph-based analysis for unseen data-flow and control-flow variants.
  • LLM-Assisted Rule Generation: MELT infers rules from pull requests, while SPELL augments this process with LLM-generated input-output-context triplets and iterative LLM refinement.SPELL first applies anti-unification to construct an initial rule.
  • Comparison with SPELL: SPELL differs from this study because it uses synthetic examples in one DSL and assumes multiple examples of the same transformation.The present evaluation uses real-world code changes and directly infers rules from them.
  • Security and Optimization Applications: Other systems use LLMs, retrieval, or abstract configuration formats to generate vulnerability-detection rules.Examples include CodeQL transformation rules and natural-language-driven vulnerability rules.
  • Open Challenges: Low-resource and domain-specific languages remain challenging because they are underrepresented in LLM training data.Domain-specific pre-training, fine-tuning, and retrieval-augmented generation are proposed responses.

9 Conclusion

The paper presents a broad empirical assessment showing that LLMs can synthesize useful transformation rules, but performance depends on task complexity, code-change structure, and DSL expressiveness. The authors conclude that rule synthesis should be reviewed and refined within semi-automated workflows, with further evaluation and feedback-based refinement as future work.

  • Conclusion: The study evaluates three LLMs across three transformation DSLs and six benchmarks covering multiple software-evolution tasks.It is designed as a comprehensive assessment in realistic settings.
  • Conclusion: LLMs generate syntactically valid rules with strong but imperfect performance, especially for localized edits rather than function-level repairs or multi-statement migrations.Effectiveness varies with transformation complexity, code-change structure, and DSL expressiveness.
  • Conclusion: Generated rules often show high reusability, but models tend toward overly specific rules instead of using metavariables and higher-level abstractions.The paper also reports formatting failures, over- and under-generalization, and missing edits.
  • Conclusion: The anti-unification baseline performs worse overall than LLMs on syntactic and semantic correctness.The supplied conclusion passage contrasts the baseline with LLM performance on correctness.
  • Conclusion: The central question shifts toward the task and representation conditions under which rules are sound, correct, generalizable, and reusable.The authors recommend semi-automated workflows in which rules are reviewed, refined, and applied at scale.
  • Future Work: Future work will extend evaluation to additional benchmarks, models, and DSLs and investigate iterative feedback for correcting generated rules.Possible deployment settings include IDEs and CI pipelines.
Loading 2609.03592v1…