Source-linked AI summary
InferFix: End-to-End Program Repair with LLMs
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, Alexey Svyatkovskiy
TL;DR
Bugs make software repair costly, while prior LLM approaches have not provided a reliable end-to-end solution for categorized critical defects. InferFix combines static analysis, retrieval-augmented prompts, and a finetuned Codex generator, achieving top-1 fix-generation accuracy of 65.6% in C# and 76.8% in Java.
Problem
Prior LLM program-repair approaches learned general bug-fixing patterns but did not provide a reliable end-to-end solution suitable for productization.
Method
InferFix combines Infer for bug detection, localization, and classification with a contrastive retriever and a finetuned Codex generator using augmented prompts.
Results
65.6% top-1 accuracy was achieved for generating fixes in C# and 76.8% in Java on InferredBugs, outperforming strong LLM baselines.
Takeaways & Limitations
InferFix was deployed internally at Microsoft as a GitHub action and Azure DevOps plugin within continuous-integration pipelines.
Takeaways & Limitations
Reported percentages require generated patches to exactly match the original developer’s token-by-token fix, excluding different valid patches.
Abstract
from arXiv · showhide
Software development life cycle is profoundly influenced by bugs: their introduction, identification, and eventual resolution account for a significant portion of software cost. This has motivated software engineering researchers and practitioners to propose different approaches for automating the identification and repair of software defects. Large language models have been adapted to the program repair task through few-shot demonstration learning and instruction prompting, treating this as an infilling task. However, these models have only focused on learning general bug-fixing patterns for uncategorized bugs mined from public repositories. In this paper, we propose InferFix: a transformer-based program repair framework paired with a state-of-the-art static analyzer to fix critical security and performance bugs. InferFix combines a Retriever -- transformer encoder model pretrained via contrastive learning objective, which aims at searching for semantically equivalent bugs and corresponding fixes; and a Generator -- a large language model (Codex Cushman) finetuned on supervised bug-fix data with prompts augmented via bug type annotations and semantically similar fixes retrieved from an external non-parametric memory. To train and evaluate our approach, we curated InferredBugs, a novel, metadata-rich dataset of bugs extracted by executing the Infer static analyzer on the change histories of thousands of Java and C# repositories. Our evaluation demonstrates that InferFix outperforms strong LLM baselines, with a top-1 accuracy of 65.6% for generating fixes in C# and 76.8% in Java. We discuss the deployment of InferFix alongside Infer at Microsoft which offers an end-to-end solution for detection, classification, and localization of bugs, as well as fixing and validation of candidate patches, integrated in the continuous integration pipeline to automate the software development workflow.
1 INTRODUCTION
InferFix addresses the time-consuming, error-prone manual repair step by combining static analysis with a language-model-based repair framework for critical bugs. It extends prior LLM repair approaches toward a reliable end-to-end workflow.
- Manual patch generation remains time-consuming and error-prone after traditional analyses detect and localize bugs.
- Existing LLM repair approaches use few-shot demonstrations or instruction prompting, but have not provided a reliable end-to-end solution suitable for productization.
- Static analysis can identify critical security and performance issues and produce machine-readable reports suitable for patch-generation models.
- The framework targets Null Pointer Dereference, Resource Leak, and Thread Safety Violation bugs because they create critical performance, reliability, and security issues.
- InferFix combines a contrastively pretrained retriever over historic bugs and fixes with a finetuned 12-billion-parameter Codex model.
2 MOTIVATING EXAMPLE
InferFix is presented as an automated continuous-integration workflow that detects and localizes bugs, prepares relevant context, generates a patch, validates it, and returns it as a bug-fixing pull request.
- InferFix is designed to reduce the developer effort required to detect, localize, and fix errors in large modular projects before merging.
- In the continuous-integration workflow, Infer runs analysis after a pull request triggers testing and building, and InferFix proposes a patch when bugs are detected.
- The approach combines static analysis for bug detection, localization, and classification with a finetuned 12-billion-parameter Codex model for fix generation.
- For a detected bug, preprocessing retains the buggy method and relevant imports, class information, and surrounding method context.
- Retrieval augments the prompt with semantically similar buggy snippets and fixes before the finetuned Codex model generates a patch.
3 DATASET
InferredBugs is a metadata-rich Java and C# dataset built by running Infer over repository histories and comparing analyzer reports across commits. Its metadata supports localized, categorized repair research, while its extracted examples include challenging multi-line bugs.
- InferredBugs contains bugs detected with Infer, whose semantic analysis is based on Separation Logic.
- The extraction analyzed approximately 6.2k Java and C# repositories and more than 1 million commits.
- Each dataset instance includes bug type, multi-granularity location information, and linked change history for detected or fixed issues.
- The pipeline compares Infer reports from previous and current commits to classify issues as introduced, fixed, or preexisting.
- 8,280 bug patches were identified from 2,937 repositories, including 259 filtered null-dereference patches and 462 filtered resource-leak patches.
- The dataset includes multi-line bugs, creating a challenging case for program repair tools.
4 BASELINES
The baselines evaluate LLM-based program repair through demonstration prompting, code completion, and instruction prompting, using exact string match between generated patches and ground-truth fixes.
- Baseline performance is evaluated by exact string match between each generated patch and the ground-truth fix.
- 4.1 Demonstration Prompting: Demonstration learning prepends two answered prompts before the buggy code to show Codex how to approach program repair.
- 4.2 Conditional Language Modeling: Zero-shot conditional language generation uses next-token prediction to rewrite a program without bugs.
- 4.3 Instruction Prompting: Instruction learning adds a natural-language task description to the prompt, with experiments directing the model to output only the code snippet.
5 INFERFIX FRAMEWORK
InferFix combines static analysis, semantic retrieval, and a finetuned language-model generator to produce program fixes from analyzer-enriched prompts and historic bug-fix information.
- InferFix uses three modules: static analysis for bug detection, localization, and classification; retrieval of similar bugs and fixes; and generation of fixes.
- 5.1 Bug Detection & Classification Module: Infer performs Separation Logic-based analysis over control-flow graphs and produces compositional method summaries to identify defects.
- 5.2 Retrieval Module: The retriever searches for semantically equivalent vulnerable code and retrieves fix candidates using cosine similarity between query and buggy-code embeddings.
- 5.2 Retrieval Module: The retrieval database stores encoded buggy snippets as keys and corresponding fixes as values, with similarity computed from transformer embeddings.
- 5.2 Retrieval Module: Contrastive pretraining teaches the bidirectional encoder to distinguish semantically similar same-type bugs from irrelevant bugs of different types.
- 5.3 Generator Module: Codex Cushman is finetuned on supervised bug-fix data whose prompts include bug localization, categorization, extended context, and retrieved similar fixes.
6 PROMPT AUGMENTATION
InferFix augments a finetuned Codex generator with bug annotations, localized context, syntax hierarchies, focal methods, and retrieved structurally similar fixes. These prompt features progressively improve repair accuracy across Java and C# evaluations.
- 6 PROMPT AUGMENTATION: InferFix constructs prompts from prioritized syntax hierarchies, focal context, bug annotations, retrieved hints, and marked buggy regions.The framework’s augmentation strategy combines relevant code context with structured bug information and similar historical fixes.
- 6.1 Basic Prompt: Task-oriented finetuning on buggy and fixed InferredBugs methods outperforms few-shot baselines, with 11–55% relative accuracy improvements across Java bug types.Demonstration learning reaches 19–25% Java accuracy, while instruction learning with Davinci reaches 40–53%.
- 6.2 Bug Type Annotations: Bug-type annotations improve repair accuracy across all evaluated categories and both languages by 2.7–5.6% relative.The annotation is prepended to the basic buggy-method prompt.
- 6.3 Bug Localization: Bug localization markers added to type-annotated prompts improve accuracy across categories by up to 3.4%, especially for larger methods.Infer supplies analyzer locations and surrounds the buggy region with <START_BUG> and <END_BUG> markers.
- 6.4 eWASH extended context: eWASH syntax hierarchies and focal-method context further improve patch-generation accuracy by 7.2–7.8% for Java and 4.0–6.7% for C#.These additions prioritize relevant code within transformer context limits and retain invoked method implementations.
- 6.5 Enriching Context with Hints: Retrieved hints use semantically similar fixes, same-bug-type filtering, and a 60% minimum similarity threshold to enrich the generator prompt.Two nearest neighbors are extracted by default, with identifier obfuscation reducing dependence on naming.
- 6.6 Inference: InferFix decodes ten candidate patches with nucleus sampling at top_p 1.0 and temperature 0.7, then ranks them by sequence log probability.This inference configuration balances prediction diversity and quality.
7 RESULTS
InferFix fixes 57–82% of evaluated Java and C# bug categories with its top-1 prediction and exceeds the best baseline by 8.6–13% in absolute terms. The reported exact-match percentages conservatively count only patches identical to the developer’s token sequence.
- 7 RESULTS: 57–82% of the three evaluated bug categories are fixed by InferFix with the top-1 prediction across Java and C#.These results are reported on the InferredBugs dataset against LLM baselines.
- 7 RESULTS: 8.6–13% absolute performance separates InferFix from the best-performing baseline, Finetuned Codex.The comparison is reported for the evaluated bug categories.
- 7 RESULTS: The evaluation percentages count only generated patches that exactly match the original developer’s token-by-token fix.Correct patches using different token sequences are not counted, making the reported figures conservative.
- 7 RESULTS: InferFix’s top-1 results support its integration into the software development cycle for proposing fixes for critical bugs.The paper connects these results with potential productivity and reliability benefits.
8 DEPLOYMENT
InferFix is deployed with Infer in Microsoft’s Azure DevOps and GitHub continuous integration pipeline. It proposes candidate patches as individually validated pull requests, allowing developers to accept or decline recommendations.
- 8 DEPLOYMENT: InferFix’s CI deployment automates bug detection and fixing for Microsoft internal projects, reducing manual identification and repair effort.The deployment is integrated into Azure DevOps and GitHub pipelines.
- 8 DEPLOYMENT: Each configurable candidate patch is packaged as a separate pull request and individually validated through the CI pipeline.Validation includes build and testing checks described in the deployment workflow.
- 8 DEPLOYMENT: Developers retain the option to accept or decline each recommended fix.This preserves developer control over the served patch recommendations.
- 8 DEPLOYMENT: Internal CI integration has provided workflow benefits through seamless validation and reduced complexity for developers.The paper reports expansion of InferFix integration to additional projects.
9 RELATED WORK
Prior program-repair systems learn from generic, unclassified bugs and generally omit bug-type information. InferFix instead combines bug-type-aware task finetuning with retrieval-based prompt augmentation using historical fixes.
- Earlier systems learn bug-fixing patterns from generic, unclassified bugs mined from open-source change histories.
- InferFix uses bug-type information from the Infer static analyzer to generate fixes tailored to specific bug categories.
- The framework combines retrieval-based prompt augmentation with task-oriented finetuning and large language models.
- Related work also explores pretraining and prompt transformations for code infilling and program editing.
- InferFix augments prompts with similar fixes retrieved from a historical bug database and extended information.
10 CONCLUSION
InferFix is an end-to-end framework pairing Codex with a static analyzer to repair critical security and performance bugs in Java and C#. Its experiments outperform strong LLM baselines, and its deployment integrates repair into Microsoft’s continuous integration workflow.
- InferFix combines Codex with a state-of-the-art static analyzer to repair critical security and performance bugs in Java and C#.
- The InferredBugs dataset contains metadata-rich bugs extracted from change histories of thousands of Java and C# repositories using Infer and InferSharp.
- InferFix was deployed internally at Microsoft as a GitHub action and Azure DevOps plugin within the continuous integration pipeline.
- The internal deployment significantly improved the software development workflow for Microsoft Developer Division projects.