Source-linked AI summary
Applying CodeBERT for Automated Program Repair of Java Simple Bugs
Ehsan Mashhadi, Hadi Hemmati
TL;DR
The paper investigates automated repair of Java simple bugs in the ManySStuBs4J dataset using CodeBERT. After fine-tuning on small and large datasets, the approach reproduces developer fixes in 19–72% of cases and supports varied-length patches across bug types.
Problem
The paper targets automated repair of Java simple bugs, where each bug appears in one statement and its fix lies within that statement.
Method
The approach fine-tunes the pre-trained CodeBERT model on buggy code as source and fixed code as target using ManySStuBs4J datasets.
Results
19–72% of cases received fix codes exactly matching developer implementations across the evaluated dataset types.
Takeaways & Limitations
The approach generates variable-length fixes for different bug types, including types with few training examples, without language-specific parsers or substantial post-processing.
Abstract
from arXiv · showhide
Software debugging, and program repair are among the most time-consuming and labor-intensive tasks in software engineering that would benefit a lot from automation. In this paper, we propose a novel automated program repair approach based on CodeBERT, which is a transformer-based neural architecture pre-trained on large corpus of source code. We fine-tune our model on the ManySStuBs4J small and large datasets to automatically generate the fix codes. The results show that our technique accurately predicts the fixed codes implemented by the developers in 19-72% of the cases, depending on the type of datasets, in less than a second per bug. We also observe that our method can generate varied-length fixes (short and long) and can fix different types of bugs, even if only a few instances of those types of bugs exist in the training dataset.
I. INTRODUCTION
The paper applies CodeBERT to automated repair of Java simple bugs in ManySStuBs4J, motivated by the similarity between program repair and sequence-to-sequence learning. It reports strong accuracy across dataset variants and identifies support for varied bug types, dataset sizes, and patch lengths.
- Automated program repair changes existing buggy programs automatically, motivating machine-learning approaches based on sequence-to-sequence learning.
- CodeBERT is a bimodal pre-trained language model trained on natural and programming languages using masked language modeling and replaced token detection.Its pre-training corpus includes 6.4M unimodal codes across several programming languages.
- The approach fine-tunes CodeBERT to generate fixes for single-statement Java bugs in the ManySStuBs4J dataset.
- 72% and 68.8% accuracy were achieved on the large and small datasets, respectively, while unique datasets reached 23.7% and 19.65%.The approach does not require special bug-location tokens or context lines.
- The technique works with small and large datasets and can be applied to other datasets and programming languages without language-specific lexers or parsers.CodeBERT pre-training reduces the need to train a model from scratch on a large local dataset.
- CodeBERT fixes different bug types with few training examples, generates unseen tokens, and produces both short and long patches.The reported examples include wrong Boolean parameters and incorrectly ordered function parameters.
II. RELATED WORK
Earlier repair systems use search, mined fix patterns, compiler-oriented neural repair, or machine translation. The paper distinguishes its CodeBERT approach by avoiding search-based techniques, special bug tokens, separate context inputs, and language-specific parsing.
- Search-based systems mutate buggy code with operators and search for fixes, as in GenProg and MutRepair.
- Compiler-error repair tools such as DeepFix and TRACER target errors verifiable through compilation or language-specific parsing, whereas this approach generates patches for logical errors.
- Pattern-based systems mine recurrent patterns, successful patches, or similar code to obtain repair ingredients.Examples include HDRepair, Prophet, Genesis, and DeepRepair.
- SequenceR uses recurrent encoder-decoder networks, copy mechanisms, context abstraction, and special tokens to indicate bug boundaries.
- Other neural machine-translation approaches use attention, code abstraction, convolutional networks, ensembles, or surrounding context to generate patches.The cited systems include Ratchet, Tufano et al., CoCoNuT, and DLFix.
III. EXPERIMENT
The experiment section presents the datasets, experiment design, and results used to evaluate the proposed repair approach.
- The study explains its datasets, experiment design, and results.
A. Dataset
The study uses small and large ManySStuBs4J collections of single-statement Java bugs, then constructs duplicate and unique variants for evaluation. The datasets are randomly split into training, validation, and test sets.
- The small and large ManySStuBs4J versions contain 10,231 and 63,923 single-statement bug instances, respectively.They were mined from 12,598 and 86,771 bug-fix commits.
- The study uses a Duplicate subset because some Missing Throws Exception, Delete Throws Exception, and Change Modifier instances lack proper before- and after-fix source code.
- A Unique dataset removes duplicate buggy-code and fixed-code pairs from both dataset sizes.The resulting datasets retain more than 91% of the original large data and 80% of the original small data.
- The datasets are randomly divided into training, validation, and test sets using an 80%, 10%, 10% ratio.
B. Design
The design fine-tunes CodeBERT with buggy Java code as input and developer fixes as targets, using a Transformer encoder with a decoder for generation. Evaluation uses exact-match accuracy, alongside a simple LSTM baseline and comparisons of input requirements, runtime, and language portability.
- Architecture: CodeBERT uses a multi-layer bidirectional Transformer and supports code sequences up to 510 tokens directly.The 512-token maximum includes two internal tokens, leaving 510 tokens for code input.
- Generation setup: The model receives buggy code as the source sequence and fixed code as the target sequence for generation.A Transformer decoder is attached to CodeBERT's encoder, and the task uses only the code segment of the input format.
- Baseline: The study trains a simple LSTM-based sequence-to-sequence model as a baseline without abstraction or copy mechanisms.The baseline follows a bidirectional LSTM encoder, LSTM decoder, global attention, and copy selector.
- Evaluation: Accuracy measures whether generated fixes exactly match developer-implemented fixes.Testing-based metrics were not used because the dataset projects lacked complete, reliable test suites.
- Comparison dimensions: The design also compares required input formats, training and inference time, and additional effort for supporting other programming languages.These comparisons include related approaches such as SequenceR, DLFix, and CoCoNuT.
1) Answer to RQ1:
CodeBERT achieves substantially higher accuracy than the simple baseline across duplicate and unique ManySStuBs4J datasets, while remaining applicable to varied fix lengths and fast per-bug inference. Direct comparison with prior work is limited because the datasets are not readily compatible, but the reported unique-dataset accuracy falls within the range of related tools.
- Accuracy results: 72% and 68.8% accuracy are achieved on duplicate datasets, compared with 23.27% and 19.65% on unique datasets.These are top-1 accuracy results across four datasets.
- Baseline comparison: The simple baseline reaches 2.2%-17.65% accuracy, while CodeBERT performs better, especially on relatively small datasets.The paper attributes the small-data benefit to fine-tuning a pre-trained model.
- Prior-work context: Direct comparison with previous work is not possible because the proposed dataset is not readily usable with those approaches.Reported prior-work accuracies provide context rather than a direct benchmark comparison.
- Execution time: Generating patches for 2,449 unique bugs takes about 10 minutes with this technique, whereas the compared abstraction process would take more than 938 hours.The comparison uses Sequencer's reported 23 seconds per bug for abstracting buggy context.
- Patch generation: CodeBERT generates patches of up to 510 tokens and uses byte-level byte-pair encoding to handle unseen words without custom abstraction or post-processing.The tokenizer progressively falls back to character-level embeddings for unseen words.
- Execution time: Patch generation takes less than a second per bug after fine-tuning, although training takes 5.5-9 hours and inference runs take 1.6-20 minutes.The timing measurements used a server with four NVIDIA Tesla V100 SXM2 16GB GPUs.
- Evaluation boundary: Dynamic validation through test cases requires additional time that is excluded from the reported execution time.The method's timing excludes validating generated patches with dynamic analysis.
2) Answer to RQ2:
CodeBERT fixes diverse Java simple bugs, including cases with few local examples, unseen project patterns, and generated code absent from training data. It produces both short and long fixes across bug categories.
- Bug diversity and generalization: 60% of SWAP_BOOLEAN_LITERAL bugs were fixed in the unique large dataset despite only 103 local training instances.The authors attribute this capability to CodeBERT’s large pre-training dataset.
- Novel code generation: The model generated fixes containing code absent from both local training and evaluation datasets.One example adds a logical AND operator and operand to produce mReadOwners!=null && includingOwners.
- Bug categories: SWAP_ARGUMENTS denotes using a correct method with arguments in the wrong order.The paper identifies this as a common Java bug because Java lacks named parameters.
- Fix length: The approach generated fixes ranging from small patches to 193 tokens, including up to 50% success for patches longer than 100 tokens in the small-duplicate dataset.The reported long-patch result refers to 100% accurate patches.
IV. CONCLUSION AND FUTURE WORK
The paper presents CodeBERT fine-tuned for Java simple-bug repair and reports exact developer-matching fixes across dataset variants. It also identifies practical scope advantages and proposes broader future evaluation.
- Conclusion: CodeBERT fine-tuned on small and large ManySStuBs4J datasets generated developer-matching fixes in 19–72% of cases.The authors describe the approach as viable for different bug types.
- Conclusion: The approach avoids special-token, short-token-length, unknown-vocabulary, and post-processing limitations.These properties are presented as making the method more practical.
- Conclusion: Fix effectiveness was not negatively impacted by the number of instances from a bug type or project in local training data.The conclusion also reports variable-length fix generation.
- Future work: Future work includes applying the approach to Defects4j and using testing-based metrics and additional CodeBERT-supported programming languages.These directions expand datasets, evaluation, and language coverage.