Source-linked AI summary
Neural Transfer Learning for Repairing Security Vulnerabilities in C Code
Zimin Chen, Steve Kommrusch, Martin Monperrus
TL;DR
Deep-learning vulnerability repair is constrained by scarce confirmed vulnerability-fix data. The paper proposes VRepair, which pretrains on a large C bug-fix corpus and fine-tunes on vulnerability fixes; transfer learning then outperforms small-dataset training and other alternatives.
Problem
Confirmed vulnerability-fix datasets are too small to be meaningfully used for training deep-learning models.
Method
VRepair first trains on a large bug-fix corpus, then tunes its model on a curated vulnerability-fix dataset.
Results
Transfer learning outperforms training only on vulnerability fixes, reaching 21.86% versus 7.86% on Big-Vul and 22.73% versus 11.29% on CVEfixes.
Takeaways & Limitations
Knowledge learned from bug fixing can be transferred to vulnerability fixing in C.
Takeaways & Limitations
VRepair currently targets intraprocedural vulnerabilities in C functions and identifies only the first vulnerable line for input.
Abstract
from arXiv · showhide
In this paper, we address the problem of automatic repair of software vulnerabilities with deep learning. The major problem with data-driven vulnerability repair is that the few existing datasets of known confirmed vulnerabilities consist of only a few thousand examples. However, training a deep learning model often requires hundreds of thousands of examples. In this work, we leverage the intuition that the bug fixing task and the vulnerability fixing task are related and that the knowledge learned from bug fixes can be transferred to fixing vulnerabilities. In the machine learning community, this technique is called transfer learning. In this paper, we propose an approach for repairing security vulnerabilities named VRepair which is based on transfer learning. VRepair is first trained on a large bug fix corpus and is then tuned on a vulnerability fix dataset, which is an order of magnitude smaller. In our experiments, we show that a model trained only on a bug fix corpus can already fix some vulnerabilities. Then, we demonstrate that transfer learning improves the ability to repair vulnerable C functions. We also show that the transfer learning model performs better than a model trained with a denoising task and fine-tuned on the vulnerability fixing task. To sum up, this paper shows that transfer learning works well for repairing security vulnerabilities in C compared to learning on a small dataset.
1 INTRODUCTION
Vulnerability repair is difficult to learn with deep models because confirmed vulnerability-fix datasets are small. VRepair addresses this by transferring knowledge from large-scale bug fixing to vulnerability fixing and reports better performance than alternative training strategies.
- Motivation: Confirmed vulnerability-fix datasets are scarce, although deep-learning models typically require much larger training corpora.A related system was trained and evaluated on 279 manually identified vulnerabilities, whereas translation models have used over 41 million sentence pairs.
- Approach: VRepair transfers knowledge from a large bug-fixing dataset to a smaller vulnerability-fixing dataset.The model is first trained on bug fixing and then tuned on vulnerability fixing.
- Approach: The source corpus contains over 21 million C bug-fixing commits extracted from 892 million GitHub events.The events span 2017-01-01 through 2018-12-31 and are processed with a heuristic to identify bug-fix commits.
- Results: 21.86% accuracy on Big-Vul and 22.73% on CVEfixes demonstrate improved vulnerability-fixing performance with transfer learning.VRepair also outperforms denoising pre-training and improves final-model stability.
- Results: VRepair outperforms models trained only on the small vulnerability dataset, on a large generic bug-fix corpus, or with denoising pre-training.The authors also report that transfer-learning performance is stable.
- Availability: The authors share the code and data to facilitate replication and future research.
2 BACKGROUND
The paper introduces vulnerability concepts and the Transformer-based sequence-to-sequence foundations used by VRepair. It frames transfer learning as adapting knowledge from a data-rich source task to a related target task with limited data.
- Vulnerabilities: A software vulnerability is a code weakness that an attacker can exploit to perform unauthorized actions.Examples include buffer overflows and SQL injection.
- Vulnerability identifiers: CVE IDs uniquely identify vulnerabilities, while CWE IDs classify them into general software-weakness categories.For example, CVE-2019-9208 is categorized as CWE-476, NULL Pointer Dereference.
- Seq2seq learning: A seq2seq model encodes an input sequence into an intermediate representation and decodes that representation into an output sequence.The encoder maps X to H, and the decoder generates Y given H.
- Transformers: Transformers use self-attention to connect tokens efficiently and compute hidden states in parallel.They also use positional embeddings because parallel computation does not inherently encode token position.
- Transfer learning: Transfer learning improves a related target task by adapting knowledge learned from a source task with more available training data.The source-trained model is tuned on the target task using its limited dataset.
- VRepair foundation: VRepair uses network-based transfer learning because bug-repair data can pretrain a Transformer representation also suitable for vulnerability repair.The target vulnerability-repair problem maps naturally to a sequence-to-sequence formulation.
3 VREPAIR: REPAIRING SOFTWARE VULNERABILITIES WITH TRANSFER LEARNING
VRepair is a Transformer-based vulnerability repair system that transfers knowledge from large-scale bug fixing to scarce vulnerability fixing data. It represents repairs as compact token-context changes and uses multi-stage training and inference to generate patches.
- Overview: VRepair targets automatic repair of intraprocedural vulnerabilities in C functions using a Transformer architecture.The system has source-domain training, target-domain training, and inference phases.
- Transfer-learning pipeline: The model first learns from a large bug-fix corpus, then tunes its weights on a high-quality vulnerability-fix dataset.This transfers knowledge about manipulating code and generating patches from bug fixing to vulnerability fixing.
- Inference: At inference, VRepair receives a vulnerable function, marked location, and CWE token, then generates multiple context-diff patch proposals.Each predicted patch is applied to the identified vulnerable function.
- Code representation: A context size of 3 was selected because pilot experiments found it represented most patches without ambiguity.Shorter contexts can reduce output length but risk multiple interpretations.
- Code representation: The token context diff represents additions, deletions, and replacements across multiple locations using short output sequences.Unlike full-function generation or single-line changes, the representation supports complex multi-line repairs.
- Inference: Beam search orders likely model outputs, while VRepair introduces a beam tailored to its code representation.The specialized beam is part of VRepair inference.
4 EXPERIMENTAL PROTOCOL
The evaluation methodology is organized around four research questions that define how VRepair’s approach will be assessed.
- Experimental protocol: The experimental protocol defines four research questions and specifies how the approach will be evaluated.The questions cover training regimes, transfer learning, denoising pre-training, and data-split strategies.
4.1 Research Questions
The research questions compare single-domain training with transfer learning and examine whether transfer learning outperforms denoising pre-training under different data splits.
- RQ1: RQ1 measures vulnerability-fixing accuracy when training only on the small target dataset or only on the larger bug-fixing source dataset.Both settings are treated as single-domain training.
- RQ2: RQ2 evaluates whether source-domain training followed by target-domain tuning mitigates the small vulnerability-dataset problem.The source domain is bug fixing and the target domain is vulnerability fixing.
- RQ3: RQ3 compares transfer learning with denoising pre-training for vulnerability fixing.The comparison focuses on the accuracy of the resulting models.
- RQ4: RQ4 examines how different data-split strategies affect transfer-learning and target-domain models.The outcome of interest is vulnerability-fixing accuracy.
4.2 Datasets
The experiments combine a newly mined large C bug-fix corpus with two independently collected vulnerability-fix datasets. These datasets support controlled training, validation, and testing of transfer learning.
- Bug Fix Corpus: The bug-fix corpus was mined from 892 million GitHub events collected between 2017-01-01 and 2018-12-31.The process filtered development activity and extracted C bug-fix commits.
- Bug Fix Corpus: 650,499 unique function-plus-specification samples remained after deduplication and token-context-diff processing.The corpus initially contained 1,838,740 function-level changes and 655,741 samples after duplicate-function removal.
- Vulnerability Fix Datasets: Big-Vul contains 3,754 vulnerabilities across 348 projects, while CVEfixes contains 5,365 vulnerabilities across 1,754 projects and 180 CWE IDs.Both datasets contain confirmed vulnerabilities with CVE IDs and are used for vulnerability-fix tuning.
- Vulnerability Fix Datasets: The two vulnerability datasets were collected in independent studies, supporting evaluation across separate sources.All research questions were conducted on both Big-Vul and CVEfixes.
4.3 Methodology for RQ1
RQ1 compares source-domain bug-fix training with target-domain vulnerability-fix training using random splits of Big-Vul and CVEfixes. Both approaches are evaluated with beam size 50 and sequence accuracy.
- Big-Vul and CVEfixes vulnerable–fixed function pairs are randomly divided into training, validation, and testing sets.The splits are mutually exclusive; Big-Vul uses 70%, 10%, and 20% proportions.
- The source-domain model trains on Btrain and validates on Bval before evaluation on vulnerability-fix test data.Evaluation uses VRepair beam size 50 for each test example.
- Sequence accuracy equals 1 when any of 50 prediction sequences matches the ground truth, and 0 otherwise.The reported metric is averaged over all test examples.
- The target-domain model trains directly on Big-Vulrand_train with early stopping on Big-Vulrand_val.Its predictions are evaluated on Big-Vulrand_test using the same beam-search procedure.
- The hypothesis is that source-domain training will have lower test sequence accuracy than target-domain training because bug fixes differ from vulnerability repairs.The comparison tests whether a large bug-fix corpus can perform as well as a small vulnerability-repair dataset.
4.4 Methodology for RQ2
RQ2 compares transfer learning against training only on the small vulnerability-fix dataset. The transfer-learning protocol first trains on bug fixes, then continues training on Big-Vul vulnerability fixes with a reduced learning rate.
- The study compares transfer learning with a model trained only on the small vulnerability-fix dataset.The comparison addresses concerns that small training data may limit deep-learning generalization.
- Transfer learning first trains models on Btrain with early stopping on Bval, then uses those models for target-domain training.The target phase continues training on Big-Vulrand_train and selects the model using Big-Vulrand_val.
- The target-domain learning rate is one tenth of the source-domain learning rate.This reduction follows the described standard practice for target-domain learning.
- Final models are evaluated on Big-Vulrand_test with VRepair beam size 50 and sequence accuracy.The same evaluation procedure is applied after target-domain training.
4.5 Methodology for RQ3
RQ3 compares transfer learning with denoising pre-training followed by vulnerability-fix fine-tuning. The denoising baseline is trained on synthetic corrupted functions derived from the bug-fix corpus.
- The comparison uses PLBART, a denoising-based pre-training technique, as the alternative to transfer learning.Both approaches are subsequently fine-tuned with target-domain vulnerability-fix training.
- Tables 4 and 5 report the numbers of top-10 CWE examples in the time-based Big-Vul and CVEfixes test splits.The supplied table captions identify dataset and split contents but do not provide outcome values.
- Denoising pre-training corrupts full functions using token masking, deletion, and infilling, with the original function as the repair target.Infilling replaces multiple consecutive tokens with one <MASK> token, with length sampled from a Poisson distribution.
- The denoising pre-training data contains 728,730 Pretrain samples and 10,000 Preval samples.Models are trained on Pretrain and stopped early using Preval.
- The final denoising models are evaluated on Big-Vulrand_test and compared with transfer-learning models.The evaluation follows the target-domain training protocol.
4.6 Methodology for RQ4
RQ4 studies how random versus time-based data splits affect vulnerability-fixing performance. The time-based setup trains on past vulnerabilities and tests on later ones.
- Big-Vul and CVEfixes are divided using random and time-based strategies.Both transfer-learning and target-only models are trained under each strategy.
- The time-based split orders each dataset by CVE publication date and assigns later vulnerabilities to testing.Big-Vul spans vulnerabilities collected from 2002 to 2019.
- Big-Vul’s time-based split contains 2,272 training, 302 validation, and 603 testing examples.Testing covers publication dates from 2018-01-01 through 2019-12-31, while validation covers 2017-06-01 through 2017-12-31.
- The time-based strategy simulates training on past vulnerability fixes and repairing future vulnerabilities.The intended proportions are roughly 70% training, 10% validation, and 20% testing.
- Test sequence accuracy is reported for every split strategy and both training regimes.The models follow the protocols used in the earlier methodology sections.
5 EXPERIMENTAL RESULTS
Across the experiments, transfer learning generally outperformed training only on vulnerability fixes, improved performance across most CWE categories, and produced more stable results across data splits. It also outperformed generic denoising pre-training and enabled fixes that target-only models missed.
- RQ1: Source versus target training: 18.24% on Big-Vulrand test and 15.98% on CVEfixesrand test were achieved by source-domain-only models, exceeding target-domain-only results of 7.86% and 11.29%.The comparison shows that a large bug-fix corpus can provide useful vulnerability-repair capability even before tuning.
- CWE-level performance: Transfer learning improved performance over source-domain training across almost all CWE categories, with exceptions including CWE-119, CWE-190, and CWE-264 in one test setting.The model favored generalization across vulnerability types over specialization, which could reduce performance for some specific categories.
- Overall performance: 21.86% on Big-Vulrand test and 22.73% on CVEfixesrand test were the highest overall accuracies achieved by transfer learning.These results surpassed target-domain-only training and supported transferring knowledge from bug fixing to vulnerability repair.
- Examples: Transfer learning correctly predicted fixes that target-only models missed, including an integer-overflow patch for CVE-2016-9754.The paper presents this example as evidence that source-domain training helps with a more sophisticated vulnerability.
- RQ3: Denoising pre-training: 21.86% versus 13.36% on Big-Vulrand test and 22.73% versus 13.34% on CVEfixesrand test favored transfer learning over denoising pre-training.The comparison used pre-training datasets of similar size and favored the task-specific bug-fixing source domain.
- RQ4: Stability across data splits: 16.23% to 22.73% for transfer learning versus 0.13% to 11.29% for target-only models showed greater stability across random and time-based splits.Target-only performance varied substantially with the data split, whereas transfer-learning performance remained in a higher range.
6 ABLATION STUDY
The ablation study examines how decoding, architecture, fault localization, repair scope, and the copy mechanism affect VRepair’s performance. Results identify beam search, Transformer architecture, fault localization, and copying as important design choices.
- Decoding: 21.86% test pass rate was achieved with beam size 50, compared with 8.96% from a single model output.The comparison uses the Big-Vulrand test dataset.
- Architecture: The Transformer model outperformed the bidirectional RNN model on the Big-Vul evaluation.
- Fault localization: 23.9% test sequence accuracy was obtained when all vulnerable lines were identified for the model.Without vulnerable-line identification, performance was described as poor; localization also reduces context-matching interpretations.
- Repair scope: 28.39% test sequence accuracy for contiguous-block repairs solved 16.42% of test data because single-block repairs comprise 57.84% of the dataset.The resulting 16.42% is below the golden model’s 21.86%.
- Vocabulary: The copy mechanism enables VRepair to handle out-of-vocabulary tokens with a vocabulary size of 5000.
7 RELATED WORK
Related work includes learning-based, template-based, program-analysis-based, and cyber-reasoning approaches to vulnerability repair. VRepair differs by learning repairs for multiple vulnerability types in C and evaluating on confirmed CVE vulnerabilities.
- Learning-based repair: Vurle learns repair templates from AST edits and was evaluated on 279 vulnerabilities, whereas VRepair uses deep learning and was evaluated on 3754 vulnerabilities.
- Learning-based repair: Other learning-based work includes GAN-based repair evaluated on synthetic vulnerabilities and DeepFix, which uses seq2seq learning to fix compiler errors.
- Learning-based repair: SeqTrans is the closest related work, using a Transformer seq2seq model with attention, copying, bug-fix pretraining, and vulnerability-fix fine-tuning for Java.VRepair instead targets C, uses copying rather than BPE, evaluates two vulnerability datasets, and represents multiline fixes compactly.
- Non-learning repair: Template- and analysis-based systems repair specific vulnerability classes using mechanisms such as symbolic execution, dynamic data-flow analysis, safety properties, constraints, static analysis, or SAT solving.Examples include IntRepair, SoupInt, Senx, ExtractFix, MemFix, and LeakFix.
- Non-learning repair: Cyber-reasoning systems such as Mayhem and Fuzzbuster generate binary patches using runtime property checking, symbolic execution, or fuzz testing.ClearView instead learns behavioral invariants from normal execution to detect and repair failures.
- VRepair’s distinction: VRepair learns arbitrary fixes across multiple vulnerability types rather than manually designed strategies for each type.Its evaluation targets confirmed vulnerabilities with CVE IDs, distinguishing them from functional-bug datasets such as Defects4J.
- Transfer learning: Transfer learning had been studied in software engineering, but the paper states that prior work had not used it to generate code fixes.
8 CONCLUSION
The paper proposes VRepair, which pretrains on a large C bug-fix corpus and fine-tunes on curated vulnerability fixes to address small vulnerability datasets. Experiments report substantially higher accuracy than training only on the small vulnerability datasets, while future work considers larger source and target datasets.
- Conclusion: VRepair first trains neural networks on a large bug-fix corpus and then fine-tunes on a curated vulnerability-fix dataset.
- Conclusion: 21.86% versus 7.86% accuracy on Big-Vul and 22.73% versus 11.29% on CVEfixes were reported for transfer learning versus small-dataset-only training.
- Future work: Future work includes using larger source-domain datasets, training on all GitHub code changes, and enlarging target datasets with vulnerabilities from issue trackers and static analyzers.