PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on Open & Industry Data

Moritz Mock, Thomas Forrer, Barbara Russo

arXiv:2509.09313v1cs.SE

TL;DR

The paper addresses limited evidence on whether vulnerability detectors developed using open-source data work on industrial code. It evaluates CodeBERT across open and industry datasets, develops AI-DO for code review, and concludes that the approach is feasible for companies seeking earlier vulnerability detection.

  • Problem

    The authors report that cross-domain evaluation between open data and industry data had not been performed, leaving CodeBERT’s transfer between these settings to be examined.

    Source: §2.2 · p. 3

  • Method

    The study evaluates CodeBERT on three consistently annotated datasets with alternative balancing strategies and develops AI-DO for integration into a company’s CI/CD code-review workflow.

    Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 6

  • Results

    The authors conclude that their approach is feasible and valuable for companies aiming to detect vulnerable code earlier in development.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The paper’s supported practical conclusion is that vulnerability detection can be shifted earlier in the development process in the studied company setting.

    Source: §6 · p. 14

    Results may be specific to the datasets, and selecting PHP for the company use case may limit broader generalizability.

    Source: §5 · p. 14

Abstract

from arXiv · show

Deep learning solutions for vulnerability detection proposed in academic research are not always accessible to developers, and their applicability in industrial settings is rarely addressed. Transferring such technologies from academia to industry presents challenges related to trustworthiness, legacy systems, limited digital literacy, and the gap between academic and industrial expertise. For deep learning in particular, performance and integration into existing workflows are additional concerns. In this work, we first evaluate the performance of CodeBERT for detecting vulnerable functions in industrial and open-source software. We analyse its cross-domain generalisation when fine-tuned on open-source data and tested on industrial data, and vice versa, also exploring strategies for handling class imbalance. Based on these results, we develop AI-DO(Automating vulnerability detection Integration for Developers' Operations), a Continuous Integration-Continuous Deployment (CI/CD)-integrated recommender system that uses fine-tuned CodeBERT to detect and localise vulnerabilities during code review without disrupting workflows. Finally, we assess the tool's perceived usefulness through a survey with the company's IT professionals. Our results show that models trained on industrial data detect vulnerabilities accurately within the same domain but lose performance on open-source code, while a deep learner fine-tuned on open data, with appropriate undersampling techniques, improves the detection of vulnerabilities.

1 Introduction

Security experts often review code manually near delivery, while static analyzers may sit outside the development pipeline. This study evaluates CodeBERT across industrial and open-source data and develops an integrated recommender to support code review.

1 Introduction

  • Security experts often manually review code near delivery and may rely on black-box static analyzers that are not integrated into the development pipeline.They may create custom rules for application-specific scanning.Source: §1 · p. 1
  • The study evaluates CodeBERT for vulnerable-function detection within and across industrial and open-source data, including strategies to mitigate class imbalance.It also examines integration into a company’s DevOps environment.Source: §1 · p. 2
  • The contributions include a cross-domain evaluation using industry and open datasets and making those datasets available..Source: §1 · p. 2
  • AI-DO supports reviewers by detecting vulnerabilities during code review, and the authors surveyed developers about the tool.The tool is designed for a company’s DevOps environment.Source: §1 · p. 2

2 Related Work

Prior work spans transformer-based vulnerability detectors and evaluations across datasets, but the reviewed literature identifies no cross-domain evaluation between open and industry data. This work examines CodeBERT trained on open data and applied to industry code.

2.1 AI Models for Vulnerability Detection

  • Transformer-based vulnerability detectors include CodeBERT-based LineVul and LineVD, with LineVD framing statement-level detection as node classification over program dependency graphs.The reviewed work also describes CodeBERT’s code-focused pretraining and LineVul’s function- then line-level detection.Source: §2.1 · p. 3

2.2 Cross-Domain Performance Evaluation of Datasets

  • Evaluations across datasets can reveal performance loss, with prior work reporting a drop of up to 91% after a dataset shift.The passage also notes that combining multiple SAST tools increases vulnerability-detection accuracy in Java projects.Source: §2.2 · p. 3
  • The authors report no prior cross-domain evaluation between open data and industry data, and study CodeBERT trained on open data and applied to industry data [7].Fig. 1 depicts dataset creation by fusing two static-analyzer annotation types across 40 open-source and 8 industry projects.Source: §2.2 · pp. 3–4, Figure 1 · p. 4

3 Methodology

The methodology creates consistently annotated industrial and open-source PHP datasets, fine-tunes CodeBERT using alternative balancing strategies, and evaluates performance within and across domains.

3.1 Dataset creation and annotation

  • Three datasets were created because no publicly available PHP-function dataset used a consistent vulnerability-annotation methodology across open and industrial code.They comprise one industry dataset, one technology-agnostic open-source dataset, and one technology-similar open-source dataset.Source: §3.1 · p. 4
  • Functions were extracted with TreeSitter, deduplicated, and annotated using SemGrep and SonarQube, with Info or Minor and Warning findings excluded as vulnerability indicators.PMD-CPD used a 30-token window; functions with at least 99% Jaccard similarity were removed.Source: §3.1 · pp. 4–5
  • Dataset quality was addressed through accuracy, uniqueness, consistency, completeness, and currentness checks using shared analyzer versions and settings across datasets.Open-source data were aligned in time with the industry dataset, and function records were traceable to source locations.Source: §3.1 · p. 5

3.2 CodeBERT fine-tuning

  • CodeBERT was fine-tuned for 10 epochs with an 80%/10%/10% train-validation-test split and four training strategies: no balancing, two undersampling approaches, and weighted loss.Validation and test sets retained their natural class distributions.Source: §3.2 · p. 6

3.3 Cross-Domain Evaluation

  • Performance was assessed on test data using precision, recall, and F1 across balancing techniques.Cross-domain analysis tests open-data fine-tuning on the Industry Dataset and compares it with same-dataset performance.Source: §3.3 · pp. 6–7
  • CodeBERT is fine-tuned separately on the Industry Dataset, Generic Open-source Dataset, and Technology-similar Open-source Dataset, which support study of learning transfer.The evaluation tests models trained on the two open datasets against the Industry Dataset and compares results with same-dataset performance.Source: §3.1 · p. 4, §3.3 · p. 7

4 Case Study

The case study evaluates CodeBERT across open-source and industrial PHP data, then integrates an open-data-trained model into AI-DO. Technology-agnostic open-source training with undersampling better controlled false negatives on unseen industrial data, with false-positive tradeoffs.

4.1 Data collection

  • The case study used a representative PHP ERP project at Würth Phoenix, alongside GOD (209,532 functions), TOD (240,876), and company ID (65,385).Source: §4 · p. 7, §4.1 · p. 8

4.2 Cross-Domain Performance Evaluation

  • For same-dataset tests, NB or WLF favored F1 score and precision, while undersampling was generally preferred to increase recall.Source: §4.2 · pp. 9–10

4.3 AI-DO

  • AI-DO integrates CodeBERT into a GitHub Action that extracts changed functions with git diff and TreeSitter, then tags vulnerable functions on pull requests.The fine-tuned model runs offline and can be replaced or periodically updated.Source: §4.3 · p. 11
  • Among 13 surveyed professionals, enthusiasm varied with vulnerability-detection experience and role-specific needs; architects wanted vulnerability-type details and reasoning.Source: §4.3 · pp. 11–12

4.4 Answers to the Research Questions

  • Company-trained CodeBERT performed comparatively well on unseen industrial data, with at most a 10% F1-score decrease versus open-source-trained and tested CodeBERT; weighted loss narrowed the gap.Source: §4.4 · pp. 12–13
  • URSC undersampling on technology-agnostic open-source data controlled false negatives on unseen industrial vulnerabilities better than same-system industrial training, but increased false positives.The authors note that false positives may burden teams with unnecessary investigations and reduce trust in AI-DO.Source: Figure 2 · p. 9, §4.4 · p. 13

5 Threats to Validity

The validity discussion addresses whether the experimental setup supports comparing portability across open and industry data, and identifies limits on generalizability and interpretation.

5 Threats to Validity

  • The datasets used the same annotation policy and model setup, while sample-size and vulnerability-count differences were addressed through balancing and weighting.Source: §5 · p. 13
  • Results may be dataset-specific, and using PHP as the target language may limit broader generalizability.Source: §5 · p. 14
  • Possible shared code between open and industry datasets could not be identified, although duplicate code was mitigated with PMD-CPD.Source: §5 · p. 14
  • Survey results may favor the work because the first author had prior contact with participants; replication would strengthen those findings.Source: §5 · p. 14

6 Conclusion and Further Work

The study evaluates CodeBERT across industrial and open-source vulnerability data and develops AI-DO for reviewers. Industrial-trained models performed poorly on general open-source code, while open-data training with undersampling improved industrial vulnerability detection.

6 Conclusion and Further Work

  • Industrial-trained models performed well in-domain but poorly on general open-source code, while open-data training with undersampling increased vulnerability detection in industrial data.Source: §6 · p. 14
  • The authors conclude the approach is feasible and valuable for companies seeking to detect vulnerable code earlier in development.Source: §6 · p. 14
  • Surveyed developers were open-minded about AI-DO but raised concerns about trust and vulnerability localization.Source: §6 · p. 14
  • Three datasets—one company dataset and two open-source datasets—shared an annotation policy; individual fine-tuning and balancing strategies supported transfer analysis.Source: §6 · p. 14
  • Future work includes multiple programming languages, transformer models, and annotation types, leveraging the new multi-annotation dataset.Source: §6 · p. 14
Loading 2509.09313v1…