PaperlayerView exact revision on arXiv

Source-linked AI summary

Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models

Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang, Jane Cleland-Huang

arXiv:2102.04411v2cs.SE

TL;DR

The paper addresses inaccurate and costly automated traceability under limited labeled data and runtime constraints. It proposes T-BERT, which transfers knowledge from code search through staged pre-training, intermediate-training, and traceability fine-tuning. Single-BERT is most accurate, while Siamese-BERT offers comparable accuracy with faster execution, and all T-BERT models outperform classical IR baselines.

  • Problem

    Automated traceability is constrained by limited labeled links and the high training and execution costs of deep-learning models.

  • Method

    T-BERT uses BERT relation classifiers and transfers knowledge from code search through intermediate-training before traceability fine-tuning.

  • Results

    Single-BERT achieved the best accuracy, Siamese-BERT delivered comparable accuracy with faster execution, and all three T-BERT models outperformed classical IR baselines.

  • Takeaways & Limitations

    Intermediate-trained T-BERT models can be applied to software-engineering traceability despite limited task-specific training data.

  • Takeaways & Limitations

    Experiments covered only Python projects and three OSS projects, limiting generalization to other languages and projects.

Abstract

from arXiv · show

Software traceability establishes and leverages associations between diverse development artifacts. Researchers have proposed the use of deep learning trace models to link natural language artifacts, such as requirements and issue descriptions, to source code; however, their effectiveness has been restricted by availability of labeled data and efficiency at runtime. In this study, we propose a novel framework called Trace BERT (T-BERT) to generate trace links between source code and natural language artifacts. To address data sparsity, we leverage a three-step training strategy to enable trace models to transfer knowledge from a closely related Software Engineering challenge, which has a rich dataset, to produce trace links with much higher accuracy than has previously been achieved. We then apply the T-BERT framework to recover links between issues and commits in Open Source Projects. We comparatively evaluated accuracy and efficiency of three BERT architectures. Results show that a Single-BERT architecture generated the most accurate links, while a Siamese-BERT architecture produced comparable results with significantly less execution time. Furthermore, by learning and transferring knowledge, all three models in the framework outperform classical IR trace models. On the three evaluated real-word OSS projects, the best T-BERT stably outperformed the VSM model with average improvements of 60.31% measured using Mean Average Precision (MAP). RNN severely underperformed on these projects due to insufficient training data, while T-BERT overcame this problem by using pretrained language models and transfer learning.

I. INTRODUCTION

Automated traceability remains limited by semantic gaps, scarce labeled links, and deep-learning runtime costs. T-BERT applies BERT-based relation classification to generate more accurate, trustworthy links between natural-language and programming-language artifacts.

  • Manual trace-link creation is costly, leaving traceability data incomplete, inaccurate, and underused.
  • Existing automated methods often rely on word matching and perform poorly when related artifacts have a semantic gap.
  • Limited project-specific golden links constrain deep-learning training, while multi-layer neural networks are slower than traditional retrieval methods.
  • T-BERT uses BERT-based relationship classifiers to link natural-language artifacts such as issues with programming-language artifacts such as change sets.
  • The study compares single, twin, and siamese architectures for NLA-PLA traceability.

II. PROBLEM STATEMENT

The paper frames architecture selection, training improvement, and knowledge transfer as questions for improving NLA-PLA traceability. It investigates whether code search can provide resource-rich intermediate-training data for the downstream task.

  • BERT pre-training transfers contextual knowledge to specialized downstream tasks through fine-tuning.
  • RQ1 asks which single, twin, or siamese BERT relation classifier best balances accuracy and efficiency for NLA-PLA traceability.
  • The study investigates training techniques intended to improve accuracy without reaching a low-performance glass ceiling.
  • RQ3 asks whether knowledge from a resource-rich retrieval task can improve downstream NLA-PLA tracing.
  • Intermediate-training uses paired doc-strings and functions from code search because these artifacts provide ample training data.

III. APPROACH

T-BERT uses a three-stage pipeline: code-language pre-training, code-search intermediate-training, and traceability fine-tuning. Three relation-classifier architectures encode artifact pairs differently before classification.

  • A. Introduction to BERT and Language Models: BERT learns contextual token representations through transformer layers and masked language modeling before downstream fine-tuning.
  • A. Introduction to BERT and Language Models: The pipeline first trains a code language model, then intermediate-trains T-BERT on code search, and finally fine-tunes it for traceability.
  • C. T-BERT Architectures: Twin encodes natural-language and programming-language artifacts separately, pools their representations, concatenates them, and classifies the joint vector.
  • C. T-BERT Architectures: Siamese uses one BERT sequentially for the two artifacts, then pools and concatenates their representations for classification.
  • C. T-BERT Architectures: Single concatenates annotated natural-language and programming-language text into one sequence before BERT encoding and pooling.

IV. MODEL TRAINING

Training combines multilingual code-search data, OSS issue-commit data, code-language pre-training, and intermediate binary classification. The resulting models are evaluated using mined trace links and dynamic negative sampling.

  • A. Three Step Training •: Pre-training and intermediate-training use function definitions paired with doc-strings from CodeSearchNet across six programming languages.
  • A. Three Step Training •: Fine-tuning data come from OSS issues and commits, with ground-truth links mined from issue identifiers in commit messages.
  • A. Three Step Training •: Code BERT is pre-trained by masking 15% of tokens and recovering them from surrounding context.
  • A. Three Step Training •: The study uses a released pretrained code BERT model as the source language model for T-BERT relation classifiers.
  • A. Three Step Training •: Intermediate-training classifies whether a doc-string correctly describes its paired function.
  • A. Three Step Training •: Dynamic sampling balances positive and negative code-search pairs and refreshes negative examples each epoch.

B. Negative Sampling

The authors identify overfitting from random negative sampling as a cause of declining trace-link accuracy and propose Online Negative Sampling to provide higher-quality negatives dynamically.

  • B. Negative Sampling: Trace-link accuracy can peak and then decline as training continues, producing a performance glass ceiling.The authors attribute this decline to overfitting on mostly easy negative examples.
  • B. Negative Sampling: Online Negative Sampling adds high-quality negative samples to alleviate the overfitting problem.
  • B. Negative Sampling: ONS generates negative examples dynamically at the batch level instead of creating the training dataset at each epoch's beginning.

V. EXPERIMENTAL EVALUATION

The experiments use CodeSearchNet and three GitHub projects to evaluate T-BERT on code search and issue-to-commit traceability, with links mined from commit and pull-request metadata.

  • V. EXPERIMENTAL EVALUATION: The evaluation includes CodeSearchNet functions and docstrings, focusing on Python functions.CodeSearchNet is publicly available and covers six programming languages.
  • V. EXPERIMENTAL EVALUATION: The traceability datasets come from the Python projects Pgcli, Flask, and Keras, using issues as sources and commits as targets.The projects were selected because they are popular, actively maintained, and contain issue identifiers on commits.
  • V. EXPERIMENTAL EVALUATION: Golden links were mined from issue tags embedded in commit messages and from accepted pull requests connecting issues and commits.
  • V. EXPERIMENTAL EVALUATION: The mined link set may be incomplete, so the authors prune the dataset to retain artifacts appearing in the link set.
  • V. EXPERIMENTAL EVALUATION: The implementation used PyTorch 1.6.0, HuggingFace Transformers 2.8.0, and one NVIDIA GeForce GTX 1080 Ti GPU.

C. Evaluation Metrics

The study evaluates traceability with F-scores and MAP@3, while using MRR and Precision@K for ranking and user-oriented code-search assessments.

  • C. Evaluation Metrics: F-scores combine precision and recall, with F2 preferred when recall matters more than precision.The best F-scores are reported by enumerating thresholds.
  • C. Evaluation Metrics: MAP measures the ranking of relevant artifacts, while MAP@3 counts only artifacts ranked in the top three positions.
  • C. Evaluation Metrics: MRR averages the reciprocal rank of the first related target artifact for each query.It is the standard metric for the CodeSearchNet benchmark.
  • C. Evaluation Metrics: Precision@K measures how many related artifacts appear in the top K results, with K values from 1 to 3.Higher Precision@K indicates users are more likely to find at least one related target artifact within those results.
  • C. Evaluation Metrics: Because MRR and Precision@K ignore recall, the traceability evaluation uses only F-score and MAP@3.Since most queries have fewer than three correct links, a perfect MAP@3 score approximates 100% recall.

VI. RESULTS AND DISCUSSION

The results compare six T-BERT variants with classical tracing methods and TraceNN on CodeSearchNet, using accuracy, efficiency, and learning-curve analyses.

  • VI. RESULTS AND DISCUSSION: The first evaluation tests T-BERT architectures on datasets with adequate labeled examples.
  • VI. RESULTS AND DISCUSSION: Six T-BERT variants are compared with VSM, LDA, LSI, and TraceNN on the CodeSearchNet dataset.The variants differ by relation-classifier architecture and negative-sampling technique.
  • VI. RESULTS AND DISCUSSION: Each CodeSearchNet query pairs a docstring's correct function with 999 unrelated functions, requiring retrieval from 1,000 candidates.
  • VI. RESULTS AND DISCUSSION: Table II reports T-BERT performance on the CodeSearchNet Challenge dataset, while Table III reports training and testing time.Testing time in Table III is recorded for a test set containing 100 queries.
  • VI. RESULTS AND DISCUSSION: Learning curves were examined over the first 35,000 optimization steps, with evaluations every 1,000 steps on 200-example development sets.

B. Evaluate NLA-PLA Traceability

T-BERT models effectively address NLA-PLA traceability, outperforming information-retrieval baselines while exposing a trade-off between accuracy and evaluation efficiency. SINGLE* is most accurate, whereas SIAMESE and TWIN are substantially faster and more scalable.

  • CodeSearchNet: SINGLE*, TWIN*, and SIAMESE* achieve F scores around 0.6, MAP scores around 0.8, and Precision@3 around 0.9 on CodeSearchNet.The models return related functions in around 9 of 10 queries, with correct answers ranked first in 75% to 80% of cases.
  • CodeSearchNet: SINGLE* achieves the best performance across all CodeSearchNet metrics, although the gap among the three T-BERT* architectures is small.All three T-BERT* models clearly outperform the VSM, LSI, and LDA baselines.
  • NLA-PLA traceability: All T-BERT* models outperform information-retrieval baselines on the NLA-PLA traceability challenge, with SIAMESE* and SINGLE* reaching F scores above 0.95 and MAP of 0.99 on Keras.The Keras result indicates that T-BERT can provide perfect tracing results in some scenarios.
  • Efficiency: SINGLE requires around 20,000 seconds to evaluate 100 queries, compared with around 3,000 seconds for TWIN and SIAMESE.For the full CodeSearchNet test set, SINGLE took an estimated 6,000 hours, while TWIN and SIAMESE took around 20 hours in practice.
  • Efficiency: SINGLE has O(N^2 ∗K) feature-vector construction complexity, whereas TWIN and SIAMESE use O(N ∗K) plus O(N^2), making them one order of magnitude more scalable.SIAMESE is identified as the most appropriate architecture for NLA-PLA traceability because it approaches SINGLE accuracy while retaining TWIN-like low time complexity.
  • Comparison with prior models: TNN underperforms all BERT models and VSM on all three evaluated open-source projects.The study attributes the RNN model's poor performance partly to insufficient training data and the open vocabulary of programming languages.

E. RQ3: To what extent can T-BERT leverage transfer knowledge from code search to software traceability

Intermediate training on code search transfers knowledge to improve T-BERT traceability performance, with architecture-dependent convergence behavior.

  • A T-BERT model trained with transferred knowledge was compared against a model without transferred knowledge in Table IV.
  • F2 increased from 0.654 to 0.748 and MAP increased from 0.728 to 0.779 for SIAMESE on Pgcli after intermediate training.Similar improvements were observed across other datasets and T-BERT architectures.
  • SINGLE converged earliest, SIAMESE more slowly, and TWIN slowest, indicating different capacities for transferring knowledge.The convergence comparison is shown in Fig. 5.

VII. RELATED WORK

Related work spans neural, information-retrieval, heuristic, and machine-learning approaches to software traceability and code search, while prior studies had not compared the three T-BERT architectures.

  • No prior study had compared TWIN, SIAMESE, and SINGLE architectures.
  • Code-search research included an RNN approach that reported 0.6 MRR on a test set with 100 queries.

VIII. THREATS TO VALIDITY

The study’s validity is constrained by its Python-only experiments, three-project evaluation, and potentially incomplete OSS trace links; replication materials and broader future evaluations are provided.

  • Experiments covered only Python projects, and evaluation used only three OSS projects, limiting generalizability across languages and projects.
  • Mining explicitly related issue and commit IDs may miss true links involving hidden dependencies among related issues.
  • The study reports SINGLE as most accurate but slower, while SIAMESE provides similar accuracy with faster execution.
  • Datasets, a complete T-BERT implementation, and execution instructions are provided to support replication and reproducibility.
  • Future work will evaluate more diverse project domains, programming languages, and software artifact types.
Loading 2102.04411v2…