PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically Enhanced Software Traceability Using Deep Learning Techniques

Jin Guo, Jinghui Cheng, Jane Cleland-Huang

arXiv:1804.02438v1cs.SE

TL;DR

Software traceability is laborious and error prone, and existing automated methods can miss semantic relationships between artifacts. The paper uses word embeddings and an RNN tracing network to learn artifact semantics from domain data; BI-GRU achieved higher MAP than VSM and LSI on an industrial PTC dataset.

  • Problem

    Automated traceability methods struggle with artifact semantics and domain knowledge, while manual trace-link construction is time consuming and error prone.

  • Method

    The approach learns domain word embeddings and trains an RNN tracing network on validated trace links to predict links between software artifacts.

  • Results

    BI-GRU achieved MAP .598, 41% higher than VSM (MAP = .423) and 32% higher than LSI (MAP = .451).

  • Takeaways & Limitations

    Given an initial training set of trace links, the tracing network provides a fully automated and highly scalable approach for industrial trace-link generation.

  • Takeaways & Limitations

    The evaluation focused on one Positive Train Control domain and cannot establish generalizability because dataset characteristics may affect effectiveness.

Abstract

from arXiv · show

In most safety-critical domains the need for traceability is prescribed by certifying bodies. Trace links are generally created among requirements, design, source code, test cases and other artifacts, however, creating such links manually is time consuming and error prone. Automated solutions use information retrieval and machine learning techniques to generate trace links, however, current techniques fail to understand semantics of the software artifacts or to integrate domain knowledge into the tracing process and therefore tend to deliver imprecise and inaccurate results. In this paper, we present a solution that uses deep learning to incorporate requirements artifact semantics and domain knowledge into the tracing solution. We propose a tracing network architecture that utilizes Word Embedding and Recurrent Neural Network (RNN) models to generate trace links. Word embedding learns word vectors that represent knowledge of the domain corpus and RNN uses these word vectors to learn the sentence semantics of requirements artifacts. We trained 360 different configurations of the tracing network using existing trace links in the Positive Train Control domain and identified the Bidirectional Gated Recurrent Unit (BI-GRU) as the best model for the tracing task. BI-GRU significantly out-performed state-of-the-art tracing methods including the Vector Space Model and Latent Semantic Indexing.

I. Introduction

The paper addresses inaccurate and labor-intensive software traceability by using deep learning to capture artifact semantics and domain knowledge. Its tracing network learns word and artifact representations from domain data and generates trace-link probabilities.

  • Motivation: Manual trace-link construction is time consuming and error prone, while automated methods often struggle with semantic mismatches between related artifacts.A PTC example requires domain knowledge to connect differently worded requirements and design artifacts.
  • Motivation: DoCIT improved PTC trace retrieval over VSM but required a hand-built ontology, manually defined heuristics, and a conventional parser.These dependencies made DoCIT effective but fragile and difficult to transfer to new domains.
  • Approach: The proposed approach first learns domain word embeddings, then trains an RNN tracing network on validated links to compare artifact semantics and predict link probabilities.Words become vector representations, RNN outputs represent artifact semantics, and the network evaluates pairs of artifacts.
  • Scope: The approach requires an initial set of manually constructed trace links and therefore cannot be used in an entirely green field domain.The paper focuses on using an initial project training set to automate additional links as the project proceeds.
  • Evaluation: The study evaluates RNN configurations and compares the best configuration against standard trace-retrieval baselines on an industrial PTC dataset.The research questions concern RNN configuration and whether it improves trace-link accuracy over baseline techniques.

II. Deep Learning for Natural Language Processing

Deep learning represents language through layered nonlinear computation, while word embeddings encode distributional relationships and RNNs process sequences using recurrent hidden states. Standard RNNs remain difficult to train when sequences contain long dependencies.

  • Deep learning: Deep learning uses multilayer nonlinear processing to learn increasingly abstract data representations rather than relying solely on manually selected features.Backpropagation adapts internal parameters so layers compute useful representations.
  • Word embeddings: Word embeddings represent words as continuous high-dimensional vectors in which similar words are close and linguistic relationships can be encoded.Embedding models learn from unlabeled text using word co-occurrence statistics.
  • Recurrent neural networks: RNNs process sequences by combining the current input with the previous hidden state, allowing outputs to depend on current and prior inputs.The same recurrent unit and parameters are reused across time steps.
  • Recurrent neural networks: Standard RNNs degrade on long dependencies because exploding or vanishing gradients make them difficult to train.Gradient clipping addresses exploding gradients, while LSTM and GRU variants target vanishing gradients.

D. Long Short Term Memory (LSTM)

LSTM extends recurrent networks with a memory cell and gates that regulate information flow. This design preserves selected information over longer dependencies and motivated its use for semantic reasoning.

  • LSTM architecture: LSTM recurrent units include a memory cell vector and gates that control when information is read from or written to memory.The sigmoid-based gates regulate information throughput through point-wise multiplication.
  • LSTM architecture: The input and forget gates combine a memory candidate with retained prior memory to update the cell state.The input gate controls what candidate information is remembered, while the forget gate controls what prior information remains.
  • LSTM architecture: The output gate controls how the retained memory cell state contributes to the unit output.The paper describes this as the final stage of the LSTM unit computation.
  • LSTM behavior: LSTM retains information until the forget gate erases it, enabling more effective handling of long-term dependencies.Its prior success on semantic-relatedness tasks motivated adoption for tracing semantics.
  • LSTM and GRU comparison: Figure 2 contrasts LSTM and GRU units by showing LSTM’s input, forget, and output gates with its memory cell alongside GRU’s reset and update gates.The comparison highlights the distinct internal structures of the two recurrent-unit types.

E. Gated Recurrent Unit (GRU)

GRU controls recurrent information flow with a simpler gated unit that lacks a dedicated memory cell. Its update mechanism blends prior and candidate outputs, embedding long-term information in hidden states.

  • GRU architecture: GRU uses reset and update gates to control information flow without a dedicated memory cell vector.This gives GRU a simpler unit structure than LSTM.
  • GRU architecture: The reset gate filters the previous hidden output before the unit computes a candidate output from that result and the current input.The candidate output is based on gated prior information together with the current input.
  • GRU architecture: The update gate linearly interpolates between the previous output and the candidate output.It balances how much the current output is updated from each source.
  • GRU behavior: GRU embeds long-term information directly into hidden output vectors and has achieved competitive NLP results with LSTM.The paper compares GRU and LSTM to identify the more suitable model for tracing.

F. Other RNN Variables

The tracing network converts source and target artifacts into semantic vectors and compares them to predict trace-link probabilities. It explores bidirectional and multilayer RNN designs for generating those representations.

  • F. Other RNN Variables: Bidirectional RNNs process artifact sequences in both forward and backward directions, allowing outputs to use past and future sequence information.
  • F. Other RNN Variables: The tracing process ranks candidate target links for each source artifact so a human analyst can review and accept the correct links.
  • F. Other RNN Variables: Word Embedding layers map artifact words to vectors, while RNN layers transform each artifact sequence into a semantic vector.
  • F. Other RNN Variables: Semantic Relation Evaluation layers compare the source and target semantic vectors and produce the probability that a valid trace link exists.
  • F. Other RNN Variables: Network configurations vary the RNN unit type, hidden dimensions, layer count, bidirectionality, and other RNN settings.

B. Training the Tracing Network

The tracing network is trained as a regularized categorical prediction model using validated artifact-pair labels and stochastic gradient descent. Training behavior is controlled through optimization hyper-parameters and repeated epochs.

  • B. Training the Tracing Network: The network minimizes regularized negative log likelihood as its objective loss function.
  • B. Training the Tracing Network: Training examples are labeled as links or non-links, and the loss evaluates the probability assigned to each example’s correct category.
  • B. Training the Tracing Network: L2 parameter regularization is included to prevent overfitting, with λ controlling the regularization strength.
  • B. Training the Tracing Network: Stochastic gradient descent updates network parameters across epochs until the loss becomes sufficiently small or stops decreasing.
  • B. Training the Tracing Network: Hyper-parameters include learning rate, gradient clipping, regularization strength, mini-batch size, and epoch count.

A. Data Preparation

Data preparation combines domain text for word embeddings with Positive Train Control artifact pairs for tracing-network training and evaluation. The extreme scarcity of valid links requires balanced sampling during training.

  • A. Data Preparation: The embedding corpus combines 52.7MB of PTC-domain text with an optional 19.92GB Wikipedia dump.
  • A. Data Preparation: The PTC dataset contains 1,651 Software Subsystem Requirements as source artifacts and 466 Software Subsystem Design Descriptions as targets.
  • A. Data Preparation: Only 0.18% of 769,366 artifact pairs are valid links, creating a highly unbalanced training dataset.
  • A. Data Preparation: Each training epoch includes all valid links and an equal randomly selected number of non-links to construct a balanced training set.
  • A. Data Preparation: Initial experiments found the dynamic balanced-sampling technique effective for training the tracing network.

B. Model Selection and Hyper-Parameters Optimization

The study selects network configurations and hyper-parameters through a constrained search, then evaluates the best tracing network against optimized VSM and LSI baselines. MAP is computed at full recall, alongside precision–recall analysis.

  • B. Model Selection and Hyper-Parameters Optimization: Because exhaustive configuration search was too costly, the study first manually identified configurations expected to perform well.
  • B. Model Selection and Hyper-Parameters Optimization: Word embeddings were trained as either 50-dimensional PTC-only vectors or 300-dimensional vectors trained on PTC plus Wikipedia text.
  • B. Model Selection and Hyper-Parameters Optimization: The RNN comparison evaluated GRU, LSTM, BI-GRU, and BI-LSTM variants with one or two layers, using 30 or 60 hidden dimensions.
  • B. Model Selection and Hyper-Parameters Optimization: Mean Average Precision measures how highly valid links are ranked across source-artifact queries.
  • B. Model Selection and Hyper-Parameters Optimization: MAP is computed on the test dataset after returning all valid links, corresponding to 100% recall.
  • B. Model Selection and Hyper-Parameters Optimization: The tracing network is compared with VSM and LSI, whose preprocessing and parameters are also optimized using a Genetic Algorithm.
  • B. Model Selection and Hyper-Parameters Optimization: Precision–recall curves show trade-offs across similarity or probability thresholds, with curves farther from the origin indicating better performance.

V. Results and Discussion

The evaluation identifies BI-GRU as the strongest tracing-network configuration and shows that domain-specific word embeddings can support effective, computationally economical tracing.

  • The study reports the best configurations for each RNN unit type and compares the tracing network against VSM and LSI.It also examines performance with a larger training set.
  • BI-GRU achieved the overall best performance, while GRU and BI-GRU converged faster and reached smaller loss than LSTM and BI-LSTM.Bidirectional models also performed slightly better than their unidirectional counterparts on training and development data.
  • Three of the four best configurations trained word embeddings on the PTC corpus alone.The authors suggest that general Wiki contexts can weaken domain-specific semantic reasoning when training data is insufficient for disambiguation.

B. Does the tracing network outperform leading trace retrieval algorithms?

The BI-GRU tracing network outperformed VSM and LSI in mean average precision and at higher recall levels. Gate behavior illustrates how GRU dimensions process sequential artifact words into semantic representations, although semantic associations can still produce false positives.

  • .598 MAP was achieved by the tracing network, 41% higher than VSM and 32% higher than LSI.The differences were statistically significant for both comparisons, with p < .001.
  • The tracing network outperformed VSM and LSI at higher levels of recall, with precision improving notably above 0.2 recall.The authors attribute this improvement to extracting semantic information and reasoning over artifact associations.
  • GRU gate behavior varies across sentence-semantic-vector dimensions as words are processed sequentially.Figure 6 tracks reset gates, update gates, and outputs for the 24th and 12th dimensions.
  • The network can assign high probability to invalid links when artifacts share valid associations but differ in domain-specific meaning.One example received 97.27% probability despite referring to external rather than internal events.
  • Precision reached a glass ceiling above 0.27 because some false-positive links could not be ruled out.

C. How does the tracing network react to more training data?

Increasing the training set from 45% to 80% improved tracing performance on the smaller test set. The tracing network remained significantly better than VSM and LSI, and the authors conclude that larger training sets yield greater accuracy improvements.

  • The experiment increased the training dataset to 80% while retaining a 10% test set for evaluation.The process was repeated five times to reduce the effect of random data selection.
  • .834 MAP with the larger training set exceeded .803 with the smaller training set.The larger training set also improved both precision and recall, especially at low recall levels.
  • The tracing network significantly outperformed VSM and LSI on the larger training-set division, achieving MAP = .834 versus .625 and .637.Pairwise Wilcoxon tests used Bonferroni p-value adjustments, with p < .001 for both comparisons.
  • The authors conclude that trace-link accuracy generally improves over VSM and LSI, with larger training sets producing more marked improvements.They expect further improvements from reconfiguring the tracing network for larger training sets.

VII. Threats to Validity

The evaluation has two main validity threats: results come from one Positive Train Control domain, and the evaluation trace matrix may not be completely correct. The authors also describe dataset-specific factors that could affect effectiveness and plan evaluation on additional datasets.

  • Threats to Validity: The study focused on a single Positive Train Control domain because large industrial datasets and experimentation across algorithms were difficult to obtain.The authors therefore state that they cannot claim generalizability.
  • Threats to Validity: Dataset size, vocabulary characteristics, and individual artifact properties may have made the approach more or less effective.The PTC dataset contained regulatory text authored by multiple engineers and developers, but its specific characteristics may still have influenced the results.
  • Threats to Validity: The authors plan to evaluate the approach on additional datasets and improve precision using more representative negative training examples.The conclusion also reports higher MAP, precision, and recall than VSM or LSI on the studied industrial dataset.
  • Threats to Validity: The authors cannot guarantee that the trace matrix used for evaluation is 100% correct.It was supplied by industrial collaborators and used in their project to demonstrate coverage of regulatory codes.
  • Threats to Validity: MAP, Recall, and Precision are accepted research standards, and comparisons used VSM and LSI configured with a Genetic Algorithm.These choices were intended to avoid comparison against a weak baseline.
Loading 1804.02438v1…