PaperlayerView exact revision on arXiv

Source-linked AI summary

What Works Better? A Study of Classifying Requirements

Zahra Shakeri Hossein Abad, Oliver Karras, Parisa Ghazi, Martin Glinz, Guenther Ruhe, Kurt Schneider

arXiv:1707.02358v1cs.SE

TL;DR

Automated classification of natural-language requirements remains challenging because terminology and sentence structures vary. The paper preprocesses requirements and evaluates classifiers for FR/NFR classification and NFR sub-categories. Preprocessing improved both tasks, and classifier performance differed across sub-classification methods.

  • Problem

    Variation in terminology and sentence structure makes automated classification of natural-language requirements into FRs and NFR sub-categories challenging.

    Source: §I · p. 1

  • Method

    The paper applies rules to standardize and normalize requirements, then evaluates machine-learning methods for FR/NFR classification and NFR sub-classification.

    Source: §I · p. 1

  • Results

    FR/NFR accuracy rose from 89.92% to 95.04%; preprocessing also improved NFR sub-classification, where BNB performed best and BTM did not perform well.

    Source: 3) Na¨ıve Bayes Classification · p. 6, §VII · p. 6

  • Takeaways & Limitations

    Preprocessing improves performance on both FR/NFR classification and NFR sub-category classification in this study.

    Source: §VII · p. 6

    The dataset lacks several requirement types, including compatibility, compliance, safety, and cultural, environmental, or physical constraints; only one requirement is labeled PO.

    Source: §VI · p. 6

Abstract

from arXiv · show

Classifying requirements into functional requirements (FR) and non-functional ones (NFR) is an important task in requirements engineering. However, automated classification of requirements written in natural language is not straightforward, due to the variability of natural language and the absence of a controlled vocabulary. This paper investigates how automated classification of requirements into FR and NFR can be improved and how well several machine learning approaches work in this context. We contribute an approach for preprocessing requirements that standardizes and normalizes requirements before applying classification algorithms. Further, we report on how well several existing machine learning methods perform for automated classification of NFRs into sub-categories such as usability, availability, or performance. Our study is performed on 625 requirements provided by the OpenScience tera-PROMISE repository. We found that our preprocessing improved the performance of an existing classification method. We further found significant differences in the performance of approaches such as Latent Dirichlet Allocation, Biterm Topic Modeling, or Naive Bayes for the sub-classification of NFRs.

I. INTRODUCTION

The paper addresses the difficulty of classifying inconsistent natural-language requirements and studies preprocessing and machine-learning approaches to FR/NFR classification. Prior work covers automated classification and NFR extraction, while this study compares additional methods and their precision and recall.

I. INTRODUCTION

  • Different terminology and sentence structures make automated classification of natural-language requirements into FRs and NFR sub-categories challenging and error-prone.Source: §I · p. 1
  • The study tests preprocessing rules that standardize and normalize requirements, and evaluates machine-learning methods for classifying NFRs into sub-categories.Source: §I · p. 1

II. RELATED WORK

  • Knauss and Ott compared manual, semi-automatic, and fully automatic classification, reporting the semi-automatic approach as the best balance of quality and effort.Source: §II · p. 1
  • Prior work studied NFR extraction and quality concerns; this paper additionally compares classifiers’ precision and recall for NFR sub-categories.Source: §II · pp. 1–2

III. THE CHALLENGE AND RESEARCH QUESTIONS

The study uses a labeled repository dataset to examine FR/NFR classification and NFR sub-classification. Its research questions focus on preprocessing sentence characteristics and the effect of classifier choice.

A. Context and Data Set

  • The OpenScience tera-PROMISE dataset contains 625 labeled requirements: 255 FRs and 370 NFRs, with eleven NFR sub-categories.Source: A. Context and Data Set · p. 2

B. Research Questions

  • RQ1 asks whether preprocessing grammatical, temporal, and sentimental sentence characteristics affects FR/NFR classification precision and recall.Source: B. Research Questions · p. 2
  • RQ2 asks how the chosen machine-learning method affects precision and recall when classifying NFRs into the dataset’s sub-categories.Source: B. Research Questions · p. 2

IV. PREPROCESSING OF REQUIREMENTS SPECIFICATIONS

The preprocessing aims to reduce inconsistency in requirements specifications using sentence features and co-occurrence relations. It includes POS tagging and abstraction of context-specific users and products.

IV. PREPROCESSING OF REQUIREMENTS SPECIFICATIONS

  • The preprocessing combines rich sentence features and latent co-occurrence relations to reduce inconsistency in requirements specifications.Source: §IV · p. 2

A. Part Of Speech (POS) Tagging

B. Entity Tagging

  • Entity tagging replaces context-specific product and user names with PRODUCT and USER, then reduces associated noun phrases to those labels.Source: B. Entity Tagging · p. 2

C. Temporal Tagging

Temporal tagging is motivated by the relevance of time to several NFR types. The preprocessing recognizes temporal expressions and normalizes patterns in requirements.

C. Temporal Tagging

  • Time helps characterize availability, fault tolerance, and performance requirements, and SUTime recognizes and normalizes temporal expressions using the TIMEX3 standard.Source: C. Temporal Tagging · p. 2

DURAT ION

  • Duration examples include availability for 24 hours per day and 365 days per year, annotated as temporal durations.Source: DURAT ION · p. 3

D. Co-occurrence and Regular Expressions

The preprocessing uses co-occurrence and regular expressions to weight influential words for each NFR type.

D. Co-occurrence and Regular Expressions

V. ANALYSIS AND RESULTS

On 625 requirements, preprocessing improved FR/NFR classification and substantially improved NFR sub-classification results across the evaluated algorithms. The analysis also describes the feature extraction and classification workflow.

A. RQ1- Influence of grammatical, temporal and sentimental sentence characteristics on FR/NFR classification

  • The workflow cleans encoding and formatting errors, applies Stanford Parser part-of-speech tagging, and extracts five syntactic features.The five are adjective, adverb, verb-modifying adverb, cardinal, and degree adjective/adverb counts.Source: A. RQ1- Influence of grammatical,… · p. 3
  • Using a > 0.8 occurrence-probability cutoff, the unprocessed data retained cardinal and degree adjective/adverb counts, while processed data retained cardinal and adverb counts.Source: A. RQ1- Influence of grammatical,… · p. 3
  • The classifier uses ten features for each dataset, with the syntactic features differing between processed and unprocessed data.Both lists also include adjective, adverb, modal, determiner, verb, preposition, singular noun, and plural noun keyword features.Source: A. RQ1- Influence of grammatical,… · p. 3
  • A Java feature-extraction prototype supplies the features to Weka’s J48 C4.5 decision tree, with a minimum leaf size of 6.Source: A. RQ1- Influence of grammatical,… · p. 3
  • For all six machine learning algorithms, NFR sub-classification results were dramatically better on preprocessed data than raw data.The passage refers to overall precision and recall in Table IV, which is not among the supplied table passages.Source: A. RQ1- Influence of grammatical,… · p. 4
  • 94.40% of processed requirements were correctly classified, versus 89.92% of unprocessed requirements, a 4.48% improvement.The processed data had weighted precision of 0.95 and recall of 0.94; the unprocessed data had weighted average precision and recall of 0.90.Source: A. RQ1- Influence of grammatical,… · pp. 3–4, Table III · p. 4, Table II · p. 4

B. RQ2- Classifying Non-functional Requirements

The section describes machine learning algorithms for classifying NFRs and assesses their performance using recall and precision.

B. RQ2- Classifying Non-functional Requirements

1) Topic Modeling:

Topic modeling produced more intuitive word lists with BTM than LDA, although BTM performed worse at NFR subclassification. Clustering tests found the dataset clusterable, but clustering algorithms classified NFRs poorly.

1) Topic Modeling:

  • LDA classifies documents using word co-occurrence frequencies, whereas BTM models word-word co-occurrence patterns suited to short, sparse requirements.The paper describes BTM as exploring word-word, or biterm, patterns.Source: 1) Topic Modeling · p. 4
  • BTM generated more intuitive topic word lists than LDA but performed much worse at NFR subclassification.The authors suggest corpus-level modeling and independently generated biterms may explain the difference.Source: 1) Topic Modeling · p. 4

2) Clustering:

  • Hierarchical clustering repeatedly merges the closest clusters, while the hybrid method uses hierarchical cluster centers to initialize K-means with k=10.K-means alone assigns documents randomly to k bins and is sensitive to the initial centroid selection.Source: 2) Clustering · pp. 4–5
  • H=0.1 indicated clusterability, yet clustering algorithms classified NFRs poorly, suggesting their sub-categories were not well separated.Figure 3(a) presents the Hopkins statistic; panels (b–d) show cluster assignments and silhouette values.Source: 2) Clustering · p. 5, Figure 3 · p. 5
  • Silhouette analysis assesses cluster cohesion; Figure 3(b–d) reports each cluster’s assigned requirement count and silhouette value.Values around zero indicate observations lie between clusters and have low cohesion.Source: 2) Clustering · p. 5, Figure 3 · p. 5

3) Na¨ıve Bayes Classification:

The section reports Binarized Naïve Bayes as the strongest tested method for NFR sub-classification and describes how preprocessing and sentence patterns relate to classification performance.

3) Naïve Bayes Classification:

  • The section presents Naïve Bayes classification using conditional probabilities over classes and document feature vectors.Source: 3) Na¨ıve Bayes Classification · p. 5
  • BNB replaces term frequencies with Boolean presence/absence features, reflecting the stated importance of word occurrence over frequency for sentiment classification.Source: 3) Na¨ıve Bayes Classification · p. 5
  • The BNB evaluation used five runs of 5-fold cross-validation; reported overall accuracy was just over 90% (p-value 2.2e-16).All NFR categories except Portability had recall from 75% for Legal requirements to 97% for security and performance requirements.Source: 3) Na¨ıve Bayes Classification · p. 5
  • Classes FT, L, MN, O, and SC were often predicted as Usability, which the authors associate with overly general usability terminology.The authors identify a need for additional or better sentiment patterns to distinguish Usability from similar NFR categories.Source: 3) Na¨ıve Bayes Classification · p. 6
  • Preprocessing improved FR/NFR classification accuracy from 89.92% to 95.04%.Source: 3) Na¨ıve Bayes Classification · p. 6
  • Preprocessing strongly affected NFR sub-classification; both precision and recall doubled for LDA and BNB.Source: 3) Na¨ıve Bayes Classification · p. 6
  • Among LDA, BTM, Hierarchical, K-means, Hybrid, and Binarized Naïve Bayes, BNB had the highest performance for sub-classifying NFRs.Source: 3) Na¨ıve Bayes Classification · p. 6
  • BTM generally modeled short-text corpus themes better than LDA but performed poorly for NFR sub-classification.Source: 3) Na¨ıve Bayes Classification · p. 6

VI. LIMITATIONS AND THREATS TO VALIDITY

The study’s findings are constrained by the dataset used to develop and evaluate preprocessing, the available NFR categories, and data quality issues. The authors also note that class imbalance may affect findings.

Data Analysis Limitations

  • The preprocessing model was developed and evaluated on the RE Data Challenge dataset, creating the study’s biggest validity threat.The authors cite sentence-structure features and less context-dependent regular expressions as mitigation.Source: §VI · p. 6
  • The work depends on the dataset creators’ choice of NFR sub-categories, although the preprocessing can be adapted to other categories.The authors aim to expand co-occurrence rules and regular expressions as they gain insights from real-world specifications.Source: §VI · p. 6

Dataset Limitations

  • The dataset contains mislabeled requirements, does not properly distinguish quality requirements from constraints, and has selection bias, including absent categories and only one PO requirement.The authors leave repeating the study on a higher-quality dataset to future work.Source: §VI · p. 6

Data Analysis Limitations

  • The unbalanced NFR dataset may affect findings; the authors say preprocessing and n-fold cross-validation mitigate several data-distribution problems.They identify noisy data, borderline examples, class overlap, dataset shift, and small disjuncts among the issues addressed.Source: §VI · p. 6

VII. CONCLUSION AND IMPLICATIONS

The authors conclude that preprocessing improves both FR/NFR classification and NFR sub-classification. Binarized Naïve Bayes performed best among the tested popular machine-learning algorithms for NFR sub-categorization, while usability differentiation needs better patterns and sentence structures.

VII. CONCLUSION AND IMPLICATIONS

  • Preprocessing improves performance for both FR/NFR classification and NFR sub-classification.Source: §VII · p. 6
  • Binarized Naïve Bayes performed best among popular machine-learning algorithms for classifying NFRs into sub-categories.Source: §VII · p. 6
  • Although BTM generally works better than LDA for extracting short-text topics, it does not perform well for NFR sub-classification.Source: §VII · p. 6
  • Differentiating usability requirements from other NFR types needs additional or better sentimental patterns and sentence structures.Source: §VII · p. 6

Related papers

Loading 1707.02358v1…