PaperlayerView exact revision on arXiv

Source-linked AI summary

Rapid quality assurance with Requirements Smells

H. Femmer, D. Méndez Fernández, S. Wagner, S. Eder

arXiv:1611.08847v1cs.SE

TL;DR

Requirements defects can contribute to project time and cost overruns, motivating faster feedback on detectable pitfalls. This paper defines and evaluates a lightweight Requirements Smells approach based on ISO language criteria, reporting average precision around 0.59 and average recall of 0.82.

  • Problem

    Requirements defects can lead to project time and cost overruns, motivating faster feedback to help discover certain types of pitfalls in requirements artifacts.

    Source: §1 · p. 2

  • Method

    The paper defines Requirements Smells as quality-violation indicators and proposes a lightweight detection approach based on ISO 29148 natural-language criteria.

    Source: §7.1 · p. 35

  • Results

    Average precision was around 0.59 across smells and average recall was 0.82; both varied across smells.

    Source: §7.1 · p. 35

  • Takeaways & Limitations

    Practitioners reported negative impacts of detected defects on estimation and implementation, while smell relevance varied by case, indicating detection should be tailored to project or company context.

    Source: §7.1 · p. 35

    The study analyzes a first set of smells based on an interpretation of ISO criteria and its findings cannot be generalized to all contexts.

    Source: §7.4 · p. 37

Abstract

from arXiv · show

Bad requirements quality can cause expensive consequences during the software development lifecycle, especially if iterations are long and feedback comes late. %-- the faster a problem is found, the cheaper it is to fix. This makes explicit the need of a lightweight detection mechanism of requirements quality violations. We aim at a light-weight static requirements analysis approach that allows for rapid checks immediately when requirements are written down. We transfer the concept of code smells to Requirements Engineering as Requirements Smells. To evaluate the benefits and limitations, we define Requirements Smells, realize our concepts for a smell detection in a prototype called Smella and apply Smella in a series of cases provided by three industrial and a university context. The automatic detection yields an average precision of 59% at an average recall of 82% with high variation. The evaluation in practical environments indicates benefits such as an increase of the awareness of quality defects. Yet, some smells were not clearly distinguishable. Lightweight smell detection can uncover many practically relevant requirements defects in a reasonably precise way. Although some smells need to be defined more clearly, smell detection provides a helpful means to support quality assurance in Requirements Engineering, for instance, as a supplement to reviews.

Appendix A Requirements Checklist 41 · 1. Introduction

The introduction motivates rapid, lightweight requirements-quality checks to complement reviews and presents Requirements Smells as detectable symptoms of defects in natural-language requirements. It outlines the smell definitions, Smella prototype, and empirical evaluation across industrial and academic contexts.

1. Introduction

  • Requirements defects such as ambiguity, incompleteness, vagueness, and loopholes can cause project overruns or subjective acceptance judgments [56].Some defects, including completeness issues, require domain knowledge to uncover.Source: §1 · p. 2
  • Requirements reviews require relevant stakeholders, domain expertise, and manual reading, and their outcomes depend on reviewer quality.The authors argue that reviews are time-consuming and costly, and superficial defects may distract reviewers.Source: §1 · p. 2
  • Rapid feedback on automatically detected Requirements Smells is proposed as lightweight quality assurance that can complement reviews.The approach targets concrete symptoms of quality defects despite the imperfect semantics of natural-language requirements.Source: §1 · p. 2
  • The paper defines and integrates Requirements Smells into RE quality assurance, presents the Smella prototype, and empirically investigates the approach’s usefulness.The investigation considers technical accuracy and practical, context-specific relevance.Source: §1 · pp. 2–3
  • The eight defined smells use ISO 29148’s Requirements Language Criteria, including loopholes and ambiguous adverbs, as their basis.The paper identifies these definitions as the basis for its smell analysis.Source: §1 · p. 2
  • The evaluation spans three industrial contexts and an academic context, including 51 requirements artifacts produced by students.The study examines detectable quality defects and how smell detection could support industrial quality assurance.Source: §1 · p. 3

Previously published material

The article extends its workshop paper with a precise definition of Requirements Smell, a tool-supported approach integrated into QA, and broader empirical investigations.

Previously published material

  • The article extends workshop paper with a richer discussion and precise definition of Requirements Smell, plus a tool-supported analysis approach integrated into the QA process.Source: Previously published material · p. 3
  • The authors add an industrial case study and an academic investigation to examine detection accuracy, detectable quality defects, and practitioner views on integrating smell detection in formal and agile QA processes.The accuracy investigation covers precision, recall, and practical relevance.Source: Previously published material · p. 3

Outline

The paper moves from prior work and Requirements Smells to tool realization, quality-assurance integration, empirical evaluation, and conclusions.

Outline

  • Sections 3–4 define Requirements Smells derived from ISO 29148 and introduce their tool realization, Smella.Section 3 follows a review of previous work in Section 2.Source: Outline · p. 3
  • Sections 5–7 discuss integrating smell detection into quality assurance, report the empirical evaluation, and conclude the paper.Source: Outline · p. 3

2. Related work

Related work traces the Requirements Smell concept to software smell research, surveys manual and automatic requirements quality assurance, and identifies empirical and ISO 29148 coverage gaps addressed by this study.

2.1. The notion of smells in software engineering

  • Requirements Smells extend the smell metaphor from code and other software artifacts to requirements engineering, defining concrete, visible symptoms as indicators of quality problems.Fowler and Beck introduced code smells as symptoms such as duplicated code, while later work applied the concept to test and use-case artifacts.Source: §2.1 · p. 3

2.2. Quality assurance of software requirements

  • Manual requirements QA includes quality-characteristic classifications and checklists, as well as constructive language approaches and analytical techniques such as reviews.These approaches and defect lists support the view that requirements artifacts are vulnerable to quality defects.Source: §2.2 · pp. 3–4
  • For redundancy detection, reported precision and recall approach 1 depending on artifact and method; RETA reports Precision > 0.85 and Recall > 0.9 for template conformance.RETA’s ambiguity detection is not further evaluated, and its template-based approach, like AQUSA and Circe, requires structured requirements; AQUSA reports precision of 0.63-1 for defects such as missing rationales.Source: §2.2 · p. 4
  • For ambiguity-related defects, Chantree et al. report precision of 54%-75%, Gleich et al. report 34%-97% across gold standards, and Krisch and Houdek report 12% for weak-word detection.Gleich et al.’s combined ambiguity metrics do not distinguish ambiguity types, preventing direct comparison; the approaches target violations related to, but not identical with, the smells studied here.Source: §2.2 · p. 5

2.3. Discussion

  • Prior automatic QA lacked realistic empirical evidence, differentiated precision and recall for ISO 29148 defects, and qualitative feedback from engineers, although detection appropriateness depends on context.The authors note that accuracy in natural-language requirements depends on context and the impact of each finding.Source: §2.3 · p. 6
  • For most ISO 29148 language-quality defects, no detection tool yet exists, and, to the authors’ knowledge, no differentiated empirical precision-and-recall analysis exists; existing approaches use proprietary, experience-based, or directly measurable quality definitions.Source: §2.3 · p. 6
  • Technically, Requirements Smell detection reuses NLP methods such as lemmatization, part-of-speech tagging, and dictionaries, adding morphological analysis for German superlatives and comparatives.The ISO 29148 rules required neither parsing nor ontologies.Source: §2.3 · p. 6
  • The study addresses these gaps with systematic analyses of defect distribution, precision, recall, and relevance, plus practitioner evaluation under realistic conditions.The evaluation covers existing and new quality defects from ISO 29148 and extends the authors’ earlier empirical work.Source: §2.3 · pp. 6, 8

3. Requirements Smells

The section defines Requirements Smells as locatable, detectable indicators of possible requirements-quality violations and derives a set of language-based smells from ISO 29148. Their findings require contextual judgment, and the paper focuses on natural-language smells with automatic detection mechanisms.

3.1. Requirements Smell terminology

  • A Requirements Smell indicates a possible requirements-quality violation, has a concrete location and detection mechanism, and does not necessarily signal a defect.Requirements quality is framed in terms of potential effects on lifecycle activities that rely on the artifact; whether a finding is problematic depends on context and follow-up quality assurance.Source: §3.1 · p. 8
  • The paper focuses on natural-language Requirements Smells whose detection can be executed automatically.The authors argue that automation is important for smell detection to provide practical benefits.Source: §3.1 · p. 8

3.2. Requirements Smells based on ISO 29148

  • Using ISO 29148 as its baseline, the paper derives smells from all of its requirements language criteria but leaves a precise, complete set for future work.The authors note that Subjective Language, Ambiguous Adverbs and Adjectives, and Non-verifiable Terms are strongly related; they use the standard's examples as their current basis.Source: §3.2 · p. 9
  • The defined smells include Subjective Language, Ambiguous Adverbs and Adjectives, Loopholes, Open-ended Non-verifiable Terms, and Superlatives.They cover subjective wording, unspecific modifiers, imprecisely limited fulfillment, terms offering developers choices, and comparisons with all other systems.Source: §3.2 · pp. 9–10
  • The remaining defined smells are Comparatives, Negative Statements, Vague Pronouns, and Incomplete References.These concern relations to specific systems or prior situations, statements of capabilities not provided, unclear pronoun relations, and references readers cannot follow.Source: §3.2 · p. 10

4. Smella: A prototype for Requirements Smell detection

Smella is a ConQAT-based prototype that detects Requirements Smells to support manual quality assurance. It turns requirements artifacts into annotated text, identifies findings, and presents them for human review.

4. Smella: A prototype for Requirements Smell detection

  • The detection process parses requirements into items, annotates language, identifies findings, and presents a human-readable visualization and summary.It is implemented on ConQAT, extended with NLP features, and is intended to support manual quality assurance.Source: §4 · pp. 10–11

4.1. Requirements parsing

  • Smella accepts Word, Excel, PDF, plain-text, and CSV files, parsing them according to format and selecting configured Excel or CSV columns as smell-detection input.Known templates can reveal missing content items, but the paper focuses on ISO-standard natural-language smells.Source: §4.1 · p. 11

4.2. Language annotation

  • Language annotation combines POS tagging, morphological analysis, dictionaries, and lemmatization to provide features for smell detection.POS tagging identifies word roles and substituting pronouns; morphological analysis detects adjective or adverb comparison forms. Dictionary matching uses lemmas, which preserve distinctions such as “use” versus “useful” that stemming can blur.Source: §4.2 · pp. 11–12

4.3. Findings identification

  • Findings are identified through grammatical annotations or dictionary matches against word lemmas, while incomplete references are not detected by the current tool.For example, the Superlatives Smell is flagged when morphological analysis identifies a superlative adjective.Source: §4.3 · p. 12

4.4. Findings presentation

  • Smella displays findings in the full artifact in a spell-checker style, treating them as contextual indications and providing explanations and possible improvements.Users can review and set finding statuses, blacklist rejected findings, and filter displayed smells; colored treemaps show artifact-level finding hotspots to guide in-depth reviews.Source: §4.4 · pp. 12, 15

5. Requirements Smell detection in the process of quality assurance

Requirements Smell detection can support both constructive and analytical quality assurance, but its integration should reflect the socio-economic context. It raises awareness and supports communication without requiring every detected smell to be resolved.

5. Requirements Smell detection in the process of quality assurance

  • Smell detection is not tied to a particular quality-assurance task or software process model, so its integration should be tailored to the socio-economic context.Source: §5 · p. 15
  • Authors can use smell detection to identify and remove potential smells before release, while reviewers can use it to prepare analytical tasks such as Fagan inspections.Making potential smells visible in advance may benefit inspections involving several reviewers, including phased inspections.Source: §5 · p. 15
  • Smell detection does not enforce resolution; it raises awareness and makes reasoning about decisions transparent, recognizing that authors and reviewers may judge a smell differently.A finding clear to its author may be unclear to its target readers.Source: §5 · p. 15
  • Smella lets stakeholders comment on findings and state whether and why they accept, reject, or need to resolve a smell, fostering communication and continuous feedback between authors and reviewers.Source: §5 · pp. 15–16

6. Evaluation

The evaluation used an exploratory multi-case study spanning industrial and academic cases to examine the approach in realistic practical settings.

6. Evaluation

  • The researchers chose practical case studies over controlled experiments and largely followed Runeson and Höst for study design and reporting.The study included both industrial and academic cases.Source: §6 · p. 16

6.1. Case study design

The case study examines whether automatic Requirements Smells analysis helps assure requirements-artifact quality, using research questions on smell prevalence, relevance, defects, and QA use.

6.1.1. Research questions

  • The four research questions address smell prevalence, detection accuracy and practical relevance, correspondence with requirements defects, and how practitioners might integrate detection into QA.Relevance is divided into technical accuracy (false positives and false negatives) and whether practitioners would react to findings and change requirements.Source: §6.1.1 · pp. 16–17

6.1.2. Case and subjects selection

  • Cases were selected opportunistically to maximize variation across realistic contexts, combining three industrial cases with an academic case involving student teams.The academic course used a shared set of functionality and provided more specifications and reviews than the industrial cases.Source: §6.1.2 · p. 17

6.1.3. Data collection procedure

  • Researchers collected requirements artifacts, ran Smella, classified detected findings as true or false positives, and inspected sampled artifacts for false negatives.For recall checks, researchers selected one artifact per case and grouped four ambiguity-related smells for manual inspection; dictionary-coverage false negatives were excluded from recall because dictionaries were considered a configuration task.Source: §6.1.3 · pp. 17–18
  • Practitioners rated selected true-positive findings for relevance, prior awareness, and whether and when they would resolve them; researchers also interviewed practitioners and collected student review reports.Selected findings covered each smell with at least two examples where artifacts allowed, and student teams reviewed other teams’ artifacts without seeing the smell findings.Source: §6.1.3 · p. 18

6.1.4. Analysis procedure

  • They reported smell findings per 1,000 artifact words, false-positive rates overall, false-negative ratios of detected to manually identified findings, and practitioner-rated irrelevant findings; user-story findings were reported by role, feature, and reason.Source: §6.1.4 · pp. 18–19
  • Researchers compared student-review defects with smell findings in a random 20% artifact sample and manually coded interview notes to analyze practitioners’ QA feedback.Source: §6.1.4 · p. 19

6.1.5. Validity procedure

  • Validity procedures included researcher peer debriefing, paired classifications, independent reclassification of 10% of findings, practitioner clarification of relevance, student review guidance, and member checking.Researchers also maintained an audit trail, iterated defect classifications to agreement, and checked practitioner interview interpretations with interviewees.Source: §6.1.5 · p. 19
  • To support external validity, case selection aimed to vary domains, sizes, and requirements-documentation practices.Source: §6.1.5 · p. 19

6.2. Results

The results examine requirements-smell findings and detection performance across industrial and academic cases, as well as practitioners’ assessments of those findings. Frequency, accuracy, and perceived relevance varied by smell and context.

6.2.1. Case and subjects description

  • The cases covered automotive, chemical business systems, and agile web user stories, plus a university student case using use cases and textual requirements.Practitioners involved in the company, domain, and specifications were selected for the industrial cases; the same experts rated findings and discussed quality assurance.Source: §6.2.1 · pp. 20–21

6.2.2. RQ 1: How many Requirements Smells are present in the artifacts?

  • More than 11,000 findings appeared in over 250,000 words—roughly 44 per 1,000 words—and findings strongly correlated with artifact size (Spearman correlation 0.9).Every artifact contained findings, ranging from 5 in the smallest case to 572 in the largest.Source: §6.2.2 · p. 21, Figure 7 · p. 22
  • Vague pronouns were most frequent at 25 findings per 1,000 words, followed by negative words at 6 and loopholes at 4; repeated “should” phrasing drove many outlier loophole findings.The automotive artifacts averaged 26 findings per 1,000 words, versus 41 for Wacker and TechDivision and 43 for student artifacts.Source: §6.2.2 · p. 23

6.2.3. RQ 2.1: How accurate is the smell detection?

  • Precision was 0.48 across all inspections and 0.59 accounting for unequal smell samples; average recall was 0.82, while superlative recall was 0.5 on few data points.Recall ranged from 0.84 to 0.95 for four of the five investigated smells. Because checking a finding often took only a few seconds, the authors considered even low-precision detection potentially useful.Source: §6.2.3 · pp. 25–27
  • False positives arose from grammatical flaws, German pronoun ambiguity, and comparative or negative terms used in conditions; NLP lemmatization and tagging errors also caused missed findings.Comparatives in about 48% of examined cases indicated the intended relative-property issue, while a similar share described conditions instead.Source: §6.2.3 · pp. 25–26
  • Across smell detectors, average precision was around 59% and average recall was 82%, with both varying between smells.Most smell detections achieved more than 80% recall. Whether precision and recall suffice in industry needs further research; reviewing findings was quick and required low investment in the cases studied.Source: §6.2.3 · p. 27, Figure 10 · p. 28, Table 8 · p. 29

6.2.4. RQ 2.2: Which of these smells are practically relevant in which context?

  • TechDivision practitioners considered 65% of discussed findings relevant, were unaware of 45%, would act on 50%, and would act immediately on 40%.They associated relevance with lengthy discussions and unnecessary estimation iterations, and expressed strong interest in exploring smell detection; they considered legal-binding findings less relevant than practitioners in Cases A and B.Source: §6.2.4 · pp. 29–30
  • Practitioners often saw underspecified stories as unimplementable, treated “should” as low-risk and not immediately urgent, and would act on 2 of 3 negative statements that revealed unclear requirements.They recalled estimation discussions about completing underspecified stories; the “should” issue was considered less urgent because developers understood it as imperative and stories were not legally relevant in their context.Source: §6.2.4 · p. 29
  • Some apparent defects depended on domain knowledge, described process constraints rather than story defects, or appeared in the user-story reason part, which practitioners treated as supplemental business-value context.Practitioners would not change the process-requirement story, though they agreed its smell violated common user-story practice; reason-part findings were considered irrelevant to testing or direct implementation.Source: §6.2.4 · p. 30

Further observations of quality defects in different parts of a user story

Findings were concentrated in the reason part of user stories, although practitioners considered them less relevant. Smell detection can flag representation and content defects and support quality assurance, but it cannot identify every defect and must fit existing workflows without adding effort.

Further observations of quality defects in different parts of a user story

  • At 64 findings per 1,000 words, the reason part had nearly twice the feature part’s rate and nearly 70% more than the average requirement.The reason part contained 53% of findings despite accounting for 34% of words; practitioners considered findings there less relevant, and the authors suggest location could help prioritize smells.Source: Further observations of quality… · p. 30

6.2.5. RQ 3: Which requirements quality defects can be detected with smells?

  • Smells flagged representation and content defects—including understandability, legally binding terminology, and underspecified non-functional requirements—but did not indicate every defect.Some terminology, use-case singularity, and structural defects may be detectable, while domain errors require formalized domain concepts and vague defects need precise definitions for automation.Source: §6.2.5 · pp. 31, 33

6.2.6. RQ 4: How could smells help in the QA process?

  • Practitioners agreed smell detection could support analytical or constructive quality assurance, provided it adds no effort, for example through existing tool-chain integration.They also described on-the-fly use to support requirement writers and potentially reduce effort-estimation time by helping prevent later misinterpretations.Source: §6.2.6 · pp. 33–34

6.2.7. Evaluation of validity

  • Validity was qualified by limited domain coverage and human judgment in classifications, while varied artifacts and peer debriefing and member checking helped mitigate threats.For false-positive classifications, inter-rater Cohen’s kappa rose from 0.53 to 0.72; a separate false-positive comparison yielded 88% agreement.Source: §6.2.7 · pp. 34–35

7. Conclusion

The paper proposes lightweight Requirements Smells detection based on ISO 29148 and evaluates its use for quality assurance, finding that it can reveal relevant defects while leaving limitations in accuracy, context, and taxonomy clarity.

7.1. Summary of conclusions

  • The authors define a Requirements Smell as a quality-violation indicator with a concrete location and detection mechanism, and propose a lightweight approach based on ISO 29148 criteria.Their prototype uses part-of-speech tagging, morphological analysis, and dictionaries, and they outline integration into quality assurance.Source: §7.1 · p. 35
  • Across cases, detection averaged 0.59 precision and 0.82 recall, with findings ranging from 22 to 67 per 1,000 words and substantial variation.The approach analyzed requirements totaling 250k words; precision ranged from 0.26 to 0.96, and recall from 0.5 to 0.95. Evidence from reviews, practitioner interviews, and independent manual reviews indicated that smells can identify relevant defects across forms of requirements, domains, and methodologies.Source: §7.1 · pp. 35–36

7.2. Relation to existing evidence

  • The study adds detailed empirical evaluation of ISO 29148 quality factors and considers how requirements defects may be addressed through automatic analysis more broadly.Prior empirical studies of the standard’s violations were lacking, while related studies reported varied precision and recall; the paper also provides a broader perspective beyond starting from a particular quality model.Source: §7.2 · p. 36

7.3. Impact/Implications

  • For practitioners, smell detection can reduce obvious issues in reviews and provide immediate feedback, but low precision in some smells can create unnecessary checking work.The authors advise focusing on highly accurate smells; findings’ relevance varies by case, so detection should be tailored to the project or company context.Source: §7.1 · p. 35, §7.3 · p. 36
  • For researchers, the taxonomy is a starting point, but several smells were difficult to distinguish and require critical refinement.The paper also presents smell detection as a way to quantify selected defects in large requirements samples while accounting for context sensitivity.Source: §7.3 · pp. 36–37

7.4. Limitations

  • The findings are a first step rather than generalizable results, because the study covers a first set of smells and cannot represent all applicable contexts.The smells are based on the authors’ interpretation of sometimes imprecise ISO 29148 criteria, and other smells with different characteristics may exist.Source: §7.4 · p. 37

7.5. Future work

  • Future work should clarify and extend the taxonomy, study full QA-process integration, and evaluate whether smell detection helps engineers improve requirements artifacts.The link between detected issues and their impact also needs careful evaluation in practice.Source: §7.5 · p. 37

Appendix A. Requirements Checklist

Appendix A presents a checklist for students’ requirements reviews. The checklist is based on [13] and [53] and translated from German.

Appendix A. Requirements Checklist

  • Table A.11 contains the requirements-review checklist.Source: Table A.11 · p. 47
  • The checklist is intended for students’ requirements reviews.Source: Table A.11 · p. 47
  • Anke Drappa, Patricia Mandl-Striegnitz, and Holger Röder created the checklist based on [13] and [53] and translated it from German.Source: Table A.11 · p. 47

Related papers

Loading 1611.08847v1…