PaperlayerView exact revision on arXiv

Source-linked AI summary

The Quamoco Product Quality Modelling and Assessment Approach

Stefan Wagner, Klaus Lochmann, Lars Heinemann, Michael Kläs, Adam Trendowicz, Reinhold Plösch, Andreas Seidl, Andreas Goeb, Jonathan Streit

arXiv:1611.04433v1cs.SE

TL;DR

Existing quality models often separate abstract quality attributes from concrete assessments, leaving gaps in measurement and overall evaluation. Quamoco combines a meta model, base model, and assessment approach; validations found correspondence with expert opinions, strongest for maintainability.

Abstract

from arXiv · show

Published software quality models either provide abstract quality attributes or concrete quality assessments. There are no models that seamlessly integrate both aspects. In the project Quamoco, we built a comprehensive approach with the aim to close this gap. For this, we developed in several iterations a meta quality model specifying general concepts, a quality base model covering the most important quality factors and a quality assessment approach. The meta model introduces the new concept of a product factor, which bridges the gap between concrete measurements and abstract quality aspects. Product factors have measures and instruments to operationalise quality by measurements from manual inspection and tool analysis. The base model uses the ISO 25010 quality attributes, which we refine by 200 factors and 600 measures for Java and C# systems. We found in several empirical validations that the assessment results fit to the expectations of experts for the corresponding systems. The empirical analyses also showed that several of the correlations are statistically significant and that the maintainability part of the base model has the highest correlation, which fits to the fact that this part is the most comprehensive. Although we still see room for extending and improving the base model, it shows a high correspondence with expert opinions and hence is able to form the basis for repeatable and understandable quality assessments in practice.

I. INTRODUCTION

Quamoco addresses the gap between abstract software-quality definitions and concrete assessments with an operationalised quality model. It combines a meta model, an assessment method, a Java- and C#-focused base model, and initial validation against expert opinions.

I. INTRODUCTION

  • Software quality remains controversial and often unsatisfactory, with economic impacts including higher maintenance costs, resource consumption, long test cycles, and user waiting times.Source: §I · p. 1

A. Quality Models – Benefits and Shortcomings

  • Abstract quality models lack practical assessment support, while domain-specific models often lack links to higher-level goals; existing assessment methods can also be too abstract or lack a solid theoretical basis.Source: A. Quality Models – Benefits and… · p. 1

B. Research Objective

  • Quamoco aims for a widely applicable, highly operationalised model connecting general quality attributes with specific analysis approaches, tools, manual analyses, and a defined assessment method.Source: B. Research Objective · p. 1

C. Contribution

  • The project contributes a meta model, an integrated assessment method, a base model elaborated for Java and C#, and initial validation showing correspondence with expert opinions.Source: C. Contribution · p. 2

II. RELATED WORK

Prior quality models evolved from attribute taxonomies toward more elaborate structures, but measurement and assessment links remain incomplete. Quamoco builds on earlier work and a three-iteration development of its quality-model concepts.

II. RELATED WORK

  • Quality-model research progressed from decomposing quality attributes to distinguishing product components and externally visible attributes, with later calls for explicit meta models,.Source: §II · p. 2
  • Critiques find quality taxonomies ambiguous and insufficiently measurable; a survey found fewer than 28% of companies use standard models, while 71% developed their own,,,.Source: §II · p. 2
  • Quality-analysis tools and dashboards cover specific aspects or display metrics, but are not systematically linked to quality models and attributes.Source: §II · p. 2
  • Squale offers a hierarchical model and assessment tools, whereas Quamoco distinguishes itself with an editable model and flexible integration of tools and manually collected data.Source: §II · p. 2
  • Quamoco's complete approach builds on prior work describing quality models, metric classification, assessment, and tool support, –.Source: §II · p. 2

III. QUALITY MODEL CONCEPTS

  • The quality-model concepts and meta model were developed over three years in three iterations, with corresponding evaluations.Source: §III · p. 2

A. Usage of the Quality Models

Quamoco uses a quality model to define goals, measurements, and interpretations, then tailors the model to a project. Its concepts connect measurable product properties to abstract quality goals and support assessment across contexts.

A. Usage of the Quality Models

B. General Concepts

  • Quamoco separates abstract quality aspects from measurable product factors, such as code duplication measured through clone coverage.Source: B. General Concepts · p. 3
  • Multiple quality-aspect hierarchies can express different views, including product quality and quality in use, allowing models to be tailored for different stakeholders.Source: B. General Concepts · p. 3
  • Impacts connect product factors to quality aspects and describe whether a factor's presence or absence influences an aspect positively or negatively.Source: B. General Concepts · p. 3
  • Measures quantify product factors in a context, while separate instruments implement those measures through manual inspection or tools such as FindBugs.Source: B. General Concepts · p. 3
  • Evaluations aggregate measurements into quality assessments and also let teams trace important quality aspects downward to influential factors and their measures.Source: B. General Concepts · p. 3
  • Modules divide large models into a root module and extensions for technologies, programming languages, and domains.Source: B. General Concepts · pp. 3–4

C. Meta Model

The meta model formalises quality-model concepts around factors and their relationships. It links measurable product factors to quality aspects while supporting evaluations, measures, and instruments.

C. Meta Model

  • Figure 1 depicts the meta model's core: Factor branches into Quality Aspect and Product Factor, each refinable into a separate directed acyclic graph, with impacts directed from product factors to quality aspects.Source: Figure 1 · p. 4, C. Meta Model · p. 4
  • Each factor describes a property of an entity and has an evaluation; product factors can use associated measure values, and measures may have multiple instruments for different collection contexts.Source: C. Meta Model · p. 4
  • The detailed meta model was implemented as an EMF model and became the basis for the quality-model editor.Source: C. Meta Model · p. 4

IV. BASE MODEL

The Quamoco base model makes software quality assessable through a detailed model intended for broad applicability, with concrete measures and instruments for Java and C# systems. It combines tool-based analysis with manual assessment and established quality factors.

IV. BASE MODEL

  • The base model is intended to support tool-based quality assessment across a wide range of software products, with detailed analysis tools for Java and C# systems.Experts from academia and industry developed it through workshops over three years as a basis for further specialization and adaptation.Source: §IV · p. 4

A. Contents

  • The model contains 286 factors, 526 measures and 542 instruments, including instruments supplied by 12 tools.Of its factors, 221 have evaluations and 202 define impacts on other factors.Source: A. Contents · p. 4

1) Rules of Static Code Analysis Tools:

  • The FindBugs rule for equality to Double.NaN measures expression applicability, a factor that influences functional correctness and analysability.The rule checks Java floating-point equality comparisons with Double.NaN, which always evaluate false.Source: 1) Rules of Static Code Analysis Tools · pp. 4–5

2) Established Generic Factors:

  • Naming-convention assessment combines automated checks with manual review, while source-code duplication negatively affects analysability and modifiability.Duplication is measured through clone coverage and cloning overhead.Source: 2) Established Generic Factors · p. 5

B. Modularisation

Quamoco modularizes general quality concepts, language-specific instruments and domain extensions. Its assessment method uses stakeholder preferences and utility synthesis to turn measurements into product-level assessments.

B. Modularisation

  • The object-oriented module shares 64 factors across Java and C#, while Java adds 1 and C# adds 8 language-specific factors.Modules separate general metrics from language-specific measurement instruments and can connect domain-specific models to the root model.Source: B. Modularisation · p. 5

V. QUALITY ASSESSMENT APPROACH

  • The assessment method synthesizes measurement data to assess overall product quality, addressing practical requirements identified through a literature review and consortium survey.The survey’s design, execution and results are reported separately.Source: §V · p. 5, A. Practical Challenges · p. 5

A. Practical Challenges

  • Quamoco adapts MAUT/MAVT because it meets most identified requirements, including comprehensibility, stakeholder preferences, incomplete information and compensation among quality aspects.The authors state that the approach can be adjusted to meet all the requirements.Source: A. Practical Challenges · p. 5

B. Quality Assessment Method

  • Utility maps a factor’s measurements to preference values, and higher-level factor utilities synthesize the utilities of their direct sub-factors.A factor’s utility can be non-monotonic; for comment density, utility rises to a threshold and then falls. Figure 2 pairs model-operationalisation activities with corresponding product-assessment activities.Source: B. Quality Assessment Method · pp. 5–6

1) Defining Measures / Measurement:

Measurements are linked to instruments and normalized to make values comparable across systems. Utility functions then score measured factors using thresholds derived from baseline systems.

1) Defining Measures / Measurement:

  • Each measure is paired with an appropriate instrument and reviewed by two measurement experts for normalization; normalized measurements make systems of different sizes comparable.For Java Platform 6, 6 detections over 2,759,369 lines of code yield a normalized value of 2.17 · 10^-6.Source: 1) Defining Measures / Measurement · p. 6
  • Interval arithmetic determines a range of possible factor outcomes when measurement data are incomplete.The measures quantify the lowest-level factors in the quality model hierarchy.Source: 1) Defining Measures / Measurement · p. 6

2) Defining Utility Functions / Scoring:

  • Linear increasing or decreasing utility functions map normalized measurements to scores from 0 to 1, with thresholds derived from at least 10 baseline systems and reviewed by measurement experts.For M4, the thresholds are min = 0 and max = 8.5 · 10^-6; Java Platform 6 receives U(M4) = 0.74.Source: 2) Defining Utility Functions / Scoring · p. 6

3) Defining Factor Weights & Aggregation Operator / Aggregation:

Quamoco assigns relative weights to adjacent factors and measures, aggregates their utilities, and supports interpretation of assessment results. The project also provides an integrated tool chain for modelling and assessment.

3) Defining Factor Weights & Aggregation Operator / Aggregation:

  • Experts rank sibling factors and the Rank-Order Centroid method converts those rankings into weights that sum to 1; weighted sums aggregate utilities up the quality-model hierarchy.In the example, M4 receives weight 0.25 and the resulting utility for F1.1 is 0.85.Source: 3) Defining Factor Weights &… · pp. 6–7

4) Defining / Applying Interpretation Models:

VI. TOOL SUPPORT

  • The model and assessment method are described as ready to use with minimal project effort, and Quamoco provides an integrated tool chain for modelling and assessment.The tool chain includes a quality-model editor and an assessment engine.Source: §VI · p. 7

A. Quality Model Editor

The quality model editor supports editing and validating models against Quamoco constraints. The assessment engine generates analyses from a model, presents results at multiple levels, and supports batch execution; the validation section frames two research questions.

A. Quality Model Editor

  • The model editor uses separate files for quality-model modules, enabling concurrent work and tree-view navigation with form-based editing.Source: A. Quality Model Editor · p. 7
  • The editor provides model validation, error and warning navigation, and context-sensitive help for meta-model concepts and modelling best practices.Validation checks include unreferenced elements and missing module dependencies.Source: A. Quality Model Editor · p. 7

B. Quality Assessment Engine

  • The assessment engine generates a ConQAT configuration from a quality model and can combine it with source code and optional manual results.Assessors can extend the tooling with custom analyses.Source: B. Quality Assessment Engine · p. 7
  • The engine presents assessment results in the model hierarchy, a treemap, or an HTML report, and supports batch runs for continuous integration.Batch execution can detect quality decay early.Source: B. Quality Assessment Engine · p. 7

VII. EMPIRICAL VALIDATION

  • The empirical validation asks whether the base model detects quality differences across systems or subsystems and improvements over time.Source: §VII · p. 7

A. Comparison of Software Products and Subsystems

The base model’s product rankings closely match an independent expert-based ranking, while subsystem comparisons are less conclusive. The validation uses expert rankings and is limited by the small and specific set of systems assessed.

B. Making Quality Improvements Visible

The base model reflected experts’ expectations of quality improvement in one industrial software project. Its assessment showed steady improvement across later versions, while the result’s generalizability is limited by the single project.

VIII. CONCLUSIONS

Quamoco combines an operational quality meta model, a broad base model, and a transparent assessment approach. Its assessments fit expert opinion, with the strongest results limited to maintainability; the model and its evaluation still need broader technology coverage and further study.

VIII. CONCLUSIONS

  • Quamoco contributes a meta model, a base model operationalised for Java and C#, and a transparent, repeatable quality assessment approach.The approach was evaluated in empirical studies.Source: §VIII · p. 9
  • Assessment results fit expert opinion, but the strongest results are limited to the maintainability part of the model.The authors report further empirical studies to investigate remaining weaknesses.Source: §VIII · p. 10
  • Open-source tools support both building operationalised quality models and performing quality assessments.Source: §VIII · p. 10
  • The base model and its evaluation currently cover only a small number of technologies, and further empirical studies are underway.Industry partners are conducting drill-downs in which system experts rate the usefulness of assessment results.Source: §VIII · p. 10
Loading 1611.04433v1…