Source-linked AI summary
Operationalised product quality models and assessment: The Quamoco approach
Stefan Wagner, Andreas Goeb, Lars Heinemann, Michael Kläs, Constanza Lampasona, Klaus Lochmann, Alois Mayr, Reinhold Plösch, Andreas Seidl, Jonathan Streit, Adam Trendowicz
TL;DR
Software quality models often describe abstract characteristics or concrete measurements without connecting them, and standard models can be difficult to operationalise and customise. Quamoco develops modular operationalised models, an assessment method, and tools; its evaluations report substantial practitioner understanding and reasonable agreement with expert opinion, while noting limits to the evidence.
Problem
Quality models often describe abstract characteristics without connecting them to concrete measurements, and standard models may lack detailed measures and require customisation.
Source: §1 · p. 2, §2.1 · p. 4Method
Quamoco combines an explicit meta-model, a broad base model operationalised for Java and C#, an integrated assessment approach, and tool support.
Source: §9.1 · p. 35Results
Assessments largely matched expert opinion, especially for maintainability, and practitioners found the model and assessments highly understandable, though evidence was not fully conclusive.
Source: §9.1 · p. 35Takeaways & Limitations
The work shows that detailed operationalised models can bridge abstract quality characteristics and concrete measures, though creating them required substantial effort.
Source: §9.2 · p. 35The evaluation used open-source software and cannot establish whether the approach holds for safety-critical systems.
Source: §7.1.5 · p. 28
Abstract
from arXiv · showhide
Software quality models provide either abstract quality characteristics or concrete quality measurements; there is no seamless integration of these two aspects. Reasons for this include the complexity of quality and the various quality profiles in different domains which make it difficult to build operationalised quality models. In the project Quamoco, we developed a comprehensive approach for closing this gap. It combined constructive research, which involved quality experts from academia and industry in workshops, sprint work and reviews, with empirical studies. All deliverables within the project were peer-reviewed by two project members from a different area. Most deliverables were developed in two or three iterations and underwent an evaluation. We contribute a comprehensive quality modelling and assessment approach: (1) A meta quality model defines the structure of operationalised quality models. It includes the concept of a product factor, which bridges the gap between concrete measurements and abstract quality aspects, and allows modularisation to create modules for specific domains. (2) A largely technology-independent base quality model reduces the effort and complexity of building quality models for specific domains. For Java and C# systems, we refined it with about 300 concrete product factors and 500 measures. (3) A concrete and comprehensive quality assessment approach makes use of the concepts in the meta-model. (4) An empirical evaluation of the above results using real-world software systems. (5) The extensive, open-source tool support is in a mature state. (6) The model for embedded software systems is a proof-of-concept for domain-specific quality models. We provide a broad basis for the development and application of quality models in industrial practice as well as a basis for further extension, validation and comparison with other approaches in research.
1. Introduction
Software quality models and concrete measurements remain disconnected, while assessment methods may be too abstract and fixed to support consistent, context-sensitive quality improvement. Quamoco addresses this gap with operationalised, modular quality models, an assessment method, and supporting tools.
1. Introduction
- Software quality remains controversial and insufficiently understood, and poor product quality carries economic costs including maintenance, resource use, testing delays, and user waiting times.Source: §1 · p. 2
1.1. Quality Models – Benefits and Shortcomings
- Quality models describe abstract characteristics, while concrete measurements often lack links to higher-level goals; existing assessment procedures may be non-operational, and models are often difficult to adapt across contexts.These gaps make quality problems harder to explain and can inhibit assessment or produce inconsistent results.Source: §1.1 · p. 2
1.2. Research Objective
- The project aimed to connect generic quality characteristics with specific analyses and measurements through modularised models, focusing on product quality and static analyses or manual reviews.Dynamic testing was excluded because it requires domain-specific test cases and execution environments.Source: §1.2 · p. 2
1.3. Contribution
- Quamoco contributes a meta-model, a base model elaborated for Java and C#, an integrated assessment method, empirical validations, tool support, and a domain-specific extension.The base model provides a foundation for more specific quality models.Source: §1.3 · pp. 2–3
1.4. Terms and Definitions
- A quality aspect is an area of stakeholder interest, while a product factor is an observable property of an entity in a software product.The paper treats ISO/IEC quality characteristics as quality aspects and allows other decompositions as well.Source: §1.4 · p. 3
1.5. Outline
- The article moves from related work and the meta-model to the base model, assessment method, tools, empirical evaluation, an embedded-systems extension, and conclusions.Source: §1.5 · p. 3
2. Related Work
Prior work ranges from hierarchical quality characteristics to richer models and analysis tools, but the literature describes gaps in operational assessment, tool support, and adaptable model foundations. Quamoco builds on this work with a product model, modularisation, and configurable tool support.
2.1. Hierarchical Models
- Hierarchical models and standards decompose quality into characteristics, but detailed measures remain limited and fewer than 28% of surveyed companies use standard models while 71% developed variants.Source: §2.1 · pp. 3–4
2.2. Richer Models
- Richer models add structure beyond characteristics and metrics, yet the surveyed approaches had not established a corresponding assessment method, tool support, and general base model.Examples discussed include Dromey’s product components, QMOOD design properties, probabilistic quality attributes, and the Squid approach.Source: §2.2 · p. 4
2.3. Quality Measurement and Tool Support
- Analysis tools and dashboards address specific quality aspects or visualize measurements, but generally lack comprehensive assessment and explicit links between metrics and quality aspects.Squale offers a more comprehensive model and assessment support, but lacks product-component detail and modularisation; CAST pools measures into severity classes.Source: §2.3 · pp. 4–5
- Quamoco distinguishes itself with a product model for technical factors, modularisation, model-management tools, and flexible integration of measurement tools or manually collected data.Source: §2.3 · p. 5
2.4. Preliminary Work
- The authors’ prior work developed a meta-model, explored assessment and tool support, and produced a tool chain; this article adds a detailed base-model development account, domain models, adaptation, tools, and validations.Source: §2.4 · p. 5
3. Quality Model Concepts
Quamoco connects quality goals to measurable product properties and organizes the concepts into a meta-model. Its quality models support assessment, context-specific modularization, and interpretation of results.
3.1. Usage of Quality Models
- Quality models support quality control by linking relevant quality requirements, assurance activities, assessments, and product rework in one source of quality information.For assessment, measurements are interpreted and aggregated into quality-aspect and overall-product utilities, then mapped to intuitive assessments such as school grades.Source: §3.1 · p. 6
3.2. General Concepts
- Quality aspects express abstract product-wide goals, while measurable product factors describe properties of product parts, bridging quality goals to concrete measurements.Examples include maintainability as a quality aspect and source-code duplication measured by clone coverage as a product factor.Source: §3.2 · pp. 6–7
- Multiple quality-aspect hierarchies can represent different quality views, while product factors can impact quality aspects positively or negatively.Factors included in assessments need at least one impact on a quality aspect; some factors are used only to structure the model.Source: §3.2 · p. 7
- Measures quantify product factors, instruments implement measures, and evaluations aggregate measurements and factor impacts into quality assessments.Separating instruments from measures allows a measure to be collected manually or with different tools in different contexts.Source: §3.2 · pp. 7–8
- Modularization separates a large model into compatible modules for general concepts, technologies, programming paradigms, and domains.Modules can be selected and extended for a context, while a separate adaptation method guides project-specific choices of quality aspects, factors, and measures.Source: §3.2 · p. 8
3.3. Meta-Model
- The meta-model formalizes factors, their refinements, and impacts, with an Impact relating a Product Factor to a Quality Aspect.Factors have associated entities and may have evaluations; measures can link to multiple product factors and instruments.Source: §3.3 · p. 8
4. Base Model
The base model consolidates software-source-code quality knowledge into an operationalized model intended for broad use and low-effort extension. Its development combined complementary modeling directions with iterative review, testing, and validation.
4. Base Model
- The base model captures general software-source-code quality and provides measures and instruments for assessing Java and C# systems with little adaptation or configuration.It is intended to apply in principle to any kind of software and to be extended for specific contexts.Source: §4 · p. 9
4.1. Development of the Base Model
- The project involved at least 23 industry and academic participants, and later two-week sprints brought together four to eight people from multiple partners.The third iteration focused on consolidating and completing the model after earlier work shifted toward broader partner involvement.Source: §4.1 · p. 9, §4.1.1 · p. 10
4.1.1. Approach
- Development combined top-down modeling from quality aspects with bottom-up modeling from existing measures and static-analysis tools to support comprehensiveness.Top-down work helps identify missing factors and measures, while bottom-up work starts from available measures.Source: §4.1.1 · p. 9
- Three yearlong iterations used consortium feedback, reviews by two uninvolved reviewers, and regular workshops to refine and validate the model.Nightly assessments on open-source projects were added to find model problems quickly.Source: §4.1.1 · pp. 9–10
4.1.2. Execution
- The final iteration selected Java and C# tools, derived product factors from measures, assigned and reviewed impacts, and assessed measure coverage in workshops.Reviewers checked whether each factor's impacts would remain valid if assigned directly to each measure, adjusting factor granularity where needed.Source: §4.1.2 · pp. 10, 12
4.2. Contents
The base model combines a broad set of quality factors and measures with tool support for Java and C#. It includes static-analysis rules, manual and automated best-practice checks, and a separate example module for accessibility guidelines.
4.2. Contents
- 92 entities and 284 factors form the model, with 233 evaluated factors and 490 impacts linking 201 factors to quality aspects.Factors can impact multiple quality aspects, and measures can contribute to multiple factors.Source: §4.2 · p. 13
- 526 measures and 544 instruments operationalize 194 measured factors, including 536 tool-provided instruments from 12 tools.FindBugs provides 361 modeled Java rules and Gendarme 146 C# rules; eight instruments are manual.Source: §4.2 · p. 13
4.2.1. Rules of Static Code Analysis Tools
- The model emphasizes static-analysis rules and best practices, while the validity of other software metrics remains controversial.Automatic analysis can produce false positives, which the authors say are accounted for in utility mapping and lower weights for some measures.Source: §4.2 · p. 13, §4.2.1 · p. 14
- Static-analysis rules are linked through product factors to quality aspects, as in a NaN equality-check rule affecting functional correctness and analysability.The authors note that empirical evidence validating assignments of rules to factors was not available to them.Source: §4.2 · p. 13, §4.2.1 · p. 14
4.2.2. Programming Best Practices
- Best-practice factors combine automated and manual checks when a property, such as consistent and meaningful identifier use, is only partly assessable automatically.Code duplication is measured by clone coverage and cloning overhead and impacts analysability and modifiability.Source: §4.2.2 · p. 14
4.2.3. Guidelines
- The WCAG 2.0 graphical-interface module models technology-independent accessibility success criteria for UI components, but is presented as an example rather than a complete model.It is separated from the base model because the guideline is large and only a small fraction was represented.Source: §4.2.3 · p. 15
4.3. Modular Structure
The base model uses modules to separate reusable, context-independent concepts from technology-specific elements. This structure supports independent module development and explicit modeling of language commonalities and differences.
4.3. Modular Structure
- The root module holds reusable quality aspects and technology-independent factors, while object-oriented and language modules add shared and specific concepts.The object-oriented module provides 64 common factors reused by Java and C#, which add 1 and 8 language-specific product factors, respectively.Source: §4.3 · p. 15
- General measures can be defined separately from concrete language-specific instruments, keeping concepts distinct from their implementations.For example, a general number-of-classes measure can be paired with a Java tool in the Java module.Source: §4.3 · p. 15
- Explicit modules let teams work independently across technologies and domains and permit experimental modules without changing stable base-model contents.The C and C++ modules were not yet mature, while a GUI module tested non-source-code factors.Source: §4.3 · p. 15
5. Quality Assessment Approach
Quamoco’s assessment method turns measured properties into an overall quality assessment through four steps: measurement, evaluation, aggregation, and interpretation. Its operationalisation adapts measures, utility functions, weights, and interpretation to the assessment context.
5.1. Quality Assessment Method
- Quamoco assesses software through measurement, evaluation, aggregation, and interpretation, with operationalisation defining the choices needed to apply each step.Operationalisation can be adapted to context, including measures, utility functions, factor weights, and the interpretation model.Source: §5.1 · p. 16
5.1.1. Measurement
- Measurements are normalised to support comparability across products, although the chosen normalisation measure can affect cross-product comparisons.For class-related measures, the authors use abstract measures such as number of classes rather than lines of code when appropriate.Source: §5.1.1 · p. 16
5.1.2. Evaluation
- Utility functions evaluate lowest-level factors using simple increasing or decreasing linear functions bounded by thresholds for utility 0 and 1.Thresholds are determined by benchmarking measures across products; incomplete data are handled with best- and worst-case values that express uncertainty as a range.Source: §5.1.2 · p. 19
5.1.3. Aggregation
- Weighted-sum aggregation combines utilities bottom-up through the quality-model hierarchy, using weights to express the relative importance of sibling elements.The example assigns M4 and M5 weights of 0.25 and 0.75, respectively, yielding U(F1.1) = 0.85.Source: §5.1.3 · p. 20
5.1.4. Interpretation
- Interpretation maps ratio-scale utility values to a more intuitive assessment scale, exemplified by German school grades from 1 to 6.Visualisations can help users navigate detailed results; a sunburst supports hierarchy exploration, while a Kiviat diagram supports comparison across systems or subsystems.Source: §5.1.4 · p. 20
5.2. Operationalisation of the Base Model
- The base model’s operationalisation was prepared for assessment but may still need context-specific adaptation.The authors calibrated Java evaluation thresholds using open-source projects and based quality-aspect weights on a survey with more than 100 responses.Source: §5.2 · pp. 21–22
- Across 120 systems, assessments spanned grades 1–6, with few systems at the extremes and a slight tendency toward good results.The grade distribution was 1: 9%, 2: 37%, 3: 31%, 4: 13%, 5: 6%, and 6: 4%.Source: §5.2 · p. 22
6. Tool Support
Quamoco provides tools to edit and adapt quality models, run assessments, and inspect results. Its support spans model construction through assessment visualisation, with adaptation tasks partly automated and partly left to user judgment.
6. Tool Support
- The tool chain supports model building and editing, context-specific adaptation, system assessment, and visualisation of assessment results.The tool support is available from the Quamoco website.Source: §6 · p. 22
6.1. Quality Model Editor
- The editor supports modular quality-model editing, model validation, and operationalisation activities such as defining measures, utility functions, and weights.It can run the assessment engine and load results for visualisation.Source: §6.1 · p. 22
6.2. Adaptation Support
- The adaptation wizard matches models to a stated goal and context, removes irrelevant elements, and generates tasks for remaining adjustments.Users perform tasks requiring decisions, such as classifying a newly added measure as a normalisation measure.Source: §6.2 · p. 23
6.3. Quality Assessment Engine
- The assessment engine generates a ConQAT configuration from an adapted model and assesses source code alongside manual measurements.It outputs browser-readable HTML reports and result files, and can run in batch mode for continuous integration.Source: §6.3 · pp. 23–24
6.4. Visualisation
- Visualisations support tracing results from abstract quality aspects to measures and comparing assessment results across systems.Sunburst segments encode factor importance by angle, hierarchy by position, and results by colour; Kiviat spokes represent top-level factors and data points show system results.Source: §6.4 · pp. 24–25
7. Empirical Validation
The empirical validation examines whether the Java base model yields valid overall assessments, detects quality change over time, and is accepted by practitioners. The C# model is excluded because its calibration was based on only 23 systems.
7. Empirical Validation
- Validation focuses on the Java base model because it was thoroughly calibrated, while the C# model is excluded due to weak calibration based on 23 systems.The authors frame three research questions around validity, detecting improvements over time, and practitioner acceptance.Source: §7 · p. 25
- The first two research questions test technical usefulness in typical scenarios, while the third addresses practitioner-perceived usefulness and applicability.The evaluation limits its first two questions to overall quality assessment results, including identifying low-quality products or releases and tracking improvement initiatives.Source: §7 · p. 25
- The evaluation sections report each study’s goal, design, execution, results, and the most relevant validity threats.Raw validation data were available upon request and, where possible, through.Source: §7 · p. 25
7.1. Comparison of Software Products
The Java base model’s ranking of five open-source products closely matched an independent expert ranking, though the validation is narrow in scope. The comparison uses expert judgments as its criterion and evaluates consistency between ordinal rankings.
7.1.1. Design
- The study compares base-model assessments with independent expert judgments because no direct quality measurements were available for the assessed products.The expert criterion was a ranking of five open-source Java products based on independent ratings from nine experienced Java experts.Source: §7.1.1 · p. 26
7.1.2. Execution
- The authors assessed five named Java products and compared their overall-quality ordering with the Linzer Software-Verkostung ranking.Manual measures were collected using base-model specifications and independently reviewed before assessment.Source: §7.1.2 · p. 27
7.1.3. Results
- The base model’s ranking showed a significant positive correlation with the Linzer Software-Verkostung ranking (Spearman’s rho r = 0.9, p-value 0.05), close to a perfect correlation of 1.The two systems’ rankings differed only in the order of systems with close ranking results.Source: §7.1.3 · p. 27
7.1.4. Interpretation
- The authors judge the overall assessments largely consistent with the independent criterion, but did not test whether fewer measures or factors would produce the same results.The base-model results incorporate all measures and product factors considered in the quality model.Source: §7.1.4 · p. 27
7.1.5. Threats to Validity
- The validation covers five medium-sized open-source Java systems, limiting generalisability beyond that setting.The authors also note that the expert criterion may not fully represent product quality, although expert rankings showed strong concordance (Kendall’s W = 0.6635, α = 0.01).Source: §7.1.5 · pp. 27–28
- Because the study excludes industrial-strength software, the authors cannot conclude whether the approach applies to safety-critical systems.They cite known internal-quality differences between safety-critical and non-safety-critical software.Source: §7.1.5 · p. 28
7.2. Sensing Quality Improvements
The base model was tested on four versions of an automation-domain Java system whose developers had improved quality through their own experience. Assessments tracked those improvements, though the evidence comes from one project.
7.2.1. Design
- The study examined four versions of an automation-domain Java product after its team began improving quality through developer experience rather than static-analysis tools.The base model assessed versions 2.0.1 through 2.2.1, and the team reviewed the results in an interview without raising concerns.Source: §7.2.1 · p. 28, §7.2.2 · p. 28, §7.2 · p. 28
7.2.3. Results
- The results showed steady improvements for versions 2.0.2, 2.1.0, and 2.2.1, with an improvement of 0.46 from version 2.0.1 to version 2.2.1.Source: §7.2.3 · p. 28
7.2.4. Interpretation
- The assessments’ rising quality grades were an indicator that the base model reflected experienced developers’ understanding of software quality.The developers relied on their understanding of software quality during the improvements.Source: §7.2.4 · p. 29
7.2.5. Threats to Validity Conclusion Validity.
- The result’s generalisability is limited to one project where developers improved quality without static-analysis tools.The improvements relied on the developers’ experience.Source: §7.2.5 · p. 29
- The study mitigated influence from the base model or static-analysis findings by ensuring developers had neither helped build the model nor used static analysis.The analysis was ex post, with no relation between improvement actions and the analysis.Source: §7.2.5 · p. 29
- The external-validity boundary is one industry project whose developers explicitly invested in quality without static-analysis tools.The project’s improvements were based on developer experience.Source: §7.2.5 · p. 29
7.3. Acceptance by Practitioners
Practitioners generally considered the base model suitable and easier to adopt and understand than ISO/IEC 9126/25000, while agreeing it was easy to use but not that it was useful. The study used eight interviews, and validity of assessment results remained uncertain to participants.
7.3.1. Design
- The interviews assessed transparency, comprehensibility, consistency with experts’ perceptions, overall suitability, technology acceptance, and comparison with ISO/IEC.Closed questions used a five-point scale from -2 to 2, with follow-up comments.Source: §7.3.1 · p. 29
7.3.2. Execution
- The eight interviews involved a model-editor walkthrough, assessment-result drill-down for a system in each expert’s work environment, and closing questions.The model was compared with ISO/IEC 25010 as a reference point.Source: §7.3.2 · pp. 29–30
7.3.3. Results
- Participants considered the model suitable for measuring and evaluating quality, but assessments were neither appropriate nor inappropriate against their own perceptions.H1 held for all transparency sub-aspects and overall suitability, but not for consistency with own perception.Source: §7.3.3 · p. 30
- Practitioners agreed the model was easy to use, but did not agree that it was useful.The participants were eight industry practitioners with 4–20 years of software-development experience, interviewed separately.Source: §7.3.2 · p. 29, §7.3.3 · p. 30
- Interviewees strongly agreed the base model was easier to adopt and had a more transparent quality definition than ISO/IEC 9126/25000.Overall, they found it suitable and acceptable for their work.Source: §7.3.3 · p. 30, §7.3.4 · p. 30
7.3.4. Interpretation
- Experts were undecided about assessment validity and whether the model would increase productivity; the authors call for further investigation.The authors suggest limited time and lack of adaptation to experts’ specific contexts may explain uncertainty about validity.Source: §7.3.4 · pp. 30, 32
7.3.5. Threats to Validity
- The authors note low statistical power, a non-representative practitioner sample, possible interview contamination, and participants whose daily work involved quality assessment only indirectly.Each participant assessed a different system tied to their area of expertise.Source: §7.3.5 · p. 32
8. Specific Quality Model for Embedded Systems
The embedded-systems quality model extends the base model with C and C++ modules and embedded-specific factors. Its assessments matched an expert’s product ordering, while retaining a boundary around general quality factors outside the embedded domain.
8. Specific Quality Model for Embedded Systems
- The ESQM extends the base model with embedded-specific factors, developed from literature analysis of guidelines and quality standards for embedded and safety-critical systems.The requirements drew on functional-safety, medical-device, railway, MISRA C/C++, and JSF AV C++ standards and guidelines.Source: §8 · pp. 32–33
8.1. Contents
- The model uses the meta-model’s modularisation, separating a core C module from a C++ module, with measures quantifying factors that refine or impact other factors and quality aspects.The model was calibrated using normalized measurements from 17 open-source C and C++ projects.Source: §8.1 · p. 33
- The ESQM contains 162 leaf-level product factors, 336 measures, 177 impacts on requirements, and 128 impacts on ISO/IEC 25010 quality aspects.The model also provides aggregation functions for all ISO/IEC 25010 quality aspects to calculate an overall quality statement.Source: §8.1 · p. 33
8.2. Validation
- The ESQM assessment and the expert ranked products A, C, and B in the same order, although the automatic assessment gave better grades.The comparison covered three industrial software products and one embedded-software quality expert; Table 8 presents the comparison.Source: Table 8 · p. 34, §8.2 · p. 34
- The assessment-order match does not establish matching grade values: the automatic method gave better grades, possibly because ESQM omits general factors independent of embedded systems.The authors assume the expert considered these general aspects in the rating.Source: §8.2 · p. 34
8.3. Comparison with Base Model
- Many base-model properties were reused with only minor changes, while 14 properties were removed; C/C++ entities required considerable restructuring.The changes addressed language-specific entities and properties, including elements not relevant to C/C++.Source: §8.3 · pp. 34–35
9. Summary & Conclusions
Quamoco presents operationalised, adaptable quality models and an assessment approach, with evidence of agreement with expert opinion and practitioner understanding. The conclusions also emphasize the effort involved, uneven model coverage, and areas for further development.
9.1. Summary
- The approach combines an explicit meta-model, a broad technology-independent base model operationalised for Java and C#, and a transparent, repeatable assessment approach.The meta-model structures factors, impacts, measures, and modules; the base model captures statically measurable product factors and their impacts on quality.Source: §9.1 · p. 35
- Quamoco provides open-source tooling for model creation and assessment, plus an embedded-software model demonstrating use in a specific context.The embedded model reuses the base model’s infrastructure and can be built with less effort than starting from scratch.Source: §9.1 · p. 35
9.2. Conclusions
- Empirical analyses found reasonable agreement between expert opinion and model-based assessments, alongside increased practitioner understanding, though results were not fully conclusive.Assessments especially matched expert opinion for maintainability; opinions on validity and potential productivity improvement remained inconclusive.Source: §9.2 · p. 35, §9.1 · p. 35
- Building the models and supporting tools took three years and about 23 people, an effort many companies may prefer to avoid by adapting an existing model.Source: §9.2 · p. 35
- The base model covers most ISO/IEC 25010 quality characteristics with varying depth; reliability and security assessments may improve with dynamic analysis.Dynamic assessment of these aspects requires a running system in a suitable environment.Source: §9.2 · p. 36
- Industrial deployment showed the tooling could manage quality and costs across larger projects, while SME use revealed a mismatch requiring better environmental adaptation.The financial software provider deployment handled Java, .NET, C++, and COBOL in an integrated way; a separate SME project found partial disagreement with expert opinion.Source: §9.2 · p. 36
9.3. Future Work
- Future work includes broadening the base model and validation beyond the limited paradigms and technologies studied, and conducting further empirical studies.Source: §9.3 · p. 36
- The tools remain below commercial or mature open-source application standards, while static-analysis-heavy measures limit dynamic assessment and motivate extensions for testing and other dynamic analyses.The authors also identify metric aggregation as an area for further investigation.Source: §9.3 · p. 36
Appendix A. Benchmarking Approach
The appendix describes benchmarking rules for setting utility-function thresholds using measurement distributions. It illustrates how outlier handling and distribution shape affect the selected thresholds.
Appendix A. Benchmarking Approach
- The benchmarking procedure determines minimum and maximum thresholds for utility functions using normalized measurement results from baseline systems.The passage identifies Q1 and Q3 as the 25th and 75th percentiles and IQR as their inter-quartile range.Source: Appendix A · p. 40
- Outliers beyond Q3 + 1.5 × IQR or below Q1 − 1.5 × IQR are excluded to avoid extreme thresholds that can cluster evaluation results and reduce differentiation.Source: Appendix A · pp. 40–41
- For the bitwise-add measure, 108 of 110 systems had a normalized value of 0, producing a simple jump function at 0.The measure is based on rule-based static analysis and normalized by lines of code.Source: Appendix A · p. 40
- Clone coverage uses the minimum and maximum non-outlier values, 0.0 and 0.57, as thresholds; missing-JavaDoc values are more evenly distributed and use non-outlier thresholds.The appendix describes three measure histograms, with values to the right of dotted lines treated as outliers.Source: Appendix A · p. 41, Figure A.11 · p. 41