PaperlayerView exact revision on arXiv

Source-linked AI summary

To Automatically Map Source Code Entities to Architectural Modules with Naive Bayes

Tobias Olsson, Morgan Ericsson, Anna Wingkvist

arXiv:2109.09525v1cs.SE

TL;DR

Source-to-module mappings are often absent and costly to create or validate, limiting a practical component of static architecture conformance checking. The paper evaluates a Naive Bayes classifier that combines source-code dependency and text information. It reports better performance than the compared state-of-the-art functions across all subject systems, with less parameterization.

  • Problem

    Source-code entity mappings to architectural modules are often missing, and creating or validating them manually requires too much effort.

    Source: §1 · p. 1

  • Method

    The authors train a Naive Bayes classifier using source-code information, including dependencies and identifier names, to map entities to architecture modules.

    Source: §1 · p. 1, §2.4 · p. 4

  • Results

    NBAttract has a higher median F1 score and lower variance than CountAttract, IRAttract, and LSIAttract across initial-set sizes in all subject systems.

    Source: §9 · p. 16

  • Takeaways & Limitations

    The authors present NBAttract as a candidate for static architecture conformance tools and a baseline for research into more advanced mapping techniques.

    Source: §9 · p. 16

    The authors note that F1 may be misleading for some purposes, and NBAttract’s precision-recall balance depends on adjusting its classification threshold.

    Source: §6 · p. 13

Abstract

from arXiv · show

Background: The process of mapping a source code entity onto an architectural module is to a large degree a manual task. Automating this process could increase the use of static architecture conformance checking methods, such as reflexion modeling, in industry. Current techniques rely on user parameterization and a highly cohesive design. A machine learning approach would potentially require fewer parameters and better use of the available information to aid in automatic mapping. Aim: We investigate how a classifier can be trained to map from source code to architecture modules automatically. This classifier is trained with semantic and syntactic dependency information extracted from the source code and from architecture descriptions. The classifier is implemented using multinomial naive Bayes and evaluated. Method: We perform experiments and compare the classifier with three state-of-the-art mapping functions in eight open-source Java systems with known ground-truth-mappings. Results: We find that the classifier outperforms the state-of-the-art in all cases and that it provides a useful baseline for further research in the area of semi-automatic incremental clustering. Conclusions: We conclude that machine learning is a useful approach that performs better and with less need for parameterization compared to other approaches. Future work includes investigating problematic mappings and a more diverse set of subject systems.

1. Introduction

Static architecture conformance checking needs mappings from source entities to architectural modules, but such mappings are often absent and costly to create or validate manually. This paper combines source-code information in a machine-learning classifier and compares it with existing mapping approaches.

1. Introduction

  • Manual source-to-module mappings are often missing, and creating or validating them can require too much effort.The paper presents automatic mapping support as a promising way to increase the usefulness of static architecture conformance checking in practice.Source: §1 · p. 1
  • Existing automated mapping functions use syntactic dependencies or semantic similarity, while the authors combine multiple information sources as classifier features.The approach uses dependency graphs and identifier names; the study also examines adding syntactic and semantic information to existing text-based functions.Source: §1 · p. 1
  • The study evaluates whether a Naive Bayes classifier performs better than state-of-the-art mapping functions and how syntactic and semantic information affects text-based functions.The introduction also describes a comparison with three state-of-the-art mapping functions.Source: §1 · pp. 1–2
  • The authors report that their approach outperforms the state of the art, requires a smaller initial set and fewer parameters, and combines syntactic with semantic information.They also describe an open-source implementation and analysis of initial sets and attraction-function parameterization.Source: §1 · p. 2

2. Background

The paper frames mapping as assigning source-code entities to architectural modules so implementation dependencies can be compared with allowed architectural dependencies. It situates NBAttract within semi-automatic incremental clustering and describes its use of dependency and text information.

2. Background

  • Structural architecture organizes source code into cohesive modules, and mapping entities to modules enables comparison of implementation dependencies with allowed architectural dependencies.The source model contains entities and dependencies; the architecture model contains modules and allowed module dependencies.Source: §2 · p. 2

2.1. Semi-automatic Incremental Clustering

  • Orphan adoption iteratively maps unmapped entities using an initial manual mapping, attraction values, automatic decisions, and manual review of unresolved entities.Naming, structure, style, and semantics are identified as criteria that can guide mapping.Source: §2.1 · pp. 2–3

2.2. Initial Set and Orphan Set Construction

  • Small initial sets need representative composition, while larger sets provide more training data but require significant manual effort; general heuristics for composing small sets remain unresolved.The paper also describes filtering candidate orphans using a dependency-ratio threshold chosen by the end-user.Source: §2.2 · p. 3

2.3. Attraction Functions, Training, and Mapping

  • Existing attraction functions use dependency counts, information retrieval similarity, or classification; some require cohesion assumptions, user-set parameters, or criteria for automatic mapping.CountAttract weights dependency types; IRAttract uses cosine similarity, LSIAttract uses latent semantic indexing, and HuGMe describes attraction-based mapping conditions.Source: §2.3 · pp. 3–4, §2.4 · p. 4

2.4. Concrete Dependency Abstraction

  • Concrete Dependency Abstraction turns implementation dependencies into textual terms encoding source module, dependency type, and target module, which can be combined with identifier terms for classification.NBAttract builds a probabilistic model from mapped entities and estimates each orphan’s probability of belonging to each module.Source: §2.4 · p. 4

3. Method

The study compares four attraction functions, including Naive Bayes, across eight Java systems with known mappings, testing the effect of dependency terms and performance across varied initial mappings. It evaluates mappings against ground truth using precision, recall, and F1.

3. Method

  • The experiment compares CountAttract, IRAttract, LSIAttract, and NBAttract, omitting MQAttract based on prior findings that CountAttract outperforms it.It extends earlier work by adding the two information-retrieval functions and examining CDA.Source: §3 · pp. 4–5
  • Each run randomly selects an initial mapping and attraction-function parameters, then iteratively maps orphans without human correction.The evaluation compares resulting mappings with known mappings; automatic errors and unmapped orphans may remain.Source: §3 · p. 5

3.1. Experiment Outline

  • The experiment outline generates initial mappings by randomly removing entities while retaining at least one mapped entity per module, then repeats runs until 50,000 executions.The experiment initializes each attraction function, calculates orphan-to-module attractions, maps eligible orphans, and records data.Source: §3.1 · p. 5

3.2. Subject Systems

  • The evaluation covers eight open-source Java systems with documented architectures and mappings, drawn from prior studies, workshop contributions, and a contributor-provided case.The systems span tools, libraries, desktop applications, and a web application.Source: §3.2 · p. 6

3.3. Data Analysis

  • Performance is assessed with precision and recall, combined as F1; statistical tests examine F1 differences and effect sizes across initial-set sizes.The Wilcoxon rank-sum test uses p < 0.05, with median differences and r reported; visualizations show medians and percentile limits across the interval.Source: §3.3 · p. 6

4. Implementation of the Attraction Functions

The attraction functions use dependencies, extracted source-code terms, or both to score orphan-to-module matches. Their implementations differ in how they weight dependencies, represent text, and derive attraction values.

4.1. Attraction Calculation

  • CountAttract uses compiled-source dependencies, with system-specific dependency weights optimized genetically and rounded to five coarse values.The evaluated weights come from {0, 0.25, 0.5, 0.75, 1}; attraction values determine automatic mapping under the described threshold procedure.Source: §4.1 · p. 7

4.2. Term Generation

  • IRAttract, LSIAttract, and NBAttract derive terms from compiled source code, including names and string constants, but not comments.Terms are split on common identifier casing boundaries, stemmed, and filtered to words at least three characters long; dependency terms can also be added.Source: §4.2 · p. 7
  • IRAttract represents normalized term frequencies as vectors and scores orphan-to-module matches with cosine similarity.Term frequencies are normalized by each document’s maximum term frequency rather than using TF-IDF.Source: §4.2 · p. 7
  • LSIAttract reduces a raw-frequency term-document matrix to a low-rank space, then uses cosine similarity to calculate attractions.Its rank is set to the number of modules.Source: §4.2 · pp. 7–8
  • NBAttract trains multinomial Naive Bayes on module documents and uses classification probabilities as attraction values.The implementation uses term occurrence rather than term frequency, based on initial experiments.Source: §4.2 · pp. 7–8

5. Results

Dependency terms generally improve the text-based attraction functions, with three exceptions among the tested cases. The results section compares CDA variants and four attraction functions across systems, mapping sizes, and performance measures.

5.1. The Effect of CDA

  • CDA terms improve performance in 21 of 24 cases, with exceptions for IRAttract in Lucene and A.UML and LSIAttract in Lucene.For NBAttract in ProM, the comparison between CDA and no CDA is not statistically significant; the other reported comparisons are significant.Source: §5.1 · p. 8, Figure 3 · p. 8

5.2. Attraction Functions Comparison

  • The attraction-function comparison uses the best CDA versions of NBAttract, IRAttract, and LSIAttract, plus filtered CountAttract data; all reported comparisons are significant at p < 0.05.The section presents distributions across systems, running median F1 across initial-set sizes, and overall precision, recall, and F1.Source: §5.2 · pp. 8, 10, Figure 5 · p. 10, Figure 6 · p. 10

5.1. The Effect of CDA

  • Figure 4 plots running median F1 and the 75th- and 25th-percentile limits against initial mapping size for each system and attraction function, with and without CDA.Figure 3 compares the functions with and without CDA for each system; its paired plots distinguish CDA from non-CDA cases.Source: Figure 3 · p. 8, Figure 4 · p. 9
  • Table 3 reports CDA-versus-no-CDA F1 significance tests, standardized effect sizes, median differences, and sample counts.Its caption states that results are significant at p < e−16 and marks negative median differences as cases where CDA is not beneficial.Source: Table 3 · p. 10

6. Analysis

NBAttract outperforms the other attraction functions across systems and initial mapping sizes, with less score variation. Adding CDA terms generally helps, though its effect varies by system and attraction function.

6. Analysis

  • NBAttract outperforms the other functions across systems and initial-set sizes, with less variation in its scores.Its median score exceeds the other functions’ 75th-percentile scores for every system.Source: Figure 7 · p. 11, §6 · p. 13
  • CDA use produced a positive effect in 21 of 24 cases: in 13, the median CDA result exceeded the non-CDA counterpart's 75th percentile, while eight were positive but below it.Source: §6 · p. 12
  • Lucene and ProM have high, consistent mapping results, where naming may be a strong signal and mixing information sources may lower performance.The authors suggest consistent terms in these systems and identify naming as a possible contributor, while noting that the contributing factors are uncertain.Source: §6 · pp. 12–13
  • The authors recommend CDA for text-based classifiers that can balance term frequencies so CDA terms do not dominate classification.They report that CDA’s overall benefit can weaken as the initial set grows for IRAttract in Ant, ArgoUML, and JabRef, and say more experiments are needed.Source: §6 · p. 13
  • All attraction functions vary more near the beginning and end of initial-set intervals, indicating dependence on initial-mapping composition.The authors note that functions require a representative initial mapping; some entities are particularly hard to map and can strongly affect scores.Source: §6 · p. 13
  • No attraction function has a perfect median score, and F1 may be misleading when mapping goals differ; NBAttract’s probability threshold can balance precision and recall.The experiment fixed NBAttract’s threshold at 0.9, which seemed to favor recall over precision; the threshold’s effect merits empirical investigation.Source: §6 · p. 13

7. Threats to Validity

The study’s validity is bounded by its Java subject systems, seed-set design, evaluation choices, and uncertainty in mappings and comparator implementations. These constraints qualify how its performance results should be interpreted.

External validity

  • Across Java systems, source organization may reflect architecture, while mostly small-to-medium subjects leave transfer to other languages, radically different structures, and very large systems uncertain.The authors also note that erosion and diverse architectural decisions could make large systems harder to map.Source: §7 · p. 13

Initial sets

  • Random initial sets may not resemble expert-selected seeds, so the reported F1 scores may not reflect real-world performance, although they support comparisons across easier and harder cases.Experts might map easy entities or representative examples first, leaving harder cases for automation.Source: §7 · pp. 13–14

Human intervention

  • The evaluation omits human correction during mapping, whose effect could not be assessed because no easy mechanism for checking mapping correctness was available.The authors suggest that a larger initial set could help assess human guidance.Source: §7 · p. 14

Construct validity

  • Overall F1 can conceal poor performance on minority classes, including entities in tiny modules, and the analysis does not assess those class-level problems.The authors emphasize effect sizes, reported numerically through median difference and r-value and visually through plots.Source: §7 · pp. 14–15

Ground truth

  • Ground-truth mappings may be subjective: four systems use expert mappings, while four use researcher-derived mappings that were not validated by architects or developers.The authors nevertheless consider most mappings likely to be high quality, citing consistency checks against commits and discussions.Source: §7 · p. 14

Internal validity

  • The attraction-function implementations follow the authors’ interpretation of prior publications; IRAttract and especially LSIAttract are insufficiently described, though reproduced results and large performance gaps make ranking changes seem unlikely.LSI’s small number of architecture modules and unclear initialization choices contribute to this uncertainty.Source: §7 · p. 14

Parameterization

  • CountAttract depends on expert-set dependency weights and parameters, and its performance is sensitive to those choices; experts may not find optimal values in practice.The study assumes experts would select roughly optimal weights, which the authors consider improbable for every parameter.Source: §7 · p. 14

Statistical analysis

  • Large datasets make very low p-values unsurprising, and repeated comparisons ideally call for an adjusted threshold; the authors say adjustment would not change significance here.They retain potential outliers and use a test robust to their effects, alongside plots across initial-set sizes.Source: §7 · p. 14

8. Related Work

Related work spans orphan adoption, attraction functions, and architecture recovery. The paper positions NBAttract against methods using structural or textual signals and approaches combining heterogeneous information.

Orphan adoption

  • Tzerpos and Holt frame orphan adoption as mapping new entities and orphan kidnapping as deciding when already mapped entities should be remapped.Their approach uses naming, structure, and style criteria, with style classification limited to two styles for kidnapped orphans.Source: §8 · p. 15
  • In an industrial case, Tzerpos and Holt’s technique suggested changing 46 mappings; developers agreed in 33 cases and considered the other 13 valid but not optimal.For those 13, developers said entity restructuring was needed to justify the original modules.Source: §8 · p. 15

Supervised mapping

  • Bibi et al. found Naive Bayes, k-nearest neighbors, and the original structural method produced the same results; the paper argues their Naive Bayes probability estimate is too simplistic to learn training patterns.NBAttract instead learns dependency patterns and can use identifier semantics.Source: §8 · p. 15

HuGMe

  • HuGMe adds iterative human intervention and uses convergent and divergent dependencies; its evaluation found CountAttract better overall than MQAttract, particularly for limiting suggested modules.The comparison covered four systems and varied initial-set sizes and CountAttract parameters.Source: §8 · p. 15

Attraction functions

  • CountAttract assumes low coupling and high cohesion, although modules may instead reflect other design goals, including documented rationales that violate that principle.Candela et al. reported low structural and conceptual cohesion and coupling in many studied systems.Source: §8 · p. 15
  • Bittencourt et al. found LSIAttract combined with CountAttract had the best overall F1 in their four-system evolutionary evaluation, while noting that automatic-mapping faults need attention.They considered semi-automatic mapping feasible and its evolutionary overhead lower than manual work.Source: §8 · p. 15

Architecture recovery

  • Architecture-recovery clustering can combine heterogeneous information, but it works from implementation while Reflexion modeling lets architects express intended architecture and multiple views.The cited work also describes AAR clustering as potentially suitable for SACC.Source: §8 · pp. 15–16
  • Other recovery approaches combine textual and syntactic features or contextual and structural information; Naim et al.’s approach requires developer feedback for acceptable architecture recovery.Misra et al. found all features performed best when no explicit feature-selection approach was available.Source: §8 · p. 16

9. Conclusions

The paper introduces NBAttract, a Naive Bayes attraction function combining dependency and semantic information. Experiments report stronger mapping performance than the compared methods, while identifying qualifications and future work.

Conclusions

  • NBAttract has higher median F1 and lower variance than CountAttract, IRAttract, and LSIAttract across initial-set sizes in all subject systems.The authors also describe it as requiring only a classification threshold and as a useful baseline for further research.Source: §9 · p. 16
  • CDA’s dependency terms generally help text-based attraction functions, but can dominate IRAttract and LSIAttract in some cases; NBAttract can adapt their value, unlike IRAttract without manual adjustment.The authors therefore do not generally recommend CDA with IRAttract.Source: §9 · p. 16
  • CountAttract’s dependency weights significantly affect performance, and NBAttract outperforms the other approaches even when they use optimized weights.The authors consider good weights difficult for experts to set, especially across dependency types.Source: §9 · p. 16

Future work

  • The study is limited to Java and mostly small-to-medium systems, so larger and more diverse systems, especially those whose source organization does not mirror architecture, need evaluation.The authors also call for further investigation of problematic mappings and precision.Source: §9 · p. 16
  • Some entities were never mapped correctly unless included in the initial set, motivating study of whether hard cases are edge cases or reflect human mapping errors.The authors suggest identifying such cases so they can be assigned to a human.Source: §9 · p. 16

Related papers

Loading 2109.09525v1…