PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Automatic Quantification of Bayesian Networks for Software Decision Support: Comparing WSA and RNM in a Software R&D Organization

Mirko Perkusich, João Nunes, Emilia Mendes, Emanuel Dantas, Ademar Sousa, Danyllo Albuquerque, Kyller C. Gorgônio, Angelo Perkusich

arXiv:2609.31275v1cs.SE

TL;DR

Software decisions often rely on expert-driven Bayesian networks when historical data are sparse, but quantifying their conditional probability tables requires many judgments and comparative operational evidence is limited. This case study compares WSA and RNM as complete pipelines in two contexts within one software R&D organization, using walkthroughs and retrospective decisions. WSA matched more walkthrough expectations, while both methods ranked selected alternatives near the top despite materially different probability distributions.

  • Problem

    Comparative evidence on semi-automatic CPT methods in operational software decision processes is limited, although CPT quantification can require many probability judgments.

    Source: §1 · p. 3

  • Method

    An embedded case study compared WSA and RNM pipelines in two software decision contexts using expert-defined walkthroughs and retrospective reconstructions, with graph, priors, and evidence fixed within each context.

    Source: §3.1 · p. 10, §7 · p. 27

  • Results

    WSA matched 7/7 versus RNM’s 4/7 walkthrough expectations in Context A and 6/8 versus 4/8 in Context B; both ranked selected alternatives near the top, despite median TVDs of 0.304 and 0.340.

    Source: §7 · p. 27

  • Takeaways & Limitations

    For comparable models, method selection and validation should consider node semantics, available expert judgments, calibration, complete distributions, and intended downstream use.

    Source: §7 · p. 27

    Evidence is bounded by one organization and two purposefully selected ordinal models, and the study did not compare total elicitation effort or sensitivity to alternative RNM calibration settings.

    Source: §7 · p. 28

Abstract

from arXiv · show

Expert-driven Bayesian networks can support recurring software decisions when historical data are limited, but quantifying their conditional probability tables (CPTs) requires many probability judgments. Semi-automatic methods reduce direct elicitation, yet practitioners have little comparative evidence from operational decision processes. We report an embedded case study comparing the Weighted Sum Algorithm (WSA) and Ranked Nodes Method (RNM) as complete elicitation-and-quantification pipelines in two contexts within a single software research and development (R&D) organization: feature selection for Internet of Things projects and user interface design selection. Within each context, the pipelines shared the graph, root priors, and decision evidence. We evaluated them through 15 expert-defined model-walkthrough scenarios and retrospective reconstructions of alternatives recorded in decision meetings. WSA matched 7/7 and 6/8 walkthrough expectations, whereas RNM matched 4/7 and 4/8. Both pipelines placed the selected alternatives near the top of the retrospective rankings. Their complete distributions nevertheless differed: the median total variation distance was 0.304 across four distinct feature patterns and 0.340 across seven distinct design patterns, rising to 0.890 for one design pattern. These results concern two ordinal value-estimation models in one organization. They show that similar shortlist behavior does not imply equivalent representations of uncertainty. For comparable software decision-support models, method selection and validation should consider node semantics, the judgments experts can provide, calibration requirements, complete output distributions, and the intended downstream use of the probabilities.

1. Introduction

The paper addresses limited operational evidence for comparing semi-automatic CPT quantification methods in software decisions. It compares WSA and RNM in two organizational contexts using expert-defined walkthroughs and recorded decisions.

1. Introduction

  • CPT elicitation can become burdensome as parent configurations multiply, while repeated probability judgments risk fatigue, inconsistency, and cognitive burden.Semi-automatic methods reduce direct input but encode relationships differently and require distinct expert judgments and calibration choices.Source: §1 · pp. 2–3
  • Operational comparisons of WSA and RNM are limited, especially in software decision processes, and reduced elicitation alone does not establish organizational suitability.Practitioners also need evidence about expected behavior, decision support, and uncertainty representation for downstream use.Source: §1 · p. 3
  • The embedded case study compares WSA and RNM in feature-selection and user-interface design contexts, holding each context’s graph, root priors, and decision evidence fixed.The study reuses VALUE records; earlier studies used WSA models, while RNM quantifications and a paired comparison were not reported there.Source: §1 · pp. 3–4
  • The study asks whether each method reproduces expert-defined modal walkthrough expectations and how its rankings and distributions correspond to stakeholder selections in historical meetings.Walkthrough labels are expert-defined expectations, while historical choices are decisions rather than objective measurements of value.Source: §1 · p. 3
  • The study contributes an operational account of both pipelines, a paired evaluation, and practice-oriented selection and validation criteria informed by distribution-level comparison.Its evidence is bounded by two models in one organization.Source: §1 · p. 4

2. Background and related work

This section frames CPT quantification as a choice among methods with different assumptions, expert inputs, and evaluation needs. It introduces expert-driven quantification, semi-automatic method families, WSA, RNM, and prior comparison evidence.

2.1. Expert-driven BN quantification

  • Expert-driven BNs are useful when representative data are scarce or organization-specific judgments matter, but CPT size and row-by-row elicitation can make quantification costly and inconsistent.A node with r states and parents with q_i states has r times the product of q_i free probabilities after row normalization.Source: §2.1 · p. 5

2.2. Families of semi-automatic CPT methods

  • Semi-automatic CPT methods reduce elicitation through reduced causal models, interpolation, weighted functions, pairwise comparisons, or other judgment formats, but suitability depends on assumptions and node semantics.The reviewed approaches also include fuzzy-logic procedures and methods combining observations with expert judgment; WSA and RNM address cases without sufficiently representative data.Source: §2.2 · pp. 5–6

2.3. Weighted Sum Algorithm

  • WSA elicits compatible parent configurations, child distributions, and relative parent weights, then combines assessed distributions through weighted sums to fill remaining CPT rows.Its practical output also depends on elicitation interface and facilitation, not only the completion algorithm.Source: §2.3 · pp. 6–7

2.4. Ranked Nodes Method

  • RNM maps ordinal states to unit-scale intervals and uses weighted parent values with a doubly truncated distribution to allocate probability across child states.Its expression, weights, variance, and calibration choices affect the resulting CPT; variants include weighted-minimum, weighted-maximum, and weighted-mean relationships.Source: §2.4 · p. 7

2.5. Empirical comparisons and research gap

  • Earlier WSA–RNM comparisons identified trade-offs rather than a consistently dominant method, motivating evaluation within the decision processes the pipelines are meant to support.The present study adds a paired evaluation of expected behavior, recorded-alternative rankings, and complete distributions with graph, priors, and evidence held fixed.Source: §2.5 · pp. 7–9
  • Prior comparisons include software-effort estimation and human-reliability modeling; the study evaluates full distributions alongside modes and rankings because those summaries can conceal differences.Table 1 summarizes prior work most closely related to the WSA–RNM comparison.Source: §2.5 · pp. 8–9, Table 1 · p. 8

3. Research method

This embedded case study compares WSA and RNM quantifications in two ongoing software decision contexts, holding each context’s graph and evidence fixed. It evaluates expert-defined walkthrough expectations and retrospective rankings and distributions for recorded alternatives.

3.1. Study design and units of analysis

  • The study compared paired WSA and RNM quantifications in two organizational decision contexts, evaluating walkthrough responses and retrospective alternative rankings and distributions.The graph and evidence were fixed within each context, so the comparison concerned the complete elicitation-and-quantification pipelines.Source: §3.1 · pp. 9–10

3.2. Organizational setting

  • The study took place within ongoing project work at a university-based software R&D laboratory, where feature and interface-design choices were recurring multi-stakeholder decisions.Teams worked in a Scrum-based process with project managers, developers, partner representatives, and cross-project quality and user-experience groups.Source: §3.2 · pp. 10–11

3.3. The VALUE framework and Value tool

  • The VALUE framework links stakeholder-defined value factors, actual decision records, and organization-specific Bayesian models through five activities.This study focused on constructing and evaluating the models, using earlier factor and decision records as inputs.Source: §3.3 · pp. 11–12

3.4. Selection and description of the decision contexts

  • Context A covered feature selection for two Internet-of-Things applications, while Context B covered user-interface design selection for a project-management application.The contexts supplied recorded alternatives and factor assessments for the paired evaluations.Source: §3.4 · pp. 12–13

3.5. Participants and study preparation

  • Two domain experts per context reviewed the models, while three Bayesian-network researchers supported quantification; experts had prior involvement in the relevant VALUE activities.Experts received a 20-minute introduction to Bayesian-network graphs, probabilistic reasoning, and what-if analysis.Source: §3.5 · pp. 13–14

3.6. Model construction and quantification

  • Within each context, WSA and RNM shared the graph, root priors, and manually elicited one-parent CPTs; only their generated multi-parent CPTs differed.Root priors came from recorded assessments, while experts reviewed graph structure and elicited direct judgments for one-parent CPTs.Source: §3.6 · pp. 14–15

3.7.1. RQ1: walkthrough-scenario agreement

  • Walkthrough evaluation counted exact matches between each model’s modal output and expert-defined expected labels across seven Context A and eight Context B scenarios.The scenarios were model-review judgments, not independent observed outcomes or a test set.Source: §3.7.1 · p. 17

3.7.2. RQ2: retrospective selection concordance

  • Retrospective evaluation ranked alternatives by P(High), compared rankings with recorded selections, and measured distributional differences using total variation distance.The analysis concerned ranking concordance and distributional differences, not objective value classification or prospective predictive accuracy.Source: §3.7.2 · pp. 17–18

4. Results

WSA matched more expert-expected walkthrough labels than RNM in both contexts. Both pipelines ranked selected alternatives near the top, while producing different probability distributions.

4.1. RQ1: WSA agreed with more walkthrough labels in both contexts

  • WSA matched 7/7 and 6/8 walkthrough expectations, compared with RNM’s 4/7 and 4/8.The walkthrough labels were expert-defined modal expectations.Source: §4.1 · p. 18, Table 5 · p. 19

4.2. RQ2: both pipelines placed selected alternatives near the top but produced different distributions

  • In Context B, selected designs 2 and 7 tied highest under both pipelines, yet TVD reached 0.890 across seven patterns (median 0.340).The modal states differed for three patterns.Source: §4.2 · p. 19
  • In Context A, selected features ranked near the top, but feature-pattern TVD ranged from 0.250 to 0.380 (median 0.304) despite matching modal states across all four patterns.Feature 3 tied under both methods with rejected features 2, 4, and 5.Source: §4.2 · p. 18, Table 6 · p. 20

5. Discussion

The discussion separates walkthrough agreement, retrospective rankings, and full-distribution comparisons, then draws practice and research implications from their differences. It emphasizes that similar rankings can coexist with different probability allocations.

5.1. Interpretation of the findings

  • WSA reproduced more walkthrough expectations, while both methods placed selected candidates near the top; their distributions nevertheless diverged, reaching TVD 0.890 in Context B.The authors distinguish scenario agreement from retrospective selection concordance and note that the full distributions add information beyond modes and rankings.Source: §5.1 · p. 20
  • RNM’s low or absent simulated probability mass on Low reflects the complete implemented pipeline, so the study cannot attribute it to any single RNM component.Elicited extreme-state targets, WMIN/WMAX expressions, parent weights, variance, and calibration search jointly determined the CPTs.Source: §5.1 · p. 21

5.2. Relationship to previous evidence

  • Prior evidence aligns with higher WSA walkthrough agreement but differs in outcome definition; RNM settings remain consequential, and similar rankings do not establish equivalent distributions or downstream decisions.Baker (2009) measured prediction error against observed software effort, whereas this study measured agreement with scenario expectations; Laitila and Virtanen (2020) describe sensitivity to user-selected settings, and Rohmer (2020) distinguishes CPT derivation from evaluating its assumptions.Source: §5.2 · pp. 22–23

5.3. Implications for software decision-support practice

  • Choose a pipeline to fit node semantics and the judgments experts can provide, then validate model behavior, rankings, and complete distributions according to intended use.WSA suits representative distributions and relative parent influence; RNM suits ordinal relationships with defensible expressions, weights, variance, and calibration targets. An audit trail and representative CPT pilot can support review.Source: §5.3 · pp. 23–24
  • The authors suggest grounding proposed quantification inputs in project records and explicit value-factor definitions, while leaving contextual interpretation and acceptance to stakeholders.This is presented as a possible LLM-assisted workflow for preparing WSA or RNM judgments.Source: §5.3 · p. 24

5.4. Implications for research on BN quantification

  • Future comparisons should evaluate methods as elicitation pipelines, separate expected behavior, rankings, and distribution agreement, and test whether parameter changes affect downstream decisions.The authors recommend prospective evaluation, balanced facilitation and information budgets, separate effort measurement, and sensitivity analyses of WSA and RNM inputs; replication across organizations and decision types is still needed.Source: §5.4 · pp. 24–25

6. Threats to validity and study limitations

The validity discussion limits interpretation to company-specific ordinal constructs, internally defined walkthrough expectations, and retrospective selection concordance. It also identifies implementation differences, bounded sampling, transferability, and reproducibility constraints.

Construct validity

  • “Overall Value” is a company-specific three-state ordinal construct, and walkthrough labels are internally developed expectations rather than observed outcomes.Historical selections are an imperfect proxy for value because dependencies, capacity, strategy, or negotiation may also shape decisions.Source: §6 · p. 26

Internal validity

  • WSA and RNM differed in facilitation, interaction mode, tools, and calibration, while some Context B priors came from meetings later than the reconstructed alternatives.The retrospective findings are model reconstructions, not temporally held-out performance.Source: §6 · p. 26

Conclusion validity

  • One organization, two selected contexts, and 15 walkthrough scenarios limit inference; alternatives share participants, model structure, priors, and meetings and are not independent replications.The descriptive, within-context agreement counts and TVD summaries are not population estimates.Source: §6 · p. 26

External validity

  • Organization-specific value factors, graphs, participants, and calibration targets constrain transfer; current workflows and other organizations, topologies, state cardinalities, and decision types need further evaluation.The contextual account from one professional does not establish that the original factors or comparative performance transfer unchanged.Source: §6 · p. 26

Reliability and reproducibility

  • Confidential organizational records and WSA artifacts are unavailable, RNM calibration settings are incomplete, and alternative calibration sensitivity and commensurate total-effort comparisons were not analyzed.The related-work search was targeted rather than systematic, so relevant studies may have been missed.Source: §6 · p. 27

7. Conclusion

The conclusion reports different outcomes across evaluation levels: WSA matched more walkthrough expectations, both pipelines ranked selected alternatives near the top, and their full distributions sometimes differed substantially. It recommends selection and validation based on model semantics, available judgments, and intended use.

7. Conclusion

  • WSA matched more walkthrough expectations than RNM, both ranked stakeholder-selected alternatives near the top, and median TVD reached 0.304 in Context A and 0.340 in Context B, with 0.890 for one design pattern.Similar shortlist behavior did not imply equivalent representations of uncertainty.Source: §7 · p. 27
  • Practitioners should match WSA or RNM to node semantics and available expert judgments, then check behavior, rankings, and complete distributions according to how the BN will be used.The conclusion also recommends recording elicited inputs, calibration choices, and software settings for later review and maintenance.Source: §7 · pp. 27–28
  • The evidence is bounded by one organization, two ordinal models, retrospective records, and differently implemented pipelines; total effort and sensitivity to alternative RNM calibration settings were not compared.The authors call for prospective replication, balanced facilitation, separate effort measurement, and tests of downstream effects.Source: §7 · p. 28

Data availability

The article reports model and analysis artifacts, but the repositories do not provide a complete case-study replication package or confidential organizational records.

Data availability

  • Public repositories provide RNM prototype and algorithm artifacts and the WSA algorithm, but not an executable replication package for the case study.Source: Data availability · p. 28
  • The article reports the model graphs, factor definitions, retained RNM parameterizations, and analyzed output distributions.Source: Data availability · p. 28
  • Underlying organizational decision records and detailed elicitation artifacts are confidential and unavailable publicly.Source: Data availability · p. 28

Appendix A. Practitioner consultation on current software decision contexts

Appendix A presents six questions and the VIRTUS technical lead’s complete written responses, translated from Portuguese into English, with the respondent’s authorization.

Appendix A. Practitioner consultation on current software decision contexts

  • The appendix reproduces six questions and the complete responses from a VIRTUS technical lead consulted about current feature-prioritization and interface-design practices.The questions and responses were translated from Portuguese into English, and the respondent authorized publication.Source: Appendix A · p. 29
  • The respondent authorized publication of the complete written responses.Source: Appendix A · p. 29
  • The consultation concerns current feature-prioritization and interface-design practices.Source: Appendix A · p. 29

Appendix A.1. What is your experience with these decisions?

The respondent is a VIRTUS technical lead with approximately nine years of experience in software decision activities, including five years at VIRTUS.

Appendix A.1. What is your experience with these decisions?

  • The respondent has worked on these activities for approximately nine years, including the last five at VIRTUS.Source: Appendix A.1 · p. 29
  • As a VIRTUS technical lead, the respondent participates in requirements elicitation and prioritization, feature definition, and product-interface decisions.Source: Appendix A.1 · p. 29
  • The respondent’s experience includes projects involving requirements, features, and product interfaces.Source: Appendix A.1 · p. 29

Appendix A.2. Are these two types of decisions present in current projects?

A recent project combined research objectives with usability, response-time, and interface-modernization goals, and involved team and client discussions about design alternatives.

Appendix A.2. Are these two types of decisions present in current projects?

  • The recent project sought to improve usability and response time while modernizing a client-used system with a complex workflow and device-related performance problems.Source: Appendix A.2 · p. 29
  • The technical team, design team, and client discussed new components and interface alternatives using client references and designer proposals.Source: Appendix A.2 · p. 29
  • The technical lead evaluated proposals by balancing client expectations, implementation feasibility, and usability.A more elaborate visual solution could introduce unnecessary complexity for users.Source: Appendix A.2 · p. 29

Appendix A.3. What information supports these choices?

Current software choices draw on client needs, screen objectives, user feedback, and team assessments; uncertainty is handled through professional discussion and client validation. LLM tools now support preliminary work, while people retain review and final validation responsibilities.

Appendix A.3. What information supports these choices?

  • Client needs, screen objectives, user feedback, and technical and design team assessments support interface choices.Source: Appendix A.3 · p. 30

Appendix A.4. How is uncertainty handled when it arises?

  • When information is insufficient, professionals discuss usability, complexity, and development effort; disagreements are refined through discussion and client validation before development.A proposal is internally reviewed and then presented to the client for validation, avoiding implementation effort on a solution not yet aligned with user expectations.Source: Appendix A.4 · p. 30, Appendix A.3 · p. 30

Appendix A.5. What has changed in recent years?

  • LLMs can structure initial features, suggest implementation order, and prepare a first roadmap version, reducing the work needed to reach that version.The team reviews the proposal and subsequently validates it with the client.Source: Appendix A.5 · p. 30
  • LLM-based tools have reduced interface implementation effort and allow alternatives to be tried and implemented more quickly, while professional assessment and client validation remain necessary.Source: Appendix A.5 · pp. 30–31

Appendix A.6. Do AI tools, including LLMs, participate in these decisions?

  • An internal LLM tool supports interface alternatives, requirements, initial features, and suggested priorities while addressing data protection and confidentiality needs.Professionals review suggestions and retain responsibility for final decisions and applicable product-owner or client validation.Source: Appendix A.6 · p. 31
Loading 2609.31275v1…