PaperlayerView exact revision on arXiv

Source-linked AI summary

Replications, Revisions, and Reanalyses: Managing Empirical Evidence in Software Engineering

Julian Frattini, Jannik Fischbach, Davide Fucci, Michael Unterkalmsteiner, Daniel Mendez, Robert Feldt, Richard Torkar

arXiv:2609.05983v1cs.SE

TL;DR

Individual studies and replication portfolios can be limited as decision support, while synthesis in software engineering remains uncommon. The paper proposes a framework for relating and evolving empirical evidence. It applies that framework to show how evidence relationships can indicate a field’s research frontier.

  • Problem

    Research synthesis in software engineering is uncommon, and replication portfolios do not necessarily support knowledge application or help reconcile contradictory findings.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    The authors use method engineering to build a framework that defines quantitative evidence, relates evidence items, and models the evolution of variance theories.

    Source: §1 · p. 2, §3 · p. 6

  • Results

    The framework relates evidence and helps researchers assess relative internal, external, and conclusion validity, revealing the evolution and current frontier of research on a phenomenon.

    Source: §3 · p. 6, §7 · p. 21

  • Takeaways & Limitations

    The framework offers research communities a way to orchestrate collaboration across studies and systematically evolve empirical evidence in software engineering.

    Source: §3 · p. 6, §7 · p. 21

    The framework integrates quantitative evidence; qualitative evidence can prompt hypothesis revision but is not included as evidence in synthesis.

    Source: §6.3.2 · p. 20

Abstract

from arXiv · show

One aspired outcome of empirical research on quantitative data is a variance theory, i.e., a quantification of the effect of an independent on a dependent variables. The validity of variance theories stems from the synthesis of multiple pieces of evidence, which increases its validity beyond the findings of a single study. However, research synthesis in SE is rare and if done mostly limited to purely narrative syntheses. At best, researchers perform meta-analyses to synthesize variance theories from several quantitative results. But even meta-analyses only produce reliable results when synthesizing exact replications yet fail to generalize from variations. We aim to extend the frontier of research synthesis beyond the state-of-the-art to systematically manage empirical evidence and its evolution. We apply method engineering to construct a framework for research synthesis from proven, individual method fragments. The framework allows researchers to put new evidence in a clear relation to an existing body of evidence and systematically expand knowledge about a studied phenomenon. We demonstrate the application of this framework to two fields of research by explicitly modeling the relationship between existing pieces of evidence. The framework puts three types of evolution of evidence into relation: (1) replications investigate the same hypothesis in a new context to improve external validity, (2) revisions challenge an existing hypothesis to improve internal validity, and (3) reanalyses replace analysis methods to improve conclusion validity. Through a systematic evolution of evidence and clear assessment criteria for each dimension of validity, the proposed framework can determine the frontier of a field of research. The framework provides a perspective to systematically evolve empirical evidence in SE, supporting more constructive and productive advances in our field.

1 INTRODUCTION

Empirical software engineering needs synthesis to turn context-limited primary studies into decision support, but synthesis is uncommon and existing approaches have important constraints. The paper proposes a framework to relate and systematically evolve evidence beyond traditional meta-analysis.

1 INTRODUCTION

  • Individual primary studies have limited scope and context, so synthesis is needed between empirical knowledge creation and its application to practitioners.The intended support includes decisions about tools, techniques, and approaches.Source: §1 · p. 2
  • Synthesizing quantitative research can produce a variance theory that quantifies how independent variables affect dependent variables and supports practitioner decisions.The cited TDD synthesis of 27 studies found a small positive effect on quality but little to no discernible effect on productivity.Source: §1 · p. 2
  • Research synthesis is rare in software engineering and is mostly narrative; meta-analyses are also uncommon and can be inconclusive when evidence contradicts.Current approaches also struggle with complex relationships and may invite unreflected statistical choices.Source: §1 · p. 2
  • The authors use method engineering to assemble a framework that defines quantitative evidence and evidence relationships, then derives how variance theories emerge and evolve.They demonstrate the framework in active fields of software engineering research.Source: §1 · p. 2

2 RELATED WORK

Research synthesis combines findings across primary studies to support knowledge application and develop variance theories. This related-work section describes why synthesis is needed and outlines its steps.

2 RELATED WORK

  • Variance theories synthesize quantitative evidence to estimate how variables affect one another, but may fail to exceed individual studies in validity.Their intended role is to provide practitioners with reliable decision support.Source: §2 · p. 2

2.1 Research Synthesis

  • Single studies are limited by context, while portfolios of replications strengthen findings but do not necessarily support knowledge application or reconcile contradictory results.Research synthesis aggregates and integrates findings about the same phenomenon.Source: §2.1 · p. 3
  • Shepperd’s synthesis process includes formulating a research problem, locating evidence, appraising evidence quality, and synthesizing and interpreting findings.The final step extracts relevant data and draws conclusions about the initial research question from accumulated evidence.Source: §2.1 · p. 3

5. Reporting: disseminating the results in a research report

The related work describes synthesis methods and variance theories, then reviews how synthesis can strengthen validity and where current methods fall short. Its discussion covers external, internal, conclusion, and construct validity.

5. Reporting: disseminating the results in a research report

  • Narrative synthesis summarizes findings textually, vote-counting classifies effect direction or evidence strength, and meta-analysis pools quantitative results.Structured synthesis aggregates heterogeneous evidence using primary studies’ belief values; IPD-S retains study membership but requires raw data from every study.Source: §5 · pp. 3–4

2.2 Theory Building

  • Software engineering distinguishes theories for understanding, process theories, and variance theories, which quantify the strength of effects between variables.Variance theories require synthesis of quantitative evidence and can support decisions such as whether to adopt a technique.Source: §2.2 · p. 4

2.3 Validity of Variance Theories

  • Variance theories aim to exceed single studies in validity across external, internal, and conclusion dimensions, assessed using the dimensions applied to controlled experiments.The paper notes that construct validity instead limits a variance theory to the validity of its individual evidence.Source: §2.3 · p. 4, §2.3.4 · p. 6

2.3.1 External Validity

  • Synthesizing studies can broaden the contexts covered and make an effect more likely to generalize, but variation across replication configurations complicates synthesis.Systematic replications can identify whether a varying dimension influences a phenomenon, but not quantify its significance.Source: §2.3.1 · pp. 4–5

2.3.2 Internal Validity

  • Revisions can improve internal validity by changing causal assumptions, but current synthesis methods struggle with complex relationships beyond two variables.Post-hoc subgroup analyses do not capture moderating or interaction effects.Source: §2.3.2 · p. 5

2.3.3 Conclusion Validity

  • Reanalysis can improve conclusion validity by applying a better-fitting statistical method to available raw data, but synthesis lacks guidance for selecting that method.The cited Bayesian reanalysis violated fewer statistical assumptions than the frequentist baseline and produced more meaningful insights.Source: §2.3.3 · pp. 5–6

2.3.4 Construct Validity

  • Construct validity constrains synthesis when studies measure a construct differently, and the framework focuses on effects between operationalizations rather than concepts.The framework provides no systematic improvement along this validity dimension.Source: §2.3.4 · p. 6

3 GOAL AND METHOD

The paper proposes a framework to improve the validity of variance theories by systematically relating evidence across studies. It assembles existing method fragments while excluding construct-validity work for which no proven fragment was available.

3 GOAL AND METHOD

  • The framework aims to improve variance theories’ internal, external, and conclusion validity and help research communities coordinate collaboration across studies.It is presented as a tool for systematically improving validity beyond individual pieces of evidence.Source: §3 · p. 6
  • Method engineering assembles the framework from existing fragments for replication, causal model comparison, and statistical reanalysis.The method is specified as a process that guides researchers through ordered stages and activities.Source: §3 · p. 6
  • The framework excludes construct validity because no proven method fragment addresses it.Consequently, its variance theories quantify relationships between operationalizations rather than concepts.Source: §3 · pp. 6–7
  • Qualitative triangulation may be valuable, but it does not quantify effect strength.The paper distinguishes this value from its framework’s aim of quantifying effects.Source: §3 · p. 6
  • The assembled method is conceptual in abstraction and process-oriented, modeling the relationships and order of synthesis stages and activities.It is intended to guide a researcher conducting synthesis.Source: §3 · p. 7

4 FRAMEWORK FOR RESEARCH SYNTHESIS

The framework models evidence as hypotheses, data, and analysis methods, then relates studies through replications, revisions, and reanalyses. These relationships support systematic evolution of variance theories as evidence accumulates.

4.1 Evidence

  • A quantitative evidence piece comprises a hypothesis, data, and an analysis method; DAGs represent variables and assumed causal links, while adjustment choices affect effect estimates.Evidence aims to estimate the average causal effect of the main exposure on the outcome; adjustment sets can help avoid confounding.Source: §4.1 · pp. 7–8

4.2.1 Replication

  • Replications retain the hypothesis and analysis method but use new data; agreeing conclusions can improve external validity, while disagreement signals a causal leak requiring revision.Agreement can be assessed by aggregating p-values or comparing coefficient confidence intervals, depending on the method; determining agreement is non-trivial.Source: §4.2.1 · p. 10

4.2.2 Revision

  • A revision proposes a competing hypothesis for the same phenomenon, potentially adding or changing variables and requiring new data when needed variables were not recorded.The competing hypotheses are evaluated by model comparison, with revisions intended to improve internal validity.Source: §4.2.2 · p. 10
  • Deconfounding revisions use testable implications and compare adjusted effect estimates, whereas precision revisions compare out-of-sample predictive power.These evaluation purposes are not interchangeable: greater predictive power does not establish that evidence is deconfounded.Source: §4.2.2 · p. 11
  • Future studies should use the hypothesis with greatest internal validity and collect variables in its adjustment set.This is the framework’s stated way to optimize the internal validity of the resulting variance theory.Source: §4.2.2 · p. 11

4.2.3 Reanalysis

  • Reanalysis applies a different analysis method to the same data under the same causal assumptions, aiming to improve conclusion validity.Reanalyses may adapt more advanced methods from other disciplines; a hypothesis revision can also make a previous analysis method ineligible.Source: §4.2.3 · pp. 11–12

4 FRAMEWORK FOR RESEARCH SYNTHESIS

  • An evidence evolution graph models how evidence about a phenomenon develops through related studies over time.The framework defines evidence and relationships between evidence pieces as the basis for describing synthesis and evolution.Source: §4 · p. 7, §4.3 · p. 12

4.3 Evolution

  • In the example graph, conflicting replications leave a revision to h2 as a dead end, while h3’s proposed confounder a becomes the most internally valid hypothesis.Future replications under h3 should collect data about a; a reanalysis may use a different method if the outcome’s distribution supports it.Source: §4.3 · p. 12, Table 1 · p. 12

5 APPLICATION

The framework is applied to test coverage and requirements quality to relate individual evidence and reveal synthesis problems. The examples trace how replications, revisions, and reanalyses change conclusions about those phenomena.

5.1 Test Coverage affecting Test Suite Effectiveness

  • Test-coverage studies produced conflicting findings when controlling for suite size, while a later revision proposed unobserved test quality as another explanation without empirical evidence for h4.Inozemtseva and Holmes found the association diminished after controlling for size, whereas Gopinath et al. found it persisted.Source: §5.1 · p. 14

5.2.1 Evolution as Published

  • The published sequence weakened e1’s strong effect to 34% in e2, replicated e1 with e3, then put the more-missing estimate at 45% in e4, still below half.The e3 replication used a within-subject experiment with 25 mostly industrial participants.Source: §5.2.1 · pp. 15–16

5.2.2 Evolution as supposed to publish

  • The pure Bayesian reanalysis largely agreed with the frequentist test, though it was more cautious, contrary to the claim that Bayesian statistics explained the difference.Separating reanalysis from revision makes the comparison between e1 and e2 interpretable.Source: §5.2.2 · p. 16
  • Adding missing entities as a mediator narrowed the confidence interval without changing the conclusion; adding skill widened it across zero, with no strong evidence h2 was more internally valid than h2a.The linear-model residuals were not iid, supporting greater conclusion validity for the linear mixed model.Source: §5.2.2 · pp. 16–17
  • Skill random effects explained substantially more outcome variance than fixed effects, motivating h2c, which retains skill but drops academic and industrial experience.The passage reports marginal R2 = 0.184 and conditional R2 = 0.561.Source: §5.2.2 · p. 17
  • 14% fewer, 47% equal, and 39% more missed associations were attributed to passive voice in the final e1.4 evidence.This combines the most internally valid hypothesis with the Bayesian analysis method of highest conclusion validity.Source: §5.2.2 · p. 18

6 DISCUSSION

The framework assembles method fragments into a coherent approach to improving the internal, external, and conclusion validity of variance theories.

6 DISCUSSION

  • The framework assembles method fragments into a coherent approach to improving the internal, external, and conclusion validity of variance theories.Source: §6 · p. 18

6.1 Implications

The framework aims to improve how evidence is created, synthesized, and applied. Its implications include guiding follow-up studies and making synthesized findings more useful to practitioners.

6.1 Implications

  • The framework provides terminology and a process for synthesizing evidence while valuing replications, revisions, and reanalyses.The authors argue this may encourage follow-up research rather than ending empirical investigations after a single study.Source: §6.1 · p. 18
  • The framework can help future studies gauge the knowledge frontier and choose variables and analysis methods with stronger validity.The hypothesis with highest internal validity can inform which variables to collect, while the analysis method with highest conclusion validity can guide analysis.Source: §6.1 · p. 18
  • Systematically synthesized findings may be more digestible and convincing to practitioners than isolated studies.The authors expect focused, integrated results with maximum internal, external, and conclusion validity to have a higher chance of practical application.Source: §6.1 · p. 18

6.2 Use Cases

The use cases progress from framing a new study against existing evidence to modeling a field’s evidence evolution and using it to guide follow-up study design.

6.2.1 Framing new Evidence

  • New evidence can be framed as a replication, revision, or reanalysis by changing one evidence component at a time.Agreeing replications improve external validity; revisions can improve internal validity through more precise or deconfounded ACE estimates; reanalyses can improve conclusion validity when they violate fewer statistical assumptions.Source: §6.2.1 · pp. 18–19

6.2.2 Systematic Literature Reviews

  • An evidence evolution graph can show which causal assumptions have the strongest support and which are dead ends, informing future research directions.Applying the framework across quantitative studies about one phenomenon makes the evolution of its variance theory explicit.Source: §6.2.2 · p. 19

6.2.3 Informing a follow-up Study

  • Researchers can use an evidence graph to select variables and analysis methods for a follow-up study.The test-coverage example points to collecting test-suite size and potentially quality aspiration; the requirements-quality example favors Bayesian analysis methods because they violate fewer statistical assumptions.Source: §6.2.3 · p. 19

6.3 Limitations

The framework has limits tied to the validity of its component methods, its focus on quantitative evidence and operationalizations, and its exclusion of primary-study quality appraisal.

6.3.1 Reliance on Fragments’ Validity

  • The framework’s validity depends on the validity of the method fragments from which it is assembled.The authors consider this threat constrained because prior research supports the validity of replications, meta-analysis, statistical causal inference, and reanalysis.Source: §6.3.1 · p. 20

6.3.2 Limitation to quantitative Evidence

  • Only quantitative evidence qualifies as an individual piece in the synthesis; qualitative evidence can trigger a hypothesis revision but is not synthesized.The framework’s variance-theory focus excludes qualitative evidence from synthesis because it lacks quantification.Source: §6.3.2 · p. 20

6.3.3 Construct Validity

  • The framework addresses effects between operationalizations, not concepts, so its resulting variance theories have limited applicability.The test-coverage example concerns operational test coverage rather than the broader concept of test adequacy criteria.Source: §6.3.3 · p. 20

6.3.4 Quality Appraisal

  • The framework relates and compares included evidence but does not address the non-trivial appraisal and selection of primary studies.Careful selection of which studies to include in a synthesis remains an ongoing challenge.Source: §6.3.4 · p. 20

6.4 Future Work and Roadmap

The roadmap envisions applying the framework to existing and new literature reviews, supporting ongoing synthesis, and making synthesized knowledge accessible to practitioners.

6.4 Future Work and Roadmap

  • Existing systematic literature reviews could be replicated with the framework when their evidence details are available, while new reviews are its expected main impact.The authors anticipate using evidence evolution graphs to describe research progress and inform study design and dissemination.Source: §6.4 · p. 20
  • Researchers are encouraged to organize evidence in a visual graph, with a possible online platform for submitting, integrating, and maintaining new evidence.The envisioned platform could automate synthesis for evidence sharing a hypothesis and analysis method, shifting synthesis from static and manual toward dynamic and continuous.Source: §6.4 · p. 20
  • Making recent synthesized variance theories available to practitioners could improve knowledge translation compared with presenting one-off studies.This is framed as a potential benefit of access to synthesized knowledge through such platforms.Source: §6.4 · p. 21

7 CONCLUSION

The framework relates evidence to assess validity and trace how research on a phenomenon evolves, revealing its current frontier.

7 CONCLUSION

  • Research synthesis can produce variance theories with greater validity than individual pieces of evidence.These theories support knowledge translation.Source: §7 · p. 21
  • The framework relates evidence about a phenomenon and supports comparing its internal, external, and conclusion validity.Source: §7 · p. 21
  • Relating multiple pieces of evidence reveals research evolution and indicates the phenomenon’s current frontier.The authors envision this orchestration supporting knowledge generation, synthesis, and application.Source: §7 · p. 21

FUNDING

The work acknowledges support from the KKS foundation and two projects at Blekinge Institute of Technology.

FUNDING

  • The work was supported by the KKS foundation through the S.E.R.T. Research Profile project and the GIST project at Blekinge Institute of Technology.Source: FUNDING · p. 21
Loading 2609.05983v1…