Replication Studies: Not Just a Copy

1 INTRODUCTION

• The field lacks an easily understood account of replication designs, motivations, and findings, amid inconsistent terminology and often implicit links to earlier work.Replication matters for assessing whether findings remain reliable, where they generalise, and under what conditions they change.Source: §1 · p. 1
• A single replication label obscures consequential design choices because studies can preserve some components while adapting or adding others.Studies with the same label may change different parts of an experiment, while differently labelled studies may remain comparable in practical respects.Source: §1 · p. 1
• REPVIS1 had not been systematically applied to a broader corpus, leaving its ability to represent practical design diversity untested.The paper frames three questions around representing replication designs, characterising existing practice, and informing future planning and reporting.Source: §1 · p. 2
• The authors refined REPVIS1 using a development corpus, evaluated REPVIS2 on a separate corpus, and characterised 86 studies from 51 papers.The paper contributes a validated design space, a corpus characterisation, and corpus-grounded implications for future research.Source: §1 · p. 2

2.1 Replication in Visualisation and Related Fields

• Replication can examine research validity and revisit findings across methods, analyses, populations, and settings, but its meanings vary across research traditions.Visualisation examples include crowdsourcing evaluation, reanalysis, and evaluating neural networks on tasks previously studied with humans.Source: §2.1 · p. 2

2.2 Describing and Designing Replication

• Broad labels and strategies describe replication aims, while reviews and tools address practice or protocol comparison; REPVIS2 focuses on consistently recording design correspondence to a reference.Earlier work includes a visual tool for comparing study protocols and reviews of replication practice in HCI, visualisation, and AR/VR.Source: §2.2 · p. 2

2.3 Developing Design Spaces

• Multidimensional design spaces organise comparable aspects of visualisation tasks and designs, connecting descriptions of existing work with decisions about future studies.Published-literature reviews have also produced structured evaluation guidance that subsequent reviews extended.Source: §2.3 · p. 2

3 RESEARCH METHODOLOGY

• The workflow moves from developing and freezing REPVIS2 to validation on a separate corpus, then characterisation of the combined studies.The design space is descriptive rather than prescriptive: it represents possible correspondences without ranking designs as better or more valid.Source: §3 · p. 2, §3.1 · p. 3

3.1 Rationale

• The research question establishes the primary replication relationship, while dimensions separately describe how the replication design corresponds to, differs from, or extends its reference.The aim is to represent the information needed for that relationship, not every reported study detail.Source: §3.1 · p. 3

3.2 Phase I: Development Corpus and REPVIS2 Development

• The development corpus contained 37 papers after scope screening, and coding disagreements and difficult cases informed a candidate REPVIS2 frozen before validation.The collection began with 24 papers and expanded to 49 through backward and forward snowballing; 12 were excluded by consensus.Source: §3.2 · p. 3

3.3 Phase II: Validation Corpus and Validation Procedure

• Keyword searches of titles and abstracts identified 14 new validation papers containing 21 replication studies, while an audit showed many missed papers lacked replication terms in searchable fields.The search covered VisPubs publications across IEEE VIS, EuroVis, and visualisation-related CHI research.Source: §3.3 · p. 3

3.4 Phase III: Corpus Coding and Characterisation

• The combined corpus comprised 51 papers and 86 studies coded through paired reading of each replication report and its reference report.One primary reference was selected per study when several predecessors were involved, and all authors reviewed the codes until consensus.Source: §3.4 · p. 3

4.1 Evaluation Criteria

• Evaluation assessed whether each replication–reference pair was representable, whether existing levels classified observed relationships, and whether the structure remained stable.The development cases informed refinement, while the validation corpus tested the frozen candidate on cases not used in development.Source: §4.1 · pp. 3–4

4.2 Refining the Dimensions

• REPVIS2 replaces the broad EXPERIMENT dimension with STIMULI, TASK, PROCEDURE, INTERFACE, and ENVIRONMENT, alongside DATA, PARTICIPANT, and ANALYSIS.Separating components makes mixed relationships visible, such as retaining a task while changing other experiment components.Source: §4.2 · p. 4

4.3 Selecting the Comparison Scale

• The three levels distinguish identical, similar, and different relationships according to whether direct or meaningful comparability is preserved.A binary same/different scale could not distinguish meaningful comparison from changes beyond comparability; validation required no additional level.Source: §4.3 · p. 4

4.4 The Validated REPVIS2 Design Space

• Each profile describes one replication–reference pair, with eight practical dimensions coded independently and research-question layers separating the primary relationship from additional questions.RESEARCH TEAM and RESEARCH QUESTION provide context rather than adding practical implementation dimensions.Source: §4.2 · p. 4, §4.4 · p. 5
• Fully instantiated designs yield 38 possible profiles, although some combinations may be impossible or inappropriate in a study context.Design changes record modifications within the primary replication relationship; design additions record components beyond it, including those pursuing additional questions.Source: §4.4 · p. 5
• All 21 validation studies were represented without adding a dimension, comparison level, or structural change, supporting coverage and structural sufficiency for that corpus.This does not establish that future cases could never require refinement.Source: §4.4 · p. 5

5 CHARACTERIZING REPLICATION PRACTICES

• REPVIS2 characterizes each replication against its reference across eight independently coded dimensions, and the resulting profile can guide planning of retained and varied choices.The example profile retained STIMULI, TASK, PROCEDURE, INTERFACE, and ANALYSIS while changing ENVIRONMENT and PARTICIPANT; components beyond the replication core are recorded as additions.Source: §5 · p. 5

5.1 Corpus Overview

• The 51 papers appeared between 2007 and 2026; the median gap between replication and reference studies was 7.5 years, with a range of zero to 41 years.The corpus included 86 studies referring to 48 primary papers; Cleveland and McGill’s 1984 study was the reference for 15 studies across eight papers.Source: §5.1 · p. 5, Figure 4 · p. 5

5.2 Motivations, Reporting, and Outcomes

• The most common motivations were explaining effects or conflicting findings (21 studies) and testing generalisability across designs or settings (20); 41 results were consistent, 30 partly consistent, and 15 not consistent.Reported outcomes varied within motivations, including among generalisability tests across designs or settings: 13 consistent, four partly consistent, and three not consistent.Source: §5.2 · pp. 5–6, Figure 5 · p. 6
• Consistent results accounted for 43.3% of independent-team studies and 57.7% of overlapping-team studies; these descriptive differences do not support causal interpretation.The comparison also grouped results by explicit replication claim; absence of an explicit claim did not exclude a study from the replication scope.Source: §5.2 · p. 6, Figure 6 · p. 7

5.3 Design Space Profiles

• TASK was identical in 50 studies, while ENVIRONMENT and PARTICIPANT were most often different (47 and 44); five other dimensions were most often similar.STIMULI was similar in 75 studies, PROCEDURE in 71, INTERFACE in 49, DATA in 82, and ANALYSIS in 69; the studies occupied 44 distinct profiles.Source: §5.3 · p. 7, Figure 7 · p. 7
• The most common departure combination covered every non-DATA dimension except TASK (26 studies, 30.2%); three leading combinations covered 50 studies (58.1%).These combinations had mixed reported outcomes, and the summary records where departures occurred, not their severity.Source: §5.3 · p. 7

5.4 Recurring Design Choices

• Collecting new observations (84 studies), changing the interface (72), and adjusting administration (70) were the most frequent design changes.Five studies reused existing evidence: two without new observations and three alongside new observations; reuse did not necessarily mean identical DATA.Source: §5.4 · pp. 7–8, Table 2 · p. 7
• Additions occurred in 63 studies (73.3%), most often additional performance evidence (29), participant measures (24), qualitative accounts or recordings (23), and additional outcomes or comparisons (23).Additions supplied evidence about performance, participants, and context alongside the primary replication.Source: §5.4 · p. 8, Table 3 · p. 8

6.1 RQ1: Representing Replication Designs

• The 21 validation studies required no additional dimension, comparison level, or structural change, supporting REPVIS2’s coverage of these cases.Its eight dimensions and comparison levels make mixed relationships visible, while the two research-question layers distinguish changes within the replication core from additional inquiry.Source: §6.1 · p. 8
• REPVIS2 distinguishes design profiles that broad labels cannot specify and records identical, similar, or different relationships across eight dimensions.“Similar” captures change that preserves a meaningful basis for comparison; REPVIS2 describes design choices but is neither a substitute for detailed accounts nor a checklist guaranteeing a sound study.Source: §6.1 · p. 8

6.2 RQ2: Patterns in Replication Practice

• Across 86 studies, task was often retained while procedure, interface, environment, participant population, evidence, or analysis changed; motivations and reported outcomes were mixed.Studies occupied 44 profiles, and recurring motivations included testing generalisability and robustness, evaluating methods, and examining effects or conflicting findings; additions supplied further evidence.Source: §6.2 · p. 8
• A small set of graphical-perception and perceptual-ranking studies recurred as references, while most primary references appeared in only one replication paper.The corpus therefore includes both repeated investigations of established findings and more isolated returns to earlier work.Source: §6.2 · p. 9

6.3 RQ3: Implications for Future Planning and Reporting

• Researchers can start from a reference study and claim, specify correspondences to preserve or vary, and record the rationale for changes and additions before data collection.Because several dimensions often change together, the corpus cannot isolate individual design effects; low frequency alone also does not establish a useful research gap.Source: §6.3 · p. 9

6.4 Future Work

• REPVIS2 can structure clearer reporting, but cannot recover unreported details, and its transfer beyond visualisation still requires evaluation.The corpus is constructed rather than exhaustive; screening, reference selection, and coding involved judgement, and no formal inter-rater reliability statistic was calculated. The analysis is descriptive, not causal, and prospective planning has not been evaluated in practice.Source: §6.3 · p. 9, §6.4 · p. 9

• REPVIS2 represents replication through eight practical dimensions and three comparison levels, alongside research-question and research-team context.Application to a separate validation corpus supports its coverage and structural sufficiency.Source: §7 · p. 9
• The 86 studies occupied 44 profiles, often retaining a comparable task while adapting study conditions, evidence, or analysis; reported outcomes remained mixed.Many studies also used replication as a basis for further measures, conditions, comparisons, and analyses; these patterns do not establish a preferred design or a simple design–result relationship.Source: §7 · p. 9
• REPVIS2 supports characterising and reporting existing studies and planning future replications by making their relationship to prior work explicit.Source: §7 · p. 9