PaperlayerView exact revision on arXiv

Source-linked AI summary

Sampling in Software Engineering Research: A Critical Review and Guidelines

Sebastian Baltes, Paul Ralph

arXiv:2002.07764v6cs.SE

TL;DR

Software engineering research often lacks representative sampling, while sampling frames, randomness, and representativeness are frequently misunderstood. The paper reviews sampling in recent high-quality SE research, synthesizes a practical primer, and develops guidelines; it finds probability sampling is rare and identifies a generalizability crisis. It recommends pursuing more representative samples where generalizability is desired without broadly disparaging non-probability sampling, especially in qualitative research.

  • Problem

    Software engineering research has limited representative sampling, missing or biased sampling frames, and frequent misunderstandings of sampling, randomness, and representativeness.

  • Method

    The paper synthesizes sampling knowledge into an SE-specific primer, reviews sampling in 120 recent articles, and develops guidelines for conducting, reporting, and reviewing sampling.

  • Results

    Probability sampling was rare: only 17 articles employed it, and 11 of those sampling stages were subsamples of non-probability samples.

  • Takeaways & Limitations

    Researchers should work toward more representative samples where generalizability is desired while recognizing that non-probability sampling can be appropriate, particularly for qualitative research.

  • Takeaways & Limitations

    The findings may not generalize safely to substantially different venues, older research, or research conducted in future years.

Abstract

from arXiv · show

Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.

1 Introduction

Sampling is central to software engineering research, but the field faces missing sampling frames, poorly understood strategies, and widespread misunderstandings of representativeness. The paper responds with a primer, an investigation of sampling practice, and guidelines for improvement.

  • Motivation: Sampling matters in both quantitative and qualitative software engineering research, although its role differs across research traditions.For positivist studies it determines external validity; for interpretivist and constructivist studies, poor site or topic selection can hinder data collection and theory building.
  • Motivation: Many software engineering phenomena lack suitable sampling frames, while some studies misuse random sampling or misunderstand representativeness.These three observations motivate the paper’s critical review of sampling practice.
  • Paper purpose: The paper provides an SE-specific sampling primer, investigates sampling practice, and proposes guidelines for improving sampling.The goals cover sampling concerns and techniques, the field’s current state, and better research practice.
  • Paper purpose: The paper discusses recurring sampling errors without identifying individual articles because it aims to foster consensus rather than division.Examples include misusing “random” and overstating sample representativeness.

2 Sampling: A Primer

The primer defines sampling concepts and surveys probability and non-probability approaches relevant to software engineering research. It emphasizes that no sampling technique is universally best and that method choice depends on purpose and context.

  • Core concepts: Sampling selects a sample from a population, usually using an incomplete sampling frame; representativeness describes how closely sample properties resemble the population.These concepts provide the framework for evaluating sampling choices.
  • Core concepts: Different sampling approaches suit different purposes and circumstances, so there is no single best technique or hierarchy of techniques.The primer focuses on strategies most applicable to software engineering research.
  • Non-probability sampling: Non-probability sampling includes convenience, purposive, referral-chain, and respondent-driven sampling.These approaches do not employ randomness.
  • Non-probability sampling: Convenience sampling is fast, inexpensive, and requires no sampling frame, but it is controversial because it can threaten generalizability.Items are selected based on availability or expedience.
  • Non-probability sampling: Purposive sampling uses deliberate, non-random selection to apply expert judgment or ensure representativeness on a specific dimension, but remains subjective and opportunistic.It can select information-rich or diverse cases without requiring a complete sampling frame.
  • Non-probability sampling: Snowball sampling helps locate items absent from a sampling frame but can bias results toward highly connected people or artifacts.In software engineering it is also used to supplement literature-search keywords through backward and forward citation searching.

2.2 Probability sampling

Probability sampling uses randomness to give population items equal selection probabilities and can support statistical generalization, but software engineering often lacks comprehensive sampling frames. The primer distinguishes several probability and multistage strategies and their trade-offs.

  • Probability sampling: Probability sampling uses randomness, meaning each population item has an equal probability of selection; arbitrary recruitment or arrival-order assignment is not random.Practically, selection without a random number generator is probably not random.
  • Probability sampling: Probability sampling can support statistical generalization, but its major software engineering challenge is the absence of comprehensive sampling frames.Researchers otherwise select population subsets to estimate population parameters statistically.
  • Probability sampling: Simple random sampling selects items entirely by chance, giving each item an equal chance of inclusion.The example assigns developers numbers and uses a random number generator to select interviewees.
  • Probability sampling: Systematic random sampling selects every xth item after choosing a random starting point, but recurring patterns aligned with x can bias the sample.The example reads messages 47, 147, 247, and so on.
  • Probability sampling: Panel sampling studies the same sample at least twice and is probabilistic only when the panel is randomly selected.The example re-interviews the same developers one year after an intervention.
  • Multistage sampling: Multistage sampling intentionally combines strategies, including stratified or quota sampling and cluster sampling.Stratified random sampling randomly selects within purposively chosen strata, whereas cluster sampling draws items from selected groups.

2.4 Sampling in qualitative research

Qualitative research selects sites and data sources in ways that vary across research traditions, often using non-probability sampling. The paper argues that sampling need not imply statistical generalization and can support generalization from data to theory.

  • Qualitative sampling: Qualitative researchers select both research sites and data sources, but traditions such as grounded theory, phenomenology, and ethnography describe this selection differently.Some qualitative researchers use the term “sampling,” while others avoid it because of its statistical associations.
  • Qualitative sampling: Sampling does not imply statistical generalization; predominantly qualitative approaches typically use non-probability sampling to support non-statistical generalization from data to theory.Predominantly quantitative studies, especially questionnaire surveys, sometimes use probability sampling for statistical generalization.
  • Qualitative sampling: Selecting a site for an interesting property is purposive sampling, while interviewing whoever is available is convenience sampling.The paper treats selection of sites and data sources as a form of sampling.
  • Qualitative sampling: Grounded theory’s theoretical sampling selects items based on an emerging theory and directs what the researcher studies next.A developing category such as interpersonal conflict can guide subsequent data selection.

2.5 Sampling Frameworks and Algorithms

Software engineering studies often involve multistage sampling that does not fit a simple population–frame–sample model. Clear, replicable procedures and explicit discussion of potential bias matter more than forcing every study into that framework.

  • A sampling framework traditionally comprises a population, sampling frame, and sample, with each nested within the preceding level.
  • Multistage examples can involve successive restrictions by artifact type, project characteristics, language, and random selection, making the framework difficult to map.
  • Random selection after extensive filtering does not by itself justify claiming that the resulting sample represents the original population.
  • Researchers should identify the target population, describe each sampling step clearly, and explain how every step could bias generalization.
  • Scripts or precise manual algorithms should be archived or reported so others can reproduce the sample; qualitative researchers should instead explain their selection reasoning when it cannot be algorithmized.

2.6 Representativeness

Representativeness concerns how a sample’s relevant properties resemble those of a target population, not whether a particular sampling technique was used. Randomness can support representativeness, but neither guarantees it nor makes non-probability sampling inherently inferior.

  • Representativeness is a mutual property of a sample and target population, rather than a property of a sampling technique.
  • Representativeness is a primary criterion for positivist, falsificationist, and Bayesian research, but not for epistemologies that reject statistical generalization.
  • Representativeness is dimension-specific, so a sample may match a population on one parameter but not another.
  • Probability sampling does not guarantee representativeness, while non-random or non-probability samples can better match dimensions of interest.
  • No sample size can overcome bias in a sampling frame that excludes relevant parts of the target population.
  • Random selection is most reliable when the sample is large and the sampling frame is unbiased, yet it still produces representativeness only most of the time.
  • Researchers should not imply representativeness merely from professional participants, and may dismiss statistical generalization only if the manuscript avoids implying it elsewhere.

3 Method

The study uses a critical review to examine sampling in recent, influential software engineering research. It manually analyzes a stratified random sample of papers and codes sampling, methods, units of observation, and authors’ justifications.

  • The critical review evaluates methodological issues in a sample of primary studies rather than aggregating all relevant studies or estimating causal relationships.
  • The review investigates common sampling approaches, authors’ justifications, empirical methodologies, and units of observation, with sampling as its main focus.
  • The sampling frame covers full technical-track papers published from 2014 through 2019 in ICSE, FSE, TSE, or TOSEM.
  • A Python script identified 1,830 eligible papers, from which five papers per outlet–year stratum were selected using a true-random number generator.
  • The authors acknowledge that their purposive outlet selection limits representativeness because the venues are highly competitive and English-language, with no objective basis for the precise outlet and time choices.
  • The review analyzed 120 articles through manual coding of sampling descriptions, stages, origins, populations, frames, sample sizes, methods, and units of observation.

4 Results and Discussion

Sampling in the reviewed software engineering literature was dominated by purposive and convenience approaches, while probability sampling was uncommon and often applied within previously non-random samples. Sample origins and quality justifications were also frequently limited or unclear.

  • Sample sizes ranged from 1 to 819,000, with a median of 22; reported sampling-frame sizes ranged from 3 to 2,000,000, with a median of 395.
  • 4.1 RQ1: Sampling techniques used: 149 sampling stages used purposive sampling and 23 used convenience sampling, while only 17 used probability sampling.
  • 4.1 RQ1: Sampling techniques used: 13 stages used simple random sampling and 4 used stratified random sampling; 14 analyzed their entire sampling frame.
  • 4.1 RQ1: Sampling techniques used: Eight of 13 simple-random stages and three of 4 stratified-random stages sampled from previously non-random samples.
  • 4.1 RQ1: Sampling techniques used: 62 sampling stages used existing datasets, 31 sampled online resources, and 27 of those 31 online-resource stages used purposive sampling.
  • Of 120 articles, 86 justified some sample-quality criteria, most commonly by describing artifacts as real or real-world.
  • The reviewed studies were predominantly quantitative, with 126 quantitative versus 16 qualitative studies; 98 quantitative studies evaluated software tools, models, or machine-learning approaches.
  • 4.4 RQ4: Units of observation: Most studies examined code artifacts, including GitHub projects, commits, and packages, while other observations included bug reports, test logs, students, developers, and platform workers.

5 Discussion

The review finds that probability sampling is rare in software engineering research, while sampling practices and terminology are often misunderstood. It also identifies sampling-frame limitations and recommends clearer reporting, context-appropriate evaluation, and improved sampling practices.

  • Purposive and convenience sampling were the most common strategies, while only seventeen articles used probability sampling.Eleven of the seventeen corresponding probability-sampling stages were sub-samples of non-probability samples.
  • Probability sampling is difficult because suitable sampling frames rarely exist for software engineering populations.Researchers often study GitHub projects, Microsoft developers, or Huawei test logs because comprehensive lists of relevant populations are unavailable.
  • Sampling terminology and procedures are frequently misunderstood, including misuse of “random” and unsupported claims of representativeness or generalization.Reported problems include treating arbitrary selection as random, assuming random samples are representative regardless of size or frame bias, and generalizing without a representative sample.
  • Non-probability sampling should not be dismissed when representativeness is not the study goal, especially in qualitative research.The review notes that qualitative studies may pursue analytical rather than statistical generalization and that disparaging non-probability sampling can be misguided.
  • The review’s conclusions primarily generalize to recent, high-quality research in four venues, not necessarily to other venues or time periods.The authors also note that some extracted methodology and sampling classifications required interpretation, despite auditing ambiguous cases.

6 Recommendations

The paper recommends aligning sampling practices with a study’s goals and philosophical position, while improving transparency, bias assessment, and sampling frames. It also urges researchers and reviewers to avoid overstating representativeness and to recognize that generalization is not the aim of every study.

  • The proposed strategy addresses a generalizability crisis, but the paper also identifies an over-generalization crisis in which findings and samples are overstated.
  • Sampling strategies should match the study’s philosophical position, purpose, context, constraints, and intended knowledge claims.
  • Researchers should not call ad hoc or biased-frame sampling random, or treat small, real, professional, or deliberately selected samples as automatically representative.
  • Software engineering research should identify its target population and describe a replicable multistage sampling algorithm, including how each step could introduce bias.
  • Studies seeking generalization should explicitly argue for representativeness and explain efforts to mitigate sampling bias, especially in questionnaires, repository mining, and exploratory data science.
  • Researchers can assess stability through bootstrapping, replicate analyses across sampling frames or repositories, and use multiple sources to reduce review bias.
  • Long-term improvements require better sampling frames and curated corpora, while acknowledging that corpora may remain biased but can make those biases better understood.
  • Reviewers should assess sample adequacy, strategy consistency, stated limitations, and whether the sampling approach supports the paper’s claims.

7 Related Work

Related work has examined sampling prevalence, barriers, evaluation concepts, and context-specific recommendations. This paper extends that work by integrating representativeness, sampling-frame bias, mitigation techniques, and reviewer guidance.

  • Earlier studies found probability sampling uncommon in software engineering research, with purposive and convenience sampling frequently used in GitHub studies.
  • Prior work discusses barriers to random questionnaire sampling and the need for population parameters when assessing representativeness.
  • The sample study concept limits external-validity criticism to studies that aim for statistical generalization.
  • Existing representativeness criteria include sample coverage and comparison with known target-population parameters, though those parameters may be unavailable.
  • The paper argues that some prior sampling-frame recommendations do not address representativeness or biased sampling frames directly.
  • Its extension combines a six-pronged improvement strategy with bias-mitigation approaches and specific guidelines for reviewers.

8 Conclusion

The paper contributes a sampling primer, an empirical review, representativeness analysis, and guidelines for conducting, reporting, and reviewing sampling in software engineering. It concludes that better sampling is needed, while acknowledging important scope and evidential limitations.

  • The paper contributes an SE-grounded introduction to sampling that is broader than previous SE discussions and more tailored than treatments in reference disciplines.
  • Its review of recent research shows that probability sampling is rare and often relies on unknown or nonrepresentative sampling frames.
  • Randomness does not equal or guarantee representativeness, and a nonprobability sample can sometimes be more representative than a probability sample.
  • The paper argues that insufficient representative sampling undermines software engineering research, while rejecting studies solely for unavoidable nonrepresentativeness can be capricious.
  • The authors recommend more honest reporting, bias mitigation where generalization is desired, and better sampling frames without disparaging appropriate nonprobability sampling.
  • The empirical review covers four top venues over six years, so its sample is unlikely to represent the broader software engineering field.
  • Some guidelines are not directly supported by empirical evidence because they are meta-science recommendations that are difficult to test experimentally.
  • The authors hope the primer, empirical results, and recommendations raise awareness and help improve sampling in software engineering research.

9 Data Availability

Supplementary materials are archived on Zenodo and include the article list, extracted data, analyses, and scripts used in the study.

  • The supplementary materials include an Excel spreadsheet containing the complete article list, extracted data, and analyses.
  • The archive also contains scripts used to retrieve sampling-frame and sample information.
  • The project received support from the National Sciences and Engineering Research Council of Canada.
Loading 2002.07764v6…