PaperlayerView exact revision on arXiv

Source-linked AI summary

Empirical Standards for Software Engineering Research

Paul Ralph, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, Robert Feldt, Antonio Filieri, Breno Bernard Nicolau de França, Carlo Alberto Furia, Greg Gay, Nicolas Gold, Daniel Graziotin, Pinjia He, Rashina Hoda, Natalia Juristo, Barbara Kitchenham, Valentina Lenarduzzi, Jorge Martínez, Jorge Melegati, Daniel Mendez, Tim Menzies, Jefferson Molleri, Dietmar Pfahl, Romain Robbes, Daniel Russo, Nyyti Saarimäki, Federica Sarro, Davide Taibi, Janet Siegmund, Diomidis Spinellis, Miroslaw Staron, Klaas Stol, Margaret-Anne Storey, Davide Taibi, Damian Tamburri, Marco Torchiano, Christoph Treude, Burak Turhan, Xiaofeng Wang, Sira Vegas

arXiv:2010.03525v2cs.SEcs.GL

TL;DR

Peer review is unreliable, and opaque or inconsistent criteria leave authors and reviewers with different expectations. The paper presents community-generated empirical standards and a structured-review approach; it expects these to make review more effective, reliable, transparent, and fair. It also warns that partial adoption can leave some reviewers disregarding the shared expectations.

  • Problem

    Peer review is described as unreliable and ineffective, while unclear or incompatible evaluation criteria leave authors uncertain about how their work will be judged.

    Source: Introduction · p. 3

  • Method

    A diverse team of domain experts generated empirical standards, which venues can use in a structured review process.

    Source: Design Process · p. 6, Full Adoption (Structured Review) · p. 9

  • Results

    The standards are intended to transform free-form review into structured evaluation and make peer review more effective, reliable, transparent, and fair.

    Source: Conclusion · p. 15

  • Takeaways & Limitations

    The standards are intended to transfer the power to set expectations from individual reviewers to the scientific community as a whole.

    Source: Conclusion · p. 16

    With partial adoption, some reviewers may disregard the shared expectations, potentially making the process seem even more unjust to authors who comply.

    Source: Partial Adoption · p. 9

Abstract

from arXiv · show

Empirical Standards are natural-language models of a scientific community's expectations for a specific kind of study (e.g. a questionnaire survey). The ACM SIGSOFT Paper and Peer Review Quality Initiative generated empirical standards for research methods commonly used in software engineering. These living documents, which should be continuously revised to reflect evolving consensus around research best practices, will improve research quality and make peer review more effective, reliable, transparent and fair.

MARCH 4

Peer review is unreliable and often leaves its criteria opaque, creating disconnects between authors, reviewers, and community norms. The paper proposes brief, method-specific empirical standards to make expectations shared and review more structured while preserving justified deviations.

Introduction

  • Reviewers often lack method-appropriate criteria, and opaque or incompatible standards leave authors unsure how manuscripts will be evaluated.This disconnect is presented as a source of unpredictable and frustrating peer review.Source: Introduction · p. 3

Empirical Standards

  • Empirical standards specify expected attributes for particular methods rather than relying on vague criteria such as “soundness.”Examples include random assignment for experiments and a clear chain of evidence from interviewee quotations to proposed concepts for qualitative surveys.Source: Empirical Standards · pp. 3–4
  • Standards allow justified deviations and do not replace expert judgment or micromanage style.They focus on methodological substance and include examples of acceptable deviations.Source: Empirical Standards · p. 4
  • Using the same rubric makes evaluation criteria visible to authors and reviewers and lets the scientific community set expectations collectively.This is intended to make peer review fairer and more transparent.Source: Empirical Standards · p. 4

Components of the Standards

Two Crucial Rules for Reviewers

The paper asks reviewers to apply standards flexibly and avoid introducing criteria beyond them. It describes standards drafted and revised through a community process involving subject-matter experts.

Two Crucial Rules for Reviewers

  • Reviewers should allow justified deviations and easy-to-fix mistakes rather than using standards as a “gotcha” test.The paper proposes asking whether a deviation makes sense in context and whether a problem would be easy to fix in the camera-ready copy.Source: Two Crucial Rules for Reviewers · p. 5
  • Reviewers should not impose criteria absent from the standards, but may submit a standards issue when a critical flaw is not covered.This rule is intended to prevent reviewers from applying their own non-standard criteria.Source: Two Crucial Rules for Reviewers · pp. 5–6

Design Process

  • Subject-matter experts drafted standards from published guidance where possible; drafts were edited, circulated for feedback, and revised.The initiative reported more than 50 volunteers, and the resulting standards were generated by a large, diverse team of domain experts.Source: Design Process · p. 6

Evolution and Governance

The standards are intended to evolve through community feedback, with expert maintainers responsible for reviewing proposed changes. Venues can adopt them accidentally, partially, or fully, but informal use may create conflicts when expectations are undisclosed.

Evolution and Governance

  • The standards are living documents that researchers can refine through reported issues and change requests as community views evolve.Each standard is expected to have expert maintainers who review issues and change requests.Source: Evolution and Governance · pp. 6–7

Adoption Models

  • Venues may adopt standards accidentally, partially, or fully, and fuller adoption is expected to accelerate improvements in review and paper quality.Reviewers may use standards independently because they reduce the effort of generating ad hoc criteria, while researchers may use them to design studies and manuscripts.Source: Adoption Models · p. 7, Accidental Adoption · p. 7

Accidental Adoption

  • Accidental adoption may cause conflict when authors are judged against standards they were not told about and reviewers lack support applying them.The paper identifies formal venue adoption as a way to avoid these conflicts.Source: Accidental Adoption · p. 8

Partial Adoption

The proposal ranges from encouraging venues to adopt empirical standards to replacing essay-style reviews with structured, standards-based decisions. The structured approach makes criteria visible to authors and reviewers while reserving follow-up questions and revisions for identified gaps.

Partial Adoption

  • Venues can tentatively adopt the standards by adding them to guidance, collecting authors’ method selections, training reviewers, and revising decision forms.The proposed decision forms offer three choices and avoid asking reviewers directly for accept or reject recommendations.Source: Partial Adoption · p. 8
  • Partial adoption is easy to implement, but reviewers who disregard the standards can make compliance seem more unjust, prompting consideration of structured review.The passage names EasyChair and HotCRP as examples of editorial systems that need no major changes for this approach.Source: Partial Adoption · p. 9

Full Adoption (Structured Review)

  • Structured forms replace rambling review essays with brief criticism or revision text, while optional free-form comments do not affect decisions or revision evaluation.The general process also includes author checklists, initial checks of method selection, and reviewer forms generated from the standards.Source: Full Adoption (Structured Review) · pp. 9–10
  • Dynamic forms reveal diagnostic follow-ups when an essential attribute is missing, allowing justified deviations and specific shortcomings to receive different treatment.For limitations sections, the proposed pathways distinguish absent explanations, adequate sections, and inadequate sections that warrant revision.Source: Full Adoption (Structured Review) · p. 10
  • Full adoption is expected to increase acceptance and reduce revisions by aligning authors’ and reviewers’ expectations.Revision lists address missing essential attributes, excluding ad hoc requirements; one editor or lead reviewer can check completion.Source: Full Adoption (Structured Review) · p. 11
  • Reviewers answer mostly yes-or-no questions against a standard, while venue decision rules determine acceptance, revision, or rejection rather than reviewer recommendations.The system compiles reviewers’ answers and sends a review summary, revision to-do list, or rejection reasons.Source: Full Adoption (Structured Review) · pp. 9–11

Cost and Benefits of Empirical Standards

  • Empirical standards make evaluation criteria more transparent by giving authors and reviewers a shared account of expected study practices.Vague criteria leave reviewers to invent rubrics, making the real evaluation criteria secret from authors.Source: Cost and Benefits of Empirical Standards · p. 11

Benefits for Editors / Program Chairs

Structured standards are presented as a way to make review clearer and more consistent for editors and program chairs. For reviewers, they reduce the work of setting expectations and help focus review and revision.

Benefits for Editors / Program Chairs

  • Standards promise editors and program chairs more consistent, thorough, readable reviews and clearer control over paper expectations.The listed benefits also include easier analysis of reviewer reliability and recruitment, fewer low-quality submissions, and faster publication.Source: Benefits for Editors / Program Chairs · p. 11

Benefits for Reviewers

  • Reviewers avoid constructing expectations, writing essays, and copyediting papers.Work that can be done by one person, such as checking keywords, is assigned to a non-reviewer.Source: Benefits for Reviewers · pp. 11–12
  • Reviewers can focus discussion on disagreements, with a third reviewer needed only when the first two do not agree sufficiently.Known expectations are also presented as reducing obviously deficient submissions and repeat reviews.Source: Benefits for Reviewers · p. 12
  • Shared standards clarify venue expectations and let reviewers assess compliance rather than debate how a study should have been conducted in principle.They are also said to reduce manuscript movement between venues when many venues use the same standards and rejection rates are lower.Source: Benefits for Reviewers · p. 12

Benefits for Researchers

The standards are presented as making review more predictable and actionable for researchers while also serving as checklists for study design and manuscript preparation. The section also describes benefits for research training and literature reviews.

Benefits for Researchers

  • Standards make venue expectations transparent and reviews more predictable, actionable, and consistent, with fewer mixed messages.The passage also lists higher acceptance rates, fewer major revisions, and faster publication as benefits attributed to expectations being known in advance.Source: Benefits for Researchers · p. 12
  • Researchers can use standards as checklists for designing studies and polishing manuscripts, while reviewers cannot micromanage style or impose requirements inconsistent with established norms.The passage also says standards mitigate bias against qualitative, exploratory, and industry-focused research and stop cross-paradigmatic criticism.Source: Benefits for Researchers · p. 12
  • Standards are described as useful for graduate-student training and for assessing primary studies in systematic literature reviews.Source: Benefits for Researchers · p. 12

Benefits to Society

The section attributes broader research and resource benefits to empirical standards, including better studies, reduced waste, and less destructive reviewing.

Benefits to Society

  • Standards are presented as supporting better, more rigorous, and more ambitious studies.Source: Benefits to Society · p. 12
  • The passage says standards reduce wasted resources from forgotten or ignored critical practices and from bad reviewing and re-reviewing.Source: Benefits to Society · p. 12
  • The passage links standards to less HARKing and publication bias because statistical significance is not a criterion, and to fewer scientists and graduate students burning out from destructive reviewing.Source: Benefits to Society · p. 12

Eliminating Frustration

The section argues that empirical standards address frustrations caused by mismatched expectations and reviews that focus on style or omit essential practices. It presents structured review and community-developed standards as ways to focus evaluation on substance and shared norms.

Eliminating Frustration

  • Empirical standards address frustration when authors and reviewers disagree about appropriate research practices and when essential practices are omitted or glossed over.The passage says structured review ensures detailed technical reviews because the process does not allow another alternative.Source: Eliminating Frustration · pp. 12–13
  • Structured review with empirical standards addresses style-focused reviews by directing attention to methodological substance.The section also says separating standard creation from review of a specific paper facilitates broader debate about how studies should be performed and presented.Source: Eliminating Frustration · p. 13
  • The project’s main cost is building and maintaining standards and the system that generates review forms.The passage notes that volunteers have contributed time and expertise, while a prototype structured-review system is under development and more help is needed.Source: Costs · p. 13
  • Full adoption requires more fundamental review-process changes and collaboration with management systems such as Easychair and HotCRP.Source: Costs · p. 13

Challenges with Structured Review

Structured review should guide expert judgment without becoming inflexible box-ticking: reviewers can assess justified deviations, while concerns about the standards themselves should prompt revision rather than ad hoc criteria. The authors describe steps to broaden participation and reduce bias, while acknowledging those steps may not eliminate it.

Challenges with Structured Review

  • Empirical standards direct reviewers but do not replace human judgment about what is reasonable and convincing.Reasonable deviations remain acceptable.Source: Challenges with Structured Review · p. 13
  • Follow-up questions about whether deviations are justified are intended to balance structured review with flexibility.Poor implementation could instead produce superficial, inflexible box-ticking.Source: Challenges with Structured Review · p. 13
  • Reviewers should report problems not covered by the standards and accept the paper meanwhile, avoiding unfair rejection and arbitrary reviewer-made criteria.The authors argue that a flawed standard can be corrected once identified, whereas allowing unrestricted overrides invites abuse.Source: Challenges with Structured Review · p. 14

Justice, Equality, Diversity and Inclusion

  • To mitigate bias, the initiative used open recruitment, sought underrepresented participants, involved more than 40 experts, and made feedback and changes transparent.It also aimed for method-specific standards and a general standard that is philosophy- and method-agnostic.Source: Justice, Equality, Diversity and… · p. 14
  • The authors acknowledge that their steps are likely insufficient to eliminate bias and welcome further suggestions.Source: Justice, Equality, Diversity and… · p. 14

The Scientific Basis of Empirical Standards

The paper argues that peer review’s low agreement reflects insufficient structure, and proposes community standards to guide evaluations while retaining expert judgment. It presents structured review as a way to make peer review more consistent and to shift expectations from individual reviewers to the community.

The Scientific Basis of Empirical Standards

  • An example uses multiple experts to rate documentation quality and estimates agreement with Cohen’s Kappa or Krippendorff’s Alpha [7].The example illustrates how agreement among raters informs confidence in expert ratings.Source: The Scientific Basis of Empirical… · pp. 14–15
  • Low inter-rater agreement among peer reviewers is presented as evidence that review is capricious and unscientific [8].The paper argues that task structure can improve consistency, drawing on prior work on decision rules [9].Source: The Scientific Basis of Empirical… · p. 15
  • Empirical standards add structure through specific methodological questions and decision rules, with the goal of making peer review more reliable.The paper compares this approach to coding schemes and decision rules that improve the reliability of expert-judgment studies.Source: The Scientific Basis of Empirical… · p. 15
  • The authors cite checklist research in reporting and healthcare as evidence that checklists can improve consistency and performance [10, 11].Source: The Scientific Basis of Empirical… · p. 15

Conclusion

  • The conclusion presents empirical standards as a route from free-form essays to structured evaluation, intended to improve review effectiveness, reliability, transparency, and fairness.The authors describe this as radical rather than incremental change.Source: Conclusion · pp. 15–16
  • The standards are intended to reduce frustration from mismatched author and reviewer expectations and transfer expectation-setting from individual reviewers to the community.Source: Conclusion · p. 16

Disclaimers

The standards are intended to apply without privileging a particular methodology, epistemological position, analysis approach, or data type.

Disclaimers

  • Each study should be reviewed on its own terms, without privileging a methodology, epistemological position, analysis approach, or data type.Source: Disclaimers · p. 16

Glossary

The glossary defines methodological, research-quality, and data-protection terms, while the accompanying credits identify the editor and contributors.

Glossary

  • Anonymization removes or transforms identifying elements to prevent re-identification, whereas pseudonymization retains a separate map that can reconnect records to individuals.De-identification is often used for either process, though the glossary recommends specifying which applies.Source: Glossary · p. 17
  • Qualitative-research terms include chain-of-evidence, mapping raw data to theoretical concepts, and theoretical sampling, selecting data based on emerging theory.Source: Glossary · pp. 17–18
  • External validity concerns generalization to a target population, while construct validity asks whether measures represent the research aims.Source: Glossary · p. 17
  • Reliability concerns consistent measurement across time or raters, while objectivity concerns freedom from researcher bias through standardized procedures.Source: Glossary · p. 17
  • Replicability concerns repeating data collection and analysis on new data, whereas reproducibility means recreating results from the original data and source code.The glossary notes that qualitative research is typically neither replicable nor reproducible.Source: Glossary · p. 18

Content Contributors

Related papers

Loading 2010.03525v2…