Source-linked AI summary
Breaking the Alphabet: Rethinking File Ordering in Code Review
Md Shamimur Rahman, Zadia Codabux, Chanchal K. Roy
TL;DR
The paper addresses limited evidence about how alphabetical file ordering affects cognitive load and perceived thoroughness in multi-file code reviews. It surveys professional developers across open-source projects using mixed quantitative and qualitative methods, finding that alphabetical ordering often conflicts with reviewers’ cognitive needs and prompting expectations for more adaptable tooling.
Problem
The paper asks how changed-file ordering influences review behavior, comprehension, cognitive effort, and perceived effectiveness when tools default to alphabetical ordering.
Method
The study conducts a mixed-methods survey of 1,355 developers associated with 182 open-source projects, analyzing closed- and open-ended responses.
Results
The study finds widespread dissatisfaction with alphabetical ordering because it fragments context, increases navigation effort, and undermines confidence in defect detection.
Takeaways & Limitations
The findings support reviewer-centric tools that align file organization with human cognition and provide semantically or logically grouped sequences.
Takeaways & Limitations
The survey relies on self-reported, cross-sectional data and may be affected by recall, social-desirability, and selection biases.
Abstract
from arXiv · showhide
Effective code review is central to maintaining software quality, yet there is limited research about how the ordering of changed files in Pull Requests (PRs) influences review effectiveness. Most popular code review tools default to alphabetical ordering, favoring predictability over contextual relevance. While prior studies examined how file position shapes reviewer attention, it remains unclear how such ordering influences cognitive load and perceived review thoroughness. This study presents the first large-scale survey of 1,355 professional developers across 182 widely used open-source projects to investigate how file ordering impacts review behavior, comprehension, and perceived effectiveness. Our mixed-methods analysis reveals that only 10.2% of reviewers consider alphabetical ordering optimal, underscoring a cognitive misalignment in their interpretation of code changes. Although some developers appreciate its predictability, more than half (57.6%) report that it increases context switching, disrupts logical reasoning, and contributes to review fatigue, and 63.9% expressed concern that the default ordering may cause them to miss bugs. We further identify key challenges in multi-file reviews and elicit developers' expectations for improved tooling, including dependency-aware grouping and customizable file ordering (requested by 66% of reviewers). These findings highlight the need for reviewer-centric interface designs that better align tool behavior with human cognition.
1 Introduction
This study examines whether default alphabetical file ordering aligns with reviewers’ cognitive strategies and affects review effectiveness in multi-file PRs. It uses a large-scale survey to investigate navigation behavior, perceived cognitive burden, and usability challenges.
- Motivation: Prior research shows that file position affects reviewer attention, with earlier files receiving more attention and comments.Controlled evidence also found reviewers were 64% less likely to detect defects in last-positioned files, although that setting did not reflect real-world review complexity.
- Motivation: Existing research has proposed ordering files by change size, salient classes, tests, or semantic links to the PR description, but effectiveness varies across contexts and reviewer preferences.The central concern is whether file ordering preserves review quality and prevents overlooked bugs.
- Research gap: Alphabetical ordering may increase cognitive burden by disrupting context, reasoning, and reviewers’ ability to retain information across files.The study specifically examines whether this ordering contributes to missed defects or reduced review efficiency.
- Study approach: The study surveys 1,355 professional developers from 67 countries and 182 popular open-source projects about file ordering, review behavior, perceived effectiveness, cognitive effort, and usability.Participants report prioritization practices, perceptions of alphabetical ordering, and challenges in multi-file reviews.
2 Background and Related Work
Prior work establishes that file position shapes reviewer attention and navigation, while cognitive constraints can reduce review thoroughness. This study addresses the resulting evidence gap by examining reviewers’ perceptions, reasoning, coping strategies, and usability experiences.
- Cognitive foundations: Code review requires sustained attention, mental-model construction, reasoning about delocalized bugs, and comprehension of large or multi-file changes.These demands can exceed working-memory capacity and interact with biases such as anchoring, confirmation, and availability effects.
- Position and attention: Attention decrement causes focus to decline across sequential files, producing positional bias and uneven scrutiny during review.Prior studies found that top-positioned files receive more comments and that bugs in first files were 64% more likely to be detected than bugs in last files.
- Ordering strategies: Reviewers often follow alphabetical file order, but related work recommends grouping related changes, prioritizing large diffs, and surfacing files linked to descriptions or tests.Reviewers also deviate from alphabetical order, with 44.6% showing non-alphabetical commenting sequences in one study.
- Research gap: Existing studies mainly measure observable behavior, leaving reviewers’ perceptions, reasoning, and subjective experiences of navigation insufficiently explained.The present study targets this gap through a large-scale survey of ordering perceptions, coping strategies, usability challenges, and expectations for tool support.
3 Methodology
The study uses an online mixed-methods survey of active open-source contributors and professional developers to examine navigation, ordering perceptions, review effectiveness, multi-file challenges, and desired tooling improvements. The questionnaire combines closed- and open-ended items and applies quantitative and qualitative analyses.
- Research questions: The survey addresses five research questions covering navigation behavior, alphabetical-order perceptions, perceived review effectiveness, multi-file challenges, and expectations for improved tooling.The questions examine both alignment with default ordering and the role of customizable ordering mechanisms.
- Survey design: Researchers conducted an online survey following established empirical software-engineering guidelines.The questionnaire was designed to address the study’s research questions through closed- and open-ended items.
- Survey design: The questionnaire covered review strategies, default-ordering perceptions, cognitive load, change impact, adaptive behaviors, external tool use, and alternative ordering preferences.It also collected background information relevant to participants’ review experiences.
- Validation: Survey face validity was reviewed by two industry practitioners and two software-engineering researchers before iterative revision and ethics approval.The reviewers had substantial professional or research experience in software engineering and code review.
- Recruitment: The sample targeted 182 mature repositories selected using popularity and activity metrics, supplemented by snowball sampling through contributors’ professional networks.Detailed project-selection criteria and recruitment statistics were provided in supplementary material.
- Analysis: Quantitative analyses used descriptive statistics, chi-square tests, logistic regression, effect sizes, odds ratios, and 95% confidence intervals.The analyses controlled for demographic and experiential covariates and used α=0.05 for significance testing.
- Analysis: Qualitative responses were analyzed through manual categorization and thematic analysis independently conducted by two evaluators using open coding.Concise factual responses were assigned to predefined or emergent categories.
4 Evaluation and Results
Across the survey, reviewers often deviated from alphabetical ordering and generally viewed context-driven or customizable sequences as better aligned with comprehension and review quality. Alphabetical ordering was associated with cognitive inefficiency, uncertainty about missed bugs, disrupted context, and additional effort during multi-file reviews.
- 42.4% of reviewers adhered to alphabetical ordering, while the majority adopted alternative, context-driven strategies for multi-file PRs.
- Only 10.2% considered alphabetical ordering optimal, compared with 53.9% who considered it suboptimal and 35.9% who were neutral.
- 63.9% expressed concern or uncertainty that alphabetical ordering could contribute to missed bugs.
- Reviewers associated non-alphabetical and customizable ordering with improved comprehension and review effectiveness, independent of experience-driven differences.
- Arbitrary ordering disrupted dependency reasoning, separated implementation from test files, and encouraged premature or irrelevant feedback.
- About 41% mentally reordered files or used external aids, indicating that reviewers adapted cognitively to ordering they found misaligned with their reasoning.
5 Discussion
The findings indicate that alphabetical file ordering is misaligned with reviewers’ cognitive strategies, adding cognitive burden and disrupting comprehension. They support reviewer-centric, adaptive interfaces that group files semantically or by dependencies while allowing customization.
- Tool-Cognition Misalignment: Alphabetical ordering forces reviewers to mentally reconstruct semantic relationships, adding cognitive load and separating related files such as implementation–test pairs.This misalignment can produce premature judgments and lower-quality feedback.
- Reviewer-Centric Tool Design: Reviewer-centric tools should provide dependency-aware, semantically grouped, and customizable file ordering rather than rigid defaults.The study reports that no single ordering suits all contexts or review goals.
- Reviewer-Centric Tool Design: Empirically validated ordering preferences can inform interfaces intended to enhance comprehension, efficiency, and reviewer well-being.
- Practical Implications: Disciplined PR authoring practices, including atomic changes, logical grouping, clear descriptions, and limited PR size, can mitigate cognitive load.The discussion also recommends strategically managing review effort through responsibility rotation and reviewer training.
- Broader Implications: The findings extend cognitive-ergonomics concerns beyond code review by showing that tool design shapes reasoning, collaboration, and satisfaction.
6 Threats to Validity
The study’s validity is constrained by subjective survey constructs, self-reported responses, sampling and cross-sectional design, and limited generalizability beyond the open-source ecosystem. Statistical associations should therefore be interpreted as correlational rather than causal.
- Construct Validity: “File ordering optimality,” “review effectiveness,” and “cognitive load” are subjective constructs that participants may interpret differently.Expert review, defined scales, open-ended responses, and triangulation were used to mitigate this threat.
- Internal Validity: Self-reported perceptions and behaviors may introduce recall and social-desirability biases despite methodological safeguards.
- Internal Validity: The cross-sectional survey and recruitment through public GitHub addresses and snowball sampling may limit representativeness and capture only one point in time.
- Internal Validity: The statistical analyses identify associations among experience, ordering preferences, and review perceptions but cannot establish causal relationships.
- External Validity: Findings may not generalize from popular open-source projects to closed-source environments, specialized tools, or organizations with overriding review protocols.
7 Conclusion
The study finds widespread dissatisfaction with alphabetical file ordering among professional developers in popular open-source projects, driven largely by difficulty tracking dependencies and prioritizing critical changes. Respondents strongly preferred semantically or logically grouped sequences.
- The survey of 1,355 professionals across 182 popular open-source projects found widespread dissatisfaction with alphabetical file ordering.
- Respondents associated dissatisfaction with difficulties tracking inter-file dependencies and prioritizing critical changes.
- Respondents strongly preferred semantically or logically grouped file sequences.