Cognitive Biases in Software Engineering: A Systematic Mapping Study

• Cognitive biases are linked to problems in design, testing, requirements engineering, and project management, and can inform improved practices, methods, and artifacts.Examples include confirmation-biased unit tests and optimism bias in project estimates.Source: §1 · pp. 1–2
• Software engineering research on cognitive biases was disconnected, and no analogous review of that research existed.The authors present synthesis as a way to make findings more cumulative and communicable.Source: §1 · p. 1
• The study reviews and synthesizes software engineering research on cognitive biases, including publication patterns, antecedents, effects, and mitigation approaches.The authors argue that enough studies had accumulated to connect previously disconnected research and identify patterns.Source: §1 · p. 1
• Bias mechanisms can include heuristics, emotion, social influences, and noisy information processing, but the causes of many biases remain poorly understood.The primer also describes fast, intuitive thinking as bias-prone, while noting that biases can arise through other phenomena.Source: §1 · p. 2
• Debiasing can target the task rather than the person, but effort estimation can involve interconnected biases that a single redesign may not address.Planning Poker targets anchoring, not optimism bias; person-focused warnings may be ineffective, while extended training can be expensive.Source: §1 · p. 2

3 RESEARCH OBJECTIVES AND METHOD

• A systematic mapping study was chosen to categorize and aggregate knowledge dispersed across disconnected studies with inconsistent terminology or theoretical concepts.The protocol followed established guidelines.Source: §3 · p. 2

3.1 Research questions

• The research questions cover investigated biases, antecedents, effects, debiasing, research methods, publication patterns, researchers, and targeted SWEBOK areas.The bias categories and descriptions are presented in a table adapted from an information-systems taxonomy.Source: §3.1 · p. 2, Table 1 · p. 3

3.4 Primary study selection

• Study selection combined database screening with recursive reference checking and prolific-author searches, adding 16 and nine studies, respectively, to the initial 40.Reference checking took four iterations; two authors piloted screening with medium-to-high agreement (Cohen’s Kappa = 0.77).Source: §3.2 · p. 2, §3.4.2 · pp. 3–4, §3.5.2 · p. 5

3.3.1 Database selection and search string formatting

• Five databases were searched using full-text queries, and all ten studies selected in advance as relevant were found with the revised software AND “cognitive bias” search.The revised search returned 826 records; searches were completed on 20 December 2016.Source: §3.3.1 · p. 3

3.5.4 Searching specific bias names

• Searching for 18 additional bias names found one new study, while backward snowballing on that study found none.The search followed comparison with biases investigated in an information-systems review but absent from the primary studies.Source: §3.5.4 · p. 5

3.6 Quality assessment

• Quality assessment addressed reporting practices rather than methodological rigor; excluding one non-empirical study left 65 studies, with mean quality scores of 84% for empirical and 85% for non-empirical studies.The final set comprised 47 empirical and 18 non-empirical studies.Source: §3.6 · p. 5

3.7 Data extraction

• The authors coded study passages for investigated biases and discussions of antecedents, effects, or debiasing, then organized and reviewed the classifications against the research questions.NVivo was used to organize and analyze qualitative data and bibliographic information.Source: §3.7 · p. 5

4 RESULTS

• Across the primary studies, 37 distinct cognitive biases were investigated, but antecedents were considered for only 11; anchoring (26), confirmation (23), and overconfidence (16) were most common.Source: §4.1 · pp. 5–6
• All eight bias categories were represented, with interest and stability biases most investigated and social biases studied in only two studies; category coding had 55% agreement.Categories could overlap, and some biases did not fit any category, so the authors assigned each to its most appropriate category.Source: §4.1 · pp. 5–6
• Anchoring was linked to inexperience, uncertainty, limited business knowledge, and inflexible clients, while optimism varied with role, task difficulty, and risk distance.Experts appeared less susceptible to anchoring than novices; developers were more optimistic than managers estimating web tasks, and near-future risks were judged more accurately than distant ones.Source: §4.1 · pp. 6–7

4.3.1 Anchoring and adjustment bias

• Anchoring can propagate errors from reused artifacts, impede adjustments to changed requirements, and distort project estimates, contributing to budget and schedule overruns.A study of students found apparent anchoring on concurrent coursework, but did not establish whether the resulting designs were poor or inappropriate.Source: §4.3.1 · p. 9
• Confirmation bias affects documentation searches, trade-off studies, design changes, and testing, where it can increase defects and lead testers to favor tests that confirm expected behavior.During debugging, it can also lead programmers to misidentify causes of program failure.Source: §4.3.1 · p. 9

4.3.3 Hyperbolic discounting

• One study argued that hyperbolic discounting favors quick fixes and simple refactoring over more ambitious revisions; availability bias can steer searches toward familiar documentation terms and locations.Availability bias also appears in project decisions based on easily recalled information or in the absence of lessons learned from earlier projects.Source: §4.2.3 · p. 7, §4.3.3 · p. 9
• Availability bias can make searches inefficient when memorable past experience mismatches the current software [P13, P17], while hyperbolic discounting is linked to neglecting maintenance for short-term gains [P36].Availability bias was also associated with overrepresenting memorable code features and preferring a familiar but less suitable programming language [P32].Source: §4.3.3 · p. 10
• Framing dubious desiderata as requirements may fixate designers on them and reduce creativity [P19], while experts labeled as attentional-biased overlooked diagram contradictions by focusing on relationships [P28].The review questions the attentional-bias explanation because the study did not explain what recurring thoughts caused the experts to ignore surface semantics.Source: §4.3.3 · p. 10, §4.3.6 · p. 10

4.3.7 Representativeness

• Representativeness may lead professionals to overlook error-producing tests, increasing defect density, and to mistake notable code features for frequent ones [P7, P32].The proposed mechanism is judging from a few properties or stereotypes rather than representative evidence.Source: §4.3.7 · p. 10, §4.3.3 · p. 10
• One study proposed that miserly information processing prompts uncritical acceptance of client requirements, bandwagon effects acceptance of team-leader decisions, and status quo bias defense of prior illogical choices [P21].The review found no empirical research investigating these proposed causal relationships.Source: §4.3.7 · pp. 10–11, §4.3.3 · p. 10
• Overconfidence was linked to neglected requirements analysis, superficial or incorrect understanding, and project failure from poor time and resource estimates [P3].The review cautions that confirmation bias, miserly information processing, and anchoring may also contribute to poor understanding and estimation.Source: §4.3.3 · p. 10, §4.3.7 · p. 11

4.4 What debiasing approaches are investigated?

• A primary study speculates that burndown charts, bottom-up planning, demonstrations, daily meetings, flexible planning, and stakeholder feedback may guard against multiple biases.The passage presents these as a study's speculation, not as validated interventions.Source: §4.4 · p. 11
• Techniques target only six of the 37 investigated biases, and none has strong empirical evidence of effectiveness; all are untested suggestions.The authors caution that proposed techniques may also have unintended consequences.Source: §4.4 · p. 11
• Availability-bias suggestions include FAQs, spelling conventions, ontology-based documentation, traceability, contextual framing, and stakeholder-involved retrospectives [P13], [P17], [P18], [P32], [P53], [P63], [P65].The paper notes that detailed experience records may become overlong and quickly outdated [P53].Source: §4.4.1 · p. 11
• For confirmation bias, suggestions include seeking evidence of problems in testing, generating and evaluating alternative designs, and using traceability [P32], [P37], [P16].The paper observes that few studies propose ways to mitigate confirmation bias despite its frequent investigation.Source: §4.4.2 · p. 11
• Model-based forecasting may avoid anchoring in expert-based effort estimates, but models can have statistical bias and variance and may need unavailable project data [P53], [P63].The review confines its coverage to software-engineering research, although forecasting is also studied in operations research.Source: §4.4.2 · p. 11
• Planning Poker and generating multiple design candidates are proposed to reduce anchoring [P38], [P33], but evidence on group estimates using multiple anchors is mixed.The authors report no known research on how multiple design candidates affect anchoring.Source: §4.4.2 · p. 11
• Directed questions can prompt scrutiny of initial estimates and counter overconfidence [P3]; asking about task effort yielded less optimistic estimates than asking about work-period output [P47].Raising awareness of anchoring alone is described as a weak intervention [P37].Source: §4.4.2 · pp. 11–12

4.4.5 Representativeness

• For representativeness, proposed approaches include consistent interfaces, alternative-solution prompts, retrospectives, warnings, and training [P20], [P53].These techniques are described as not evaluated.Source: §4.4.5 · p. 12

4.5 What research methods are used?

• Most empirical studies used laboratory experiments, while predominantly qualitative approaches such as case studies were underrepresented.Nine empirical papers reported multiple studies: seven reported multiple experiments and two used multimethod approaches.Source: §4.5 · p. 12
• Other studies used pre-existing datasets or did not report enough methodological detail to be classified.Source: §4.5 · p. 12

4.6 When were the articles published?

• Primary-study publications increased unevenly after 2001, peaked in 2010, and numbered only two in 2016, possibly because the December 2016 search missed unindexed studies.The 2010 spike was mostly due to four publications coauthored by Calikli and Bener.Source: §4.6 · pp. 12–13, Figure 3

4.7 Where were the articles published?

• Journals or academic magazines published 58% of the articles; the Journal of Systems and Software had the most, with five.Source: §4.7 · p. 13

4.8 Who is conducting the research?

• Of 109 authors, 81 had one paper, 16 had two, and eight had three; Magne Jorgensen was the most prolific, while Calikli and Bener coauthored eight papers.Jorgensen and Molokken collaborated on four publications.Source: §4.8 · p. 13

4.9 Which SWEBOK knowledge areas are targeted by research on cognitive biases?

• SE management was investigated in 21 primary studies, followed by software construction in 12 and software design in 11; requirements, design, testing, and quality were underrepresented.Source: §4.9 · p. 13

5 DISCUSSION

• The study identifies four research-improvement priorities: more qualitative and multimethod work, neglected topics, cross-study integration, and clearer treatment of confusion.Qualitative and exploratory work can examine how biases manifest and are perceived in practice; experiments alone do not establish effects on real projects.Source: §5.1 · p. 13
• The review finds neglected biases, categories, and SE knowledge areas; 21 biases appeared in only one or two studies, and few studies proposed debiasing techniques.Examples of underexplored biases include those studied in information systems but not SE, while social and memory biases are largely ignored.Source: §5.1 · p. 13
• Confusion arises from inconsistent or overlapping bias labels, questionable causal claims between similar phenomena, and misuse of terms such as heuristics and fallacies.The discussion also notes articles using terms incorrectly, including treating Parkinson’s Law as a cognitive bias.Source: §5.1 · p. 15
• Researchers applying reference-discipline theories should study them deeply and may need psychology mentors; editors should consider expert review from that discipline.The authors consulted three psychology professors to check their interpretations and terminology.Source: §5.1 · p. 15
• Task-focused debiasing may be quicker and cheaper than changing people, but reviewed studies provide no compelling evidence that a specific technique will work in a future project.The paper lists proposed SE techniques, including Planning Poker for anchoring, but cautions that they are not extensively validated and may have unintended consequences.Source: §5.3 · pp. 15–16
• Extensive training may be impractical for effort estimation, while repeatedly generating alternative designs may be a more practical educational exercise.The discussion suggests teaching biases alongside common problems and practices, and framing planning poker through the causes of estimation problems and potential remedies.Source: §5.3 · p. 16
• Synthesis is constrained by uncertain relationships among biases, inconsistent terminology, and possible omissions from database, category, and publication-bias limits.The authors warn that relevant studies may have been missed despite search precautions, particularly because unpublished works were excluded.Source: §5.5 · p. 16

5.6 Future research

• Future work should study bias processes and context-specific effects, adapt or develop debiasing approaches, and empirically evaluate their effectiveness.The proposed methods include qualitative studies for social and cognitive processes, statistical modelling for effects, and experiments or action research to evaluate debiasing artifacts.Source: §5.6 · p. 17

6 CONCLUSION

• 65 primary studies investigate 37 cognitive biases; interest and stability biases receive the most attention, while social and decision biases receive the least.The authors also identify software project management as an area of sustained interest and point to design, testing, and devops as opportunities for further research.Source: §6 · p. 17
• Psychological and sociological mechanisms underlying many biases remain poorly understood, and debiasing approaches cover only six biases.None of the identified approaches were empirically evaluated, and few were grounded in psychological theory.Source: §6 · p. 17
• Controlled experiments are the most common method, but the review found no replications, experiment families, or meta-analyses.Source: §6 · p. 17
• Many primary studies are disconnected: they do not cite earlier SE work on the same bias or conceptualize that bias consistently.Source: §6 · p. 17
• The authors conclude that awareness alone does not counter cognitive biases, while simple interventions can often debias professionals.They argue that stronger theoretical understanding of problems and practices can make effective interventions easier to develop.Source: §6 · p. 17

APPENDIX A DEFINITIONS OF COGNITIVE BIASES INVESTIGATED IN SE

• Anchoring and adjustment concern excessive reliance on an initial value; endowment, fixation, framing, halo, and hindsight describe ownership valuation, narrowed focus, response shifts, global-to-specific judgment, and hindsight predictability.Source: Appendix A · p. 20
• Attentional bias reflects recurring thoughts; invincibility bias overtrusts ability; mere exposure increases liking; miserly processing avoids complex thought; other entries describe following information and neglecting probability.Source: Appendix A · pp. 20–21
• Belief perseverance maintains refuted beliefs; overconfidence overestimates personal skill or control; confirmation bias favors prior beliefs; contrast and default biases alter perception or choice.Source: Appendix A · p. 20
• Hyperbolic discounting favors immediate rewards; IKEA effect values self-made items; impact bias overestimates future feelings; information and infrastructure biases concern unnecessary information and existing infrastructure.Source: Appendix A · p. 20
• The appendix also describes normalcy and optimism tendencies, primacy and recency, availability, and reducing inference to similarity judgments.Source: Appendix A · pp. 20–21
• Other entries cover differing perceptions, privileging model syntax over semantics, rejecting contradictory information, maintaining the status quo, and sunk-cost investment.Source: Appendix A · p. 21
• The list includes time-based short-term thinking, bandwagon alignment, positive-outcome weighting, validity from repetition, and underestimating negative outcomes.Source: Appendix A · pp. 20–21

APPENDIX

• Appendix B is identified as the list of primary studies.Source: APPENDIX · p. 22
• The visible bibliography includes a 2004 entry by DeFranco-Tommarello and Deek on collaborative problem solving and groupware for software development.Source: APPENDIX · p. 22
• Visible entries include Ralph's works on core theories for software engineering and debiasing software development, and Shull's work on engineering values.Source: APPENDIX · p. 22