PaperlayerView exact revision on arXiv

Source-linked AI summary

A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models

Tomáš Kliegr, Štěpán Bahník, Johannes Fürnkranz

arXiv:1804.02969v7stat.MLcs.AIcs.LG

TL;DR

Human interpretation of machine-learning models may be affected by cognitive biases, particularly when people assess rules learned from data. This review surveys twenty biases and possible debiasing techniques. It concludes that machine-learning-specific empirical validation remains a critical next step.

  • Problem

    The paper addresses the limited systematic discussion of how cognitive biases may affect human understanding of machine-learning models, especially inductively learned rules.

    Source: §1 · p. 1, §8 · p. 51

  • Method

    The authors review twenty cognitive biases and discuss how they may affect rule interpretation and how their effects might be mitigated.

    Source: §1 · p. 1, §8 · p. 51

  • Results

    The review identifies twenty biases, heuristics, and effects that can give rise to systematic errors when inductively learned rules are interpreted.

    Source: §8 · p. 51

  • Takeaways & Limitations

    The review provides a psychological basis for further research on inductive rule-learning algorithms and communication of their results.

    Source: §8 · p. 51

    Empirical validation of the selected biases in machine-learning settings is a critical next step.

    Source: §7.1 · p. 48

Abstract

from arXiv · show

While the interpretability of machine learning models is often equated with their mere syntactic comprehensibility, we think that interpretability goes beyond that, and that human interpretability should also be investigated from the point of view of cognitive science. The goal of this paper is to discuss to what extent cognitive biases may affect human understanding of interpretable machine learning models, in particular of logical rules discovered from data. Twenty cognitive biases are covered, as are possible debiasing techniques that can be adopted by designers of machine learning algorithms and software. Our review transfers results obtained in cognitive psychology to the domain of machine learning, aiming to bridge the current gap between these two areas. It needs to be followed by empirical studies specifically focused on the machine learning domain.

1. Introduction

The paper asks how cognitive biases may affect human understanding of machine-learning models, especially inductively learned rules. It reviews relevant psychological research and possible debiasing approaches, while noting the need for machine-learning-specific empirical studies.

1. Introduction

  • Prior psychology research addresses bias in decision-making, but follow-up work on cognitive phenomena in rule induction and machine learning has been sparse.Earlier work by the authors examined whether minimizing rule complexity or length increases interpretability.Source: §1 · p. 2
  • The review examines how twenty cognitive biases and heuristics may distort interpretation of inductively learned rules and asks what might help mitigate those effects.The discussion centers on symbolic rules, which are often treated as interpretable, and considers whether their presentation can affect perceived trustworthiness, safety, or fairness.Source: §1 · p. 2
  • The paper situates its review among work on rule learning, psychology, bias-selection criteria, recommendations, and review limitations.It previews twenty biases, developer-oriented recommendations, and future research directions.Source: §1 · p. 2

2. Background and Related Work

The section frames individual rules as focused objects for studying how psychological processes shape interpretation. It connects rule learning to cognitive-science accounts of reasoning and outlines how the review understands bias, debiasing, and interpretability.

2. Background and Related Work

  • The review focuses on individual learned rules as small, self-contained knowledge items, enabling focused analysis of changes such as adding a condition.It situates this choice amid use of rules to explain black-box models and renewed interest in directly learned rule models.Source: §2 · p. 3, §2.1 · p. 3

2.1. Motivation

  • Research at the AI–psychology intersection is limited; this review applies psychological findings to machine-learning rule interpretation and surveys twenty biases, broader than the four addressed in.The review lacks experimental validation specific to machine learning, as does the cited work by Miller.Source: §2.1 · pp. 3–4

2.2. Decision Rules in Machine Learning

  • Rule-learning outputs are characterized by confidence and support, which quantify correct predictions among covered cases and the number or proportion of supporting cases.Confidence is also called precision, while coverage counts cases matching the rule body.Source: §2.2 · p. 4

2.3. Decision Rules in Cognitive Science

  • Cognitive-science accounts connect a rule’s confidence to the conditional probability of its consequent given its antecedent, and its support to the evidence sample size.The Wason 2-4-6 task illustrates hypothesis testing: participants may seek confirming rather than disconfirming evidence.Source: §2.3 · pp. 5–6

2.4. Cognitive Bias

  • The review treats biases broadly, including thinking, judgment, and memory phenomena, and notes that heuristics may fail when applied outside suitable contexts.It presents debiasing approaches including educational, workplace, and forcing-function strategies, with some intended to shift processing toward conscious reasoning.Source: §2.4 · pp. 6–7

2.5. Measures of Interpretability, Perceived and Objective Plausibility

  • Because syntactic model size does not capture semantics and no generally accepted interpretability measure exists, the authors use perceived rule plausibility as a measure.Plausibility is linked to whether a rule seems reasonable or probable and reflects domain semantics and knowledge.Source: §2.5 · pp. 7–8

3. Motivational Example

The example shows how different aspects of a rule can shape perceived plausibility, from attribute relevance and logical connectives to confidence and support. It also introduces possible training, algorithm, and representation changes to mitigate bias.

3. Motivational Example

  • Rule evaluation requires judging attribute predictiveness and coverage, interpreting AND as intersection, and weighing confidence against support.The example illustrates biases that may affect each of these processes.Source: §3 · p. 8
  • Low support may be overlooked despite high confidence, while the same information can trigger opposing plausibility effects depending on the analyst and context.For instance, a director condition may increase plausibility through information bias but decrease it through ambiguity aversion when the director is unfamiliar.Source: §3 · p. 9
  • The review groups debiasing approaches into user training, learning-algorithm adaptation, and changes to model representation or user interface.Training can improve statistical reasoning and understanding of sample size; algorithm adaptation can use confidence intervals and omit weakly grounded rules.Source: §3 · p. 10
  • The example is motivational rather than a general guideline; the paper revisits its biases and gives more concrete recommendations later.Some quantitative results and participant responses from earlier user studies are used for illustration.Source: §3 · pp. 8, 11

4. Scope of Survey

The survey selects biases by identifying rule properties that may influence perceived plausibility. It covers twenty heuristics and biases, while leaving room for future expansion.

4. Scope of Survey

  • The review selects biases that can interact with rule length, interest measures, condition or rule order, predictive properties, logical connectives, missing values, or rule conflicts.The criterion is whether activation of a bias could affect perceived rule plausibility.Source: §4 · p. 11
  • Learning heuristics can shape rule properties by trading off coverage or support against implication strength or confidence.This connects algorithm choices to properties considered in the bias-selection criteria.Source: §4 · p. 11
  • The review covers twenty judgmental heuristics and cognitive biases, with future work potentially adding effects such as labelling and overshadowing.The authors selected biases they considered most relevant to machine-learning research under their criteria.Source: §4 · p. 11

5. Review of Cognitive Biases

The review relates twenty cognitive biases to how people interpret inductively learned rules, organizing them by affected rule-model element and underlying mechanism. It also proposes debiasing techniques, while noting limited applied-psychology grounding and the need for further validation.

5. Review of Cognitive Biases

  • The review covers twenty biases and their proposed explanations, organizing them by affected rule-model element—from literals to whole models—and underlying mechanism.It also discusses possible debiasing techniques for each bias.Source: §5 · pp. 11–12, Figure 4 · p. 12
  • The review proposes a debiasing technique for each bias, while noting that some strategies may apply across multiple biases.Its frequency-format adaptation is intended especially for base-rate neglect and insensitivity to sample size, and also addresses misunderstanding “and.”Source: §5 · pp. 12–13
  • The proposed techniques have limited grounding in applied psychological research and need further validation; the review includes only one specifically rule-learning debiasing method because applicable work is scarce.Developing a rule-learning-specific technique for each of the twenty biases is outside the article’s scope.Source: §5 · pp. 12–13

5.1. Conjunction Fallacy and Representativeness Heuristic

  • 85% of Tversky and Kahneman’s subjects chose the conjunction in the Linda problem; the fallacy has also appeared across settings and types of participants.The conjunction rule says a conjunction cannot be more probable than either constituent.Source: §5.1 · pp. 13, 15
  • Representative conditions may make a rule seem more plausible even when its objective discriminatory power is lower, and the literals must form a coherent representative description.The latter point is illustrated by user-choice justifications reported in.Source: §5.1 · p. 16
  • Frequency formats can reduce or eliminate conjunction fallacies, but raw frequencies may prompt mental ratio computation and increase errors, while reference values can distort support.The proposed rule-learning wording presents frequencies alongside precomputed confidence and support ratios and requires further experimental validation.Source: §5.1 · p. 17

5.2. Misunderstanding of “and”

Misunderstanding “and” can make rule conditions seem to increase coverage and plausibility when users interpret conjunction as union. The review also discusses averaging as a distinct heuristic that can obscure how condition probabilities combine, alongside wording and representation strategies and their costs.

5.2. Misunderstanding of “and”

  • Interpreting “and” as union rather than intersection can make each added condition seem to increase rule coverage and plausibility, encouraging preference for rules with more conditions.About a quarter of participants in cited experiments interpreted “and” as union rather than intersection.Source: §5.2 · p. 18
  • Connecting propositions rather than categories, using wording such as “and also,” or showing semantics visually may reduce ambiguity around “and.”Decision trees are suggested as a visual alternative to rules, though support for this proposition is limited.Source: §5.2 · p. 19, §5.3 · p. 19

5.3. Averaging Heuristic

  • The averaging heuristic offers an alternative account of conjunction judgments: people may average component probabilities instead of multiplying them.One cited experiment found that a model averaging component likelihoods accounted for approximately 49% of variance in subjects’ conjunctions.Source: §5.3 · p. 19
  • Averaging can obscure the effect of a low-probability condition: rules with the same average condition probability may have different joint probabilities under independence.The examples contrast three- and two-condition rules and compare two equal-length rules with different component probabilities.Source: §5.3 · pp. 19–20
  • Probability reminders may reduce conjunction fallacies, while logarithm formats may counter weighted averaging at the cost of training and greater mental effort.The review notes that these costs may outweigh the benefits.Source: §5.3 · p. 20

5.4. Disjunction Fallacy

The review links two judgment biases to rule interpretation: preference for more specific rules can discount statistical validity, while base-rate neglect can make rules with equal support and confidence seem equally plausible. It suggests making attribute hierarchies and class priors more visible to analysts.

5.4. Disjunction Fallacy

  • Analysts may prefer more specific attributes in hierarchical feature spaces, even when those rules have fewer supporting instances and weaker statistical validity.In user feedback, some participants preferred longer rules as more descriptive or specific.Source: §5.4 · pp. 20–21
  • Making taxonomic relations explicit and explaining the larger supporting samples behind general attributes are proposed debiasing strategies.The interface could notify analysts that one category subsumes another, complemented by training in rules.Source: §5.4 · p. 21

5.5. Base-rate Neglect

  • Support and confidence do not communicate the prior probability of a rule’s consequent, although that base rate matters when judging plausibility.In the example, the engineer and lawyer base rates are 1.8% and 7.2%, respectively.Source: §5.5 · pp. 21–22
  • Presenting lift can expose differences hidden by equal confidence: Rule 1 has lift 50, while Rule 2 has lift 12.5.Both rules have 90% confidence, but the engineer base rate is 1.8% and the lawyer base rate is 7.2%.Source: §5.5 · p. 22
  • The review also proposes showing rule statistics as natural frequencies and using coverage diagrams to help analysts account for base rates.The authors suggest presenting natural frequencies alongside percentages and adapting Euler-circle diagrams.Source: §5.5 · pp. 22–23

5.6. Insensitivity to Sample Size

The review argues that analysts may undervalue support as evidence of estimate reliability and favor confidence instead. It connects this tendency and confirmation bias to rule-learning interfaces that expose sample-size information, priors, and opportunities to revisit judgments.

5.6. Insensitivity to Sample Size

  • Insensitivity to sample size can lead analysts to overlook that higher support makes a rule’s confidence estimate more reliable.The bias may favor a higher-confidence rule over one with higher support, all other factors equal.Source: §5.6 · p. 23
  • When confidence and support were both shown, participants favored a rule with slightly higher confidence despite another rule’s much higher support; a similar study confirmed the finding.No participant rated Rule 2 as more plausible, although it had 25.9% support versus Rule 1’s 1.95%.Source: §5.6 · p. 24
  • Interfaces could communicate how support affects confidence-estimate accuracy and provide confidence intervals; the review hypothesizes that frequency formats may reduce, but not eliminate, the bias.A variation of frequency-format debiasing was used in, yet the effect remained.Source: §5.6 · p. 25

5.7. Confirmation Bias and Positive Test Strategy

  • Confirmation bias may lead analysts to select rules that support an existing hypothesis and disregard applicable rules with opposing conclusions.The review suggests delayed review and editing, as well as explicit prompts to consider evidence both for and against a hypothesis.Source: §5.7 · p. 26
  • The authors propose showing class priors alongside support and confidence; Figure 8 depicts this addition to EasyMiner’s interface.The selected-rule contingency table can provide the prior, but the authors describe calculating it as inconvenient; it also presents frequencies in a format considered easier to understand.Source: §5.7 · pp. 26–27, Figure 8 · p. 27

5.8. Availability Heuristic

The review considers how ease of recall, repetition, and brief exposure may shape judgments of rules. It connects these effects to rule characteristics and presentation practices, while noting limits on directly applicable debiasing methods for mere exposure.

5.8. Availability Heuristic

  • The availability heuristic uses ease of recalling relevant instances as a cue to judge class frequency or event probability, which can bias judgment when that cue is invalid.The review illustrates the mechanism with the finding that about 70% of participants incorrectly judged that “R” appears more often in the first position than the third.Source: §5.8 · p. 27
  • Rules whose matching instances are easier to recall may seem more plausible, and analysts may even ignore antecedent conditions whose matches are harder to remember.The review proposes alerting analysts to irrelevant reasons for ease of recall as a possible debiasing step.Source: §5.8 · p. 28

5.9. Reiteration Effect, Effects of Validity and Illusiory Truth

  • Repeated, overlapping rules may increase belief in a shared rule or subrule; even rejected weak rules may make similar rules more acceptable later.The review connects this possibility to redundant association-rule outputs and reports that a 30-minute delay can let initially negative information have a positive influence.Source: §5.9 · p. 29
  • Pruning redundant rules or initially presenting clusters of rules can reduce repeated exposure to similar statements.The review also proposes showing which rules share coverage, since independent sources of gossip increased perceived veracity.Source: §5.9 · p. 29

5.10. Mere Exposure Effect

  • Brief exposure while scrolling through lists of thousands of rules may influence judgments even when analysts do not realize they have seen a rule.The review also notes a possible mere-exposure effect for rules containing frequently occurring words in text mining or sentiment analysis.Source: §5.10 · pp. 30–31
  • The review found no directly applicable debiasing techniques for mere exposure, though it suggests limiting subliminal exposure by changing interface operation.The reported reversal of the effect involved fearful participants viewing spider pictures, making it difficult to transpose to rules.Source: §5.10 · p. 31

5.11. Overconfidence and underconfidence

The section explains how confidence, recognition, and information-seeking biases may shape judgments of learned rules, including their perceived plausibility and predictive value.

5.11. Overconfidence and underconfidence

  • Rules with high confidence but low support may have their predictive ability substantially overrated, paralleling algorithms’ preference for overly specific rules.The authors suggest this similarity means human-in-the-loop solutions may not alleviate overfitting.Source: §5.11 · p. 32

5.12. Recognition Heuristic

  • Recognized antecedent attributes may attract preference under rule-review time pressure, though recognition can sometimes be useful and feedback offers examples of its application.In feedback, participants cited prior experience with anise smell and familiarity with Switzerland when judging rules.Source: §5.12 · p. 33

5.13. Information Bias

  • Information bias may lead users to configure larger rule lists and longer rules with low-value attributes, and participants also judged longer presented rules as more plausible.Exposing the importance of individual rule conditions is proposed as a way to counter this bias.Source: §5.13 · pp. 34–35

5.14. Ambiguity Aversion

The section applies ambiguity aversion and confusion of the inverse to rule interpretation, describing how unfamiliar conditions and implication wording can affect judgments.

5.14. Ambiguity Aversion

  • Analysts may prefer rules without unfamiliar conditions; feedback includes a mushroom rule judged less plausible because it contained the unknown trait creosote.The authors propose requiring justifications, particularly when a rule conflicts with existing knowledge, and making condition meanings accessible.Source: §5.14 · pp. 35–36

5.15. Confusion of the Inverse

  • Confusing an implication with its inverse may make characteristic rules seem preferable to short discriminative rules, especially when attributes can appear on either side of association rules.Training in probabilistic theory was reported as ineffective against the confusion of the inverse; the authors recommend unambiguous wording for implications.Source: §5.15 · p. 37

5.16. Context and Tradeoff Contrast Effects

The section discusses how choice context, negative information, and presentation order may influence analysts’ judgments of learned rules, alongside possible interface and output changes.

5.16. Context and Tradeoff Contrast Effects

  • Showing inferior rules alongside a rule with confidence 0.7 could increase that rule’s plausibility through the tradeoff contrast effect.The effect may not apply when people have well-articulated preferences; removing rules with extremely low confidence or support is proposed as a debiasing step.Source: §5.16 · p. 38

5.17. Negativity Bias

  • Negative wording may attract attention to a rule: participants’ rationales referred to a foul-smell condition, even though some preferred the alternative rule.The authors caution that this observation is not necessarily proof of the bias.Source: §5.17 · p. 39

5.18. Primacy Effect

  • Analysts may favor rules presented first, especially when list order does not reflect rule quality; presenting relevant rules first or supporting note-taking are proposed debiasing strategies.In a health-information retrieval study, an interface that reshuffled document order eliminated the order effect, unlike the baseline and annotation interfaces.Source: §5.18 · pp. 39–41

5.19. Weak Evidence Effect

The section considers how weak predictors and condition counts can distort judgments of learned rules, and discusses ways to present or convey evidence strength.

5.19. Weak Evidence Effect

  • A weak condition added to a rule with a strong predictor may reduce its perceived plausibility; numerical expressions were most suitable for expressing uncertainty in the cited study.That study concerned jurors assessing trial evidence; the authors caution that conveying evidence through language or visualizations requires care.Source: §5.19 · pp. 41–42

5.20. Unit Bias

  • Analysts may treat rule conditions as similarly important despite differences in discriminatory power, and feedback included judgments based on the number of conditions.The authors suggest communicating each condition’s discriminatory power, for example by listing how many dataset instances meet it.Source: §5.20 · p. 43

6. Recommendations for Rule Learning Algorithms and Software

The section offers practitioner-facing ways to reduce bias when rule-based models are presented to users, drawing on conversational principles and cognitive-science findings. Recommendations address rule wording and measures, rule collections, user interaction, and education.

6.1. Adherence to conversational rules (maxims)

  • Table 2 links conversational maxims to biases they may help reduce, but the authors characterize the alignment as indicative rather than exclusive.The section emphasizes following Grice’s conversational rules when machine-learning explanations are communicated to general audiences.Source: Table 2 · p. 44, §6.1 · p. 44

6.2. Representation of a rule

  • Use unambiguous AND and IF–THEN wording, and review negation carefully because it can demand more effort and be forgotten.The cited research and the authors’ experiments support clarifying AND; the review suggests discouraging NOT where possible.Source: §6.2 · p. 44
  • Explain conditions and their predictive power, omit redundant or low-relevance conditions, order them by strength, and report numerical measures rather than verbal substitutes.The authors also suggest confidence intervals where applicable and note that natural frequencies may be easier to understand than probabilities or ratios.Source: §6.2 · p. 45

6.3. Rule models

  • Reduce unnecessary rule-model content through pruning or size controls, group similar rules to limit reiteration, and sort displayed rules by strength.The best definition of rule strength for mitigating bias remains unclear because applicable research is sparse.Source: §6.3 · pp. 45–46

6.4. User Engagement

  • User interfaces can expose conflicting domain knowledge, elicit checked annotations, allow rule-selection revision, and offer search instead of scrolling to limit repeated exposure.A diverse initial preview can be supported by clustering rules and displaying a representative from each cluster.Source: §6.4 · p. 47

6.5. Bias inoculation

  • Education in logic, hypothesis testing, and critical assessment can reduce fallacies in some tasks, but remains disputed and is not sufficient as a sole measure.Brief tutorials on particular biases have also reduced fallacy rates in some studies.Source: §6.5 · p. 47

7. Limitations and Future Work

The review maps possible limitations and research directions, particularly the need to validate cognitive-bias effects in machine-learning settings. It also calls for study of domain knowledge, user differences, debiasing trade-offs, phenomena beyond biases, and models beyond learned rules.

7.1. Validation through human-subject experiments

  • Bias effects in machine-learning interpretation need empirical validation: the review’s links from cognitive-science findings to rules rely largely on the authors’ machine-learning experience.Prior examples used questionnaires; the authors propose complementary observational studies in operational software and note that systematic evidence on effect size remains limited.Source: §7.1 · p. 48

7.2. Role of Domain Knowledge

  • Domain knowledge may help debias rule interpretation, and its relationship to bias activation or inhibition warrants systematic study.The review notes that familiarity with a topic area affects the presence or strength of the reiteration effect.Source: §7.2 · p. 49

7.3. Individual Differences

  • Bias susceptibility may vary with user traits and background, so further work on machine-learning users and personality assessment could help target debiasing education.The cited discussion connects some biases with numeracy, working memory, motivation, and personality; it also notes that confirmation bias is not necessarily related to general intelligence.Source: §7.3 · p. 49

7.4. Effectiveness and negative effects of debiasing techniques

  • Before deploying debiasing, establish that the targeted bias occurs and matters, verify improvement over baseline, and weigh effectiveness against cost and feasibility.The review notes that very few empirical studies address bias incidence in rule learning and almost none examine possible negative effects of debiasing strategies.Source: §7.4 · pp. 49–50

7.5. Extending Scope Beyond Biases

  • The review leaves human rule-induction phenomena beyond classified biases, including research on Wason’s 2-4-6 problem, for future investigation.It also identifies negation as a candidate for focused empirical work because studies outside machine learning indicate slower reasoning and possible forgetting.Source: §7.5 · p. 50

7.6. Extending Scope Beyond Inductively Learned Rules

  • The underlying concern may extend beyond inductively learned rules to other interpretable model forms and reasoning processes, but their specific requirements and users’ expertise need further study.The discussion names visual, structural, mathematical, and probabilistic models, as well as deduction, search-based planning, and Case-Based Reasoning.Source: §7.6 · pp. 50–51

8. Conclusion

The review connects cognitive-science research to the interpretation of learned rules, covering twenty biases, heuristics, and effects and proposing possible debiasing measures. It concludes that machine-learning-specific empirical evaluation remains a priority.

8. Conclusion

  • The review covers twenty cognitive biases, heuristics, and effects that can give rise to systematic errors when inductively learned rules are interpreted.The authors describe this as an initial systematic discussion of cognitive biases in relation to rule-learning interpretability.Source: §8 · p. 51
  • The authors transfer prior debiasing findings into proposed machine-learning measures, including frequency formats hypothesized to address confidence-over-support errors and misunderstanding AND.The proposed effect of frequency formats is a hypothesis, not an empirical result established by this review.Source: §8 · p. 51
  • Empirical evaluation of cognitive-bias effects in machine learning remains a particular priority, as the review found only very limited work specifically in this domain.Source: §8 · p. 51
Loading 1804.02969v7…