PaperlayerView exact revision on arXiv

Source-linked AI summary

Explanation-Based Human Debugging of NLP Models: A Survey

Piyawat Lertvittayakumjorn, Francesca Toni

arXiv:2104.15135v3cs.CLcs.AIcs.HCcs.LG

TL;DR

Debugging NLP models is difficult when errors arise from data, learning processes, or opaque model behavior. This survey defines EBHD, organizes prior work across context, workflow, and experimental setting, and synthesizes human-factor findings and open challenges. It concludes that feedback quality, trust, realistic bugs, and deployment remain central concerns.

  • Problem

    The central problem is how to use human knowledge to fix NLP model bugs through feedback on explanations.

    Source: §1 · p. 1

  • Method

    The survey defines EBHD, presents a general framework, categorizes existing studies across three dimensions, and reviews human factors and open problems.

    Source: §1 · p. 1, §2.2.3 · p. 8, §5 · p. 12

  • Results

    EBHD research reports that human feedback can support model improvement, but effective debugging depends on explanation understanding, feedback quality, and suitable update processes.

    Source: §1 · p. 3, §3.1 · p. 9, §3.6 · p. 11

  • Takeaways & Limitations

    EBHD should account for feedback providers’ imperfect and selective judgments, while matching provider roles and interaction strategies to trust and domain-knowledge constraints.

    Source: §3.2 · p. 10, §3.6 · p. 11

    Existing evaluations often use easy or unrealistic bugs, and EBHD has not yet been widely deployed outside research environments.

    Source: §4.2 · p. 11

Abstract

from arXiv · show

Debugging a machine learning model is hard since the bug usually involves the training data and the learning process. This becomes even harder for an opaque deep learning model if we have no clue about how the model actually works. In this survey, we review papers that exploit explanations to enable humans to give feedback and debug NLP models. We call this problem explanation-based human debugging (EBHD). In particular, we categorize and discuss existing work along three dimensions of EBHD (the bug context, the workflow, and the experimental setting), compile findings on how EBHD components affect the feedback providers, and highlight open problems that could be future research directions.

1 Introduction

The survey defines explanation-based human debugging (EBHD) as using human feedback on model explanations to fix or mitigate NLP model bugs, connecting explainability, human-in-the-loop learning, and knowledge integration. It introduces a general framework and reviews prior work, human factors, and open challenges.

Motivation

  • EBHD uses human feedback on explanations to fix or mitigate bugs in trained NLP models, including spurious correlations, labeling errors, and undesirable out-of-distribution behavior.The survey distinguishes this from debugging that only identifies error causes.Source: §1 · pp. 1–2

Contribution

  • The survey is the first overview to connect explainability, interactive human-in-the-loop learning, and knowledge integration for NLP model debugging.Source: §1 · p. 1

Survey Scope

  • The survey organizes prior work by EBHD choices, examines human factors affecting debugging and provider satisfaction, and identifies open research challenges.Source: §1 · pp. 2–3

General Framework

  • EBHD follows three steps: provide explanations, collect human feedback, and use that feedback to update the model, either once or iteratively.The framework includes the inspected model, feedback providers, and workflow components.Source: §1 · pp. 2–3, Figure 1 · p. 2

Illustrative Example

  • Ribeiro et al. (2016) used LIME word-relevance explanations and crowdsourced feedback to remove artifact words from training data, retraining an SVM for three rounds with improved generalization after each round.The example applies EBHD to 20Newsgroups, whose artifacts could cause incorrect model associations.Source: Figure 2 · p. 2, §1 · p. 3

Human Roles

  • Feedback providers may be developers, domain experts, or end users, and EBHD can occur before deployment or through automated post-deployment updates.The survey emphasizes trust, frustration, and expectations when end users provide feedback.Source: §1 · p. 3

2 Categorization of Existing Work

The survey categorizes existing EBHD studies by debugging context, workflow, and experimental setting to characterize the field’s current state.

Categorization

  • Existing EBHD studies are compared across debugging context, workflow, and experimental setting.Context covers tasks, models, and bug sources; workflow covers the three framework steps; setting covers human engagement.Source: §2 · p. 3

2.1 Context

The surveyed EBHD literature is concentrated in text classification and traditional models, often using artifact-driven bugs, while several challenging tasks and modern settings remain underexplored.

2.1 Context

  • The EBHD context comprises the target task, inspected model, and bug source that the debugging process aims to address.Source: §2.1 · pp. 3–4

2.1.1 Tasks

  • Most studies address single-input text classification, while NLI, visual question answering, and table question answering appear less often.Text-classification applications include email, topic, spam, sentiment, and transcript-coding tasks.Source: §2.1.1 · pp. 4–5
  • Text classification is common because lay participants can more easily understand its explanations and provide feedback, whereas other tasks may require linguistic expertise.Authorship attribution and deceptive-review detection are exceptions that are also unsuitable for lay-participant EBHD.Source: §2.1.1 · p. 4

2.1.2 Models

  • Early studies used interpretable bag-of-words Naive Bayes, logistic regression, and SVM models, while only two surveyed papers tested pretrained language models such as BERT and RoBERTa.Other studies examined CNNs, bidirectional LSTMs, fastText, and attention-based question-answering models.Source: §2.1.2 · p. 5

2.1.3 Bug Sources

  • Natural artifacts are the dominant bug source, producing spurious correlations that undermine generalizability; other studies simulate bugs with small training subsets, wrong labels, or out-of-distribution tests.Injected decoy artifacts are proposed as a further NLP direction based on related computer-vision work.Source: §2.1.3 · p. 5

2.2 Workflow

EBHD workflows use explanations to expose potential NLP model bugs, collect human feedback, and update models, with choices about explanation scope, feedback form, update mechanism, and iteration shaping the process.

2.2 Workflow

  • Explanations provide interpretable insights that can reveal model misbehavior missed by outputs or evaluation metrics.Source: §2.2.1 · p. 5

2.2.1 Providing Explanations

  • Local explanations target individual predictions, whereas global explanations describe the model overall; global explanations may miss less-obvious bugs in complex models.FIND worked more effectively on significant gender bias than on less-obvious dataset shift when using global explanations.Source: §2.2.1 · pp. 5–6
  • Local explanations can burden feedback providers, motivating example selection or ranking, while grouped explanations offer an unexplored middle ground between local and global scopes.Existing selection strategies include targeting incorrect predictions, choosing non-redundant examples, and maximizing informativeness.Source: §2.2.1 · p. 6
  • EBHD uses input-, example-, rule-, or adversarial-based explanations generated either inherently by transparent models or post hoc through model analysis or perturbation.Input-based explanations are most common, followed by example-based explanations.Source: §2.2.1 · pp. 6–7

2.2.2 Collecting Feedback

  • Feedback may identify relevant words, adjust importance scores, validate labels, assess features or rules, or provide attention and compositional reasoning signals.Compositional feedback can express relations among words and requires more expertise from feedback providers.Source: §2.2.2 · pp. 7–8

2.2.3 Updating the Model

  • Human feedback updates models by directly adjusting parameters, improving training data, or influencing retraining through model-specific or model-agnostic techniques.Parameter adjustment is fast and avoids retraining but should be reversible and checked for generalization; training-data changes apply regardless of model complexity.Source: §2.2.3 · p. 8

2.2.4 Iteration

  • Iterative explain-feedback-update cycles can address vital bugs before finer ones, but local improvements may accumulate into inferior overall performance.Each update should be checked so it does not overwrite beneficial effects from earlier iterations.Source: §2.2.4 · p. 8

2.3 Experimental Setting

EBHD experiments use in-person participants, crowdsourcing, or simulation, trading off behavioral insight, scale, response quality, cost, and realism.

2.3 Experimental Setting

  • In-person studies closely observe participant behavior but typically involve fewer than 100 participants.Source: §2.3 · pp. 8–9
  • Crowdsourcing enables large-scale feedback collection but requires quality controls because response quality can vary.Controls include qualifications, multiple annotations, participant training, and attention checks.Source: §2.3 · p. 9
  • Simulation is faster and cheaper than studies with real humans but may not reflect deployment effectiveness or capture inaccurate, noisy, interrupted, or frustrated feedback.Simulated studies therefore cannot thoroughly examine these human factors.Source: §2.3 · p. 9

3 Research on Human Factors

Human factors shape whether explanation-based human debugging succeeds: explanations must support accurate understanding without overload, feedback must be useful despite being incomplete, and workflows must manage trust, frustration, and expectations.

3.1 Model Understanding

  • Explanation formats differ in how well they support model understanding: rule- and keyword-based, why-oriented, and interactive explanations generally outperform similarity-based, why-not, or static alternatives, though interactivity and model internals can add cognitive workload.Users may misunderstand why absent words increase confidence, while exposing internals can make them doubt their understanding despite providing more information.Source: §3.1 · p. 9

3.2 Willingness

  • Human feedback can extend beyond labels to feature edits, weight tuning, and feature combinations, drawing mainly on commonsense and English-language knowledge.Participants considered label-only feedback inadequate for injecting this background knowledge into models.Source: §3.2 · pp. 9–10
  • Human feedback is useful but selective and imprecise: people focus on features that violate expectations and, especially as lay users, struggle to correct explanations quantitatively.Human explanations more often express causal relationships than probabilities.Source: §3.2 · p. 10

3.3 Trust

  • Explanations do not reliably increase trust: they can expose low-quality models or wrong reasons, sometimes warn users about faults, and feedback can either restore or reduce trust depending on the study.Explanations may calibrate trust to model quality, but subtle flaws remain difficult for users to detect.Source: §3.3 · p. 10

3.4 Frustration

  • Poor-model explanations can increase frustration, while allowing feedback reduces it; excessive detail can overload users and undermine their ability to detect model flaws.The reviewed recommendation is to reveal model internals on request rather than expose everything by default.Source: §3.4 · p. 10

3.5 Expectation

  • Users may expect improvement after interacting with a model regardless of whether they saw explanations or gave feedback, so systems should communicate updates and preferably show changes incrementally in real time.Incremental updates let feedback providers check whether their feedback had the expected effect.Source: §3.5 · pp. 10–11, §3.6 · p. 11

3.6 Summary

  • The survey recommends matching feedback collection to stakeholders, using understandable and request-driven explanations, checking feedback before model updates, and communicating improvements after updating.Collective feedback and provider verification can mitigate feedback that is incomplete, incorrect, or inaccurate.Source: §3.6 · p. 11

4 Open Problems

The survey identifies open problems spanning EBHD’s language and task coverage, bug realism, efficiency, evaluation reliability, and deployment. It calls for broader settings, more challenging bugs, end-to-end efficiency analysis, standardized comparisons, and evaluation with real users.

4.1 Beyond English Text Classification

  • EBHD studies have used only English datasets, motivating research beyond English despite the practical difficulty of multilingual explanation analysis and participant recruitment.The survey points to multilingual data availability and growing NLP-community awareness as factors that may support future work.Source: §4.1 · p. 11
  • Most EBHD work targets text classification, but other tasks require task-specific debugging methods because NLI bugs can involve syntactic heuristics rather than word-level artifacts.The survey specifically names reading comprehension, question answering, and natural language inference as underexplored tasks.Source: §4.1 · p. 11

4.2 Tackling More Challenging Bugs

  • Existing evaluations often use obvious, easily preprocessed artifacts, leaving EBHD’s effectiveness on infrequent real-world bugs uncertain and potentially making local explanations inefficient.When models are already reasonably good, feedback may produce only minor improvements.Source: §4.2 · p. 11
  • Open problems include reconciling conflicting human feedback, injecting new knowledge into opaque models, and transferring EBHD techniques across modalities.The survey distinguishes confirming or removing learned information from adding knowledge that may not appear in explanations.Source: §4.2 · pp. 11–12

4.3 Analyzing and Enhancing Efficiency

  • No selected study evaluates the efficiency of explanation generation, human feedback, and model updating together, especially for costly black-box and iterative settings.Prior work mainly measures explanation-generation or human-feedback efficiency, while model-retraining costs remain insufficiently analyzed.Source: §4.3 · p. 12

4.4 Reliable Comparison across Papers

  • A standard benchmark is needed because user-study variation and limited HCI practice make cross-study comparisons of explanation types and feedback mechanisms unreliable.Most existing work compares pre- and post-debugging performance or configurations within a single framework.Source: §4.4 · p. 12

4.5 Towards Deployment

  • Deployment remains limited, motivating integration with visualization systems, adherence to human-AI interaction guidelines, and evaluation with potential end users rather than only simulations or crowdsourcing.The survey identifies human factors as important in real deployment settings.Source: §4.5 · p. 12

5 Conclusion

The survey presents EBHD as a framework for analyzing and improving NLP models through explanations and human feedback. It synthesizes existing work, human-factor findings, design practices, and open problems to guide future research.

5 Conclusion

  • The survey presents a general EBHD framework for NLP models and analyzes existing studies through its components to characterize the field’s state of the art.It also summarizes human-factor findings, suggests design practices, and identifies open problems for future studies.Source: §5 · p. 12
  • The survey aims to guide researchers and support examination of future EBHD studies as the research area continues developing.Source: §5 · p. 12

Related papers

Loading 2104.15135v3…