Source-linked AI summary
On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
Vivian Lai, Chenhao Tan
TL;DR
Critical decision tasks may benefit from machine learning but are not always suitable for full automation, making human performance and agency important research concerns. This paper studies explanations and predictions in deceptive-review detection using assistance levels spanning human agency to automation, finding that predicted labels substantially improve accuracy and explanations can moderate the performance–agency tradeoff.
Problem
Machine learning can perform well in critical tasks, but evidence is needed on how its assistance affects human performance and human agency when humans remain final decision makers.
Method
The paper uses deceptive-review detection as a testbed and evaluates human decisions under progressively stronger assistance involving explanations, predicted labels, and machine-accuracy statements.
Results
∼21% relative improvement in human accuracy occurs with predicted labels, while explicit strong-accuracy priming yields ∼46% relative improvement; explanations with labels achieve a similar effect to strong-accuracy statements.
Takeaways & Limitations
Explanations of machine predictions may moderate the tradeoff between improved human performance and retained human agency.
Takeaways & Limitations
The participant sample from Amazon Mechanical Turk may not represent the broader population, and the explanations come from a linear SVM classifier and nearest neighbors.
Abstract
from arXiv · showhide
Humans are the final decision makers in critical tasks that involve ethical and legal concerns, ranging from recidivism prediction, to medical diagnosis, to fighting against fake news. Although machine learning models can sometimes achieve impressive performance in these tasks, these tasks are not amenable to full automation. To realize the potential of machine learning for improving human decisions, it is important to understand how assistance from machine learning models affects human performance and human agency. In this paper, we use deception detection as a testbed and investigate how we can harness explanations and predictions of machine learning models to improve human performance while retaining human agency. We propose a spectrum between full human agency and full automation, and develop varying levels of machine assistance along the spectrum that gradually increase the influence of machine predictions. We find that without showing predicted labels, explanations alone slightly improve human performance in the end task. In comparison, human performance is greatly improved by showing predicted labels (>20% relative improvement) and can be further improved by explicitly suggesting strong machine performance. Interestingly, when predicted labels are shown, explanations of machine predictions induce a similar level of accuracy as an explicit statement of strong machine performance. Our results demonstrate a tradeoff between human performance and human agency and show that explanations of machine predictions can moderate this tradeoff.
1 INTRODUCTION
The paper frames machine learning assistance as a spectrum from human agency to automation, using deception detection to study how explanations and predictions affect human decision-making. It reports that assistance can improve performance, but stronger machine influence creates a tradeoff with human agency.
- Motivation: Machine learning can support decisions in ethically and legally sensitive tasks where full automation is often undesirable.Machines may identify patterns that are unsalient, unknown, or counterintuitive to humans, while humans retain final decision authority.
- Research design: The authors propose assistance levels spanning full human agency to full automation, including explanations alone, predicted labels, and labels paired with strong-accuracy statements.These presentations progressively increase the influence of machine predictions while humans continue making final decisions.
- Testbed: The study uses genuine-versus-deceptive hotel-review classification as a constrained testbed for human predictions with machine assistance.The task is based on textual information and isolates how humans interact with machine predictions.
- Findings: ∼21% relative improvement in human accuracy occurs when predicted labels are shown, compared with slight improvement from explanations alone.Explicit machine-accuracy priming produces ∼46% relative improvement, exceeding the improvement from predicted labels alone.
- Findings: Explanations paired with predicted labels achieve a similar effect to an explicit statement of strong machine accuracy.The findings suggest explanations can moderate the tradeoff between human performance and human agency.
- Findings: Participants are more likely to trust machine predictions when an accuracy statement is provided, even when the statement suggests poor machine performance.Frequency explanations can help humans calibrate accuracy numbers, but the authors do not recommend untruthful presentations.
2 RELATED WORK
The related work situates the study at the intersection of interpretable machine learning, deception detection, and misinformation research. It emphasizes direct evaluation of human performance on a difficult prediction task and limits the empirical setting to deceptive review detection.
- Interpretable machine learning: Interpretable machine learning seeks to improve transparency because blindly following black-box predictions can have serious consequences in domains such as medicine and justice.The literature includes example-based and feature-based explanation approaches.
- Positioning: The paper differs from much interpretability research by evaluating whether explanations improve human performance in the end task.Prior work often evaluates model understanding, model debugging, or machine performance instead.
- Deception detection: Deception detection in online reviews is difficult because reliable ground-truth labels are challenging to obtain.Ott et al. created a sizable dataset by asking Mechanical Turk workers to write deceptive reviews.
- Scope: Fake-news detection differs from deceptive-review classification because it may require skills such as fact checking and involves difficult class definitions.The paper therefore treats textual genuine-versus-deceptive review classification as a sandbox for studying human interaction with machine assistance.
3 EXPERIMENTAL SETUP AND HYPOTHESES
The study evaluates human deception-detection performance under progressively stronger forms of machine assistance, from explanations to predicted labels and accuracy information. It measures both human accuracy and trust in machine predictions.
- Experimental setup: The user study tests human judgments of genuine versus deceptive hotel reviews under different machine-assistance conditions.Turkers label reviews from a held-out test set in a between-subject design.
- Varying levels of machine assistance: Control provides only the review, preserving full human agency without machine information.
- Varying levels of machine assistance: Feature-based explanations highlight influential words using either uniform highlighting or a heatmap of model-weight magnitudes.The model is linear, and the heatmap uses darker blue for more heavily weighted words without indicating weight sign.
- Varying levels of machine assistance: Predicted-label assistance reveals the machine’s label, while stronger priming additionally reports machine accuracy of 87%.The interface variants differ in whether they show explanations, predicted labels, and explicit machine-performance information.
- Evaluation metrics: The evaluation measures human accuracy and trust, with trust defined as the percentage of instances in which humans follow the machine prediction.Trust can be computed only when predicted labels are available.
4 RESULTS
The study evaluates how assistance levels affect human deception-detection accuracy, trust, and individual differences. Predicted labels substantially improve performance, while explanations can improve trust and moderate reliance on machine accuracy statements.
- Trust: Humans trusted correct machine predictions more than incorrect ones in all five setups, but the difference was statistically significant in only two.This breakdown distinguishes trust in machine predictions from overall human accuracy.
- Human Accuracy: Showing predicted labels is crucial for improving human performance, whereas explanations alone only slightly improve accuracy.Control performance was 51.1%, no better than chance; explanation-based improvements were significant for highlight and heatmap, but not examples.
- Human Accuracy: 74.6% accuracy followed an explicit statement of strong machine accuracy, representing a 46% relative improvement over control.The reported comparison is predicted label w/ accuracy versus the 51.1% control condition.
- Human Accuracy: 69.7% and 72.5% accuracy for predicted label + examples and predicted label + heatmap exceeded 61.9% for predicted label w/o accuracy.Heatmap performance was comparable to predicted label w/ accuracy, although human performance remained below machine performance of 87.0%.
- Trust: Feature- and example-based explanations increase trust in machine predictions, with predicted label + heatmap producing trust similar to predicted label w/ accuracy.Random heatmaps also significantly increased trust, suggesting that explanations can influence reliance without explicitly stating strong machine accuracy.
- Heterogeneity in Human Perception and Performance: Approximately 42% of participants correctly estimated their performance, while 18% overestimated and 40% underestimated it.Participants generally found it difficult to estimate their own performance.
- Heterogeneity in Human Perception and Performance: Participants who found hints useful outperformed those who did not in 5 of 8 setups, while females generally outperformed males in 8 of 9 setups.The hint-usefulness difference was statistically significant in the predicted label + heatmap group; gender differences were not statistically significant.
5 VARYING STATEMENTS OF MACHINE ACCURACY
The study varies stated machine accuracy while holding machine predictions unchanged, then tests whether frequency explanations alter human accuracy and trust. Lower stated accuracy generally reduces both measures, and frequency explanations make 50% accuracy resemble showing no accuracy statement.
- Varying Statements of Machine Accuracy: Human accuracy and trust generally decline as stated machine accuracy decreases, despite unchanged machine predictions.The authors separate these experiments because the accuracy statements are not true and do not recommend this manipulation as part of their assistance spectrum.
- Frequency Explanations: Frequency explanations reduce human accuracy by approximately 7% and trust by approximately 10% for the 50% accuracy statement.The accuracy difference had p=0.003, while the trust difference had p<0.001.
- Frequency Explanations: With frequency explanations, the 50% accuracy condition did not differ significantly from predicted label w/o accuracy on either human accuracy or trust.The authors conclude that frequency explanations can make a 50% accuracy statement function similarly to showing no machine accuracy.
6 CONCLUDING DISCUSSION
The paper finds that machine assistance can improve human deception-detection performance, but stronger assistance creates a tradeoff with human agency. It also highlights concerns about trust, explanation design, ethical consequences, and generalizability.
- Main findings: Explanations alone slightly improve human performance, while predicted labels significantly improve it; explicitly stating strong machine performance improves it further.The authors frame these assistance levels as varying degrees of machine priming.
- Agency: Explanations may moderate the tradeoff between improved human performance and retained human agency.The paper treats this as an initial empirical step toward understanding human interaction with machine assistance.
- Trust: Human trust tracks machine correctness, but irrelevant explanations or low-accuracy statements can also increase trust.The authors caution that additional details do not necessarily justify trusting machine predictions.
- Future directions: Limited resources for teaching explanations may help explain why explanations alone produced only modest gains.Suggested directions include tutorials, rules, examples, and training sequences with explanations.
- Future directions: Narrative rationales are proposed as a way to make feature-based and example-based explanations more effective.Qualitative study of how participants interpret hints could clarify requirements for useful narratives.
- Ethics: Providing predicted labels, especially with machine-performance information, risks reducing human agency and raises ethical concerns.The paper calls for input from machine learning researchers, legal scholars, and society.
- Limitations: The study’s Amazon Mechanical Turk sample may not represent the broader population, and its explanations come from linear SVM and nearest-neighbor models.The authors note that explanations from nonlinear classifiers may be more difficult to interpret.
- Limitations: Generalizability is difficult to assess because the testbed involves machines substantially outperforming humans and possible false beliefs about deception detection.The authors leave open whether a principled framework can compare tasks such as deception detection, bail decisions, and medical diagnosis.
A.1 Amazon Mechanical Turk Setup
The Mechanical Turk study applied participant-quality criteria and required training before the main deception-detection task and exit survey.
- Recruitment: Participants had to be U.S.-based, complete at least 50 HITs, and maintain an approval rate of at least 99%.These criteria were intended to support English fluency and participant quality.
- Training: Training showed three reviews with feedback after each prediction, using the same interface as the experiment.Participants then labeled 20 reviews before completing an exit survey.
- Survey: The exit survey collected self-estimated performance and demographic information, including age, gender, education, and online-review experience.Survey screenshots appear in Figures 15 and 16.
A.2 Experiment Interfaces
The appendix documents additional interfaces and study visualizations for the experimental setups, including duration and review-ordering figures.
- Experiment interfaces: Additional interfaces cover control, highlight, examples, predicted label without accuracy, and predicted label plus examples.Predicted label plus random heatmap uses the same interface as predicted label plus heatmap, with randomly highlighted words.
- Study duration: Figure 7 presents the cumulative distribution of study duration across nine experimental setups.The supplied caption identifies the setup count but does not state a comparative outcome.
- Review ordering: Figure 8 presents average accuracy with respect to review ordering across nine experimental setups.The supplied caption identifies the comparison dimensions but does not state a comparative outcome.
A.3 Individual Differences
Individual-difference analyses examine hint usefulness, gender, education, age, and review-writing experience across experimental setups. The clearest reported pattern is that females generally outperform males, although no gender difference is statistically significant.
- Hint usefulness: Participants who found hints useful performed better in 5 of 8 experimental setups.This pattern occurred across representative assistance conditions including heatmap, examples, and predicted-label setups.
- Gender differences: Females generally outperform males in 8 of 9 experimental setups, but none of the differences is statistically significant.The comparison is reported in Figure 10.
- Education background: Education level shows no clear relationship with deception-detection ability.High school graduates perform best in predicted label without accuracy but worst in examples; groups were sparse, so no statistical testing was conducted.
- Age group: Age groups show no clear performance trend.Participants aged 61 and above perform best with predicted label without accuracy but worst with predicted label plus heatmap; no statistical testing was conducted.
- Review experience: Review-writing experience shows no clear overall trend.Except in control and predicted label plus random heatmap, the best-performing group is usually weekly or frequent reviewers; no statistical testing was conducted.