PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment

Jonathan Dodge, Q. Vera Liao, Yunfeng Zhang, Rachel K. E. Bellamy, Casey Dugan

arXiv:1901.07694v1cs.HCcs.CY

TL;DR

Fairness assessment of ML systems depends on human judgments, yet people need explanations that make fairness problems understandable and actionable. The paper empirically tests four explanation styles with a real recidivism-risk model and more than 160 Mechanical Turk workers. It finds that explanation effects vary with the fairness issue and users’ prior positions, motivating personalized or hybrid explanations.

  • Problem

    Human scrutiny is needed to identify and remedy ML fairness problems, but black-box systems and limited user-centered evidence make explanation design difficult.

  • Method

    The study programmatically generates four explanation styles for a real ML model and compares judgments across model-wide and case-specific fairness conditions.

  • Results

    Explanation styles affect perceived fairness and the exposure of fairness problems, while individual prior positions influence reactions to different explanations.

  • Takeaways & Limitations

    Effective fairness explanations have no one-size-fits-all design and may require hybrid, personalized approaches combining model overview with case scrutiny.

  • Takeaways & Limitations

    The study used crowdworkers rather than actual tool users and omitted explanation elements such as confidence.

Abstract

from arXiv · show

Ensuring fairness of machine learning systems is a human-in-the-loop process. It relies on developers, users, and the general public to identify fairness problems and make improvements. To facilitate the process we need effective, unbiased, and user-friendly explanations that people can confidently rely on. Towards that end, we conducted an empirical study with four types of programmatically generated explanations to understand how they impact people's fairness judgments of ML systems. With an experiment involving more than 160 Mechanical Turk workers, we show that: 1) Certain explanations are considered inherently less fair, while others can enhance people's confidence in the fairness of the algorithm; 2) Different fairness problems--such as model-wide fairness issues versus case-specific fairness discrepancies--may be more effectively exposed through different styles of explanation; 3) Individual differences, including prior positions and judgment criteria of algorithmic fairness, impact how people react to different styles of explanation. We conclude with a discussion on providing personalized and adaptive explanations to support fairness judgments of ML systems.

1 INTRODUCTION

As ML systems increasingly support consequential decisions, fairness scrutiny requires human judgment supported by effective explanations. This study examines how explanation styles and individual differences shape fairness judgments.

  • ML systems used in consequential domains can replicate or amplify historical biases, increasing concern about discriminatory outcomes.
  • Although researchers develop non-discriminatory algorithms, less attention has addressed the human scrutiny needed to identify and remedy fairness issues.
  • Black-box models make fairness problems difficult to identify because users often need information about decision-making processes, not only outputs.
  • Explanations can support human-in-the-loop fairness work by helping people identify issues and potentially increasing trust in the underlying system.
  • The paper empirically studies how explanation styles, fairness-problem types, and individual differences affect judgments, advocating user-centric and personalized approaches.

2 BACKGROUND

The paper builds on fairness and explainability research by studying how people perceive fairness and explanations together. It addresses prior work's limited attention to descriptive judgment, individual differences, and realistic ML settings.

  • Algorithmic fairness can involve discrimination when otherwise comparable individuals receive different decisions because of a sensitive attribute.
  • Unfair algorithms may arise from biased sampling, labeling, representation, optimization, distribution shifts, or omitted domain constraints.
  • The study uses a data preprocessor that transforms biased data while penalizing large feature changes and retaining the original features.
  • Unlike normative methods that optimize prespecified fairness criteria, the paper descriptively studies how people judge fairness and perceive normative debiasing.
  • XAI seeks to make complex systems more transparent, but explanations may be pedagogical rather than faithful representations of the underlying algorithm.
  • Prior work distinguishes global descriptions from local justifications and identifies soundness and completeness as explanation dimensions with a length-complexity tradeoff.
  • The paper extends prior work by automatically generating four explanation types for a real ML model and quantitatively testing their effects on fairness judgments.

3 STUDY OVERVIEW

The study tests how four explanation styles affect fairness judgments of a real recidivism-risk model. It compares model-wide and case-specific fairness issues while examining user characteristics and prior fairness positions.

  • The study asks whether explanation styles differ in perceived fairness and in their ability to expose model unfairness and case-level discrepancies.
  • It also examines how cognitive style and prior positions on ML decision support and race affect fairness judgments across explanations.
  • The experiment uses a real recidivism-risk dataset and treats the criminal-justice context as a probe of fairness judgments rather than studying actual COMPAS users.
  • The four programmatically generated explanations include global influence and demographic styles and local sensitivity and case-based styles.
  • The study focuses on racial discrimination and compares raw-data and debiased-data models to test whether explanations expose model-wide fairness differences.
  • A motivating case-level fairness issue occurs when changing a protected race variable changes a prediction for otherwise comparable cases near the decision boundary.

4 SYSTEM OVERVIEW

The system is a logistic-regression classifier trained on COMPAS data to predict reoffending, with race treated as the fairness-sensitive feature. The study evaluates its accuracy, debiasing, and disparate-impact cases before presenting explanations.

  • The binary classifier predicts whether a COMPAS individual will reoffend using Scikit-learn logistic regression.
  • The model uses race, age, charge degree, prior convictions, and juvenile-conviction status, with categorical features dummy coded against reference categories.
  • The data are split into 4,222 training samples and 1,056 testing samples.
  • 67.1% raw-data accuracy and 67.6% processed-data accuracy were comparable to reported COMPAS accuracy.
  • Debiasing reduced the African American feature coefficient from 0.177 to -0.036, with zero indicating no feature effect.
  • Disparate impact was identified when changing race changed a prediction; 23 raw-data cases met this criterion and were near the decision boundary.
  • The user study sampled all 8 unique impacted cases and 16 unique non-impacted cases because each participant saw only a small explanation sample.

4.2 Explanation Generation

The study generated four explanation styles programmatically for comparable ML predictions, including global descriptions of the model and training data and local explanations based on feature changes or similar cases.

  • Generation approach: The authors automatically generated explanation versions from Binns et al.’s templates to control representation and presentation across the same data points.The generation methods were designed for broad application to ML prediction models using relational features.
  • Global explanations: Influence-based explanations describe the logistic-regression decision boundary using discretized feature coefficients represented as strings of ‘+’ and ‘-’.Coefficients were discretized into 11 buckets, and the explanation is global because the boundary is shared across samples.
  • Global explanations: Demographic-based explanations summarize how training data matching each feature category is distributed relative to the predicted label.They report the percentage of matching training cases with the same label as the presented example and are global across samples on each boundary side.
  • Local explanations: Sensitivity-based explanations modify each presented sample feature-by-feature until the prediction changes, then report the feature change required to alter the output.This local explanation justifies an individual decision through the changes needed to produce a different result.
  • Local explanations: Case-based explanations use nearest-neighbor searches to show matching-label percentages for exact matches or the features and label of the nearest case.This local explanation was modified from prior work to accommodate exact matches in the study’s large training set.

5 METHODOLOGY

The study used a mixed-design online experiment in which 160 US Mechanical Turk workers judged fairness across cases, data-processing conditions, and four explanation styles while also reporting individual differences.

  • Design: Each participant completed 6 randomized fairness-judgment trials, including 2 disparately impacted and 4 non-impacted cases.Data processing and explanation style were between-subject variables, while disparate impact was manipulated within subjects.
  • Participants: 160 US Mechanical Turk workers were randomly assigned to 8 conditions combining 2 data-processing treatments and 4 explanation styles.Participants had completed more than 1000 tasks with at least a 98% approval rate.
  • Procedure: Participants reviewed each individual, predicted re-offense themselves, and then evaluated the ML prediction’s fairness in a criminal-justice bail-decision scenario.The preliminary prediction task was intended to ensure careful consideration of the features used by the model.
  • Measures: The study measured prior positions on general ML decision support, race as a feature, and need for cognition as individual-difference factors.Prior positions were measured with averaged semantic-differential ratings, while need for cognition used four items from the standard scale.

6 RESULTS: QUANTITATIVE

Fairness judgments differed by data processing, disparate impact, explanation style, and participants’ prior positions. Local explanations better exposed case-specific discrepancies, while sensitivity-based effects depended on views about race and ML fairness.

  • Explanation, data processing, and disparate impact: Sensitivity-based explanations most prominently exposed disparate impact in raw data, but the corresponding three-way interaction was not statistically significant before accounting for individual differences.The authors identify unmeasured individual differences as one possible explanation for the nonsignificant interaction.
  • Individual differences: After adding individual differences, the explanation-style-by-processing-by-impact interaction became significant, F(3, 144) = 2.96, p = 0.03.The expanded model also found significant main effects of ML position, race position, data processing, and disparate impact.
  • Individual differences: Sensitivity-based explanations lowered perceived fairness mainly among participants who considered using race unfair, whereas race-fair or neutral participants did not rate raw-data predictions less fairly than processed-data predictions.Race-fair or neutral participants comprised 33.1% of the sample.
  • Individual differences: Case-based explanations were rated marginally less fair than influence-based and demographic-based explanations, while global explanations further increased confidence among people who trusted ML systems.The case-based comparisons were F(1, 153) = 3.51, p = 0.06, and F(1, 148) = 3.20, p = 0.08.

7 RESULTS: QUALITATIVE

The qualitative analysis examined participants’ written justifications to understand how they made fairness judgments and how they responded to the four explanation styles.

  • Qualitative analysis: The authors open-coded participants’ fairness-judgment justifications into themes about reasoning processes and feedback on the four explanation styles.These two theme groups address how participants judged fairness and how they evaluated the explanations.

7.1 How is fairness judgment made?

Participants judged fairness through varied criteria, including trust in ML, algorithmic features, process understanding, and perceived data quality. Individual positions on ML use and fairness shaped these judgments.

  • Participants varied in reliance on explanations and depth of reasoning about algorithmic processes when judging fairness.Some relied on general trust or distrust of ML systems and gave consistent ratings across cases.
  • General trust or distrust of ML systems sometimes dominated fairness judgments across cases.Some participants valued objective data and large datasets, while others preferred human judgment.
  • Participants frequently evaluated fairness based on the features used by the algorithm.Some separated confidence in the algorithmic process from concerns about features such as age and race.
  • Participants criticized limited features and questioned whether additional information could ensure fair assessments.Suggestions included more detailed prior-record information, while others doubted that software could capture relevant environmental factors.
  • Many participants based fairness judgments on their understanding of the algorithm’s process and how data were used.Global explanations prompted scrutiny of feature weights, probability calculations, historical context, and process transparency.
  • Participants in demographic- or case-based conditions sometimes questioned whether the underlying data were biased.These explanation styles rely on distributions of similar cases, which directed attention toward data quality.

7.2 Explanation styles

Explanation styles prompted different forms of scrutiny and confidence. Global process explanations supported detailed evaluation, sensitivity explanations were concise but exposed controversial features, and case-based explanations were often judged least fair.

  • Influence based: Influence-based explanations describe how each feature contributes to the algorithm’s decision-making process.Their detail increased confidence for some participants but also exposed more information to heterogeneous fairness standards.
  • Influence based: Distribution-based global explanations justify decisions with data distributions without exposing the algorithm’s process.Participants sometimes found these distributions convincing, but others considered them insufficiently connected to outcomes and noticed potential data bias.
  • Sensitivity based: Sensitivity-based explanations were concise and directed attention to features relevant to a particular decision.They were easy to process for uncontroversial decisions but heightened concern when race was explicitly implicated in disparate impact.
  • Case based: Case-based explanations were judged the least fair because they offered little process information and often relied on small or weakly supportive comparison groups.Participants also differed on whether comparisons with identical cases constituted a fair decision process.

8 DISCUSSION

The discussion emphasizes that fairness judgments are heterogeneous and concern both unfair models and case-specific discrepancies. It argues for explanation styles tailored to fairness problems and user positions, while noting important study limitations.

  • 8.1 Supporting different needs of fairness judgment: Fairness judgments involve multiple dimensions beyond evaluating model features, including unfair models and discrepancies among cases.The study highlights models learned from biased data and fairness differences across feature-space regions.
  • 8.1 Supporting different needs of fairness judgment: Local explanations are more effective for exposing case-level fairness discrepancies, whereas global explanations increase confidence in understanding the model.The authors propose combining global model evaluation with local scrutiny of individual cases.
  • 8.1 Supporting different needs of fairness judgment: Different feature-space regions may exhibit different fairness levels and types of fairness issues.The authors call for fine-grained sampling and explanation designs, including interactive fairness-improvement workflows.
  • 8.1 Supporting different needs of fairness judgment: Process-oriented explanations generally received preference, while data-oriented explanations can justify decisions through distributions without revealing how decisions were made.Influence- and sensitivity-based explanations connect features to decisions, whereas case- and demographic-based explanations emphasize data distributions.
  • 8.2 Individual differences and descriptive fairness: Participants’ prior positions on algorithmic fairness influenced their reactions to explanation types.The study distinguishes general fairness positions from positions about the fairness of particular features.
  • 8.1 Supporting different needs of fairness judgment: 16% of test data exhibited contradictory case-based claims, while 24% had insufficient justification from near-balanced label matches.These frequencies indicate inherent unsoundness in the case-based explanation failure modes studied.
  • 8.2 Individual differences and descriptive fairness: Close to one third of participants considered race neutral or fair to use, showing variation in positions on a protected variable.The authors connect this variation to the distinction between normative and descriptive fairness.
  • 8.3 Limitations: The study used crowdworkers rather than judges who would be actual users, and it omitted several explanation elements such as confidence.These choices limit direct generalization to the intended users and unexplored explanation styles.

9 CONCLUSION

The study finds that explanation effectiveness depends on fairness issues and user profiles rather than following a one-size-fits-all solution. It supports hybrid, personalized explanations that combine model overview with case scrutiny.

  • Explanation effectiveness depends on the fairness issue and user profile rather than following a one-size-fits-all solution.The conclusion emphasizes differences between global model explanations and local decision justifications.
  • Hybrid explanations may support accurate fairness judgment by combining model overview with scrutiny of individual cases.The conclusion links this design to human-in-the-loop fairness assessment.
  • Individuals’ prior positions on algorithmic fairness influence how they react to explanation types.The authors therefore call for a personalized approach to explaining ML systems.
Loading 1901.07694v1…