Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning

1 INTRODUCTION

• The paper situates its proposal within a debate over interpretability’s scope, noting that ambiguity complicates evaluating and regulating opaque systems.It also describes its examination of automated healthcare decision-making as an application of the approach.Source: §1 · pp. 1–2
• Model-centric explanations may identify past healthcare costs as important without explaining why a readmission algorithm underestimates risk for a racial group.The paper’s example links lower historical costs to systemic inequities and unmet healthcare needs, not necessarily better health.Source: §1 · p. 2
• Lower costs can reflect inadequate access to care and unmet needs, which may lead to higher readmission rates the algorithm does not account for.The paper argues that interpreting the prediction requires considering both model-specific features and healthcare affordability disparities.Source: §1 · p. 2
• The paper proposes socio-structural explanations, drawing on social philosophy to extend model-centric interpretation in domains with substantial societal impact.It argues that, in high-stakes domains, socio-structural analysis could be necessary to understand outputs and uncover societal biases.Source: §1 · p. 2

2 INTERPRETABLE ML AND ITS DISCONTENTS

• The section frames explanations as elements that help make a model’s prediction or decision understandable to humans.It presents the model output as the explanandum and visualizations or linguistic descriptions as the explanans.Source: §2 · p. 2
• Intrinsic interpretability limits model complexity, whereas post-hoc methods are applied after training.Short decision trees and rule-based systems are examples of intrinsic approaches; SHAP is listed among post-hoc methods.Source: §2 · p. 2
• Local explanations address individual predictions or decisions, often using feature attribution or counterfactual instances.The section illustrates these approaches with pixel attributions for image classification and a loan-approval counterfactual.Source: §2 · p. 3
• Global explanations describe average model behavior across possible inputs, using methods such as partial dependence plots and accumulated local effects.They can help users understand general mechanisms in the data, debug a model, or assess behavior across scenarios.Source: §2 · p. 3
• Mechanistic interpretability seeks to reverse-engineer neural-network algorithms by examining weights and activations.The section mentions circuit analysis and dictionary learning to identify subnetworks implementing particular behaviors.Source: §2 · p. 3
• These approaches offer different perspectives, and the choice of method depends on the interpretability goal and the model being analyzed.The section describes a landscape ranging from instance-level explanations to overarching principles and computational mechanisms.Source: §2 · p. 3

3 SOCIAL STRUCTURES AND SOCIO-STRUCTURAL EXPLANATIONS

• Garfinkel’s curved-grading example shows why Mary’s thoughtful paper alone does not explain her sole A: her paper was best relative to peers under a curve limiting As to one [19, p. 41].The explanation combines the structural grading rule with Mary’s performance relative to classmates.Source: §3 · p. 5
• Social structures—such as institutional rules, routines, resource systems, and cultural norms—shape individual action by both enabling and constraining it [67, p.111].They can help explain persistent patterns by situating individual outcomes within broader systems of interconnected social forces.Source: §3 · pp. 5–6
• In an ML-powered hiring example, SHAP says the model recommends male X over female Y because X’s 10 years of tech leadership adds +0.4, versus Y’s 7 years adding +0.2.Similar explanations for a population of females might fail to capture the full picture.Source: §3 · p. 6
• The model’s historical training data—85% male executives from 2000–2020—reflects past hiring practices and systemic exclusion that can perpetuate male dominance in leadership [8].The socio-structural account connects the lack of female representation to norms and institutional practices that reinforce perceptions of leadership as suited to men.Source: §3 · pp. 6–7
• The model reduces a candidate’s score by 0.1 per year for gaps longer than 6 months, while women’s disproportionate family-care burden can lead to more frequent and longer interruptions.In the example, 40% of female candidates had career gaps averaging 2.5 years.Source: §3 · p. 7
• By overlooking geography, the model also misses how family-care expectations can constrain women’s job options by proximity to home and need for flexible hours.The passage notes that these constraints can disadvantage highly qualified women.Source: §3 · p. 7

• Possible interventions include capping the contribution of continuous experience, valuing diverse career paths, adding profiles of successful executives with career gaps, and requiring human review for certain lower-ranked candidates.The proposed review policy applies when career gaps primarily drive a ranking difference greater than 0.2; the training-data example adds 500 profiles, at least 50% women.Source: §3 · p. 7
• Integrating socio-structural explanations into ML transparency can move beyond superficial model-centric solutions and address fundamental causes underlying ML outputs.Source: §3 · p. 7
• The interventions and their effectiveness depend on the problem context, social-system complexity, available data quality, and analytic goals; socio-structural explanations are not always applicable.Source: §3 · p. 7
• The importance of socio-structural explanations extends beyond ML systems to a broader range of automated decision-making.Source: §3 · p. 7

4 SOCIO-STRUCTURAL EXPLANATIONS OF RACIAL BIAS IN HEALTH-CARE ALGORITHMS

• The algorithm assigned Black patients lower risk scores despite their being equally in need as White patients, because it used healthcare costs as a proxy for need.The paper identifies conditioning on costs as the mechanism by which bias arises in this case.Source: §4 · pp. 7–8
• The care provided to Black patients cost an average of USD 1,800 less per year than care for White patients with the same number of chronic health problems.At a given risk score, Black patients were considerably sicker than White patients, according to signs of uncontrolled illnesses.Source: §4 · p. 8
• Understanding why the algorithm discriminates requires considering both its model-level choices and the broader socio-structural context in which it operates.Obermeyer et al. show that the algorithm used healthcare costs as a proxy for need in a system marked by racial disparities.Source: §4 · p. 8
• By encoding healthcare costs as a proxy for need, the algorithm encodes disparities in care access and quality and underestimates Black patients’ health risks.The paper describes this as reproducing structural racism through data embodying systemic inequalities.Source: §4 · p. 8
• Model-centric explanations might identify cost as influential but would not expose the historical and systemic inequalities underlying the racial bias.The socio-structural interpretation considers the societal context, including historical biases, norms, and institutional practices.Source: §4 · p. 9

• Model-centric explanations can be inadequate for judging whether models benefit or harm people, especially in high-stakes domains shaped by complex social structures and limited domain-specific theoretical understanding.Source: §5 · p. 9
• Many practitioners may not feel equipped to analyze social structures, and constraints in time, training, incentives, or resources can hinder use of that knowledge.The gap between technical expertise and socio-structural understanding is described as a hurdle to developing beneficial ML systems.Source: §5 · p. 9
• Transparency in socially salient domains should extend beyond model-centric details to include socio-structural factors.Source: §5 · p. 9
• Producing more representative labels and objectives requires domain understanding, the ability to identify and extract relevant data, and capacity to iterate.Source: §5 · p. 9
• Socio-structural understanding can inform the ML lifecycle from problem formulation and data collection through development, deployment, and ongoing monitoring.Social structures can influence the questions asked, data collected, and choice of objective functions.Source: §5 · p. 9
• Socio-structural interpretations can sometimes reveal causally relevant reasons for an algorithm’s behavior toward a population and support work toward socially aware, beneficially aligned systems.Source: §5 · p. 9