Source-linked AI summary
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Wesley Hanwen Deng, Manish Nagireddy, Michelle Seng Ah Lee, Jatinder Singh, Zhiwei Steven Wu, Kenneth Holstein, Haiyi Zhu
TL;DR
Research has rarely examined how industry practitioners actually use fairness toolkits, despite their growing availability. This paper studies practitioners working with existing toolkits through task-based think-aloud interviews and an anonymous survey. It finds needs for better contextualization and support for communication and collaboration around fairness efforts.
Problem
Although open-source fairness toolkits have grown, little research has examined how practitioners actually use them in practice.
Source: §1 · p. 1Method
The authors observed practitioners using two fairness toolkits during a real-world ML task in think-aloud interviews and supplemented those observations with an anonymous online survey.
Source: §1 · p. 2, §3.1.1 · p. 3Results
Takeaways & Limitations
The findings inform design implications for future fairness toolkits intended to address complex socio-technical problems.
Source: §6 · p. 9The study examined individual practitioners; future work is needed on how organizational teams collectively use toolkits on real-world tasks.
Source: §5.2 · p. 9
Abstract
from arXiv · showhide
Recent years have seen the development of many open-source ML fairness toolkits aimed at helping ML practitioners assess and address unfairness in their systems. However, there has been little research investigating how ML practitioners actually use these toolkits in practice. In this paper, we conducted the first in-depth empirical exploration of how industry practitioners (try to) work with existing fairness toolkits. In particular, we conducted think-aloud interviews to understand how participants learn about and use fairness toolkits, and explored the generality of our findings through an anonymous online survey. We identified several opportunities for fairness toolkits to better address practitioner needs and scaffold them in using toolkits effectively and responsibly. Based on these findings, we highlight implications for the design of future open-source fairness toolkits that can support practitioners in better contextualizing, communicating, and collaborating around ML fairness efforts.
1 INTRODUCTION
Open-source fairness toolkits have proliferated, but little research has examined how practitioners use them in practice. This study explores that use through a task-based think-aloud study and survey, identifying needs and design opportunities for better support.
1 INTRODUCTION
- Fairness toolkit usability is difficult because practitioners must formulate real-world questions, draw on distributed expertise, and navigate socially contested definitions and context-dependent fairness.Prior work had not observed practitioners directly using a toolkit within a real ML task.Source: §1 · p. 1
- Eleven practitioners completed a real-world education-resource modeling task and think-aloud interviews with two toolkits; a survey of 56 practitioners supplemented the observations.The interviews examined first-time exploration of toolkit APIs and attempts to address fairness in participants’ models.Source: §1 · p. 2
- Practitioners need toolkit support to contextualize fairness issues and communicate complex analyses to non-technical colleagues.The study also identifies four requirements: learning about fairness, rapid onboarding, pipeline integration, and using toolkits as code repositories.Source: §1 · pp. 1–2
2 BACKGROUND AND RELATED WORK
Prior research documented practitioner needs and evaluated toolkit capabilities, but had not examined practitioners using toolkits to perform a task. This section situates the study and introduces its two toolkits.
2.1 Understanding Practitioners’ Needs and Challenges around ML Fairness
- Related work examined practitioner challenges, organizational barriers, and gaps between fairness research solutions and industry needs.Prior studies include interviews, surveys, and co-design workshops.Source: §2.1 · p. 2
- Earlier studies used retrospective interviews, surveys, or researcher-curated results; this study addresses the remaining gap of observing practitioners using toolkits to perform a task.The paper characterizes its study as the first task-based exploration of how practitioners learn about and work with fairness toolkits and APIs.Source: §2.1 · p. 2
2.2 Open Source Fairness Toolkits
- Fairness toolkits provide ready-to-use metrics and mitigation algorithms to help practitioners assess and potentially reduce unfairness in ML systems.The paper introduces AIF360 and Fairlearn, selected as useful and well-documented toolkits in prior assessment.Source: §2.2 · p. 2
2.2.1 AIF360.
- AIF360 offers broad coverage, with over 71 bias-detection metrics and 9 mitigation algorithms, but IBM describes its intended use as limited to allocation or risk-assessment problems with well-defined protected attributes.The toolkit spans pre-processing, in-processing, and post-processing.Source: §2.2.1 · p. 2
2.2 Open Source Fairness Toolkits
- Fairlearn is community-driven and focuses on negative group impacts, specifically allocation harms and quality-of-service harms; it also offers domain-specific guides and a detailed user guide.Its community includes a weekly call practitioners can join through Discord.Source: §2.2.2 · p. 3
3 METHODS
The study combined a task-based think-aloud interview with an anonymous online survey. Practitioners worked on a Portuguese student-performance task, explored two fairness toolkits, and reported their experiences and workplace context.
3.1.1 Participants.
- Eleven industry practitioners completed the study after purposive and snowball recruitment targeted people with prior professional fairness challenges but no prior toolkit use.Participants averaged five years of programming and three years of ML experience; 41 responded to recruitment, 23 followed up, 14 scheduled, and 12 attended.Source: §3.1.1 · p. 3
3.1.2 Study Design.
- The task asked participants to model which Portuguese secondary-school students needed tutoring, using a roughly 650-instance dataset with demographic, social, school, and academic features.The dataset was chosen to be less familiar and to prompt reflection on sensitive features and socioeconomic factors.Source: §3.1.2 · p. 3
3.1.3 Interview Study Protocol.
- The 60-minute semi-structured interview included 30 minutes exploring Fairlearn and AIF360, followed by implementing an assessment or mitigation approach while thinking aloud.Researchers also asked about practical use and workflow obstacles; interview analysis used inductive thematic coding of about 11.5 hours of recordings and transcripts.Source: §3.1.3 · pp. 3–4, §3.1.4 · p. 4
- Before interviews, participants explored the data, formulated the ML problem, trained a model, and answered preparatory questions in a Colab notebook.The task took 30–60 minutes by participant self-report.Source: §3.1.3 · p. 3
3.2 Anonymous Online Survey
- The anonymous online survey asked about practitioner backgrounds, toolkits, and workplace use; 56 of 71 starters completed at least one section, while 21 completed the entire survey.Question-level sample sizes varied from 21 to 56, and the authors note possible dropout related to expertise, confidence, or sensitive workplace questions.Source: §3.2 · p. 4
4 FINDINGS
The findings cover three phases of toolkit use and identify a cross-cutting need for better support in contextualizing, communicating, and collaborating around fairness work.
4 FINDINGS
- Across preparation, toolkit exploration, and toolkit use, practitioners wanted more support contextualizing, communicating, and collaborating around ML fairness efforts.The authors supplement interview observations with online survey results and discuss design implications after each phase.Source: §4 · p. 4
4.1 Preparing to Evaluate Fairness
Practitioners relied on personal experience to identify fairness concerns, but recognized the need for domain expertise; toolkit capabilities also shaped some problem formulations. The findings motivate broader lifecycle support and tools that facilitate interdisciplinary work and responsible use.
4.1.1 Participants’ analysis choices were heavily influenced by their
- Nine of eleven participants treated sex as a sensitive feature, and seven of those nine tried to mitigate bias by removing or ignoring sensitive features without considering proxy attributes.The paper notes that omitting sensitive features may still lead to disparate outcomes.Source: §4.1.1 · p. 5
4.1.2 Some participants (re)formulated the ML problem to a format
- Five of seven participants who initially formulated regression problems switched to classification after finding that both toolkits better supported classification.Only two participants raised concerns about reformulating their problem because of toolkit limitations.Source: §4.1.2 · p. 5
4.1.3 Participants were conscious about the need for expert guidance
- Participants wanted domain experts to help identify sensitive features and interpret analysis in context; 21 of 27 survey respondents selected legal or regulatory experts, and 20 selected business domain experts.Participants also sought input from dataset builders, education experts, and legal experts.Source: §4.1.3 · p. 5
- The authors recommend extending toolkit support to EDA and problem formulation, while facilitating interdisciplinary conversations and warning users about common fairness pitfalls.They also suggest checklists to help practitioners avoid forcing problem formulations to fit toolkit constraints.Source: §4.1.4 · pp. 5–6
4.2 Exploring and Learning about Toolkits
Practitioners sought toolkits for varied purposes, including learning fairness concepts, getting started quickly, fitting tools into existing workflows, and examining code for reuse. The authors propose design approaches that address these needs while balancing low barriers to entry with responsible use.
4.2.1 Some participants wanted to be able to use toolkits as ed-
- Some participants treated toolkits as learning resources, valuing explanations of fairness concepts and references to research rather than mitigation code alone.One participant appreciated AIF360’s broad overview of fairness techniques and its designated reference section.Source: §4.2.1 · p. 6
4.2.2 Some participants wanted rapid onboarding to fairness toolkits
- Participants sought quick tutorials, Quickstarts, or existing notebooks, and 14 of 24 survey respondents said they would look for the fastest available solution when encountering fairness issues.Practitioners described time constraints as part of their day-to-day work.Source: §4.2.2 · p. 6
4.2.3 Participants emphasized the importance of being able to in-
- Ten of eleven interview participants emphasized that workplace adoption depended on easy integration into existing ML workflows; participants saw Fairlearn’s resemblance to scikit-learn as helpful.Participants valued familiar API classes and functions because scikit-learn was already used to build and evaluate their models.Source: §4.2.3 · pp. 6–7
4.2.4 Some participants wanted to use fairness toolkits as code repos-
- Some participants used toolkit repositories as references for building their own tools, especially when existing toolkits lacked domain-specific methods for recommendation systems.Among 19 survey respondents who preferred building or extending tools, 14 selected the need to understand low-level implementation as a reason.Source: §4.2.4 · p. 7
- The authors propose in-tool learning, familiar syntax, contextual warnings and checklists, and well-documented worked examples to support diverse needs without promoting irresponsible use.They also recommend a flexible, clearly documented codebase for practitioners seeking extensible code.Source: §4.2.5 · p. 7
4.3 Attempting to Use the Toolkits
Practitioners tried fairness toolkits in concrete work and wanted support that made fairness issues easier to understand, communicate, and address in organizational contexts. Their attempts also exposed risks from using examples without adapting them to context.
4.3.1 Participants desired better support for communication around
- Participants valued dashboard visualizations for scanning fairness issues and feeling more comfortable with the toolkit.Some also saw dashboards as useful under time constraints.Source: §4.3.1 · p. 7
- Participants saw visualizations and shareable reports as ways to explain fairness analysis to non-engineers, while all but two of 24 survey respondents (92%) said stakeholder demand mattered.Survey respondents also described external pressure from superiors and the need for reports or scorecards that could be shared with stakeholders.Source: §4.3.1 · pp. 7–8
- Participants found fairness terminology unintuitive and wanted explanations that situated metrics and toolkit functions in relevant contexts.Participants and a survey respondent noted that complex metrics can be hard to explain to business stakeholders.Source: §4.3.2 · p. 8
4.3.3 Participants frequently copied code directly from toy examples
- Most participants copied tutorial or documentation code directly, while only one attempted to reason through metric or sensitive-feature choices.Because fairness is context-dependent, the authors recommend use- and context-specific guidance, including resources on regulatory, domain, and social contexts.Source: §4.3.3 · p. 8, §4.3.4 · p. 8
- The authors recommend toolkits support cross-functional collaboration with tailored visualizations and reports for non-engineers.They argue that designing for teams and organizations, not only engineers, may help address challenges in gaining organizational buy-in.Source: §4.3.4 · p. 8
5 DISCUSSION AND FUTURE DIRECTIONS
The discussion frames fairness toolkits as potential socio-technical supports for collaboration, while emphasizing that the study's scope and evidence leave important questions for future research. It calls for broader evaluation across contexts and of teams' longer-term use.
5.1 Fostering interdisciplinary communication and collaboration
- The authors advocate toolkits that connect diverse organizational roles and outside stakeholders, and that prompt reflection on whether technical intervention or ML is appropriate.They note that the study centered on a technical approach to educational resource allocation and left little room to question whether ML was appropriate.Source: Fostering interdisciplinary… · pp. 8–9
5.2 Future directions for evaluating fairness toolkits
- Future evaluations should include more diverse regions, domains, and organizational contexts, and examine practitioners' use on complex tasks and teams.The authors call for longer-term ethnographic studies of in-house use to understand toolkits' uses, limitations, and potential impacts.Source: §5.2 · p. 9
6 CONCLUSION
The paper reports an empirical exploration of industry practitioners' attempts to use fairness toolkits and identifies needs for contextualization, communication, and collaboration in organizational settings.
6 CONCLUSION
- Think-aloud interviews and an anonymous survey revealed practitioners' need for toolkit support in contextualizing fairness issues and collaborating with non-technical peers.The authors also present design implications for toolkits addressing complex socio-technical problems.Source: §6 · p. 9