PaperlayerView exact revision on arXiv

Source-linked AI summary

Spotting (and Missing) Algorithmic Bias: Investigating User Understanding in a Fairness Assessment Tool

Anna Verheyden, Yizhe Zhang, Robin De Croon, Simone Stumpf, Katrien Verbert

arXiv:2609.35781v1cs.HC

TL;DR

Fairness tools have involved non-technical stakeholders, but whether those stakeholders understand the metrics has not been tested. FairAware, co-designed with HR-domain experts, was evaluated in a mixed-methods study; most participants identified the most disadvantaged group, while subjective and objective understanding were not well aligned. The findings support comprehension checks before relying on stakeholder input in higher-stakes decisions.

  • Problem

    Fairness tools involve non-technical stakeholders, but prior prototypes had not tested whether users understand the metrics they interact with.

    Source: Measuring Fairness and Involving… · p. 2

  • Method

    The authors co-designed FairAware with hiring-domain stakeholders and evaluated it in a mixed-methods study of 70 HR employees and job seekers.

    Source: Introduction · p. 1

  • Results

    Across 70 participants, most identified the most disadvantaged group; task time was the only predictor, and subjective understanding correlated only weakly with objective understanding.

    Source: Conclusion · p. 10

  • Takeaways & Limitations

    Fairness tools should include comprehension checks before stakeholder input is relied on for higher-stakes decisions.

    Source: Conclusion · p. 10

    The study was online via Prolific, conducted in one session with synthetic data, and participants’ judgments had no consequences.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Fairness metric selection is typically left to data scientists, but which biases are problematic and which metric captures them best depends on stakeholders' experience and domain knowledge. This calls for involving non-technical stakeholders, but the research prototypes built for this purpose so far have not tested whether these stakeholders form accurate mental models of the metrics they interact with or can act on them to identify biases. We present FairAware, a fairness assessment tool co-designed with Human Resources (HR) domain experts. We evaluate stakeholders' understanding through a mixed-methods study with 70 participants (35 HR employees, 35 job seekers), measuring objective and subjective understanding, cognitive load, bias identification accuracy, and open-ended feedback. Most participants correctly identified the most disadvantaged group, with task duration being the only significant predictor. We also found a gap between subjective and objective understanding, with both groups performing similarly across all measures. These results suggest that fairness assessment tools for non-experts are usable for identifying biases but need built-in checks on understanding before stakeholders make higher-stakes decisions.

Introduction

Fairness metric choice is a context-dependent value judgment, yet stakeholders with relevant domain expertise or lived experience may lack the technical background to interpret the metrics. FairAware was co-designed with hiring stakeholders, and its evaluation examines whether non-experts understand fairness metrics and identify bias.

Introduction

  • More than fifty fairness metrics have been proposed, many mutually incompatible, so selecting which metrics to apply is a context-dependent value judgment.Stakeholders’ domain expertise or lived experience positions them to make that judgment, but they may lack the technical background to interpret the metrics.Source: Introduction · p. 1
  • Earlier work developed tools to elicit fairness preferences and frameworks for stakeholder participation, but had not investigated stakeholders’ metric understanding or ability to identify biases.The paper asks whether interfaces presenting fairness metrics help non-technical stakeholders interpret them correctly and identify potential algorithmic biases.Source: Introduction · p. 1
  • FairAware is a fairness assessment dashboard co-designed with non-technical hiring stakeholders and evaluated in a mixed-methods study of 35 HR employees and 35 job seekers.The study asks what affects users’ subjective and objective understanding and bias identification, and how users employ the dashboard to understand and find biases.Source: Introduction · p. 1
  • The study found that most participants identified the most disadvantaged group correctly, with task duration the only significant predictor.Subjective understanding tracked prior familiarity rather than objective understanding, and dashboard interaction logs did not predict either.Source: Introduction · pp. 1–2

Background and Related Work

Fairness assessment depends on context-sensitive choices about metrics and affected groups, prompting work to involve non-technical stakeholders. This paper focuses on whether stakeholders understand the metrics and can act on them, while also developing a mental-model instrument for evaluating that understanding.

Measuring Fairness and Involving Stakeholders to Assess Fairness

  • More than fifty fairness metrics reflect different ethical viewpoints and can be mutually incompatible, making metric and attribute selection context-dependent choices usually made by data scientists.Group-level metrics compare outcomes between privileged and unprivileged groups; individual-level metrics ask whether similar people are treated similarly.Source: Measuring Fairness and Involving… · p. 2
  • Stakeholder-facing fairness prototypes elicit preferences or facilitate deliberation, but assume users understand the metrics and had not been evaluated in recruitment.The paper focuses on testing stakeholders’ understanding and ability to act on metrics in the recruitment domain, validating objective understanding and its relation to subjective understanding and cognitive load.Source: Measuring Fairness and Involving… · p. 2

Measuring Understanding

  • Mental-model questions can test factual knowledge, prediction of system behavior, or free-text articulation; multiple-choice formats may overestimate understanding through recognition or guessing.Prior work in explainable AI includes approaches to measuring users’ mental models.Source: Measuring Understanding · p. 2
  • Because no validated instrument measures mental models of fairness metrics specifically, the paper constructs one balancing fact, simulation, and articulation questions.Existing questionnaires vary in scoring and item composition, are context-specific, and lack shared construction guidelines, according to the authors.Source: Measuring Understanding · p. 2

Methods

The authors co-designed FairAware with hiring-domain stakeholders and used it in an online mixed-methods study with HR employees and job seekers. The methods combine configurable fairness assessment tasks with measures of understanding, cognitive load, qualitative feedback, and interaction logs.

Methods

  • A human-centered co-design process with 15 participants preceded an online mixed-methods study with 70 participants: 35 HR employees and 35 job seekers.The research received ethics approval, and participants provided informed consent.Source: Methods · p. 2

Data and recommendation algorithm.

  • FairAware assessed a synthetic job-matching recommender that produced binary vacancy suggestions from candidate–vacancy skill similarity, with records covering seven co-selected attributes.The tool addressed unequal access to vacancy suggestions, rather than résumé screening or candidate ranking; common biases in age, gender, and language fluency were incorporated.Source: Data and recommendation algorithm · pp. 2–3

The FairAware dashboard.

  • The dashboard lets users inspect group fairness scores, choose attributes and metric weights, explore metric explanations, and view attribute relationships and data insights.The overall fairness score combines rescaled metric values using the user’s chosen priorities; the dashboard presents four fairness metrics and supports user-defined attribute binning and re-weighting.Source: The FairAware dashboard · pp. 3–5

Participants.

  • The study recruited 35 HR professionals and 35 job seekers through Prolific, with eligibility and screening criteria and two quality-based participant replacements.Participants had to be English-fluent, at least 18, and meet an 80% task-approval threshold; screening questions validated HR expertise.Source: Participants · p. 5

Procedure.

  • In a roughly 71-minute online session, participants completed a tutorial and pre-study measures, three dashboard tasks, then post-study measures and open-ended feedback.The tasks were to identify the most biased group, configure metric weights to reflect a fairness policy, and investigate a possible cause of bias.Source: Procedure · p. 5

Data collection and analysis.

  • Objective understanding was measured with a newly constructed pre/post mental-model instrument, alongside subjective understanding, cognitive load, qualitative responses, and interaction logs.The nine-item instrument balanced fact, simulation, and articulation questions; researchers also coded feedback and task responses and logged page visits, clicks, searches, parameter changes, and time.Source: Data collection and analysis · pp. 5–6

Results

Participants across stakeholder groups showed similar measured outcomes, yet subjective confidence often diverged from objective understanding. Most identified the most disadvantaged group, but identifying bias did not necessarily mean they understood the metrics or were ready for higher-stakes decisions.

Subjective understanding tracked confidence, not comprehension.

  • Subjective understanding (SU) correlated only weakly with post-task objective understanding (OU), despite stronger links to familiarity and cognitive load.Some participants rated their understanding highly while giving superficial or mistaken answers; others rated themselves cautiously despite strong responses.Source: Subjective understanding tracked… · p. 6

Familiarity helped subjective understanding, but not objective understanding.

  • Familiarity correlated weakly or not at all with OU, and cognitive load did not correlate with OU despite its association with SU.The authors note that interface-navigation effort may contribute to low OU, but the observed patterns make it an unlikely sole explanation.Source: Familiarity helped subjective… · pp. 6, 8, Figure 7 · p. 7

Most participants were able to spot a bias, but not because of understanding it.

  • 42 of 70 participants correctly identified the most disadvantaged group; time on task was the biggest predictor of success (p = .011), while pre- and post-task OU, SU, and CL did not predict identification.The authors caution that first-session bias identification is not decision-ready and that appropriate use requires more interaction time.Source: Most participants were able to spot a… · p. 8

Dashboard use produced a small gain in objective understanding.

  • Learning Gain was significant across the full sample (W = 592.0, Z = −2.040, p = .041, r = 0.244), driven by job seekers but not HR employees.The authors describe the absolute gain as modest; Phase 2 lasted an average of 16.61 minutes.Source: Dashboard use produced a small gain in… · p. 8

Helpfulness reports showed what logs or dashboard use miss.

  • Home Page helpfulness reports were associated with higher OU (M = 58.1% vs. 39.3%; p = 0.002), while Information Page use did not distinguish OU.Twenty-two participants named the Home Page and 28 named the Information Page as helpful.Source: Helpfulness reports showed what logs… · p. 8
  • The Experiment Page was polarizing: 14 participants found it most helpful and 11 least helpful, while job seekers more often named it helpful than HR employees.The authors suggest interactive configuration can be valuable once users have internalized the static overview.Source: Helpfulness reports showed what logs… · pp. 8–9

Self-reported usefulness did not directly translate to actual usage.

  • For the Causal Analysis Page, 16 participants named it least helpful, while only 10 used it for its intended attribute-influence task.Some participants who said it was helpful did not use it for that task or could not correctly interpret its graph.Source: Self-reported usefulness did not… · p. 9

Exploration depth varied, repeating explanations remains valuable either way.

  • Usage logs did not correlate with OU, SU, or cognitive load, partly because similar hover and click behavior reflected either exploration or confusion.Participants valued keeping tooltips, tutorials, and hover explanations accessible throughout use.Source: Exploration depth varied, repeating… · p. 9

Discussion

Non-technical stakeholders can identify biased groups, but task duration—not measured understanding—predicted success. The discussion calls for explicit comprehension checks and cautions that the study’s scope and measurement instrument limit its conclusions.

Beware of and counteract Mount Stupid.

  • Subjective understanding rose with familiarity and fell with cognitive load, while objective understanding did neither.The authors caution that simplifying an interface may reduce effort to verify or improve understanding.Source: Beware of and counteract Mount Stupid · p. 9
  • Familiarity with fairness metrics correlated most strongly with subjective understanding but was unrelated to objective understanding.The authors suggest lightweight, auto-scorable checks to identify flawed mental models and tailor explanations.Source: Beware of and counteract Mount Stupid · p. 9

Dashboard use does not ensure users understand fairness metrics.

  • Hover counts, clicks, and page visits did not predict objective understanding, while reported helpfulness did not always match actual use.Only two of seven participants who rated the causal graph most helpful used it for the bias-cause task.Source: Dashboard use does not ensure users… · p. 9

Algorithmic bias finding is achievable but takes time.

  • Task duration was the only significant predictor of basic fairness-assessment success, not objective or subjective understanding.The authors suggest shorter, repeated assessments rather than one longer session, which could increase cognitive load.Source: Algorithmic bias finding is achievable… · p. 9

Consensus-building frameworks need to account for miscalibrated participants.

  • Stakeholders may hold strong opinions based on incorrect mental models, complicating frameworks that assume stronger opinions reflect deeper engagement.The authors suggest calibrating participation with a brief understanding check before weighting opinions.Source: Consensus-building frameworks need to… · pp. 9–10

Transferring FairAware to other domains.

  • FairAware’s structure could transfer to other binary decisions with sensitive attributes, but meaningful metrics and groups require domain co-design.The authors name loan approval and benefits triage as possible settings.Source: Transferring FairAware to other domains · p. 10

Future Work.

  • The authors call for longitudinal real-world studies, tests of more complex assessment tasks, and research on understanding in group deliberations.These directions address how mental models and bias-identification accuracy change with repeated use, task complexity, and group decisions.Source: Future Work · p. 10

Limitations.

  • The study used four metrics, synthetic data, and a one-time online session without consequences; its understanding instrument also lacks a validated standard for cross-questionnaire comparison.The instrument distinguished understanding levels, but absolute scores cannot be directly compared with other mental-model questionnaires.Source: Limitations · p. 10

Conclusion

The paper tests whether non-technical stakeholders understand fairness metrics through FairAware and a mixed-methods study. Results show a gap between subjective and objective understanding, motivating comprehension checks before stakeholder input is relied upon.

Conclusion

  • The authors co-designed FairAware, built a mental-model instrument, and studied 35 HR employees and 35 job seekers.The study used mixed methods to test the assumption that stakeholders interpreting fairness metrics understand them.Source: Conclusion · p. 10
  • Across 70 participants, subjective understanding correlated with familiarity but only weakly with objective understanding; time spent, not understanding, predicted biased-group identification.HR employees were not better calibrated than job seekers.Source: Conclusion · p. 10
  • The authors recommend workflow comprehension checks, calibrated weighting of stakeholder preferences, and treating single-session assessments as preliminary.They state that a confident user who has quickly read but not grasped the metrics does not meet the EU AI Act’s meaningful human oversight requirement.Source: Conclusion · p. 10
  • The mental-model instrument offers a starting point for adaptation to other tools and metrics, alongside a stakeholder-facing tool and evidence that comprehension checks are needed.The authors connect these contributions to more responsible stakeholder integration in fairness assessment.Source: Conclusion · p. 10

Appendix A: Dataset Construction

The authors constructed synthetic hiring data with known group disparities to support ground-truth evaluation of fairness detection.

Appendix A: Dataset Construction

  • The synthetic dataset contains 10.000 job candidates and 34 vacancies, with candidate–vacancy fit measured by skill overlap on a continuous 0–1 scale.The dataset is described as representative of the local labor market.Source: Appendix A · p. 13
  • Candidates are characterized by demographics, education, experience, resume gaps, skills, and languages, with nationality drawn independently of other attributes.Spoken-language assignment depends partly on nationality, and candidates may receive additional sampled languages.Source: Appendix A · p. 13
  • The data encode disparities for women, candidates aged 54+, and candidates who do not speak the local language, enabling ground-truth checks of whether tools flag introduced biases.The construction changes skills, resume-gap probability, experience, or education probabilities by group.Source: Appendix A · p. 13

On the screening criteria

Participants were screened for reading fluency and group eligibility, with additional domain-knowledge checks for HR employees.

On the screening criteria

  • The screening criteria table covers English fluency, approval history, and separate eligibility checks for HR employees and job seekers.English fluency was required because FairAware contains textual explanations; approval history was used to screen for answer quality.Source: On the screening criteria · p. 13, Table 1 · p. 13

Domain screening questionnaire

  • HR participants needed a hiring- or inclusion-related role and at least three correct answers on five domain questions to continue.The threshold accommodated variation in HR roles and countries; the authors state the chance of guessing correctly was 10,35%.Source: Domain screening questionnaire · pp. 13–14

Q1. Which of the following best describes your involvement in hiring candidates?

  • The HR role question distinguished direct hiring involvement, high-level recruiting or inclusion roles, and no relevant involvement.Those without direct hiring or ethics/inclusion involvement in an HR context were redirected to Prolific.Source: Q1. Which of the following best… · p. 14

Q3: What is a key benefit of conducting a trend analysis in HR planning?

The appendix collects the study questionnaires, including pre- and post-task mental-model measures, subjective-understanding and cognitive-load items, and open-ended feedback. The mental-model questions probe metric concepts, dashboard interpretation, and users’ ability to reason about fairness information.

Q3: What is a key benefit of conducting a trend analysis in HR planning?

Demographics questionnaire

The mental model questionnaire, including scoring

  • The mental-model score totals 12.5 points and is later converted to a percentage; questions test impact-ratio and consistency concepts.The rubric distinguishes group-level impact ratio from individual-level consistency, including whether similar people are treated similarly.Source: Table 2 · p. 15, The mental model questionnaire,… · p. 15
  • Participants explain a weighted overview score and match preset weighted scores to the presets’ metric priorities.The questionnaire also asks how the weighted score changes when group balance shifts while other metrics and their weights remain fixed.Source: The mental model questionnaire,… · p. 15
  • Causal-graph questions ask which attributes would change when attribute A changes and which would be most influenced.The first question is scored only when all answers are correct; the second asks participants to identify the attribute with the greatest influence.Source: The mental model questionnaire,… · p. 16
  • A rubric asks whether participants understand why multiple metrics add value, including that metrics capture different aspects and cannot all be optimized simultaneously.Source: The mental model questionnaire,… · p. 17

Notes on the mental model questionnaire

Qualitative questions

The qualitative-question section gathers participants’ feedback about dashboard components, confusion, and accessibility, alongside NASA-TLX cognitive-load scores. It also reports a gap between recognizing an impact-ratio answer and explaining the concept.

Qualitative questions

  • The feedback prompts ask which dashboard parts helped or hindered understanding, what caused confusion, and how the experience could be more accessible.Source: Qualitative questions · p. 17
  • 40% of pre-task participants who answered the impact-ratio multiple-choice item correctly scored zero on the open-ended explanation; post-task, 37.2% of correct-MCQ respondents did so.The table caption identifies the comparison as open-ended impact-ratio scores conditional on multiple-choice correctness.Source: Table 5 · p. 18
  • Table 6 reports NASA-TLX subscale scores for 70 participants (N=70), with scores recorded on a 7-point scale; no individual subscale names are given in the cited passage.Source: Table 6 · p. 18

Appendix E: Codebook of the Semi-Structured Interviews

The interview codebook organizes eight themes and 52 codes derived from five semi-structured co-design interviews. Its categories cover mental models, agency, dashboard components, navigation, the causal graph, and preferences.

Appendix E: Codebook of the Semi-Structured Interviews

  • Eight themes and 52 codes were derived from five semi-structured interviews and grouped across dashboard and participant-response topics.The groups include mental models and clarifications, agency, overview and weighting components, metric explanations, flow, causal graphs, and metric, attribute, or data preferences.Source: Appendix E · p. 18
  • The codebook separates codes about mental models or expectations from codes about clarifications and phrasing.The table captions describe what participants expected the system to do and count how many of five participants raised clarification or phrasing codes.Source: Table 7 · p. 19, Table 8 · p. 20
  • Agency codes capture whether users would adapt group configurations or metric weights.The corresponding table reports the number of participants, out of five, who raised each code.Source: Table 9 · p. 20
  • The codebook tables classify interview codes about overview indicators, general interface, hover cards, weighting, metric explanations, flow, and the causal graph.The causal-graph table covers three participants who opened the graph; the other listed tables count codes raised by up to five participants.Source: Table 10 · p. 21, Table 11 · p. 22, Table 12 · p. 23
  • Three participants named age, nationality, or language as the most important attribute; one participant requested a blue-collar/white-collar split.The latter request reflects dissatisfaction with results shown per individual job rather than at the preferred granularity.Source: Appendix E · p. 26
Loading 2609.35781v1…