Source-linked AI summary
Bias and Discrimination in AI: a cross-disciplinary perspective
Xavier Ferrer, Tom van Nuenen, Jose M. Such, Mark Coté, Natalia Criado
TL;DR
AI systems increasingly make consequential decisions, but the relationship between bias and discrimination remains context-dependent and unresolved across disciplines. This paper surveys and synthesizes technical, legal, social, and ethical literature, concluding that addressing the problem requires robust cross-disciplinary collaboration.
Problem
The relationship between bias and discrimination in AI is not always clear, and no single discipline has resolved the problem independently.
Method
The paper conducts a critical survey and synthesis of literature, combining technical perspectives with social, legal, and ethical ones.
Results
The paper identifies technical, legal, social, and ethical challenges in attesting and addressing discrimination in AI.
Takeaways & Limitations
Finding solutions to discrimination in AI requires robust cross-disciplinary collaborations and sensitivity to differing perspectives on desirable implementation.
Abstract
from arXiv · showhide
With the widespread and pervasive use of Artificial Intelligence (AI) for automated decision-making systems, AI bias is becoming more apparent and problematic. One of its negative consequences is discrimination: the unfair, or unequal treatment of individuals based on certain characteristics. However, the relationship between bias and discrimination is not always clear. In this paper, we survey relevant literature about bias and discrimination in AI from an interdisciplinary perspective that embeds technical, legal, social and ethical dimensions. We show that finding solutions to bias and discrimination in AI requires robust cross-disciplinary collaborations.
I. INTRODUCTION
AI-based automated decision systems can affect large groups in consequential domains, creating risks of systematic disadvantage and discrimination. The paper responds with an interdisciplinary approach spanning technical, social, legal, and ethical perspectives.
- Automated systems make consequential decisions affecting domains such as credit scores, insurance payouts, and health evaluations.
- Discrimination occurs when automation places certain groups or people at a systematic disadvantage based on characteristics such as income, education, gender, or ethnicity.
- As AI techniques increasingly support delegated decisions, digital discrimination has become a serious problem that no single discipline has resolved independently.
- The paper surveys and synthesizes literature by supplementing technical perspectives with social, legal, and ethical ones.
- It discusses how disciplines might collaborate and highlights interdisciplinary challenges in addressing discrimination in AI.
II. BIAS AND DISCRIMINATION
Bias is a deviation from a standard that can support classification, whereas discrimination is unfair or unequal treatment; therefore, bias does not necessarily produce discrimination. Whether an outcome is discriminatory depends on its deployment context and intended task.
- Bias is a deviation from a standard and can be necessary for identifying statistical patterns and classifying differences.
- The paper focuses on problematic instances of bias that may lead to discrimination in AI-based automated decision-making systems.
- Modelling bias can arise from deliberate algorithmic processing choices or from using objective categories for subjective judgements.
- Training bias can reproduce prejudices in past decisions or arise when datasets inadequately represent different populations.
- Usage bias can result when algorithms are applied outside their intended population or when outputs are misinterpreted.
- Assessing discrimination requires context about deployment and task, making simple comparisons of demographic hiring ratios insufficient.
III. MEASURING BIASES
Bias attestation requires examining the full algorithmic process, but access to data, source code, and deployment details is often limited. Technical approaches therefore range from process-focused analysis to dataset and output-based measurement.
- Assessing algorithmic bias requires examining modelling assumptions, training and test data, and adequacy for the specific context and task.
- Procedural approaches identify bias in an algorithm’s decision-making process, while relational approaches identify biased decisions in datasets or outputs.
- Procedural interventions are difficult because complex learning structures are hard to trace and source code is rarely available.
- Explainable AI and inherently interpretable models can help reveal problems in algorithmic code and behaviour.
- Relational methods can compare sub-population outcomes, use counterfactual explanations, or examine links between inputs and biased outputs without accessing the algorithmic process.
B. Bias Metrics
Bias metrics use statistical, similarity-based, and causal approaches to evaluate relationships among groups, individuals, attributes, and algorithmic outcomes. Each approach depends on formalising fairness or relevant relationships in ways that can be difficult to generalise.
- Bias metrics are organised into statistical measures, similarity-based measures, and causal reasoning.
- Statistical measures examine associations between demographic distributions, predicted outcomes, and achieved outcomes.
- Group fairness requires equal proportions of distinct groups to receive each possible algorithmic outcome.
- Similarity measures assess whether individuals with comparable relevant attributes receive comparable classifications or output distributions.
- Causal reasoning uses directed causal graphs to assess how protected attributes affect algorithmic outputs.
IV. ATTESTING AND ADDRESSING DISCRIMINATION
Technical approaches attest discrimination by comparing algorithmic outcomes across protected or proxy-defined groups, but selecting relevant groups and interpreting differences remains context-dependent. The paper argues that clearer definitions require legal, social, and ethical considerations alongside technical methods.
- Identifying groups: Technical approaches first identify sub-populations using protected or proxy attributes, but intersecting attributes make group selection non-trivial.Proxy attributes can be strongly correlated with protected attributes, and groups may result from multiple intersecting features.
- Measuring differences: Bias metrics compare outcomes between selected groups, with differences linked to protected attributes potentially indicating discriminatory decisions.
- Accounting for context: Explanatory attributes encode conditions under which differentiating between users is considered acceptable, helping technical systems address contextual uncertainty.The IBM AI Fairness 360 toolkit is cited as one implementation containing bias-detection and mitigation techniques.
- Legal proxies: Legal rules such as the US four-fifths and Castaneda rules are used as general prima facie evidence, but the interpretive boundary between bias and discrimination remains unresolved.
- Intervention: Removing protected attributes can reduce discriminatory use but may substantially reduce accuracy and leave correlated information available to the algorithm.
- Open questions: Determining how much bias constitutes discrimination and how historical inequities should be treated requires a cross-disciplinary approach.
V. LEGAL PERSPECTIVE
The legal perspective frames discrimination through affected populations, comparison groups, measures, and prima facie thresholds, while newer regulation adds procedural regularity and explanation rights. Existing legal frameworks nevertheless struggle with data-driven categories and the interpretive question of when bias becomes discrimination.
- Legal scope: Anti-discrimination laws vary across countries and protect groups sharing one or more legally recognized attributes.
- Legal tests: Discrimination trials determine the affected population, comparison groups, discrimination measure, and threshold constituting prima facie evidence.
- Regulatory responses: The GDPR addresses automated decision-making through procedural regularity, accountability, and a right to meaningful explanations.
- Legal limitations: Current laws are poorly equipped for algorithmic discrimination because data-driven systems aggregate profiles and may produce arbitrary categories rather than real communities.Procedural and relational explanations do not by themselves resolve when potential bias constitutes discrimination.
VI. SOCIAL PERSPECTIVE
The social perspective treats digital discrimination as dependent on social and historical context, with usage bias challenging general-purpose AI. It also highlights intersectionality, contested transparency, and the need for explanations that let affected people assess automated systems.
- Socio-cultural context: Understanding digital discrimination requires attention to the social and historical conditions informing its meaning and to the context of implementation.Bias in usage challenges generalist AI solutions.
- Intersectionality: Intersectionality can make digital discrimination reinforce existing inequalities as multiple identities and exclusionary experiences interact.Race, gender, and class can interact in labor markets to generate new identity categories.
- Transparency and evaluation: Transparency has cultural and ideological dimensions, and no standard methodology exists for ethically evaluating bias classifications across contexts.People may differ over whether algorithmic discrimination is immoral or whether computers can be assigned moral responsibility.
- Public engagement: Public explanations can help individuals form their own judgments and anticipate or respond to how algorithmic systems profile them.
VII. ETHICAL PERSPECTIVE
The ethical perspective separates unfairness from unlawfulness and emphasizes that moral standards evolve with technology. Ethical frameworks span autonomous-system standards, social morality, and individual engagement, but their technical operationalization remains unclear.
- Ethics beyond law: Discrimination may be unfair without being unlawful, while moral standards change historically and can lag behind technical developments.The paper calls for reassessing moral standards in light of AI’s discriminatory risks and benefits.
- Ethical frameworks: The IEEE Global Initiative and related frameworks provide more grounded structures for developing ethical solutions to intelligent-technology implementation.
- Levels of ethics: Ethical questions about AI can be organized across interconnected levels including social morality and individual engagement.The social-morality level involves many reference points and the cultivation of emotional responses such as guilt, indignation, and empathy.
- Individual responsibility: Individuals and associations retain a role in moral judgment through practices such as developing their own codes of practice.How these ethical levels can be operationalized technically is not yet clear.
VIII. OPEN CHALLENGES
Addressing and attesting digital discrimination remains constrained by technical, legal, social, and ethical challenges. The paper argues that progress requires sensitivity to differing perspectives and robust cross-disciplinary collaboration.
- VIII. OPEN CHALLENGES: Technical limits make it difficult to automatically determine the relationship between bias and discrimination.Legal classifications, social inequalities, differing computational attitudes, and ethical reassessment add further challenges.
- VIII. OPEN CHALLENGES: Algorithmic discrimination exposes gaps in current legislation, which is poorly equipped for its classificatory complexities.
- VIII. OPEN CHALLENGES: AI design and evaluation reflect different perspectives, concerns, and goals rather than a predefined path.The paper recommends sensitivity to distinctions about desirable implementation and a dialogical orientation toward design.
- VIII. OPEN CHALLENGES: Finding solutions to discrimination in AI requires robust cross-disciplinary collaborations.
A. How Much Bias Is Too Much?
Whether biased AI decisions count as discriminatory depends on context, the groups compared, and competing values about merit and equality.
- A. How Much Bias Is Too Much?: Whether bias constitutes discrimination depends on deployment context, comparison groups, and competing individualist-meritocratic or outcome-egalitarian values.Technical systems often simplify these interpretive issues by borrowing legal definitions and thresholds as general evidence.
B. Critical AI Literacy
Critical AI literacy requires users to understand how data, explanations, and moral rules shape automated decisions. The paper connects this need to socio-technical approaches, public confidence, and democratic participation.
- B. Critical AI Literacy: Critical AI literacy should include knowledge of dataset attributes and the ability to compare explanations with the moral rules underlying decisions.
- B. Critical AI Literacy: Because decision systems construct users as data subjects, improving literacy is not solely a technical exercise.
- B. Critical AI Literacy: Socio-technical approaches can address literacy by considering technical dimensions alongside the complex social contexts of deployment.
- B. Critical AI Literacy: Building public confidence and democratic participation requires explainable AI, better Human-AI interaction, socio-technical platforms, and public engagement.
- B. Critical AI Literacy: AI may help detect and mitigate discrimination through data-driven learning or models that encode socio-ethical and legal principles.