Source-linked AI summary
FairAIED: Navigating Fairness, Bias, and Ethics in Educational AI Applications
Zhipeng Yin, Sribala Vidyadhari Chinta, Zichong Wang, Matthew Gonzalez, Wenbin Zhang
TL;DR
Fairness research in educational AI remains fragmented across assumptions, methods, and application contexts, while existing surveys often separate technical fairness from educational settings. This survey synthesizes fairness research into an education-centered framework and examines practical challenges including censored outcomes and fairness-utility trade-offs. It concludes by highlighting challenges around privacy, digital access, and adoption.
Problem
Fairness studies in educational AI use differing assumptions and contexts, while existing surveys often omit either educational grounding or fairness as a central lens.
Source: §1 · p. 2Method
The survey synthesizes bias sources, fairness definitions, mitigation strategies, evaluation resources, and ethical considerations within an education-centered framework and taxonomy.
Source: §1 · p. 3Results
The review identifies practical challenges around censored outcomes and fairness-utility choices, alongside personalization and privacy, the digital divide, and educator and student adoption.
Source: §1 · p. 4, §10 · p. 42Takeaways & Limitations
The survey outlines future directions for comprehensive fairness frameworks and interdisciplinary collaboration to support equitable educational AI.
Source: §10 · p. 42Undersampling can discard valuable majority-class information, potentially compromising predictive accuracy and fairness.
Source: §6.1.1 · p. 27
Abstract
from arXiv · showhide
The integration of AI in education holds immense potential for personalizing learning experiences and transforming instructional practices. However, AI systems can inadvertently encode and amplify biases present in educational data, leading to unfair or discriminatory outcomes. As researchers have sought to understand and mitigate these biases, a growing body of work has emerged examining fairness in educational AI. These studies, though expanding rapidly, remain fragmented due to differing assumptions, methodologies, and application contexts. Moreover, existing surveys either focus on algorithmic fairness without an educational setting or emphasize educational methods while overlooking fairness. To this end, this survey provides a comprehensive systematic review of algorithmic fairness within educational AI, explicitly bridging the gap between technical fairness research and educational applications. We integrate multiple dimensions, including bias sources, fairness definitions, mitigation strategies, evaluation resources, and ethical considerations, into a harmonized, education-centered framework. In addition, we explicitly examine practical challenges such as censored or partially observed learning outcomes and the persistent difficulty in quantifying and managing the trade-off between fairness and predictive utility, enhancing the applicability of fairness frameworks to real-world educational AI systems. Finally, we outline an emerging pathway toward fair AI-driven education and by situating these technologies and practical insights within broader educational and ethical contexts, this review establishes a comprehensive foundation for advancing fairness, accountability, and inclusivity in the field of AI education.
1 Introduction
AI in education offers personalized instruction and automated support, but systems can reproduce training-data biases and produce unfair outcomes. This survey unifies fragmented fairness research in an education-centered framework while addressing practical challenges and future directions.
1 Introduction
- AI systems can disadvantage marginalized students through biased essay grading, overprediction of risk, and adaptive platforms that limit challenging materials.These examples connect fairness concerns to assessment, early warning, and personalization.Source: §1 · p. 2
- Prior fairness and educational-AI surveys use differing assumptions and contexts, often separating technical fairness from educational applications or overlooking fairness in education surveys.Censored outcomes and the lack of unified protocols for choosing fairness-utility trade-offs further limit existing frameworks' applicability.Source: §1 · p. 3
- The survey integrates bias sources, fairness definitions, mitigation strategies, evaluation resources, and ethics into a coherent education-centered framework.Its taxonomy adds multi-level bias, capturing issues that affect individuals and groups simultaneously.Source: §1 · p. 3
- The review highlights practical challenges including partially observed outcomes, fairness-utility choices, privacy, digital access, and adoption by educators and students.It outlines future directions around these challenges.Source: §1 · p. 4
- The paper is organized from review methodology and bias case studies through fairness definitions, mitigation, evaluation resources, ethics, and future directions.Sections 2–9 cover these topics in sequence.Source: §1 · p. 4
2 Literature Selection Criteria
The review selects literature on AI-driven education, with particular attention to fairness, bias, and ethics, using a systematic screening and synthesis process.
2 Literature Selection Criteria
- The review searches five databases for relevant literature from the past decade and screens, evaluates, and extracts studies using set criteria.Multiple reviewers conduct the process to minimize selection bias.Source: §2 · p. 4
- The search targets AI-driven educational applications, especially work concerning fairness, bias, and ethics.Search terms include “artificial intelligence in education” and “AI fairness in education.”Source: §2 · p. 4
- The review synthesizes the selected literature to provide an overview and inform future research.Source: §2 · p. 4
3 AI in Education: Bias Perspectives and Case Studies
The section organizes educational-AI bias by its primary unit of harm: group, individual, or both. Case studies illustrate how biases arise from data, model assumptions, and interactions with students, and how individual and group harms can intertwine.
3 AI in Education: Bias Perspectives and Case Studies
- The taxonomy distinguishes group-level, individual-level, and multi-level bias, the last covering fairness dynamics that affect both individuals and groups.Figure 1 presents these three categories; the section uses case studies to illustrate them.Source: §3 · p. 5, Figure 1 · p. 5
3.1 Group-level Biases
- Group-level cases include admissions models favoring privileged schools and grading systems scoring ESL students lower when their essays are underrepresented in training data.The section also identifies historical, mapping, and representation bias as group-level forms.Source: §3.1 · pp. 6–7
3.2 Individual-level Biases
- Individual-level cases include assessment distortion when essay datasets favor Standard American English and personalized platforms that reinforce students’ existing strengths or struggles.The section names measurement and confirmation bias as individual-level forms.Source: §3.2 · p. 9
3.3 Multi-level Biases
- Multi-level bias captures fairness interactions where interventions at one level can affect the other, requiring attention to individual and group outcomes together.Source: §3.3 · p. 10
- The section also describes LLM-related medical-education bias as individual inaccuracies that may reinforce group inequities when training data underrepresent some populations.Source: §3.3 · p. 10
- Early-warning systems can carry group-level over-identification into individual misclassification, while career guidance can translate gender stereotypes into constrained recommendations.The cases connect group-originated patterns to specific students’ interventions, experiences, or career options.Source: §3.3 · pp. 11–12
- Adaptive platforms can turn unequal engagement into unequal personalization, as active students shape recommendations and less-engaged marginalized students receive less support.The section identifies this feedback dynamic as interaction bias.Source: §3.3 · p. 12
- Censored records from transfers, leave, or unresolved enrollment can disproportionately affect disadvantaged and immigrant students, with incomplete trajectories misread as dropout risk.The section frames censored outcome bias as spanning individual, group, and multi-level fairness.Source: §3.3 · p. 13
4 Notations
The section establishes notation for binary educational prediction tasks, protected groups, model predictions, and classification outcomes.
4 Notations
- The framework represents a binary target as Y = {+, −} and a binary protected attribute as A ∈ {a, ā}, with a denoting the disadvantaged group.Examples include pass/fail and socioeconomic or gender groups.Source: §4 · p. 14
- Predictions are written as Ŷ = {+, −}, with a+ and a− and ā+ and ā− distinguishing predicted outcomes for disadvantaged and advantaged students.Source: §4 · p. 14
- The notation defines TP, FN, TN, and FP in terms of educational outcomes such as success, passing, failure, and at-risk status.Source: §4 · p. 14
5 Fairness Definitions in AI in Educational Settings
The section connects educational AI fairness to operational definitions and metrics at group, individual, and multi-level scales. It also describes a framework for evaluating fairness interventions against predictive utility.
5.1 Group Fairness
- Group fairness measures disparities in outcomes or prediction errors across demographic groups, with metrics including Statistical Parity, Equal Opportunity, Predictive Parity, and Predictive Equality.Concordance Imparity additionally compares group-level predictive discrimination when outcomes may be censored.Source: §5.1 · pp. 14–17
5.2 Individual Fairness
- Individual fairness asks whether students with similar academic abilities and experiences receive equal treatment, and distance-based measures test prediction similarity for students with similar educational features.The Lipschitz condition scales input-space distances by a constant; ranking-based measures instead compare similarity rankings, including NDCG@k and ERR@k.Source: §5.2 · pp. 18–19
- Discounted Cumulative Fairness (DCF) extends ranking-based individual fairness by incorporating censorship information when comparing predicted-outcome rankings with student-similarity rankings.It is intended for settings such as longitudinal or incomplete learning datasets where outcomes may be only partially observed.Source: §5.2 · pp. 19–20
5.3 Multi-level Fairness
- Multi-level fairness examines whether fairness at one level is maintained at another, through individual-to-group and group-to-individual approaches.Individual Group Disparity (IGD) measures the largest relative deviation between group-level fairness scores to assess how consistently individual fairness is distributed across demographic groups.Source: §5.3 · pp. 20–21
- Constraint Concordance Difference (CCD) compares individual ranking consistency before and after group fairness constraints, with lower values indicating less individual fairness loss.It can identify students disproportionately affected by enforcing group-level constraints.Source: §5.3 · p. 21
5.4 Unified Fairness-Utility Protocol
- Fairness Bonded Utility (FBU) evaluates interventions against a trade-off baseline and categorizes their joint fairness and predictive-accuracy effects into five effectiveness levels.Its levels range from jointly improving both measures to jointly worsening both.Source: §5.4 · p. 22
- Larger areas in FBU’s “impressive” region indicate more favorable fairness–accuracy trade-offs, enabling comparisons of educational fairness strategies.The framework quantifies interventions relative to a baseline.Source: §5.4 · p. 23
6 Approaches to Mitigating Biases
This section surveys approaches to mitigating bias in educational AI, organizing methods by fairness goal and implementation stage. It also covers methods that address individual and group fairness together, including ranking and graph-based decisions.
6.1.1 Pre-processing
- Pre-processing mitigates group bias by reweighting student instances or resampling underrepresented and dominant groups, but undersampling can discard information and compromise accuracy and fairness.Reweighing adjusts the influence of disadvantaged and dominant groups; resampling can oversample or undersample. The trade-off is retaining information while improving representation.Source: §6.1.1 · pp. 25–27
6.1.2 In-processing
- In-processing embeds group-fairness goals in training through loss penalties, explicit optimization constraints, or adversarial models that reduce demographic information in predictions.Regularizers can enforce multiple criteria, including equal opportunity and demographic parity; constraint-based methods have been applied to essay scoring and dropout early-warning systems. Educational applications of adversarial debiasing reduced racial disparities in achievement predictions and gender bias in course recommendations.Source: §6.1.2 · pp. 27–29
6.1.3 Post-Processing
- Post-processing improves group fairness after training by adjusting decision thresholds or perturbing outputs, including to reduce disparities and protect sensitive information.Threshold adjustment targets criteria such as demographic parity or equal opportunity; outcome perturbation introduces noise or systematic changes to predictions and can use differential privacy.Source: §6.1.3 · pp. 29–30
6.2 Individual Fairness Methods
- Metric-consistency promotes individual fairness by penalizing different predictions for students with similar task-relevant educational characteristics, using neighborhood-based regularization.Dwork et al. formulated the similar-individuals principle; later work learned task-specific similarity metrics and incorporated consistency penalties into training [89].Source: §6.2 · pp. 30–31
- Ranking-based individual fairness preserves relative ordering among similar students; DCF and IFS extend this approach to censored educational outcomes.Traditional ranking methods assess alignment between input-based and predicted rankings, including with NDCG@k and ERR@k. DCF incorporates censored survival information, and IFS embeds DCF as a fairness constraint in survival models [62].Source: §6.2 · pp. 31–32
- Counterfactual fairness requires a student’s prediction to remain consistent when only a sensitive attribute changes, using causal models to examine and mitigate educational prediction bias.Kusner et al. formalized the approach; educational applications include uncovering inequities [95] and generating counterfactual samples for fairness-aware representations [96].Source: §6.2 · pp. 32–33
- Pairwise and graph-based constraints penalize prediction differences among students with similar educational profiles, supporting individual fairness in peer grading and collaborative learning.Lahoti et al. [93] constructed pairwise fairness graphs from educational data, while Kang et al. proposed fairness-aware graph learning for student networks.Source: §6.2 · p. 33
6.3 Multi-level Fairness Methods
- The multi-level subsection begins by warning that treating individual and group fairness separately can introduce bias; its example shows applicant distances changing under individual-fairness constraints alone.For three applicants with equal input-space distances of 3, the passage reports post-constraint distances of 20, 70, and 90; the explanation of this shift continues beyond the supplied passage.Source: §6.3 · p. 33
- Without group awareness, individual fairness constraints can differ across groups, disadvantaging a female applicant; adding group fairness positions her closer to the admitted comparator.The example motivates jointly addressing both fairness levels so similar students receive similar educational opportunities.Source: §6.3 · p. 34
- GFIC combines group and individual fairness through randomized equitable rankings, while GEIF equalizes individual fairness loss across demographic groups.These approaches aim to prevent a few students from bearing disproportionate fairness adjustments in educational allocations.Source: §6.3 · p. 34
- Fair Graph U-Net [101] embeds fairness in graph pooling and probabilistic rankings, while Faircox dynamically calibrates group and individual constraints during ranking optimization.The methods target equitable student representations and balanced adjustments in downstream decisions such as group assignments, support, awards, scholarships, or admissions.Source: §6.3 · p. 35
7 Analytical Tools and Datasets
The section surveys analytical tools for evaluating, visualizing, diagnosing, and mitigating bias, then organizes educational fairness datasets by the contexts and levels of analysis they support.
7.1 Analytical Tools
- AIF360 and Fairlearn support fairness evaluation and mitigation, while TensorFlow Fairness Indicators target large-scale systems and WIT enables interactive demographic comparisons.FairTest identifies unintended sensitive-attribute correlations, FairML offers feature-level interpretability, and Aequitas audits binary-classification decisions.Source: §7.1 · pp. 35–36
7.2 Datasets
- Dataset demographic coverage, granularity, and interaction scope shape fairness assessments; the survey groups benchmarks into IALS, ILAD, and PIFD categories.Table 1 is identified as covering datasets used in fairness and bias research in AIED.Source: §7.2 · p. 36, Table 1 · p. 37
- International assessment and longitudinal datasets support cross-national and extended-trajectory fairness analyses, while PERSUADE enables subgroup analysis of essay-scoring measurement bias.Examples include TIMSS, PISA, NELS:88, ELS:2002, and PERSUADE.Source: §7.2 · pp. 36, 38
- Interaction-based datasets capture fine-grained learner activity, supporting analysis of personalization, dropout prediction, and adaptive learning across demographic subgroups.Examples include EdNet, KDD Cup 2015, OULAD, TutOR, MOOCdb, and CMU DataShop.Source: §7.2 · p. 38
- Policy and institutional datasets support macro-level analyses of educational equity, policy effects, and resource distribution using institutional demographic data.Examples include CCD, IPEDS, and the fairness-oriented Aequitas WP7 dataset.Source: §7.2 · p. 38
8 Ethical Considerations and Policy Guidelines for AI in Education
The section addresses fairness, equity, and privacy as ethical concerns in educational AI and surveys guidelines, policies, and regulations intended to guide responsible integration. It highlights bias risks and student-data protections alongside global and local frameworks for ethical practice.
8.1 Ethical Dilemmas
- Educational AI ethics must address bias and fairness, privacy and data protection, informed consent, and compliance when systems use sensitive student information.The section links careful design and review to avoiding the perpetuation or amplification of existing social biases.Source: §8.1 · p. 40
8.2 Guidelines, Policies, and Regulations
- International and U.S. ethical guidelines address transparency, accountability, fairness, inclusiveness, and privacy in educational AI deployment.Examples include UNESCO recommendations, European Commission guidance, and the U.S. Department of Education toolkit.Source: §8.2 · p. 40
- Policy initiatives span international, national, state, and institutional levels, while formalized regulations include ITU and U.S. Department of Education initiatives.The section presents these documents as a framework for ethical, equitable, and transparent AI use in education.Source: §8.2 · pp. 40–41
9 Challenges and Future Directions
The section identifies privacy, digital access, and educator and student adoption as challenges for equitable educational AI, and calls for international collaboration and shared standards.
Personalization versus data privacy
- Personalization requires learner data that may be sensitive or retained by models, while limited generalizability and privacy risks raise concerns about learner exposure and acceptance.The section calls for restricting instructors’ access to learner data and for institutions to be transparent about data privacy.Source: §9 · p. 41
Digital divide in AI integration education
- Unequal access to infrastructure, devices, and reliable internet can aggravate educational inequalities, particularly in developing countries.The section calls for equitable access to technologies, internet resources, and targeted training, with coordination across government, education, industry, and civil society.Source: §9 · p. 41
Teacher and student adoption
- Effective AI integration requires teacher training and support for students, including customized guidance that reflects students’ metacognitive awareness.The section recommends workshops, online resources, and collaborative learning communities for educators and students.Source: §9 · p. 42
Global Collaboration and Standardization
- International partnerships and standardized guidelines could help harmonize fairness efforts by sharing datasets, analytical tools, and evaluation methods across educational contexts.The section links this cooperation with consistency and transparency in international education.Source: §9 · p. 42
10 Conclusion
The survey uses a taxonomy of individual-, group-, and multi-level bias to organize its review of fairness in educational AI. It highlights practical challenges and calls for interdisciplinary and cross-sector cooperation toward equitable, ethically responsible integration.
10 Conclusion
- The survey organizes its synthesis of educational AI fairness, bias sources, and mitigation strategies using a taxonomy of individual-, group-, and multi-level bias.The framework guides analysis of fairness definitions and mitigation strategies in educational contexts.Source: §10 · p. 42
- The review highlights personalization and data privacy, the digital divide, and educator and student adoption as challenges for equitable educational AI.These challenges are presented as critical but overlooked.Source: §10 · p. 42
- The survey advocates sustained joint efforts among researchers, educators, and policymakers to support ethically responsible and equitable AI integration in education.It emphasizes interdisciplinary collaboration and cross-sector cooperation.Source: §10 · p. 42