PaperlayerView exact revision on arXiv

Source-linked AI summary

EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics

Jiaxu Zhao, Bahar Radmehr, Fares Fawzi, Tanya Nazaretsky, Tanja Käser

arXiv:2609.12949v1cs.AI

TL;DR

LLM tutors may provide different levels or kinds of pedagogical support depending on student demographics, but prior audits and tutoring benchmarks had not jointly tested pedagogical quality and demographic fairness. EduFair-Bench evaluates paired tutoring dialogues across demographic conditions and measures both pedagogical behavior and bias. Across five tutors, capability and fairness are largely orthogonal, and pedagogy-specific RL redistributes rather than removes disparities.

  • Problem

    Whether LLM tutors provide the same quality of pedagogical support across demographic groups remains underexplored, despite the risk of reinforcing or amplifying educational inequities.

    Source: §1 · p. 1

  • Method

    EduFair-Bench pairs a multi-domain question bank with controlled tutor–student simulations across demographic conditions, scores dialogues on pedagogical and conversation measures, and analyzes paired group differences.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across five tutors, capability and demographic fairness are largely orthogonal, pedagogy-specific RL redistributes rather than removes bias, and disparities concentrate in scaffolding, corrective feedback, and mistake recognition.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    EduFair-Bench supports behavior-level fairness auditing of educational LLMs, with findings indicating that stronger pedagogical capability does not imply greater fairness.

    Source: §6 · p. 8

    Because the benchmark uses a single fixed LLM student, its observed bias patterns should be validated against human-subjects studies before strong claims about real learners.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed as tutors, but it is unclear whether they support all students equally well. We introduce \textbf{EduFair-Bench}, a benchmark for auditing the pedagogical fairness of LLM tutors---whether tutoring quality varies systematically with student demographics. EduFair-Bench pairs a multi-domain question bank (mathematics, physics, chemistry) with a controlled simulation in which a fixed LLM student interacts with each tutor across nine demographic levels spanning four dimensions: gender, immigration background, first language, and socioeconomic status (SES). Tutoring quality is scored on five turn-level pedagogical metrics and four conversation-level dimensions, using an LLM judge validated against three-annotator consensus on 180 tutor turns. Bias is measured via paired Wilcoxon signed-rank tests and bootstrap effect-size confidence intervals. Two ablations (demographic cues conveyed through names; conflicting demographic information between tutor and student) disentangle tutor-driven from student-driven bias. Across five tutors, we find that model capability and demographic fairness are largely orthogonal: the smallest model is the most consistent while the four more capable tutors all exhibit wide demographic gaps with no clear capability-to-fairness ordering, pedagogy-specific RL training redistributes rather than removes bias, and language- and immigration-related cues produce larger gaps than gender- and SES-related cues.

1 Introduction

The paper asks whether LLM tutors provide equitable pedagogical support across student demographics, a question prior audits and tutoring benchmarks had not jointly addressed. EduFair-Bench combines pedagogical fairness evaluation with controlled demographic conditions and ablations to distinguish tutor-driven effects from student-simulator confounds.

1 Introduction

  • Prior educational LLM audits focused on surface differences, while existing tutoring benchmarks measured quality without testing whether it is equitably distributed across student groups.The overlooked strategies include answer revelation, scaffolding, actionable next steps, and error correction tone.Source: §1 · p. 1
  • EduFair-Bench jointly evaluates tutoring quality and demographic fairness using a multi-domain question bank, four demographic dimensions, nine metrics, and a structured bias pipeline.The benchmark evaluates both pedagogical quality and fairness across student groups.Source: §1 · p. 2
  • The Implicit and Opposite ablations distinguish name-mediated demographic cues from conflicting tutor and student information, helping separate tutor-driven bias from student-simulator confounds.Source: §1 · p. 2
  • Across five tutors, capability and fairness are largely orthogonal, pedagogy-specific RL redistributes rather than removes bias, and demographic effects vary across pedagogical dimensions.The tutors span 7B to 70B scales and four training paradigms.Source: §1 · p. 2

2 Related Work

Related work covers LLM tutoring, fairness audits in NLP and educational AI, and benchmarks for tutoring quality. These lines of work leave a gap: prior audits do not evaluate pedagogical strategies, and tutoring benchmarks do not assess demographic fairness.

LLM-based tutoring systems

  • Recent LLM tutors draw on tutoring corpora, pedagogy-oriented reinforcement learning, and deployed systems such as LearnLM and Khanmigo.MathDial and SocraticLM provide tutoring corpora, while some RL methods optimize pedagogical reward signals.Source: §2 · p. 2

Fairness in NLP and educational AI

  • Prior work documents demographic bias across NLP, educational AI, educational explanations, tutoring feedback, and deployed commercial tutors, but not bias in pedagogical strategies.The omitted strategies include error diagnosis, answer withholding, next-step guidance, and communicational tone.Source: §2 · p. 2

Tutoring evaluation benchmarks

  • Tutoring benchmarks measure pedagogical quality, but do not examine how that quality varies across student populations.EduFair-Bench extends this evaluation infrastructure with a fairness dimension.Source: §2 · p. 3

3 EduFair-Bench

EduFair-Bench uses a multi-domain question bank and controlled tutor–student simulations across demographic conditions, then scores dialogues and tests paired group differences. Its pipeline combines question selection, paired dialogue generation, pedagogical evaluation, and bias analysis.

3.1 The EduFair-Bench Question Bank

  • The question bank covers mathematics, physics, and chemistry, with questions serving as anchors for generated tutoring conversations.Source: §3.1 · p. 3

3.1.1 Source Corpora

  • Mathematics questions come from SocraTeach, while physics and chemistry questions come from SciQ, whose corpora differ in dialogue and metadata availability.SocraTeach includes reference tutoring dialogues and error metadata; SciQ includes questions, answers, and supporting evidence passages.Source: §3.1.1 · p. 3

3.1.2 Data Filtering

  • Filtering selects multi-turn math problems with student errors and excludes trivial, prohibitively difficult, or answer-disclosing dialogue paths; science items are separated by domain and filtered for low-information questions.The math criteria yield 907 questions from 35,151 items; up to 500 questions per science domain are sampled from 11,679 items after confidence-based classification.Source: §3.1.2 · p. 3

3.2 Tutoring Dialogue Generation

  • The benchmark pairs five tutors with a fixed simulated student and questions across nine demographic levels in four dimensions, generating paired dialogues capped at ten turns.The dimensions are gender, immigration background, first language, and SES; holding question and student fixed attributes differences to tutor responses to demographic signals.Source: §3 · p. 3, §3.2 · p. 4

3.2.1 Experimental Conditions

  • The Explicit condition gives tutors direct demographic statements, Implicit conveys demographics through names, and Opposite varies tutor-perceived and student-stated gender to examine mismatches.The four demographic dimensions are treated as independent axes, so each paired contrast isolates a single signal.Source: §3.2.1 · p. 4

3.2.2 Simulated Tutor & Student

  • All tutors use the same question bank, student, and prompts apart from demographic information; the fixed LLaMA-3.1-8B-Instruct student follows a fixed-ability, low-motivation persona.The fixed student is intended to isolate tutor-side variation, while the Opposite condition provides a handle on residual student-side effects.Source: §3.2.2 · pp. 4–5

3.3 Scoring Dialogues

  • Dialogue scoring combines five turn-level pedagogical metrics with four conversation-level metrics, using an LLM judge validated against three annotators on 180 tutor turns.Conversation-level measures are extracted deterministically; the validation reports judge–human agreement from 0.79 to 1.00 across dimensions.Source: §3.3 · p. 5

3.4 Bias Measurement

  • Paired Wilcoxon tests assess binary contrasts, Friedman tests followed by corrected pairwise tests assess multi-level dimensions, and bootstrap intervals quantify effect sizes.Analyses require at least 15 paired questions; a fairness violation is recorded when the 95% confidence interval excludes zero.Source: §3.4 · p. 5

4 Experimental Settings

The experimental settings compare five LLM tutors spanning model families, scales, and training paradigms under controlled decoding and evaluation conditions. GPT-5-mini is evaluated on a stratified subsample, while open-weight tutors use the full question bank.

4 Experimental Settings

  • Five tutors span instruction-tuned, pedagogy-tuned, reasoning-tuned, and proprietary frontier models, from 7B to 70B scales.The set includes LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct, TutorRL-7B, DeepSeek-R1-Distill-Llama-70B, and GPT-5-mini.Source: §4 · p. 6
  • GPT-5-mini uses a stratified 30% subsample per domain because of API cost, while the four open-weight tutors use the full bank.The GPT-5-mini subsample contains 272 math, 150 physics, and 150 chemistry questions.Source: §4 · p. 6
  • The student and judge are decoded at temperature 0, tutors at 0.1, and all models use a 2048-token limit.Source: §4 · p. 6

5 Results

Across the experiments, demographic gaps vary by tutoring behavior and subject, with language and immigration cues producing larger disparities than gender or SES. Name-only cues also affect pedagogy, and mismatched tutor–student gender signals generally amplify disparities.

5.1.1 Pedagogical bias profiles

  • In mathematics, bias reaches 0.144 for step_scaffolded and 0.169 for actionability, while corrective_tone exceeds 0.10 for every model in chemistry and physics.Corrective_tone ranges from 0.102–0.168 in chemistry and 0.129–0.294 in physics; mistake_recognize bias is also higher there for all models except LLaMA-3.1-8B.Source: §5.1.1 · pp. 6–7
  • Pedagogy-specific RL redistributes bias: TutorRL-7B reduces corrective_tone and answer_leakage effects but increases mistake_recognize bias relative to Qwen2.5-7B.For example, corrective_tone falls from 0.294 to 0.163 in physics, while mistake_recognize rises from 0.150 to 0.212 there.Source: §5.1.1 · p. 7

5.1.2 Bias across demographic dimensions

  • First-language and immigration effects are largest, especially in physics; First Language reaches 0.281 for DeepSeek-R1-70B, while Immigration reaches 0.216 for DeepSeek-R1-70B.Gender and SES disparities are smaller and more diffuse; DeepSeek-R1-70B also has the smallest reported gender effect, 0.022 in chemistry.Source: §5.1.2 · p. 7
  • Qwen2.5-7B exceeds the 0.10 threshold in 7/12 domain–dimension cells, versus once for LLaMA-3.1-8B; RL also raises First Language disparities across all domains.TutorRL-7B versus Qwen2.5-7B First Language effects rise from 0.069 to 0.144 in math, 0.114 to 0.177 in chemistry, and 0.081 to 0.137 in physics.Source: §5.1.2 · p. 7

5.2 Are names alone sufficient to trigger bias?

  • Name-mediated pedagogical bias is strongest in physics and usually under combined gender-plus-ethnicity cues, while conversation-level effects remain comparatively small.Mean pedagogical effects are approximately 0.074 in math, 0.071 in chemistry, and 0.142 in physics; combined cues reach 0.205 for LLaMA-3.1-8B and 0.185 for Qwen2.5-7B.Source: §5.2 · p. 7
  • Name-cue sensitivity varies by tutor: LLaMA-3.1-8B reaches 0.205 pedagogical bias under Implicit cues, while GPT-5-mini has smaller Implicit than Explicit effects.DeepSeek-R1-70B is the main conversation-level outlier, with elevated asymmetry in chemistry and physics.Source: §5.2 · pp. 7–8

5.3 Is bias driven by tutor-side demographic conditioning?

  • Mismatched gender conditioning produces higher pedagogical bias in 11/15 tutor–domain cells, and conversation-level effects follow the same pattern.Only four cells reverse the pattern, with Correct exceeding Wrong by at most 0.043; these reversals span all three domains and appear model-dependent.Source: §5.3 · p. 8

6 Conclusion

EduFair-Bench audits tutoring fairness with controlled demographic variation and human-validated pedagogical metrics. Across five tutors, demographic disparities vary across behaviors, cues, and training paradigms.

6 Conclusion

  • EduFair-Bench combines a multi-domain benchmark, controlled demographic variation, human-validated metrics, and paired non-parametric bias analyses.The benchmark is released to support behavior-level fairness auditing of educational LLMs.Source: §6 · p. 8
  • Across five tutors, disparities concentrate in scaffolding, corrective feedback, and mistake recognition; name-only cues suffice, and mismatched conditioning further amplifies divergence.The paper also reports that name cues produce substantial disparities particularly in physics and under combined gender–ethnicity cues.Source: §6 · p. 8
  • Greater pedagogical capability does not imply greater fairness, and pedagogy-specific RL redistributes rather than removes demographic disparities.Source: §6 · p. 8

Limitations

The audit treats demographic dimensions as independent axes, with only partial intersectional coverage through name cues. It also relies on simulated rather than human students, so classroom-facing claims require validation with human-subjects studies.

Limitations

  • Four demographic dimensions are tested independently; name-based gender–ethnicity cues provide only partial intersectional coverage, leaving a full four-dimension audit for future work.The paper notes that harms across attributes may interact non-additively and cannot be assumed to sum from isolated contrasts.Source: Limitations · p. 9
  • A single fixed LLM student controls student-side variation, but its error patterns, persistence, and help-seeking may differ from real learners.The authors call for validation against human-subjects studies before making strong claims about classroom deployment.Source: Limitations · p. 9

Ethical Considerations

The paper frames tutoring fairness audits as ethically important and describes EduFair-Bench as reproducible. It also documents constraints in demographic operationalization and the human annotation protocol.

Ethical Considerations

  • Because differential tutoring can shape learning, perceived ability, and later opportunities, the authors regard bias audits before classroom deployment as an ethical prerequisite.EduFair-Bench is designed to make that audit reproducible.Source: Ethical Considerations · p. 9
  • Coarse, often binary demographic categories erase within-group heterogeneity; the authors encourage extensions with richer operationalizations.The name cues may conflate names with race and ethnicity and are used as stimuli, not as ground truth about individuals.Source: Ethical Considerations · p. 9
  • The benchmark is intended for auditing, tutor comparison, and mitigation research, not as certification that real students will be treated equitably.Its effect-size thresholds apply only to the stated configuration.Source: Ethical Considerations · p. 9

A.1 Annotator Information

  • Three education researchers calibrated rubrics through four workshops; final pre-discussion labels used frozen rubrics, and annotators were blind to demographic condition and tutor identity.The annotators had dialogue-annotation experience and knew the Grade 8–9 STEM content.Source: §A.1 · p. 12

A.2 Annotation Agreement

This section describes the annotation framework and controlled simulated-student design used to assess tutor responses across demographic cues. It also identifies the agreement and judge-alignment measures reported for 180 tutor turns and explains how fixed-student comparisons and independently manipulated gender cues probe tutor sensitivity and residual student-side effects.

A.2 Annotation Agreement

  • Table 3 reports pre-discussion Krippendorff’s α and LLM-judge–human alignment on the 180-turn calibration set; the supplied passage gives no values.Annotation rubrics were developed in four calibration-workshop rounds involving three annotators and disagreement analysis on 18 conversations (180 tutor turns).Source: Table 3 · p. 12, §A.2 · p. 12, B Annotation Rubrics · p. 12

B Annotation Rubrics

  • The rubrics use binary labels, with −1 where applicable, and define a student mistake to include incorrect commitments or uncertainty without a committed answer.A correct conclusion, clarifying question, or social-only turn is not a mistake; rubrics include decision procedures and examples across mathematics, physics, and chemistry.Source: B Annotation Rubrics · p. 12

B.1 answer_leakage

  • A tutor response is labeled 0 for no leakage when it confirms or rephrases student-produced content, gives a directional hint without revealing the answer, or offers only metacognitive guidance.Source: §B.1 · p. 12

B.2 step_scaffolded

  • Step scaffolding is labeled 1 when the tutor assigns a specific, meaningful next sub-task for the student to perform without supplying its answer in the same turn.Scaffolding can co-occur with answer leakage when the tutor’s prompt reveals another key step.Source: §B.2 · p. 13

B.3 actionability

  • Actionability is labeled 1 when the turn leaves the student a clear next action requiring substantive cognitive work, and 0 for praise, vague checks, or work already done by the tutor.Annotators judge the student’s immediate next action at the end of the tutor’s turn.Source: §B.3 · pp. 13–14

B.4 corrective_tone; B.5 mistake_recognize

  • Corrective tone distinguishes softened from blunt corrections, while mistake recognition labels whether the tutor targets a prior error; both use −1 when not applicable.Recognition may be implicit through a targeted guiding question, but asking the student to redo work without targeting the error does not count.Source: §B.4 · p. 14, §B.5 · p. 14

C Prompt Templates; C.1 Base Prompt Components; C.2 Condition-Specific Demographic Components

  • Prompt templates pair fixed student and tutor base components with one demographic signal per session, delivered through explicit framing, a connoted name, or—in the gender-only Opposite condition—independently set student and tutor identities.The base framing specifies grade and domain; the student persona is struggling and low-motivation, while the tutor is instructed to guide discovery rather than give away answers.Source: C Prompt Templates · p. 14, §C.1 · pp. 14–15, §C.2 · p. 15

D Simulated Student: Design Rationale

  • A fixed LLM student enables scalable controlled comparisons, with paired dialogues on the same question holding the student constant so variation is attributable to the tutor.The simulated learner is designed to make plausible grade-level errors, respond to hints, and struggle across turns, creating opportunities to assess pedagogical metrics.Source: D Simulated Student: Design Rationale · p. 16
  • A fixed student avoids conflating tutor demographic sensitivity with student persona sensitivity, and the Opposite condition independently varies the student’s stated gender and the tutor’s perceived gender.This provides an additional way to assess residual student-side effects.Source: D Simulated Student: Design Rationale · pp. 16–17

E Data Filtering Details

The appendices document question filtering, the judge’s rubric, and a calibration example that illustrates judge–human disagreements. The rubric defines six output fields, including conditional scoring for student mistakes.

E Data Filtering Details

  • SciQ questions are filtered and classified using weak keyword labels followed by a TF-IDF logistic-regression classifier.The filter removes empty support passages and overly definitional questions; classifier features combine question and support text.Source: E Data Filtering Details · p. 17

F LLM Judge Prompt and Error Analysis

F.1 Judge System Prompt and Rubric

  • The judge outputs six fields; the prior-student-mistake field gates two metrics, while four other fields map directly to Table 2 metrics.Source: §F.1 · pp. 17–18

F.2 Worked Example: Dialogue, Labels, and Judge Rationales

  • The calibration example shows two human–judge disagreements, both on the final tutor turn of a Qwen2.5-7B chemistry conversation.Source: §F.2 · p. 18

F.3 Disagreement Analysis

In the calibration example, the judge over-credits engagement when a tutor asks a probing question and supplies its answer in the same turn. These disagreements inflate engagement scores but do not drive paired demographic bias effects.

F.3 Disagreement Analysis

  • Both final-turn disagreements involve the judge scoring scaffolding and actionability as 1 when human annotators score them as 0.The tutor states the answer after asking the probing question, leaving no substantive cognitive task for the student.Source: §F.3 · p. 18, Table 4 · p. 19
  • The same self-answering trigger appears in a milder middle-turn actionability disagreement, with the judge over-scoring engagement-style metrics.The rubric’s conservative instruction for answer leakage has no equivalent for scaffolding or actionability.Source: §F.3 · pp. 18, 20
  • Because the over-scoring affects both groups symmetrically in paired comparisons, it inflates engagement levels without introducing a between-group bias difference.Calibration Krippendorff’s α is 0.79 for scaffolding and actionability and 0.94–1.00 for the other three dimensions.Source: §F.3 · p. 20

G Additional Results

The appendix supplies numerical backing for the main-text figures, with tables detailing effect sizes across metrics, models, domains, demographic dimensions, and ablation conditions.

G Additional Results

  • Tables 9–12 report signed rank-biserial r with 95% bootstrap confidence intervals for each turn-level metric by demographic dimension.The intervals help assess which individual metric gaps are significant at the cell level versus only after aggregation.Source: G Additional Results · p. 20, Table 9 · p. 22, Table 10 · p. 22
  • Tables 5 and 7 provide |r| values for Implicit name contrasts and Opposite student–tutor gender matches or mismatches, respectively.Their captions identify within-model maxima across pedagogical and conversation-level columns.Source: Table 5 · p. 20, G Additional Results · p. 20, Table 7 · p. 21
  • Table 6 reports mean absolute rank-biserial correlation per pedagogical and conversation-level metric, averaged across four demographic dimensions under Explicit.It marks desirable, undesirable, and direction-neutral bias and bolds |r| ≥0.10.Source: G Additional Results · p. 20, Table 6 · p. 21
  • Table 8 gives Explicit-condition |r| averaged over five turn-level metrics for each model, domain, and demographic dimension.Source: G Additional Results · p. 20, Table 8 · p. 21

Related papers

Loading 2609.12949v1…