PaperlayerView exact revision on arXiv

Source-linked AI summary

Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets

Yan Ma, Lizhuo Zhang

arXiv:2609.29625v1cs.LG

TL;DR

Public educational prediction benchmarks are rarely assessed for structural reliability before modeling, leaving limited evidence about whether they support generalizable claims. This paper audits seven datasets with four readiness checks, finding that three pass all four while four fail at least one.

  • Problem

    Evidence about whether public educational prediction benchmarks are structurally reliable enough to support generalizable modeling claims is limited before modeling begins.

    Source: §1 · p. 1, §5 · p. 13

  • Method

    The paper audits seven datasets with four readiness checks, applying repeated resampling, permutation-null, and group-holdout procedures uniformly.

    Source: §3 · p. 5, §5 · p. 13

  • Results

    Three of seven datasets passed all four readiness checks, while four failed at least one; apparent signal often did not survive stronger cross-group generalization tests.

    Source: §5 · pp. 13–14

  • Takeaways & Limitations

    The protocol exposes benchmark reliability problems before benchmark claims are treated as evidence of generalizable educational AI performance.

    Source: §5 · p. 14

    The seven-dataset gradient is partly confounded by sample size, feature-target proximity, and task-construct differences, and operational thresholds remain uncalibrated across a large benchmark corpus.

    Source: §4.2 · p. 12

Abstract

from arXiv · show

Across seven public educational prediction datasets, three passed all four pre-modeling reliability checks; the remaining four either failed group-aware generalization tests or lacked the provenance metadata needed to run them. One dataset was initially classified as failing but corrected after excluding group-identifier features from the holdout matrix, demonstrating that the audit can distinguish genuine cross-group confounding from feature-encoding artifacts. Each dataset was audited before model optimization using four checks: baseline gap, split instability, null separation, and metadata adequacy under group-aware holdout. The dominant failure mode was not weak iid performance alone but cross-group fragility: in the clearest case, UCI Student declined from iid R-squared 0.242 to group-holdout R-squared -0.097, while Higher Ed collapsed from 0.041 to -8.79. Increasing model complexity did not remove this pattern: ensemble models improved structurally sound datasets but amplified instability or failed under group holdout on fragile ones. An exploratory cross-dataset comparison further showed that stronger profiles clustered in larger, richer-grouped, performance-proximal datasets, while random-split performance severely overstated deployable signal in fragile datasets. Classification-metric sensitivity analyses reached the same substantive conclusions. The results show that benchmark reliability in educational AI is constrained less by algorithm choice than by data structure, group heterogeneity, and evaluation design. A reusable pre-modeling audit offers a minimum quality gate before public educational datasets support strong benchmark or deployment claims.

1 Introduction · 2 Materials and Methods

The paper audits seven public educational prediction datasets to test whether their benchmark claims withstand reliability and group-generalization checks. Three pass all four dimensions, and the authors identify data structure and evaluation design—not insufficient model sophistication—as the main threat to benchmark validity.

1 Introduction

  • Nested educational groups can let models learn contextual shortcuts, while shared automated pipelines for LLM-derived features and targets can produce artifact-driven predictive performance.The introduction frames pre-modeling reliability checks as important because benchmark audits remain uncommon in educational prediction.Source: §1 · p. 1
  • Three of seven datasets pass all four audit checks, pointing to data structure and evaluation design rather than insufficient model sophistication as the main threat to benchmark validity.The authors also present the reusable audit as a precondition for making strong benchmark claims about future datasets.Source: §1 · p. 2

2.1 Audit Protocol

  • The four-dimension audit tests baseline advantage, split stability, separation from a permutation null, and whether metadata permits group-aware validation.It assigns structured readiness profiles from Fragile (0/4) to Strong (4/4), rather than aggregating dimensions into a single score.Source: §2.1 · p. 2, §1 · p. 2

2.2 Operational Decision Rules

  • Baseline advantage requires positive ∆MAE and a beat rate ≥0.90 across ≥50 splits; split instability fails at I ≥1, while group-holdout adequacy requires retaining ≥50% of iid R2.The thresholds are empirically motivated, not universal, and the reported profiles remain robust to threshold variations within ±20%.Source: §2.2 · p. 2

2.3 Datasets

  • The seven-dataset sample spans N = 145 to 32,593, multiple educational settings and target types, and grouping structures from 2 to 22 groups, plus one dataset without grouping metadata.It is intended as a stress test across common educational prediction dataset types, not exhaustive coverage of educational AI.Source: §2.3 · pp. 3–4

2.4 Models and Evaluation

  • The audit compares a mean-only baseline, linear and ridge regression, Random Forest, and Gradient Boosting, using linear regression as the primary detector of weak signal and group-sensitive leakage.Across 100 random 80/20 splits, it evaluates performance and instability, estimates permutation nulls on 30 splits, and uses leave-one-group-out holdout where metadata are available. UCI Student’s G1 and G2 features are excluded to prevent target-proxy leakage.Source: §2.4 · p. 4
  • After removing group-identifier features, xAPI-Edu’s linear group-holdout R2 rises from −2.03 to 0.484, changing its profile from Mostly Passing (3/4) to Strong (4/4).The correction shows that the audit can distinguish genuine cross-group fragility from artifacts caused by group-collinear feature encoding.Source: §2.4 · p. 4

2.5 Classification-Metric Sensitivity Check

  • For four datasets with binary or ordinal targets, the sensitivity analysis evaluates classification metrics under iid resampling and group-aware holdout.It reports accuracy, macro-averaged AUC-ROC, and an accuracy-based instability ratio to test whether audit conclusions depend on metric choice.Source: §2.5 · p. 5

3 Results

The audit found that cross-group generalization—not random-split performance alone—separated structurally sound educational datasets from fragile ones. Removing group-identifier features corrected xAPI-Edu’s apparent failure, while classification-metric checks confirmed the substantive conclusions.

3.1 Cross-Dataset Audit Summary

  • Three of six testable datasets showed cross-group failure, while OULAD, Dropout, and corrected xAPI-Edu passed all four audit dimensions.MM-TBA could not be assessed for group-aware generalization because grouping metadata were unavailable; four of seven datasets failed metadata adequacy. The audit summary reports the four-dimension scores and readiness profiles.Source: Table 2 · p. 5, §3.1 · p. 5, Figure 2 · p. 6

3.2 Per-Dataset Results

  • OULAD, Dropout, and xAPI-Edu had positive group-holdout R2, while Entrance Exam, UCI Student, Higher Ed, and MM-TBA showed distinct forms of fragility.OULAD’s group-holdout R2 was 0.410 across 22 cohorts, Dropout’s was 0.527 across 16 of 17 course programs, and xAPI-Edu’s was 0.484 after correcting the Topic-encoding artifact. UCI Student fell from iid R2 = 0.242 to group-holdout R2 = −0.097; Higher Ed had linear group-holdout R2 = −8.79 and RF = −13.0. MM-TBA lacked grouping metadata and had iid R2 = −0.067 despite a MAE-based beat rate of 0.80.Source: §3.2 · pp. 7–8

3.3 Cross-Dataset Structural Patterns and iid-to-Group Collapse

  • Strong datasets had small iid-to-group gaps, whereas metadata-adequacy failures showed larger gaps that could make random-split performance misleading for unseen groups.The exploratory comparison linked stronger profiles descriptively to more usable groups and features closer to the outcome construct; with only seven datasets, it is not a formal meta-regression. Higher Ed fell from iid R2 = 0.041 to group R2 = −8.79, UCI Student from 0.242 to −0.097, and Entrance Exam from 0.438 to −0.060.Source: §3.3 · p. 8, Figure 5 · p. 9

3.4 Model-Complexity Ablation

  • Ensemble models amplified instability on fragile MM-TBA, improved iid performance but still collapsed under group holdout on Higher Ed, and improved structurally sound datasets without introducing instability.On MM-TBA, instability rose from 2.2 to 4.6; on Higher Ed, ensemble instability fell from 0.90 to 0.18–0.23 and iid R2 rose from 0.04 to 0.52, but group-holdout performance still collapsed. On OULAD, RF MAE was 0.193 versus linear MAE 0.296.Source: §3.4 · p. 8

3.5 Classification-Metric Sensitivity Check

  • Classification-metric checks retained Strong profiles for OULAD, Dropout, and xAPI-Edu, while Entrance Exam remained the weakest and fragile case.The Strong datasets had iid accuracy 0.74–0.91, Iacc ≤0.14, and group-holdout AUC ≥0.86. Entrance Exam had Logistic iid accuracy 51.2% on a 4-class task against a 30.5% majority baseline, and group-holdout AUC = 0.68–0.73; its low accuracy partly reflects the difficulty of 4-class ordinal prediction.Source: Table 4 · p. 10, §3.5 · p. 10

3.6 Feature-Attribution Stability

  • Feature attributions were most stable on OULAD and more variable on UCI Student and Higher Ed, consistent with—but not validating—their audit profiles.OULAD’s top-10 linear coefficients had SD ≤0.007 with zero sign flips; Higher Ed coefficient SD reached 0.134 and RF importance CV ranged from 0.18 to 0.41. This exploratory analysis used IID splits only and does not establish a causal link to cross-group instability.Source: §3.6 · p. 10

4 Discussion

Across the seven audited datasets, three passed all four checks; cross-group fragility, rather than weak iid performance alone, was the dominant failure mode. The conclusions remained substantively unchanged across model complexity and classification-appropriate metrics.

4 Discussion

  • Three of seven datasets passed all four checks, while cross-group fragility—not weak iid performance alone—was the dominant reliability failure.The study cautions that this is an exploratory audit and should not be over-generalized to all educational prediction benchmarks.Source: §4 · p. 11

4.1 Findings

  • Higher Ed had R2 < −8 across all models, while UCI Student’s group-holdout R2 ranged from −0.097 to 0.018; MM-TBA lacked grouping metadata.Entrance Exam’s linear-model failure (R2 = −0.060, retention −14%) was partly remediable (RF R2 ≈0.32), while xAPI-Edu passed after group-identifier features were removed.Source: §4.1 · p. 11
  • On Higher Ed, RF raised iid R2 from 0.04 to 0.52 but group-holdout R2 remained −13.0; on MM-TBA, instability rose from I = 2.2 to 4.6.Ensembles partly recovered Entrance Exam’s group-holdout performance (RF R2 ≈0.32 versus linear −0.060), while structurally sound datasets gained without added instability.Source: §4.1 · p. 11
  • Stronger profiles clustered in larger datasets with richer grouping and performance-proximal features, but group-holdout severity also depended on group count and heterogeneity.Despite similar sample sizes, UCI Student (N = 649; iid R2 = 0.24) showed persistent cross-school confounding, whereas xAPI-Edu (N = 480; iid R2 = 0.61) passed; UCI’s two distinct schools contrast with xAPI-Edu’s 12 topics on a common platform.Source: §4.1 · p. 11
  • UCI Student’s iid R2 = 0.242 fell to school-holdout R2 = −0.097, while Higher Ed’s fell from 0.041 to −8.79 across all models.Entrance Exam’s gap was narrower (0.438 to −0.060) and partly closed by ensembles; random-split results can therefore overstate performance in unseen educational contexts.Source: §4.1 · p. 11
  • The audit is a minimum quality gate, with different remediation paths: fragile datasets need provenance or grouping review, while partly passing datasets need broader group evidence and group-aware validation.For example, MM-TBA needs teacher- or classroom-level identifiers; Higher Ed needs more courses and larger held-out samples; Entrance Exam may benefit from ensembles, whereas UCI Student needs stronger cross-school evidence. Even strong datasets should report group-aware results for deployment-oriented comparisons.Source: §4.1 · pp. 11–12
  • Group-holdout collapse does not by itself establish confounding: it may reflect distribution shift, and small group counts make leave-one-group-out evaluation an extrapolation stress test.The proposed diagnostic is to compare group-level feature distributions when all models collapse; when regularized or ensemble models recover from linear-model collapse, overfitting is a more parsimonious explanation.Source: §4.1 · p. 12

4.2 Limitations

  • The audit’s thresholds and profile labels remain provisional: permutation estimates use 30 of 100 splits, group counts vary, and equal weighting may not fit every domain.Seven datasets are insufficient to calibrate thresholds or establish generality; borderline assignments could change under alternative cutoffs.Source: §4.2 · p. 12

5 Conclusion

Across seven public educational datasets, only OULAD, Dropout, and xAPI-Edu passed all four readiness checks; the other four failed at least one, usually metadata adequacy. The audit finds that apparent signal often fails stronger cross-group generalization tests.

5 Conclusion

  • OULAD, Dropout, and xAPI-Edu passed all four readiness checks, whereas the remaining four datasets failed at least one criterion and most failed metadata adequacy.The audit asks whether public educational prediction benchmarks are structurally reliable enough to support generalizable modeling claims; across the seven datasets, the answer was usually no.Source: §5 · p. 13
  • Higher Ed and UCI Student showed structural fragility, Entrance Exam had linear-model-only failure, xAPI-Edu passed after feature-artifact correction, and MM-TBA lacked grouping metadata.Higher Ed had R2 < −8 across all models, while UCI Student had near-zero holdout; on Entrance Exam, ensembles partly remediated −14% retention. Ensembles amplified instability or collapsed under group holdout on fragile datasets, but Entrance Exam RF R2 ≈0.32 versus linear −0.060.Source: §5 · p. 13
  • Stronger audit profiles clustered in larger, better-grouped releases with proximal features, and classification-metric sensitivity analysis confirmed profile robustness to metric choice.The audit assigns profiles from Fragile to Strong, with each level corresponding to different remediation needs.Source: §5 · p. 13
  • Higher Ed fell from 0.041 iid to −8.79 group holdout, UCI Student from 0.242 to −0.097, and Entrance Exam from 0.438 to −0.060.The iid-to-group collapse gap was small for Strong datasets but substantial for datasets with metadata-adequacy failures, showing that random-split performance can overstate deployable signal.Source: §5 · p. 13
  • The reusable checklist tests baseline gap, split instability (I < 1), null separation across admissible splits, and metadata adequacy under group-aware holdout.It also checks whether provenance fields needed for leakage-resistant validation are present in the release.Source: §5 · p. 13
  • Passing all four checks supports cautious benchmark claims; one or two failures may warrant targeted remediation, while three or more indicate a research-stage resource needing substantial revision.The protocol exposes structural reliability problems before benchmark claims are treated as evidence of generalizable educational AI performance.Source: §5 · p. 14

Author contributions statement

Y.M. and L.Z. shared conceptualization, methodology, validation, data curation, and manuscript review; other responsibilities were divided between them.

Author contributions statement

  • Y.M. led formal analysis, investigation, and the original draft, while L.Z. led software, visualization, and supervision.Y.M. and L.Z. shared conceptualization, methodology, validation, data curation, and writing—review and editing; all authors reviewed and approved the manuscript.Source: Author contributions statement · p. 14

Funding

The study was funded by an Education Department of Hunan Province, China grant supporting empirical research on college English classroom environments and student engagement.

Funding

  • The study received funding from the Education Department of Hunan Province, China, under project HNJG-2022-0608 on classroom environment changes and student engagement.The audit methodology supports empirical evaluation of classroom engagement prediction models funded under the grant.Source: Funding · p. 14

S1 Model-Complexity Ablation

The model-complexity ablation compares instability and beat rate across linear and nonlinear models over repeated data splits.

S1 Model-Complexity Ablation

  • Supplementary Table S1 reports instability ratio (I) and beat rate for linear and nonlinear models across 100 repeated 80/20 splits.Source: Table S1 · p. 17

S2 EDA Summary

All seven datasets had zero missing values, and ordinary exploratory data analysis did not reveal red flags that would pre-emptively disqualify the four datasets that failed the audit. Their fragility emerged only through the four-dimension audit.

S2 EDA Summary

  • MM-TBA, Entrance Exam, Higher Ed, and UCI Student each failed at least one audit dimension, despite showing no distinctive EDA red flags.The cited red flags include extreme imbalance and redundant features.Source: Table S2 · p. 18
  • All seven dataset releases contained zero missing values.Source: Table S2 · p. 18
  • The four failing datasets’ fragility was revealed through the four-dimension audit, not by pre-emptive EDA screening.Source: Table S2 · p. 18

S3 Train-Test R2 Gap

The train-test R2 gap analysis indicates that UCI Student’s group-holdout failure is not attributable to ordinary overfitting, while OULAD shows small train-test and group-holdout gaps.

S3 Train-Test R2 Gap

  • UCI Student’s linear-model train-test R2 gap was 0.105, small relative to the iid-to-group-holdout R2 decline from 0.242 to −0.097.This supports the conclusion that its group-holdout failure is not attributable to ordinary overfitting.Source: Table S3 · p. 19
  • OULAD had small train-test and iid-to-group-holdout gaps.Source: Table S3 · p. 19

S4 Group-Holdout Uncertainty

Across-group group-holdout performance showed uniform collapse for UCI Student, while OULAD generalized positively but with variation across cohorts.

S4 Group-Holdout Uncertainty

  • UCI Student’s linear-model group-holdout R2 SD was 0.109 (−0.206 to 0.012), while OULAD’s across-cohort SD was 0.162 (0.041 to 0.654).The ranges indicate uniform collapse for UCI Student across 2 schools and positive but variable generalization for OULAD across 22 cohorts.Source: Table S4 · p. 20

S5 Linear-Model Coefficient Stability

Across 100 repeated 80/20 splits, OULAD coefficients were directionally stable, while UCI Student and Higher Ed showed occasional sign flips and greater coefficient uncertainty.

S5 Linear-Model Coefficient Stability

  • OULAD’s top-10 features had zero sign flips and low SD across 100 repeated 80/20 splits.The sign-flip rate measures the proportion of splits where a coefficient’s sign differs from the sign of its mean coefficient.Source: Table S5 · p. 21
  • UCI Student showed 3% sign flips for f9 and 4% for f2; Higher Ed showed 2% for f9, with SDs indicating greater uncertainty.UCI Student’s SDs were proportionally larger than OULAD’s; Higher Ed’s SDs reflected greater uncertainty about effect magnitude.Source: Table S5 · p. 21

S6 Random Forest Importance Stability … S9 Supplementary Figures

Supplementary analyses show that feature-importance stability and cross-model attribution agreement are strongest on OULAD, while supplementary figures illustrate split-level performance, instability, and sensitivity to audit cutoffs. The feature-name mapping confirms that group-identifier columns were excluded from the attribution analyses.

S6 Random Forest Importance Stability

  • OULAD’s top-10 Random Forest importance CVs are all below 0.12, compared with 0.08–0.20 on UCI Student and 0.18–0.41 on Higher Ed.These coefficients of variation across 100 repeated 80/20 splits indicate greater split sensitivity in the smaller datasets.Source: Table S6 · p. 22

S7 Feature-Name Mapping

  • The feature-name mapping identifies zero-indexed preprocessed columns and excludes school and Course ID group-identifier columns from the attribution feature matrix.The mapping covers UCI Student and Higher Ed group identifiers, as well as the three datasets included in the attribution stability analysis.Source: Table S7 · p. 23

S8 Cross-Model Attribution Agreement

  • OULAD’s cross-model top-10 attribution agreement was ρtop10 = 0.817 (SD 0.065), while UCI Student and Higher Ed showed weaker, more variable agreement.The comparison uses linear-model absolute standardized coefficients and Random Forest impurity-based importances across 100 repeated 80/20 splits.Source: Table S8 · p. 24

S9 Supplementary Figures

  • Supplementary Figure S1 shows MM-TBA and Higher Ed with mean linear-model R2 near or below zero and relatively wide spread across 100 random 80/20 splits.The figure complements Table 2 and notes that these datasets have Fragile / Partial classifications.Source: Figure S1 · p. 25
  • Strong datasets OULAD and Dropout have instability ratios I < 0.03, whereas MM-TBA has a broad, negative-centered distribution with I = 2.21.The figure plots linear-model R2 across 100 random 80/20 splits and reproduces the Table 2 instability pattern.Source: Figure S2 · p. 25
  • At the chosen instability cutoff I = 1.0, MM-TBA is flagged; stricter cutoffs below about 0.9 also flag Higher Ed, while the chosen beat-rate cutoff 0.90 flags two datasets.More lenient beat-rate cutoffs below about 0.84 reduce the flagged count; MM-TBA remains the weakest case, and Higher Ed is the only borderline dataset near the operational thresholds.Source: Figure S3 · p. 26
Loading 2609.29625v1…