Source-linked AI summary
PEARL: A Task-Aware Framework for Evaluating Differentially Private Synthetic Educational Data
Xianghui Meng, Yujing Zhang, Jionghao Lin
TL;DR
Personalized learning needs learner data, but those data are privacy-sensitive and synthetic releases may not preserve information required for specific educational tasks. PEARL combines validity, privacy, predictive-usefulness, and task-suitability checks into explicit approval decisions; across 96 settings, only 12 releases passed all applicable checks, while sequential and knowledge-tracing failures showed that privacy protection alone does not ensure task usefulness.
Problem
Existing evaluations assess privacy and predictive usefulness separately, leaving unclear whether DP synthetic learner data remain suitable for their intended educational tasks.
Method
PEARL evaluates DP synthetic educational datasets through validity, privacy, predictive usefulness, task-specific suitability, and applicable subgroup-fairness checks, recording rejection reasons.
Results
Only 12 of 96 releases passed all applicable PEARL checks, while tested synthetic knowledge-tracing datasets failed to preserve meaningful next-response patterns.
Takeaways & Limitations
Privacy protection is necessary but does not ensure preservation of important outcome groups, subgroup performance, learning-sequence structure, or educational task usefulness.
Takeaways & Limitations
Findings may not generalize beyond six public datasets, a limited set of generation methods, and three broad applications, and PEARL thresholds are study-specific.
Abstract
from arXiv · showhide
Personalized learning systems rely on real learner data, including performance, behavior, and demographic information, but these data are highly privacy-sensitive. Differentially private (DP) synthetic data can support system development and educational research while reducing exposure of individual learners. Existing evaluations, however, assess privacy and predictive usefulness separately, without determining whether synthetic learner data remain usable for the intended personalized learning task. We introduce PEARL (Privacy-Equivalence Audit and Release Ledger), which approves a DP synthetic educational dataset only when it passes all required checks of validity, privacy protection, predictive usefulness, and suitability for the intended educational task, while recording why each rejected dataset fails. Across 96 study settings, each defined by a dataset, data-generation method, privacy budget, and random seed, only 12 produced synthetic datasets that passed all applicable PEARL checks. Many privacy-protected datasets were rejected for omitting important outcome groups, such as withdrawn students, or for failing to preserve the order of learning activities. Fairness analysis further showed that some datasets passing privacy and predictive-usefulness checks still yielded unequal at-risk prediction performance across groups defined by disability and socioeconomic background. Moreover, Deep Knowledge Tracing and Self-Attentive Knowledge Tracing learned no meaningful next-response patterns from any tested synthetic knowledge-tracing dataset, showing that privacy protection alone does not guarantee usefulness for dropout prediction, knowledge tracing, or adaptive tutoring.
1. INTRODUCTION
PEARL addresses the gap between protecting learner privacy and preserving the information needed for specific personalized learning tasks. It evaluates DP synthetic educational data across multiple checks and records explicit approval or rejection reasons.
- Research gap: Existing evaluations report privacy and predictive usefulness separately, without determining whether a release supports its intended personalized learning task.This is especially problematic when uncommon outcomes, such as early withdrawal, are omitted.
- Framework: PEARL approves synthetic data only when validity, privacy, predictive usefulness, and intended-task suitability checks all pass.It also examines subgroup fairness when protected attributes are present and records the failed check for each rejection.
- Evaluation: PEARL evaluates six educational datasets across 96 settings defined by dataset, generation method, privacy budget, and random seed.The settings span tabular records, MOOC clickstreams, and knowledge-tracing response sequences.
- Contribution: The framework integrates validity, privacy protection, predictive usefulness, task suitability, subgroup fairness, and a release ledger into one transparent decision.Its evaluation is designed to identify failures such as omitted outcome groups, distorted subgroup performance, unreliable patterns, and lost sequence information.
2. RELATED WORK
Educational synthetic-data evaluation must address privacy and utility challenges specific to rare outcomes, ordered learning histories, and knowledge-tracing relationships. Existing parallel metrics do not establish whether a particular release is suitable for its intended use.
- Differential privacy: Differential privacy limits an individual record’s influence on a release, but stronger privacy generally adds noise and can reduce fidelity to original patterns.Existing synthesizers include count-based methods that perturb frequency tables or low-dimensional statistics.
- Educational data challenges: Student datasets may lose important outcome groups such as dropout, while sequential logs require preserving ordered learner histories.Knowledge tracing additionally depends on relationships among skills, responses, and subsequent performance.
- Privacy evaluation: Membership inference attacks test whether a record contributed to a released synthetic dataset or model, commonly using nearest-neighbor similarity and AUC.An AUC of 0.5 represents chance-level discrimination.
- Evaluation gap: Prior privacy–utility benchmarks place membership inference, downstream prediction, and fidelity metrics such as TSTR side by side.These metrics do not determine whether a specific release should be certified for a particular educational task.
3. METHOD
The method evaluates six educational datasets with tabular and sequential DP synthesizers through PEARL’s validity, privacy, utility, and task-specific checks. Approval requires passing every applicable check, while completed but failing runs remain rejected releases.
- Datasets: The evaluation covers UCI 697, UCI 856, OULAD, ACT-MOOC, ASSISTments, and EdNet across tabular, clickstream, and knowledge-tracing data.The primary datasets include 101 records in UCI 856, 2,100 records in UCI 697, 1,820 OULAD students, and 2,000 ACT-MOOC learners.
- Synthesizers: Tabular methods include DP-Histogram, DP-Marginals, MST, and DP-CTGAN, while sequential methods include DP-Markov and order-free DP-HistSeq.Tabular privacy budgets range from ε ∈ {0.5, 1.0, 2.0, 5.0, 10.0}; sequential budgets use ε ∈ {1.0, 5.0, 10.0}.
- PEARL protocol: PEARL assesses each synthetic dataset through sequential Validity, Privacy, Utility, and Task-Specific Suitability checks.The protocol also applies subgroup-fairness analysis to tabular datasets containing protected attributes.
- Privacy check: A release fails privacy if membership-inference AUC exceeds 0.55 or any synthetic record exactly copies an original record.The 0.55 threshold is a benchmark screening rule rather than a universal privacy guarantee.
- Utility check: The utility check trains models on synthetic data, tests them on held-out real data, and compares macro-F1 with models trained on real data.Releases are rejected for omitting target groups, producing no predictive performance, or lacking a usable real-data comparison.
- Study protocol: The study evaluates 96 settings, each defined by a dataset, generation method, privacy budget, and random seed.A completed run that fails an applicable check is counted as a rejected synthetic dataset rather than a missing result.
4. RESULTS
Across 96 study settings, PEARL approved 12 synthetic datasets, while rejected releases commonly failed privacy, predictive-usefulness, or task-specific checks. Task-aware evaluation exposed failures that standard predictive utility could miss, including missing outcome groups, distorted sequence order, and unequal subgroup performance.
- 4.2 RQ2: PEARL Certifies 12 Releases and Documents Where Rejected Releases Fail: 12 of 96 study settings produced synthetic datasets that passed all applicable PEARL checks.The approved releases included nine OULAD or UCI 697 tabular datasets and three ACT-MOOC datasets; no UCI 856 setting passed.
- 4.2 RQ2: PEARL Certifies 12 Releases and Documents Where Rejected Releases Fail: 45 settings failed privacy, 34 failed predictive usefulness, and 12 failed task-specific suitability, with failures allowed to overlap.Privacy failures reflected membership-inference AUC above 0.55; utility failures included missing outcome groups or near-zero performance; task failures reflected lost ordered skill–response patterns.
- 4.3 RQ3: Task-Specific Checks Separate Sequential Utility from Classifier Utility: ACT-MOOC DP-Markov retained 92.2%–99.7% of real-data classifier performance across privacy budgets and preserved sequential-model performance.Classifier retention ranged from 0.922 to 0.997, while GRUSeq retention ranged from 0.943 to 0.978.
- 4.3 RQ3: Task-Specific Checks Separate Sequential Utility from Classifier Utility: DP-HistSeq produced retention above 1.0 but was not considered stronger sequentially useful because it removes event order by design.The apparent performance advantage was treated as a label-related artifact rather than evidence of preserved sequence information.
- 4.3 RQ3: Task-Specific Checks Separate Sequential Utility from Classifier Utility: DKT and SAKT learned no meaningful next-response patterns from any tested synthetic knowledge-tracing dataset.Real sequences achieved macro-F1 of approximately 0.61; DP-HistSeq removed skill-practice order, while DP-Markov lacked paired skill–correctness information.
- 4.3 RQ3: Task-Specific Checks Separate Sequential Utility from Classifier Utility: Some releases passed privacy and predictive-usefulness checks but still produced unequal at-risk prediction across student groups.PATE-GAN at ε = 10.0 failed fairness in three of five seeds, with a mean worst-group recall gap of 0.244.
5. Discussion
PEARL’s discussion shows that synthetic-data approval must be tied to the intended educational task, because privacy and predictive usefulness do not ensure preserved outcomes, sequences, or fairness. The framework also has bounded generalizability because its evaluation settings, methods, datasets, thresholds, and privacy attacks were limited.
- Task-Aware Decisions Depend on Dataset Characteristics: Only 12 of 96 releases passed all applicable PEARL checks, with the ledger identifying privacy risks, missing outcome groups, low predictive usefulness, and lost learning-sequence information.The approval rate and rejection reasons show why task-aware auditing adds information beyond privacy protection alone.
- Predictive usefulness does not guarantee sequential suitability: Synthetic data can support sequential learning only when generation preserves the transitions required by the intended model.DP-Markov supported sequential learning on ACT-MOOC, whereas DP-HistSeq’s high retention could reflect simplified labels or artifacts after removing event order.
- Predictive usefulness does not guarantee sequential suitability: Neither DP-HistSeq nor DP-Markov preserved the paired skill-and-correctness information required by DKT and SAKT.The knowledge-tracing results indicate that predictive usefulness does not establish suitability for models requiring relationships among skills, responses, and subsequent performance.
- Fairness: Fairness evaluation should combine subgroup performance gaps with absolute subgroup performance, because a small gap can conceal uniformly poor predictions.PEARL could use both a maximum subgroup-gap threshold and a minimum subgroup-performance requirement, especially for early-warning systems.
- Implications for educational data use: Approval should apply to the evaluated synthetic dataset and specified task, not automatically to every dataset produced with the same method and privacy budget.The intended use, applicable privacy and fairness criteria, and task-specific performance requirements should be defined before evaluation.
- Limitations and future research: The evidence may not generalize beyond six public datasets, limited generation methods, three broad applications, and study-specific thresholds.The primary privacy check used nearest-neighbor membership inference, and stronger attacks were not applied to every setting.
6. Conclusion
The conclusion argues that differentially private synthetic educational data should be evaluated according to their intended task, not privacy protection alone. PEARL integrates task-specific preservation checks into a transparent approval process for deciding when learner data are usable.
- 6. Conclusion: Synthetic educational data should be evaluated according to the task they are expected to support, because privacy protection alone does not ensure preservation of groups, relationships, fairness, or learning sequences.PEARL provides a transparent basis for deciding when further data generation or evaluation is required.