Source-linked AI summary
Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements
Oliver G. B. Garrod, Robin A. A. Ince, Meng Liu, Mohamed Huti, Moritz Boos, Amy Waldock, Dominic Andrews, Paul Atherton
TL;DR
Existing educational filters often reduce educational value to one scalar, despite differences in accuracy, engagement, structure, audience suitability, and instructional purpose. Edu-QuRating distills pairwise LLM preferences into reusable criterion-specific scorers and applies them to corpus filtering and GRPO reward modeling. The resulting scores recover held-out preferences and support higher observed aggregate pre-training accuracy than the FineWeb-Edu baseline, while combined rewards improve judged pedagogical quality and instruction following.
Problem
Single-score educational filters do not distinguish dimensions such as accuracy, engagement, pedagogical structure, audience suitability, and foundational-literacy support in already educational corpora.
Method
Edu-QuRating defines educational rubrics, distills LLM-labelled pairwise document preferences into sequence-classification Edu-QuRaters, and uses their scores for filtering and GRPO rewards.
Results
Edu-QuRating-based pre-training mixtures achieved higher observed aggregate accuracy than the FineWeb-Edu baseline, while combined Edu-QuRater and answer-structure rewards improved judged pedagogical quality and instruction following.
Takeaways & Limitations
Explicit educational criteria can guide scalable selection of educational pre-training data and provide reference-answer-free feedback for generated educational responses.
Takeaways & Limitations
The findings are limited by upstream-corpus coverage, single-run pre-training comparisons, a small-scale GRPO evaluation, and evaluation criteria overlapping with the FL-Teacher rubrics.
Abstract
from arXiv · showhide
Educational data filters have become a practical way to improve language-model pre-training, but most filters treat educational value as a single scalar property. This may be too broad for some applications, especially if the data set already features a high density of educational material. Useful learning material needs to be accurate, engaging, well structured, and appropriate for the intended audience and application (e.g. learner- vs teacher-facing). Following QuRating (Wettig et al. 2024), we introduce Edu-QuRating: a pipeline for multi-dimensional educational data scoring and curation. Edu-QuRating defines education-specific rubrics, uses an LLM judge to label sampled document pairs and distills those pairwise preferences into reusable Edu-QuRaters, which can score individual text chunks on a set of educational criteria. Across two sequence-classification base models and six educational criteria, the best Edu-QuRater recovers held-out GPT-4.1-mini pairwise judgements with mean accuracy 0.917. We then apply the resulting scorers in two applications. First, we investigate the potential of Edu-QuRaters for corpus filtering to improve pretraining of small language models. We scored 322.25M FineWeb-Edu-Fortified documents to obtain a filtered pre-training mixture. In matched single-run pre-training comparisons, models trained with Edu-QuRating-based mixtures reached higher observed aggregate accuracy across nine benchmarks than the FineWeb-Edu baseline, with gains concentrated in particular tasks. Second, we used Edu-QuRater scores as reward terms for GRPO post-training. In held-out pairwise judge evaluations, combining Edu-QuRater and answer-structure rewards produced responses preferred to the Qwen3-4B base model on both pedagogical quality and instruction following.
1 Introduction
Educational filtering improves language-model training, but a single educational score cannot capture the dimensions that determine whether material fits particular learning goals. Edu-QuRating addresses this by distilling pairwise LLM judgements into inspectable, reusable scores for distinct educational criteria.
- Educational filtering can improve smaller language models on knowledge- and reasoning-heavy benchmarks, making data curation a central training choice.
- A document may be engaging but inaccurate, well structured but unsuitable for a target audience, or educational without supporting foundational literacy.
- Edu-QuRating specializes QuRating’s pairwise preference distillation for fine-grained educational judgements on already educational data.
- Its rubrics cover factual accuracy, pedagogical structure, engagement, learner-level suitability, and 14 additional foundational-literacy dimensions.
- The resulting scores are evaluated for preference recovery, corpus filtering in small-language-model pre-training, and reward modeling for GRPO educational-response fine-tuning.
2 Related Work
Prior work established scalable web-data curation, distilled LLM pairwise judgements into reusable scorers, and identified multiple components of educational usefulness. Edu-QuRating combines these lines for educational corpus filtering and open-ended response rewards.
- Web-corpus research shows that filtering, deduplication, mixture design, and documented curation can strongly influence language-model performance and reproducibility.
- QuRating samples document pairs, obtains LLM preferences, distills them into single-document QuRaters, and applies the scores to scalable data selection.
- The QuRater pipeline replaces costly repeated LLM evaluation with a smaller model that predicts reusable Bradley–Terry scores for new documents or responses.
- Reading research characterizes educational usefulness through interacting components including decoding, linguistic comprehension, word recognition, and language comprehension.
- Research on small language models links capable systems to carefully curated or textbook-quality pre-training corpora and experimental training methodology.
- Unlike verifiable-answer settings, educational response quality is open-ended, motivating learned rewards for pedagogical structure, engagement, factual accuracy, and foundational literacy.
3 Developing Edu-Qurater Scoring Models
Edu-QuRating distills criterion-specific pairwise educational judgements into reusable single-text scorers. The models recover held-out preferences accurately and produce structured, inspectable variation across educational materials and dimensions.
- Pipeline overview: Edu-QuRating uses criterion-specific LLM pairwise preferences to train multi-output sequence-classification models that score texts independently.The core model predicts six educational dimensions, while separate models target student-facing and teacher-facing foundational literacy.
- Educational dimensions: The Core-Ed rubric measures overall educational orientation, learner-level suitability, factual accuracy, lesson engagement, and pedagogical structure.These dimensions are intended to distinguish texts that differ in correctness, sequencing, accessibility, or engagement.
- Educational dimensions: The foundational-literacy rubric families provide seven student-facing and seven teacher-facing output dimensions alongside six core educational dimensions.Student-facing scoring treats text as practice material, whereas teacher-facing scoring evaluates whether it models explicit teaching practices.
- Pairwise preference distillation: 0.917 mean accuracy is achieved by the best Edu-QuRater on held-out GPT-4.1-mini pairwise preferences across six criteria.Both base models exceed 0.86 accuracy on every criterion, and Gemma-3-4B-PT improves uniformly over Sheared-LLaMA-1.3B.
- Pairwise preference distillation: 200k pairwise examples provide a practical training scale, reducing validation loss to 0.245 versus 0.307 at 20k examples.Increasing the dataset to 300k or 400k examples yields almost no additional gain in the reported baseline setup.
- Out-of-sample score analysis: Across roughly 15k external educational materials, scores vary by metadata and remain distinct across dimensions rather than reproducing one generic educational-quality signal.Overall Education Level and Primary-level Suitability correlate most strongly with education-level metadata, while four principal components explain 95% of score variance.
4 Application: Improving small model performance with an Edu-QuRater filtered pre-training corpus
Edu-QuRating scores 322.25M FineWeb-Edu-Fortified rows and filters them using multidimensional educational-quality criteria before matched small-model pre-training. At 30k steps, all Edu-QuRating mixtures exceed the FineWeb-Edu baseline in aggregate accuracy, but gains are concentrated in selected tasks.
- Corpus scoring and filtering: 322.25M FineWeb-Edu-Fortified rows were scored with a Gemma-3-4B-PT Edu-QuRater after chunking documents into 512-token segments and averaging chunk scores.The run covered all 95 Common Crawl subsets.
- Corpus scoring and filtering: The filtering conditions used Factual Accuracy, Lesson Engagement, and Pedagogical Structure, while educational-level scores served as audience-targeting metadata.The three quality dimensions were thresholded for corpus filtering; level dimensions were not treated as standalone quality signals.
- Corpus scoring and filtering: The conjunctive 50-50-50 rule retained 51.94M rows, or 16.12% of the scored corpus, by requiring all three quality scores to exceed their dataset medians.A separate stricter-pedagogy condition used 41.6th-, 34.4th-, and 69.6th-percentile thresholds for accuracy, engagement, and structure.
- Matched pre-training comparison: Matched mixtures fixed FineMath-3Plus at 10% and Stack-Edu-Python at 20%, varying only the remaining 70% web-text component.The web-text slice was FineWeb-Edu, Edu-QuRating-filtered data, stricter-pedagogy filtered data, or a stricter-pedagogy/DataComp-LM mixture.
- Matched pre-training comparison: At 30k steps, the FineWeb-Edu baseline reached 0.3806 mean accuracy across nine tasks, versus 0.3903 for regenerated 50-50-50, 0.3962 for stricter pedagogy, and 0.3961 for Edu-QuRating plus DCLM.The stricter-pedagogy and mixed conditions were effectively tied overall, but the former led education-related tasks while the latter led general tasks.
- Task-level results: Task-level gains were concentrated in ARC-CF and HellaSwag, while MMLU-family improvements were near zero, CommonsenseQA was mixed, and Pedagogy CDPK depended on filtering variant.OpenBookQA, BoolQ, and WinoGrande improved for all Edu-QuRating conditions, but by smaller margins.
5 Application: Edu-Quraters as GRPO Reward Models
Edu-QuRater scores were tested as GRPO rewards for educational responses, alongside answer-structure rewards and held-out evaluation of pedagogical quality and instruction following. Combining both reward types achieved the strongest observed balance across the two evaluation criteria.
- Reward configurations: GRPO fine-tuning optimized responses using answer-structure, Edu-QuRater, or combined reward functions.Edu-QuRater rewards included Core-Ed and FL-Teacher score combinations, while the combined configuration also used reference-based answer-structure feedback.
- Evaluation: The evaluation used 73 held-out examples and pairwise judge comparisons against Qwen3-4B on pedagogical quality and instruction following.Items received education-expert quality control and were tagged for relevance to the FL-Teacher dimensions.
- Results: 68.24% pedagogical-quality and 64.86% instruction-following win rates were achieved by answer-structure rewards alone against Qwen3-4B.This condition served as a baseline for subsequent reward configurations.
- Results: 77.70% pedagogical-quality win rate paired with 40.54% instruction-following win rate for Edu-QuRating rewards alone.The pattern is consistent with proxy-reward over-optimization, although the evaluation does not establish that mechanism.
- Results: 81.08% pedagogical-quality win rate and 68.24% instruction-following win rate were obtained by combining Edu-QuRating and answer-structure rewards.Both reported win rates were above parity with the Qwen3-4B base model.
6 Discussion
The discussion presents Edu-QuRating as a scalable framework for educational quality assessment, metadata labeling, pre-training data selection, and reward modeling. It also emphasizes scope limitations and the need for real-world educational validation.
- Contributions: Edu-QuRating adapts pairwise preference distillation to educational data curation and applies the resulting scorers to pre-training filtering and GRPO reward modeling.The framework uses explicit educational criteria to score existing educational data and generated responses.
- Model-training applications: All tested Edu-QuRating pre-training mixtures exceeded the FineWeb-Edu baseline in observed aggregate accuracy, while combined rewards improved both judged pedagogical quality and instruction following.Edu-QuRater rewards can be computed without reference answers.
- Broader applications: The framework supports quality assessment and metadata labeling for selecting content according to factual accuracy, pedagogical structure, engagement, audience, and instructional purpose.The discussion highlights learner- and teacher-facing foundational-literacy rubrics as examples of use-specific curation.
- Limitations: The pre-training evidence is limited by one run per mixture and an incomplete set of mixture combinations, so it does not establish the best possible mixture.The reported best mixture was only the strongest among those tested.
- Limitations: Real-world classroom benefits were not evaluated, leaving educator assessments and learner-outcome studies as important future evidence.The authors identify this validation as beyond the scope of the proof-of-concept technical report.
A External Score Profiles
The external score-profile figures visualize how Edu-Qurater scores are distributed across educational metadata and how multiple rubric dimensions relate to one another. The supplied captions specify the groupings and dimensionality but not substantive outcomes.
- Student-facing profiles: The FL-Student Edu-Qurater score distributions are divided by education level and material type metadata.Panel A uses Education Level metadata, while panel B uses Material type metadata.
- Teacher-facing profiles: The FL-Teacher Edu-Qurater score distributions are divided by education level and material type metadata.The figure uses the same two metadata groupings for teacher-facing scores.
- Cross-dimension structure: The correlation matrix and PCA summarize relationships among twenty combined dimensions from Core-Educational, Student-, and Teacher-Facing Foundational Literacy Edu-Quraters.The analysis covers document-weighted chunk scores from 15k sourced educational materials.