Source-linked AI summary
Tuned Models of Peer Assessment in MOOCs
Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, Daphne Koller
TL;DR
MOOCs need scalable and accurate grading for complex open-ended work, yet peer grades can substantially differ from staff assessments. The paper develops probabilistic models that estimate and correct grader biases and reliabilities. These methods improve accuracy over median-based grading and connect grader quantities with educationally relevant variables.
Problem
MOOCs need scalable grading for complex open-ended assignments, but peer grading does not always match human-expert results.
Method
The paper develops probabilistic peer-grading models that estimate submission scores, grader biases, and grader reliabilities from observed assessments.
Results
Over 30% lower RMSE than median-based peer-grading baselines was achieved, while grader quantities also related to educationally relevant variables.
Takeaways & Limitations
Maintaining grader-specific bias, reliability, and uncertainty estimates can make large-scale peer grading more dependable, accurate, and fair.
Takeaways & Limitations
Selecting the best grader for a particular submission remains open and may depend on writing style and cultural or linguistic background.
Abstract
from arXiv · showhide
In massive open online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to human experts. In this paper, we develop algorithms for estimating and correcting for grader biases and reliabilities, showing significant improvement in peer grading accuracy on real data with 63,199 peer grades from Coursera's HCI course offerings --- the largest peer grading networks analysed to date. We relate grader biases and reliabilities to other student factors such as student engagement, performance as well as commenting style. We also show that our model can lead to more intelligent assignment of graders to gradees.
1. INTRODUCTION
MOOCs need scalable grading for complex assignments, but peer assessment can diverge substantially from expert grades. This paper models grader bias and reliability to improve accuracy and examine broader educational relationships.
- Motivation: 43% of HCI submissions received peer grades more than 10 percentage points from staff grades.Some submissions differed by as much as 70 percentage points.
- Contribution: Over 63,000 peer grades form the largest peer-grading networks analyzed to date.The dataset enables analyses that would be difficult to validate with smaller networks.
- Contribution: Over 30% lower RMSE than median-based peer-grading baselines was achieved by modeling grader biases and reliabilities.The probabilistic models estimate submission grades and grader-specific tendencies, then compensate for grader idiosyncrasies.
- Contribution: Maintaining submission-grade uncertainty estimates supports fairer scores by targeting similar confidence across students.Fairness is defined here as the system reaching a similar confidence level about every student’s grade.
- Contribution: Grader bias and reliability relate to educationally relevant variables, including grading quality, course participation, and formative and summative assessment.The paper examines both what predicts better grading and how peer grading affects later participation.
- Significance: Because peer grading is structurally similar across MOOCs and traditional classrooms, the findings may inform practices in both settings.The paper also characterizes peer-assessment dynamics in a newer, potentially more accessible educational setting.
2. DATASETS
The study analyzes two consecutive Coursera HCI offerings using calibrated, anonymized peer grading for weekly creative web-design assignments. The dataset includes thousands of students and staff-graded ground-truth submissions for validation.
- Data collection: Two consecutive Coursera HCI offerings supplied weekly peer assessments of creative web-design assignments.The courses were taught by Stanford professor Scott Klemmer.
- Peer-grading protocol: Calibration required students to correctly assess a training submission before grading classmates’ work.Each student evaluated five randomly selected submissions on every assignment.
- Peer-grading protocol: Peer grading was anonymized so students could not identify their evaluators or evaluatees.
- Course refinement: HCI2 introduced a modified rubric and separated English- and Spanish-language grader groups.The changes addressed shortcomings in the original scheme and concerns about non-native-language grading and patriotic grading effects.
- Sample: The English offerings included 3,607 students in HCI1 and 3,633 in HCI2 who submitted at least one assignment.
- Validation: Each assignment used 3–5 staff-graded ground-truth submissions that received about 160 student assessments on average.Students were not told which assigned submission belonged to the ground-truth set.
3. PROBABILISTIC MODELS OF PEER GRADING IN MOOCS
The paper formulates probabilistic peer-grading models that estimate true submission scores while compensating for grader bias and reliability. The models incorporate temporal bias patterns, relationships between student performance and grading ability, and uncertainty while addressing fairness and overfitting.
- Model formulation: The models estimate latent true scores, grader biases, and grader reliabilities from observed peer grades.Bias captures systematic inflation or deflation, while reliability reflects precision after bias correction.
- Model formulation: PG1 places priors on biases, true scores, and reliabilities, allowing individual grader tendencies to be estimated and corrected.The model assumes average grader bias is zero and uses distributions over latent variables.
- Model formulation: PG2 propagates grader bias across assignments because consecutive-assignment bias estimates have Pearson correlation 0.33, whereas reliability correlation is low.It normalizes grades to z-scores to remain robust to assignment-specific artifacts.
- Model formulation: PG3 links a student’s submission score to grading reliability, while constraining reliability through a single parameter to reduce overfitting.The model is motivated by the observed tendency for higher-scoring students to be somewhat more reliable graders.
- Design considerations: The models maintain uncertainty estimates to support fairer scoring and avoid relying on demographic variables that may be inaccurate or unfair.Fairness is framed as reaching similar confidence levels about every student’s grade.
- Inference and evaluation: The evaluation compares probabilistic models with Coursera’s baseline, which assigns each student the median of four peer grades without modeling grader bias or reliability.The probabilistic approach is intended to address the baseline’s omission of individual grader characteristics and prior grade distributions.
4. EXPERIMENTAL RESULTS
Probabilistic peer-grading models substantially improve accuracy, support confidence-aware grader allocation, and reveal relationships between grading behavior, student characteristics, and future participation.
- Accuracy: 33% and 31% RMSE reductions lowered errors from 7.95 to 5.30 in HCI1 and from 6.43 to 4.73 in HCI2.In HCI2, students within 10 percentage points of staff grades increased from 88% to 97%.
- Accuracy: Model PG3 outperformed other models on most metrics, while estimating grader bias alone produced 95% of the RMSE reduction.PG2, which models temporal coherence, also yielded among the best results.
- Accuracy: With five grades per student, RMSE could fall to 4.19 for HCI1 and 4.36 for HCI2.The evaluation simulated increasing each student’s received grades from four to five.
- Fairness and efficiency: Confidence-calibrated belief distributions enable reallocating graders toward submissions with uncertain scores to improve fairness of feedback.The model’s confidence estimates were conservative: predictions assigned 90–95% confidence were correct over 95% of the time.
- Fairness and efficiency: After two grading rounds, 15% of submissions had highly confident estimates, while 54% remained uncertain after all rounds.This indicates that some submissions could receive fewer grades while uncertain submissions receive more or better graders.
- Grader behavior: Including grader bias and reliability improved future-participation prediction from AUC 0.93 to 0.98.These grading-related properties captured engagement information not present in assignment grades alone.
5. RELATED WORK
The paper builds on information-aggregation, educational testing, Bayesian peer-grading, crowdsourcing, and conference-review assignment research while emphasizing the distinct constraints of MOOC peer grading.
- Information aggregation: The models extend a tradition of aggregating information from noisy human labelers, including item-response theory and crowdsourcing approaches.Related applications include educational essay scoring, cultural anthropology, human computation, and crowdsourcing.
- Educational testing: MOOC peer grading requires more constrained per-grader models because each student evaluates only a handful of assignments.This contrasts with dedicated graders who may evaluate hundreds of essays, enabling richer per-grader models.
- Peer grading: Earlier Bayesian peer-grading work in smaller classrooms modeled grader bias but did not address task reallocation or links between bias and student engagement.The paper positions these issues as extensions beyond that prior setting.
- Crowdsourcing: Like crowdsourcing, MOOC peer grading balances label accuracy against labor cost, but graders are also the items being graded.This grader–gradee overlap motivates models such as PG3 rather than directly importing standard labeler–item assumptions.
- Conference review: Conference paper assignment also fuses opinions about open-ended work, but relies more heavily on reviewer expertise and citation-link information than peer grading does.A submission in peer grading is described as less sensitive to personal preferences because many students can grade it.
6. DISCUSSION AND FUTURE WORK
The paper presents methods intended to make large-scale peer grading more dependable, accurate, and efficient, while identifying unresolved challenges in grader assignment, incentives, and score transparency.
- Discussion: Up to 30% improvement in peer grading accuracy comes from maintaining grader-specific bias and reliability estimates.These quantities also provide insight into peer grading as formative and summative assessment.
- Future Work: Selecting the best grader for a particular submission remains an open problem involving writing styles and cultural or linguistic backgrounds.This is especially important for globally distributed MOOC cohorts.
- Future Work: Incentivizing students who spend inadequate time grading to provide careful, high-quality feedback remains a major unresolved issue.A scoring mechanism that makes grades depend on grading performance may provide a justified incentive, while its game-theoretic properties remain for future work.
- Future Work: It remains unclear how to communicate scores produced by a complicated peer grading model to students.Greater sophistication may create concerns about transparency, although students might also feel more satisfied with their overall grade.
- Discussion: Reliable grading and feedback for open-ended assignments becomes increasingly critical as MOOCs become more widespread.Peer grading is presented as the most scalable solution shown to be effective.