Source-linked AI summary
Measuring Student Self-Assessment against Viva-Demonstrated Mastery in a Large First-Year Programming Course
Sakshi Sharma, Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade, Rohit Sharma, Meenakshi V, S. R. S. Iyengar
TL;DR
Overall self-ratings provide limited insight into which tasks or students account for mismatches, so this study compares task-level programming self-reports with viva-demonstrated mastery. Self-report was highly accurate in aggregate, while the remaining overestimation was concentrated among a minority of students and the hardest tasks.
Problem
Overall self-ratings compared with examination scores provide limited evidence about which tasks or students account for mismatches in reported and demonstrated mastery.
Source: §3.2 · p. 5Method
The study compared students’ reported programming-task mastery with oral-viva demonstrations, verifying each reported task against the student’s performance.
Source: §4.1 · p. 6Results
Self-report was highly accurate in aggregate, but remaining overestimation was concentrated among a minority of students and at the hardest end of the task ladder.
Source: §5 · p. 7, §6 · p. 13Takeaways & Limitations
Task-level comparison can support selective verification and task-specific feedback where reported mastery is more likely to diverge from demonstrated mastery.
Source: §6 · p. 13Verification followed self-report, so under-reporting was unobservable; the study covers one course, one academic year, and one institution.
Source: §4.2 · p. 7, §5.6 · p. 12
Abstract
from arXiv · showhide
Mastery-based education increasingly places the reporting of learning progress in students' hands, who record task completion on learning dashboards. The usefulness of such self-reports depends on how closely reported mastery corresponds to demonstrated competence. Most evidence on student self-assessment compares an overall self-rating with an overall examination score and therefore provides limited evidence about which tasks or which students account for the mismatch. This study examines first-year students' self-assessment against viva-demonstrated mastery at the level of individual tasks across a ladder of sixty programming tasks. The study draws on a large first-year programming course taught in 2023, involving 203 students and 12 examiners, in which every reported task was verified through an oral viva. Because a task entered the Viva only after it was reported, the design is one-sided and captures over-estimation but not under-estimation. Of 11,093 reported tasks, 10,885 (98.1%) were demonstrated, indicating a high degree of correspondence between self-report and demonstrated mastery. The remaining 208 overestimations were not evenly distributed. A small number of students accounted for most of the errors, and they occurred mainly on difficult tasks near the end of the task ladder rather than on higher-point tasks. This task-level analysis shows that high overall self-assessment accuracy can coexist with specific areas where reported and demonstrated mastery diverge. It also provides a practical basis for directing additional verification and formative feedback toward students and tasks where such divergence is more likely.
1 Introduction
Mastery-based learning tracks progress through individual competencies, but student reports are judgments rather than direct evidence of demonstrated competence. Comparing self-reports with demonstrations at the task level can reveal where reported and demonstrated mastery diverge.
1 Introduction
- Mastery-based education represents learning outcomes as smaller tasks, while end-of-course examinations provide evidence of overall achievement too late to track each competency during learning.Periodic tracking can help students understand what they have and have not mastered.Source: §1 · pp. 1–2
- Student reports can make task-level progress visible and reduce instructors’ need to record every mastery claim, but require comparison with an independent demonstration.A reported task may not be demonstrated when checked.Source: §1 · p. 2
- Overall self-ratings compared with examination scores provide little information about which competencies or tasks account for mismatches.A task-level comparison can locate where reported mastery diverges from demonstrated mastery.Source: §1 · p. 2
- The study compares first-year students’ task-level programming self-reports with oral-viva performance at cohort, student, and task levels.It focuses particularly on instances where reported and demonstrated mastery diverge.Source: §1 · p. 2
2 Literature Survey
Prior research documents variation in self-assessment accuracy but often compares overall ratings with overall scores, limiting insight into task-specific mismatches. This study uses task-level comparison in a mastery-based programming setting to examine where self-reports diverge from oral demonstrations.
2 Literature Survey
- Self-assessment correspondence varies across learners, tasks, and assessment conditions, and can be high when students have explicit criteria.A metasynthesis found moderate correspondence overall.Source: §2 · pp. 2–3
- Overall self-ratings compared with examination or instructor scores can show general over- or under-estimation, but not identify the tasks responsible.A learner may judge simpler competencies accurately while misjudging a more demanding one.Source: §2 · p. 3
- Knowledge-in-pieces describes novices’ knowledge as loosely connected and context-dependent, which may let students recognize part of a solution but fail to integrate it during demonstration (diSessa, 1988, 1993).Task-level comparison can make this point of divergence observable.Source: §2 · p. 3
- Progress dashboards make task completion visible, but their mastery signal depends on the accuracy of student reports; guidance on verifying task-by-task reports remains limited.Oral assessment can complement automated programming assessment by asking students to explain and defend their work.Source: §2 · p. 3
3 Methodology and Methods
The study compares reported programming-task mastery with viva demonstrations, defining calibration narrowly as the accuracy of reported abilities. It examines cohort, student, and task patterns, after checking the task ladder and examiner variation as criteria for interpretation.
3.1 The design
- Each reported task was examined in an oral viva, but unreported tasks were never assessed, so the design captures overestimation rather than underestimation.Students could reattempt a reported task; only its final outcome was retained, making overestimation a persistent gap rather than a single failed attempt.Source: §3.1 · p. 4
3.2 Research questions
- The research questions ask how well self-assessment aligns with viva-demonstrated mastery and how overestimation varies across the cohort, students, and tasks.Task-level variation is considered by ladder position and point value.Source: §3.2 · p. 5
3.3 Checking the criterion
- The task ladder is checked as a measurement scale using item facility, corrected item-total correlations, KR-20, and a two-parameter logistic item-response model.These checks examine task performance, internal consistency, and task difficulty and discrimination.Source: §3.3 · p. 5
- Viva consistency is assessed by comparing examiner confirmation rates and summarizing between-examiner variation, but examiners assessed their own students.Examiner and cohort are therefore not independent, so this is a check on observed criterion variation rather than a fully crossed reliability estimate.Source: §3.3 · p. 5
3.4 Analytical methods
- Cohort analysis reports agreement and overestimation, while student-level analysis examines error counts and concentration against an exposure-adjusted null model.The null model holds task difficulty fixed and randomizes which students reporting each task fail to demonstrate it; intervals use 5,000 bootstrap resamples.Source: §3.4 · p. 5
- Task-level analysis compares early with harder late tasks, tests association with ladder position, and separately examines task point value.Spearman correlation is secondary because ladder position is confounded with task difficulty and curricular time.Source: §3.4 · pp. 5–6
- No formal multiplicity correction is applied; interpretation rests on effect magnitude and uncertainty rather than borderline significance alone.The study reports confidence intervals for observed effects.Source: §3.4 · p. 6
4 Data Collection and Analysis
The study analyzes a 2023 first-year programming course in which students reported tasks from a sixty-task ladder and each reported task was verified in an oral viva. Overestimation was limited to reported tasks, and the analyses used a scripted pipeline on de-identified course records.
4.1 The course and the task ladder
- The 2023 course used a sixty-task programming ladder, and the analytic sample comprised 203 students across five engineering branches.Students were assigned to a branch and appeared in both self-report and viva records; 12 teaching assistants conducted the vivas.Source: §4.1 · p. 6
- Each reported task was checked through an oral viva requiring runnable, correct code and an answer explaining that code.Students could reattempt until passing, but only the final outcome was retained; random instructor audits almost always upheld assistants’ decisions.Source: §4.1 · p. 6
- The tasks carried course-grade points and appeared on a public leaderboard; values ranged from 1 to 25 across the ladder.Point values were instructor-assigned rather than determined by difficulty alone; the mastery score was the points-weighted percentage of tasks demonstrated.Source: §4.1 · p. 6
4.2 The paired grid and its one-sided structure
- The paired records form a 203 × 60 grid, but because viva assessment followed self-report, demonstrated-but-unreported tasks cannot occur in the data.The analysis consequently uses the one-sided overestimation rate rather than a symmetric comparison of disagreements.Source: §4.2 · pp. 6–7
- 208 of 11,093 reported tasks were overestimations, a rate of 1.9%.The one-sided design observes reported tasks that were not demonstrated, but cannot observe underestimation.Source: §4.2 · p. 7
4.3 Reproducibility, data, and ethics
- The analysis pipeline was scripted with fixed seeds, and de-identified secondary course records were reported in aggregate.The task set, point weights, and ladder order are course-specific; data and code were to be released on acceptance.Source: §4.3 · p. 7
5 Results and Discussion
Self-reports closely matched viva-demonstrated mastery overall, but the remaining over-estimations were concentrated among a minority of students and on the hardest tasks. The results support selective verification, while the one-sided design and limits of the viva criterion bound interpretation.
5 Results and Discussion
- Self-report was highly accurate in aggregate, while remaining over-estimations clustered among a minority of students and at the hardest end of the task ladder.Table 2 summarizes the research questions, methods, and headline results.Source: §5 · p. 7, Table 2 · p. 8
5.1 The viva criterion and its limits
- The viva criterion showed KR-20 = 0.94 across all tasks and 0.95 across tasks with response variance, but reliability also reflects students’ progress along the ladder.The task ladder had a strong ceiling effect: 15 tasks were demonstrated by every student, and its information was concentrated at the difficult end.Source: §5.1 · pp. 7–8
5.2 SQ1: Self-assessment is highly accurate in aggregate
- Of 11,093 reported tasks, 10,885 (98.1%) were demonstrated; the design detects over-estimation but cannot observe under-reporting.The 98.1% figure is the confirmation rate for reported mastery, not a general measure across all possible task judgements.Source: §5.2 · p. 8
5.3 SQ2: Over-estimation is concentrated in a minority of students
- Among 203 students, 159 (78%) had no observed over-estimations, while the remaining 44 accounted for all 208 errors; the least-calibrated 10% accounted for 67%.The Gini coefficient was 0.85, exceeding the exposure-adjusted null value of 0.57; among students reporting all nine hard late tasks, 83 of 89 were perfectly calibrated.Source: §5.3 · p. 9, Figure 2 · p. 10
5.4 SQ3: Over-estimation concentrates on the hardest tasks
- All 208 over-estimations occurred on tasks O52 to O60, whose mean over-estimation rate was 17.6%; the O52 boundary was identified descriptively from observed data.Even within the hard tail, 82% of reported tasks were demonstrated; the 25-point easy task O45 had no over-estimations, while hard O60 had six among 99 reports.Source: §5.4 · pp. 9–10, Figure 3 · p. 11
5.5 Discussion
- The task-level pattern is consistent with mismatch emerging on demanding tasks, but the proposed knowledge-in-pieces explanation is not a causal finding.The study did not measure the cognitive processes behind discrepancies, and points, a public leaderboard, and graded completion leave metacognitive judgement and strategic reporting indistinguishable.Source: §5.5 · p. 11
- Selective verification could target tasks where mismatch is more likely, rather than checking every claim or labeling individual learners as inaccurate.A discrepancy can also provide formative information about the specific competency where reported mastery and performance diverge.Source: §5.5 · p. 12
5.6 Limitations
- The study covers one course, year, and institution, so exact rates are local; its one-sided design misses under-estimation and other design limits constrain interpretation.Only final viva outcomes were retained, examiner and branch effects could not be separated, and the course provided additional support to struggling students.Source: §5.6 · p. 12
6 Conclusion
Task-level comparisons found high overall correspondence between reported and viva-demonstrated mastery, but over-estimation clustered among a small group of students and harder tasks. The analysis supports targeted verification and feedback while identifying the value of testing whether this pattern holds across courses and institutions.
6 Conclusion
- Over-estimation clustered among a small group of students and at the harder end of the task ladder, while task point value showed less direct association with mismatch.Aggregate self-assessment accuracy can therefore conceal where and for whom self-report becomes less reliable.Source: §6 · p. 13
- Task-level measurement makes self-report usable as a scalable first layer of progress reporting while enabling verification and formative feedback to target likely mismatches.The observed self-report–viva gap can provide task-specific feedback that links mastery assessment with opportunities for recalibration.Source: §6 · p. 13
- Future work should test whether the pattern holds across courses and institutions and develop a general framework for selective verification and feedback.Source: §6 · p. 13