Scaling Peer Assessments: An Integrity Report from a Large Engineering Internship

• In large classrooms, instructors have limited capacity to assess each learner’s understanding and provide meaningful feedback.Common progress measures may not fully reflect the depth of a student’s understanding.Source: §1 · p. 1
• Peer validation lets students assess peers, explain concepts, question reasoning, identify gaps, and provide immediate feedback.Assessing another learner can engage the assessor with the subject while giving the learner a chance to articulate and defend understanding.Source: §1 · p. 2
• In peer assessment at scale, familiarity, reciprocity, and differing judgments may weaken consistency and reliability.The central question is whether distributed assessors can maintain a meaningful, consistent validation standard.Source: §1 · p. 2
• The study examines a large, multi-institutional internship model where students who demonstrated understanding were authorized to validate peers.Students could move from being assessed to becoming assessors.Source: §1 · p. 2
• The study investigates participants’ perceptions of standards, peer conduct, challenges, and possible improvements to scalable peer validation.It seeks empirical evidence about the model’s feasibility, reliability, and challenges.Source: §1 · p. 2

• Peer marks can correspond closely to teacher marks, and peer assessment has shown positive academic-performance effects across studies.Peer evaluation can also benefit the assessor as a learning activity (Lundstrom and Baker, 2009; Li et al., 2010; Van Popta et al., 2017).Source: §2 · p. 2
• Peer assessment may develop learners’ capacity to judge quality and use feedback effectively.On this account, judging peers against a standard is itself practice in the capability the activity aims to build.Source: §2 · p. 3
• Reassuring peer-assessment findings commonly come from instructor-moderated settings with numeric marks and inspectable samples.In one overseen setting, reciprocity accounted for about one per cent of rating variance.Source: §2 · p. 3
• Large-scale peer grading studies report bias alongside encouraging peer–staff agreement, with feedback on bias reducing it.Staff reference grades and numeric judgments enabled checking and calibration; anonymous arrangements were associated with more critical feedback.Source: §2 · p. 3
• The literature offers little empirical description of cascading binary validation and limited evidence on participant-selected remedies in that setting.Agreement with instructor marks is unavailable when no instructor mark exists, while aggregation, calibration, and moderation presuppose numeric judgments.Source: §2 · p. 3

3.1 Research questions

• The research questions address reported integrity problems, differences between accounts of own and peer conduct, preferred changes, and validator-finding difficulty.The principal question asks what participants reported about integrity and possible improvements.Source: §3.1 · pp. 3–4

3.2 Context and the validation cascade

• In the second phase, 583 interns undertook short oral discussions of unmarked problems, with reciprocal validation prohibited.The instructor validated two mentors, who helped seed a self-propagating cascade; passing a discussion granted authority to validate others, with no routine confirmation step.Source: §3.2 · p. 4, Figure 1 · p. 5

3.3 Instrument and participants

• The exit survey’s peer-validation section included six five-point Likert items, two ordinal items, a multi-select item, and free-text fields.Of 314 submitted surveys, 241 respondents consented to research use and 238 were routed to the section; negatively worded items indicated reported problems when agreed with.Source: §3.3 · p. 4, Figure 2 · p. 6

4.1 Results

• 79.8% agreed they solved problems they could not previously solve, and 66.8% said the problems were sufficiently demanding to be worthwhile.These are participant perceptions, not measures of learning.Source: §4.1 · p. 6
• 29.0% reported at least one integrity lapse; reduced effort, a lowered validation bar, and prohibited reciprocal validation were endorsed by 18.1%, 16.0%, and 13.4%, respectively.Reduced effort and a lowered bar were strongly associated (ρ = 0.62); the 13.4% reciprocity figure is best interpreted as a lower bound.Source: §4.1 · pp. 6, 9
• 88.7% believed at least a few validations occurred without proper examination, a perception that does not establish the actual prevalence of lax validation.37.8% believed about a quarter or more were granted without proper examination; the authors distinguish perceived legitimacy from accuracy.Source: §4.1 · p. 7, §4.2 · p. 9
• 50.0% selected post-validation solution discussion to make validation more rigorous, about twice as often as auditing (24.8%) or mentor validation (26.5%).The ordering remained unchanged when respondents selecting “nothing” were excluded; discussion was selected by 63.1% of the remaining 168.Source: §4.1 · pp. 7–8, Figure 3 · p. 8
• 77.3% reported some difficulty finding a validator, but this difficulty was not associated with reciprocal validation or reduced effort.Its association with perceived laxity did not survive correction for multiple testing.Source: §4.1 · p. 8

4.2 Discussion

• Participants’ preference for solution discussion is consistent with learning through judging quality, but the survey cannot show that discussion would reduce lax validation.The authors note that supervision consumes expert time, while a single post-validation session can reach the cohort; participants selected discussion regardless of whether they reported a lapse.Source: §4.1 · p. 8, §4.2 · p. 9
• Reported reciprocity was low and the distributed system did not collapse into collusion, yet the authors identify validation without teaching as the subtler concern.Their conclusion is that scaling judgement is not the same as scaling learning.Source: §4.2 · p. 10

4.3 Limitations

• Because this is a one-time self-report study without validator-assignment records or a direct learning measure, it cannot determine validation accuracy or whether the activity improved understanding.The authors identify recording the validation graph as the key improvement for testing how rigour changes across the cascade.Source: §4.3 · p. 10

5 Conclusion

• Where qualified assessors are scarce, the authors frame peer validation as a necessity rather than a preference.Source: §5 · p. 10
• Among 238 consenting participants, most regarded the activity as worthwhile, a minority reported standards slipping, and a large majority suspected lax validations in others.Source: §5 · p. 10
• Difficulty finding validators was widespread but was not associated with the reported problems.Source: §5 · p. 10
• Participants chose instruction over supervision by roughly two to one, regardless of whether they reported a lapse.Source: §5 · p. 10
• The authors call for future work to record the validation graph and compare cohorts with and without a post-validation discussion.Source: §5 · p. 10