PaperlayerView exact revision on arXiv

Source-linked AI summary

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh, Salman Khan, Naeemullah Khan

arXiv:2609.29333v1cs.CLcs.AIcs.CY

TL;DR

The paper tests whether LLM graders can grade real computer-science exams reliably despite scarce qualified graders and prompt variation. Across two exams, strict-prompt effects differ by exam for open-weight models, while pooled LoRA fine-tuning brings five models to parity or better with a human grader.

  • Problem

    LLM grading is appealing when exams require hundreds of grader-hours and qualified graders are scarce, but common metrics do not show which model, prompt, and persona configurations grade usably or withstand small prompt edits.

    Source: §1 · p. 1

  • Method

    The study evaluates closed and open-weight models across prompt and persona configurations on two dual-graded exams, then tests LoRA fine-tuning using examples pooled across the exams.

    Source: §1 · p. 1, §3 · p. 3, §8.1 · p. 6

  • Results

    The strict preamble pushes 14 of 17 open-weight models out of the CV graded band, while on the ML exam it worsens 10 of 17 and improves seven; pooled LoRA brings five models to parity or better with a human grader.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For these exams, grader performance should be checked under the intended prompt, and pooled fine-tuning on in-house dual-graded examples is a tested alternative to prompt engineering.

    Source: §9 · p. 8, §11 · p. 9

    Few-shot demonstrations leave five CV and three ML students in the evaluated cohorts; ML Flash-Lite comparisons use only a matched subsample because prompt-length failures cause biased dropout.

    Source: §10 · p. 8

Abstract

from arXiv · show

One long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce; LLM graders are a tempting alternative. To show its pitfalls we grade a practical Computer Vision exam ($570$ dual-graded students) under $171$ configurations spanning closed and open-weights models; the best reaches mean absolute error $1.64/35$, below the $2.61/35$ two human graders achieve against each other. The catch is the prompt: a short ''strict grader'' preamble drives $14$ of $17$ open-weights models out of the graded band ($\text{MAE} \ge 8$), three stopping grading altogether. The damage traces to the preamble's two credit-withholding sentences, not to tone or model scale; one of them, ''never give partial credit'', alone makes two of three probed models stop grading. The closed flagships of three vendors shift calibration under it but stay in the band. In $162$ further configurations on a second, independent Machine Learning exam from another course ($1{,}038$ dual-graded students), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over-mark: the vulnerability replicates, but its direction is exam-specific. Light LoRA fine-tuning repairs it: one adapter on the two exams' pooled $\sim 3{,}900$ graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes ($\le 0.32$ MAE). We release the anonymised dataset, full ablation grid, and grading, fine-tuning and analysis pipelines.

1 Introduction

Across two real exams, the study finds that LLM grading performance depends on prompt and exam context: a strict preamble destabilizes many open-weight models, while pooled LoRA fine-tuning substantially reduces that sensitivity.

1 Introduction

  • Six full-cohort configurations—four Gemini and two GPT-5.5—match or beat the CV exam’s human-grader floor of 2.61/35.The introduction reports the best configuration at 1.64/35 in the paper context.Source: §1 · p. 2
  • On the CV exam, a strict-grader preamble pushes 14 of 17 open-weight models out of the graded band, with three stopping grading entirely.The reported damage is attributed to the two credit-withholding policy sentences, not the preamble’s tone or model scale.Source: §1 · p. 2
  • On a second Machine Learning exam, the preamble worsens 10 of 17 models, while seven improve, showing that its effect depends on exam context.Three models leave the graded band and one refuses; the improvements are associated with the exam’s neutral-prompt over-marking.Source: §1 · p. 2
  • A pooled LoRA adapter trained on approximately 3,900 examples brings five open models to parity or better with a human grader and limits harsh-persona sensitivity to at most 0.32 MAE.The reported results cover models from 4B to 30B parameters across both exams.Source: §1 · p. 2

2 Related Work

Prior work establishes LLM judging and persona sensitivity, but the paper places student-code grading against real exams and dual-grader ground truth, then tests meaningful prompt edits across exams.

LLMs as judges and graders

  • LLM judging has been used for benchmark evaluation, while programming auto-grading often tests functional correctness or correlates model scores with human raters on small datasets.The closest cited benchmark compares ChatGPT and GPT-4 with human tutors.Source: §2 · p. 2
  • This study evaluates student-code grading on two released exams against two independent human graders rather than a curated gold reference.Its few-shot examples use the best closed grader’s scores and rationales, not the human graders’ marks.Source: §2 · p. 2

Prompt brittleness and personas

  • Earlier work reports prompt-format sensitivity and persona-dependent toxicity, whereas this study perturbs prompt meaning and replays it on a second exam.The vulnerability replicates, but its direction and failure mode do not.Source: §2 · p. 2

Instruction hierarchies and conflicts

  • The structured score channel can abandon an instruction while the prose channel continues following the rubric, connecting the observed behavior to instruction arbitration.The paper notes that a conflicting-instruction account fits this pattern only as a hypothesis; prior work discusses instruction hierarchy and conflict [11].Source: §2 · p. 2

3 Experimental Setup

The experiments compare closed and open-weight graders across two dual-graded exams, varying model and prompt configurations while scoring predictions against grader averages.

3 Experimental Setup

  • The CV exam covers 570 students on a 35-point base scale, while the second exam is from another course and covers 1,038 students.The CV exam’s grader-average total is used as ground truth; the ML exam has three questions and an approximately 65-point scale.Source: §3 · p. 2
  • The CV grid contains 171 configurations across 33 closed and 138 open-weight runs, including models from 7B to 480B parameters.The runs include four Gemini models and GPT-5.5, GPT-5.4, and Claude Opus 5 among the closed models, plus 17 open-weight variants.Source: §3 · p. 2
  • The CV configurations vary prompt components, personas, few-shot demonstrations, temperature, and other settings, while the headline metric is MAE against the grader-average total.The analysis also reports mean signed error and uses 95% percentile bootstrap confidence intervals from 2,000 resamples.Source: §3 · p. 3
  • The default prompt supplies the question, rubric, reference solution, notebook, grading guidelines, and scorable-item breakdown; personas are short preambles.The strict persona asks for minimum defensible scores and says not to give partial credit when a task does not run correctly.Source: §3 · p. 3

4 The Human-Grader Floor

The paper uses the two graders’ disagreement as a comparative floor for interpreting AI grading error, while noting that the comparison favors AI and varies across grader pairs.

4 The Human-Grader Floor

  • 2.61/35 is the CV exam’s human-grader floor: the graders’ totals have MAE 2.61 against each other across 570 submissions.An AI configuration at or below that MAE makes no more aggregate error against the grader average than the graders make against each other; the floor favors AI because averaging cancels some grader noise.Source: §4 · p. 3
  • Per-pair disagreement ranges from 0.85 to 3.55 MAE, so the pooled floor masks substantial variation across the 10 fixed grader pairs.The reported range is a 4.2× difference in per-pair MAE.Source: §4 · p. 3

5 The Strict-Persona Collapse

A strict-grader preamble pushes most tested open-weight models out of the graded band, with failures attributed to its credit-withholding policy sentences rather than its tone. The collapse recurs on a second exam, but the strict wording can also improve scores when it cancels neutral over-marking.

5.1 The collapse, quantified

  • A strict-grader preamble pushes 14 of 17 open-weight configurations out of the graded band across all six represented families, with MAE rising 1.69×–7.11×; damage is not ordered by model scale.Three models stop grading outright: Llama-3.1-8B and Mistral-Small-24B award 0.00/35 on average, while GLM-4-9B awards 0.56.Source: §5.1 · p. 3

5.2 Attribution: the policy sentences, not the adjective

  • Only strict includes the two credit-withholding policy sentences, and changing the adjective alone causes no consistent MAE shift.The frame alone moves each of three probed models less than two points, while both policy sentences together produce MAEs of 20.26–26.04.Source: §5.1 · p. 4, §5.2 · p. 4

5.3 Mechanism: three failure modes, one MAE range

  • Strict failures divide into blanket zeroing, selective field collapse, and uniform severity, which MAE alone cannot distinguish.The proposed channel-conflict mechanism remains unresolved: withdrawing the breakdown worsens collapse in five of five families, and the grid cannot separate competing explanations.Source: §5.3 · p. 4

5.4 The collapse replicates on the ML exam; its direction does not

  • On the ML exam, four models cross the graded-band threshold under strict, and Llama-3.1-8B again refuses.The exam-specific threshold is MAE ≥15.7; using a fixed MAE ≥8 threshold would also classify eleven neutral baselines as out of band.Source: §5.4 · p. 4
  • Seven models improve under strict on the ML exam, where neutral prompts over-mark for 15 models; none improve on the CV exam.Across 16 non-refusal ML pairs, neutral bias predicts the strict effect (r = −0.73, permutation p = 0.002), consistent with calibration rather than robustness.Source: §5.4 · p. 4, Figure 1 · p. 5
  • The 480B Qwen’s apparent robustness reflects correction of neutral over-marking, while the 235B Qwen stays in band under both personas on both exams.The 480B model over-marks by 9.52 points at neutral; the authors note that both models above 106B belong to Qwen, confounding family with scale.Source: §5.1 · p. 3, §5.4 · p. 4
  • A single strict preamble separates competitive open-weight graders from 14 models out of band, motivating remedies that avoid forbidding partial credit or use in-house labels.The paper’s proposed remedies are to avoid the partial-credit prohibition or fine-tune lightly on in-house labels.Source: §5.4 · p. 4

6 Closed-Model Configurations

The closed-model baseline prompt is already at a joint optimum, and tested prompt-component removals do not reliably improve it. Across vendors, strict wording shifts calibration without generally pushing models out of the graded band.

6 Closed-Model Configurations

  • Three closed configurations beat a human grader in paired tests, while three others are at parity; all six initially met or undercut the 2.61 human floor.The 2.61 floor comparison is not like-for-like, so the six were retested with a paired third-grader statistic.Source: §6 · p. 5
  • On Flash-Lite, thinking closes about three-quarters of the gap to D01, while removing the reference solution costs about 0.8 MAE.Removing guidelines or rubric breakdown changes MAE by at most 0.12; strict plus thinking recovers most of Flash-Lite’s strict penalty.Source: §6 · p. 5

6.1 The asymmetry holds across vendors

  • GPT-5.5, GPT-5.4, and Claude Opus 5 remain in band under strict, showing calibration shifts rather than the open-weight collapse.For example, GPT-5.5 shifts from 2.43 to 4.43 on the CV exam and from 3.54 to 3.70 on the ML exam.Source: §6.1 · p. 5

7 Open-Weights Configurations Under Neutral Prompts

Open-weight performance under neutral prompts varies by model and exam rather than increasing predictably with scale. GLM-4-32B supplies the strongest open-weight result in this grid, but its strict-prompt collapse limits robustness.

7 Open-Weights Configurations Under Neutral Prompts

  • Neutral baselines span 2.85–7.49 MAE, and scaling is non-monotone across tested Qwen models.Qwen2.5-Coder improves from 7B to 14B, then regresses at 32B; removing the reference solution recovers 3.12 on this exam but moves the model +0.16 on the ML exam.Source: §7 · p. 6
  • GLM-4-32B’s best configuration reaches 2.68 MAE, the grid’s strongest open-weight result, while its strict configuration collapses to 9.66.The best result removes the rubric breakdown and uses a neutral persona; the strict-surviving best is Qwen3-Coder-480B at 3.06 MAE.Source: §7 · p. 6
  • Model choice beats size: the 30B-A3B beats the 80B model, and neutral MAE across models on the two exams is uncorrelated (ρ = −0.07).The weakest model on this exam ties statistically for best on the other, underscoring that exam-level performance does not predict across exams.Source: §7 · p. 6

8 Fine-Tuning Repairs the Strict-Persona Collapse

The authors fine-tune five small open models with LoRA on grader labels from two exams, evaluating exam-specific and pooled adapters on held-out students. Pooled adapters reach at least human parity across both exams, transfer across exams, and largely remove sensitivity to harsh personas.

8.1 Setup

  • Five open models from 4B to 30B receive exam-specific and pooled LoRA adapters, evaluated on held-out students against grader-pair disagreement.The pooled training set combines the two exams’ training splits without exposing any adapter to a held-out student.Source: §8.1 · p. 6, §8 · p. 6

8.2 One adapter reaches human parity or better on both exams

  • Pooled breakdown adapters reach at least parity with a human grader in all 10 model–exam cells, and pooled marks adapters are significantly better in 9 of 10.No pooled-adapter cell is worse; the human-disagreement floors differ between exams, so the ML exam sets a lower bar.Source: §8.2 · p. 7

8.3 Grading skill transfers across exams

  • An adapter trained on one exam improves grading on the other, though transfer remains short of in-domain performance.Mean MAE falls from 7.95 to 5.36 on the ML exam under the CV adapter and from 5.54 to 3.57 on the CV exam under the ML adapter; pooling matches or beats specialists in all eight exam-by-recipe columns.Source: §8.3 · p. 7, Table 1 · p. 7

8.4 Fine-tuning immunises against the strict-persona collapse

  • Pooled adapters move by at most +0.32 MAE across three harsh personas, two exams, and five models, eliminating the catastrophic tail seen in base models.Under neutral prompts, the worst base mis-grades 43 of 114 CV students by more than 10 marks, while every pooled model does so for 0–1 students.Source: §8.4 · p. 7
  • The same strict wording can improve an over-grading base on the ML exam, so its effect is not a reliable calibration knob.For Qwen3-Coder-30B, strict changes MAE from 8.60 to 6.34 on the ML exam by cancelling some over-grading.Source: §8.4 · p. 7

8.5 What matters, what does not

  • Pooled marks adapters reach 1.75–2.01 MAE on the CV exam and 3.29–3.53 on the ML exam, largely independent of starting model quality.Human labels alone suffice: pooled marks matches or beats pooled breakdown on every model–exam cell, decisively on the ML exam (3.41 vs. 4.15 mean).Source: §8.5 · p. 7
  • Gemma-4-E4B is the exception to transfer gains on the ML exam, where its cross-exam adapter result is 4.79 versus a 4.66 base.Pooled-model residual errors correlate with human-grader disagreement, consistent with remaining difficulty on rubric-ambiguous students.Source: §8.5 · p. 8

9 Discussion

The strict preamble exposes a sharp difference: open-weight models are vulnerable across families, while the closed models remain in the graded band. Single-prompt evaluations can miss this fragility, motivating evaluation across instructor-written prompts and exams.

9 Discussion

  • No closed model from the three vendors leaves the graded band on either exam, while open-weight failures span all six families and are unordered by scale.The authors characterize this as an open-weights instruction-following failure; fine-tuning suggests it is zero-shot rather than simply a small-model problem.Source: §9 · p. 8
  • 2.68 versus 1.64 MAE separates the best open-weights configuration from Gemini’s best under one prompt; an instructor-style preamble widens the gap to nearly fourteen.The section argues that single-prompt evaluation would have produced a more optimistic picture.Source: §9 · p. 8
  • The authors recommend checking graders against the target exam and multiple prompts, reporting bias alongside MAE, and fine-tuning on pooled in-house examples rather than prompt-engineering.They report that Flash-Lite’s lenient and strict personas have comparable MAE but opposite-signed bias (+4.01 versus −5.44).Source: §9 · p. 8

10 Limitations

The evidence is bounded by a single university, uneven transfer comparisons, selected prompt variants, and limits in grading and statistical validation. These constraints narrow institution-level generalization and some model- and cohort-level comparisons.

10 Limitations

  • Both exams come from one university, so replication spans courses, cohorts, rubrics, and scales, but not institutions.Only three strict-flavoured wordings and two preset policy sentences were tested, not every credit-withholding instruction.Source: §10 · p. 8
  • Per-pair statistics on the ML exam cover only the 18-pair backbone, or 79% of the cohort, because grader pairing is hybrid rather than fixed.Nine rows with a zero for one grader against real partner marks remain in the headline floor, though they are excluded from the maximum-disagreement figure.Source: §10 · p. 8
  • The few-shot demonstrations use graders’ labels, and five CV-exam students whose work appears in the prompt remain in the evaluated cohort.On the ML exam, prompt-length failures cause biased dropout in the Flash-Lite arm, leaving only its matched subsample comparable.Source: §10 · p. 8
  • The three-way failure classification does not establish a cause: removing the rubric breakdown contradicts the conflicting-instruction interpretation, leaving the causal account open.The few-shot uplift reflects capability rather than held-out generalization because the demonstrations carry D01’s own labels.Source: §10 · pp. 8–9
  • Three CV-exam better-than-grader verdicts are marginal under normal-approximation confidence intervals, and the weakest would include zero under a t-interval.The ML-exam verdicts are less marginal, with upper confidence bounds at or below −0.30.Source: §10 · p. 9

11 Conclusion

For dual-graded exams, the authors recommend verifying graders on the exam and prompts they will face. Stronger closed models meet the human floor untuned, while open models require light exam-specific fine-tuning to withstand instructor edits.

11 Conclusion

  • Strong closed models grade at or below the human floor untuned from a neutral prompt, whereas open models reach it only after light fine-tuning on the exam’s grades.Source: §11 · p. 9
  • Between exams, the vulnerability and its repair transfer, but model ranking, persona direction, and t = 0 determinism do not.Source: §11 · p. 9
  • The authors recommend verifying a grader on its target exam under instructor-written prompts.Source: §11 · p. 9

AI Use Statement

The authors disclose using generative AI as a coding, writing, and exploratory-search assistant, while stating it did not generate research ideas, results, or reported analysis numbers. The evaluated LLM graders are distinguished from those tools: their scores are analyzed and their outputs released.

AI Use Statement

  • Opus 4.8, Opus 5.0, and Fable 5 assisted with the data pipeline, grading harness, fine-tuning scripts, analysis code, prose, and exploratory literature search.Source: AI Use Statement · p. 9
  • The authors state that generative AI did not generate research ideas, experimental results, or reported analysis numbers, which were computed from released scripts and data.Source: AI Use Statement · p. 9
  • The authors say AI-assisted code was reviewed and executed, statistical results independently re-derived, and responsibility for the final work retained by the authors.The LLM graders evaluated in the study are the object of analysis, and their outputs are released in full.Source: AI Use Statement · p. 9

Ethics Statement

The appendix details the two-exam evaluation, its scoring and run procedures, and safeguards for releasing student work and using LLM graders in consequential assessment.

Ethics Statement

  • Released datasets are anonymised by removing student identifiers and contact information; no demographic attributes are collected or released.Graders appear only as opaque identifiers, and the released artifacts do not identify students.Source: Ethics Statement · p. 9
  • The authors warn that under-grading can harm students and recommend against deployment based on one prompt’s accuracy, requiring human oversight for consequential grading.They report bias alongside MAE so the direction of harm is visible and do not present the results as licensing unsupervised assessment.Source: Ethics Statement · pp. 9–10

A Experimental Details

  • The CV exam has 570 submissions across four code questions; its 13 possible bonus points are graded but excluded from every reported metric.The three base questions carry 12, 11, and 12 points.Source: A Experimental Details · p. 13
  • The evaluation spans 171 configurations, including 17 open-weight variants from six vendors, and uses ablations, persona sweeps, alternate wording, and mechanism probes.Gemini G- and M-series runs use the first 100 students; the 3-flash-preview throughput constraint motivates matching baselines to that subset.Source: A Experimental Details · p. 13
  • Each call receives a question’s notebook text without outputs; replies are parsed as JSON, with retries and failure handling differing between Vertex and open-weight runs.Grammar-constrained decoding changed no score in a controlled A/B and cost 6.9× in latency.Source: A Experimental Details · p. 13
  • Gemini thinking is disabled for 21 of 25 ablations, enabled for three, and dynamic by default for D03 because gemini-2.5-pro rejects a zero budget.D03 is therefore treated as thinking-enabled.Source: A Experimental Details · pp. 13–14
  • Total-level Spearman ρ exceeds the weakest question’s ρ in all 37 CV and 37 ML runs with defined correlations, by median gaps of 0.20 and 0.12, respectively.Q2 is weakest on CV in 30 of 37 runs, and Q1 on ML in 27 of 37.Source: Table 2 · p. 14
  • A student with any ungraded Q1–Q3 is generally excluded from that run’s metrics; E-series reruns retain students only when all three questions were graded in at least one rerun.The reported E-series reruns average recorded per-question scores; single-call runs keep non-submitters and score a missing question zero.Source: A Experimental Details · p. 14

B Strict-Persona Collapse: Additional Tables

The appendix probes strict-prompt failures and prompt-only repairs, then reports model ablations, fine-tuning results, and per-question error patterns.

B Strict-Persona Collapse: Additional Tables

  • Withdrawing the rubric breakdown worsened all five strict models; the structural-scaffolding explanation fits L-BD but remains untested because the rubric table stayed.The same manipulation under neutral was a clean control, with four of five models shifting within ±0.25 MAE.Source: B Strict-Persona Collapse: Additional… · p. 16

E Fine-Tuning Details

  • 1.75–2.01 MAE on held-out CV students puts pooled adapters below the human floor, while prompt-only repairs recover only neutral-prompt performance.The prompt-only arbiter uses three calls per question and defers to the neutral grade.Source: B Strict-Persona Collapse: Additional… · p. 18

B Strict-Persona Collapse: Additional Tables

C Closed-Model Details

  • On the first-100 G-series, the apparent G04 win disappears against the same-subset baseline; sampling adds variance without beating one t = 0 call.D01-restricted scores 1.23 versus G04’s 1.25; mean-of-reruns MAE is 3.38 at t = 0 versus 3.34 for the single-call baseline.Source: C Closed-Model Details · p. 19, Table 13 · p. 21

D Open-Weights Details

  • Excluding the five demonstration students leaves the few-shot uplift unchanged on CV and ML, addressing whether the examples inflated the measured gains.For Qwen3-Coder-30B-A3B, the CV uplift is +1.16 both with and without those students.Source: D Open-Weights Details · p. 22

E Fine-Tuning Details

  • Pooled adapters reduce catastrophic errors to 0–1 CV students and 2–4 ML students, but CV Q3 still has 1.1–1.3 MAE versus 0.6–1.0 on Q1/Q2.The Q3 residual is on semantic segmentation, the most open-ended question.Source: E Fine-Tuning Details · p. 23

F Per-Question Error Decomposition

  • Question-level cancellation can hide directional error: L-A14’s biases sum to −1.16 while concealing 1.78 of 2.94 points of directional error.Cancellation is confined to near-unbiased runs; all 17 matched strict runs have C = 0.Source: F Per-Question Error Decomposition · p. 25

G Ground-Truth Structure

The ground-truth analysis finds that AI error tracks human disagreement differently across the two exams, while grader-pair effects complicate the human comparison. The ML replication confirms strong prompt sensitivity but shows that strict wording can either damage or improve calibration, depending on the model’s neutral bias.

G Ground-Truth Structure

  • Per-pair human disagreement and AI error diverge by exam: they are uncorrelated on CV but positively correlated on ML, with contiguous-block assignment only a candidate explanation.The CV relationship is r = −0.301 (p = 0.40); on ML it is r = +0.571 (permutation p = 0.014), and the difference is significant, but the authors do not claim a settled mechanism.Source: G Ground-Truth Structure · p. 26, H ML-Exam Replication Details · p. 31
  • Within-pair grader harshness is measurable, but between-pair differences are confounded with batch ability; the CV AI’s MAE is about 1.6 around noisy human consensus.The within-pair component accounts for η2 = 0.017, while the broader η2 estimate is design-dependent and does not transfer across exams.Source: G Ground-Truth Structure · p. 26

H ML-Exam Replication Details

  • The policy sentences, not harsh framing alone, reproduce the attribution pattern: S2 alone stops two probed models, while the dominant sentence differs by model.Swapping STRICT for RIGOROUS or FAIR while retaining both policy sentences does not produce a consistent direction; absolute damage varies by exam.Source: H ML-Exam Replication Details · pp. 29–30
  • Closed-model parity largely replicates on ML, while other effects do not: few-shot examples nearly halve one open model’s error, but prompt length causes biased dropout on Flash-Lite.The CV component-removal and temperature-zero determinism results do not transfer uniformly to ML.Source: H ML-Exam Replication Details · pp. 30–31
  • On the 1,038-student ML exam, strict wording worsens 10 of 17 open models, pushes four beyond the floor-matched band, and improves seven whose neutral prompts over-mark.The replication spans 162 configurations; the direction of the persona effect is exam-specific, and improvements can reflect calibration rather than robustness.Source: H ML-Exam Replication Details · pp. 28, 31

I All Grader Configurations

  • The appendix records all CV grader runs in Table 24, including prompt components, persona probes, sample counts, MAE, bias, confidence intervals, and behavior classes.The table covers closed- and open-weight configurations as well as separate prompt-only baselines.Source: I All Grader Configurations · p. 31, Table 24 · p. 32

J Full Limitations Inventory

The limitations inventory identifies incomplete persona coverage, serving and model confounds, and qualifications on the fine-tuning comparisons. The ML run table is a comprehensive record of configurations and metrics, with specific sample and comparability caveats.

J Full Limitations Inventory

  • Persona coverage is incomplete: only three models receive the policy 2 × 2 probe, adjective-only interactions are untested, and closed-model persona sweeps cover only Gemini.The ML exam also lacks some wording coverage for Qwen models, limiting cross-persona comparisons.Source: J Full Limitations Inventory · p. 35
  • Fine-tuning comparisons have scope constraints: the bd arm distills closed-model targets, the 30B model is not a clean size comparison, and base rows use a different harness.Held-out MAE confidence intervals are roughly ±0.45; evaluation is vLLM-served, with Gemma-4’s path cross-validated against HuggingFace generation to within about 0.1 MAE.Source: J Full Limitations Inventory · p. 35

K ML-Exam Run Table

  • Table 25 lists 162 ML configurations and their metrics, but IG18’s MAE uses survivors of prompt-length failures and is not comparable to full-cohort rows.The table also records run-specific confidence intervals, bias, behavior class, and repeated-sampling or first-100-student subsets.Source: Table 25 · p. 35, K ML-Exam Run Table · p. 35

Related papers

Loading 2609.29333v1…