PaperlayerView exact revision on arXiv

Source-linked AI summary

The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale

Wenhui Chen, Ziyao Lin, Jianlin Chen, Peiji Long, Chi Man Vong

arXiv:2609.26087v1cs.AI

TL;DR

Assessment items must satisfy multiple curriculum and quality constraints, yet fluent LLM output is not necessarily deployable. CLAIM closes the loop between generation and evaluator feedback; MCQ/MSQ generation saturates under static rules while cloze remains capability-tiered, and the final production run reached a 97.8% expert-evaluator pass rate.

  • Problem

    Deployable assessment items must jointly satisfy alignment, difficulty, answer-key, factual, and other constraints, while quality is externally defined rather than determined by the model’s own output.

    Source: §1 · p. 2

  • Method

    CLAIM uses two-stage adversarial review and recycles evaluator verdicts as contrastive exemplars and mined knowledge rules for future generation.

    Source: §1 · p. 2

  • Results

    MCQ/MSQ generation saturated under static rules, whereas cloze remained capability-tiered; the final production run reached 97.8% expert-evaluator pass.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper frames answer-key completeness as open-set boundary determination, a challenge that also recurs when the evaluator is distilled.

    Source: §1 · p. 2

    Model judges do not establish ground truth, and psychometric difficulty requires field testing with student responses.

    Source: §4.1 · p. 8, §4.3 · p. 12

Abstract

from arXiv · show

We present CLAIM, a production pipeline for K-12 assessment-item generation coupling a two-stage generate-then-attack protocol (the model drafts as a "curriculum architect", then re-enters the same conversation as a hostile adversarial reviewer), bi-directional few-shot conditioning on accepted and rejected items, the latter carrying the evaluator's diagnosis, and a knowledge dictionary of 44,844 error-correction rules mined from that feedback and retrieved per standard and item type. Across 43,227 scored items over 755 Common Core ELA standards, three item types, and ten LLMs, the pipeline reaches a 97.8% expert-evaluator pass rate on a 9,074-item production run. We then ask what that rate certifies. Re-scoring a stratified sample with three judges from other vendors, blind to the deployed verdict, reproduces the format ordering under every judge and recovers a larger open-set deficit than the deployed evaluator does; but agreement on the accept/reject binary is weak at production prevalence (kappa about 0.13), and the judges agree with each other no better. The level is therefore judge-relative, and with no student-response data our quality evidence is evaluator-judged throughout. The corpus also exposes a robust asymmetry. Multiple-choice and multiple-select generation saturate at 98% or above for both frontier models under a dozen static rules, whereas fill-in-the-blank generation is capability-tiered (82.8-96.7% across five models under a matched rule set, standards, and judge) and plateaus under prompt-only optimization, with error mass shifting between answer-key over-inclusion and omission as rules accumulate. We analyze this as open-set boundary determination, a task autoregressive decoders are structurally ill-equipped to solve, and show the asymmetry recurring when the evaluator itself is distilled: fail-recall rises from 8% to 63% while F1 saturates at 0.25.

1 Introduction

CLAIM couples generation with evaluator feedback to address the constraints that separate fluent assessment items from deployable ones. At scale, it achieves a high evaluator-pass rate, but results expose a persistent difficulty with open-ended answer keys and limits on what a single judge certifies.

1 Introduction

  • CLAIM combines adversarial self-review, accepted and rejected few-shot exemplars, and a 44,844-rule knowledge dictionary mined from evaluator feedback.Rejected exemplars retain the evaluator’s diagnosis, and rules are retrieved by standard and item type.Source: §1 · pp. 2–3
  • 97.8% of the 9,074-item final production run passed the expert evaluator, with grade×type cells ranging from 95.3–99.6%.The run covered 755 Common Core standards, three item types, three difficulty levels, and grades 1–12.Source: §1 · p. 2
  • MCQ and MSQ reached ≥98% under static rules, while matched cloze results ranged from 82.8–96.7% and rule additions shifted errors between key over-inclusion and omission.The paper frames the contrast as closed-set self-referential generation versus open-set boundary determination.Source: §1 · p. 2
  • The empirical study covers 43,227 scored items across 755 standards, grades 1–12, three item types, three difficulty levels, and ten LLMs, but reports no student-response evidence.The authors identify this as the largest reported curriculum-standard-scored item corpus, while noting the absence of student responses.Source: §1 · p. 3
  • Three blind external judges reproduced the format ordering on a stratified 390-item sample, but accept/reject agreement was weak at production prevalence.The format asymmetry was larger under external judges than under the deployed evaluator.Source: §1 · p. 3
  • Distilling the evaluator reproduced the generation-over-discrimination asymmetry: fail-recall rose from 8% to 63% while F1 saturated at 0.25.The authors interpret this as acquiring the verdict form without acquiring the accept/reject boundary.Source: §1 · p. 3
  • The authors propose rubric-based key adjudication, externalized boundary verification, and cloze generation as a discriminative stress test.Cloze spreads a model tier over 13.9 points, compared with 3.9 points for closed-set formats.Source: §1 · p. 3

2 Related Work

Related work establishes quality standards and explores automated generation, evaluation, prompt optimization, and self-correction. CLAIM’s positioning centers on persistent evaluator-derived artifacts and the production consequences of open-set answer-key determination.

2 Related Work

  • Template-based and early neural item generation established quality guidelines and automated ranking needs, while LLM work studies quality, Bloom level, difficulty, and reading comprehension [48] [50] [12] [46].The Duolingo English Test represents operational deployment of transformer-based item generation.Source: §2 · p. 3
  • Distractor generation spans PLM ranking, retrieval-prompted LLMs, and misconception-aware methods [6] [15] [16]; CLAIM instead mines feedback-derived rules and retrieves them per curriculum standard.The paper relates standard-specific rule retrieval to standard-derived knowledge artifacts [27] and learned error representations [16].Source: §2 · p. 3
  • Prior prompt-optimization approaches target scorers [45] [63] [33] [11]; CLAIM combines persistent rules, feedback memories, and negative demonstrations in one production system [49] [39] [17].Its distinction is operationalizing all three artifact types and reporting their behavior at scale.Source: §2 · p. 4
  • Prior work identifies decoder enumeration and calibration deficits [25] [32] [36], while CLAIM shows prompt optimization stalling on curriculum-defined answer boundaries.The paper distinguishes its setting from decidable reference sets because acceptable answers are defined by language and curriculum [8].Source: §2 · p. 4

3 Method

The method turns a curriculum blueprint into structured item generation, adversarial review, and feedback-driven updates. It specifies task requirements, generator and reviewer protocols, feedback channels, rule accretion, and the corpus and comparisons used for evaluation.

3.1 Task and evaluation protocol

  • Each blueprint specifies grade, CCSS ELA standard, item type, and difficulty; items must emit JSON, and pass only above 0.85 on an evaluator bottlenecked by its weakest dimension.The commercial evaluator scores more than 100 diagnostic criteria and is distinct from the generator models.Source: §3.1 · p. 4

3.2 Stage 1: The Architect

  • The Architect applies golden principles, domain and linguistic constraints, grade-specific Bloom rubrics, and analyze–blueprint–draft–validate; cloze drafts a source passage before masking and pruning mirror cues.For MCQ/MSQ, distractors are tied to misconceptions, with grammatical parallelism and option-length balance requirements.Source: §3.2 · p. 5

3.3 Stage 2: The Adversary

  • The Adversary extends the same conversation to attack the draft, testing cloze-key precision and completeness, cues, leakage, and explanations; corrected JSON replaces the draft unless review fails.MCQ/MSQ receive a lighter five-point gate, and the reviewer uses named attacks rather than a generic review instruction [21] [26].Source: §3.3 · pp. 5–6

3.4 Closing the loop: mining evaluator feedback

  • Evaluator verdicts feed accepted and rejected exemplar pools, with three of each sampled per prompt, and a 44,844-rule dictionary indexed by standard and item type.Rejected examples carry evaluator reasoning and scores; rules are retrieved with diversity-first severity sampling, up to ten per prompt.Source: §3.4 · p. 6

3.5 Iterative rule accretion

  • Cloze safeguards grew from ∼15 core rules in R0 to ∼42 in R3, with R4 adding word-bank integrity and grade-band safeguards; MCQ/MSQ retained ∼12 static rules.The cloze additions followed successive failure-driven tuning rounds.Source: §3.5 · p. 6

3.6 Corpus and experimental setup

  • The corpus spans 755 standards, grades 1–12, three item types, and three difficulty levels; 43,227 unique scored items correspond to 47,278 valid verdicts, including repeats.Ten LLMs were used, but protocols varied across the aggregate corpus; controlled claims rely on matched runs and within-model trajectories. Table 1 distinguishes primary verdicts from repeats used for reliability analysis and excludes evaluator errors.Source: §3.6 · pp. 6–7, Table 1 · p. 7

4 Results

The production pipeline passes most items across the curriculum, but open-set cloze generation remains more capability-sensitive and harder to optimize and judge than closed-set formats. Independent judging and distillation further show that pass rates are judge-relative and that discrimination remains difficult.

4.1 Full-curriculum production quality

  • 97.8% of 9,074 final-run items passed the expert evaluator; all 36 grade×type cells fell within 4.3 points (95.3–99.6%).The production run covered 755 standards across grades 1–12; Table 2 summarizes pass rate and mean score.Source: §4.1 · p. 7, Table 2 · p. 8

4.2 The closed-set/open-set asymmetry

  • Under matched conditions, Grade-12 cloze spanned 13.9 points across five models, versus 3.9 points (95.2–99.1%) for four models with MCQ measurements.The MCQ comparison uses an overlapping, not identical, model set; cloze is a discriminative stress test that reveals a sharper capability ordering.Source: §4.2 · p. 8, Table 4 · p. 10

4.3 How much should we trust the judge?

  • Repeated verdicts flipped for 6.6% of cloze and 6.2% of MCQ items, far below the 2.8–16.8-point format gap; the operationally triggered audit bounds random, not shared systematic, error.Relaxing the audit’s pairing rule changes the cloze flip rate to 8.2%, while MCQ remains at 6.2%.Source: §4.3 · p. 11
  • At production prevalence, deployed-versus-independent judge agreement fell to κ=0.13; the format ordering persisted across judges, but model judges do not establish ground truth.The independent judges also agreed with each other no better, bounding the 97.8% rate as throughput past one evaluator’s gate rather than a measured defect rate.Source: §4.3 · pp. 11–12

4.4 Prompt optimization plateaus, and the error mass trades direction

  • As rules accumulated, over-inclusion fell from 8 to 3 cases while omissions rose from 2–3 to 4, leaving boundary-error mass roughly constant in a small audit.The audit covers 10–12 failing items per round, one grade, and one generator; the authors interpret the pattern as suggestive, not proof, of a precision–recall trade-off.Source: Figure 5 · p. 13, §4.4 · p. 13

4.5 What still fails: error taxonomy

  • Only the cloze difficulty-mismatch pattern survived error-label validation: 36.0% by keyword rules and 36.7% independently, while several other apparent categories failed validation.Keyword-rule agreement was weak (κ=0.27), and the independent labeller left 21–43% of failures without a fitting class.Source: §4.5 · pp. 13–14, Table 8 · p. 14

4.6 The asymmetry recurs when distilling the judge

  • Distilling the evaluator raised fail-recall to 63.1%, but F1 stayed at 0.25 past v2 as precision fell to 15.2%, reproducing the generation-side precision–recall trade-off.This result comes from a single 14B student, so it establishes the asymmetry in that setting, not scale-invariance.Source: §4.6 · pp. 14–15, Table 9 · p. 15

5 Analysis and Discussion

The discussion distinguishes self-consistent closed-set checks from cloze’s external semantic boundary, traces cloze failures to decoder behavior, and derives design implications while qualifying the evidence.

5.1 Closed-set vs. open-set generation

  • Cloze key construction must recover a fixed semantic boundary, unlike an MCQ’s self-consistent check; models checked membership better than they enumerated answers on mechanically decidable tasks [8].The paper frames cloze as discrimination embedded in generation.Source: §5.1 · p. 16

5.2 Three mechanisms

  • Three decoder properties map to cloze errors: sequential enumeration, predictive rather than discriminative representations, and uncalibrated membership thresholds.Sequential generation can omit rare synonyms; near-neighbor confusion contributes to over-inclusion, while instruction-following degrades under heavy rule loads.Source: §5.2 · pp. 16–17

5.3 Collocational cue leakage

  • High-frequency sentence frames can make a blank answerable from collocation alone, producing self-defeating items and difficulty mismatch.The paper reports that the R3 collocational-cue ban produced the largest reduction of any rule addition in that error class.Source: §5.3 · p. 17

5.4 Design implications

  • For open-set keys, the authors recommend rubric-based adjudication or external candidate verification rather than exhaustive key enumeration.The proposed alternatives include semantic matching against a rubric or a bidirectional-encoder verifier or dictionary/thesaurus tool call.Source: §5.4 · p. 17
  • 13.9 points separate models on cloze, versus 3.9 points on near-ceiling MCQ; the authors propose cloze as a capability probe.A dozen static rules saturate MCQ/MSQ, while additional rules’ marginal return is concentrated in cloze and the audited errors trade direction rather than shrink.Source: §5.4 · p. 17
  • Evaluator feedback can become standard-indexed guardrails through a knowledge dictionary and negative exemplars, without marginal labeling cost.The passage proposes storing structured critic feedback and retrieving it by task key.Source: §5.4 · pp. 17–18

5.5 Limitations

  • Weak judge agreement makes the headline pass level judge-relative; results also cover only ELA, CCSS, and English, with method and model partly confounded in aggregate comparisons.Human validation and student-response field testing were not reported; the round-by-round audit covers one grade and generator.Source: §5.5 · pp. 18–19

6 Conclusion

The conclusion presents CLAIM’s production-scale pass rate as throughput past one evaluator gate, not as a measured defect rate. It identifies open-set answer-key construction as a limit of prompt optimization and calls for calibrated boundaries.

6 Conclusion

  • Closing the loop between an LLM generator and structured evaluator reaches 97.8% evaluator-pass at full-curriculum scale, but independent judges agree weakly on accept/reject and no student-response evidence is reported.Prompt optimization plateaus on open-set answer-key construction, trading over-inclusion against omission rather than reducing both.Source: §6 · p. 19

Ethics Statement

Generated materials are synthetic K–12 ELA assessment items; the authors recommend human review before classroom deployment.

Ethics Statement

  • No student data were used or collected, and evaluator pass rates measure evaluator-defined quality rather than learning outcomes.The authors note that generated items may still encode subtle biases and recommend human review before classroom deployment.Source: Ethics Statement · p. 19

A Production Results by Grade

The supplied appendix material identifies the production table’s scope, excerpts cloze rule families, and gives an abridged Stage-2 adversarial prompt.

A Production Results by Grade

  • Table 10 reports final-production pass rate (%) by grade and item type for Claude Opus 4.6; the supplied caption gives no cell values.The main text identifies Table 10 as the per-grade breakdown.Source: Table 10 · p. 24

B Cloze Rule Families (Excerpt)

  • Cloze rules require answer-array consistency and block answer leakage, including answers outside a displayed word bank.The excerpts require every variant in the explanation to appear in the key, morphology to match instructions, and accepted answers not to appear in student-visible text.Source: B Cloze Rule Families (Excerpt) · p. 25
  • Other cloze rules target collocational cues, verbatim retrieval, difficulty step counts, and semantic suitability of candidate keys.The excerpts require restructuring cueable blanks, downgrading or redesigning retrieval-level items, recounting cognitive operations, and deleting meaning-shifting substitutions.Source: B Cloze Rule Families (Excerpt) · p. 25

C Stage-2 Attack Prompt (Cloze, Abridged)

  • The Stage-2 cloze prompt directs a hostile reviewer to test answer precision and completeness, cue leakage, self-consistency, and explanation claims.It instructs the reviewer to return corrected JSON only.Source: C Stage-2 Attack Prompt (Cloze,… · p. 25

D Example Mined Knowledge Rule

The mined rule corrects letter-shape misconceptions by specifying the forms of uppercase B and lowercase p.

D Example Mined Knowledge Rule

E Error-Category Taxonomy

The taxonomy keyword-matches evaluator feedback into error classes, but independent relabelling reveals over-firing and substantial failures left undescribed.

E Error-Category Taxonomy

  • The taxonomy assigns labels by matching feedback vocabulary, including phonetics, spelling, factual error, distractor quality, and difficulty mismatch.The rule can over-fire when feedback mentions a category’s vocabulary while faulting something else.Source: E Error-Category Taxonomy · pp. 25–26
  • The keyword rules have high recall where they fire but often low precision, consistent with over-firing when category vocabulary is merely mentioned.Overall primary-label κ is 0.27, mean Jaccard is 0.37, and exact label-set agreement is 0.14.Source: E Error-Category Taxonomy · p. 26, Table 11 · p. 26
  • The ten keyword-matched classes leave substantial feedback undescribed: independent relabelling assigns “other” 36% of the time versus 8% for the rule.“Other” captures unmatched feedback; labels may overlap because an item can receive multiple classes.Source: E Error-Category Taxonomy · pp. 25–26, Table 12 · p. 27
  • Independent relabelling finds cloze difficulty-mismatch concentration persists, while MCQ distractor-quality concentration reverses.Difficulty mismatch is +18.7 points by rule and +24.0 independently; distractor quality is +80.7 by rule and -7.3 independently.Source: E Error-Category Taxonomy · p. 26, Table 12 · p. 27

F Corpus Composition

The production corpus is near-balanced across grades and formats, but its requested difficulty labels do not establish empirical student difficulty.

F Corpus Composition

  • The 9,074-item run is organized by grade and format, with pass rates also reported by format and labelled difficulty.Grades 1, 3, 9, and 12 have extra cloze items from reruns during the production window.Source: F Corpus Composition · p. 28, Table 13 · p. 28
  • Cells are near-balanced by construction, except for extra cloze items in grades 1, 3, 9, and 12 from production-window reruns.Source: F Corpus Composition · p. 28, Table 13 · p. 28
  • “Easy” and “hard” are requested generator labels, so similar pass rates show evaluator judgments of appropriateness, not measured student difficulty.Empirical difficulty would require student-response data, which the paper lacks.Source: F Corpus Composition · p. 28, Table 13 · p. 28

G Generation-Method Audit

The method audit finds the aggregate model comparison confounded by generation method, while the two-model evidence does not estimate a reliable method effect.

G Generation-Method Audit

  • The aggregate model table pools single-pass and two-pass items, materially weakening its use as a model comparison.The corpus records two_pass_verify and single_pass_knowledge_grounded methods.Source: G Generation-Method Audit · p. 28
  • Only two models were measured under both methods, with opposing changes: Kimi K2.5 gains 6.2 points and GPT-OSS-120B loses 3.7.The samples are unbalanced, so these results do not support estimating the method effect.Source: G Generation-Method Audit · p. 28
  • The matched capability comparison uses Table 4 because its five models share the two-pass method, rule set, standards, judge, and grade.Several other models were run exclusively under one method, leaving method and model partly collinear.Source: G Generation-Method Audit · p. 28, Table 14 · p. 29

H Independent-Judge Cross-Validation: Details

Independent judges preserve the format ordering and identify a larger cloze deficit than the deployed evaluator, while agreement on binary acceptability remains limited. The cross-validation uses blinded re-scoring with a distinct rubric and examines both format effects and rejection patterns.

H Independent-Judge Cross-Validation: Details

  • The 390-item sample combines 100 randomly drawn items per format with 30 deployed failures per format to support κ estimation despite low production failure rates.Judges saw neither deployed verdicts nor scores and used an independent rubric that rejected only deployment-blocking defects.Source: H Independent-Judge Cross-Validation:… · p. 29
  • Gemini 3.1 Pro and GPT-5.1 agree at κ=0.40; their agreement with the deployed evaluator is 0.43 and 0.33, while pairs involving Qwen3-235B are near 0.11.The authors characterize binary acceptability as a low-agreement judgment among competent judges, limiting what any single-judge pass rate certifies.Source: H Independent-Judge Cross-Validation:… · p. 29
  • The format ordering holds under every judge, and the two frontier judges find a larger cloze deficit than the deployed evaluator.The deployed evaluator is the most permissive on cloze, making its measured format gap the smallest of the four.Source: H Independent-Judge Cross-Validation:… · p. 29, Figure 8 · p. 30
  • Both frontier judges reject 9.9% of deployed-passed cloze items, versus 1.0% of MCQ items and none of the MSQ items.Answer-key incompleteness dominates cloze rejections and is rare on MCQ.Source: H Independent-Judge Cross-Validation:… · pp. 29–30, Table 16 · p. 31
  • Qwen3-235B accepts 99.7% of items and recovers 8.3% of deployed failures, reproducing verdict form with little accept/reject discrimination.It also provides a check that an always-accepting judge cannot manufacture the format effect seen by the other judges.Source: H Independent-Judge Cross-Validation:… · p. 30
Loading 2609.26087v1…