Source-linked AI summary
AI-Enabled Quality Assurance for Multiple-Choice Assessment Items
Steven Moore, Nicholas Diana
TL;DR
Scaling MCQ generation leaves the assessment quality of generated or existing items unresolved, and different flaws require distinct forms of review. This focused review maps automated detection and downstream evidence across fourteen reports, finding uneven evidence across criteria and stages. It concludes that detection, remediation, and assessment outcomes require separate evaluation.
Problem
Scalable MCQ authoring does not resolve how to determine whether generated or existing items are suitable for a particular assessment.
Source: §1 · p. 1Method
Results
Evidence is uneven across criteria and stages; TruthfulQA revisions reduced items with at least two detected flaws from 97% before revision to 64–68% afterward, while 56–68% had newly detected flaws.
Source: §4.4 · p. 6, §7 · p. 7Takeaways & Limitations
Evaluate detection, remediation, and assessment outcomes separately, then test how they work together; rubric flags can organize human attention before deployment, with expert review and response-based validation.
Source: §1 · p. 1, §7 · p. 7The bounded corpus is not exhaustive, screening lacked independent replication, and the authors did not rerun detectors or reanalyze participant data.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Generating multiple-choice questions is increasingly scalable, but establishing their assessment quality remains difficult. We present a focused narrative review of automated item-writing flaw detection, revision, psychometric screening, and NLP benchmark auditing. Database searches, citation retrieval, and nominated sources yield fourteen research reports reviewed in full text. We distinguish surface checks from content-sensitive judgments and map a 19-criterion rubric to detection methods and reported evidence. High label-level accuracy often coexists with weak positive case detection, while rubric definitions and reference standards vary. Revision evidence is mixed, and the associations reported in prior work do not establish the effects of repair. We propose evaluating quality assurance as a sequence of independently validated decisions, with criterion-specific reporting, calibrated human review, and outcome-based assessment of revisions.
1 Introduction
Scaling MCQ authoring does not remove the need to establish assessment quality: different item-writing flaws require different forms of review. This review maps automated checks to rubric criteria and separates flaw detection, item revision, and measurement outcomes.
1 Introduction
- MCQs can be grammatically inconsistent, implausible, or allow a defensible second answer, while fluency and answer-key agreement do not establish that an item measures its intended construct.Reference-overlap metrics such as BLEU and ROUGE assess similarity to reference text, not distractor defensibility or construct measurement (Moore et al., 2024).Source: §1 · p. 1
- The Item-Writing Flaws approach assigns named flags to specific problems, and SAQUET operationalizes a 19-criterion rubric using rules, NLP, and selective LLM verification (Moore et al., 2024).Related applications address revision, psychometric and IRT analysis, and audits of MCQs used to evaluate language models, but make different evidentiary claims.Source: §1 · p. 1
- The review maps automated IWF application criterion by criterion, compares detection with downstream evidence, and proposes linking flags to assessment decisions.It distinguishes detecting a rubric violation, improving an item, and improving measurement; success at one stage does not establish success at the next.Source: §1 · p. 1
2 Review Scope and Method
The authors conducted a focused narrative evidence map using database searches, citation retrieval, and nominated sources. Fourteen reports were reviewed in full text, with extraction designed to preserve differences in samples, rubrics, evaluation units, and outcomes.
2 Review Scope and Method
- Eligible studies empirically evaluated automated MCQ flaw detection or correction, or connected automated flaw annotations to authoring, psychometric, or benchmark outcomes.Generation-only studies, exam-taking studies without evaluated flaw screening, general reviews, and unrelated assessment formats were excluded; related technical-flaw rubrics were not assumed equivalent to all 19 IWF criteria.Source: §2 · p. 2
- Fourteen reports were reviewed in full text: seven papers met the database-search criteria, and seven additional reports came from seed, citation-linked, and nominated sources.The searches returned 212 records; after duplicate and version removal, 210 reports were screened and 203 excluded.Source: §2 · p. 2
- Extraction recorded item samples, reference standards, rubric versions, detectors, metric denominators, and outcomes, prioritizing criterion-level performance, positive-class F1, and chance-corrected agreement.Dataset reuse and incompatible units precluded pooling performance or summing sample sizes; missing metrics remained unreported, and aggregate accuracy was not treated as recall.Source: §2 · p. 2
3 What the Rubric Operationalizes
The IWF rubric provides a vocabulary for specific avoidable item flaws, not a complete theory of assessment quality. Its application depends on the information available, source-specific definitions, and item format.
3 What the Rubric Operationalizes
- IWF criteria describe avoidable cueing, ambiguity, or irrelevant task demands, but do not establish content coverage, curricular alignment, fairness, or intended cognitive demand.SAQUET’s routes include visible-pattern checks, NLP relationships among words and options, and selective LLM verification for judgments a pattern alone cannot resolve (Moore et al., 2024).Source: §3 · p. 2
- Equally named criteria are not necessarily operationalized identically: SAQUET, Costello et al. (2018), BenchMarker (Balepur et al., 2026), and NBME-based evaluations use differing definitions or thresholds.The review preserves source definitions rather than treating similarly named scores as interchangeable.Source: §3 · p. 2
- Rubric applicability depends partly on learner-available information: a transcript-based question may rely on a video diagram and fail as a standalone item.Firoozi et al. (2025) implement six guidelines, including character-count ordering, unlike SAQUET’s numerical or chronological sequencing rule.Source: §3 · p. 2
- Quality assurance should record item format and intended learner population, and allow uncertain or not-applicable judgments, because legitimate design choices can otherwise appear defective.Multiple defensible answers may violate a single-best-answer item but not an intentionally scored multi-select task; underscores in code need not be sentence-completion flaws.Source: §3 · pp. 2–3
- Table 1 is an evidence map rather than a system ranking, and reported performance applies to the specified task and reference standard.Samples, rubric definitions, model versions, and outcomes differ across its rows.Source: Table 1 · p. 3
4 Evidence across the Assessment Workflow
Detection metrics can conceal weak positive-case identification, and revision findings vary across evaluation settings. Psychometric associations and benchmark changes motivate targeted review but do not establish the effects of repair.
4.1 Detection and the accuracy denominator
- 94.13% accuracy on 5,149 item-criterion decisions is close to the 93.09% always-negative baseline; another ambiguity signal had 92.9% specificity but only 35.3% sensitivity.SAQUET also reported 38% exact match, domain micro-F1 of .59–.67, and 75.3% agreement on a separate acceptability threshold; these measure different tasks.Source: §4.1 · pp. 3–4
- GPT-4 detected 12 of 44 expert-identified flaws (27.3%), while rule-based and LLM flag counts in another study were not error rates against expert labels.The dental study had no defect-free controls for estimating specificity and reported no separately validated repair-success rate.Source: §4.1 · pp. 4–5
4.2 Revision and author interaction
- Revision evidence was mixed: optimized sequences were preferred for stems but not options, and evaluator groups differed in their ratings of revised questions.The stem comparison used intentionally flawed inputs; another study found no significant aggregate difference among external evaluators, while calibrated study-author ratings favored revisions.Source: §4.2 · p. 5
4.3 Psychometric and cognitive grounding
- Difficulty screening reached R2 = .216 overall and .030 in mathematics, but a flag–difficulty association is not the effect of fixing that flag.A low-difficulty screening example had precision .801 and recall .472, leaving missed cases and unexplained variation.Source: §4.3 · p. 5
- Confusion-category flags were associated with longer response times across three datasets, but the USMLE indirect estimate was approximately 2.5 times the aggregate estimate and was domain-specific.The category-level associations are not validated weights for individual criteria; observational item-level data cannot establish a causal cognitive mechanism.Source: §4.3 · pp. 5–6
4.4 NLP benchmark auditing
- Gemini 2.5 Pro’s writing-flaw F1 was .66 on NLP annotations and .39 on educational annotations; filtering writing errors also changed model rankings beyond random filtering.These annotation results are not whole-item correctness, and filtered subsets may differ in content and difficulty.Source: §4.4 · p. 6
- TruthfulQA revision reduced items with at least two detected flaws from 97% to 64–68%, yet newly introduced flaws were detected in 56–68% of revised items.Automated assessments do not establish expert-confirmed repair quality or improved measurement.Source: §4.4 · p. 6
5 Research Priorities
The review prioritizes portable rubric definitions, consequence-aware policies, and validation of the full human–AI review and revision workflow. Proposed evaluations should connect defect removal to assessment outcomes without treating flags or easier items as success by themselves.
Make definitions and uncertainty portable
- Release exact rubric wording, applicability rules, model versions, examples, annotation disagreements, intended populations, and item purposes.Evaluation sets need enough positive cases per criterion; six SAQUET criteria have fewer than ten positive reference cases, limiting what their F1 values establish.Source: §5 · p. 6
Prioritize consequences, validate the policy
- Prioritize flags by correctness, assessment consequence, and checking cost, rather than treating every flag as equally important.Cognitive-category evidence motivates prioritization, but does not validate universal severity weights; priorities should be tested on held-out domains and independently judged revisions.Source: §5 · p. 6
Evaluate the complete human-AI workflow
- Evaluate the complete human–AI workflow against unaided review, flags alone, flags with explanations, and suggested revisions.Measure expert-confirmed defect removal, introduced defects, reviewer time, escalation, and missed problems among unflagged items; later pilots can assess measurement outcomes.Source: §5 · p. 6
Design paired item comparisons
- Use paired item comparisons that preserve the learning objective, change the targeted flaw, and assign learners to only one version.Shared anchor items can support a common scale; removing a cue may increase difficulty while improving validity, so neither faster responses nor easier items alone define success.Source: §5 · p. 6
6 Limitations
The review’s bounded search and screening process does not establish exhaustive literature coverage, and the authors did not independently replicate screening, rerun detectors, or reanalyze participant data. Its evidence base and comparisons are also constrained by small or reused annotations, differing rubrics and evaluation units, and observational cognitive findings.
Review scope
- The literature search was bounded, relied partly on nominated sources, and used AI-assisted screening without independent replication.The corpus is not exhaustive, and the first author co-authored five of the fourteen reports, including the toolkit anchoring Table 2.Source: §6 · p. 7
Evidence base
- The authors did not rerun detectors or reanalyze participant data, and much of the evidence uses small or reused annotation sets.Comparisons span different rubric versions and evaluation units.Source: §6 · p. 7
Interpretation and validation
- Cognitive findings are observational, while the proposed workflow and consequence-based priorities still require validation.Source: §6 · p. 7
7 Conclusion
AI-enabled item-writing-flaw review offers interpretable screening support, but evidence varies across criteria and stages of use. The review calls for separate validation of detection, remediation, and assessment outcomes, followed by testing how they work together.
Conclusion
- AI-enabled item-writing-flaw review offers interpretable support for screening multiple-choice questions, but evidence is uneven across criteria and stages of use.Source: §7 · p. 7
- Surface checks are often tractable, while content-sensitive judgments and reliable revision remain harder.Source: §7 · p. 7
- Evaluate detection, remediation, and assessment outcomes separately, then test how they work together.Rubric flags can organize human attention before deployment; expert review and response-based validation establish whether items serve their intended purpose.Source: §7 · p. 7