Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks

• Reliable LLM-as-a-Judge techniques and psychometric methods remain scattered across papers, leaving practitioners to compose evaluation methods ad hoc and repeatedly reimplement them.The cited techniques include ensemble judging (Verga et al., 2024), position-bias mitigation (Wang et al., 2024a), and abstention; educational measurement and psychometrics include rubric design and inter-rater reliability (McKeown & Lenarcic Biss, 2018; Brookhart, 2018).Source: §1 · p. 1
• Analytic rubrics score independent criteria, reducing criterion conflation and halo effects, enabling criterion-level reliability measurement, and providing structured optimization signals (Lee et al., 2025; Wei et al., 2025).The passage names Cohen’s κ and weighted κ as reliability measures.Source: §1 · p. 2
• Autorubric is an open-source framework for mixed binary, ordinal, and nominal criteria, with atomic grading, bias mitigation, configurable ensemble aggregation, few-shot calibration, and reliability metrics.It provides opinionated defaults informed by LLM-as-a-judge, education testing, and psychometrics literature.Source: §1 · p. 2
• CHARM-100 is a synthetically authored chatbot dataset with binary, ordinal, and nominal criteria, per-sample reference labels, and a 50-item second-annotation agreement audit.Source: §1 · p. 2
• The framework is validated on three benchmarks, with exploratory configuration-sensitivity analysis across three judge-model families.Source: §1 · p. 2
• Two downstream applications use per-criterion rubric scores as optimization signals for agent skill improvement and reinforcement learning with rubric-based rewards.Source: §1 · p. 2

Rubric design

• Autorubric defaults to analytic rubrics, scoring criteria independently rather than assigning one holistic score.Rubrics pair criteria specifying what is evaluated with performance-level descriptions; criteria can also have configurable weights.Source: §2 · p. 3

Criterion types

• Criteria may be binary, bounded ordinal, or nominal, with explicit numeric values for multi-choice options and type-appropriate reliability metrics.Continuous-valued criteria are intentionally excluded; bounded categorical options are used instead.Source: §2 · p. 3

Weighting and aggregation

• Configurable positive and negative weights support rewards and anti-pattern penalties, with abstention handling determining which criteria contribute to the normalized score.Positive weights define the maximum attainable score when present; with only negative weights, the score starts at 1 and falls with triggered penalties, while no remaining nonzero weights yields 0.Source: §2 · p. 3

Evaluation modes and judging strategies

• Autorubric defaults to pointwise, reference-free judging and separates each criterion into its own LLM call to reduce conflation and halo effects.It supports optional references and diverse-model panels with majority, weighted, unanimous, or any-vote aggregation; mean inter-judge agreement is tracked as a reliability indicator.Source: §2 · p. 3

Calibration and reasoning

• Few-shot calibration uses verdict-balanced training examples, while reasoning effort is configurable because evidence for its benefits is mixed.Reasoning can be controlled through thinking levels or explicit token budgets per judge.Source: §2 · p. 3

Position bias

• Position bias affects LLM judges; Autorubric randomizes option order using explicit numeric values and reproducible per-item seeds.The per-item seed is derived from the master seed, making reruns reproducible.Source: §3 · p. 4

Low reliability

• Individual judgments show unexplained variance and stochastic self-inconsistency; per-criterion ensemble voting reduces variance and model-specific biases.A single master seed, persisted in the experiment manifest, controls non-LLM randomness. The cited work on variance is Feuer et al. (2025), and the work on self-inconsistency is Haldar & Hockenmaier (2025).Source: §3 · p. 4

Criterion conflation

• Joint evaluation can conflate distinct quality constructs; Autorubric makes a separate LLM call for each criterion.Concurrent execution and prompt caching offset evaluation throughput and cost, respectively. The cited studies are Lee et al. (2025) and Wei et al. (2025).Source: §3 · p. 4

Uncertainty

• When a submission lacks enough information to assess a criterion, Autorubric supports CANNOT_ASSESS with configurable SKIP, ZERO, PARTIAL, or FAIL strategies.This implements selective prediction’s option to abstain instead of forcing a label.Source: §3 · p. 4

Interpretability and auditability

• Autorubric requires an explanation for every verdict and preserves per-judge reasoning, but explanations are post-hoc rationalizations, not a substitute for human review.The explanations enable audits and downstream optimization; reviewers can inspect rationales behind ensemble disagreements. The caution cites Turpin et al. (2023).Source: §3 · p. 4

4.1 College-level chemistry grading (RiceChem)

• 5-shot calibration raises RiceChem held-out micro-accuracy from 78.0% (639/819) to 80.7% (661/819), a descriptive 2.7-point gain.RiceChem contains 1,240 human-graded responses across 27 binary criteria; the few-shot result remains below the 86.8% fine-tuned RoBERTa+MNLI baseline. Because decisions are nested within student responses, the gain is descriptive rather than cluster-adjusted inference.Source: §4.1 · pp. 4–5, Table 2 · p. 5

4.2 Deep research system evaluation (ResearcherBench)

• Both ResearcherBench judges rank Gemini > OpenAI > Grok3, but the Gemini–OpenAI gap is significant only under Sonnet-4.5 (p = 0.003, d = 0.39), not Gemini-3-Flash (p = 0.219).The top-two confidence intervals overlap. The benchmark covers 65 questions and 931 weighted binary criteria; the judges evaluated three systems using 5,586 criterion-level judgments.Source: §4.2 · p. 5, Table 3 · p. 6
• On ResearcherBench, judges agree more on question difficulty than system quality: per-question Spearman ρ = 0.54–0.82, while criterion-level agreement is κ = 0.53 (78.5% raw).Critical-analysis criteria have 27.9% disagreement versus 14–15% for enumeration and depth; Gemini-3-Flash is 5–6× cheaper while preserving the aggregate ranking.Source: §4.2 · pp. 5–6

4.3 Chatbot assessment with heterogeneous criteria (CHARM-100)

• CHARM-100 introduces 100 synthetic, annotated chatbot conversations with six criteria spanning binary, ordinal, and nominal scales, designed to expose reliance on a single quality score.Reference labels have mean normalized entropy 0.92, and a second annotation pass covers a stratified 50-item subset.Source: §4.3 · p. 6
• On CHARM-100, factual accuracy reaches 87.0% exact accuracy, while ordinal criteria reach 38–58% exact but 85–93% adjacent accuracy; response-length accuracy is 81.0%.Naturalness has the highest reported type-appropriate κ (0.719, quadratic-weighted); response length detects brevity better than verbosity (0.70 vs. 0.14 recall). The judge also shows positive bias (+0.170) despite score-level Spearman = 0.810; type-specific κ weighting means these results do not establish a general reliability hierarchy.Source: §4.3 · p. 6, Figure 2 · p. 7

4.4 Configuration sensitivity

• Configuration effects on CHARM-100 vary by judge family, with ablations and ensembles shifting outcomes inconsistently rather than supporting a universal mitigation stack.Under Default, pooled exact accuracy/κ is 60.4%/0.679 for Gemini, 51.0%/0.462 for GPT, and 19.4%/−0.001 for LLaMA. These single-run comparisons are descriptive and include model-output variability; cross-family aggregation trails Gemini alone on exact accuracy and κ but has slightly lower score RMSE and higher cost.Source: §4.4 · pp. 7–8, Table 4 · p. 7

5 Application: agent skill improvement

• Per-criterion rubric measurements served as an optimization signal in a demonstration peer-reviewing 10 scientific papers with Llama 3.1 8B against a 10-criterion binary rubric.The rubric covered outcome quality, style, efficiency, and a factual-misrepresentation penalty; the loop graded reviews, turned failing criteria into feedback, and sent it to a revision LLM.Source: §5 · p. 8
• A single rubric-guided revision raised the vague skill’s score from 0.47 to 0.85, versus 0.82 for the expert-curated skill; bootstrap confidence intervals overlap.The GPT-5.4 revision scored 0.86. The revised-versus-curated comparison does not establish superiority, and independently sampled revision arms do not isolate revision-model self-preference.Source: Figure 3 · p. 8, §5 · p. 8
• With GPT-5.4-mini grading fresh reviews, the vague-to-revised score rose from 0.58 to 0.74, while the expert-curated score was 0.78.Across both graders, improvement over the vague skill persisted, but the runs do not isolate revision-model or grading-model effects.Source: §5 · p. 8

6 Application: reinforcement learning with rubric-based rewards

• The run trains Qwen/Qwen3-4B-Instruct with LoRA on 402 complex_if_single_turn_v5 prompts, using gemini-3-flash-preview’s normalized rubric score as reward.The subset averages 7.44 expert-curated binary criteria per prompt and is split 80/20 for training and validation; 50 training steps cost approximately $12.Source: §6 · p. 9
• A separate GPT-5.4-mini check gives a smaller positive difference of +0.023, but is inconclusive and does not isolate judge coupling.The nominal paired Wilcoxon p = 0.243; the check changes both sampled responses and judge.Source: §6 · p. 9
• 0.795 vs. 0.756 (+0.039) is the mean validation rubric score at step 25 versus the base model on the 81-prompt split.Step 25 was the highest-scoring of ten monitored checkpoints; the nominal paired Wilcoxon p = 0.032 is unadjusted for checkpoint selection and descriptive, not an independent test. Mean rubric scores on training data rise from 0.774 at epoch 1 to 0.825 at epoch 3, then decline in epochs 4–5.Source: Figure 4 · p. 9, §6 · p. 9
• Style, phrasing, and format criteria show the largest gains through the selected checkpoint, while some content and factual criteria remain unresolved.Responses scoring 1.0 increase from 21/81 to 30/81, and mean validation response length falls 33% (1,097 to 733 tokens).Source: §6 · p. 9, Figure 4 · p. 9
• On IFEval, eight of nine instruction types show non-negative transfer; punctuation gains +4.5pp and detectable content gains +3.8pp, but transfer is not statistically significant.McNemar p = 0.28.Source: §6 · p. 9, Figure 4 · p. 9

7 Additional Related Works

• Autorubric draws criterion-level rubric principles from McKeown & Lenarcic Biss (2018) and psychometric diagnostics from Comer (2009) and He et al. (2025a), but leaves quality assurance to authors.Position bias, prompt sensitivity, and self-preference bias (Panickssery et al., 2024; Wataoka et al., 2024) lack educational precedent and are handled through Section 3 mitigations.Source: §7 · p. 10
• Autorubric targets user-defined analytic rubrics without task-specific calibration data, using generation-time mitigations rather than prior supervised or batch-calibration approaches.RULERS learns from 200+ labeled examples per dataset with fixed rubrics (Hong et al., 2026); option shuffling and verdict-balanced few-shot design address concerns analogous to pairwise decision balancing and batch-prior correction (Zhou et al., 2024a; Zhou et al., 2024b). Unlike Xia et al. (2025), which estimates confidence from inter-model agreement for factoid QA, Autorubric’s ensemble surfaces agreement as a reliability diagnostic.Source: §7 · p. 10

8 Discussion and conclusion

• On CHARM-100, factual accuracy had 87% exact accuracy, ordinal criteria had 38–58% exact and 85–93% adjacent accuracy, and naturalness had type-appropriate κ of 0.719.Criterion identity is confounded with type for the single binary and nominal criteria, and type-specific κ variants are not directly comparable; the paper recommends matching format to construct and reporting type-appropriate metrics.Source: §8 · p. 10
• Per-criterion scores diagnose judge agreement and disagreement and provide actionable feedback for skill revision and reinforcement-learning optimization.On CHARM-100, recall was 0.70 for brevity versus 0.14 for verbosity; pass rates in a peer-review skill loop rose from 0.47 to 0.85, and per-criterion RL rewards yielded a positive validation-selected difference on AdvancedIF with directionally positive transfer to IFEval.Source: §8 · p. 10
• Autorubric provides a common implementation, but consistency cannot establish construct validity.Source: §8 · p. 10
• The paper describes a March 2026 peer-review snapshot and reports that Autorubric gained capabilities by the August 2026 camera-ready submission and continues to evolve.Newer capabilities include agent-harness integration through a skill without coding, rubric-quality assessment, and rubric induction.Source: §8 · p. 10

Evaluation limitations

• LLM judges cluster at ordinal scale extremes, depressing exact accuracy on graded criteria, while some constructs resist binary categorization.Coarser scales or adjacent accuracy may partially mitigate clustering; multi-choice criteria address non-binary constructs but require more careful rubric design.Source: Limitations · p. 11
• Gemini-3-Flash at 5–6× lower cost matched Sonnet-4.5’s aggregate ResearcherBench ranking, but overlapping top-two bootstrap CIs and calibration needs limit cheaper judges.Inexpensive judges may be insufficient for criterion-level analysis requiring absolute calibration.Source: Limitations · p. 11
• Ensembling costs k× LLM calls, while same-model ensembles yield small, model-dependent changes and verbosity bias remains unmitigated.Adaptive ensembling for low-confidence items, length-controlled evaluation, and post-hoc calibration are potential directions.Source: Limitations · p. 11
• Single-run CHARM configurations confound mitigation comparisons with stochastic output variation, and tested mitigations cannot compensate for the weakest judge.Judge-family differences exceed within-model ranges for Gemini and GPT; the experiments do not identify causal mitigation effects.Source: Limitations · p. 11
• Experiments and default prompts are English-only, so adapting criterion prompts and few-shot exemplars to other languages remains untested.The framework is language-agnostic in design, but the study does not test other languages.Source: Limitations · p. 11

Optimization limitations

• The optimized skill is tightly coupled to its rubric and model, and factual-grounding failures persist in the tested setup.A different rubric may reward different tradeoffs.Source: Limitations · p. 11
• AdvancedIF uses the validation split both to select step 25 and estimate its gain, leaving the nominal p-value unadjusted and no independent test split evaluated.The independent-judge check regenerates responses, so it cannot separate judge effects from generation variability.Source: Limitations · p. 11
• The framework assumes well-designed rubrics, while assessing whether criteria measure their intended constructs remains an open problem.Source: Limitations · p. 11

• LLMs served as evaluation judges, revision models, and a reinforcement-learning reward model in the experiments.A language model also generated CHARM-100 conversations and reference labels, followed by a recorded second annotation pass on 50 items.Source: Generative AI Use Disclosure · p. 11
• Gemini 3 Pro Preview and Claude Sonnet 4.5 helped organize references, proofread drafts, make plots, and lightly rewrite the manuscript.Source: Generative AI Use Disclosure · p. 11
• The authors reviewed the resulting prose, figures, data artifacts, and citations and take full responsibility for the paper’s contents.Source: Generative AI Use Disclosure · p. 11

Ethics Statement

• Autorubric exposes agreement and per-criterion explanations, recommends human review for low-agreement cases, and cautions against unsupervised use in high-stakes assessment.Organizations should validate against human judgments for their populations and scale oversight to decision stakes; scores may obscure uncertainty and systematic error.Source: Ethics Statement · p. 12

A Design space taxonomy

• Figure 5 maps rubric-based LLM evaluation design across five dimensions, distinguishing Autorubric defaults, supported options, and covered but unimplemented paradigms.Green, blue, and gray dashed pills encode those categories, respectively.Source: Figure 5 · p. 16

B Framework details

• Appendix listings target Autorubric v1.5.3, while reported ResearcherBench and CHARM runs are historical artifacts from earlier or contemporaneous code versions, not v1.5.3 reruns.Later prompt and default changes describe current API behavior and are not retroactively attributed to those predictions.Source: B Framework details · p. 16

B.1 Code examples

• Appendix code examples illustrate analytic weighted rubrics, binary/ordinal/nominal criteria, positive and penalty weights, and multi-judge aggregation.The examples target v1.5.3 and show weighted, majority, unanimous, and any aggregation options.Source: §B.1 · pp. 16–18
• Further examples demonstrate balanced few-shot calibration, option shuffling for position-bias mitigation, strategies for unassessable criteria, and access to per-criterion explanations.The listings illustrate API capabilities rather than results from the reported experiments.Source: §B.1 · pp. 19–21

B.2 Educational measurement and psychometrics

• Educational measurement principles call for unidimensional, behaviorally anchored, construct-aligned criteria, but Autorubric neither verifies unidimensionality nor enforces alignment.Criterion-level decomposition makes requirements auditable, while multi-choice labels can provide behavioral anchors.Source: §B.2 · p. 21
• Autorubric provides Cohen’s κ for categorical judgments, quadratic weighted κ for ordinal scales, and Earth Mover’s Distance for distribution analysis, but not intraclass correlation.Reliability is necessary but insufficient for validity; content, criterion, and construct validity require separate assessment.Source: §B.2 · p. 21

B.3 Production infrastructure

• Production support includes caching, resumable checkpointing, provider rate limits, cost and timing tracking, and recorded configuration, though exact reconstruction also requires retaining code and inputs.The manifest does not serialize every prompt, model, or grader option.Source: §B.3 · p. 22

B.4 Default evaluation prompts / B.6 Correctness verification / C Rubric evaluation report example

• Default prompts require independent, evidence-based criterion judgments and reserve CANNOT_ASSESS for genuine uncertainty; correctness checks matched hand calculations.Prompts also instruct judges to review all options, avoid defaulting to middle choices, and use only specific evidence; the suite contains more than 400 tests. A sample report gives 92.2% accuracy, 0.95 precision, and 0.87 recall.Source: §B.4 · pp. 24–25, §B.6 · p. 26, C Rubric evaluation report example · p. 27

D.1 Dataset and conversion details / D.2 Per-question criteria and weights / D.5 Score distributions

• RiceChem conversion retained only positive criteria (Q1–Q4: 8, 6, 7, and 6), and score summaries identify Q4 as easiest and Q2 as most challenging.Blank or blank-flagged submissions were excluded (1, 11, 10, and 2 from Q1–Q4); binary annotations map TRUE to MET and FALSE to UNMET.Source: §D.1 · p. 28, Table 10 · p. 29, Table 14 · p. 32

D.3 Per-question and per-criterion reliability

• In the paired 5-shot rerun, Q3 had the highest aggregate κ (0.831), Q1 the lowest (0.435), and Q2 and Q4 moderate agreement (κ ≈0.49).A separate diagnostic replicate found six of seven Q3 criteria above κ=0.76, near-zero κ for very-high-MET-rate criteria despite high accuracy, and κ = −0.385 for Q1’s partial-credit criterion. The latter reflected credit for partial explanations that human graders did not award; high base rates also explain why Q4’s 86.2% accuracy coexisted with low κ.Source: §D.3 · p. 30, Table 11 · p. 30, Table 12 · p. 31

D.4 Weight inference

• Inferred weights clustered around 1 or 2 points per criterion; Q3’s R2 was 0.542, versus at least 0.985 for Q1, Q2, and Q4.Q3’s lower fit reflected 13 of 15 students receiving full marks despite all individual criteria being marked false, plus manual adjustments on 51 of 298 converted responses (17.1%).Source: §D.4 · pp. 31–32, Table 13 · p. 31

D.6 Paired comparison and exploratory few-shot sweep

• Accuracy rose from 78.0% (639/819) at 0-shot to 80.7% (661/819) at 5-shot on the same held-out decisions.The comparison had 54 decisions improve and 32 degrade; because decisions from each response are correlated, the counts and per-question differences are descriptive, not a significance claim.Source: §D.6 · p. 34

D.7 Cold-start results / D.8 Prior results from Sonkar et al.

• Prior in-distribution results from Sonkar et al. (2024) show RoBERTa-large + MNLI substantially outperforming zero-shot GPT-4; cold-start results evaluate training on three questions and testing on the fourth.The cold-start setup simulates rubric application to a new question type.Source: §D.7 · p. 34, §D.8 · p. 35, Table 17 · p. 35

E.1 Dataset description / E.2 Rubric structure

• ResearcherBench contains 65 questions across 34 AI research subjects, with responses from 11 AI systems; expert-designed criteria use integer weights 1–3.Weights 1, 2, and 3 represent nice-to-have details (35% of criteria), supporting points (51%), and core findings (14%), respectively.Source: §E.1 · p. 35, §E.2 · p. 36

E.3 Coverage score / E.4 Systems evaluated

• ResearcherBench’s weighted binary coverage score matches Autorubric’s normalized EvaluationReport.score; evaluated systems were OpenAI DeepResearch, Gemini-2.5-Pro DeepResearch, and Grok3 DeepSearch.No score transformation is needed; raw_score stores the unnormalized weighted sum.Source: §E.3 · p. 36, §E.4 · p. 36

E.4 Systems evaluated

• The benchmark also includes Grok3 DeeperSearch, Perplexity Deep Research, GPT-4o Search Preview, Sonar Reasoning Pro, Claude, Doubao, Mita, and Perplexity Sonar; three popular systems were selected to keep costs reasonable.Source: §E.4 · p. 36

E.5 Cross-judge agreement analysis; E.6 Paired permutation test for system ranking

• The judges agree on the full three-system ranking for only 15 of 65 questions (23.1%); Gemini–OpenAI ranking significance also depends on judge choice.They agree on the top-ranked system 57% of the time and the bottom-ranked system 52%; mean per-question Spearman ρ is 0.433 over 61 questions. The paired test finds a difference under Sonnet-4.5 (p = 0.003, d = 0.39) but not Gemini-3-Flash (p = 0.219).Source: §E.5 · p. 37, §E.6 · p. 37, Table 20 · p. 37

E.7 Per-criterion agreement by weight band; E.8 Disagreement taxonomy

• Agreement varies by criterion and system, while disagreement is elevated for critical-analysis and quantitative-threshold criteria.Weight-1 criteria have the lowest κ (0.477); Grok3 has the highest system-level agreement (κ = 0.654), while Gemini DeepResearch has the lowest (κ = 0.398), especially on core criteria (κ = 0.314). Critical-analysis disagreement is 27.9% versus 21.5% overall, and quantitative-threshold disagreement is 36.7% on N = 30; the keyword-derived analysis is descriptive and does not establish that rewriting criteria improves reliability.Source: Table 21 · p. 37, §E.7 · p. 38, Figure 10 · p. 38

E.9 Cross-judge score differences; E.10 Cost analysis; E.11 Rubric statistics

• Gemini-3-Flash assigns higher mean coverage scores than Sonnet-4.5 for all three systems, while evaluation of all systems by both judges costs $130.06.The equal-weight score difference is +0.103, but varying magnitudes and the lack of independent adjudication do not establish which judge is better calibrated. Gemini-3-Flash is 5.7–6.3× cheaper; criteria counts range from 6 to 21 per question (mean 14.3).Source: §E.9 · p. 39, §E.10 · p. 39, Table 23 · p. 39

F CHARM-100 dataset; F.1 Motivation; F.2 Dataset overview; F.3 Annotation schema

• CHARM-100 addresses single-scale benchmark limits by evaluating binary, ordinal, and nominal criteria together, with type-appropriate agreement metrics.Each sample has six reference labels; they are operational annotations, not independently adjudicated truth. Ordinal criteria use quadratic weighted κ, while nominal and binary criteria use unweighted Cohen’s κ; specificity N/A cases are excluded under SKIP. Reproducing the March 2026 label space with v1.5.3 requires auto_na_option=False.Source: F CHARM-100 dataset · p. 42, §F.1 · p. 42, §F.2 · p. 42

F.4 Design principles; F.5 Topic and domain coverage

• CHARM-100 deliberately combines conflicting criterion outcomes and varied response quality, and its description-derived quality strata are heuristic audit metadata, not released annotations.Its 100 samples use unique system prompts across more than 35 topic domains and more than 25 assistant personas. Post hoc topic clusters cover 6–17 items each (normalized entropy H/ log 9 = 0.98) and are not intended to estimate deployment traffic.Source: §F.4 · pp. 43–44, §F.5 · p. 44

F.6 Ground truth label distributions

• CHARM-100 label distributions range from relatively balanced ordinal criteria to skewed response-length and factual-accuracy labels, and their designed prevalences are not deployment base rates.The four ordinal majority-class baselines are 27–36% with normalized entropies of 0.93–0.98; response length and factual accuracy have majority baselines of 66% and 72%.Source: §F.6 · pp. 44–45, Table 28 · p. 45

F.7 Response anti-pattern taxonomy

• CHARM-100’s response-failure taxonomy tests factual, helpfulness, naturalness, and length judgments, including compound failures that require independent criterion assessment.Factual examples include subtle hallucinations and misleading framing; helpfulness examples include evasive or circular answers, while naturalness and length examples include stilted prose, excessive disclaimers, and buried useful content.Source: §F.7 · pp. 45–46

F.8 Edge cases; F.9 Sample format; F.10 Summary statistics; F.11 Second-annotation agreement

• The benchmark includes borderline cases and single-turn samples; a second-annotation audit found criterion-specific agreement but is not a full-dataset human-reliability estimate.The audit covered a fixed, description-stratified 50-item subset: naturalness κ = 0.870, response length κ = 0.506, and specificity κ = 0.565. Its descriptive macro-average κ = 0.687 combines differently weighted coefficients and is not interpreted against a qualitative threshold. Samples contain one system, user, and assistant message; assistant responses range from 9 to 497 words (mean 167).Source: §F.8 · p. 46, §F.9 · p. 46, Table 29 · p. 47

F.12 Limitations

• The benchmark uses synthetic, English-only, single-turn conversations and a fixed six-criterion rubric, limiting ecological validity and generalization to other languages, dialogue settings, or rubrics.Agreement was measured on a fixed 50-item subset, with criterion-level κ estimates based on n = 50 except specificity (n = 47); the remaining 50 items have reference labels only. The dataset is also a static snapshot that misses quality changes over time and model versions.Source: §F.12 · p. 48

G.1 Confusion matrices

• Ordinal CHARM-100 predictions clustered toward the highest category; factual accuracy had 0.97 recall and 0.86 precision, while specificity exact accuracy was 32/81.Specificity adjacent accuracy was 70/81, and response-length classification detected brevity with 0.70 recall but detected verbosity with 0.14 recall. Specificity results exclude cases with N/A on either side.Source: Figure 15 · p. 49, Table 32 · p. 49, Table 36 · p. 52

G.2 Aggregate metrics; G.3 N/A handling analysis

• The judge predicted N/A 16 times against 9 reference labels, with 10 false positives; aggregate mean bias was positive, indicating higher ratings than the reference on average.N/A-status agreement was 87.0% and Cohen’s κ was 0.412; SKIP aggregation omits predicted N/A verdicts from each item’s weighted-score denominator. No inferential test was reported for mean bias.Source: §G.3 · p. 49, Table 38 · p. 52

G.4 Configuration-sensitivity details

• The cross-family majority-vote panel achieved 57.1% pooled exact accuracy but showed no general accuracy or reliability advantage over the strongest single judge, Gemini (60.4% exact accuracy, κ = 0.679).The ensemble had κ = 0.626, ρ = 0.769, and RMSE = 0.200; Gemini’s RMSE was 0.213.Source: §G.4 · pp. 50–51

H Agent skill improvement details

• Rubric-guided skill revision raised the small model’s score from 0.47 to 0.85, but did not establish superiority over the expert-curated skill.The revised skill’s point estimate of 0.85 slightly exceeds the curated estimate of 0.82, but their confidence intervals overlap and no superiority test was conducted. Concise reviews fell from 100% to 50%, and factual misrepresentation persisted in 40–60% of reviews in the 10-paper setup.Source: H Agent skill improvement details · p. 54

H.1 Cross-model robustness probes

• Cross-model probes found similar revision scores of 0.85 and 0.86 under Gemini grading and a vague-to-revised increase from 0.58 to 0.74 under GPT grading.Revised-versus-curated confidence intervals overlapped in every reported comparison, with no superiority or equivalence test. The GPT run used freshly generated reviews, so differences conflate review sampling and grader choice and do not estimate judge calibration or bias.Source: §H.1 · p. 55, Table 40 · p. 55

I Cross-benchmark reliability

• CHARM-100’s ordinal criteria had 44.4% exact accuracy versus 87% for factual accuracy and 81% for response-length accuracy, but ordinal adjacent accuracy was 87.3%.Because criterion identity is confounded with type and κ weighting differs by type, the results do not support binary criteria as a universal default; CHARM-100 does not compare scale widths.Source: I Cross-benchmark reliability · p. 56, Figure 19 · p. 57

J RL training details

• Validation scores peaked at 0.795 at step 25, while training scores fell from 0.825 at epoch 3 to 0.698 at epoch 5, alongside later entropy decline.Step 25 was selected from ten monitored checkpoints rather than by a prespecified stopping rule; validation fell to 0.629 by step 45. The diagnostics show no simple monotonic verbosity-exploitation pattern but cannot exclude other reward misspecification.Source: J RL training details · p. 57

J.1 Cross-judge robustness probe

• The probe grades freshly regenerated base-model and step-25 responses on all 81 validation prompts with GPT-5.4-mini, so it changes both response realization and judge and cannot isolate judge coupling.The original training reward and evaluation judge was gemini-3-flash-preview; generation used temperature 1.0 to match training conditions.Source: §J.1 · p. 57
• Neither this probe nor the IFEval transfer results provides a confirmatory estimate of the selected checkpoint’s in-domain effect.Step 25 was selected after observing the checkpoint curve, and no independent test split was evaluated.Source: §J.1 · p. 58, Figure 21 · p. 59
• Under GPT-5.4-mini, the trained-model mean rounds to 0.795 and the base-model mean is 0.772 vs. 0.756 under Gemini, reducing the observed difference from +0.039 to +0.023.The nominal paired two-sided Wilcoxon result is inconclusive (p = 0.243; Cohen’s d = 0.13, 95% bootstrap CI [−0.09, 0.35]); 25 prompts improve, 38 are unchanged, and 18 regress.Source: §J.1 · pp. 57–58, Table 41 · p. 58