PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

Berke Arda, Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mubashara Akhtar, Mrinmaya Sachan

arXiv:2610.28767v1cs.CLcs.AIcs.LG

TL;DR

Scalar benchmark scores obscure the varied skills underlying LLM performance. SkillEval discovers interpretable skills and uses a cognitive diagnosis model to estimate per-skill abilities; its profiles support prediction, routing, and efficient evaluation of unseen models.

  • Problem

    Average benchmark scores collapse performance across items into a scalar, obscuring fine-grained capability differences such as distinct algebra and geometry profiles.

    Source: §1 · pp. 1–2

  • Method

    SkillEval discovers shared skills from problems and uses a neural cognitive diagnosis model with a question-to-skill mapping to estimate each model’s skill-level abilities.

    Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 3

  • Results

    SkillEval produces stable, interpretable profiles that predict model performance and support routing; skill-conditioned routing matches the strongest LLM at 4× lower inference cost.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Skill profiles support analysis of benchmark skill coverage, co-mastery patterns, and skill-level differences between base and instruction-tuned models.

    Source: §1 · p. 2, §6 · p. 10

    Skill-based restrictions limit fine-grained modeling of individual items: IRTNet achieves +3.2 points in pooled AUC on unseen items.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the diverse scope of skills covered by benchmark datasets. To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis. The framework integrates advances from two areas: (1) it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per-skill abilities of LLMs. We conduct experiments on 3,811 LLMs evaluated on 9,523 items across five benchmarks spanning diverse domains (MATH, BBH, GPQA, MuSR, IFEval). Compared to the traditional average-based evaluation paradigm, SkillEval produces reliable, more calibrated and informative yet concise model profiles that more accurately predict model behavior on unseen items. We also demonstrate the utility of SkillEval in two downstream tasks: (1) LLM routing, which dynamically assigns each query to a capable but efficient model, reducing inference cost by over 60% while achieving improved performance, and (2) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM. Furthermore, we show that these skill profiles provide useful insights, for example, how benchmarks share skills and how instruction-tuned models differ from their base models.

1 INTRODUCTION

Scalar benchmark scores obscure fine-grained capability differences, motivating a more informative, human-understandable way to estimate LLM abilities. SkillEval discovers latent skills and estimates skill-level abilities, with reported gains in prediction, routing, and adaptive evaluation.

1 INTRODUCTION

  • Average scores collapse item-level performance into one number and can hide different skill profiles, even for models with similar math accuracy.The example compares scores of 0.63 and 0.65 despite potentially different algebra and geometry profiles.Source: §1 · pp. 1–2
  • SkillEval combines automatic discovery of shared skills from problems with a neural cognitive diagnosis model that estimates per-skill abilities from associated item performance.The evaluation covers 3,811 models and 9,523 items across MATH, BBH, GPQA, MuSR, and IFEval.Source: §1 · p. 2
  • SkillEval predicts model performance at item and benchmark levels, supports cost-efficient routing, and profiles unseen models with fewer adaptively selected answers.It reports AUC = 81.73 and Pearson r = 0.989; at τ=0.80, it matches the strongest LLM at 25% of its cost, while 20 adaptive answers match the mean benchmark-score error of 100 random answers.Source: §1 · p. 2
  • The framework is presented as producing human-understandable skills and stable ability estimates, alongside useful analyses of benchmark skill coverage and model differences.The described analyses include co-mastery patterns and skill-level differences between base and instruction-tuned models.Source: §1 · p. 2

2 PROFILING LLMS’ SKILLS WITH COGNITIVE DIAGNOSTIC MODELS

SkillEval replaces a single average ability score with profiles over latent skills, using an item-to-skill Q-matrix and a neural cognitive diagnosis model. It discovers and clusters skills from item text, then estimates item response probabilities from model mastery and item parameters.

2.1 PROBLEM STATEMENT

  • SkillEval maps model-item scores to per-model skill profiles, avoiding a single scalar that can obscure skill variation and be biased by item difficulty distributions.Each profile is a K-dimensional vector of proficiency over the underlying skills.Source: §2.1 · pp. 2–3

2.2 Q-MATRIX CONSTRUCTION

  • The Q-matrix assigns items to skills extracted from question text, then clusters similar skill strings and marks an item-skill link when an extracted skill belongs to that cluster.Full-list extraction averages 2.9 skills per item; the selected K=100 setting balances interpretability and performance.Source: §2.2 · p. 3

2.3 NEURAL CDM

  • The neural CDM estimates each model’s probability of answering an item correctly from skill mastery, item discrimination and difficulty, and the item’s Q-matrix row.Non-negative PosLinear weights and non-decreasing sigmoid activations ensure increasing mastery cannot decrease predicted correctness, holding other inputs fixed.Source: §2.3 · p. 4

2.4 LEARNING ITEM PARAMETERS FROM TEXT

  • To predict on unseen items, SkillEval derives item discrimination and difficulty from text features using a lightweight neural network rather than learned item-specific parameters.The pretrained encoder is frozen; only the projection parameters are optimized.Source: §2.4 · p. 4

3 EVALUATION OF SKILL PROFILES

This section evaluates the quality and stability of discovered skill profiles and tests whether they predict held-out model behavior. SKILLEVAL’s benchmark-level predictions are more accurate within each benchmark than IRTNet’s, despite similar pooled correlations.

3 EVALUATION OF SKILL PROFILES

  • The experiments use 3,811 LLMs, 9,523 items, and five benchmarks, with K=100 skills as the main configuration.The authors chose K=100 as a balance between interpretability and performance.Source: §3 · p. 4

3.1 QUALITY OF DISCOVERED SKILLS

  • The 100 extracted skills match Common Core semantic clarity, are more fine-grained with lower granularity std. (0.29 vs. 0.55), and are more diverse (0.83 vs. 0.76).In human validation, annotators accepted 652 of 780 skill–question assignments (83.6%) by majority vote.Source: §3.1 · pp. 4–5, Table 1 · p. 4

3.2 STABILITY

  • SKILLEVAL’s ability profiles correlate 0.96 across retrainings, versus approximately 0.00 for IRTNet; after alignment, IRTNet’s correlations rise only to 0.10–0.43.Per-dimension correlation tests whether each dimension assigns similar relative values to LLMs across runs.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 PREDICTIVE VALIDITY

  • IRTNet leads SKILLEVAL by 3.2 AUC and 1.8 accuracy points on OOD item prediction, while skill-linked dimensions better support stable model comparisons.SKILLEVAL’s restriction of items to their assigned skill dimensions makes profiles interpretable and stable, but limits item-specific modeling and leaves predictions sensitive to Q-matrix errors.Source: §3.3 · pp. 6–7, Table 3 · p. 6
  • 0.977 vs. 0.852 on MATH and 0.966 vs. 0.718 on IFEval: SKILLEVAL is more accurate within every benchmark despite similar pooled correlations to IRTNet.SKILLEVAL’s predictions stay close to the diagonal, while IRTNet’s spread more widely and tend to overestimate accuracy on MATH, GPQA, and MuSR.Source: §3.3 · p. 6, Figure 2 · p. 6

4 APPLICATIONS

SkillEval-based routing improves query-level model selection while balancing accuracy and inference cost. For unseen LLMs, adaptive question selection supports accurate skill profiling with fewer answers than random selection.

4.1 ROUTING LLMS VIA ABILITY PROFILES

  • SKILLEVAL achieves the highest Acc@1 at 65.67%, versus 61.57% for the strongest single LLM, 61.10% for IRT 2PL, 62.05% for KNN, and 63.08% for IRTNet; its gains over IRT 2PL are largest on MuSR (+12.5 points) and IFEval (+14.0 points).Across three retrainings, its Acc@1 is (65.46 ± 0.34)%.Source: §4.1 · pp. 7–8, Table 4 · p. 7
  • At τ=0.80, SKILLEVAL reaches 99.9% of the strongest single LLM’s accuracy at 25% of its cost; at τ=0.95, it exceeds that accuracy by 3.7 percentage points at 37% of its cost.At matched cost budgets in the plotted range, SKILLEVAL also achieves higher accuracy than IRTNet.Source: Figure 3 · p. 8, §4.1 · p. 8
  • On 723 of 1,905 test items (38.0%), an LLM with at most 13B parameters answers correctly while the training-selected strongest single LLM fails.These inversions occur in every benchmark and span 41 skill clusters.Source: §4.1 · p. 8

4.2 PROFILING UNSEEN LLMS

  • For unseen-LLM profiling, SkillEval freezes the trained diagnostic network and estimates only each new model’s K-dimensional profile from N answers to training-pool items.The evaluation uses 200 held-out LLMs and separate test items excluded from profile estimation.Source: §4.2 · p. 8
  • Adaptive question selection chooses each next item using the current profile’s predicted correctness probabilities, then updates the profile from all answers observed so far.The selection score favors uncertain items and items assigned to several skills; both selection methods use the same final estimation procedure.Source: §4.2 · pp. 8–9
  • With 20 adaptive answers, benchmark-score error is 0.061, comparable to 100 random answers; adaptive selection also yields higher profile agreement at every tested budget.After 100 answers, profile agreement is 0.53 for adaptive selection versus 0.45 for random selection.Source: Figure 4 · p. 9, §4.2 · p. 9

5 DIAGNOSTIC ANALYSES

The diagnostic analyses compare estimated skill mastery in matched base and instruction-tuned models and map skill coverage across benchmarks. The reported comparisons also distinguish statistical significance from causal evidence.

Skill-level differences between base and instruction-tuned models

  • Among 81 matched pairs, instruction-tuned models have mastery more than 0.01 higher on 38 skills and more than 0.01 lower on 36; 14 skills differ significantly after Bonferroni correction.Eleven significant differences are positive, mostly on instruction-following skills; three are negative, with magnitudes at most 0.06. IFEval has the largest benchmark mean difference (∆=+0.126, p<10−10).Source: §5 · pp. 9–10, Figure 5 · p. 10
  • The base-versus-instruction-tuned comparisons are observational and do not isolate instruction tuning’s causal effect; statistical significance does not establish a large effect.Source: §5 · p. 10

Skill coverage across benchmarks

  • Thirty of 100 skills occur in only one benchmark, including 13 unique to IFEval, 9 to GPQA, and 6 to MATH; MATH and IFEval together cover 87 skills, rising to 98 with GPQA.BBH adds one further skill despite contributing 5,761 items; withholding IFEval removes training items for its unique skills, and routing performance on IFEval deteriorates.Source: §5 · p. 10

6 CONCLUSION

SKILLEVAL builds named-skill mastery profiles for LLMs and uses them to predict behavior, route queries, and reveal differences that aggregate scores miss. Its analyses span benchmark examples, model comparisons, and skill-cluster construction.

6 CONCLUSION

  • Across 3,811 LLMs and five benchmarks, SKILLEVAL produces profiles over 100 named skills, with skill estimates correlating 0.96 across retraining runs.It predicts benchmark scores at Pearson r=0.989; routing matches the strongest LLM at four times lower inference cost, and 20 adaptively chosen answers yield mean benchmark-score error comparable to 100 random answers.Source: §6 · p. 10

A RELATED WORK

  • Prior psychometric and LLM-evaluation methods often use scalar or continuous latent ability representations, whereas SKILLEVAL models mastery over semantically named skills.The paper positions its contribution as scaling LLM-derived skill extraction across five benchmarks and fitting a CDM over those skills.Source: A RELATED WORK · pp. 13–14

C SKILL CLUSTER EXAMPLES

  • Skill profiles distinguish conjunctive from compensatory requirements: BBH accuracy is 74% when both skills are mastered versus 22–33% with only one, while either IFEval skill can suffice.On the IFEval item, accuracy is 97% with word-count skill alone and 42% with Urdu skill alone, compared with a 12% baseline.Source: C SKILL CLUSTER EXAMPLES · p. 15, Table 7 · p. 15

D Q-MATRIX CONSTRUCTION DETAILS

  • The Q-matrix uses multiple extracted skills per item, and HAC clustering was selected for highest downstream test AUC at each tested K despite a lower silhouette score.Multi-skill assignment removed empty columns and reduced under-populated clusters; question-only versus solution-aware extraction showed no clear aggregate judge preference among retained items.Source: D Q-MATRIX CONSTRUCTION DETAILS · pp. 15, 17

E TRAINING DETAILS

  • Training uses binary cross-entropy on the 80% training split, with non-negative weights and Adam optimization for 15 epochs.The main model takes about three hours to train on a single NVIDIA RTX 2080 Ti.Source: E TRAINING DETAILS · p. 17

F FULL K-SWEEP RESULTS

The K-sweep finds diagnostic fit largely unchanged across cluster counts, while routing accuracy rises modestly. For unseen models, adaptive answer selection improves both benchmark-score error and profile agreement over random selection at tested budgets.

F FULL K-SWEEP RESULTS

  • Test AUC stays at 0.716–0.721 across K=50–500, while routing Acc@1 rises from 0.640 at K=50 to 0.676 at K=500; K=100 is the default.The authors choose K=100 to retain interpretable named clusters with about four times fewer parameters than K=500; Table 8 reports the sweep.Source: F FULL K-SWEEP RESULTS · p. 17, Table 8 · p. 18

G PROFILING UNSEEN LLMS: DETAILS

  • The unseen-model procedure freezes the diagnostic network and item parameters, then estimates a new model’s K-dimensional profile from its answers to training-pool items.Its prior is based on the raw profiles of 3,048 training LLMs, with covariance shrunk 10% toward its diagonal.Source: G PROFILING UNSEEN LLMS: DETAILS · p. 17

K Test AUC Acc@1 Acc@3 Acc@5 Acc@10

  • For unseen LLMs, adaptive selection beats random selection on benchmark-score error and profile agreement at every tested budget from 5 to 500 answers.The evaluation uses the first 200 of 763 held-out LLMs; paired 95% bootstrap intervals favor adaptive selection on both measures.Source: K Test AUC Acc@1 Acc@3 Acc@5 Acc@10 · p. 18
  • Per-LLM test-item AUC changes only from 0.68 with no answers to 0.69 with all answers, so the authors assess additional answers using score error and profile agreement.With zero answers, every LLM receives the same profile, so this AUC does not require estimating differences between LLM skills.Source: K Test AUC Acc@1 Acc@3 Acc@5 Acc@10 · p. 18

H PER-BENCHMARK ROUTING

SkillEval’s routing advantage over the strongest-model and IRT 2PL baselines appears across all five source benchmarks, with the largest stated gains on MuSR and IFEval.

H PER-BENCHMARK ROUTING

  • SkillEval beats the strongest-model baseline and IRT 2PL on all five benchmarks, with its largest gains over IRT on MuSR (+12.5 pts) and IFEval (+14.0 pts).Source: H PER-BENCHMARK ROUTING · p. 18

I WEAK-BEATS-STRONG INVERSIONS

On held-out items, many queries have a capable smaller-model option even when the globally strongest model fails. These cases span all five benchmarks and 41 skill clusters.

I WEAK-BEATS-STRONG INVERSIONS

  • 723 of 1,905 held-out items (38%) have a correct answer from at least one LLM with ≤13B parameters when the global strongest model fails, spanning 41 skill clusters.The benchmark counts are BBH 329, MATH 156, GPQA 155, MuSR 64, and IFEval 19.Source: I WEAK-BEATS-STRONG INVERSIONS · pp. 18–19

J CROSS-BENCHMARK TRANSFER

Holding out MuSR and IFEval exposes sharply different transfer: SkillEval performs comparably to the strongest-model baseline on MuSR but falls below random on IFEval. The analysis also finds skill mastery retains information beyond overall and benchmark-level accuracy, though not beyond finer subtask accuracy.

J CROSS-BENCHMARK TRANSFER

  • Training uses MATH, BBH, and GPQA, with 8,231 training items and 1,292 held-out items from MuSR and IFEval.The experiment holds out both benchmark families entirely while retaining K=100 clusters, 3,811 LLMs, and the main run’s training schedule.Source: J CROSS-BENCHMARK TRANSFER · p. 19
  • AUC 0.469 (test accuracy 0.460) on combined held-out items indicates chance-level transfer; by benchmark, Acc@1 is 0.470 on MuSR versus 0.467 for the strongest-model baseline, but 0.410 on IFEval versus 0.437 random and 0.731 strongest-model.The MuSR score remains below the full-training reference of 0.544; the IFEval result is 32 points below the strongest model.Source: J CROSS-BENCHMARK TRANSFER · p. 19, Table 11 · p. 20
  • Fourteen of IFEval’s 62 assigned skills receive no response-based training, but this coverage gap does not fully explain its transfer failure.Twenty-three percent of IFEval items involve at least one untrained skill, while the share is below 1% for MuSR; most IFEval items still involve trained skills.Source: J CROSS-BENCHMARK TRANSFER · p. 19
  • 38% of held-out items (723/1,905) have at least one model with 13B or fewer parameters outperforming the global strongest model, across 41 skill clusters.The items span BBH (329), MATH (156), GPQA (155), MuSR (64), and IFEval (19).Source: Figure 6 · p. 20

K CONTROLS FOR THE VALIDITY OF Θ

  • Mean partial correlation is 0.35 after controlling for overall training accuracy and 0.135 after controlling for benchmark accuracies, but falls to 0.021 after controlling for subtask accuracy.The corresponding descriptive placebo means are 0.22, -0.02, and the passage reports substantial overlap with subtask structure for the final comparison.Source: K CONTROLS FOR THE VALIDITY OF Θ · p. 20

L EMPIRICAL CO-MASTERY PATTERNS

The co-mastery analysis turns asymmetric skill mastery into directed links, then checks whether the resulting structure is robust and useful for distinguishing models. The graph is statistical rather than causal, and individual links vary with the mastery threshold.

L EMPIRICAL CO-MASTERY PATTERNS

  • An LLM counts as mastering a skill when its predicted probability of a correct answer, averaged over that skill’s items, exceeds 0.5.The estimated mastery θ alone is interpreted relative to item difficulty and does not specify expected accuracy.Source: L EMPIRICAL CO-MASTERY PATTERNS · pp. 20–21
  • Among models mastering dice-roll combinatorics, 52 of 55 also master factorial divisibility, while only 52 of 167 divisibility masters also master dice-roll combinatorics.The resulting conditional shares are P(divisibility | dice) ≈ 0.95 and P(dice | divisibility) ≈ 0.31, so the link points from divisibility to dice rolls.Source: L EMPIRICAL CO-MASTERY PATTERNS · p. 21
  • A link from A to B requires P(A | B)−P(B | A) > 0.3 and an estimated odds ratio above 10, excluding links driven only by differing mastery frequencies.The odds ratio uses a Haldane correction that adds 0.5 to each count to keep it finite when a count is zero.Source: L EMPIRICAL CO-MASTERY PATTERNS · p. 21
  • The filtered graph retains 304 links among 86 skills, but only 29 direct links persist across mastery cut-offs of 0.4, 0.5, and 0.6.All links point from more commonly mastered skills to less commonly mastered ones; their direction does not establish a learning prerequisite.Source: L EMPIRICAL CO-MASTERY PATTERNS · p. 21
  • The co-mastery graph agrees with a second graph on all but one pair both graphs order, and label shuffles produce at most 51 links versus 1,031 originally.Across retained links, median P(A | B) is 0.87 and median P(B | A) is 0.26; the authors note the graph can suggest skills to assess within groups sharing a strength.Source: L EMPIRICAL CO-MASTERY PATTERNS · p. 21
  • Across five model-family pairs, link-set Jaccard overlap ranges from 0.09 to 0.31, so detected links are only partly shared across sampled families.Whether these patterns persist in other LLM populations remains untested.Source: L EMPIRICAL CO-MASTERY PATTERNS · p. 21
  • IFEval contributes 13 unique skills and BBH one; MATH and IFEval together cover 87 of the 100 skills.MuSR’s skills are 82% covered by BBH and 82% by IFEval, while IFEval has the lowest average overlap with other benchmarks.Source: Figure 7 · p. 22, Figure 8 · p. 22

M BENCHMARK REDUNDANCY DETAILS

Benchmark skill coverage differs substantially, with IFEval contributing distinct skills and MuSR overlapping heavily with other benchmarks. Estimated mastery changes from base to instruction-tuned models also vary by benchmark and family.

M BENCHMARK REDUNDANCY DETAILS

  • MuSR’s skills are 82% covered by BBH and 82% by IFEval, while IFEval has the lowest average overlap with other benchmarks.IFEval contributes 13 unique skills, BBH contributes one, and MATH plus IFEval cover 87 of 100 curated skills.Source: M BENCHMARK REDUNDANCY DETAILS · p. 22

N DIFFERENCES BETWEEN BASE AND INSTRUCTION-TUNED MODELS: DETAILS

  • Mean mastery differences are positive on IFEval for every evaluated family and negative on MuSR for every family.MATH and GPQA differences are near-zero or slightly negative for most families; Llama shows the broadest gains, while Qwen is essentially flat.Source: N DIFFERENCES BETWEEN BASE AND… · p. 23, Figure 9 · p. 23

O LIMITATIONS

The paper bounds its claims by the populations, item pools, and evaluation designs studied. In particular, cross-benchmark transfer and skill-taxonomy construction leave important generalization and validation limits.

O LIMITATIONS

  • New-LLM experiments cover 200 of 763 held-out models, one trained network, and one adaptive rule at answer budgets up to 500.The authors do not establish an advantage for adaptive selection in every setting.Source: O LIMITATIONS · p. 23
  • The co-mastery graph captures statistical association, not causal skill prerequisites; establishing causal relations would require skill-specific intervention experiments.The observed structure may reflect skill-difficulty ordering and shared training-data signals.Source: O LIMITATIONS · p. 23
  • Q-matrix validation uses downstream NCDM AUC rather than human adjudication, and the human check tests whether assigned skills are needed, not whether needed skills are missing.Items near cluster boundaries warrant caution; HAC, the extraction prompt, and SBERT source were not separately varied.Source: O LIMITATIONS · p. 23
  • Item-wise OOD evaluation is transductive with respect to the skill representation: test-item text helps define the taxonomy, though test responses are excluded from parameter fitting.IRTNet outperforms SkillEval on held-out-item prediction, and joint exclusion of MuSR and IFEval yields below-random IFEval routing; reliable transfer to new domains remains unestablished.Source: O LIMITATIONS · pp. 23–24
  • SkillEval covers five English-language benchmarks, leaving multilingual reasoning, tool use, long-context retrieval, safety/red-team benchmarks, and open-ended generation outside its evaluated scope.Extending coverage requires skill extraction on new items, assigning skills to existing clusters or expanding K, and retraining NCDM.Source: O LIMITATIONS · p. 24

Related papers

Loading 2610.28767v1…