Source-linked AI summary
LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model
Ergan Shang, Weijing Tang, Yinqiu He
TL;DR
The paper addresses how to predict LLM performance on unseen questions when item difficulty, context, and capability demands vary and new annotations are costly. It develops contextual multidimensional IRT, combining latent model capabilities with question embeddings, and finds improved within-scenario prediction but variable cross-scenario transfer. The results support context-aware psychometric evaluation while identifying cross-scenario generalization as an open challenge.
Problem
Predicting LLM behavior on unseen questions is difficult because heterogeneous item characteristics can confound model abilities, while collecting new labels is costly.
Method
The paper uses contextual multidimensional IRT to model LLMs with latent capability profiles and question characteristics informed by embeddings.
Results
C-MIRT improves within-scenario prediction and is competitive across most scenarios, but cross-scenario performance varies and is weaker for LLM-wise than question-wise AUC.
Takeaways & Limitations
Context-aware psychometric modeling is promising for efficient and interpretable LLM evaluation, but within-scenario success does not ensure robust cross-scenario generalization.
Takeaways & Limitations
Cross-scenario prediction is substantially harder than within-scenario prediction, so the method should not replace direct testing on representative data.
Abstract
from arXiv · showhide
Evaluation of large language models (LLMs) increasingly requires predicting how a model will perform on new questions or tasks before collecting large amounts of new annotations. This problem is challenging because question difficulty, scenario, and underlying capability demands can vary substantially. Simple retrospective averages may confound model ability with item characteristics. In this paper, we study a model-based evaluation framework that combines multidimensional item response theory model with question contexts to predict LLM performance on unseen questions. The framework represents LLMs through latent capability profiles while using question content to inform item characteristics, allowing information to transfer beyond previously observed items. Empirically, we find that for within-scenario evaluation, incorporating question embeddings improves prediction relative to model-free baselines, and that multidimensional latent structure provides a richer description of capability variation than unidimensional alternatives. At the same time, our results reveal an important limitation that the generalizability does not necessarily translate into reliable prediction under cross-scenario shift. These findings suggest that context-aware psychometric modeling is a promising direction for efficient and interpretable LLM evaluation, while also highlighting cross-scenario generalization as a central open challenge.
1. Introduction
Predicting LLM performance on unseen questions matters for efficient, trustworthy evaluation, but heterogeneous item characteristics make simple averages unreliable. The paper proposes contextual multidimensional IRT and reports stronger within-scenario prediction alongside limited cross-scenario transfer.
- Rapidly changing models, benchmarks, and deployment settings make full-scale annotation pipelines difficult to maintain.
- Predicting unseen-question performance supports deployment decisions, model selection, and risk assessment while reducing costly new annotations.
- Question heterogeneity in difficulty, context, and demanded capabilities can confound model ability with item characteristics when using average correctness.
- The proposed contextual MIRT model uses question semantics to inform item parameters while representing LLMs through richer latent capability structure.
- Question embeddings improve prediction over model-free alternatives, while multiple latent dimensions describe capability variation more richly than a unidimensional approach.
- The method targets both efficient unseen-question evaluation and cross-scenario generalization, extending prior psychometric and embedding-based approaches.
2. Method
C-MIRT separates scenario-specific model abilities from question characteristics by mapping question embeddings into a latent space and combining them bilinearly. Once learned, this structure supports predictions for unseen questions within or across scenarios.
- C-MIRT separates model-side latent abilities from question-side characteristics across scenarios to predict responses on unseen questions.
- For each scenario, binary correctness observations are paired with question embeddings that are mapped into an r-dimensional latent question representation.
- A scenario-specific model intercept captures baseline success, while a latent capability vector represents each model’s abilities.
- The bilinear interaction between model capabilities and transformed question embeddings captures alignment between abilities and question requirements.
- Unlike unidimensional Rasch-style difficulty, C-MIRT allows question characteristics and model capabilities to interact across multiple latent dimensions.
- A learned feature map embeds new questions for response-probability prediction within the training scenario or across target scenarios.
3. Experiments
The experiments evaluate C-MIRT on 11 scenarios using cross-validation, measuring prediction for unseen questions within and across scenarios. C-MIRT performs strongly within scenarios, while cross-scenario transfer varies by criterion and comparator.
- Experimental setup: The benchmark retains 11 scenarios from 22 datasets across five HELM repositories, with 436 to 29,407 questions per scenario.Scenarios include wikifact and math.
- Experimental setup: Five-fold cross-validation trains on 80% of questions and tests on the remaining 20% within each scenario.Question-wise AUC ranks LLMs for each fixed question.
- C-MIRT prediction: Diagonal heatmap entries show strong within-scenario prediction, while several off-diagonal entries indicate partial cross-scenario transfer.Off-diagonal variation suggests transfer depends on source-target scenario similarity.
- C-MIRT prediction: For LLM-wise AUC, diagonal performance remains strong but off-diagonal values approach 0.5, indicating less transferable ranking of question difficulty than ranking LLMs.Cross-scenario generalization is stronger for question-wise AUC than for LLM-wise AUC.
- Comparison with competing methods: The comparison heatmaps report mean 90th-percentile AUC differences, with gray denoting nonsignificant differences and red or blue identifying the better method.Paired t-tests assess significance across five cross-validation folds.
- Comparison with competing methods: C-MIRT consistently outperforms Mean and Rasch baselines for question-wise AUC, whereas comparisons with lasso are more scenario-dependent.For LLM-wise AUC, C-MIRT remains competitive across most scenarios.
4. Conclusions and future work
The paper proposes C-MIRT to predict LLM evaluations on unseen questions. It improves within-scenario prediction, but cross-scenario generalizability depends on the source-target scenarios and evaluation criterion.
- C-MIRT is proposed for predicting LLM evaluations on unseen questions.
- Within-scenario prediction improves, whereas cross-scenario generalizability depends on the source-target scenarios and criterion.
Impact Statement
The impact statement presents C-MIRT as a supplement to LLM evaluation pipelines rather than a replacement for direct testing, benchmark design, independent validation, or human oversight.
- C-MIRT may reduce annotation and computation costs while providing structured information about model strengths and weaknesses.
- Cross-scenario prediction is substantially harder than within-scenario prediction, so predicted performance should not substitute for direct testing on representative data.
- Embedding-based item models may inherit biases from benchmark data or pretrained text representations, and predicted performance may be mistaken for reliability evidence.
A. Appendix
The appendix formulates C-MIRT through a two-stage estimation procedure, while addressing factorization identifiability with centering and orthogonal-transformation constraints. Stage 1 estimates latent representations from responses; stage 2 maps contextual embeddings to question factors for unseen-question prediction.
- Identifiability and optimization: Without embedding parameterization, latent question factors are estimated freely, and the factorization is identifiable only up to an orthogonal transformation.The equivalence is expressed through U(s) = Ū(s)O and V(s) = V̄(s)O for O in O(r).
- Estimation procedure: C-MIRT fits a low-rank logistic factorization separately for each scenario to estimate intercepts and latent row and question representations.The response matrix follows a Bernoulli model with a logistic link, and the latent representations are collected in U(s) and V(s).
- Identifiability and optimization: The model uses a centering constraint on question factors and a regularizer to address identifiability and encourage balanced factorizations.The centering constraint is V(s)⊤1_ps = 0_r.
- Identifiability and optimization: Stage 1 is optimized with projected gradient descent, updating α(s), U(s), and V(s), then projecting to enforce the centering constraint.The projection step follows each gradient-descent update.
- Estimation procedure: In stage 2, a multilayer perceptron regresses estimated question factors on contextual embeddings to learn the feature map ϕ(s).The regression uses estimated factors from stage 1 as targets and contextual embeddings e_j as inputs.
- Estimation procedure: The learned feature map predicts latent representations for unseen questions and thereby produces out-of-sample response probabilities.This transfers question-factor estimation from observed questions to questions represented by contextual embeddings.