PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable and Efficient Amortized Model-based Evaluation

Sang Truong, Yuheng Tu, Percy Liang, Bo Li, Sanmi Koyejo

arXiv:2503.13335v1cs.CLcs.AIcs.LGstat.AP

TL;DR

Subset-based average scores can confound model ability with question difficulty, while holistic LM evaluations are costly. The paper combines IRT with content-based difficulty prediction and difficulty-conditioned question generation; its experiments report greater reliability and efficiency with reduced query complexity.

  • Problem

    Holistic LM evaluations are costly, and average scores from benchmark subsets can confound model ability with question difficulty.

    Source: §1 · p. 1

  • Method

    The paper uses IRT with a content-based question-difficulty predictor for amortized calibration and a conditional question generator to build questions at target difficulties.

    Source: §1 · p. 2

  • Results

    IRT is reported as more reliable and efficient than model-free evaluation, reducing query complexity to 53% on average and up to 86% across datasets while reliably estimating ability across test sets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach makes model-based evaluation more practical by reducing difficulty-estimation costs and helping maintain a large, diverse question bank.

    Source: §6 · p. 8

    Generated-question quality depends on training data and difficulty-prediction accuracy, and AI-generated questions risk bias, making human review and refinement important.

    Source: §6 · p. 8

Abstract

from arXiv · show

Comprehensive evaluations of language models (LM) during both development and deployment phases are necessary because these models possess numerous capabilities (e.g., mathematical reasoning, legal support, or medical diagnostic) as well as safety risks (e.g., racial bias, toxicity, or misinformation). The average score across a wide range of benchmarks provides a signal that helps guide the use of these LMs in practice. Currently, holistic evaluations are costly due to the large volume of benchmark questions, making frequent evaluations impractical. A popular attempt to lower the cost is to compute the average score on a subset of the benchmark. This approach, unfortunately, often renders an unreliable measure of LM performance because the average score is often confounded with the difficulty of the questions in the benchmark subset. Item response theory (IRT) was designed to address this challenge, providing a reliable measurement by careful controlling for question difficulty. Unfortunately, question difficulty is expensive to estimate. Facing this challenge, we train a model that predicts question difficulty from its content, enabling a reliable measurement at a fraction of the cost. In addition, we leverage this difficulty predictor to further improve the evaluation efficiency through training a question generator given a difficulty level. This question generator is essential in adaptive testing, where, instead of using a random subset of the benchmark questions, informative questions are adaptively chosen based on the current estimation of LLM performance. Experiments on 22 common natural language benchmarks and 172 LMs show that this approach is more reliable and efficient compared to current common practice.

1. Introduction

The introduction motivates reliable, efficient evaluation across language-model benchmarks and presents an IRT-based approach using predicted question difficulty and generated questions. It reports lower query complexity while maintaining reliable ability estimates across test sets.

1. Introduction

  • Holistic evaluation is costly because benchmarks can contain hundreds of thousands of questions, requiring substantial model-computing and grading resources.Answer generation may take hours, days, or weeks per model; grading can require hundreds of annotator hours or thousands of dollars for LM judges.Source: §1 · p. 1
  • Average scores from benchmark subsets can confound model ability with question difficulty, making comparisons unreliable when models use different subsets.The introduction gives privacy-constrained hospital datasets and adaptively adjusted AI-security questions as examples.Source: §1 · p. 1
  • IRT models test-taker ability and question difficulty together, enabling test-invariant ability estimates across different question subsets.In this setting, test takers are LMs, items are questions, and responses are scores on generated text.Source: §1 · p. 2
  • Amortized calibration predicts question difficulty from content, reducing calibration cost complexity to constant with respect to question-bank size.The approach uses a machine-learning model as a content-based difficulty predictor.Source: §1 · p. 2
  • A conditional question generator creates questions at a target difficulty, automating construction of a diverse question bank.The generator is a language model trained to generate questions conditioned on difficulty.Source: §1 · p. 2
  • IRT reduces query complexity to 53% on average and up to 86% across datasets while reliably estimating model ability with different test sets.The study covers 22 NLP datasets and 172 LLMs, and reports greater reliability and efficiency than a model-free approach.Source: §1 · p. 2

2. Related Work

Prior work has explored subsampling, question selection, and IRT for reducing evaluation costs or improving measurement. This paper distinguishes its approach by combining amortized calibration with automated question generation for iterative evaluation.

2. Related Work

  • Prior cost-reduction approaches include model prioritization, confidence-based core sets, and question-sampling strategies (Perlitz et al., 2023; Vivek et al., 2023; Xu et al., 2024).The cited discussion notes concerns about random-subsampling error and the lack of rigorous theory or vulnerability to spurious patterns in core-set selection.Source: §2 · p. 2
  • Prior IRT applications reduce evaluation examples, study dataset saturation, generate response matrices, improve leaderboard rankings, or target natural-language inference (Vania et al., 2021; Lalor et al., 2019; Maia Polo et al., 2024; Rodriguez et al., 2021; Lalor et al., 2018).These works apply IRT to different evaluation goals and settings.Source: §2 · pp. 2–3
  • Unlike these approaches, this paper combines amortized calibration with LLM question generation to support long-term, iterative evaluation beyond static benchmarks.The method overview depicts difficulty prediction and difficulty-conditioned question generation as parts of the evaluation pipeline.Source: §2 · p. 3

3. Preliminary

The preliminary section defines Rasch IRT evaluation, then describes calibration of question difficulty and adaptive scoring of a test taker’s ability.

3. Preliminary

  • Rasch IRT models the probability of a correct answer as a logistic function of the difference between test-taker ability and question difficulty.Responses are Bernoulli: 1 for correct and 0 for incorrect.Source: §3 · p. 3
  • Calibration estimates test-taker ability and question difficulty from a response matrix, using EM or Bayesian MCMC inference.The calibrated question bank contains questions with estimated difficulty; MCMC provides a posterior distribution but is computationally expensive.Source: §3 · p. 3
  • Adaptive scoring estimates a new test taker’s ability with K ≪ N questions, selecting questions by Fisher information and updating ability from each response.The process repeats until reliability criteria are met or the question budget is depleted.Source: §3 · pp. 3–4

4. Method

The method predicts question difficulty from content to reduce the cost of calibration, then trains a difficulty-conditioned question generator to expand the bank for adaptive testing.

4.1. Amortized Calibration

  • Content-based amortized calibration predicts a new question’s difficulty, avoiding the large test-taker response collection required to calibrate each added question.It uses question content and dataset context to share parameters across questions and datasets.Source: §4.1 · p. 4

4.2. Adaptive Testing with Question Generator

  • A difficulty-conditioned question generator addresses small or skewed banks by producing questions at target difficulties, including those sought by adaptive testing.It is trained with supervised fine-tuning followed by PPO, rewarding generated questions whose predicted difficulty is close to the target.Source: §4.2 · p. 4

5. Experiments

Across the experiments, IRT-based measurement generalizes across sampled questions and models, while amortized calibration approximates traditional calibration. Adaptive testing reduces query requirements, and difficulty-conditioned question generation improves prediction-error performance over SFT.

5. Experiments

  • Across 22 datasets and 172 test takers, the single-ability Rasch model achieved 0.85 train and 0.83 test AUC; 2PL and 3PL did not outperform it.The authors attribute substantial response-prediction power to question difficulty, noting there were three orders of magnitude more questions than test takers; added parameters increase overfitting risk.Source: §5 · pp. 4–5

5.1. Generalization of Model-based Measurement

  • IRT achieved 0.78 ± 0.07 AUC versus 0.5 ± 0.07 for subset average scores, which were more sensitive to sampled questions.The Rasch score generalizes by separating ability from difficulty using historical responses.Source: §5.1 · pp. 5–6, Figure 3 · p. 6

5.2. Amortized Calibration

  • Amortized calibration closely approximated traditional calibration across datasets, train and test splits, and the reported metrics.The conclusion was robust to using Mistral 7B v0.3 embeddings instead of Llama-3-8B embeddings.Source: Figure 4 · p. 7, §5.2 · p. 7

5.3. Adaptive Testing with Question Generator

  • Adaptive testing reduced queries by an average 53% and up to 86% while achieving 95% reliability, with gains consistent across datasets.On AIRBench, Fisher-large reached 95% reliability with 31 queries; the small-bank strategy did not reach the same reliability at the same budget.Source: §5.3 · p. 7, Figure 6 · p. 8
  • Difficulty-conditioned question generation with search reduced prediction error nearly 10x versus SFT, with mean differences of 0.12 on training and 0.15 on test data.The generator is trained to produce questions conditioned on dataset description and target difficulty, and selects the best of 64 candidates at inference.Source: §5.3 · pp. 7–8, Figure 7 · p. 8

6. Conclusion, Limitations, Future Work

The paper uses IRT to separate model ability from specific test subsets, pairing amortized calibration with a conditional question generator to make model-based evaluation more practical. It also identifies dependencies and risks that require further validation and human oversight.

6. Conclusion, Limitations, Future Work

  • Amortized calibration predicts question difficulty from content, while a conditional question generator creates questions at target difficulty levels to help maintain a large, diverse question bank.Together, these methods address the cost of difficulty estimation and question-bank construction.Source: §6 · p. 8
  • Generated-question quality depends on training data and difficulty-prediction accuracy, and AI-generated questions can carry bias that calls for expert review.The paper also notes that model responses may vary with sampling parameters, few-shot examples, and chain-of-thought prompting.Source: §6 · p. 8
  • Future work includes incorporating additional response factors into IRT, improving question validation, extending IRT to non-binary assessments, and testing the approach in broader domains.Source: §6 · p. 8

Impact Statement

The impact statement presents the question generator as a possible tool for expanding or replacing question sets, while emphasizing human review to address bias risks.

Impact Statement

  • The generator could replace overused questions, expand datasets, or construct new ones, but these uses may introduce bias that can affect fairness and reliability.Source: Impact Statement · p. 9
  • Human reviewers are presented as essential for identifying and mitigating bias in generated content and preserving fairness and accuracy.Source: Impact Statement · p. 9
  • The statement frames responsible use as collaboration between human expertise and AI-generated question creation.Source: Impact Statement · p. 9

A. Summary of Datasets and Models

The appendix summarizes a broad collection of benchmarks and evaluated models. The supplied table captions identify the table scopes, while accompanying passages provide dataset descriptions and model-list material.

A. Summary of Datasets and Models

  • The dataset descriptions span tasks including safety, bias, toxicity, reasoning, question answering, reading comprehension, and factual knowledge.Examples named in the passages include AirBench, BBQ, MATH, RAFT, NarrativeQA, and TruthfulQA.Source: A. Summary of Datasets and Models · p. 11
  • The appendix includes datasets for language and social-bias evaluation, such as TwitterAAE, which measures performance in tweets as a function of speaker dialect.Source: A. Summary of Datasets and Models · p. 11
  • Table 1 is scoped to the number of test takers and questions per benchmark, while Table 2 is scoped to the complete evaluated-model list.Source: Table 1 · p. 12, Table 2 · p. 12

C. Synthetic Questions from Question Generator

This appendix presents generated-question examples and notes that the generators received limited human modification. Its accompanying figures describe embedding-model calibration comparisons and ability-estimate distributions.

C. Synthetic Questions from Question Generator

  • Figure 10 compares two embedding models across datasets and reports close metric alignment, suggesting embedding choice has minimal impact on calibration outcomes.Blue and red points denote training and test splits, respectively.Source: Figure 10 · p. 16
  • Figure 11 shows IRT ability estimates covering ground-truth ability, while subset-dependent CTT estimates split into two modes that do not cover it.The figure presents Common Sense, Raft, and AIRBench distributions.Source: Figure 11 · p. 16
  • Generated questions are shown for two base models, with a maximum token length imposed and only minimal human modification.The examples are drawn from datasets represented in the appendix.Source: C. Synthetic Questions from Question… · p. 17
  • The generators struggle to stop properly and adhere to benchmark formats, though the appendix says further training could improve these behaviors.Source: C. Synthetic Questions from Question… · p. 17

10. Entity matching

The examples present product-listing pairs with the question of whether each pair refers to the same item.

10. Entity matching

  • Each example asks whether two product listings describe the same item.The pairs include printer or ink-cartridge listings and a cordless-phone listing.Source: §10 · pp. 20–21
  • Several listings contain missing descriptions or prices, so the examples do not always provide complete matching details.The examples include descriptions or prices marked “nan.”Source: §10 · pp. 20–21
  • Some paired listings differ in product names and model identifiers.Examples include different Canon printer models and Panasonic phone model numbers.Source: §10 · pp. 20–21

11. IMDB

The passages show varied benchmark examples: movie reviews, legal text, math questions, and multiple-choice questions spanning different subjects.

11. IMDB

  • The IMDB examples include strongly negative, enthusiastic, and qualified movie reviews.The negative review calls the movie terrible, while another praises it and a third questions whether it is scary.Source: §11 · p. 21

12. LegalSupport

  • The LegalSupport example is a legal passage about a statute’s constitutionality and an Equal Protection Clause argument.It describes the government’s position and the district court’s agreement with that position.Source: §12 · p. 22

13. MATH

  • The MATH examples include an equation-solving prompt and prompts asking for decimal values.One prompt asks for a root-derived value; another asks for a value in the decimal system.Source: §13 · p. 22

14. MMLU

  • The MMLU examples ask about a temperature-rate equation, a system’s frequency response, and group theory.The prompts use multiple-choice answer options.Source: §14 · p. 22

15. RAFT (Real-world Annotated Few-Shot)

The examples span few-shot inputs, synthetic reasoning tasks, TruthfulQA prompts, and factual questions about works and authors.

15. RAFT (Real-world Annotated Few-Shot)

  • RAFT examples include account terms, a tweet, and a bibliographic record with abstract and body text.The account-terms passage includes responsibilities related to account information and password confidentiality.Source: §15 · pp. 22–23

16. Synthetic reasoning (abstract symbols)

  • Abstract-symbol examples provide rules and substitutions for tasks involving symbolic results, composition, and translation.The examples use symbols such as Y, Z, and operators, with substitutions including animal names.Source: §16 · p. 23

17. Synthetic reasoning (natural language)

  • Natural-language reasoning examples give rule sets and facts, then ask questions about inferred properties.The examples concern a poppy, a tree, and a bear.Source: §17 · p. 23

18. TruthfulQA

  • TruthfulQA examples include a medical question and generated responses about ancient Egyptians and countries’ average incomes.The latter responses include repeated, sometimes conflicting claims about rankings.Source: §18 · pp. 23–24

19. WikiFact

  • WikiFact examples ask about an author and provide repeated or detailed facts about works, including publication and bibliographic information.The passages name Elizabeth Bowen and Hannah Arendt.Source: §19 · pp. 24–25
Loading 2503.13335v1…