PaperlayerView exact revision on arXiv

Source-linked AI summary

Position: AI Evaluation Should Learn from How We Test Humans

Yan Zhuang, Qi Liu, Zachary A. Pardos, Patrick C. Kyllonen, Jiyun Zu, Zhenya Huang, Shijin Wang, Enhong Chen

arXiv:2306.10512v4cs.CL

TL;DR

Large static AI benchmarks are costly and can be distorted by contamination, low-quality items, and misleading aggregate scores. This position paper proposes psychometrics-based adaptive testing, which estimates item characteristics and dynamically selects evaluations to infer model abilities. The cited evidence includes sparse evaluations that preserve benchmark rankings or scores, while the paper notes that psychometric validity for AI still requires further research.

  • Problem

    Static benchmark scores can be costly and unreliable because aggregate results may reflect contaminated or flawed items rather than model capability.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper proposes estimating item characteristics and using interactive adaptive testing to select informative items while updating latent ability estimates.

    Source: §1 · p. 2, §3.2 · p. 7

  • Results

    50 selected items achieved 90% Kendall’s rank correlation with nearly 1,000-item full-test rankings, and as little as 3% of items could reconstruct scores across six benchmarks.

    Source: §3.2 · p. 7

  • Takeaways & Limitations

    Psychometrics offers a promising route to more efficient, interpretable, and reliable AI evaluation through ability estimation and informative item selection.

    Source: §1 · p. 2, §3.2 · p. 7, §6 · p. 9

    Further research is needed to determine whether psychometric principles fully apply to AI evaluation.

    Source: §5 · p. 8

Abstract

from arXiv · show

As AI systems continue to evolve, their rigorous evaluation becomes crucial for their development and deployment. Researchers have constructed various large-scale benchmarks to determine their capabilities, typically against a gold-standard test set and report metrics averaged across all items. However, this static evaluation paradigm increasingly shows its limitations, including high evaluation costs, data contamination, and the impact of low-quality or erroneous items on evaluation reliability and efficiency. In this Position, drawing from human psychometrics, we discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics or value of each test item in the benchmark, and tailoring each model's evaluation instead of relying on a fixed test set. This paradigm provides robust ability estimation, uncovering the latent traits underlying a model's observed scores. This position paper analyze the current possibilities, prospects, and reasons for adopting psychometrics in AI evaluation. We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.

1. Introduction

Large-scale static benchmarks are increasingly costly and unreliable as AI systems become more capable and complex. The paper proposes psychometrics-based adaptive testing to tailor evaluations, identify problematic items, and estimate latent model abilities.

  • Full-benchmark evaluation can exceed 4,000 GPU hours for one LLM, while generative-model latency and human involvement further raise economic, time, and computational costs.Source: §1 · pp. 1–2
  • Static benchmarks can obscure low-quality, erroneous, redundant, or contaminated items, undermining the reliability of aggregate scores as AI behavior becomes more complex.Source: Figure 1 · p. 1, §1 · p. 2
  • Psychometrics offers adaptive testing that models item characteristics, selects a minimal informative test for each model, and estimates latent traits underlying observed performance.Source: §1 · p. 2
  • The paper unifies prior work on performance estimation, item selection, and experimental interpretation within an adaptive-testing framework for reliable AI evaluation.Source: §1 · p. 2

2. Psychometrics Enables Scientific Evaluation

Psychometrics reframes AI evaluation from aggregate task scores to estimates of latent abilities and item value. It can improve interpretability, reduce unnecessary testing, and expose problematic or contaminated benchmark items.

2.1. Ability-Oriented Evaluation

  • Ability-oriented evaluation estimates latent traits rather than treating task scores as direct measures of capabilities such as medical ability.Source: §2.1 · pp. 2–3
  • Three-parameter IRT models response probability from latent ability and item difficulty, discrimination, and guessing parameters estimated from response data.Source: §2.1 · p. 3
  • Observed response variability can be separated from relatively stable latent traits, with Bayesian psychometrics providing ability distributions and uncertainty estimates.Source: §2.1 · p. 3
  • Adaptive testing reduces evaluation complexity by modeling dependencies among dimensions and selecting informative items near a model’s estimated ability boundary.Source: §2.1 · pp. 3–4, Figure 2 · p. 4
  • Psychometric ability estimates support statistical interpretation, cross-model comparison, multidimensional skill assessment, and alignment of results across benchmarks.Source: §2.1 · pp. 3–4

2.2. Not All Items Are Equally Important

  • Item characteristics reveal that difficulty, discrimination, and guessing differ across benchmark items, so equal-weight aggregate accuracy can misrepresent capability.Source: §2.2 · pp. 4–5, Figure 3 · p. 5
  • Negative discrimination can identify annotation errors, while psychometric quality metrics can detect redundancy, errors, and leakage in benchmark datasets.Source: §2.2 · p. 5
  • Response anomalies and guessing parameters can flag contamination, and adaptive testing can assign different item subsets to models to limit further contamination.Source: §2.2 · p. 6, Figure 4 · p. 6

3. Adaptive Testing Conceptualization for AI

The proposed adaptive-testing system first estimates benchmark item characteristics and then dynamically selects items while updating each model’s ability estimate. Fisher-information selection prioritizes items that most reduce uncertainty.

3. Adaptive Testing Conceptualization for AI

  • Adaptive evaluation has two phases: annotating item characteristics and conducting interactive dynamic testing to estimate model ability.Source: §3 · p. 6

3.1. Item Characteristics Annotation

  • Item difficulty and discrimination are estimated from model response data, recognizing that item characteristics can differ between AI systems and humans.Source: §3.1 · pp. 6–7

3.2. Interactive Dynamic Model Evaluation

  • During testing, the system updates ability estimates from prior responses and selects the next item according to a predefined criterion.Source: §3.2 · p. 7
  • Fisher Information selects the candidate item expected to provide the most information about the current ability estimate, favoring discriminative items near that estimate.Source: §3.2 · p. 7
  • 50 items selected from nearly 1,000 achieved 90% Kendall’s rank correlation with full-test rankings, while other work reconstructed scores using as little as 3% of items.Source: §3.2 · p. 7

4. Core Mechanisms Driving Adaptive Testing

Adaptive testing applies psychometric principles by treating AI evaluation as parameter estimation, exploiting benchmark interrelationships, and modeling regularities across AI systems. These mechanisms support more efficient, robust, and informative assessment.

A Parameter Estimation Problem

  • Psychometrics reframes AI evaluation as estimating latent ability from observed responses, progressively refining estimates as more items are answered.The approach is presented as a unifying framework for adaptive testing and related work on performance estimation, item selection, and result interpretation.Source: §4 · p. 7

Interconnectedness in Benchmarks

  • Adaptive testing uses benchmark interrelationships to identify inappropriate or redundant items, reducing unnecessary evaluations while focusing on items that reveal key model performance.Accounting for dependencies among tasks and items is described as improving robustness and providing deeper performance insights.Source: §4 · p. 7

Universal laws in AI systems

  • Psychometrics relies on regularities shared across AI systems, including common architectures, training methods, and potentially overlapping data, to model and predict performance.The paper cites performance prediction above 95% R2 across new model configurations as evidence of such uniformity.Source: §4 · pp. 7–8

Core mechanisms

  • Figure 5 organizes psychometric effectiveness around transformed problem formulation, benchmark interrelatedness, and universal laws in AI systems.These three mechanisms motivate the paper’s account of why adaptive testing can work for AI evaluation.Source: Figure 5 · p. 8, §4 · p. 8

5. Opportunities and Challenges

The paper presents adaptive testing as an emerging framework that can support richer AI measurement, including ability and non-ability traits. It also emphasizes that validation, adoption, and model applicability remain open challenges.

Opportunities

  • Adaptive testing can use IRT variants, cognitive diagnostic models, and neural psychometric models to estimate abilities and produce multidimensional performance reports.The appropriate measurement model should depend on the evaluation scenario.Source: §5 · p. 8

Evaluation Beyond Ability

  • Psychometric techniques can assess AI traits beyond ability, including ethics, bias, security, and robustness, using methods adapted from human cognitive assessments.Examples include Likert scales, MaxDiff, Implicit Association Tests, and Conjoint Analysis for studying decisions and biases.Source: §5 · p. 8

Alternative Views

  • The field remains early-stage because adaptive testing challenges established AI evaluation practices and its reliability and validity for AI require further research.The paper leaves open whether existing psychometric principles fully apply or whether a distinct field such as “Machine Psychometrics” is needed.Source: §5 · p. 8

Evaluation Beyond Ability

  • Psychometric models for non-ability traits are organized in Table 1 as possible techniques for evaluating AI models.The table summarizes model families and associated techniques rather than reporting a new empirical comparison.Source: Table 1 · p. 9

6. Conclusion

The paper argues that psychometrics could make AI evaluation more efficient and reliable by enabling adaptive testing with fewer, more valuable items. It presents the approach as promising while acknowledging open problems requiring community-wide effort.

Conclusion

  • Psychometrics offers adaptive AI testing that can use fewer items, identify more valuable items, and support reliable assessment despite flaws in large-scale benchmarks.The paper identifies low informativeness, contamination, low quality, and mislabeled items as obstacles in traditional evaluation.Source: §6 · p. 9
  • Adopting adaptive testing remains an open problem requiring collaborative efforts from the broader AI evaluation community.The conclusion characterizes current evidence as promising rather than definitive.Source: §6 · p. 9

Impact Statement

The paper applies psychometric principles developed for human assessment to AI evaluation, with potential benefits for efficiency, contamination mitigation, transparency, and accountability. It also warns that adaptation must avoid reinforcing bias or misrepresenting AI capabilities.

Impact Statement

  • Psychometric principles originally designed for human assessments are applied here to evaluate AI systems.The impact statement frames this application as the paper’s central focus.Source: Impact Statement · p. 9
  • Improved AI evaluation could reduce benchmarking inefficiencies, mitigate data contamination, and provide deeper insight into model performance.The stated potential impact includes greater reliability and fairness in AI systems.Source: Impact Statement · p. 9
  • In high-stakes domains such as healthcare, education, and legal decision-making, better evaluation could promote transparency and accountability in AI deployment.The statement presents these as potential ethical benefits, not guaranteed outcomes.Source: Impact Statement · p. 9
  • Adapting human-assessment methodologies requires care because misuse could reinforce biases or misrepresent AI capabilities.The impact statement identifies this as a societal risk accompanying the potential benefits.Source: Impact Statement · p. 9

A. Supplementary Clarifications and Illustrations

A.1. Key Aspects of Psychometric Analysis in AI Evaluation.

Psychometric analysis shifts AI evaluation from aggregate benchmark scores toward latent traits and item characteristics. It also contrasts unstable subset-based accuracy with ability estimation from limited, item-informed responses.

  • Psychometric analysis targets both the latent traits influencing model performance and the differing significance of benchmark items.Source: Figure 6 · p. 16
  • Traditional accuracy can be unstable on random item subsets, whereas psychometric methods infer ability from limited responses using item characteristics.For example, answering a 0.8-difficulty item incorrectly but a 0.6-difficulty item correctly places estimated ability between 0.6 and 0.8.Source: Figure 7 · p. 16

A.2. Further Explanation of How Psychometrics Mitigates the Curse of Dimensionality

Psychometrics mitigates evaluation inefficiency by replacing random subset selection with adaptive selection based on item characteristics and estimated model ability. This focuses testing on informative items while addressing uncertainty from probabilistic model judgments.

  • Random item subsets make traditional metrics unstable because they ignore item characteristics and cannot ensure subset performance matches the full dataset.Reducing the number of items therefore typically decreases evaluation precision.Source: §A.2 · p. 16
  • Adaptive psychometric testing narrows a model’s ability range by selecting items with difficulty levels inside the current estimate.After an incorrect answer to a 0.8-difficulty item and a correct answer to a 0.6-difficulty item, subsequent items can target the 0.6–0.8 range, analogous to binary search.Source: §A.2 · pp. 16–17
  • Temperature changes significantly affect ChatGPT’s response entropy, adding uncertainty to repeated judgments and complicating reliable evaluation.Higher entropy indicates greater variability in selected options.Source: Figure 8 · p. 17
  • By prioritizing informative items, psychometric methods reduce the number of responses needed without sacrificing precision and provide a scalable approach for multidimensional benchmarks.Source: §A.2 · p. 17

A.3. The Impact of Data Contamination on Item Characteristics

A controlled contamination experiment shows that item characteristics can reveal differences between contaminated and uncontaminated benchmark items. Across three benchmarks, contaminated items had significantly higher guessing factors.

  • Contaminated items had significantly higher guessing factors than non-contaminated items across MATH, NarrativeQA, and RAFT.The experiment deliberately included some items and answers in LLM test contexts, then estimated guessing factors with IRT.Source: §A.3 · p. 17

A.4. Illustrating Uncertainty in AI Evaluation

Self-regressive probabilistic models can produce inconsistent judgments when the same question is repeated. Temperature-dependent response entropy demonstrates how randomness affects evaluation reliability.

  • Temperature significantly changes ChatGPT’s response entropy, with higher entropy indicating greater variability in selected answers.Repeated prompts can yield inconsistent decisions in both content and reasoning.Source: §A.4 · p. 17

B. Case Study: A Simple Implementation of Adaptive Testing for AI Models

The case study implements adaptive testing in two phases: annotating benchmark-item characteristics, then interactively selecting informative items to estimate each model’s ability. Simulations and MATH experiments indicate that psychometric selection can reduce evaluation size while preserving ability estimates and improving ranking consistency.

Phase 1: Item Characteristics Annotation and Phase 2: Interactive Dynamic Model Evaluation

  • Adaptive testing first annotates item characteristics, then sequentially selects suitable benchmark items for interactive model evaluation.Item parameters can be estimated from response data using methods such as maximum likelihood or Bayesian estimation.Source: B. Case Study: A Simple Implementation… · pp. 17–19

Phase 1: Item Characteristics Annotation

  • Binary scoring uses correct/incorrect responses and a three-parameter IRT model with difficulty, discrimination, and guessing parameters.Polytomous scoring instead models graded outcomes, such as continuous machine-translation scores, using a graded response model.Source: B. Case Study: A Simple Implementation… · p. 18

Phase 2: Interactive Dynamic Model Evaluation

  • The system estimates current ability from prior responses and selects the next item by maximizing Fisher information, producing a final ability estimate after testing.Fisher information is presented as theoretically guaranteed and more interpretable than more complex selection algorithms.Source: B. Case Study: A Simple Implementation… · pp. 19–20

Simulation Experiment for Ability Estimation

  • Adaptive evaluation theoretically reaches the same estimation accuracy as random sampling with at most 20% of the benchmark items.The simulation measures mean squared error between estimated and artificially generated true abilities.Source: B. Case Study: A Simple Implementation… · p. 20

Comparison of Rankings with Full Dataset

  • Using about 60% of the MATH items, Fisher-based adaptive testing achieves the highest consistency with full-dataset model rankings.IRT ability estimates with random selection also correlate more strongly with full-dataset rankings than traditional accuracy, although results vary across runs.Source: B. Case Study: A Simple Implementation… · p. 20

Adaptability Analysis

  • Adaptive selection is most differentiated early in testing, while item sets for models—especially models from the same manufacturer—converge as more items are answered.The study measures this behavior with average Jaccard similarity across 20 LLMs as selected-item counts grow from 10% to 80% of MATH.Source: B. Case Study: A Simple Implementation… · p. 20, Figure 11 · p. 22

The Possibility of Data-Driven Evaluation Solutions

  • Large-scale response data from existing leaderboards could support automated systems that select items, estimate ability, and analyze anomalous behavior.The paper contrasts data-driven approaches such as meta-learning and reinforcement learning with psychometric methods, which retain theoretical robustness and interpretability under potential data bias.Source: B. Case Study: A Simple Implementation… · pp. 20–21

C. Analysis of Item Characteristics in Benchmarks

Item-characteristic analysis across GSM8K and MedQA shows that benchmark items vary substantially in difficulty, discrimination, and guessing factor. The examples illustrate how these properties distinguish straightforward, guessable items from items requiring complex reasoning and better separating model capabilities.

Overview

Implication

  • The contrasting examples support treating benchmark items as having different evaluation value rather than assuming every item contributes equally.The analysis explicitly concludes that different items hold varying levels of value in AI evaluation.Source: C. Analysis of Item Characteristics in… · pp. 23, 25, 26

GSM8K: Low Difficulty and High Guessing Factor

  • Low-difficulty GSM8K items require simple multiplication or subtraction and have high guessing factors, limiting their ability to distinguish proficiency.Examples include items with difficulty -0.609 and -0.862 and guessing factors 0.631 and 0.690.Source: C. Analysis of Item Characteristics in… · p. 23

GSM8K: High Difficulty and Low Guessing Factor

  • High-difficulty GSM8K items require multiple steps, conversions, and precise intermediate calculations, producing low guessing factors.The examples include difficulty 5.293 with guessing factor 0.022 and difficulty 5.045 with guessing factor 0.058.Source: C. Analysis of Item Characteristics in… · p. 24

MedQA: Low Discrimination and High Guessing Factor

  • Low-discrimination MedQA items can rely on predictable answers and familiar principles, allowing guessing without demonstrating nuanced medical understanding.The analysis identifies such questions as less effective at differentiating models, including an item with discrimination 0.899 and guessing factor 0.767.Source: C. Analysis of Item Characteristics in… · p. 25

MedQA: High Discrimination and Low Guessing Factor

  • High-discrimination MedQA items require integrating multiple clinical findings and applying medical principles, making correct answers harder to guess.Examples include items with discrimination 2.183 and 2.069, alongside detailed clinical scenarios and low guessing factors of 0.043 and 0.121.Source: C. Analysis of Item Characteristics in… · p. 26

Related papers

Loading 2306.10512v4…