Source-linked AI summary
How To Grade a Test Without Knowing the Answers --- A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing
Yoram Bachrach, Thore Graepel, Tom Minka, John Guiver
TL;DR
Crowdsourcing and aptitude testing need ways to infer answers, question difficulty, and participant ability when correct answers may be unavailable. The paper introduces DARE, a joint probabilistic graphical model with entropy-based adaptive testing, and reports accurate joint inference plus more efficient testing. Its evaluation is limited by reliance on an IQ dataset and an assumption that participants answer to the best of their ability.
Problem
Correct answers are often unavailable in crowdsourcing, making it difficult to assess participant ability, question difficulty, and item correctness from responses alone.
Method
DARE jointly models answers, participant abilities, question difficulties, and responses, then selects adaptive questions by expected entropy reduction.
Results
Joint inference achieved high accuracy, and active learning enabled more efficient testing than static testing in the evaluated setting.
Takeaways & Limitations
The framework can grade tests without known answers while aggregating responses according to participant abilities and question properties.
Takeaways & Limitations
Evaluation used an IQ dataset, and the model assumes participants answer to the best of their ability.
Abstract
from arXiv · showhide
We propose a new probabilistic graphical model that jointly models the difficulties of questions, the abilities of participants and the correct answers to questions in aptitude testing and crowdsourcing settings. We devise an active learning/adaptive testing scheme based on a greedy minimization of expected model entropy, which allows a more efficient resource allocation by dynamically choosing the next question to be asked based on the previous responses. We present experimental results that confirm the ability of our model to infer the required parameters and demonstrate that the adaptive testing scheme requires fewer questions to obtain the same accuracy as a static test scenario.
1. Introduction
The paper addresses how to infer correct answers, question difficulty, and participant ability when ground truth is unavailable. It proposes a graphical-model framework and adaptive testing approach for more efficient information gathering.
- Crowdsourcing aggregates many participants’ responses, but determining how to combine them into correct classifications remains an open problem.
- Without known answers, response data must support simultaneous assessment of question difficulty, participant competence, and item correctness.
- Gold-set questions can evaluate participant skill, but the paper targets settings where correct answers are not known in advance.
- The proposed framework jointly infers answers, question difficulties, and participant abilities from responses across multiple problem domains.
- The model also selects subsequent questions adaptively according to previous responses and calibrates tests using relative difficulty and discrimination.
2. Related Work
Related work spans collective intelligence, psychometric test theory, opinion aggregation, and graphical models for estimating skills or combining responses. The paper positions DARE as a general-domain model connecting these strands.
- Collective-intelligence research studies group performance, including aggregated IQ-test responses and explicit collaboration among group members.
- Test theory and item-response theory analyze participant abilities and question difficulties to improve the reliability of psychological tests.
- Opinion-aggregation research includes semantic-web methods, prediction markets, voting, and models addressing strategic manipulation of preferences.
- Prior crowdsourcing methods variously omit task difficulty, model it with expectation maximization, or represent participant expertise through question features.
- DARE instead focuses on a general domain without specific question features and combines response aggregation with adaptive IQ testing and skill estimation.
3. Joint Probabilistic Model of Difficulty, Ability, and Response
DARE is a probabilistic graphical model for jointly representing latent answers, participant abilities, question difficulties, and observed responses. Approximate inference supports missing or partial ground truth, while posterior uncertainty guides adaptive question selection.
- 3. Joint Probabilistic Model of Difficulty, Ability, and Response: DARE models correct answers, participant abilities, question difficulties, and observed responses as variables in a probabilistic graphical model.
- 3. Joint Probabilistic Model of Difficulty, Ability, and Response: Each participant has an ability, each question has difficulty and discrimination, and correctness depends on the ability–difficulty difference.
- 3. Joint Probabilistic Model of Difficulty, Ability, and Response: If a participant knows the answer, the response matches the correct answer; otherwise, it is sampled uniformly from the available choices.
- 3.2. Probabilistic Inference: The model accepts participant-question-response triples with optional ground-truth answers, covering traditional testing and crowdsourcing cases.
- 3.2. Probabilistic Inference: Approximate message passing with expectation propagation estimates posterior distributions for answers, abilities, difficulties, correctness, responses, and discrimination.
- 3.3. Active Learning and Adaptive Testing: Adaptive testing chooses responses expected to reduce posterior entropy, with posterior variance providing the practical selection criterion.
4. Empirical Analysis
The empirical analysis evaluates DARE on intelligence-test and crowdsourcing data, including inference with missing answers, partial information, and adaptive testing. Across these settings, DARE estimates answers and participant abilities, outperforms simpler models, and achieves lower adaptive-testing error than static selection.
- 4. Empirical Analysis: DARE was evaluated on Raven’s Standard Progressive Matrices, a 60-question multiple-choice intelligence test with eight possible answers per question.The dataset included 120 individuals; the mean raw score was 99.57 (STD=14.16).
- 4.1. Unobserved Correct Answers: 46 questions’ correct answers were inferred correctly when all 120 participants’ responses were provided.The model selects the mode of the inferred answer distribution; difficult items limited performance because few participants answered them correctly.
- 4.1. Unobserved Correct Answers: Model raw scores strongly correlated with true raw IQ scores, with R2 = 0.7243 and relatively small differences across participants.Model raw scores use predicted correct answers when the true answers are unavailable.
- 4.1. Unobserved Correct Answers: DARE outperformed simpler question-only and participant-only models as crowd size increased, while participant-only modeling exceeded majority voting.Performance was measured by the number of inferred answers matching the true correct answers across 10,000 sampled crowds.
- 4. Empirical Analysis: On the TREC 2011 Crowdsourcing Track dataset, DARE correctly classified 210 questions versus 206 for majority voting.The analysis used 369 questions and 84 workers; the skill-estimation scatter plot had r2 of 0.79.
- 4.2. Partial Information on Correct Answers: A larger gold-set improved DARE’s accuracy on the remaining questions by providing information that better estimates participant abilities.The experiment sampled crowds of 20 participants and varied the number of revealed questions.
- 4.3. Adaptive Skill Testing: Adaptive testing produced smaller inferred-ability RMSE than static question sets for every tested question budget.The adaptive scheme selected each next question based on earlier responses, whereas the static baseline used one fixed set per budget.
5. Conclusions and Limitations
The DARE model jointly infers participant abilities, question difficulties, and correct answers, while adaptive testing improves efficiency. The evaluation supports accurate joint inference but notes dataset and behavioral assumptions as limitations.
- The adaptive scheme had RMSE standard deviation 1.07, compared with 0.99 for the static scheme.
- Adaptive testing produced smaller inferred-ability error than static testing across different question budgets.Figure 6 compares RMSEs for static and adaptive skill testing.
- DARE jointly infers participant ability levels, question difficulty levels, and correct answers when answers are unknown.
- The evaluation used an IQ dataset, while crowdsourcing tasks may differ in difficulty homogeneity, and participants may have varying motives rather than answering to their best ability.
- Future work includes better aggregation models, tractable optimal non-adaptive tests, and inferring individual ability from group-context performance.