Source-linked AI summary
An Audit of Measurement Quality and Answer Bias in a Large Classroom-Poll Corpus
Rohit Sharma, Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade, Sakshi Sharma, Meenakshi V, SRS Iyengar
TL;DR
Classroom polls are often treated as measures of learning even though their measurement properties are rarely evaluated. This paper audits transcript-keyed classroom polls with item response theory and analyzes True/False response patterns. The polls form an easy, moderately precise scale, while answer direction is associated with difficulty and the checks rely on signals already recorded by polling systems.
Problem
Classroom polls are often treated as evidence of learning, but their item-level measurement quality and systematic answer bias are rarely assessed.
Source: §1 · pp. 1–2Method
The authors audit authentic polls using transcript-verified keys, item response theory, and analyses of True/False response structure.
Source: §1 · p. 2Results
False-keyed items had a mean correct-share of 0.640 against 0.769 for True-keyed items, and answer direction remained associated with difficulty after content and exposure controls.
Source: §5.1 · p. 7, §6 · p. 11Takeaways & Limitations
Discrimination, measurement precision, and key-direction balance can be checked from polling logs as part of item development before questions reach students.
Source: §5.2 · p. 11The analysis excludes items whose answers could not be settled from transcripts, and the authors confine ability estimates to group-level use.
Source: §5.2 · p. 11, §6 · p. 12
Abstract
from arXiv · showhide
Real-time classroom polls are widely used and increasingly generated with automated assistance, yet the questions themselves are rarely evaluated as measurements. We audit a large corpus of authentic classroom polls, 604 items across 47 sessions answered 340,668 times by 2,807 learners, as a measurement instrument. For the 539 items whose correct answer could be established and verified from the lecture transcript, we place every item and every student on a common scale using item response theory and analyse the answer structure of the True/False items. Two findings emerge. First, the polls form a coherent but easy scale of moderate precision (marginal reliability about 0.60), on which roughly a quarter of items barely separate stronger from weaker students. Second, students show a robust tendency to answer True, present at the individual level (77% of students lean True), which meets a milder tendency for items to be keyed False; as a result answer direction predicts difficulty, False-keyed items being about thirteen points harder, and the effect survives controls for item content and for selective answering. Both findings rest on signals a polling system already records, so the same checks can be run as items are generated, before they reach students.
1 Introduction
Classroom polls support engagement and retrieval while also being treated as evidence of learning, although their measurement properties are rarely considered. This audit evaluates authentic poll items as a measurement instrument and examines their quality and answer structure.
1 Introduction
- Polls serve both as pedagogical interventions and as potential measures of student learning, despite limited attention to their properties as assessment items.The introduction notes that retrieval practice can strengthen memory and embedded questions can help sustain attention.Source: §1 · p. 1
- Purpose-designed assessment items are expected to have known difficulty, distinguish levels of mastery, and use plausible, non-misleading distractors, but little is known about whether classroom polls meet these standards.The measurement gap matters as automated tools increasingly generate questions from lecture material [9], and such tools can produce fluent but incorrect output [11].Source: §1 · p. 2
- The study audits 604 authentic polls from 47 sessions, using transcript-verified keys for the 539 items whose correct answers could be established.The analysis separates item properties from student performance and examines whether response direction relates to item difficulty.Source: §1 · p. 2
2 Background Study
Prior work examines the benefits of polling, psychometric measurement, and response bias, but leaves the measurement quality of live classroom-poll questions largely unexamined. The paper connects these strands to audit polls as both measurement items and sources of answer-bias evidence.
2 Background Study
- Research on classroom response systems reports generally favourable but modest and use-dependent gains, while leaving the soundness of individual poll questions as a separate, unaddressed question.Reviews report participation, attention, and timely feedback; a meta-analysis of over fifty studies describes gains as genuine but modest and contingent on use.Source: §2 · p. 2
- Item response theory places items and respondents on a common latent scale, separating difficulty from ability while estimating discrimination and measurement precision.These methods are routine for designed examinations but, to the authors’ knowledge, had not been applied to a large corpus of live, informal classroom polls.Source: §2 · p. 2
- Acquiescence can interact with True/False key direction to affect apparent difficulty, while generated questions are more often evaluated for fluency and answerability than measurement quality.The paper identifies respondent-level answer structure, measurement auditing of live polls, and recovery of these signals from existing logs as unexamined areas.Source: §2 · pp. 2–3
3 Methodology and Methods
The study asks whether live polls form a coherent measurement instrument, whether True/False items show answer bias associated with difficulty, and whether calibration withstands self-selection. It answers these questions with item-response models and transcript-derived answer keys.
3.1 Research Questions
- The research questions test measurement coherence and precision, True/False answer bias and its relation to difficulty, and calibration robustness to students’ self-selection of polls.The study derives the quantities used to answer these questions from polling-system response logs.Source: §3.1 · p. 3
3.2 Measurement Model
- The analysis compares Rasch, two-parameter, and three-parameter item-response models, estimates student ability with EAP, and selects a model using information criteria and likelihood-ratio tests.A synthetic recovery test checks that the procedure recovers known parameters; item discrimination and student-level claims incorporate uncertainty estimates.Source: §3.2 · p. 3
3.3 Answer Key
- Correct answers were established from lecture transcripts independently of poll votes, then independently rechecked for the 90 items where the class majority disagreed with the derived key.The second reading confirmed 62 keys and corrected 28; the modelled set comprises 539 of 604 polls, and answer-bias analyses are also recomputed on 353 high-confidence True/False items.Source: §3.3 · p. 4
4 Data Collection and Analysis
The study uses polls from an online internship-orientation programme and analyzes an uneven, verified response corpus. It groups items by instructional descriptors and examines measurement assumptions and answer bias, including possible content and exposure explanations.
4.1 Study Context and the Poll System
- The 47-session online orientation programme used short live polls, predominantly True/False, to introduce and motivate interns rather than assess mastery.Its context is consistent with the corpus’s emphasis on lower-order questions.Source: §4.1 · p. 4
4.2 Data Sources and the Verified Corpus
- The verified corpus contains 539 keyed items and 311,227 binary responses from 2,373 learners, with 24.3% matrix density and highly uneven participation.Students answered a median of 36 items; the most active fifth supplied 66% of responses. IRT treats unanswered cells as missing by design, an assumption examined in the results. The export was anonymized, with salted hashes used to track learners and aggregate reporting.Source: §4.2 · p. 4, §4.3 · p. 5
4.3 Item Descriptors and Data Protection
- Items receive lecture-inferred cognitive-level and instructional-function descriptors, used only as grouping variables and covariates rather than as a contribution of the paper.The seven-category instructional-function scheme is presented as a practical shorthand, not a definitive taxonomy; Table 2 shows two representative polls and response splits.Source: §4.3 · p. 5, Table 1 · p. 5, Table 2 · p. 5
4.4 Analysis
- The analysis checks dimensionality and local independence on binary data, evaluates convergent validity, reconstructs learner-level True/False choices, and tests whether direction effects persist after accounting for content and exposure.Content is addressed by controlling for instructional function in a student-clustered incorrectness model; exposure is assessed using response counts and stratification.Source: §4.4 · pp. 5–6
5 Results and Discussion
The polls form a coherent but easy scale with moderate precision and a substantial minority of weakly discriminating items. Students’ tendency to answer True meets a mild skew toward False keys, and False-keyed items remain harder after controls for content and exposure; the signals can support routine quality checks.
5.1 Results
- The 539 items formed a coherent, easy scale: mean difficulty was b = −1.59, overall correctness was 68%, and marginal reliability was approximately 0.60.The two-parameter model was selected over the one-parameter model; adding a guessing parameter yielded negligible gain not rewarded by information criteria.Source: §5.1 · p. 6, Figure 1 · p. 7
- Twenty-two percent of items (121) had weak discrimination (a < 0.30), and 65 remained below that threshold at the upper 95% confidence limit.Weakly discriminating items were concentrated among routine recall items.Source: §5.1 · p. 6, Figure 2 · p. 8
- The scale was approximately unidimensional, with little local dependence; higher-order Bloom items were harder than Knowledge items, while Concept Reinforcement and Integration were easiest.Only 8 of 539 items had infit above 1.3, and only 1.1% of item pairs had |Q3| > 0.2.Source: §5.1 · p. 6, Figure 3 · p. 8
- Among 1,575 students answering at least ten True/False items, mean individual True-rate was 59.5%, and more than three-quarters leaned True.The distribution was centred at 0.60; the pattern is consistent with acquiescence bias.Source: §5.1 · p. 7, Figure 4 · p. 9
- False-keyed items averaged 0.640 correct versus 0.769 for True-keyed items; their difficulty penalty persisted after controls for instructional function and exposure.The function-controlled odds of an incorrect answer were 1.79 times higher for False-keyed items, and the exposure-controlled penalty was +2.30 logits.Source: §5.1 · p. 7, Figure 5 · p. 9
- Selective answering appeared largely benign for the ability scale: participation was essentially uncorrelated with ability, and answered-item difficulty varied little across ability quintiles.Harder items were only weakly under-answered (ρ = −0.24); difficulty estimates from heavily and lightly participating groups correlated ρ = 0.70 over 143 items.Source: §5.1 · p. 10, Figure 6 · p. 10
5.2 Discussion
- The polls can indicate whether middle and lower portions of a class are keeping pace, but moderate precision and uneven discrimination limit individual ranking and higher-stakes inferences.The discussion also notes that True/False items are partly answerable without command of the material.Source: §5.2 · pp. 10–11
- Calibration separates item difficulty from respondent ability, quantifies uncertainty, and tests direction effects while holding content and exposure constant; existing polling logs can support these checks before deployment.The specific magnitudes come from one programme and should not be generalized beyond this setting; the methods and signal types may apply to other polling settings.Source: §5.2 · p. 11
6 Conclusion
The audit finds that classroom polls form a coherent but easy, moderately precise instrument, while True/False items show answer-direction bias that persists after content and exposure controls. The authors propose log-based checks for item quality and key imbalance, while noting limits in the evidence and the need for replication.
6 Conclusion
- Answer direction remains associated with difficulty after content and exposure are controlled: students’ True tendency meets a mild False-keying tendency.The calibration appears largely robust to selective answering; its conclusions are limited to items with transcript-verified answers.Source: §6 · pp. 11–12
- The polls form a coherent but easy, moderately precise scale, with roughly a quarter of items discriminating weakly.The authors confine ability estimates to group-level use.Source: §6 · pp. 11–12
- Response-log checks could complement human review by flagging weakly discriminating items or key-direction imbalance before items reach students.A richer response-style model could estimate acquiescence and ability jointly rather than sequentially.Source: §6 · pp. 11–12
- Replication across instructors, subjects, and assessment stakes is needed to establish which findings generalise beyond low-stakes orientation polling.The analysis excludes items whose answers could not be settled from the lecture transcript, and the key skew is mild overall.Source: §6 · p. 12