Source-linked AI summary
Insights on Student Learning from Live Classroom Polls: More Than Right or Wrong
Rohit Sharma, Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade, Sakshi Sharma, Meenakshi V, SRS Iyengar
TL;DR
Live classroom polls are usually read through participation and correctness, leaving key-free response order and class division unexamined at real-course scale. The paper analyzes these signals across a live deployment and finds that response order is stable while disagreement maps divided questions and flags many collective errors. The two readings are largely independent, with early responders anticipating the majority only when the class is already near agreement.
Problem
At real-course scale, it remains unclear whether key-free learner response order and answer distributions are stable, informative, and related.
Source: §I · p. 1, B. Discussion · p. 6Method
The study analyzes response order and key-free answer-distribution measures in live polls, then applies author-designated answers to interpret settled items.
Source: B. The Response-Order Construct · p. 2, C. Key-Free Disagreement Measures · p. 3, A. Results · p. 5Results
Corrected split-half reliability for response order is 0.92, and the majority is wrong on 49 of 358 settled-answer items, most of which are flagged as contested.
Source: A. Results · pp. 4–5Takeaways & Limitations
Response order and class disagreement provide largely independent descriptions of learners and questions, and early responders anticipate the majority only when the class is already moving toward agreement.
Source: B. Discussion · pp. 6–7Key-free contestedness identifies division but cannot distinguish a collectively wrong majority from a class split on a hard question.
Source: A. Results · p. 5
Abstract
from arXiv · showhide
A live classroom poll is usually read through its answer key, as a count of participation and correctness. Yet the same record holds more before any key is fixed, namely who answers early and how the class divides across the options. Whether these key-free signals are stable, and what they reveal, has not been examined at the scale of a real course. We study them in a live deployment of 340,668 submissions across 603 polls in 47 sessions from roughly 2,400 learners, reading each poll for a learner's response order and for how much the class divides. A single further step brings in the author-designated key, on items with a settled answer, to interpret the divided ones. A learner's response order is a stable individual signature. It reproduces at a corrected split-half reliability of 0.92, holds at 0.87 within single sessions, and is unrelated to whether the learner is correct. Read without a key, about one poll in five does not reach a clear majority, and division marks the questions that split the class. On the settled-answer items the majority is itself wrong on 49 of 358. The key-free measure flags most of these, though division alone does not separate a collectively wrong answer from a merely hard one. The two readings are largely independent, meeting at one point where early responders anticipate the eventual majority only on questions the class has nearly settled. Together they map the learners and questions of a session without a key.
I. INTRODUCTION
Live polls are commonly reduced to participation and correctness, leaving response order and the class’s agreement or division unread. The paper asks whether these key-free signals are stable and informative at the scale of a real course.
I. INTRODUCTION
- Poll tallies of participation and correctness omit who answers early or late and whether the class agrees or splits.A class that agrees strongly and one that divides evenly can have the same participation count.Source: §I · p. 1
- Whether response-order and disagreement patterns are stable signals, rather than transient noise, remains an empirical question.The paper frames stability as necessary for using these patterns to describe a live class while the session is underway.Source: §I · p. 1
- At real-course scale, the paper asks what polls reveal before a key is applied, whether those signals are stable, and whether the learner and question readings carry the same information.Source: §I · p. 1
II. BACKGROUND STUDY
Prior work commonly reads polls through a settled key or uses disagreement as a discussion cue, while response latency and answer-distribution measures have been studied in other settings. The paper identifies an unexamined key-free response record and three open questions about its stability, information, and scale.
II. BACKGROUND STUDY
- Classroom response-system research typically treats polls as assessment moments, reading participation and correctness or using a split as a discussion cue [Caldwell, 2007; Kay & LeSage, 2009; Hunsu et al., 2016; Mazur, 1997].Peer-instruction research reports that sharp divisions yield larger discussion gains and that discussion drives improvement.Source: §II · pp. 1–2
- Computer-based testing research models response latency alongside accuracy, while learning analytics assesses reliability of learner-level indicators [van der Linden, 2007; De Boeck & Jeon, 2019; Shrout & Fleiss, 1979].Source: §II · p. 2
- Classical item indices presuppose a key, whereas distractor analysis and distribution summaries offer key-free views of answer choices.Across these traditions, the stability of live response order, key-free answer distributions at scale, and the relation between the two remain unestablished.Source: §II · p. 2
- Research on crowd judgments and social influence addresses aggregation accuracy and the effects of exposure and public response order.Source: §II · p. 2
III. METHODOLOGY AND METHODS
The study asks what a live poll reveals about learners and questions before its key is applied. It separates the two key-free readings from their relationship and treats reported associations as descriptive.
A. Research Questions
- The main research question concerns what live polls reveal about learners and questions before an answer key is applied.Source: A. Research Questions · p. 2
- RQ1 asks what learner ranking by response earliness implies, while RQ2 asks what answer spread reveals about a poll without its key.Source: A. Research Questions · p. 2
- RQ3 examines the relationship between learner and question readings; the constructs and associations are descriptive, from one observational deployment, with no causal claim.Source: A. Research Questions · p. 2
B. The Response-Order Construct
The methods define a learner-level relative response-order score and key-free measures of answer distribution. They use within-poll rankings and classify item division without treating disagreement as correctness or difficulty.
B. The Response-Order Construct
- A learner’s response-order score averages their within-poll response-time percentiles, with lower scores indicating earlier responses and higher scores later ones.Tied submissions receive average ranks; the percentile is computed within each poll, making the measure invariant to the unknown launch instant rather than a reaction time.Source: B. The Response-Order Construct · p. 2
C. Key-Free Disagreement Measures
- Key-free item measures include majority-option share, second-option share, normalized entropy, and the number of options; contested items have a second-option share of at least 30 percent.Three-group clustering orders most two-option items by split size; review of the 20 most divided items is an author judgment, not a validated instrument.Source: C. Key-Free Disagreement Measures · p. 3
D. Crossing the Two Axes
The study crosses learner response timing with key-free disagreement across cognitive polls, using a live online programme’s polling data. It asks whether divided items fill more slowly and whether early responders align with the eventual majority differently across disagreement levels.
D. Crossing the Two Axes
- The crossing matches each cognitive poll across the two analyses by session and within-session order, then relates response timing to disagreement.It tests fill speed, early-versus-late majority alignment, competing-block timing, and learner-level majority selection; associations are reported descriptively.Source: D. Crossing the Two Axes · p. 3
A. Study Context and the Poll System
- The live deployment ran over several weeks in an online summer internship programme oriented toward acclimatising and motivating incoming interns, not assessing mastery.Frequent live polls covered mood, logistics, icebreakers, and content; most content items were short two-option questions.Source: A. Study Context and the Poll System · p. 3
B. Data Sources and the Analysis Corpora
- The person and item analyses use different poll subsets: all polls for response order, and 505 cognitive fixed-choice polls for disagreement.The corpus spans 47 sessions, 603 polls, and 340,668 submissions from 2,422 learner identities; the two analyses’ response counts agree exactly on 498 polls and within two on all but one.Source: B. Data Sources and the Analysis Corpora · p. 3
- Response order uses within-poll ranks because the export lacks poll launch times, so the unknown launch instant cancels.The item axis excludes mood and icebreaker polls; the person axis uses all polls, with stated exclusions for reliability estimation.Source: B. Data Sources and the Analysis Corpora · p. 3
C. Descriptors and Data Protection
The analysis uses cognitive level and an author-designated answer as item descriptors, and protects learner identities by hashing identifiers before computation.
C. Descriptors and Data Protection
- Cognitive level, inferred from the surrounding lecture, and the author-designated correct answer are grouping descriptors; the answer is used only for items with a settled key.A voluntary five-point end-of-programme survey joined by hashed identifier supplies items used only to interpret the response-order signature.Source: C. Descriptors and Data Protection · p. 3
- Before computation beyond parsing, learner identities are hashed with a local salt, and results aggregate over anonymous keys.The authors state that no individual can be identified; example items are paraphrased and research consent was obtained before data collection.Source: C. Descriptors and Data Protection · p. 4
D. Analysis
The analysis tests response-order stability in multiple ways, measures disagreement without a key, and introduces the author-designated answer only for a separate check of collective errors.
D. Analysis
- Response-order stability is assessed with corrected split-half correlations, intraclass correlations, and a within-session split-half check.The split-half analysis averages 200 random partitions and reports Spearman and Pearson results; the within-session check holds day and shared clock fixed.Source: D. Analysis · p. 4
- Disagreement measures and clusters are computed from item response distributions, while author-designated keys are used only for 358 items with settled answers.The key-referenced step records whether the majority is correct; it interprets contested items but does not define disagreement.Source: D. Analysis · p. 4
- The crossing uses rank correlations, a poll-size-controlled analysis, a Kruskal-Wallis cluster test, and a per-poll early-minus-late comparison.The supplement contains tables on corpus composition, reliability thresholds, cluster centroids, and collective errors by disagreement band.Source: D. Analysis · p. 4
A. Results
Across live polls, response order is reliable when averaged over many polls, while response distributions reveal frequent disagreement without requiring an answer key. The two readings are largely independent, with early responders aligning more with the majority mainly on less-divided items.
A. Results
- 0.92 is the Spearman-Brown-corrected split-half correlation across learners; within-session split-half reliability is 0.87.The single-measure intraclass correlation is 0.21, compared with 0.98 for the average measure over many polls.Source: A. Results · p. 4, Figure 2 · p. 5
- Among learners with at least ten polls, the response-order score averages 0.53, with tenth and ninetieth percentiles of 0.35 and 0.73.The fastest tenth averages a within-poll percentile of 0.29 and the slowest tenth 0.78; response order is essentially uncorrelated with accuracy (0.02).Source: A. Results · p. 4, Figure 1 · p. 4
- 110 of 505 cognitive fixed-choice items (21.8 percent) have a majority share below 0.60, and 105 of 476 true-or-false items are nearly even splits.The nearly even splits have a minority share of at least 40 percent.Source: A. Results · pp. 4–5, Figure 3 · p. 5
- The three disagreement clusters comprise 101 settled, 206 moderately divided, and 198 strongly divided items.The strongly divided cluster has a centroid majority share of 0.591, second-option share of 0.407, and entropy of 0.964.Source: A. Results · p. 5, Figure 4 · p. 6
- Hand review of the twenty most-divided items found 12 cases of genuine conceptual difficulty, six of ambiguous wording, and two opinion prompts mislabeled as cognitive.The two opinion prompts’ near-even split reflects divided opinion rather than a knowledge gap.Source: A. Results · p. 5
- The majority is wrong on 49 of 358 settled-answer items (13.7 percent); key-free contestedness includes 41 of those 49 errors.Eight collective errors are not contested, including cases where up to 93 percent agreed on the incorrect answer.Source: A. Results · p. 5
- The disagreement cluster does not predict fill time: all three clusters reach 90 percent of responses at a median of 9 seconds.The Kruskal-Wallis test finds no difference (p = 0.47), and response-order score is unrelated to majority-choosing rate on settled and divided items.Source: A. Results · pp. 5–6
- The early-minus-late majority-alignment gap is 6.7 points on settled items, 6.5 on moderately divided items, and 2.6 on strongly divided items.The gap declines as disagreement rises (rank correlation −0.17, p = 1.5 × 10−4).Source: A. Results · p. 6
B. Discussion
A live poll's response record offers two key-free readings: learner response order and cohort disagreement. These readings are largely independent, meeting where early responders align more with the eventual majority on settled questions.
B. Discussion
- Early responders match the eventual majority more often chiefly when the class is already moving toward agreement.The alignment gap is large on settled items and fades as disagreement grows.Source: Figure 6 · p. 6, B. Discussion · p. 7
- Relative response order is a stable learner habit, but it is unrelated to correctness and should not be treated as ability or attentiveness.The authors caution that speed is not a sign of being right.Source: B. Discussion · pp. 6–7
- Reading disagreement without a key shows where the class divides; applying the key later distinguishes shared mistakes from questions that are simply hard.Division alone does not identify which explanation applies.Source: B. Discussion · p. 6
- The readings are largely independent: disagreement does not slow polls, and response order does not predict which option wins.They meet where early responders align more with the majority on settled items; that lead fades as the class divides.Source: B. Discussion · pp. 6–7, Figure 5 · p. 6, Figure 6 · p. 6
- Instructors can use both live readings because each describes something the other cannot: student participation patterns and questions that divide the room.The authors suggest acting on genuinely divided questions during the lesson.Source: B. Discussion · p. 7
VI. CONCLUSION
The conclusion presents response order and cohort disagreement as complementary, largely independent readings of live polls. Early responders align with the eventual majority on settled items, while future work should test broader settings and interventions.
VI. CONCLUSION
- The two key-free readings are largely independent, describing learners and items separately, and meet where early responders anticipate the majority on already-settled items.Early responders carry no such signal when the class is genuinely split.Source: §VI · p. 7
- On settled-answer items, the majority is wrong on 49 of 358, and the key-free disagreement signal flags most collective misconceptions.A divided class more often faces a hard question than a wrong answer.Source: §VI · p. 7
- Future work should add launch times, expand multiple-choice coverage, and test whether acting on either map changes cohort learning.These directions would extend the descriptive account toward reaction latency, a broader item map, and intervention.Source: §VI · p. 7