PaperlayerView exact revision on arXiv

Source-linked AI summary

How assigned AI use before class shapes active student engagement in class

Dan J. Wang, Neelam Modi Jain, Vanessa Burbano, Jorge Guzman, Daniel Keum, Soomi Kim, Bruce Kogut, Nataliya Wright

arXiv:2610.10463v1cs.HCcs.AI

TL;DR

Evidence about classroom AI’s effects is mixed, and less is known about its effects on students’ live learning behavior. In a field experiment, students prepared with a voice-based AI discussion partner; after two uses, they made about 31% more voluntary contributions in later classes.

  • Problem

    Evidence on classroom AI’s effects is mixed, and less is known about live learning behavior; class participation captures skill development and generalizes beyond class.

    Source: Introduction · pp. 2–3

  • Method

    A field experiment randomly assigned 759 students across ten course sections to prepare for some sessions with CAiSEY, a voice-based AI discussion partner.

    Source: Discussion · p. 9

  • Results

    After two CAiSEY sessions, students made about 31% more in-class voluntary contributions in later sessions.

    Source: Class Participation · p. 7, Discussion · p. 9

  • Takeaways & Limitations

    Repeated voice-based AI practice may support class discussion participation, a learning behavior connected to skill development and real-world achievement.

    Source: Introduction · pp. 2–3, Discussion · p. 9

    Interpreting IV estimates as local average treatment effects depends on assumptions; short conversations, reminders, and other preparation changes may challenge the exclusion restriction.

    Source: Methods · p. 12

Abstract

from arXiv · show

AI learning tools are rapidly entering classrooms, but evidence about whether they help students learn is mixed and rests mostly on test scores. Comparatively less research addresses whether the use of AI changes students' live learning behaviors in class. Here, we report the results of a preregistered field experiment with 759 MBA students enrolled in ten sections of a course, in which each student was randomly assigned two of ten class sessions to prepare for with a purpose-built voice-based AI discussion partner. After two uses of the AI discussion partner, students made about 31% more voluntary contributions in each later class session. Students who used the AI discussion partner more also reported greater comfort speaking up and greater perceived learning, but not greater focus or motivation. These findings suggest that repeated practice with a voice-based AI partner can meaningfully increase students' engagement in class discussion, enhancing a critical intermediate learning outcome.

Introduction

This randomized field experiment examines how preparing with a purpose-built voice AI discussion partner relates to students’ voluntary classroom contributions and reported learning experiences. It focuses on participation, comfort speaking up, and perceived learning.

Class Participation as a Learning Outcome

  • Class participation can support learning through feedback and explanation while also developing skills relevant beyond the classroom.The introduction identifies participation as reflecting self-efficacy, critical thinking, mastery, motivation, and engagement, as well as a skill that generalizes to other settings.Source: Introduction · p. 3, Class Participation as a Learning… · p. 3

AI Tutors and Communication Skills as Learning Outcomes

Main Hypotheses

  • The proposed mechanisms are greater subject mastery and lower anxiety about speaking, which could reduce fear of negative evaluation and increase participation.The study’s hypotheses also concern reported comfort, focus, motivation, and perceived learning as CAiSEY use increases.Source: Main Hypotheses · pp. 5–6

Assignment and Use of CAiSEY

  • 759 first-year MBA students were randomly assigned CAiSEY preparation before two of ten case discussions, with instructors and teaching assistants blinded to assignment.Among the 504 students whose assigned sessions were observed, 92.2%–98.9% submitted at least one conversation in each session, and 38.1%–67.0% held a conversation of at least ten minutes.Source: Assignment and Use of CAiSEY · p. 6

Class Participation

  • 31% more voluntary contributions per later class session followed a second CAiSEY use; participation did not increase in the prepared session, and the benefit appeared in later sessions.The second-use effect was positive in the next session and across all future sessions; results were robust to qualifying-use thresholds as short as four minutes.Source: Introduction · p. 2, Class Participation · p. 7
  • A second assigned opportunity was associated with additional contributions in later sessions, with predicted differences positive in every later session and cumulative gains across sessions 5–12.The reported cumulative differences were Δ = 0.652 for assignment and Δ = 1.333 for qualifying use; the predicted schedule comparison is model-based, not an observed group.Source: Class Participation · p. 7

Self-Reported Learning Experiences

  • Comfort rose from 5.56 at midterm to 5.67 at term end, and perceived learning rose from 5.82 to 5.97; focus and motivation changed little.The reported ratings describe the 627 students who completed both surveys; the passage separately describes a model relating rating changes to assignment timing.Source: Self-Reported Learning Experiences · p. 8
  • Later qualifying use was associated with greater end-of-term comfort (λ! = 0.131, SE = 0.063, p < 0.05) and perceived learning (λ! = 0.124, SE = 0.055, p < 0.05), but not focus or motivation.The positive associations for comfort and perceived learning remained significant across five-to-ten-minute qualifying-use thresholds; focus and motivation estimates remained near zero.Source: Self-Reported Learning Experiences · p. 8

Discussion

  • After two CAiSEY sessions, students made 31% more voluntary contributions per class session than the sample mean of 0.70 contributions per session.Random assignment to different session opportunities allowed the authors to interpret the comparison as causal for students who used CAiSEY versus those who had not yet used it.Source: Discussion · p. 9
  • The study reports a first causal test of pedagogically directed AI’s effect on participation, examining voice-based rehearsal rather than the text interfaces common in classroom AI studies.The authors suggest voice AI may prompt critical thinking and subsequent classroom discussion, but their findings are limited to a purpose-built tool tailored to anticipated class topics; they did not compare it with a human partner or never-users.Source: Discussion · pp. 9–10

Methods

  • The experiment randomized 759 MBA students across ten course sections to two assigned pre-class CAiSEY sessions, with teaching assistants recording contributions and cold calls.Students defended a chosen position in a spoken debate with an AI opponent, then submitted a reflection and received generated feedback; qualifying use required a conversation of at least ten minutes.Source: Methods · pp. 10–11
  • Participation models used student and section-by-session fixed effects and controlled for prior observed-session contributions; assignment instrumented qualifying use in the IV models.The authors note that interpreting IV estimates requires assumptions including an exclusion restriction, which could be challenged by short conversations, reminders, or changes in other preparation.Source: Methods · p. 12

Panel B. Sample Construction

The panel was narrowed from 759 enrolled students to 604 students with 6,003 student-sessions in the retained analytic panel. Separate participation samples required recorded assignments or prior participation.

Panel B. Sample Construction

  • 604 students and 6,003 student-sessions remained in the analytic panel; the opportunity-specific sample included 4,787 student-sessions.The pooled Same Session ITT sample included 596 students and 5,395 student-sessions.Source: Panel B. Sample Construction · p. 27
  • Attendance screening removed 716 student-sessions, and two sections lacking separate cold-call records excluded 152 students.The retained panel covered eight sections; one section's sessions with attendance coded absent were retained only when a contribution was recorded.Source: Panel B. Sample Construction · p. 27

Sample

The section describes sample definitions and regression-table conventions, including qualifying use, model samples, and fixed effects.

Sample

  • Qualifying use is a recorded conversation of at least ten minutes; the survey full panel comprises 627 students returning both surveys, while the survey-participation panel includes 503 of them.The participation panel used 4,787 student-session rows with non-missing conversation duration.Source: Sample · p. 28
  • Models 2–7 use the 504-student opportunity-specific sample, while Model 1 uses the pooled Same Session sample; all models include student and section-by-session fixed effects and prior voluntary contributions.Standard errors are clustered by student.Source: Sample · p. 29

Dependent variable

  • The table notes identify β1 and β2 as coefficients for the first and second assigned opportunities, with β1 + β2 reporting their sum in Models 6 and 7.Table S5 Panel A is captioned as reporting the first-stage effect of assignment on qualifying use.Source: Sample · p. 29, Table S5 · p. 30

Panel B. Next session

The excerpt lists negative and positive next-session estimates and indicates fixed effects in the reported columns.

Panel B. Next session

  • Seven negative estimates are listed, ranging from -0.078* to -0.151*.The excerpt does not identify the row label for this sequence.Source: Dependent variable · p. 31
  • Seven positive estimates are listed, ranging from 0.099* to 0.183*.The excerpt does not identify the row label for this sequence.Source: Dependent variable · p. 31
  • Student fixed effects and section-by-session fixed effects are marked Yes across the listed columns.Source: Dependent variable · p. 31

Panel C. All future sessions

This section defines the cumulative participation contrast and related measures, including duration-based qualifying-use thresholds and survey outcomes. The supplied passages describe the comparisons and specifications but do not provide their numerical estimates.

Dependent variable

  • Qualifying use is redefined across duration thresholds; models include student and section-by-session fixed effects, prior participation, and 4,787 sessions from 504 students.The table notes distinguish first-stage assignment effects from second-opportunity coefficients, which are incremental in the all-future-sessions model.Source: Dependent variable · p. 32
  • Δ compares assignment in sessions 3 and 5 with a hypothetical session-3-only schedule, summed over sessions 5 through 12.The one-opportunity schedule is a model counterfactual because every student received two assignments.Source: Dependent variable · p. 33, Table S6 · p. 33
  • Percentage differences divide the schedule contrast by predicted participation under the one-opportunity schedule in that session; session 2 is a shared zero-difference baseline.The specification omits the prior-participation control used in the main models.Source: Dependent variable · pp. 33–34

Panel D. Learning of material

Panel E. Four-item index

  • The four-item index is the unweighted mean of available item responses; end-of-term threshold estimates for the index are reported separately from Figure 7.The index averages answered items, and the cited threshold note reports it only in the supplementary table.Source: Dependent variable · pp. 35, 37, Panel E. Four-item index · p. 40
Loading 2610.10463v1…