PaperlayerView exact revision on arXiv

Source-linked AI summary

From Written Response to Dialogue with AI: How Activity Format, Interaction Modality, and Language Impact Student Learning and Engagement

Deepak Varuvel Dennison, Connie Zhang, Rene Kizilcec, Aditya Vashistha

arXiv:2610.07483v1cs.HCcs.CY

TL;DR

LLMs can produce written learning responses without students necessarily engaging with the content, while evidence remains limited on how AI dialogue, modality, and language combine in academic learning. This field study compares written activities with text- and voice-based AI dialogue in English-only and bilingual settings. Among students who completed the dialogic activities, engagement and affective outcomes were higher with comparable knowledge gains, but completion was lower; bilingual support particularly benefited voice participation.

  • Problem

    LLMs can generate written responses without students necessarily engaging with learning content, and little is known about how AI dialogue, modality, and language together affect learning and engagement.

    Source: §1 · p. 2

  • Method

    A field study with 305 native Kannada-speaking undergraduates compared written responses with AI dialogue in text or voice and English-only or bilingual Kannada-English settings.

    Source: §1 · p. 2

  • Results

    Students completing AI dialogue showed greater participation, time on task, interest, and self-efficacy with comparable knowledge gains, but fewer completed dialogic activities overall.

    Source: §5 · p. 13

  • Takeaways & Limitations

    AI dialogic learning design should support current language abilities and longer-term aspirations, including opportunities for accessible participation without replacing English.

    Source: §7 · p. 15

    The single-state study with a particular linguistic population limits generalization across diverse educational and linguistic contexts.

    Source: §6 · p. 15

Abstract

from arXiv · show

The widespread availability of LLMs is challenging written learning activities, as students can increasingly generate responses without necessarily engaging with the learning content. Conversational AI creates an opportunity to redesign these activities as dialogue, while multilingual capabilities may make such dialogue more accessible to students learning through a non-native language. We conducted a field study with 305 native Kannada-speaking undergraduate students at English-medium institutions in Karnataka, India. We compared written response activities with text- and voice-based dialogic activities with AI, each conducted in English-only or bilingual Kannada-English settings. Students who completed dialogic activities spent more time on the activities, contributed more, and reported greater interest and self-efficacy than those completing written responses, although fewer students completed the dialogic activities overall. Knowledge increased across all conditions, with no reliable differences in gains between activity formats or languages. Language shaped participation differently across modalities: bilingual interaction was particularly beneficial in voice dialogue, where it reduced articulation difficulties, increased turns demonstrating understanding, and reduced conversation abandonment. However, students also valued English because of its connection to their academic and professional aspirations. These findings show that designing dialogic learning with AI requires more than choosing between writing and dialogue, voice and text, or English and students' native languages. We highlight opportunities to give learners greater control over modality, information, and pace, and to use native languages as translanguaging support rather than as a replacement for English.

1 Introduction

The introduction frames AI dialogue as a response to concerns that LLMs can make written work poor evidence of students’ thinking, and asks how activity format, modality, and language shape learning and engagement. A field study compares written activities with text- and voice-based AI dialogue in English-only and bilingual settings, finding comparable knowledge gains but differences in engagement and participation.

1 Introduction

  • LLMs can generate written responses without students engaging with the learning content, making completed writing less reliable as evidence of meaningful learning.AI dialogue offers an alternative in which students develop and articulate thinking through feedback, follow-up questions, and reflection.Source: §1 · p. 2
  • The study addresses limited evidence about how replacing written activities with AI dialogue interacts with text or voice and English-only or bilingual language use.It examines engagement and learning among 305 native Kannada-speaking undergraduates at English-medium institutions in Karnataka.Source: §1 · p. 2
  • Compared with written activities, AI dialogue increased engagement, interest, and self-efficacy while producing comparable knowledge gains, but fewer students completed it overall.The study frames this as a trade-off between engagement and completion.Source: §1 · p. 2
  • Bilingual support particularly improved voice participation, reducing articulation difficulties, increasing turns demonstrating understanding, and reducing conversation abandonment; these language differences were not evident in text.Many students also valued English for improving proficiency, so the authors propose learner control over modality, information persistence, and pace, with native languages supporting rather than replacing English.Source: §1 · p. 2

2 Related Work

Related work connects AI-supported dialogue, interaction modality, and bilingual learning support, while identifying limited evidence about their combined use for academic learning in English-medium settings. The paper studies this intersection by comparing written responses with AI dialogue across modalities and language settings.

2.1 From Written Responses to Dialogue with AI

  • Prior work shows how dialogue can elicit thinking through feedback and revision, while conversational AI may bring dialogic learning to scale; little is known about replacing written responses with AI dialogue in the same academic context.The comparison covers completion, behavioral engagement, knowledge gains, interest, and self-efficacy.Source: §2.1 · p. 3

2.2 Interaction Modality in Dialogic Activities with AI

  • Text and voice impose different communication constraints and may shape participation differently, but evidence in educational settings is limited and multilingual academic learning remains understudied.Prior findings include modality-dependent preferences and engagement despite similar learning outcomes.Source: §2.2 · p. 3

2.3 AI for Bilingual Learning Support

  • AI can help non-native English-speaking students manage language demands, yet prior work largely treats it as support around academic tasks or studies language practice rather than bilingual dialogue for learning academic content.Evidence from English-learning tools alone cannot establish whether bilingual dialogue improves participation or learning in EMI academic activity.Source: §2.3 · p. 4
  • The paper brings these strands together in a real-world English-medium setting by comparing written responses with AI dialogue and examining modality, language, and students’ experiences.Prior research had separately addressed dialogic learning, text and voice affordances, and AI support for multilingual learners.Source: §2 · p. 3, §2.3 · p. 4

3 Study Methodology

The study uses a field experiment with native Kannada-speaking undergraduates at English-medium institutions, varying activity format, dialogue modality, and language while holding instructional content broadly consistent. Surveys, platform logs, discourse coding, and interviews capture learning outcomes, engagement, and student experience.

3.1 Experimental Conditions

  • The field study enrolled 305 native Kannada-speaking undergraduates in Karnataka and assigned participants to written-response or AI-dialogue activities in English-only or bilingual conditions.Dialogic activities used either text or voice, yielding six between-subjects conditions.Source: §3 · p. 4, §3.1 · p. 4

3.2 Learning Module and Study Platform

  • All conditions used the same learning module, prompts, feedback rubric, and feedback-generation prompt, while varying format, modality, and language.Dialogic feedback arrived immediately, whereas written-response feedback was delayed 12 hours to reflect conventional classroom assignments.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.3 Field Deployment and Study Procedure

  • Students completed pre-study enrollment and survey, learning activities, and a post-study survey one week later across two deployments.Of 305 enrolled students, 173 completed; the first deployment was interrupted by permanent closure of its centers after funding cuts.Source: §3.3 · p. 5

3.4 Data Collection and Analysis

  • The study combined pre/post surveys, LMS logs, and semi-structured interviews; most analyses focused on the 173 completers, while discourse analysis included all students with conversational data.Sixteen students were interviewed, sampled across conditions and engagement levels.Source: §3.4 · pp. 5, 7
  • Knowledge was assessed with 15 repeated pre/post items, while validated scales measured interest and self-efficacy and a post-study measure captured activity preference.The knowledge assessment combined multiple-choice and open-ended questions.Source: §3.4 · pp. 5–6
  • LMS logs captured activity timing, written responses, keystrokes, and conversations; engagement measures included elapsed time, student word count, and dialogue turns.Inactive intervals were not removed from time-on-task calculations.Source: §3.4 · p. 6
  • Potentially AI-assisted typed responses were flagged using unusual length and supporting indicators, with flagged words removed from word-count analyses but responses retained for turn counts.The indicators applied only to typed responses; absence of flagged voice responses was treated as evidence the shortcut was uncommon, not proof it never occurred.Source: §3.4 · p. 6
  • Discourse analysis coded student understanding and friction, tutor behavior, and conversation resolution; only resolution, demonstrated understanding, and student friction met substantial agreement for quantitative comparisons.Automated annotation was checked against human labels on 100 conversations sampled across the four dialogue groups.Source: §3.4 · pp. 6–7

4 Findings

The findings examine how dialogic AI activities and written responses related to engagement, completion, and knowledge gains, and how language effects varied by modality. They also describe students’ views of bilingual dialogue and English, alongside perceived supports and risks of AI use.

4.1 Impact on Engagement and Completion

  • Median student-produced response lengths were 102 words (voice), 58.5 (text), and 34 (written), while median activity times were 19, 14.6, and 3.8 minutes (text, voice, written).Dialogue responses remained longer than written responses whether or not AI-generated text was excluded from word counts.Source: §4.1 · pp. 7, 10, Figure 3 · p. 8
  • Voice turn counts ranged from 0–204 (IQR 13–92), versus 12–138 (IQR 15.8–42.3) in text, showing greater dispersion but no significant median-turn difference.Speaking felt more fluid for some students, while others preferred text’s ability to revisit exchanges; voice tutor interruptions and overlapping speech also pressured students to respond quickly.Source: §4.1 · p. 8
  • 43.9% vs. 20.5% of turns showed topical understanding in bilingual vs. English voice, while articulation difficulty (17.5% vs. 44.3%) and abandonment (65.4% vs. 86.7%) were lower.In text dialogue, understanding and abandonment did not differ significantly by language; students said text gave them more time to formulate English, while speaking Kannada was easier than typing it.Source: §4.1 · pp. 9–10
  • 68.6% completed written response versus 52.3% in text dialogue and 48.4% in voice, although 68% of dialogue completers preferred the study activity versus 14% in written response.The format completion gap persisted in the full-time college deployment (100% written, 83.0% text, 79.4% voice); completion and preference did not differ systematically by language.Source: §4.1 · p. 10

4.2 Learning Outcomes and Students’ Experiences with AI

  • Knowledge scores rose from 5.45 to 8.89/15 across conditions, with no reliable gain differences by format or language (dialogic vs. written: 0.07 points, p= .88).Bilingual voice had the highest adjusted knowledge score (9.91), but its 1.89-point advantage over English voice was suggestive and inconclusive (p= .052).Source: §4.2 · pp. 10–11, Figure 4 · p. 11
  • Dialogic formats exceeded written response by 0.47 points in interest and 0.29 points in self-efficacy (both p< .001).The tutor’s immediate feedback, follow-up questions, examples, and hints built on students’ responses and helped them continue reasoning.Source: §4.2 · p. 11
  • Responsive follow-up, reassurance, and a perceived lack of judgment helped students persist through uncertainty and were proposed as contributors to greater interest and self-efficacy, but not reliably greater knowledge gains.The tutor acknowledged partial answers, asked follow-up questions, offered familiar examples and hints, and reassured hesitant students.Source: §4.2 · p. 12

4.2.3 Bilingual dialogue supported interest and self-efficacy, while English retained aspirational value

  • Bilingual participants reported 0.46-point greater interest in text dialogue and 0.65-point greater interest in voice dialogue (both p < .001), plus 0.45-point greater voice self-efficacy (p = .001); written outcomes were nearly identical.In written response, adjusted bilingual-English differences were 0.03 points for interest (p = .71), 0.02 for self-efficacy (p = .80), and 0.22 for knowledge (p = .74).Source: §4.2 · p. 12
  • Students found Kannada easier for comprehension and expression, yet valued English for English-medium coursework, assessments, and its perceived academic and career benefits.This tension meant that some students preferred English dialogue to practice and improve their English, despite Kannada making interaction easier and more engaging, especially in voice.Source: §4.2 · p. 12

4.2.4 Students used AI for support but risked overreliance

  • Among submitted responses, 9.3% in written response and 7.4% in text dialogue were flagged as containing AI-generated text; interviews also described using AI to interpret questions or formulate answers.The flagged rates were 8.6% for English and 10% for bilingual written responses, and 6.8% for English and 8.1% for bilingual text dialogue; flags do not capture all AI use or establish equal levels of reliance.Source: §4.2 · p. 12
  • Students also described habitual AI use before thinking through questions and using it for short assignment answers, raising the challenge of supporting learners without displacing independent reasoning.AI’s convenience could help with language barriers while encouraging students to seek ready-made answers.Source: §4.2 · pp. 12–13

5 Discussion

AI dialogue increased engagement among students who completed it, but fewer students completed dialogic activities overall. The discussion connects modality and language choices to learner control and to students’ present abilities and future aspirations.

5 Discussion

  • Because AI can produce assignment answers with little student effort, a completed written response may not reliably evidence students’ own thinking or engagement.The authors argue for activities that elicit reasoning throughout the process.Source: §5 · p. 13
  • Dialogic-activity completers participated more, spent longer, and reported greater interest and self-efficacy than written-response completers, with comparable knowledge gains but fewer overall completions.Students said immediate feedback, hints, and encouragement helped them continue when unsure.Source: §5 · p. 13

5.1 Designing for learner control

  • Voice reduced typing effort and supported natural flow, while text enabled review and revision; voice’s limited reviewability and English processing demands could tax working memory.The discussion uses grounding constraints—including audibility, cotemporality, reviewability, and revisability—to frame these modality differences.Source: §5.1 · p. 13
  • Learner control should include switching between voice and text, reviewing responses, setting conversational pace, and correcting speech-recognition errors.The authors note that misunderstandings can divert attention from learning to interaction repair, and propose visible, editable transcriptions and an option to switch to text.Source: §5.1 · p. 14

5.2 Designing for aspirations beyond ability

  • Bilingual voice dialogue had fewer articulation-difficulty turns, more turns demonstrating understanding, and fewer abandoned conversations than English voice dialogue.Bilingual students also reported greater interest across text and voice, and greater self-efficacy in voice but not text dialogue.Source: §5.2 · p. 14
  • Kannada supported participation, but students also valued English for curriculum, examinations, and career prospects; neither language was universally preferable.The discussion frames this as a design tension between current participation and opportunities students hope to develop.Source: §5.2 · p. 14
  • The authors propose translanguaging so students can use Kannada to clarify concepts and develop ideas while continuing to communicate and practice academic English.They situate this approach in learning academic content through a non-native language, not only in language learning.Source: §5.2 · p. 14

6 Limitations and Future Work

The optional, low-stakes deployment and its setting in one state and linguistic population constrain interpretation and generalizability.

6 Limitations and Future Work

  • 173 of 305 enrolled students completed the optional, low-stakes study, and differential attrition across conditions may have influenced outcome analyses.The authors suggest testing required or graded settings and using larger studies to detect smaller effects and examine variation across students and contexts.Source: §6 · p. 15
  • Because the study involved one state and a particular linguistic population, its findings may not generalize across diverse educational and linguistic contexts.The authors call for replication across states, institutions, and linguistic communities.Source: §6 · p. 15

7 Conclusion

Among 305 Kannada-speaking students in English-medium institutions, dialogic AI activities elicited greater engagement among completers but had lower overall completion than written responses. The conclusion presents dialogic AI as a distinct format whose benefits depend on sustaining and completing interaction, and calls for designs that support both present participation and future aspirations.

7 Conclusion

  • Dialogic activities elicited greater participation, time on task, interest, and self-efficacy among completers, with comparable knowledge gains but lower overall completion than written responses.The study examined 305 Kannada-speaking students in English-medium institutions; bilingual support was particularly beneficial in voice interaction.Source: §7 · p. 15
  • The authors frame dialogic AI as a distinct learning format, not a universally better replacement for writing, with benefits depending on students sustaining and completing the interaction.Source: §7 · p. 15
  • AI-based dialogic learning should make participation accessible while preserving opportunities for students to think, express understanding, and develop valued capabilities.The conclusion connects these needs to how students can participate now and how they hope to learn and grow.Source: §7 · p. 15

A.2 Learning Module

The learning module introduced data-driven reasoning through four videos, each paired with a learning activity.

A.2 Learning Module

  • The module covered how digital systems use data to produce conclusions, suggestions, and other outputs.It was developed with the implementation partner’s pedagogy team.Source: §A.2 · p. 18
  • Four five-to-eight-minute videos covered introductory, predictive, discovery, and generative reasoning, with English visuals and terminology and Kannada-English narration.Each video was followed by a corresponding activity; the videos addressed uncertainty in predictions and the need to verify generated content.Source: §A.2 · p. 18

A.3 Survey Instruments

The pre-survey gathered demographic information and assessed students’ awareness, AI knowledge, interest, and self-efficacy using structured questions and rating scales.

Demographic Information

  • Students first provided demographic information, including their name, phone number, degree, and year of study.Source: §A.3.1 · p. 18

Awareness

  • Awareness of predictive, discovery, and generative reasoning was rated on a five-point scale from “Not at all aware” to “Extremely aware.”Source: §A.3.1 · p. 18

AI Knowledge

  • The 17-question AI knowledge section included multiple-choice and open-ended items about prediction, classification, clustering, tokenization, and generative AI.Source: §A.3.1 · p. 18

AI Interest and Self-Efficacy

  • Students rated AI interest and confidence with AI on five-point Likert scales, with items covering enjoyment, understanding, applications, and ethical issues.Source: §A.3.1 · p. 20

A.3.2 Post-Survey.

The post-survey repeated the pre-survey’s first four sections and added questions about activity preference and research interviews.

A.3.2 Post-Survey

  • Sections 1 through 4 repeated the pre-survey unchanged, except that the knowledge instruction asked students to answer based on course learning.Source: §A.3.2 · p. 20
  • The post-survey asked students to compare the course activity with usual learning activities and explain their preference, then asked whether they would consider a research interview.Source: §A.3.2 · p. 20

A.4 Conversation Analysis Codebook

The conversation-analysis codebook organized codes by the units and speakers to which they applied.

A.4 Conversation Analysis Codebook

  • The codebook groups four codes for student turns, four for agent turns, and one for whole conversations, with descriptions and labeled-unit counts.Source: §A.4 · p. 20

A.5 Agreement Between the Model and the Human Labels

The appendix describes validation of automated conversation labels, the interview protocol, and the codebooks used for conversation and interview analyses.

A.5 Agreement Between the Model and the Human Labels

  • A balanced sample of 100 conversations formed the human-labeled gold set; its 838 turns matched model outputs by conversation and turn identifiers, with speaker labels agreeing throughout.Source: §A.5 · p. 20
  • Agreement was reported as raw agreement and Cohen’s kappa, which adjusts for agreement expected by chance.Source: §A.5 · p. 20

A.6 Interview Protocol

  • The semi-structured interview guide covered students’ studies, course learning and activity experiences, language and modality preferences, and AI use.Source: §A.6 · p. 20

A.7 Interview Codebook

  • The conversation codebook reports code descriptions and counts for the gold and annotated corpora, while the interview codebook counts coded quotations across 12 interviews.Source: §A.7 · p. 20, Table 3 · p. 21, Table 5 · p. 22

A.5 Agreement Between the Model and the Human Labels

  • Table 4 reports model–human label agreement, evaluating turn-level codes on the relevant student or agent turns and conversation resolution on whole conversations.Source: Table 4 · p. 21
Loading 2610.07483v1…