Source-linked AI summary
Towards Breaking the Learning System Wall Using Multimodal Tutoring Transcriptions
Danielle R. Thomas, Marie Cynthia Abijuru Kamikazi, Ashish Gurung, Ishan Miglani, Shivang Gupta, Zachary Levonian, Conrad Borchers, Kenneth R. Koedinger
TL;DR
Differences in learning-system logs make student learning difficult to track across platforms. The paper introduces a video-based system for generating unified transcripts and describes planned alignment and classification procedures. Its transcripts capture student and tutoring activity that conventional logs may miss, while temporal, data-matching, and validation challenges remain.
Problem
Learning-system logs differ across platforms, limiting the generalizability of log-based models and the ability to monitor learning trajectories across systems.
Source: §1 · p. 1Method
The paper introduces screenplay-style transcripts of screen-recording videos that capture dialogue and on-screen actions, with planned alignment to MATHia logs and classification of learning processes.
Source: §1 · p. 2Results
The transcripts capture student behaviors, chat, external-tool use, and instructional context that may be absent from native logs, including silent actions and activity outside MATHia.
Source: §5 · pp. 6–7Takeaways & Limitations
The approach offers an initial procedure for creating structured interaction data from screen recordings, including where native learning-log data are unavailable.
Source: §5 · p. 7, §6 · p. 7Challenges include incomplete video-log matching, unvalidated integrated audio transcription, uncategorized non-MATHia events, and timing drift that currently requires manual recalibration.
Source: §2.3 · p. 2, §4 · p. 6
Abstract
from arXiv · showhide
Past research using log data has faced the "learning system wall," whereby few methods exist for generalizing models of student learning across platforms. Increasingly, online learning is captured by richer forms of data, including dialog and video, with new affordances. An example of this is remote tutoring programs, where human tutors support students who use learning systems while video conferencing. Toward better platform-general modeling of learning, we introduce an AI-driven multimodal transcription system that processes screen-recording videos into unified screenplay-style transcripts containing audio dialogue and annotated learning log actions. We describe a planned method for temporally aligning AI-generated multimodal transcripts with MATHia learning logs and for identifying and classifying student learning processes to align with MATHia logs. Lastly, we highlight challenges and potential solutions in capturing learning processes in one system, offering initial steps towards generalizing log data across diverse systems.
1 Introduction
The introduction frames limited generalizability across learning-system logs as a barrier to tracking learning across platforms. It presents video-based multimodal transcripts and planned alignment to MATHia logs as initial steps toward more generalizable learning data.
1 Introduction
- Tutoring is described as an effective educational intervention, but relatively little is known about the instructional interactions that make it effective.Remote and hybrid tutoring combine learning-system use with personalized tutor support.Source: §1 · pp. 1–2
- Different platforms define and record learning events such as attempts and hints differently, limiting the generalizability of their log data.The resulting “learning system wall” makes it difficult to monitor a student's learning trajectory when they switch systems and challenges large-scale AI modeling efforts.Source: §1 · p. 1
- Screen-recording video can provide a unified representation of student actions across platforms, offering a potential way to address differences in learning-system logs.The introduction notes that video-based representations could eventually capture actions across diverse learning platforms.Source: §1 · pp. 1–2
- The paper introduces screenplay-style transcripts that capture dialogue and on-screen actions, alongside a planned method to align transcripts with MATHia logs and classify learning processes.The longer-term goal is to explore whether this single-platform approach could extend to other learning platforms.Source: §1 · p. 2
2 Related Work
Prior work situates remote and hybrid tutoring as a combination of software instruction and human support, while noting that isolated logs omit contextual learning activity. The section also outlines temporal, sequencing, privacy, and long-video challenges for AI-generated tutoring transcripts.
2.1 Remote and Hybrid Human-AI Tutoring
- Hybrid human-AI tutoring pairs adaptive software practice with human support, and this combination has been associated with improved learning and engagement.Tutors may monitor students' progress and provide support while students work independently.Source: §2.1 · p. 2
2.2 Multimodal Transcripts of Tutoring
- Isolated software logs lack classroom and video-conferencing context, while merging audio, video, and logs can be logistically complex and suffer from low match rates.Prior work aligned minute-level ITS logs with Zoom audio transcripts to study tutoring visits and engagement.Source: §2.2 · p. 2
2.3 Challenges Facing AI-Generated Multimodal Transcription
- MLLM transcript generation faces temporal drift, missequencing of concurrent speech and actions, privacy restrictions on human video validation, and boundary information loss when long recordings are segmented.The paper proposes overlapping segments, temporal mismatch resolution, a focus on on-screen activity, and event classification mapped to MATHia logs as responses to these challenges.Source: §2.3 · pp. 2–3
3 Method
The method converts screen recordings into unified screenplay-style transcripts, then describes planned alignment with MATHia logs and classification of transcript actions into MATHia event categories. It also specifies privacy measures and acknowledges that alignment and classification remain planned procedures.
3.1 Transcript Generation
- Screen recordings are divided into overlapping segments, transcribed under structured constraints, and post-processed into a unified chronological transcript.The planned pipeline adds session metadata to prompts, requires verbatim dialogue and detailed screen actions, and deduplicates events across overlapping segments before remapping their timestamps.Source: §3.1 · pp. 3–4
- Video transcription can capture silent drawings, external-tool use, and other learning activity that conventional merged streams may miss when it creates no software log or audible speech.The table caption identifies interactive-whiteboard activity as an example of an event absent from available data streams.Source: §3.1 · p. 4, Table 1 · p. 5
3.2 Data collection
- The data-collection procedure de-identifies transcripts and logs, removes personally identifiable information, and processes videos with locally hosted MLLMs under approved IRB protocols.The stated privacy measures keep student data within a secure local environment and out of external model training.Source: §3.2 · p. 4
3.3 Planned procedure for temporal alignment
- The planned alignment procedure maps a video timestamp for an initial student action to the first corresponding MATHia log action to establish a temporal offset.If the student was already logged in before the tutor entered, the method instead uses a visually observable event and the first recorded backend action.Source: §3.3 · pp. 4–5
- Because clock drift can accumulate during a session, the planned alignment process includes manual recalibration against later visual milestones and MATHia events.The passage describes this as a response to drift from clock variance, network delays, or inaccuracies in MLLM timing.Source: §3.3 · p. 5
3.4 Planned procedure for identifying and classifying learning processes
- The planned classifier uses an approximately 50-transcript development set to associate transcript phrases with four MATHia event types, then evaluates the rules on a manually annotated holdout set.The four categories are attempt, hint_request, hint_level_change, and done; evaluation will use precision, recall, F1-score, Cohen’s Kappa, and development-to-test comparisons.Source: §3.4 · pp. 5–6
4 Challenges
The paper identifies practical challenges in creating MATHia-aligned multimodal transcripts, including incomplete data matching, unvalidated audio transcription, uncategorized non-MATHia activity, and timing drift. Manual recalibration may mitigate drift, but limits fully automated synchronization at scale.
4 Challenges
- Video and MATHia logs may not match because recordings can be incomplete or begin at different points, and the current plan validates only on-screen math-logged events, not integrated audio transcription.The plan also lacks a way to distinguish MATHia-aligned events from other learning activities, such as calculator or whiteboard use.Source: §4 · p. 6
- Transcript and log timelines can drift because transcription timing is not standardized like learning-log timing, and manual recalibration limits hands-off synchronization at scale.The paper describes manual realignment against backend logs as a way to mitigate drift.Source: §4 · p. 6
5 Discussion
The case study shows how multimodal transcripts can recover tutoring interactions and learning activity that native MATHia logs miss. The approach offers a unified record, while cross-platform generalizability remains a future goal.
5 Discussion
- The work demonstrates a planned procedure for distilling videos into learning-log data in a single-platform case study, using native MATHia logs as ground truth.Testing generalizability on other platforms remains future work.Source: §5 · p. 6
- Video transcripts recorded non-verbal behavior and ecological context, such as distraction or leaving the computer, beyond audio-only logs and native inactivity labels.MATHia labels this inactivity as “idleness.”Source: §5 · p. 6
- Screen recordings revealed silent Zoom-chat requests for help that MATHia logged only as erroneous attempts.In one session, the transcript captured a student typing, “I do, but I don’t really know how to use it.”Source: §5 · p. 6
- Table 2 describes an input/output classification for mapping AI-generated transcripts and MATHia ground truth to event types.Source: Table 2 · p. 7
- Transcripts captured student and tutor problem-solving outside MATHia, including external-tool use and whiteboard scaffolding absent from native logs.MATHia recorded a final answer or successful attempt without the external calculation or tutor’s whiteboard support that preceded it.Source: §5 · pp. 6–7
- The screenplay-style transcript provides a unified chronological record of dialogue and on-screen actions without merging separate audio, chat, and clickstream data.It also captures math and problem-solving steps with ecological context, including when native learning logs are absent.Source: §5 · p. 7
6 Conclusion
The conclusion presents an AI-driven method for turning tutoring screen recordings into screenplay-style transcripts and structured interaction data. It outlines planned alignment and event classification while identifying automation and platform-specific formatting as challenges.
6 Conclusion
- The proposed system converts tutoring screen recordings into screenplay-style transcripts that capture behaviors such as whiteboard drawings, chat messages, and external-tool use.Source: §6 · p. 7
- The planned procedure synchronizes transcripts with MATHia logs using periodic manual recalibration for cumulative clock drift, then classifies transcript actions into standardized learning events.Source: §6 · p. 7
- Automation challenges and platform-specific formatting remain, and the procedure is an initial step toward structured interaction data from other educational systems.Source: §6 · p. 7