Source-linked AI summary
Integrating AI and Learning Analytics for Data-Driven Pedagogical Decisions and Personalized Interventions in Education
Ramteja Sajja, Yusuf Sermet, David Cwiertny, Ibrahim Demir
TL;DR
The paper addresses gaps in learning analytics concerning pedagogical integration, affective understanding, and actionable feedback. It develops a GPT-4-enabled tool that analyzes engagement, emotions, topics, questioning patterns, quizzes, and learning progression, including Bloom’s taxonomy. The tool supports data-driven instructional decisions, but faculty concerns about accuracy, security, and the lack of classroom experimentation limit conclusions about real-world effectiveness.
Problem
Learning analytics often lacks integration with pedagogical practice and affective learning, while educational AI models may overlook timely, continuous student feedback.
Method
The study develops and deploys a GPT-4-enabled learning analytics tool that analyzes engagement, affective states, topics, questioning patterns, quizzes, and learning progression through Bloom’s taxonomy.
Results
The tool provides real-time, data-driven views of student engagement, stress, performance, and learning patterns to support individual and class-level instructional decision-making.
Takeaways & Limitations
The tool demonstrates the potential of integrated AI and learning analytics to support responsive, personalized educational decision-making within existing educational platforms.
Takeaways & Limitations
Faculty raised concerns about AI insight accuracy and student-data security, and the study lacked classroom experimentation to evaluate effects during real teaching.
Abstract
from arXiv · showhide
This research study explores the conceptualization, development, and deployment of an innovative learning analytics tool, leveraging OpenAI's GPT-4 model to quantify student engagement, map learning progression, and evaluate diverse instructional strategies within an educational context. By analyzing critical data points such as students' stress levels, curiosity, confusion, agitation, topic preferences, and study methods, the tool provides a comprehensive view of the learning environment. It also employs Bloom's taxonomy to assess cognitive development based on student inquiries. In addition to technical evaluation through synthetic data, feedback from a survey of teaching faculty at the University of Iowa was collected to gauge perceived benefits and challenges. Faculty recognized the tool's potential to enhance instructional decision-making through real-time insights but expressed concerns about data security and the accuracy of AI-generated insights. The study outlines the design, implementation, and evaluation of the tool, highlighting its contributions to educational outcomes, practical integration within learning management systems, and future refinements needed to address privacy and accuracy concerns. This research underscores AI's role in shaping personalized, data-driven education.
1. Introduction
The study addresses gaps in learning analytics by integrating AI, NLP, and learning-sciences perspectives to produce richer and more actionable educational insights. It develops a tool for measuring engagement, tracking learning progression, assessing affective states, and supporting real-time pedagogical interventions.
- NLP can enrich learning analytics by incorporating qualitative textual data to provide more contextual and nuanced understanding of learning processes.
- Existing AI education models often prioritize algorithm development while overlooking timely, continuous feedback to students.
- The study develops an AI-empowered learning analytics tool to measure student engagement and support pedagogical interventions through real-time, actionable insights.
- The tool aims to map individual learning pathways using AI-based predictive modeling of students’ learning progression.
- The research focuses on engagement, progression, and affective states while emphasizing inclusivity and accessibility for diverse student populations.
2. Related Work
Related work shows that learning analytics spans diverse computational and educational methods but remains insufficiently connected to pedagogy, teaching practice, scalability, and affective learning. The proposed AI-enhanced tool is positioned to address these gaps through real-time, personalized, and ethically attentive insights.
- Learning analytics encompasses methods ranging from eye-tracking and dialog analysis to surveys, log analysis, visual analytics, and educational data mining.
- Deep Knowledge Tracing forecasts student performance to identify potential learning hurdles and support targeted intervention strategies.
- Learning analytics methods often lack integration with pedagogical theory and teaching practice, limiting educators’ ability to apply their insights effectively.
- Existing learning analytics also faces scalability and generalizability challenges across educational settings and insufficiently addresses affective aspects of learning.
- The proposed tool could bridge these gaps by combining pedagogical grounding, emotional sensitivity, ethical awareness, and real-time personalized feedback.
3. Methodology
The methodology extends AI-augmented educational assistants such as VirtualTA with a flexible learning analytics tool. Its purpose is to process broader interaction data than traditional learning management systems and provide educators with comprehensive views of student learning.
- The tool extends educational intelligent assistants such as VirtualTA to collect, process, and analyze data beyond traditional learning management system information.
- Designed for flexibility, the tool can support VirtualTA, other educational chatbots, smart assistants, and learning management systems.
- The overarching aim is to give educators a comprehensive understanding of students’ performance, engagement, and learning patterns.
3.1. Data Collection
The data-collection strategy combines interaction logs, emotional analysis, topic and questioning patterns, engagement measures, and available qualitative and quantitative learning data. Preprocessing cleans and updates records, while privacy-by-design practices discard raw message content after metric computation.
- The tool captures chatbot interactions, analyzes affective states, and records emotional metrics from student communications.
- It tracks question topics, questioning patterns, time spent, quiz duration, quiz outcomes, study preferences, and engagement with learning methods.
- The strategy combines LMS metadata, assignment and quiz results, learning-material time, interaction frequency, and instructor surveys for a holistic view of student behavior and attitudes.
- Data preprocessing cleans irrelevant, duplicated, incorrect, and incomplete records before refined metrics are updated on the instructor dashboard.
- For privacy, the tool stores time-series metrics rather than message content and discards raw text after computing emotional and interaction measures.
- Temporal metric collection supports a dynamic view of students’ academic and emotional journeys while preserving student anonymity.
3.2. Metrics and Indicators
The tool uses engagement, affective-state, and learning-progression indicators to characterize students’ learning experiences, while emphasizing that these metrics are non-diagnostic and require cautious interpretation.
- The metric suite also includes inferred affective states and indicators of students’ interests, struggles, and progress.
- Engagement and learning-progression metrics are prioritized because they indicate participation, interaction, academic improvement, and learning-experience effectiveness over time.
- These indicators should not diagnose or categorize students, and the tool explicitly warns that its data is reference information rather than fully factual.
3.3. Development of Learning Analytics Tool
The GPT-4 learning analytics tool integrates with educational assistants and LMS platforms to analyze student questions, affective states, topics, cognitive progression, and broader learning data.
- GPT-4 powers an extensible analytics tool that integrates with VirtualTA, other educational assistants, and existing learning platforms.
- The tool scores stress, agitation, curiosity, and confusion from 1 to 10 by applying few-shot learning and prompt engineering to students’ VirtualTA questions.
- It identifies question topics and applies Bloom’s taxonomy after students exit VirtualTA to track learning progression and inform curriculum and resource decisions.
- LMS integration adds metadata, assessment results, learning-material usage, quiz behavior, time-on-task, and preferred study methods to the analytics system.
- Figure 1 depicts data moving from LMS and virtual-assistant extraction through preprocessing, analysis, and secure storage, with Smart AI performing emotional and Bloom-based progression analysis.
3.4. Implementation and Deployment
The deployed system combines GPT-4 inference with a web-based dashboard and backend APIs, using prompt configuration and truncated conversation history to process educational interactions.
- The tool uses GPT-4 with few-shot learning, prompt engineering, and temperature 1.0 to derive metrics across educational contexts.
- Because GPT-4 supports approximately 8192 tokens, the system uses a first-in-first-out history strategy that retains the most recent interactions.
- Its dashboard uses vanilla JavaScript and Chart.js, while Express.js and Node.js support the technical foundation for visualizing and processing analytics.
- Backend APIs process VirtualTA and GPT data in structured JSON, enabling communication between educational assistants and the learning analytics tool.
3.5. Case Study Design
The case study evaluates the tool through an anonymous University of Iowa faculty survey focused on perceived educational value, LMS compatibility, privacy, and insight accuracy rather than direct tool use.
- The evaluation surveyed University of Iowa teaching faculty about the tool’s potential to improve educational outcomes and compatibility with LMS platforms such as Canvas.
- Faculty did not interact directly with the tool, so responses reflected their understanding of its described features.
- The anonymous online survey asked about important classroom insights, student privacy and data security, and the accuracy needed for reliable teaching support.
4. Results and Discussion
The learning analytics tool combines synthetic-data analysis and faculty feedback to visualize engagement, affective states, learning progression, topics, and study methods. These outputs support individualized or class-wide instructional decisions, while faculty feedback highlights the importance of learning trajectories and frequently asked questions.
- Synthetic Data Analysis: Stress dashboards display class-wide and individual emotional levels over time alongside quizzes and assignments, enabling timely responses to changes in student well-being.The tool scores stress, curiosity, confusion, and agitation from 1 to 10 as indicators of emotional intensity.
- Synthetic Data Analysis: Topic and study-method views summarize class-wide and individual patterns, including frequently discussed subjects and use of questions, quizzes, summaries, and flashcards.Duplicate topics are removed from the class table, while interaction records support adaptation of resources to student habits and learning paths.
- Synthetic Data Analysis: The tool analyzes quiz interactions, emotional states, topics, questioning patterns, study methods, and learning progression to support personalized instructional decisions.It records quiz timing and outcomes, infers stress-related affective indicators, identifies topics and study preferences, and applies Bloom’s taxonomy to student queries.
- Synthetic Data Analysis: Quiz visualizations combine time spent per question with correct and incorrect outcomes, helping educators identify complexity, comprehension gaps, and areas for targeted teaching.These views connect question-level effort with quiz performance to guide instructional focus and student improvement.
- Learning Progression and Bloom's Taxonomy: GPT-4 maps changes in the depth and complexity of student inquiries using Bloom’s taxonomy, helping educators detect learning trends and topics needing clarification.Recall and comprehension questions indicate earlier learning stages, whereas application, analysis, synthesis, and evaluation indicate more advanced understanding.
- Case Study Results: Faculty Feedback on the LA Tool: Faculty prioritized frequently asked questions at 80% and learning progression over time at 75%, while respondents represented varied disciplines and roles.The faculty sample included 38% from STEM fields, 27% selecting “Other,” and major representation from assistant, associate, and full professors.
5. Conclusion and Future Directions
The study developed and deployed an AI-enabled learning analytics tool that integrates with educational platforms to provide comprehensive, data-driven insights into student learning. Future work must validate its real-world effectiveness, strengthen emotional analysis and privacy protections, and improve applicability and usability.
- The tool integrates with platforms such as VirtualTA and combines real-time sentiment analysis with Bloom's-taxonomy-based cognitive tracking.It captures active and passive participation alongside engagement, performance, and learning patterns.
- These capabilities support timely, personalized interventions by giving educators nuanced insights into students' engagement and learning patterns.
- Further evaluation through classroom-based experimentation is needed to validate effectiveness in real teaching environments and identify workflow challenges.Classroom studies would assess practical benefits for students and integration with teaching workflows.
- Future refinements should incorporate real classroom data, attention tracking, stronger anonymization, transparent data-use policies, and compliance with data-protection regulations.
- Expansion across educational levels, including special education, together with educator training and usability improvements, could broaden adoption across disciplines.The authors also identify emerging technologies such as virtual reality as opportunities to deepen understanding of student behavior and engagement.
- The paper presents the tool as a foundational step that still requires refinement to meet practical, ethical, and technological demands.
Declaration of generative AI and AI-assisted technologies in the writing process
The authors disclose using ChatGPT based on GPT-4 to improve manuscript language and clarity. They state that it did not generate content, citations, or fact verification, and that they reviewed the final manuscript.
- ChatGPT, based on GPT-4, was used to improve prose flow, correct grammar, and enhance clarity during manuscript preparation.
- The language model was not used to generate content, citations, or verify facts, and the authors reviewed and edited the manuscript afterward.
Appendix
The appendix includes tables listing key metrics and indicators and digital planning verbs.
- Table A1 provides an overview of key metrics and indicators.
- Table A2 lists digital planning verbs.