PaperlayerView exact revision on arXiv

Source-linked AI summary

Empowering Personalized Learning through a Conversation-based Tutoring System with Student Modeling

Minju Park, Sojung Kim, Seunghyun Lee, Soonwoo Kwon, Kyuseok Kim

arXiv:2403.14071v1cs.HC

TL;DR

LLM-based tutoring systems face challenges in assessing students and incorporating those assessments into personalized teaching. This paper combines diagnostic student modeling with prompt-engineered instructional strategies in a proof-of-concept system tested with 20 participants, finding support for personalization while identifying limits in actionable adaptation and learning-gain evidence.

  • Problem

    Prior conversation-based LLM tutors often lack systematic student assessment, limiting the personalization of adaptive tutoring strategies.

  • Method

    The paper designs and implements a proof-of-concept tutor combining diagnostic components with LLM prompting that incorporates assessment results and personalized instructional strategies.

  • Results

    The 20-participant study supports the framework’s ability to facilitate personalization, particularly through its student-modeling elements.

  • Takeaways & Limitations

    Personalized LLM tutoring can be structured around explicit student modeling covering cognitive state, affective state, and learning style.

  • Takeaways & Limitations

    Precise student assessment did not always produce discernible, actionable tutoring strategies, and the system did not ensure significant learning gains.

Abstract

from arXiv · show

As the recent Large Language Models(LLM's) become increasingly competent in zero-shot and few-shot reasoning across various domains, educators are showing a growing interest in leveraging these LLM's in conversation-based tutoring systems. However, building a conversation-based personalized tutoring system poses considerable challenges in accurately assessing the student and strategically incorporating the assessment into teaching within the conversation. In this paper, we discuss design considerations for a personalized tutoring system that involves the following two key components: (1) a student modeling with diagnostic components, and (2) a conversation-based tutor utilizing LLM with prompt engineering that incorporates student assessment outcomes and various instructional strategies. Based on these design considerations, we created a proof-of-concept tutoring system focused on personalization and tested it with 20 participants. The results substantiate that our system's framework facilitates personalization, with particular emphasis on the elements constituting student modeling. A web demo of our system is available at http://rlearning-its.com.

1 INTRODUCTION

The paper proposes a personalized conversation-based tutoring system that combines diagnostic student modeling with LLM prompting and instructional strategies. A 20-participant proof-of-concept study examined how these design elements support personalization.

  • 1 INTRODUCTION: Prior LLM tutoring systems often rely on text inputs without systematic student assessment, which may limit achievable personalization.The paper positions explicit diagnostic components as a response to this gap.
  • 1 INTRODUCTION: The system combines diagnostic student assessment with an LLM tutor prompted to incorporate assessment outcomes and instructional strategies.The assessment considers cognitive state, affective state, and learning style.
  • 1 INTRODUCTION: The personalized instructional strategies include adaptive exercise selection, state-tailored interventions, and teaching customized to learning style.The design is intended to support a plug-and-play tutor across skill domains through domain-specific prompts.
  • 1 INTRODUCTION: The proof-of-concept teaches Pronouns, Punctuation, and Transitions through onboarding, pre-test, tutoring, and post-test stages.The system was implemented for SAT Writing concepts and made available through a web demo.
  • 1 INTRODUCTION: 20 participants took part in tutoring sessions used to examine whether the system implemented personalization and aligned with its design objectives.The study also provided examples of personalization in student modeling and identified potential improvements and challenges.

2 RELATED WORK

Related work establishes LLM-based tutoring as a growing area spanning multiple skill domains, while identifying insufficient explicit student modeling as a personalization gap. The paper connects cognitive diagnosis and adaptive prompting to address that gap.

  • 2 RELATED WORK: Existing LLM tutoring systems address argumentation, reading comprehension, mathematical reasoning, and language learning, often using prompt engineering.Examples include ArgueTutor, DIRECT, MATHDIAL, and a language-learning system.
  • 2.1 Intelligent Tutoring Systems: Many prior systems track student progress through text-based cues rather than explicit student modeling, limiting diagnostic support for adaptive tutoring strategies.The paper expects this limitation to be especially relevant for zero- or few-shot LLM systems without task-specific fine-tuning.
  • 2 RELATED WORK: The paper motivates combining multi-perspective student modeling with prompts that adapt to the resulting assessments.The proposed perspectives include cognitive state, affective state, and learning style.
  • 2.2 Cognitive Diagnostic Modeling: Cognitive Diagnosis Modeling assesses examinees’ knowledge proficiency from test results and can support efficient learning-material selection in e-learning systems.The paper uses this motivation to connect assessment with item recommendation in intelligent tutoring systems.
  • 2.2 Cognitive Diagnostic Modeling: The paper employs Item Response Theory to assess students and use the assessment outcomes for adaptive exercise recommendations.IRT is presented alongside DINA as a traditional cognitive-diagnosis method.

3 DESIGN OF PERSONALIZED TUTORING SYSTEM

The system combines multi-faceted student assessment with LLM prompt-based tutoring and instructional strategies. Its proof-of-concept models cognitive, affective, and learning-style information, then uses these assessments to personalize exercises and teaching.

  • The design combines multi-faceted student assessment with LLM prompt-based tutoring that incorporates assessment outcomes and instructional strategies.
  • The user flow moves from onboarding and a 15-question SAT Writing pre-test to personalized tutoring and post-testing across three concepts.
  • 3.1 Student Modeling: Student modeling assesses cognitive state through IRT proficiency, self-reported metacognition, GPT-4 session summaries, and learning gain.
  • 3.1 Student Modeling: Affective-state assessment uses discrepancies between modeled and self-reported proficiency plus GPT-4-inferred engagement, motivation, and subject favorability, but is explicitly limited.
  • 3.1 Student Modeling: Learning-style modeling adopts Felder and Silverman’s framework but adapts only perception, processing, and understanding because the chat interface constrains input-mode adaptation.
  • 3.2 LLM Prompt-Based Personalized Tutoring System: Adaptive exercise selection uses one-dimensional 2PL IRT to choose items near a student’s estimated skill, targeting approximately 0.5 probability of correctness.The selected exercises, answers, and explanations are inserted into the prompt for the LLM tutor.

4 RESULTS

The proof-of-concept study examined personalization across 20 participants by analyzing tutoring dialogues and examples of student-model-driven instructional adaptation. The system varied tutor actions and strategies across students, while also revealing limits in linking assessment to actionable tutoring and sustaining engagement.

  • 4.1 Observation: Across 20 participants, dialogue analysis found that tutor-action proportions varied by student, indicating individualized teaching approaches.The study examined complete tutoring cycles involving participants with diverse English proficiency levels.
  • 4.1 Observation: The system personalized teaching across cognitive state, affective state, and learning style, including reinforcement when self-assessed proficiency differed from pre-test performance.One example used a stronger measured punctuation proficiency to encourage a student who rated their proficiency lower.
  • 4.1 Observation: For highly engaged students, GPT-4 recommended prompting them to explain their reasoning, and subsequent tutoring implemented that recommendation.The tutor asked a student who answered correctly to explain why “its” was appropriate.
  • 4.1 Observation: GPT-4 adapted instructional strategies to students’ learning styles across initial and subsequent sessions.Later strategies used updated learning-style assessments to recommend activities such as summarization and reinforcement.
  • 4.2 Discussion: Precise student assessment did not always yield clearly actionable tutoring strategies, possibly because the system emphasized question-answer dialogue.The authors identify the assessment-to-strategy connection as requiring closer examination.
  • 4.2 Discussion: The study also found engagement challenges in chat, reflected in word-count differences between students and the tutor and students’ insincere responses.The authors suggest more engaging, interactive content as a possible response.

5 CONCLUSION

The paper presents a personalized tutoring system centered on diagnostic student modeling and an LLM-based proof-of-concept implementation. It concludes that assessment-to-strategy alignment and user engagement remain key areas for improvement.

  • 5 CONCLUSION: The paper presents a personalized tutoring system focused on diagnostic student modeling and leveraging LLM effectiveness.The authors describe design considerations and a proof-of-concept system built from them.
  • 5 CONCLUSION: The authors identify connecting precise student assessments to effective tutoring strategies and enhancing user engagement as priorities for future refinement.These areas are presented as the basis for advancing personalized tutoring systems.

A USER FLOW OF OUR SYSTEM

The system guides learners through onboarding, adaptive pre-testing, personalized tutoring, post-testing, and repeated learning cycles across three SAT Writing concepts. Each cycle uses assessment information to select exercises and later show pre-test/post-test proficiency changes.

  • A USER FLOW OF OUR SYSTEM: Onboarding collects learning-style preferences, confidence in SAT Writing concepts, and demographic information to form an initial student profile.The onboarding survey also introduces system functionality and expectations.
  • A USER FLOW OF OUR SYSTEM: A 15-question pre-test measures proficiency across Pronouns, Punctuation, and Transitions, with five questions allocated to each concept.Pre-test results feed the IRT model and update the student’s affective state.
  • A USER FLOW OF OUR SYSTEM: IRT-based pre-test results adaptively select exercises for the ensuing tutoring session.Tutoring uses a split interface combining learning materials with an AI-tutor chat.
  • A USER FLOW OF OUR SYSTEM: During tutoring, students proceed at their own pace in a chat-centered experience until all selected exercises are covered.The session is personalized using the pre-test results.
  • A USER FLOW OF OUR SYSTEM: A post-test on the tutored concept enables side-by-side comparison with pre-test proficiency to show progress.The post-test typically contains five questions on the same knowledge concept.
  • A USER FLOW OF OUR SYSTEM: Students repeat tutoring and post-testing for the remaining concepts until all three SAT Writing concepts are completed.After each round, students choose another remaining knowledge concept.

B PROMPT

The prompt design turns student-model outputs into explicit instructions for an LLM tutor. It combines initial and updated learner information with adaptive exercises, learning-style strategies, session summaries, and turn-level tutoring rules.

  • B.1 System Prompt: The initial student model combines onboarding and pre-test information, while later prompts incorporate updated student-model information.The model includes pre-test proficiency, metacognitive proficiency, and learning style.
  • B.1 System Prompt: Prompt examples encode discrepancies between self-reported and measured proficiency as targeted encouragement for the student.The example instructs the tutor to tell a student they are better than they think according to the pre-test.
  • B.1 System Prompt: Learning-style instructions recommend brainstorming, abstract concepts, active participation, and connecting specific details with broader concepts.These strategies are framed for a student identified as Active/Intuitive/Global.
  • B.1 System Prompt: Turn-level rules require one question per turn and deeper questioning after correct answers rather than immediately advancing.For incorrect answers, the tutor should ask further questions and help the student reach the answer instead of giving it immediately.
  • B.1 System Prompt: The tutor is instructed to vary teaching methods, use analogies and visual aids, support the student, and check understanding frequently.The prompt also directs adaptation to student preferences.
  • B.2 Summary Prompt: The summary prompt asks GPT-4 to identify covered topics, proficiency, engagement, and chat-applicable action items from the dialogue.The resulting summary supplies information for subsequent tutoring decisions.

C EXPERIMENTAL RESULTS

The system analysis reports how the tutoring implementation operated and outlines future use of its methodology in more systematic experiments.

  • Future work will apply the system-analysis methodology within a more methodical experimental design to explore new research questions.
  • AUC averaged 0.65, while exercise correctness was 0.57, indicating adaptive selection aligned exercise difficulty with students’ proficiency.The system selected exercises so each student’s probability of a correct answer would approach 0.5.
  • The tutoring system’s adaptive exercise selection mechanism operated as intended by targeting exercises near each student’s proficiency level.This interpretation follows from the observed correctness ratio and the system’s exercise-selection objective.

C.0.2 Learning Gain.

Learning gain was defined as the change in average correctness probability between pre-test and post-test skill estimates. Across 20 participants, gains varied by concept and did not consistently indicate substantial improvement.

  • Learning gain was computed as p_post−p_pre, the difference between average post-test and pre-test correctness probabilities derived from IRT skill parameters.This measure was presented as more intuitive than the difference between θ_post and θ_pre.
  • Average learning gain was −0.0753 for Pronouns, 0.0159 for Punctuation, and −0.0102 for Transitions across all 20 participants.
  • The tutoring system did not guarantee notable learning gains, potentially because sessions were short and post-tests broadly assessed concept proficiency rather than directly matching taught content.The paper emphasizes that careful learning-gain measurement is needed to evaluate personalization and adapt the tutoring system.
Loading 2403.14071v1…