PaperlayerView exact revision on arXiv

Source-linked AI summary

A Prompt-Engineering Approach to Develop Scalable, Flexible, and Real-Time Hybrid Micro-Level Personalization in a General Purpose AI Teaching Assistant

Saptarshi Basu, Sandeep Kakar, Ashok Goel

arXiv:2609.03402v1cs.AI

TL;DR

LLM teaching assistants provide scalable support but offer limited personalization, motivating questions about which learner characteristics should shape responses and whether adaptations produce meaningful differences. The paper introduces a prompt-engineering framework that combines six learner dimensions with Bloom’s Taxonomy-based question analysis in structured prompts. Results provide preliminary evidence of measurable and perceptible response variation, while the small human-evaluation sample limits generalization.

  • Problem

    LLM-based teaching assistants can provide scalable, course-grounded support, but the learner characteristics, implementation strategies, and interactional effects of personalization remain open questions.

  • Method

    The framework combines six learner dimensions with Bloom’s Taxonomy question analysis and encodes them in structured prompts integrated with a RAG-based tutor.

  • Results

    NLP experiments and human evaluation showed systematic response variation across conditions and identified learner-profile parameters associated with changes in generated responses.

  • Takeaways & Limitations

    Prompt-based personalization provides preliminary evidence of adaptive behavior in LLM-powered educational agents without requiring model retraining.

  • Takeaways & Limitations

    The five-evaluator human study is exploratory, cannot estimate interaction effects reliably, and requires larger, more diverse samples for generalization.

Abstract

from arXiv · show

Artificial intelligence (AI) teaching assistants powered by large language models (LLMs) offer scalable educational support but often provide limited personalization. This study presents a prompt-engineering-based framework for personalizing general-purpose LLM/RAG-based AI teaching assistants such as Jill Watson across academic disciplines and courses. The framework adapts responses using six learner-specific dimensions: self-assessment, abstraction preference, verbosity preference, perceptual orientation, information processing style, and level of understanding, yielding 96 distinct learner profiles. Student queries are additionally analyzed using Bloom's Taxonomy to estimate cognitive complexity at the interaction level. Learner attributes and cognitive assessments are encoded in structured prompts that condition the LLM without requiring model retraining. The framework is evaluated through experiments using NLP metrics and a human study with five participants. Results show perceived differences in response style and structure across personalization conditions, with statistical analyses identifying learner attributes associated with measurable response changes. These findings provide preliminary evidence that prompt-based personalization can support adaptive behavior in LLM-powered educational agents.

Introduction

LLM-based teaching assistants can provide scalable, course-grounded support, but their flexibility leaves open how learner characteristics should drive personalization and whether adaptations change instructional interactions. This paper proposes a prompt-engineering framework that combines six learner dimensions with question-level cognitive demand to personalize responses in real time without retraining.

  • LLM-based teaching assistants can generate fluent, contextually relevant, course-grounded responses across disciplines when combined with retrieval-augmented generation.
  • The study addresses which learner characteristics should drive personalization, how adaptations should be implemented, and whether they produce meaningfully different instructional interactions.
  • The framework combines six learner dimensions with Bloom’s Taxonomy-based question analysis, yielding 96 distinct learner profiles for individual student questions.
  • Structured prompt engineering conditions an existing RAG-based tutor for real-time personalization without model retraining or domain-specific authoring.
  • NLP analyses and a five-participant human evaluation provide preliminary evidence of measurable and perceptible response differences across personalization conditions.

Literature Review

Prior adaptive systems commonly personalize content or learning pathways, while recent LLM tutors broaden personalization through conversation, retrieval, and learner modeling. This work instead targets interaction-level response form by combining learner-selected preferences with system-inferred cognitive demand.

  • It combines learner-driven preference selection with system-driven cognitive classification, linking adaptability and adaptivity at the micro level.
  • Earlier adaptive learning systems typically model mastery and adapt content selection or sequencing through a diagnose-prescribe cycle.
  • Recent LLM tutoring systems personalize difficulty, personality, instructional content, or teaching workflows through conversational interaction and retrieval-augmented generation.
  • The proposed framework combines six learner dimensions with question-level Bloom’s Taxonomy analysis for dynamic response adaptation at each interaction.
  • The approach personalizes how shared instructional content is presented rather than primarily changing content or learning pathways.

Personalization Framework Design

The framework represents learner characteristics and preferences in structured prompts integrated with Jill Watson’s RAG pipeline. Student-selected response preferences and automatically inferred question complexity jointly produce 96 profiles and enable runtime customization without retraining.

  • Framework Components: The framework represents self-assessed knowledge and question-level cognitive complexity using Bloom’s Taxonomy as learner characteristics.
  • Framework Components: Five Felder-Silverman-derived preferences—abstraction, verbosity, perception, processing, and understanding—are treated as user-selected rather than fixed classifications.
  • Framework Components: These preference dimensions control response granularity, length, communication orientation, engagement style, and organizational structure.
  • Hybrid Adaptation: The combination produces 96 learner profiles, while Bloom’s Taxonomy is automatically applied to each question for dynamic interaction-level adaptation.
  • Prompt Integration: An engineered prompt modifies response form and presentation while preserving grounding in retrieved course content within Jill Watson’s RAG pipeline.
  • Prompt Integration: Runtime preference updates enable scalable personalization without modifying or retraining the underlying LLM.

Research Questions

The study asks whether learner profiles produce different response characteristics, which learner dimensions are associated with those differences, and whether observed changes match intended personalization effects. Automated NLP analysis and human evaluation are used alongside descriptive and inferential statistics to investigate these questions.

  • Research Questions: RQ1 tests whether different learner profiles produce systematically different linguistic characteristics in generated responses.
  • Research Questions: RQ2 examines which learner dimensions are most strongly associated with variation in semantic similarity, complexity, verbosity, abstraction, and processing style.
  • Research Questions: RQ3 assesses whether observed response characteristics align with the intended effects of corresponding learner dimensions.
  • Evaluation Approach: Automated NLP analyses and human evaluation are combined with descriptive and inferential statistics, including mixed-effects models, to quantify differences and associations.

Experimental Design and NLP Evaluation

The study combines automated NLP analyses with controlled prompt configurations to test whether learner-personalization dimensions produce measurable differences in generated responses.

  • Question and Prompt Construction: Thirty student questions spanning all six Bloom’s Taxonomy categories were classified with a fine-tuned BERT-based classifier.The classifier used bert-base-uncased and achieved 0.92 test accuracy, with F1 scores of 0.88–0.94 across categories.
  • Question and Prompt Construction: Ninety-six learner profiles and one non-personalized baseline generated 2,910 responses across the 30 questions.The prompt template and model configuration remained constant while learner-profile attributes varied; GPT-4.1 ran at temperature 0.
  • Automated Evaluation: Responses were evaluated for lexical similarity, semantic similarity, linguistic complexity, and verbosity.Semantic similarity used 384-dimensional all-MiniLM-L6-v2 embeddings and pairwise cosine similarity; lexical overlap used ROUGE, complexity used textstat grade-level scores, and verbosity used lexicon_count.
  • Statistical Analysis: Descriptive analyses and ordinary least squares regression examined associations between learner dimensions and response characteristics.These analyses were conducted alongside the subsequent human evaluation to address the study’s research questions.

Human Evaluation Study Design

A five-evaluator human study assessed perceived response characteristics across selected Bloom’s categories and learner profiles, using ratings and mixed-effects models.

  • Study Design: Five evaluators assessed perceived abstraction, depth of understanding, and information-processing style, complementing the automated NLP analysis.Evaluators were current or former Georgia Institute of Technology KBAI students recruited through a pre-recruitment survey.
  • Study Design: The study examined self-assessment, abstraction, and processing style across 13 profiles, including a non-personalized baseline.One representative question was selected from each Bloom’s Taxonomy category, and evaluator data were anonymized.
  • Ratings: Three hundred ninety evaluations rated overall quality, perceived complexity, abstraction level, and processing style.Overall quality used a 0–10 scale; the other dimensions used categorical response scales.
  • Inferential Analysis: Mixed-effects models treated personalization dimensions as fixed effects and evaluator identity as a random effect.The model structure accounted for inter-rater variability while excluding the baseline from inferential analysis.
  • Limitations: Because only five evaluators participated, the findings are exploratory rather than population-level evidence.The factorial design supports estimating main effects but not interaction effects; larger and more diverse samples are needed to assess generalization and interactions.

Results and Discussion

The NLP and human-evaluation results show that prompt-based personalization changes response expression, complexity, length, and perceived instructional characteristics while retaining semantic grounding. Abstraction and processing preferences were especially aligned with intended effects, although the small evaluator sample limits generalization.

  • Qualitative response differentiation: Beginner and advanced profiles produced qualitatively different instructional strategies, contrasting sequential visualization with formal decomposition and state-space reasoning.The comparison provides qualitative evidence of response differentiation under different personalization configurations.
  • NLP response variation: High semantic similarity alongside substantially lower lexical similarity indicates that responses remain semantically grounded while varying in surface expression and structure.Pairwise cosine similarity and ROUGE scores were computed for the generated responses.
  • NLP response variation: Response length varied systematically across verbosity preferences, supporting the intended effect of verbosity personalization on output length.Figure 2 reports response word count across verbosity preference categories.
  • NLP response variation: Higher self-assessment, verbosity, reflective processing, and technical abstraction were associated with greater response complexity, while Evaluation and Analysis questions also tended to yield more complex responses.These OLS associations support systematic links between personalization dimensions, Bloom’s levels, and response characteristics.
  • Human evaluation: Human evaluations found perceptible differences in quality, complexity, abstraction, and processing style; abstraction and processing preferences aligned most consistently with intended effects, whereas self-assessment effects were weaker.Abstraction preference predicted perceived quality and complexity, and processing preference predicted perceived processing style.
  • Human evaluation: The small evaluator sample limits generalization, motivating larger studies of robustness and interactions among personalization dimensions.This limitation is stated directly in the human-evaluation results.

Path to Deployment

The personalization module is intended for integration into the existing Jill Watson architecture without modifying its core infrastructure. Development is targeted for Spring 2027, followed by a pilot in selected Georgia Tech courses in Summer 2027.

  • Path to Deployment: Integration targets the existing Jill Watson architecture without modifying its core infrastructure.The planned deployment builds on an architecture already used across multiple institutions.
  • Path to Deployment: Development and integration are targeted for completion by Spring 2027, followed by a pilot deployment in selected Georgia Institute of Technology courses in Summer 2027.

Conclusions

The paper proposes a flexible, scalable framework for real-time micro-level personalization in LLM/RAG teaching assistants. Its prompt-based design combines learner dimensions and cognitive analysis, with preliminary evaluations showing systematic response variation while leaving learning-outcome evaluation for future work.

  • Conclusions: The framework enables flexible, scalable, modular, and real-time customization of LLM/RAG-based AI teaching assistants.It emphasizes hybrid adaptability and adaptivity for interaction-level personalization.
  • Conclusions: The engineered prompt combines student cognitive ability, Bloom’s Taxonomy question complexity, and learning preferences to generate 96 question-level response configurations.The framework also supports real-time prompt adaptation and modular integration within the Jill Watson architecture.
  • Conclusions: NLP experiments and human evaluations showed systematic response variation across conditions and identified student-profile parameters associated with changes in generated responses.These findings address whether personalization changes responses, which learner factors matter, and whether observed characteristics match intended effects.
  • Conclusions: The current work focuses on personalizing responses to individual student questions; UI-integrated deployment, large-scale evaluation, A/B testing, and learning-outcome assessment remain future work.
Loading 2609.03402v1…