Source-linked AI summary
LLM-powered Multi-agent Framework for Goal-oriented Learning in Intelligent Tutoring System
Tianfu Wang, Yi Zhan, Jianxun Lian, Zhengyu Hu, Nicholas Jing Yuan, Qi Zhang, Xing Xie, Hui Xiong
TL;DR
Existing ITSs and LLM-based dialogue systems often struggle to provide proactive, goal-oriented personalization. GenMentor addresses this gap with a multi-agent framework for skill mapping, adaptive learner modeling, path scheduling, and tailored content, with evaluations and deployment reporting effective guidance and resource targeting.
Problem
Existing ITSs often struggle to provide proactive, targeted learning for learners pursuing specific professional or personal goals.
Method
GenMentor uses collaborating LLM agents to map goals to skills, model learner status dynamically, schedule evolving learning paths, and generate tailored resources.
Results
GenMentor outperformed baselines across reported skill-mapping, learning-path, and content-quality evaluations and showed effectiveness in human studies and deployment.
Takeaways & Limitations
GenMentor supports personalized, goal-oriented learning through proactive guidance, targeted resources, and adaptation to professional learners’ evolving needs.
Abstract
from arXiv · showhide
Intelligent Tutoring Systems (ITSs) have revolutionized education by offering personalized learning experiences. However, as goal-oriented learning, which emphasizes efficiently achieving specific objectives, becomes increasingly important in professional contexts, existing ITSs often struggle to deliver this type of targeted learning experience. In this paper, we propose GenMentor, an LLM-powered multi-agent framework designed to deliver goal-oriented, personalized learning within ITS. GenMentor begins by accurately mapping learners' goals to required skills using a fine-tuned LLM trained on a custom goal-to-skill dataset. After identifying the skill gap, it schedules an efficient learning path using an evolving optimization approach, driven by a comprehensive and dynamic profile of learners' multifaceted status. Additionally, GenMentor tailors learning content with an exploration-drafting-integration mechanism to align with individual learner needs. Extensive automated and human evaluations demonstrate GenMentor's effectiveness in learning guidance and content quality. Furthermore, we have deployed it in practice and also implemented it as an application. Practical human study with professional learners further highlights its effectiveness in goal alignment and resource targeting, leading to enhanced personalization. Supplementary resources are available at https://github.com/GeminiLight/gen-mentor.
1 INTRODUCTION
Existing ITSs support personalization but often remain fragmented, static, or reactive, limiting proactive guidance toward specific learner goals. GenMentor addresses this gap through multi-agent skill mapping, adaptive profiling, path scheduling, tailored content, and evaluation in practice.
- Motivation: LLM-based ITSs often respond reactively to learner queries rather than proactively guiding learners toward objectives.This reactivity also limits comprehensive learner understanding and weakens personalization.
- Motivation: Professional and lifelong learners need systems that identify knowledge deficiencies and target the skills required for specific goals.Goal-oriented learning focuses on efficiently achieving objectives rather than merely receiving information.
- Approach: GenMentor distributes ITS responsibilities across collaborating LLM agents to deliver personalized, goal-oriented learning.Its components identify skill gaps, track learner status, refine learning paths, and tailor resources.
- Evidence: Automated and human evaluations, deployment, and a professional-learner study report superior learning guidance, resource targeting, and goal achievement.The reported evidence covers identified skills, scheduled paths, generated content, and practical use.
- Approach: GenMentor combines a fine-tuned goal-to-skill identifier, adaptive learner profiling, evolvable path scheduling, and exploration-drafting-integration content generation.The learner profile incorporates cognitive status, preferences, and behavioral patterns, while interactions continuously update personalization.
2 RELATED WORKS
Related work presents ITSs as personalized tutoring systems increasingly powered by data-driven methods and LLMs. Multi-agent LLM systems extend this direction by distributing complex tasks across collaborating agents.
- Intelligent Tutoring Systems: Modern ITSs use data-driven approaches for managing materials, modeling learners, and providing personalized feedback.Earlier systems combine distinct machine-learning models, which can create technical inconsistencies and fragmented data.
- LLM Multi-agent Systems: LLM-based multi-agent systems assign diverse responsibilities to multiple agents that collaboratively tackle dynamic and complex tasks.This paradigm has been explored in software development, embodied agents, and game playing.
3 PROBLEM STATEMENT
Goal-oriented learning seeks to minimize the learner’s gap between current knowledge and the skills required for a specific objective. The proposed ITS formalizes this process through skill identification, adaptive modeling, and personalized delivery.
- Goal-oriented Learning: Goal-oriented learning focuses on efficiently achieving specific objectives such as completing projects or mastering job-related skills.The system is intended to customize pathways and content so learners acquire needed knowledge quickly.
- Formalization: Given goal G, the learner must master required skills S′, while the initial skill gap is represented as ΔS_0 = S′ − S_0.The learner profile U_0 contains initial knowledge status, preferences, and behavior patterns.
- System Objectives: The framework identifies required skills, updates learner profiles from interaction data, and generates personalized learning paths and content.These modules correspond to skill gap identification, adaptive learner modeling, and personalized resource delivery.
4 THE GENMENTOR FRAMEWORK
GenMentor is an LLM-powered multi-agent framework that maps goals to skill gaps, models learners dynamically, schedules adaptive paths, and delivers tailored learning resources. Its agents use fine-tuning, evolving optimization, simulated feedback, and exploration-drafting-integration content creation.
- Framework Overview: GenMentor distributes framework responsibilities across LLM agents for skill assessment, path generation, content curation, and profile adjustment.The system begins with skill-gap assessment and continuously adjusts learner profiles using new interactions.
- Gap Identification: A customized goal-to-skills dataset derived from job postings supports fine-tuning an LLM skill identifier.The dataset pairs job summaries treated as goals with their corresponding core skills.
- Gap Identification: The skill identifier maps goal G to required skills S′, then compares S′ with initial status S_0 to filter mastered skills and identify ΔS.This process focuses learning on skills needing improvement.
- Adaptive Learner Modeling: The learner profile U captures cognitive status, learning preferences, and behavioral patterns, and is continuously refined from interaction data.Profile updates can adjust content difficulty, activity selection, and motivational interventions.
- Path Scheduling: The path scheduler initializes and iteratively refines learning paths using the evolving learner profile and feedback from a learner simulator.The simulator role-plays learner responses to support adaptation without relying solely on direct feedback.
- Personalized Resource Delivery: The content creator explores goal-related knowledge, drafts materials around learner preferences, and integrates simulated feedback to finalize resources.It uses web search for retrieval-augmented generation and produces documents and quizzes.
5 EXPERIMENTS
GenMentor was evaluated through automated and human assessments of skill mapping, learning-path scheduling, and content generation. Across these tasks, it generally outperformed comparison methods and showed positive alignment with human judgments.
- Goal-to-skill mapping: GenMentor with GPT-4o achieved Recall 0.67, Precision 0.63, and Goal Alignment 4.28 for goal-to-skill mapping.It outperformed all baselines across the reported metrics.
- Learning-path scheduling: GenMentor with GPT-4o reached Progression 4.56 and Engagement 4.71 for learning-path scheduling.The corresponding Llama scores were 4.09 and 4.32.
- Learning-content generation: GenMentor achieved Personalization 4.17 and Content Quality 4.86 with GPT-4o for generated learning content.With Llama-based models, it achieved Personalization 4.12 and Content Quality 4.62.
- Human validation: Automated scores showed statistically significant positive correlations with human grading for 5 out of 7 metrics.The comparison assessed sampled results.
- Human preference: In pairwise human evaluation, GenMentor was preferred over CoTPrompt for skill gaps and learning paths and over OutlineRAG for learning content.The evaluation used task-specific baselines.
6 END-TO-END HUMAN STUDY
A human study with 20 professional learners evaluated GenMentor’s guidance, personalization, usability, and practical value. Participants generally rated its goal alignment, content, engagement, and interface positively, while dynamic profile matching received a comparatively lower score.
- Questionnaire findings: 18 participants agreed that GenMentor identified skills aligned with their goals, receiving a rating of 4.6 ± 0.8.Its learning path received 4.3 ± 0.8.
- Personalization: Generated content was rated useful and personalized at 4.2 ± 1.0, while dynamic profile matching received 4.1 ± 0.9.Participants described the materials as structured and targeted.
- Usability: Participants rated ease of use at 4.6 ± 0.7 and overall satisfaction at 4.3 ± 0.7.The interface was described as intuitive and easy to navigate.
- Comparative interviews: Compared with MOOCs and search-enhanced chatbots, participants valued GenMentor’s personalized paths, focused guidance, and task-specific learning support.The comparisons emphasized customization, skill-gap identification, and structured progression.
7 CONCLUSION
The paper presents GenMentor as an LLM-powered multi-agent framework for personalized, goal-oriented learning in intelligent tutoring systems. Automated evaluations, human studies, deployment, and user research collectively support its effectiveness in skill-gap identification, learning-path alignment, and tailored resource delivery.
- Conclusion: GenMentor proactively guides learners toward goals through skill-gap identification, adaptive learner profiling, and personalized resource delivery.The conclusion reports effectiveness across automated and human evaluations, deployment, and user studies.
- Conclusion: The framework demonstrates the potential of LLMs to advance personalized and goal-oriented education in professional contexts.This conclusion is framed around GenMentor’s adaptive and learner-centric design.
A APPENDIX
Supplementary resources accompany the paper’s appendix and provide additional materials for examining GenMentor.
- Supplementary resources: Supplementary resources include a demo, prompts, data, and additional insights.They are available through the project’s GitHub repository.
A.1 Details on Goal-to-skill Dataset
The goal-to-skill dataset combines filtered job postings with LLM-generated summaries, reasoning tracks, and required-skill mappings.
- Dataset construction: 58,064 job postings remained after filtering LinkedIn data for postings containing at least 500 words.The source dataset contained more than 0.12 million postings across positions.
- Dataset construction: 10,000 postings were randomly sampled for training and 200 for validation, with balanced representation across positions.
- Skill mapping: GPT4o extracted job summaries and skill lists using chain-of-thought-enabled completion to map duties to required skills and proficiency levels.Each sample contains a job summary, reasoning tracks, and required skills.
A.2 Human Validation on LLM Scoring
Human validation compared automated evaluation scores with expert grading of GenMentor’s goal-to-skill mappings, learning paths, and learning content.
- Validation design: Two experts with extensive Python development experience participated in validating the automated evaluation results.
- Validation design: Twenty randomly sampled data points were evaluated for each output type: goal-to-skill mapping, learning path, and learning content.All sampled outputs concerned Python developer occupations.
- Validation design: The experiment directly compared automated scores with human grading of GenMentor’s outputs.
A.3 Human Preference Evaluation Details
The human preference study compared GenMentor with baseline methods across skill-gap identification, learning-path scheduling, and content generation, using professional learners’ goals and resumes.
- Experiment setup: 5 participants, including 3 software engineers and 2 product managers, supplied resumes and occupation-related learning goals.Each participant repeated the process six times for distinct learning goals.
- Evaluation procedure: Participants reviewed competing outputs side-by-side and selected their preferred skill gaps, learning paths, and learning content.Preferred skill gaps were supplied as inputs for subsequent path generation.
- Results: 22 out of 30 cases favored GenMentor for skill-gap identification over CoTPrompt.This evaluation measured alignment between identified skills and learning goals.
- Results: GenMentor was chosen in 17 cases for learning-path scheduling and in 80% of cases for learning-content generation.The reported results describe goal-oriented paths and personalized, high-quality materials.
- Application workflow: The application lets learners set goals, provide background information, review skill gaps, and adjust or refine personalized learning paths.Path refinement can use updated learner profiles.