Source-linked AI summary
Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems
Weibo Gao, Qi Liu, Linan Yue, Fangzhou Yao, Rui Lv, Zheng Zhang, Hao Wang, Zhenya Huang
TL;DR
Personalized learning needs high-quality response data, yet offline data scarcity and bias can create a gap between offline metrics and online performance. Agent4Edu uses LLM-powered generative agents with learner profile, memory, and action modules to simulate practice and interact with personalized learning environments. The paper reports comprehensive evaluations of agent–human consistency and personalized-learning algorithm use.
Problem
Scarce and potentially biased offline response data can separate personalized-learning metrics from actual online performance, while existing simulators simplify answering processes and depend on real-world data.
Method
Agent4Edu combines LLM-powered generative agents, initialized from learner response data, with profile, memory, and action modules and a personalized learning environment.
Results
Comprehensive evaluations examine consistency between agents and human learners and use simulated data to evaluate and improve personalized learning algorithms.
Takeaways & Limitations
Agent4Edu provides a simulator for generating learner response data and detailed practice behaviors while supporting evaluation and improvement of intelligent educational services.
Takeaways & Limitations
Existing simulation approaches remain limited by simplified answer processes and dependence on high-quality real-world data, while the study assumes problem-solving ability is stable during learning.
Abstract
from arXiv · showhide
Personalized learning represents a promising educational strategy within intelligent educational systems, aiming to enhance learners' practice efficiency. However, the discrepancy between offline metrics and online performance significantly impedes their progress. To address this challenge, we introduce Agent4Edu, a novel personalized learning simulator leveraging recent advancements in human intelligence through large language models (LLMs). Agent4Edu features LLM-powered generative agents equipped with learner profile, memory, and action modules tailored to personalized learning algorithms. The learner profiles are initialized using real-world response data, capturing practice styles and cognitive factors. Inspired by human psychology theory, the memory module records practice facts and high-level summaries, integrating reflection mechanisms. The action module supports various behaviors, including exercise understanding, analysis, and response generation. Each agent can interact with personalized learning algorithms, such as computerized adaptive testing, enabling a multifaceted evaluation and enhancement of customized services. Through a comprehensive assessment, we explore the strengths and weaknesses of Agent4Edu, emphasizing the consistency and discrepancies in responses between agents and human learners. The code, data, and appendix are publicly available at https://github.com/bigdata-ustc/Agent4Edu.
1 Introduction
Agent4Edu addresses the gap between offline response-data metrics and online personalized-learning performance with an LLM-powered learner simulator. Its agents model learner profiles, memory, and practice actions, then interact with personalized learning environments for evaluation and improvement.
- Motivation: Offline response-data scarcity and bias can separate personalized-learning metrics from actual online performance, hindering real-world integration.Existing simulators also simplify answer processes and depend on high-quality real-world data, limiting interpretability and zero-shot generalization.
- Motivation: LLM-powered generative agents offer a route to simulate richer human practice processes with less reliance on real-world data.Their pretrained knowledge, human-like decision capabilities, and in-context learning support zero-shot simulation.
- Approach: Agent4Edu combines an LLM-powered generative agent with a personalized learning environment to simulate learner response data and support algorithm evaluation.The agent is initialized from available learner response data and uses learner profile, memory, and action modules.
- Approach: The simulator models learners’ choosing, understanding, analyzing, and answering behaviors while interacting with personalized learning services.The framework is intended to evaluate and improve intelligent tutoring algorithms, including computerized adaptive testing.
- Evaluation: Comprehensive experiments assess consistency between agents and human learners and evaluate personalized learning algorithms using simulated data.The introduction reports effectiveness from both agent and personalized-learning perspectives.
2 Related Work
Prior learner simulators address the shortage of practice data but often simplify answering processes or depend on real response data. Agent4Edu uses an LLM-powered agent to model student practice and target these limitations.
- Learner Response Data Simulation: Learner simulation seeks to address the shortage of high-quality practice data in intelligent educational systems.Prior work includes rule-based, recurrent-neural-network, and Markov-decision-process approaches.
- Learner Response Data Simulation: Memory-based simulators use manually crafted rules, while EERNN and KES use recurrent neural networks to forecast learner performance.DAISim models long- and short-term question-answering patterns as a Markov decision process.
- Learner Response Data Simulation: Most existing simulators simplify student answering processes and struggle with zero-shot simulation because they rely on real-world data.The paper positions Agent4Edu as an LLM-powered alternative intended to address both limitations.
- Generative Agents in Education: Generative agents have recently been applied to educational settings alongside task-specific and simulation-oriented agent research.The related work situates Agent4Edu within broader LLM-based generative-agent research.
3 Agent4Edu
Agent4Edu is a personalized learning simulator that combines LLM-powered learner agents with a personalized learning environment. Its agents model practice patterns and cognitive factors through profiles, multi-scale memories, and response-generation actions, while supporting external algorithm evaluation.
- 3 Agent4Edu: Agent4Edu combines LLM-powered generative agents with a personalized learning environment to simulate learner responses and evaluate personalized learning algorithms.The framework is designed to support accurate and interpretable evaluation and improvement of algorithms such as computerized adaptive testing.
- 3 Agent4Edu: Each learner agent is initialized from historical response data and uses learner profile, memory, and action modules to generate future responses on unseen exercises.The simulation does not consider repeated submissions because existing personalized learning algorithms usually assume each exercise is submitted once.
- Learner Profile Module: Learner profiles encode explicit practice styles, including activity, practice diversity, success rate, and preference, alongside implicit problem-solving ability and knowledge proficiency.The profile configures ability from observed response records using a psychological IRT model, while knowledge mastery is handled in the memory module.
- Memory Module: The memory module combines factual records, recent short-term memories, and reflective long-term summaries to preserve detailed practice history while extracting higher-level learning insights.Similar factual records are reinforced; reflection summarizes learning status, and compressed summaries filter noise and conserve operational space.
- Memory Module: Long-term memory models reinforcement and forgetting: repeatedly observed records become durable memories, while older records can decay under a thresholded forgetting function.The forgetting process resets a record’s reinforcement frequency when its decay value exceeds the predetermined threshold λ.
- 3 Agent4Edu: Combining specific factual response records with general learning summaries lets agents perceive practice processes, while the platform remains adaptable to external personalized learning algorithms.This combination distinguishes Agent4Edu from traditional simulators described as having short-term but not long-term memory.
4 Experiment
Experiments evaluate Agent4Edu as a learner-response simulator and as a tool for assessing and improving personalized learning algorithms. Results show competitive response prediction, human-like knowledge understanding and summarization, realistic success-rate distributions, and benefits for CAT services, while exercise reasoning and some ablations remain constrained.
- Experimental setup: Agent4Edu uses EduData from 500 Chinese high school students, with 18,045 time-ordered responses across 1,032 exercises and 458 knowledge concepts.Experiments use GPT-3.5-turbo and GPT-4 agents; GPT-4 simulations cover 100 learners because of cost considerations.
- Learner simulation: The evaluation compares Agent4Edu with supervised simulators and knowledge-tracing models using ACC, F1-score, and similarity between simulated and real responses.Learner records are split into 90% training and 10% testing data, while the agent updates profiles and memory from training records.
- Learner simulation: Agent4Edu with GPT-3.5-turbo is competitive with supervised baselines, especially on ACC and F1-score, while GPT-4 performs better on these metrics in the 100-learner exploration.The simulated success-rate distribution also closely approaches the real distribution, capturing learners’ practice patterns.
- Exercise understanding: Agents correctly identify the knowledge concept tested by most exercises, with GPT-4 more accurate than GPT-3.5-turbo under the same 100-learner conditions.The task selects the relevant concept from one correct and two unrelated candidates.
- Human-likeness: LLM judges find agent summarization difficult to distinguish from human responses, but exercise answering remains weaker because solving exercises requires more complex reasoning.The comparison is part of Figure 2’s evaluation of simulated response data and agent behavior.
- Ablation study: Ablations support the contribution of profile, memory, memory enhancement, forgetting, and reflection modules to response prediction, whereas their effect on knowledge prediction is not significant.The paper attributes the limited knowledge-prediction impact to GPT-3.5-turbo’s existing exercise knowledge.
- Personalized learning: Agents report higher satisfaction for MAAT among CAT strategies, and retraining IRT with Agent4Edu-generated data improves CAT services, including in zero-shot settings.The CAT evaluation uses satisfaction, appropriateness of difficulty, and learning gain metrics; Table 4 reports retrained IRT performance on EduData and EduData+.
5 Conclusion
Agent4Edu uses LLM-powered generative agents to simulate learner response data and detailed problem-solving behaviors for personalized learning. The agents support evaluation of intelligent services while revealing consistencies and discrepancies between agent and learner practice behaviors.
- Agent4Edu simulates learner response data and detailed problem-solving behaviors with LLM-powered generative agents.The agents are designed to choose, understand, analyze, and answer exercises.
- Each agent includes learner Profile, Memory, and Action modules tailored to personalized learning scenarios.
- The agents can interact with personalized learning environments to evaluate and enhance intelligent educational services.
- The evaluation examines both the strengths and weaknesses of Agent4Edu and the consistency and discrepancies between agent and learner practice behaviors.
A.1 Split of Profile Traits
Learners are divided into categorical profile groups using practice-style traits and cognitive factors. Activity and diversity are split at the median, while success rate and ability are divided into three equal-proportion categories.
- Learning activity and diversity are categorized as high or low according to whether each trait is above or below the learner-wide median.
- Success rate and ability are categorized as low, medium, or high in equal proportions based on ascending values across learners.
A.2 Split of Knowledge Proficiency
Knowledge proficiency is treated as a dynamic cognitive factor and categorized over time using response records analyzed by DNeuralCDM.
- DNeuralCDM tracks knowledge proficiency over time by analyzing each learner’s response records.
- At each step, knowledge proficiency is divided into low, medium, or high categories in equal proportions using ascending values across learners.
A.3 The Workflow of Initializing Agents based on Learner Response Data
Agent initialization begins with empty profile and memory components, then processes each learner’s response records to construct profiles, memories, proficiency estimates, and reflections.
- The workflow initializes profile, factual memory, short-term memory, and long-term memory as empty.
- For each learner, the system generates a learner profile from that learner’s data and adds response records to factual memory.
- Recent records update short-term memory, while reinforced factual-memory entries are retrieved as important facts.
- The workflow generates knowledge proficiency with DNeuralCDM and summarizes reflections during agent initialization.
B.1 IRT Model
IRT models learner ability from response data by jointly fitting ability and exercise-difficulty parameters. Agent4Edu trains and selects the model using validation accuracy, then uses the optimized ability estimate to complete each learner profile.
- Basic Description: IRT models learner ability by analyzing response data and assuming ability is stable and positively correlated with exercise responses.The objective fits learner response records to optimize ability parameters.
- Basic Description: The model predicts the probability that learner u correctly answers exercise e_i from ability θ_u and difficulty b_ei, optimized with binary cross-entropy.The loss minimizes the gap between predicted probabilities and ground-truth responses.
- Training: The IRT model is trained on training data, and the version with the highest validation ACC is selected as the tool.
- Inference: At inference, the optimized model provides each learner’s ability parameter θ_u, which is divided into levels for the learner profile.The level mapping is detailed in Appendix A.
B.2 DNeuralCDM
DNeuralCDM traces learner knowledge proficiency sequentially from chronological practice records, unlike the nonsequential IRT formulation. Agent4Edu uses its predicted concept-level proficiency in learner profiles and combines it with structured generative-agent prompts, memory, reflection, and action tasks.
- Basic Description: DNeuralCDM processes each learner’s response record chronologically to model the evolution of knowledge proficiency during practice.This distinguishes it from IRT, which analyzes exercise data nonsequentially.
- Basic Description: The model predicts performance at time t+1 from the learner’s knowledge state h_u,t and actual performance y_u,e_t at time t.The knowledge state contains proficiency values over the set of exercise-covered concepts.
- Inference: The optimized DNeuralCDM model supplies concept-level proficiency h_u,t, while the agent retrieves the proficiency for the concept tested by the current exercise.The retrieved proficiency is partitioned into levels for the learner profile.
- Learner Profile Generation: Learner profiles encode activity, knowledge diversity, preferred concepts, success rate, and analytical ability for the generative agent.The profile prompt presents these attributes as the agent’s learner profile.
- Memory Module: The memory module reinforces related factual records through a knowledge concept graph and uses reflection to summarize learning status from profile and memory information.Corrective feedback is incorporated when available.
- Taking Actions: The action workflow asks the agent whether to attempt the exercise, identify its tested concept, analyze it, and produce an answer in a fixed output format.The prompt requires four sequential tasks and specifies the exact response structure.
Appendix D. A Case Study of Agent4Edu for CAT
The CAT case study shows Agent4Edu’s learner agent receiving a profile, memories, and a recommended exercise before completing four action tasks. After an initial concept error, corrective reflection updates the agent’s learning status and the subsequent response identifies the correct concept.
- Workflow: The agent receives its learner profile, short-term memory, long-term memory, and recommended exercise before completing the action workflow.The workflow also updates learning status through reflection when necessary.
- Learner Context: In the example, the learner profile reports low activity, low knowledge diversity, low success rate, common analytical ability, and Geometry as the most-practiced concept.The long-term memory records weak performance on function-related exercises and poor Function proficiency.
- Corrective Reflection: The initial action selects Algorithm as the tested concept, but corrective reflection identifies Function as the true concept and asks the agent to revise its learning status.The reflection links the error to confusing the tested concept with the problem’s mathematical content.
- Updated Action: After reflection, the agent selects Function and correctly reasons that substituting x = 1 yields an impossible equality, so the graph does not always pass through (1, a).
- CAT Interaction: CAT recommends one exercise at each practice step using the learner agent’s most recent, continuously updated response data.