PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems

Weibo Gao, Qi Liu, Linan Yue, Fangzhou Yao, Rui Lv, Zheng Zhang, Hao Wang, Zhenya Huang

arXiv:2501.10332v2cs.CYcs.AI

TL;DR

Personalized-learning research faces scarce offline response data and a gap between offline metrics and online performance. Agent4Edu uses LLM-powered generative agents to simulate learner responses and interact with personalized-learning algorithms; experiments report its effectiveness.

  • Problem

    Scarce offline response data and potential bias in its relationship to online practice leave a gap between offline metrics and real-world performance.

    Source: §1 · p. 1

  • Method

    Agent4Edu combines LLM-powered generative agents that model learner practice patterns and cognitive preferences with a personalized-learning environment for algorithm evaluation.

    Source: §3 · p. 3

  • Results

    Experiments assess agent–human consistency and computerized adaptive testing, with results demonstrating Agent4Edu’s effectiveness.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support using generative agents and simulated data to evaluate and improve personalized-learning algorithms, including computerized adaptive testing.

    Source: §1 · p. 2

    Existing simulators often predict only correct or incorrect answers rather than detailed problem-solving processes, and require high-quality real-world response data.

    Source: §1 · p. 1

Abstract

from arXiv · show

Personalized learning represents a promising educational strategy within intelligent educational systems, aiming to enhance learners' practice efficiency. However, the discrepancy between offline metrics and online performance significantly impedes their progress. To address this challenge, we introduce Agent4Edu, a novel personalized learning simulator leveraging recent advancements in human intelligence through large language models (LLMs). Agent4Edu features LLM-powered generative agents equipped with learner profile, memory, and action modules tailored to personalized learning algorithms. The learner profiles are initialized using real-world response data, capturing practice styles and cognitive factors. Inspired by human psychology theory, the memory module records practice facts and high-level summaries, integrating reflection mechanisms. The action module supports various behaviors, including exercise understanding, analysis, and response generation. Each agent can interact with personalized learning algorithms, such as computerized adaptive testing, enabling a multifaceted evaluation and enhancement of customized services. Through a comprehensive assessment, we explore the strengths and weaknesses of Agent4Edu, emphasizing the consistency and discrepancies in responses between agents and human learners. The code, data, and appendix are publicly available at https://github.com/bigdata-ustc/Agent4Edu.

1 Introduction

Agent4Edu addresses the gap between offline metrics and online performance by using LLM-powered generative agents to simulate learner response data and interact with personalized learning algorithms. Its design targets limitations in existing simulators, which often omit detailed problem-solving processes and depend on real response data.

Introduction

  • Existing simulators often predict correctness without modeling detailed answer processes and require high-quality real-world response data, limiting reliability and zero-shot generalization (Piech et al. 2015).The paper identifies these as simplified simulation and real-data dependency limitations.Source: §1 · p. 1
  • LLM agents offer a potential alternative because their human-like capabilities and in-context learning support simulation of intricate practice processes with minimal real-world data (Wang et al. 2023c).The introduction situates this potential alongside prior generative-agent simulations of social behavior and software development (Park et al. 2023; Qian et al. 2023).Source: §1 · p. 2
  • Each agent combines learner profiles, reflective memory, and action modules to represent practice styles and cognitive factors and generate interpretable exercise-solving responses.The modules support tracking learning status, knowledge proficiency, and forgetting; tools include IRT for ability assessment and DNeuralCDM for knowledge tracing.Source: §1 · p. 2
  • Agent4Edu uses LLM-powered generative agents in configurable personalized learning environments to simulate learner responses and practice while interacting with tutoring algorithms.The paper presents it as the first generative-agent simulator focused specifically on educational scenarios and individual learners.Source: §1 · p. 2
  • The evaluation examines agent–human response consistency and computerized adaptive testing, reporting that Agent4Edu outperforms existing learner simulation methods and supports evaluation and improvement of personalized learning algorithms.Experiments assess both agent behavior and learning-algorithm outcomes using generative agents and simulated data.Source: §1 · p. 2

2 Related Work

Prior learner simulators and educational agents motivate Agent4Edu’s LLM-based practice simulation, while CAT provides a setting to evaluate generated data and explore its use in improving personalized services.

Learner Response Data Simulation

  • Earlier simulators use memory rules, RNNs, or Markov decision processes, but often simplify answering or rely on data, motivating LLM-powered practice simulation.The cited approaches include Memory-based (Reddy, Levine, and Dragan 2017), EERNN, KES (Liu et al. 2019), and DAISim.Source: §2 · p. 2

Personalized Learning Services

  • Because CAT models need high-quality practice data to estimate learner proficiency, Agent4Edu uses CAT to assess generated data and investigate whether simulated data can enhance CAT models.CAT is the representative personalized learning service selected for the study and experiments.Source: §2 · p. 2

LLM-based Agents

  • Unlike EduAgent (Xu, Zhang, and Qin 2024), which uses expert-annotated factors and omits exercise understanding and analysis, Agent4Edu extracts factors from data, captures practice styles, and simulates those processes.Other educational LLM-agent work includes SocraticLM, which uses a thought-provoking teaching paradigm for active problem-solving.Source: §2 · pp. 2–3

3 Agent4Edu

Agent4Edu combines LLM-powered learner agents with an interactive personalized-learning environment to simulate responses on unseen exercises from learners’ historical practice. Its agents represent learner characteristics, update memories as practice proceeds, and emulate exercise-solving behavior.

3 Agent4Edu

  • Agent4Edu pairs learner-simulating LLM agents with a personalized-learning environment designed to evaluate and improve algorithms such as computerized adaptive testing.The environment supports interaction with personalized learning algorithms; the framework is shown in Figure 1.Source: §3 · p. 3

3.1 Task Formulation

  • The simulator aims to generate each learner’s responses to unseen exercises from historical response data, assuming each exercise is submitted only once.Each response record includes the practiced exercise, its textual and knowledge-concept information, and the learner’s binary answer.Source: §3.1 · p. 3

3.2 LLM-powered Agent

  • Each learner has an LLM-based agent with profile, memory, and action modules for representing practice patterns and cognitive factors and simulating practice behavior.The profile captures relatively stable learner characteristics, while memory stores practice records and summarizes higher-level learning information.Source: §3.2 · p. 3

Learner Profile Module

  • Profiles combine observed practice styles with inferred problem-solving ability, while treating knowledge proficiency as a changing factor tracked in memory.Practice styles include activity, diversity, success rate, and preference; ability is inferred from response records using a trained IRT tool. Feature values are tiered, and personal identifiers are anonymized.Source: Learner Profile Module · pp. 3–4

Memory Module

  • Memory combines factual records, recent short-term observations, and long-term reinforced facts, learning summaries, and evolving knowledge proficiency.Similar knowledge concepts reinforce factual records; reflection summarizes learning, and a forgetting function can remove long-term facts when a threshold is exceeded.Source: Memory Module · pp. 4–5

Action Module

  • Agents decide whether to practice recommended exercises, identify their knowledge concepts, reason through answers, predict correctness, and trigger corrective reflection when understanding or responses mismatch.If standard answers are available, a scoring program can assess answer correctness directly.Source: Action Module · p. 5

3.3 Personalized Learning Scenarios

  • The standalone learning environment supports integrated personalized algorithms and an open interface for researchers and practitioners to add external algorithms.Experiments use computerized adaptive testing strategies, and the platform is intended to support evaluation and future collection of learner response data.Source: §3.3 · pp. 5–6

4 Experiment

Agent4Edu generates learner-response data that is competitive with supervised simulation methods and can improve CAT services. The experiments also assess agents’ exercise understanding, zero-shot behavior, and evaluations of personalized learning strategies.

4 Experiment

  • Experiments use EduData, containing 18,045 ordered responses from 500 Chinese high school students, and generally run agents with GPT-3.5-turbo.The dataset covers mathematics and physics, 1,032 exercises, and 458 knowledge concepts; GPT-4 is used for only 100 learners because of cost, with temperature set to 0.Source: §4 · p. 6

4.1 LLM-based Agent Simulation Evaluation

  • Agent4Edu (GPT-3.5-turbo) is competitive with supervised baselines, particularly on ACC and F1-score, and its simulated success-rate distribution captures learner practice patterns.The comparison includes supervised simulators and knowledge-tracing models; the success-rate analysis compares agent-simulated data with the ground truth.Source: Table 1 · p. 6, §4.1 · p. 6, Figure 2 · p. 7
  • In zero-shot simulation, agent summaries are difficult to distinguish from human responses, but exercise answers are less human-like because solving exercises requires complex reasoning.Ten randomly profiled agents answer ten selected exercises without learner response data; summary reflection remains enabled, while corrective reflection and tools are disabled.Source: Figure 2 · p. 7, §4.1 · p. 7
  • Removing the profile, memory, memory enhancement, memory forgetting, or reflection component reduces response-prediction performance, while effects on knowledge prediction are not significant.The authors attribute the limited knowledge-prediction effect to GPT-3.5-turbo’s existing knowledge being sufficient for exercise comprehension.Source: Figure 2 · p. 7, §4.1 · p. 7
  • All tested agents identify the exercise’s knowledge concept correctly in most cases, and GPT-4 is more accurate than GPT-3.5-turbo under the 100-learner condition.Knowledge prediction is evaluated by selecting the correct concept from one relevant and two unrelated candidates.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2 Personalized Learning

  • Agents report higher satisfaction with MAAT, while FSI emphasizes moderately difficult exercises likely to provide gain.The agents evaluate CAT strategies on satisfaction, appropriateness of difficulty, and whether learning gains occurred.Source: Table 3 · p. 8, §4.2 · p. 8
  • Agent4Edu-generated EduData+ enhances CAT services when used to retrain IRT models.The evaluation uses original EduData and EduData+ to assess FSI, KLI, and MAAT after recommending 5 and 10 test exercises per learner.Source: Table 4 · p. 8, §4.2 · p. 8

5 Conclusion · Appendix

Agent4Edu uses LLM-powered generative agents to simulate learner responses and problem-solving behaviors, with modules tailored to personalized learning. Its evaluation examines where agent practice behavior aligns with or differs from that of learners.

5 Conclusion

  • Agent4Edu is a personalized learning simulator that generates learner response data and detailed problem-solving behaviors.The agents use Profile, Memory, and Action modules tailored to personalized learning scenarios.Source: §5 · p. 8
  • Its agents mimic human-like exercise selection, understanding, analysis, and answering, and predict learners’ future responses.Source: §5 · p. 8
  • Generative agents can interact with personalized learning environments to evaluate and enhance intelligent services.Source: §5 · p. 8
  • The evaluation examines Agent4Edu’s strengths and weaknesses, including consistency and discrepancies between agent and learner practice behaviors.Source: §5 · p. 8
  • Future work will investigate cooperation among multiple learner agents and multimodal practice solutions using generative agents.Source: §5 · p. 8

Appendix A. Basic Setups · A.1 Split of Profile Traits

Learners are categorized by the strength of practice-style and cognitive-factor traits. Learning activity and diversity use median-based high/low splits, while success rate and ability use three equal-sized levels.

A.1 Split of Profile Traits

  • Learning activity and diversity are classified as high or low according to whether each trait is above or below the median across learners.Source: §A.1 · p. 11
  • Success rate and ability are divided into low, medium, and high categories in equal proportions, based on learners’ ascending trait values.Source: §A.1 · p. 11

A.2 Split of Knowledge Proficiency

Learner knowledge proficiency is tracked over time from response records and split into three equally sized levels based on proficiency rankings across learners.

A.2 Split of Knowledge Proficiency

  • DNeuralCDM tracks dynamic knowledge proficiency from each response record, then categorizes learners as high, medium, or low in equal proportions by ascending proficiency.Categories are assigned at each step across all learners.Source: §A.2 · p. 11

A.3 The Workflow of Initializing Agents based on Learner Response Data · Appendix B. Tools

The workflow initializes LLM-based agents from each learner’s response data, building profile and memory state, estimating proficiency, and iteratively prompting and correcting agent actions.

A.3 The Workflow of Initializing Agents based on Learner Response Data

  • For each learner, the workflow generates a profile from that learner’s data and processes the learner’s response records in sequence.Each record is added to factual memory and updates a counter for the corresponding item.Source: §A.3 · p. 11
  • Recent records update short-term memory, while reinforced facts are retrieved from factual memory as important facts.Source: §A.3 · p. 11
  • The workflow uses the DNeuralCDM tool to generate knowledge proficiency and summarizes reflections to produce the learner’s learning status.Source: §A.3 · p. 11
  • It generates a prompt for the agent to act on the next exercise, and updates learning status and prompts again if corrective reflection is required.Source: §A.3 · p. 11
  • The workflow repeats record processing for each learner, with agent actions continuing until the response requiring correction is made correctly.Source: §A.3 · p. 11
  • The initialization begins with profile, factual memory, short-term memory, and long-term memory empty.Source: §A.3 · p. 11

B.1 IRT Model

IRT models learner ability from response data under the assumption that ability is stable and positively correlated with exercise responses. Agent4Edu uses the fitted ability parameter to complete each learner profile.

Basic Description

  • IRT models learner ability as stable and positively correlated with exercise responses, fitting response data to estimate learner ability and exercise difficulty.The objective minimizes the gap between predicted and ground-truth response probabilities, usually with binary cross-entropy; joint training optimizes learner ability θu.Source: §B.1 · p. 11

Training

  • The IRT model is trained on the specified training data, and the version with the highest validation ACC is selected as the tool.The training data are detailed in the Learner Simulation Evaluation section in 4.1.Source: §B.1 · p. 11

Inference

  • At inference, each agent uses the optimized IRT model as an external tool to obtain learner ability θu, then divides it into levels for the learner profile.The profile-level details are provided in Appendix A.Source: §B.1 · p. 11

B.2 DNeuralCDM · Appendix C. Prompts

DNeuralCDM estimates evolving learner proficiency from sequential response histories, while Appendix C specifies prompts that condition agents’ profiles, memories, actions, and reflections. Placeholder definitions supply the learner, proficiency, and exercise information used by those prompts.

B.2 DNeuralCDM

  • DNeuralCDM processes each learner’s responses chronologically to estimate concept proficiency and predict the next response, with its highest-validation-ACC version used to infer proficiency for learner profiles.At inference, the agent retrieves proficiency for the concept tested by the current exercise and partitions it into profile levels.Source: §B.2 · pp. 11–12

C.1 Prompt Designing

  • The profile prompt frames the agent as a self-directed high school learner and supplies activity, knowledge diversity, practice preference, success rate, and analytical ability.These characteristics are inserted into the learner profile as placeholders.Source: §C.1 · p. 12
  • Factual-memory concept relations are judged with an RCD-based knowledge-concept graph, while reflection summarizes learning status from profile, memories, and prior status in at most 500 words.Corrective feedback, when present, is placed before the reflection instruction.Source: §C.1 · p. 12
  • The action prompt presents profile, recent facts, reinforced facts, knowledge proficiency, learning status, and exercise content, then requires four ordered tasks: attempt decision, concept selection, solution and answer, and success prediction.All four tasks are executed even if the agent declines the exercise, and the response must follow a specified output format.Source: §C.1 · pp. 12–13
  • Corrective reflection supplies feedback for wrong concept predictions or mismatches between predicted and actual correctness, then requests an updated learning-status summary based on profile and memory.The reflection is limited to 500 words and should contain no other information.Source: §C.1 · p. 13

C.2 Placeholders in Prompts

  • Placeholder examples encode activity and diversity as high or low, identify the most-practiced concept, and express success rate and IRT-derived ability in ordered categories.Success rate uses high, medium, or low; ability uses good, common, or poor.Source: §C.2 · p. 13
  • DNeuralCDM placeholders provide proficiency level and numeric value, with an unknown option, while memory contributes the latest learning status and prompts distinguish true from predicted concepts.Exercise placeholders can include its text, optional choices, reference answer, and reference analysis; concept selection offers the true concept plus two sampled distractors.Source: §C.2 · pp. 13–14
  • Exercise-content placeholders cover text, optional options, answer, and analysis, and concept-choice placeholders comprise the true concept plus two sampled distractors.These fields populate the action prompt’s exercise information and three concept options.Source: §C.2 · p. 14

Appendix D. A Case Study of Agent4Edu for CAT

The case study shows agentu interacting with CAT through a profile-, memory-, and exercise-informed sequence of four response tasks, followed by reflection that corrects a mistaken concept classification. CAT recommends one exercise per practice step using the agent’s updated response data.

Appendix D. A Case Study of Agent4Edu for CAT

  • The agent receives its profile, short- and long-term memories, and a CAT-recommended exercise, then decides whether to attempt it, classifies its concept, solves it, and estimates whether it was correct.Reflection is invoked when necessary to update learning status.Source: Appendix D · pp. 14–15
  • Agentu correctly answers the function-graph question but initially labels its concept Algorithm; corrective reflection identifies Function, and its subsequent response uses that classification.The reflection says the agent should focus more carefully on the tested concept and strengthen its understanding of function graphs and substitution.Source: Appendix D · pp. 15–16
  • CAT recommends one exercise to agentu at each practice step based on its current response data, which is updated with the agent’s latest simulated responses.Source: Appendix D · pp. 14, 16

Related papers

Loading 2501.10332v2…