Source-linked AI summary
Simulating Classroom Education with LLM-Empowered Agents
Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianxiao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, Lei Hou, Juanzi Li
TL;DR
Existing LLM educational systems have mainly addressed isolated tasks or classroom roles, leaving multi-agent classroom simulation with real users insufficiently explored. SimClass combines role-specialized agents with a classroom control mechanism and evaluates the framework in two real courses. The experiments show dynamic teacher-student and student-student interactions, improved learning outcomes associated with increased interaction, and emergent collaborative group behaviors.
Problem
Multi-agent classroom simulation with real user participation remains insufficiently explored, including its interaction quality, learning experience, and emergent group behavior.
Method
SimClass uses representative class roles, a novel class control mechanism, real-course experiments, Flanders and Community of Inquiry analyses, and qualitative group-behavior analysis.
Results
SimClass creates dynamic classrooms with lively teacher-student and student-student interactions, improved learning outcomes associated with increased interaction, and collaborative agent behaviors.
Takeaways & Limitations
LLM-based multi-agent systems show potential for simulating classroom environments for educational purposes.
Takeaways & Limitations
The experiments cover limited courses, models, quizzes, and users, so findings may be constrained by the selected courses, model, and recruited users.
Abstract
from arXiv · showhide
Large language models (LLMs) have been applied across various intelligent educational tasks to assist teaching. While preliminary studies have focused on task-specific, independent LLM-empowered agents, the potential of LLMs within a multi-agent collaborative framework for classroom simulation with real user participation remains unexplored. In this work, we propose SimClass, a multi-agent classroom simulation teaching framework. We recognize representative class roles and introduce a novel class control mechanism for automatic classroom teaching, and conduct user experiments in two real-world courses. Using the Flanders Interactive Analysis System and Community of Inquiry theoretical frameworks from educational analysis, we demonstrate that LLMs can simulate a dynamic learning environment for users with active teacher-student and student-student interactions. We also observe group behaviors among agents in SimClass, where agents collaborate to create enlivening interactions in classrooms to improve user learning process. We hope this work pioneers the application of LLM-empowered multi-agent systems in virtual classroom teaching.
1 Introduction
SimClass addresses the open question of whether LLM-based multi-agent systems can simulate classrooms with real user participation. It introduces role-based classroom control and evaluates interaction, learning experience, and emergent group behavior in real courses.
- Multi-agent classroom simulation with real user participation remains an open research problem spanning interaction quality, learning experience, and spontaneous group behavior.
- SimClass combines representative class roles with a novel control mechanism for automated classroom teaching.The framework is designed to coordinate classroom interactions while users participate in real time.
- The study conducts real-world experiments in two courses with more than 400 students and additional ablation experiments involving 48 students.
- The evaluation uses the Flanders Interaction Analysis System, Community of Inquiry theory, and qualitative analysis of agent group behaviors.
- SimClass produces lively teacher-student and student-student interactions, improved learning outcomes associated with increased interaction, and collaborative agent behaviors.Observed behaviors include collaborative teaching, discussions, emotional companionship, and discipline management.
2 Related Work
Prior educational applications of LLMs largely target specific tasks or individual classroom roles. Related work has begun exploring simulated student discussions, but without real user participation in the classroom.
- Research has applied LLMs to educational downstream tasks and classroom assistance, including personalized recommendations, teaching assistants, and AI teachers.
- LLM-based agents have also been used to simulate human decision-making and behavior in social and psychological research.
- Existing classroom applications replace selected roles by using LLMs as students for teacher training or as instructors for student teaching.
- Yue et al. explored multiple student agents for discussion assistance, but that work did not involve real users.
3 SimClass
SimClass constructs a simulated classroom from role-specialized teaching and classmate agents coordinated by a Session Controller. The controller observes classroom state and selects functions and speakers to maintain course flow.
- 3.1 Overview: The framework organizes classroom behavior around teaching and initiation, in-depth discussion, emotional companionship, and related pedagogical functions.
- 3.2 Class Role Agentization: Class roles are implemented by prompting LLMs with role descriptions, with designs informed by experienced teaching practitioners.
- 3.2 Class Role Agentization: Teaching agents deliver, supplement, and manage instruction, while classmate agents provide distinct peer personalities and classroom participation.The teacher presents scripts or answers questions; the assistant supplements information and maintains continuity; four typical classmates are initialized.
- 3.2 Class Role Agentization: Representative classmates include a Class Clown that enlivens the atmosphere and a Deep Thinker that raises challenging topics.
- 3.3 Classroom Session Controller: The Session Controller replaces a strict workflow with state-based regulation through a Class State Receptor, Function Executor, and Manager Agent.It is designed for dynamic group chat in which agents determine appropriate speaking times on the fly.
- 3.3 Classroom Session Controller: Tutoring functions are restricted to the teacher, while interacting functions can be performed by any agent and extended with new functions.
- 3.3 Classroom Session Controller: The manager observes class state, selects the next action and speaker, and coordinates interaction after user input or a waiting period.
4 Experiments
The experiments evaluate SimClass across two courses using interaction, learning-outcome, ablation, and agent-behavior analyses. Results indicate dynamic teacher-student and student-student interaction, positive associations between engagement and quiz scores, and improved user experience with classmate agents.
- Experimental Setup: SimClass was evaluated in two courses, TAGI on AI knowledge acquisition and HSU on academic and interpersonal skills development.TAGI comprised six chapters, while HSU comprised seven chapters.
- Experimental Setup: Over 400 university students used the online system, with 118 completing all chapters and 48 additional students participating in the ablation study.The completed-chapter participants included 77 in TAGI and 41 in HSU.
- Interaction Analysis: FIAS analysis found frequent teacher-student and student-student interactions, with speaking balance comparable to traditional classrooms and relatively high student-initiated interaction.The relatively low interaction-direction ratio was partly attributed to script-based teaching.
- Learning Outcome Analysis: 0.68 was the final exam score in TAGI, closely aligning with average quiz scores and indicating consistent retention of course material.Quiz scores were normalized with full marks equal to 1.
- Statistical Results: r = 0.3345 and r = 0.3349 showed significant positive correlations between normalized quiz scores and message length and message number, respectively.Both correlations had p < 0.001, associating more active messaging with higher average quiz scores.
- Ablation Analysis: Removing classmate agents significantly reduced user speech length, while removing interaction reduced experience across all three Community of Inquiry metrics.Classmate agents enhanced Cognitive Presence and Social Presence, while all systems retained good Teaching Presence.
5 Conclusion
SimClass is an LLM-based multi-agent classroom framework evaluated with real users across two courses. The experiments report dynamic collaborative teaching, increased engagement and interaction, and better learning outcomes associated with greater system interaction.
- Conclusion: SimClass is a multi-agent classroom framework that uses LLMs and collaborative agents to support dynamic teaching environments.The framework was evaluated across two courses with real users.
- Conclusion: Increased interaction with SimClass was associated with better learning outcomes, while the multi-agent setup encouraged greater student engagement.The conclusion presents these findings as results from experiments across two courses.
- Conclusion: The authors position SimClass as a contribution to further exploration of LLM-empowered education systems.The stated audience includes researchers, practitioners, and pedagogues.
6 Limitations
The initial SimClass system has limitations in latency, teacher preparation requirements, teaching-function coverage, and experimental scope. Its evaluation is also concentrated on university-level general courses with relatively homogeneous participants.
- LLM-based manager and class-role agents introduce response delays when multiple agents participate, potentially affecting user experience.The authors suggest higher-performance models as one future remedy.
- SimClass requires teacher-designed slide-script pairs, limiting automation of course preparation.Future work could automate this process.
- The framework includes a limited set of teaching functions, restricting its performance and interaction diversity.The authors propose adding more classroom interactions and technologies such as RAG, question generation, and knowledge tracing.
- The experiments used limited courses, models, quizzes, and users, so findings may depend on the selected settings and recruited participants.The authors call for broader and more diverse evaluations.
- The evaluation focused on university-level general courses and college students, creating participant-group homogeneity that may introduce bias.Future work aims to include broader users, including marginalized groups and people with learning disabilities.
7 Ethical Considerations
The study addresses privacy, informed consent, and risks from AI-generated misinformation, while recognizing that simulated teachers and classmates cannot fully substitute for human educational relationships and skills development.
- User interaction data were anonymized, participants gave informed consent, and they were briefed about the simulation, generated content, and data collection.Participants also received compensation.
- Potential hallucinations and incorrect information require careful consideration and thorough evaluation before real-world deployment.
- Multi-agent teaching systems may change how students perceive teachers because AI teachers emphasize knowledge delivery rather than adherence to human social norms.The authors state that this may bias development of students’ abilities.
- Classroom agents cannot replace human teachers’ role in comprehensive skills or real students’ role in social skills, group identity, and self-esteem.The authors call for more interdisciplinary research guided by psychology and education.
A Experiment Details
The supplementary materials document the online system, model choices, FIAS comparison procedure, labeling validation, prompts, and instruments used to assess learning and classroom experience.
- Implementation: The online system used GLM-4, ablation systems used gpt-4-vision-preview, and FIAS classification used gpt-4-turbo with default temperatures.
- FIAS: FIAS comparisons removed the silence category from both SimClass and human classroom data because silence was difficult to define and measure consistently.
- Label validation: 92% labeling accuracy was obtained when GPT-4 labels were compared with expert labels on 100 sampled data points.The eight disagreements were judged uncertain during human labeling.
- Materials: The appendix provides agent prompts and detailed designs for quizzes, surveys, and quality tests.
- Evaluation instruments: TAGI quizzes assessed chapter concept understanding with multiple-choice questions requiring all correct answers, while both courses used easier quality tests.Only TAGI included quizzes; quality tests screened participation and were verified by subject-matter experts.
- Evaluation instruments: Table 6 presents example TAGI quiz questions and marks the correct answers in bold.
- Evaluation instruments: CoI surveys evaluated cognitive, teaching, and social presence using rating guidelines intended to support consistent feedback across users.
C Supplementary Experiment Results
The user-focused FIAS matrix shows frequent interactions among users, teachers, and peers, especially question-asking and discussions with classmates.
- Users most frequently asked teachers questions and engaged in discussions with classmates.These patterns correspond to interactions (5, 8), (5, 9), and (9, 9) in the FIAS matrix.
C.2 Results of Learning Outcomes of HSU
HSU learning outcomes were examined qualitatively through anonymized student self-learning reports. The cases describe changes in academic objective setting, problem solving, and personal development.
- HSU used qualitative analysis of anonymized self-learning reports because the course focused on developing university-level skills.
- Setting Academic Development Objectives: Students described using an innovative-potential self-assessment to reduce anxiety about beginning scientific research.
- Problem Solving: One student reported focused work time increasing from 5.5 hours to 9 hours after adopting an efficiency journal.
- Personal Development: A student reported joining a professor’s research group and applying structured time-management techniques to manage daily work.
C.3 Statistical Results of Ablation Systems
The ablation systems were statistically compared using the output lengths of agents and users. The reported setup also specifies class roles and prompt templates for the compared systems.
- Table 10 reports average output lengths for users and agents across the ablation systems.
- The output-length statistics count words and distinguish classmate agents and interactions using abbreviated labels.