Source-linked AI summary
Platform-Independent and Curriculum-Oriented Intelligent Assistant for Higher Education
Ramteja Sajja, Yusuf Sermet, David Cwiertny, Ibrahim Demir
TL;DR
Students can miss course help because of scheduling conflicts and other barriers, while existing educational chatbot work is described as limited in scope. This paper presents a document-based GPT-3 assistant for course questions and evaluates its accuracy and performance, while identifying future work to broaden testing and improve the system.
Problem
Students may miss timely course assistance because of scheduling conflicts, insecurity, or shyness, and prior chatbot studies are described as limited-scope case studies.
Source: §1 · p. 2, §1.1 · p. 4Method
The authors develop a GPT-3-based framework that generates course-specific assistants from documents and uses a search model plus a completion model to answer questions.
Source: §1 · p. 3, §2.3.1 · p. 8Results
The evaluation describes accuracy-focused answering with a “Response not found” fallback and identifies course-information errors involving missing credit-hour context and inconsistent answers to similar questions.
Source: §3.2 · p. 15, §3.2.2 · p. 17Takeaways & Limitations
The framework is designed to provide course-logistics answers across communication platforms and delegate repetitive instructor duties to virtual assistants.
Source: §1 · p. 3, §4 · p. 17Future work includes a semester-long case study across multiple fields and improving accuracy through fine-tuning on bigger datasets.
Source: §4 · p. 18
Abstract
from arXiv · showhide
Miscommunication and communication challenges between instructors and students represents one of the primary barriers to post-secondary learning. Students often avoid or miss opportunities to ask questions during office hours due to insecurities or scheduling conflicts. Moreover, students need to work at their own pace to have the freedom and time for the self-contemplation needed to build conceptual understanding and develop creative thinking skills. To eliminate barriers to student engagement, academic institutions need to redefine their fundamental approach to education by proposing flexible educational pathways that recognize continuous learning. To this end, we developed an AI-augmented intelligent educational assistance framework based on a power language model (i.e., GPT-3) that automatically generates course-specific intelligent assistants regardless of discipline or academic level. The virtual intelligent teaching assistant (TA) system will serve as a voice-enabled helper capable of answering course-specific questions concerning curriculum, logistics and course policies. It is envisioned to improve access to course-related information for the students and reduce logistical workload for the instructors and TAs. Its GPT-3-based knowledge discovery component as well as the generalized system architecture is presented accompanied by a methodical evaluation of the system accuracy and performance.
1. Introduction
Students can miss timely course help because office hours and instructor availability do not fit their schedules or personal barriers. The paper proposes a document-based, course-specific assistant and positions it as a flexible alternative to limited-scope educational chatbots.
1. Introduction
- Students benefit from time and freedom for self-contemplation, which supports conceptual understanding, motivation, and creative thinking.The introduction links learning at one’s own pace to increased motivation and creative thinking skills.Source: §1 · pp. 1–2
- Students may miss help because of scheduling conflicts, insecurity, shyness, or assistant unavailability, while many study outside regular work hours.The introduction notes that delayed assistance can leave students discouraged or feeling stuck.Source: §1 · p. 2
- The framework automatically generates course-specific assistants from documents such as syllabi, answering questions about course scope, logistics, and policies.Students can interact in natural language through web platforms and messaging applications.Source: §1 · p. 3
1.1. Related Work
- Earlier work includes educational assistants such as Jill Watson and Pounce, alongside chatbots used in other instructional and campus contexts.The cited Pounce study reports disproportionate benefits for first-generation and underrepresented groups and a decreased graduation-rate gap among demographics.Source: §1.1 · pp. 3–4
- Prior educational chatbot studies are described as limited-scope case studies, while the paper identifies no complete multidisciplinary solution for transforming teaching and learning.The authors distinguish their framework by automatic generation from dynamic textual documents and independence from field and delivery technology.Source: §1.1 · pp. 4–5
2. Methodology
The methodology builds a GPT-3-based assistant from syllabus information, uses a knowledge model and question-answering pipeline, and supports multiple delivery channels. Its case study evaluates syllabus extraction and question answering using syllabi and question sets.
2.1. Natural Language Inference
- GPT-3 was selected as a cloud-based language model with a developer-friendly API, and the authors describe its few-shot use and fine-tuning options.The paper identifies Davinci as the largest GPT-3 model available to researchers and the public at the time described.Source: §2.1 · p. 5
2.2. Syllabus Knowledge Model
- The syllabus knowledge model covers nine course-information categories, with 36 competency questions augmented to 120 question forms.Categories include course and faculty information, course goals, calendar, attendance, grading, materials, and policies.Source: §2.2 · pp. 5–6
2.3. System Architecture
- VirtualTA has four components: resource curation and indexing, assistant management, intelligent services, and communication-channel integration.The architecture includes intent mapping and inference among its intelligent services.Source: §2.3 · p. 7, Figure 1 · p. 7
2.3.1. Intelligent Services
- GPT-3 retrieves syllabus snippets using competency questions; the curated information is stored in JSON, validated, and used to fine-tune question answering.The knowledge-model process is shown as a population process with instructor revision.Source: §2.3.1 · p. 7, Figure 2 · p. 8
- Question answering uses a fine-tuned SQuAD-trained search model to select relevant documents and text-davinci-002 to generate concise natural-language answers.The fine-tuned search model’s training and validation datasets each contain 1,314 entries.Source: §2.3.1 · p. 8
- The system supports multilingual questions through translation to English and back, and describes tailoring responses through fine-tuning and sentiment-based supportive messages.The sentiment feature is intended to respond to negative emotion or stress with optimistic messages and relevant resources.Source: §2.3.1 · pp. 8–9
2.3.2. Framework Integration
- The assistant is designed for plug-and-play integration, with web, Discord, and Google Assistant implementations described.The service-oriented architecture uses webhooks to connect delivery channels.Source: §2.3.2 · p. 9
2.4. Case Study Design; 2.4.1. Phase 1 - Knowledge Extraction
- The case study collected 112 syllabi, removed 12 image-format files, and used the remaining 100 to assess extraction and question mapping.Phase 1 selected 38 syllabi and asked 36 questions per syllabus to measure correct, incorrect, and partial answers.Source: §2.4 · pp. 9–10, §2.4.1 · p. 10
Before Edits
The example shows course-question and answer entries in a JSONL-style template, with a field for marking answer correctness.
Before Edits
- The template pairs course questions with answers and an isTrue field intended for TRUE, FALSE, or PARTIAL labels.Examples ask for the course name, course number, and credit hours.Source: Before Edits · p. 10
After Edits
After Phase 1 answers were manually checked and corrected, Phase 2 tested question-answering performance on an expanded set of syllabus questions and variations.
After Edits
- For the Phase 1 syllabus questions, the team manually reviewed bot answers, labeled them TRUE, FALSE, or PARTIAL, and corrected false answers for Phase 2 testing.Table 2 presents examples of the edits and highlights the changed labels.Source: After Edits · pp. 10–11
2.4.2. Phase 2 - Question Answering
- Phase 2 tested the manually corrected Phase 1 templates on 70 questions, expanded from 36 through text augmentation of question wording.Two questions were not augmented because no sensible variation was found.Source: §2.4.2 · p. 11
- Answers to the 70 questions were stored in JSONL and manually checked against the syllabus, with correct, incorrect, and partially correct responses labeled.Incorrect answers were also replaced with the correct information.Source: §2.4.2 · pp. 11–12
3. Results and Discussion
The system is demonstrated across chat, voice, multilingual, and sentiment-responsive channels. Its evaluation averages results across 38 syllabi and documents distinct knowledge-extraction and question-answering error cases.
3.1. Communication Channels
- VirtualTA spans web and Discord chat, Google Assistant voice queries, GPT-3-supported multilingual exchanges, and humorous or positive responses to negative sentiment.The Google Assistant integration uses the command “talk to Virtual T.A.” to connect to the third-party action.Source: §3.1 · pp. 12–15
3.2. Performance Evaluation
- Across 38 syllabus files, evaluation averages knowledge-extraction and question-answering performance; VirtualTA prioritizes precision and returns “Response not found” when unsure.Tables 3 and 4 report average accuracy values across the 38 files; the fallback lets students double-check answers with an instructor or TA.Source: §3.2 · p. 15, Table 3 · p. 16, Table 4 · p. 16
3.2.1. Knowledge Extraction
- Knowledge extraction errors include missing multiple TAs, substituting instructor information when no TA is listed, and missing course details in confusingly formatted syllabi.These cases were counted as incorrect when calculating the reported accuracy.Source: §3.2.1 · p. 16, Table 2 · p. 16
3.2.2. Question Answering
- Question answering errors include estimating absent credit hours from lecture frequency and sometimes giving different answers to similar questions depending on phrasing.Both cases were counted as incorrect in the reported accuracy.Source: §3.2.2 · p. 17, Table 5 · p. 17
4. Conclusion and Future Work
VirtualTA is designed to answer logistical course questions through discussion boards, third-party applications, and educational platforms. Proposed future work includes student studies, broader course-content support, and system improvements.
4. Conclusion and Future Work
- VirtualTA is designed to answer logistical questions through course discussion boards, third-party applications, or educational platforms, with access through familiar communication channels.The architecture supports any number of enrolled users; the stated aims include individualized advising and delegating repetitive instructor duties.Source: §4 · p. 17
- The authors propose a semester-long, multi-field student case study and integration with learning management systems as future work.The proposed work also includes importing course information directly from learning management systems.Source: §4 · p. 18
- Future work includes adding course-content assistance, search, quizzes, flash cards, personalized communication, and fine-tuning with larger datasets to improve accuracy and performance.The authors also propose support for different question-asking techniques and integration with mainstream communication channels.Source: §4 · p. 18
6. Appendix
The appendix presents competency questions for a syllabus knowledge graph, spanning course logistics, course information, and policies. Its examples include questions about instructor and TA contact details, course objectives, assessments, and academic conduct.
6. Appendix
- The syllabus knowledge-graph questions cover lecture times and locations, instructor and TA office locations and hours, names, and contact information.Examples ask when lectures take place and how to contact the instructor or TA.Source: Table A.1 · p. 23, §6 · p. 24
- Course-information questions address objectives, expectations, prerequisites, textbooks, and other required materials.The appendix includes questions about course objectives and whether prerequisites or corequisites are required.Source: Table A.1 · p. 23, §6 · pp. 24–25
- Policy and schedule questions cover assignment dates and late work, attendance, classroom behavior, grading, exams, and academic dishonesty.Examples include asking about assignment deadlines, the grading breakdown, and the course’s cheating policy.Source: Table A.1 · p. 23, §6 · p. 25