Source-linked AI summary
Multi-sourced AI Tutor for Non-Native English Speaker Students
Yiyin Shen, Xinyi Ai, Weibing Wang, Rogers Jeffrey Leo John, Meenakshi Syamkumar
TL;DR
NNES students can face language barriers and hesitation to seek help in computing courses. This paper presents a Mandarin-capable, multi-source RAG tutor with translation and question extraction, evaluated on 91 Piazza posts for answering student questions. The authors identify the time and resources required for human evaluation as a limitation.
Problem
NNES students face language barriers and lack of confidence that can make them hesitant to seek help from instructors or TAs.
Source: §1 · p. 1Method
Results
The system was evaluated on 91 Piazza posts and was reported to answer student questions accurately and helpfully while generating technical-term glossaries.
Source: §1 · p. 1Takeaways & Limitations
The authors conclude that their RAG tutor can provide Mandarin-speaking students with bilingual, guided answers drawing on diverse course materials.
Source: §6 · p. 6Human evaluation across the pipeline’s multiple stages is time-consuming and resource-intensive.
Source: §5.2 · p. 6
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly being adopted as teaching tools in computing education, particularly for developing AI tutors capable of answering student questions. With the rise in enrollment of international students in computing programs, researchers have begun to explore the difficulties that non-native English speaker (NNES) students face in computing courses due to language barriers. To aid such students, we developed a general-purpose AI tutor system that can process student questions in foreign languages (currently focused on Mandarin) and provide bilingual responses. Our system also incorporates the functionality of generating a glossary of technical terms and their explanations to enhance comprehension. Our approach utilizes a specialized Retrieval-Augmented Generation (RAG) pipeline with a novel question extraction mechanism, enabling us to effectively utilize a diverse array of course materials for answering student queries. We evaluated our AI tutor using 91 Piazza posts, demonstrating that it could answer 83.52% of the posts correctly and in a helpful manner. The tutor also provided high-quality translations for 76.92% of the questions and generated technical term glossaries for 64.47% of them. Our findings underscore the potential of multi-source RAG pipeline in enhancing AI tutor systems, offering support for non-native speakers that are challenging to provide with traditional supporting staff of a computing course.
1 Introduction
The tutor targets language barriers that can make NNES students hesitant to seek help, accepting Mandarin questions and returning bilingual responses with technical-term glossaries. It combines course-wide materials with translation and question extraction, and was evaluated on 91 Piazza posts.
1 Introduction
- The tutor accepts Mandarin questions and provides bilingual answers and technical-term glossaries to support NNES students facing language barriers and hesitation to seek help.The motivation describes language barriers and lack of confidence as factors that often cause hesitation to seek help from instructors or TAs.Source: §1 · p. 1
- Unlike single-source AI tutors, the system retrieves from lecture notes, project descriptions, quizzes, past exams, and the course website.The paper describes its system as a RAG-based GPT-4 tutor.Source: §1 · p. 1
- A translation stage converts Mandarin questions into English, while question extraction recovers details when posts refer only to a question number.Question extraction is designed for queries that omit question details and refer to them by number.Source: §1 · p. 1
- The study evaluates 91 Piazza posts and frames its questions around translation quality and its effect on RAG, question extraction, retrieval relevance and usefulness, and response quality.Response quality includes correctness, helpfulness, solution revelation, translation, and technical-term glossary generation.Source: §1 · p. 1
2 Related Work
Prior AI tutors use varied RAG sources and prompting strategies, while research on NNES students documents language and confidence barriers and interest in bilingual support. This work combines a multi-source tutor with Mandarin input and bilingual answers.
2.1 AI Tutors
- RAG, fine-tuning, and reinforcement learning improved answer accuracy and usefulness for Llama-2 and GPT-4 in Hicke et al..For Llama-2, accuracy rose from 0.30 to 0.58 and usefulness from 0.18 to 0.50; for GPT-4, accuracy rose from 0.40 to 0.73 and usefulness from 0.28 to 0.65.Source: §2.1 · p. 1
- Many AI tutors use RAG, but commonly draw on a single source; this tutor aims to incorporate course materials, projects, quizzes, exams, and the course website.Prior systems also include course-deployed chatbots using GPT-4 with lecture captions or few-shot learning to prevent code revelation.Source: §2.1 · p. 2
2.2 Non-Native English Speakers in Computing Education
- NNES students report vocabulary, grammar, accent, and slang barriers, alongside translating utterances and fear or nervousness about speaking.Polishchuk reports that 85.5% translated utterances, 61.25% feared making mistakes, and 47.5% felt nervous speaking in class.Source: §2.2 · p. 2
- A survey of 840 respondents found NNES students reluctant to ask questions or join discussions and preferring simplified English with visuals and inline dictionaries.The study surveyed respondents from 86 countries on a programming education website.Source: §2.2 · p. 2
- Prior studies report that bilingual instruction improved comprehension, reduced cognitive load, and boosted engagement and confidence, while this work implements a Mandarin tutor with bilingual answers.Lau and Yuen also reported improved programming understanding with bilingual instruction among K-12 students in Hong Kong.Source: §2.2 · p. 2
3 Method
The method builds a four-stage tutor pipeline for course Piazza questions: translation, question extraction, retrieval over diverse course materials, and bilingual response generation. Question extraction addresses posts that refer to course questions by number rather than reproducing their content.
3.1 Students’ Questions
- The dataset began with 620 Piazza questions from a 400-student data science course; after excluding image posts, 382 text posts were cleaned and categorized.The 238 excluded posts contained images; the text posts were assigned to logistics, conceptual, environment setup, and programming categories.Source: §3.1 · p. 2
3.2 AI Tutor Data Pipeline
- Figure 1 presents the pipeline as four stages: translation, question extraction, RAG, and response generation.The source data included original English Piazza posts and human-verified Mandarin translations.Source: §3.2 · p. 2, §3.1 · p. 2, Figure 1 · p. 3
3.2.1 Translation
- Mandarin questions are translated into English using Azure AI Translate, chosen over a Mandarin-dedicated LLM to leverage GPT-4’s programming-context capabilities.The pipeline also processes original English Piazza questions.Source: §3.2.1 · p. 2
3.2.2 Question Extraction (QE)
- Question extraction addresses posts that cite numbered project, quiz, or exam questions without giving their content, which ordinary RAG retrieval struggles to recover from number-only references.Llama 3 8B standardizes question numbers, which a rule-based algorithm uses to fetch question text from a bank built from exams and Canvas quizzes.Source: §3.2.2 · pp. 2–3
3.2.3 RAG
- The RAG vector store combines diverse course materials, and similarity search retrieves the top 10 results using both the student post and extracted questions.Past exams and Canvas quizzes are excluded from similarity search to avoid skewing retrieval away from lecture materials.Source: §3.2.3 · p. 3
3.2.4 Response Generation
- GPT-4 generates responses in English and Mandarin, using RAG results and extracted questions while being instructed to offer guidance rather than direct solutions or code.The prompt also asks it to cite source paths and provide a glossary of challenging technical vocabulary or error messages.Source: §3.2.4 · p. 3
4 Results
Across examples and quantitative evaluations, the tutor retrieved course information for Mandarin questions and generated responses with varying levels of correctness, helpfulness, translation quality, and glossary coverage. The results also show that question extraction and translation play distinct roles in supporting retrieval.
4.1 Examples
- In two examples, question extraction located the referenced exam, quiz, or programming question, and RAG-supported responses clarified a concept or offered guidance without revealing a solution.The Mandarin translations and technical-term glossaries were rated high quality in both examples.Source: §4.1 · p. 3
4.2.1 Translation
- Only 2 of 382 Mandarin posts remained untranslated; translations of the others retained their original meanings and essential information.The evaluation concluded that translation before RAG helps retrieve relevant information, as translated posts overlapped more with English retrievals than Mandarin posts.Source: §4.2.1 · p. 4, §4.2.3 · p. 4
4.2.2 Question Extraction
- Question-extraction accuracy was 97.78% for logistics, 90.74% for conceptual, 100% for environment setup, and 97.46% for programming questions.English-Mandarin QE overlap was 98.89%, 72.22%, 100%, and 97.97%; English-translated overlap was 95.56%, 74.07%, 100%, and 97.46%, respectively.Source: §4.2.2 · p. 4, Table 1 · p. 4
4.2 Quantitative Results
- The 91-post test set included 21 logistics, 14 conceptual, 9 environment-setup, and 47 programming questions, covering high-interest questions and less common queries posing unique challenges.Two native Mandarin-speaking authors evaluated all four pipeline stages; a third native Mandarin-speaking author and course instructor resolved discrepancies. English question-extraction and RAG results were compared with Mandarin and translated results by percentage overlap to serve as ground truth.Source: §4.2 · pp. 3–4, §4.2.3 · p. 4
4.2.3 RAG
- With question extraction, the average relevant results among 10 RAG retrievals were 6.1, 7.7, 7.3, and 6.6 across logistics, conceptual, environment setup, and programming questions; at least one retrieval helped 80.95%, 71.43%, 88.89%, and 89.36% of posts, respectively.English-translated posts had greater RAG overlap with English results than Mandarin posts across all four categories.Source: §4.2.3 · p. 4, Table 2 · p. 4
4.2.4 Response Generation
- Correctness was 100% for conceptual and environment setup questions, 97.83% for programming, and 80.95% for logistics; helpfulness ranged from 61.90% to 93.33%.Responses addressed 61.90% of logistics, 93.33% of conceptual, 88.89% of environment setup, and 89.13% of programming posts.Source: §4.2.4 · p. 5, Table 3 · p. 5
- No logistics, conceptual, or environment setup responses revealed direct solutions, while programming responses did so at 13.04%, primarily through code snippets.The evaluation defined solution presentation as directly providing solutions, particularly code snippets.Source: §4.2.4 · p. 5
- Mandarin response translation succeeded for 71.43% of logistics, 86.67% of conceptual, 77.78% of environment setup, and 76.09% of programming questions; glossaries appeared in 95.24%, 86.67%, 77.78%, and 86.96%, respectively.When provided, glossary terms were essential to the questions, uncommon in everyday English, and accurately explained.Source: §4.2.4 · p. 5, Table 3 · p. 5
5 Discussion and Future Work
The tutor’s evaluated responses were often helpful and included Mandarin responses with technical glossaries; future work targets classroom deployment and broader capabilities.
5.1 Our Contributions
- 83.52% average helpfulness was achieved across evaluated Piazza questions, and 82.89% benefited from question extraction or RAG.The system integrates diverse course materials rather than relying on a single source.Source: §5.1 · p. 5
- Over 76.92% of responses provided quality Mandarin responses, and 87.91% contained an effective technical term glossary.The authors suggest the system may offer NNES students a non-judgmental, accessible interface that fosters engagement and inclusivity.Source: §5.1 · p. 5
5.2 Limitations and Future Work
- Classroom deployment is planned after improving question extraction, preventing unintended answer disclosure, and shifting from single responses to conversation.The authors also plan to assess NNES students’ questioning behavior and needs after deployment.Source: §5.2 · p. 5
- Future work would expand the system to image-accompanied questions and evaluate GPT-4o and additional-language versions.The authors plan to test whether GPT-4o improves performance and whether adaptation to more languages is valid.Source: §5.2 · pp. 5–6
- Because human evaluation across pipeline stages is time- and resource-intensive, the authors will explore automated evaluation and plan to update course materials and detect active quiz or exam questions.They aim to keep the RAG vector store current throughout the semester.Source: §5.2 · p. 6
- The authors plan an interface for instructors to upload course materials and customize their own tutors.They intend this to broaden the pipeline’s reach and utility.Source: §5.2 · p. 6
6 Conclusion
The paper presents a Mandarin-focused tutor that uses RAG and question extraction to answer from varied course materials with bilingual responses and technical glossaries. Its data science evaluation underscores potential to improve educational accessibility for non-native English speakers, with classroom deployment and broader language support left for future work.
6 Conclusion
- The tutor processes Mandarin questions and delivers bilingual responses with technical glossaries, without directly revealing answers.The conclusion characterizes the responses as correct, helpful, and well-translated.Source: §6 · p. 6
- RAG and a specialized question extraction mechanism answer queries from diverse course materials, supported by prompts designed to guide GPT-4 in using both resources appropriately.Materials include Jupyter Notebooks, Python scripts, PowerPoint slides, and PDFs.Source: §6 · p. 6
- The data science question evaluation underscores the potential of AI tutors to enhance educational accessibility for non-native English speakers in computing courses.Future efforts include real-classroom deployment and expanded language support.Source: §6 · p. 6