Source-linked AI summary
VidTutorAssistant: Automating Responses to Programming Tutorial Questions
Ahmad Tayeb, Sonia Haiduc, Mohammad D. Alahmadi
TL;DR
Programming tutorial comments can outnumber creators’ capacity to answer, leaving learners without clarifications. VidTutorAssistant uses retrieved transcript context to generate replies; in its evaluation, its answers were more often correct and complete than creators’ replies, and students preferred them more often.
Problem
Programming-tutorial comment sections can contain more questions than creators can answer, leaving learners without clarifications; prior work also lacked a deployed QA tool for this setting.
Source: §1 · p. 1Method
VidTutorAssistant is a deployed web platform that classifies comments and uses retrieval-augmented generation with GPT-4 to answer questions using relevant transcript context.
Source: §1 · p. 1Results
98% of VidTutorAssistant responses were correct and 99.5% complete, versus 89% and 90% for creators’ answers; students preferred its responses 66% of the time.
Source: §4 · p. 4Takeaways & Limitations
Overall, the evaluated pipeline produced responses at least as correct and complete as creators’ answers, and students preferred its answers more often.
Source: §4 · p. 4The deployed pre-processor uses a 1,500-token transcript threshold to leave room for the comment and response prompt in GPT-4’s context window.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Programming tutorial videos on YouTube are an important information resource for software developers and students, and their comment sections have evolved into active spaces where viewers ask follow-up questions. The volume of these questions, however, often exceeds what content creators can address, leaving learners without the clarifications they need. We present VidTutorAssistant, a web platform that automates responses to viewer questions on programming video tutorials. VidTutorAssistant implements a retrieval-augmented generation pipeline that extracts a video's transcript, then segments it and embeds it. It then classifies each viewer comment as being a question or non-question, retrieves the most relevant transcript segments to each identified question via cosine similarity, and then generates an answer to the question using an LLM (GPT-4), while grounding the response using the retrieved transcript segments as context. We validate VidTutorAssistant through a study on a subset of 440 user comments selected from a larger dataset of 105,553 comments extracted from 7,522 Python and Java tutorials. VidTutorAssistant is evaluated on various criteria: a) its ability to identify the programming language in a video, achieving a 0.99 accuracy; b) its ability to classify comments into questions and non-questions, reaching a 0.96 accuracy; and c) its ability to produce correct and complete answers to questions, producing 98% correct and 99.5% complete responses, compared with 89% and 90% for the original creators' answers.
1 Introduction
Programming tutorial comments can contain more questions than creators can answer, while prior tutorial-video research has largely focused on extracting artifacts rather than responding to viewers. VidTutorAssistant addresses this gap with a deployed response-generation tool and an evaluation dataset and study.
1 Introduction
- Prior work focused mainly on extracting video artifacts, while tutorial-video QA efforts used generic videos or synthetic or manually anchored questions and lacked a deployed tool for programming tutorials.The unanswered-question volume can leave learners without clarifications, slowing learning and reducing a tutorial’s educational value.Source: §1 · p. 1
- VidTutorAssistant is a deployed web tool that uses a four-module retrieval-augmented generation pipeline with GPT-4 to answer programming-tutorial questions using retrieved transcript context.The system is intended for viewers and as a draft-generation aid for creators.Source: §1 · p. 1
- The study uses 105,553 comments from 7,522 Python and Java tutorials and a manually labeled subset of 440 comments to evaluate language identification, question classification, answer quality, and user preference.The evaluation subset includes 220 creator answers and 220 VidTutorAssistant-associated answers.Source: §1 · p. 2
2 The VidTutorAssistant Approach
VidTutorAssistant processes tutorial comments through transcript preparation, question classification, context retrieval, and response generation. Retrieved transcript context grounds the LLM’s response.
2 The VidTutorAssistant Approach
- The Video Pre-Processor and Context Provider implement retrieval, and the Response Generator uses retrieved context to ground the LLM output.GPT-4 is the current implementation, but the tool is designed to allow replacement with another LLM.Source: §2 · p. 2
2.1 Video Pre-Processor
- For transcripts longer than 1,500 tokens in the current deployment, the pre-processor splits text into 300-token segments and embeds them; shorter transcripts pass through in full.The 1,500-token threshold leaves room for the viewer comment and response prompt in GPT-4’s context window.Source: §2.1 · p. 2
2.2 Comment Classifier
- The Comment Classifier uses GPT-4 to label comments as questions or non-questions, routing questions to context retrieval and letting non-questions bypass it.Expressions of thanks and general feedback are examples of non-questions.Source: §2.2 · p. 2
2.3 Context Provider
- For questions, the Context Provider reformulates the comment as a query and retrieves the most similar transcript segment, its neighbors, and the video’s first and last segments.The first and last segments provide broader video context beyond the local match.Source: §2.3 · p. 2
2.4 Response Generator
- The Response Generator prompts GPT-4 with the comment, retrieved context, and channel, video, and programming-language metadata to produce an informative response aligned with the transcript’s tone and style.Responses can serve as drafts for creators to post directly or edit.Source: §2.4 · p. 2
3 Tool Walkthrough
The platform offers a searchable tutorial catalog and video detail pages that pair tutorial information with viewer comments and generated responses. Its implementation separates the Python pipeline backend from the Laravel web interface.
Browse Mode
- Browse Mode lets users search the tutorial catalog by title or metadata and view cards with thumbnails, duration buckets, comment-reply counts, and clickable titles.Selecting a card opens that tutorial’s detail page.Source: §3 · p. 2, Figure 2 · p. 3
Video Detail Mode
- Video Detail Mode presents tutorial metadata and an embedded video, an Ask VidTutorAssistant widget, and stored viewer comments paired with generated responses.Where available, the original creator’s reply appears beneath the AI response.Source: §3 · p. 3, Figure 3 · p. 3
Implementation
- The platform uses a Python backend for pipeline modules 1–4 and a Laravel web application with a Blade and Livewire frontend; embeddings are cached per video and stored responses and feedback use MySQL.The widget supports a model selector, with GPT-4 and GPT-4o accessed through the OpenAI API.Source: §3 · p. 3
4 Evaluation
The evaluation combines three studies of language identification, comment classification, answer quality, and student preferences. VidTutorAssistant’s answers were rated more correct and complete than creators’ replies, and students preferred them more often.
4 Evaluation
- Study 1 assessed language identification and question classification, while Study 2 compared creator and VidTutorAssistant answers for correctness and completeness.Study 3 compared participant preferences between the two answer types.Source: §4 · pp. 3–4
- The dataset comprises 105,553 comment-reply pairs from 7,522 unique videos; researchers sampled a representative 440-comment evaluation subset balanced across two programming languages and eleven video-duration buckets.Source: §4 · p. 3
- 98% of VidTutorAssistant answers were correct and 99.5% complete, compared with 89% correct and 90% complete for creators’ answers.Both response types were rated on 220 question comments.Source: §4 · p. 4
- 66% of participant reviews preferred VidTutorAssistant, 27% preferred creators, and 7% expressed no preference.The comparison covers 438 reviews by 46 computer science students; the preference split was significant among reviews expressing a preference.Source: Figure 4 · p. 4, §4 · p. 4
- Participants favored VidTutorAssistant for detailed, comprehensive explanations and a friendlier tone, while creators’ concision suited some simple questions.Overall, the studies indicate that VidTutorAssistant’s responses were at least as correct and complete as creators’ responses and were preferred more often.Source: §4 · p. 4
5 Related Tools
Related tools either extract structured artifacts from tutorial videos or answer questions about instructional videos. VidTutorAssistant focuses on programming tutorials and is deployed as a platform for generating responses to viewer questions.
5 Related Tools
- The paper identifies CodeTube, ActionNet, and psc2code as tools that extract structured artifacts from tutorial videos, but says they do not address viewer-posted comments and questions.Source: §5 · p. 4
- Prior tutorial-video QA tools differ in scope or task: they classify answerability, target non-programming software, or use synthetic and curated lecture-video questions.The passage identifies YTCommentQA, AQuA, and EduVidQA as examples.Source: §5 · p. 4
- VidTutorAssistant focuses on programming tutorials, evaluates responses against original creator replies, and is released as a deployed platform.Source: §5 · p. 4
6 Conclusion and Future Work
VidTutorAssistant is a deployed web platform that uses retrieval-augmented GPT-4 to answer questions about programming tutorials. Its responses were rated highly for correctness and completeness, and students preferred them to creator responses in the user study.
6 Conclusion and Future Work
- On 220 evaluated questions, VidTutorAssistant produced 98% correct and 99.5% complete responses, and students preferred its replies 66% of the time.Source: §6 · p. 4
- The platform automates responses to programming tutorial questions with a retrieval-augmented GPT-4 pipeline.Source: §6 · p. 4
- Planned extensions include broader language support, creator-side draft review, and comparisons with newer LLMs.The planned model comparisons include open-source and multimodal models.Source: §6 · p. 4
7 Data Availability Statement
VidTutorAssistant is freely available online, with its source code and dataset provided in a replication package. A screencast of the tool is also available.