Source-linked AI summary
Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos
S M Masrur Ahmed, Jaspal Subhlok
TL;DR
Students need lecture-video assistance that answers course-specific questions while matching an instructor’s materials and providing verifiable evidence. The paper deploys a course-isolated retrieval-augmented chatbot with chapter-summary-guided ranking and timestamped citations, then evaluates deployment behavior and retrieval. In deployment and EduVidQA testing, citations, course isolation, and refusal behavior were central design considerations, while chapter summaries produced small consistent retrieval improvements.
Problem
Search and summaries help students navigate lecture videos, but existing tools do not readily answer course-specific questions or verify answers against an instructor’s lecture.
Method
The paper reports a semester-long VideoPoints deployment of a course-isolated retrieval-augmented chatbot using chapter summaries as a ranking prior and returning timestamped citations.
Results
Chapter summaries produced small and consistent retrieval improvements, while the deployment showed citation presence, course isolation, citation support, and refusal behavior should be evaluated separately.
Takeaways & Limitations
Effective educational chatbot deployment depends on course isolation, supported citations, and alignment with students’ task-oriented study practices.
Takeaways & Limitations
The study was conducted at one institution with concentrated use, and its session-level data cannot establish unique-user counts, adoption rates, or learning gains.
Abstract
from arXiv · showhide
Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask course specific questions or verify answers against an instructor's lecture. We report a semester-long deployment of VideoPoints platform with a retrieval-augmented chatbot that answers from course lecture materials and returns timestamped citations. The chatbot retrieves only from the active course, uses chapter summaries to guide transcript ranking, and returns clickable timestamped citations. Students used it for quick lookups and exam review. Across 833 messages, 70.5% included citations, none crossed a course boundary, and when no lecture evidence matched, the chatbot usually declined rather than answering. Among the users, citations were the most consistently useful feature, while practice-question generation was the strongest unmet request. We also evaluated the design on the real-world test split of EduVidQA, a public multimodal benchmark for lecture-video question answering. Our design improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval. Together, the results show that effective deployment depends on course isolation, supported citations, and alignment with students' study practices.
1 Introduction
The paper addresses the gap between searchable lecture videos and course-specific conversational assistance by deploying and evaluating a citation-first, course-grounded chatbot. It studies student use, citation and refusal behavior, retrieval design, and perceived usefulness.
- Motivation: Searchable lecture videos support navigation, but chapter indexes cannot answer questions and general chatbots may not match an instructor’s framing.The paper also notes that general chatbots cannot answer context-specific logistical questions such as quiz timing and format.
- Design requirements: Course grounding was required: answers had to come from the active course’s lecture materials and point to relevant video segments.This requirement was treated as a deployment constraint rather than merely a technical preference.
- Research questions: The study asks how students use the chatbot, when it provides citation-supported course-isolated answers or refuses, and which study needs remain unmet.It also examines perceptions of chapter summaries, timestamped citations, trust, and exam usefulness.
- Study design: The evaluation combines production logs, retrieval traces, survey responses, citation-support and refusal auditing, and a controlled EduVidQA retrieval comparison.The benchmark uses the same corpus, questions, and retrieval budget across the system and baselines.
- Contributions: The paper contributes deployment evidence about citation-first lecture-video chatbots rather than a new RAG architecture.It separately examines grounding, retrieval design, course isolation, question reformulation, and mismatches with task-oriented student requests.
2 Related Work
Prior work improves access to lecture-video content through segmentation, search, summaries, and other structural cues, while educational RAG addresses question answering from external materials. This paper focuses on production deployment of a course-grounded conversational chatbot over lecture videos.
- Lecture video navigation: Lecture-video systems support non-linear access through topic segmentation, indexed navigation, captions, transcript search, visual anchors, and structural cues.VideoPoints and later work also use AI-generated visual and textual summaries for navigation and review.
- Lecture video navigation: Prior studies report favorable perceptions and, in some settings, improved learning outcomes when students receive lecture videos alongside AI-generated summaries.The passage presents these findings as related work on summary-supported review.
- Retrieval-augmented generation in education: Educational retrieval-augmented generation can improve course specificity but raises concerns about trust, citation quality, and mismatch with instructional context.Related systems address lecture-video question answering, course logistics, tutoring, and course assistance.
- Retrieval-augmented generation in education: This study complements prior educational chatbot work by examining a production course-grounded conversational chatbot deployed over lecture videos.Its emphasis is deployment behavior rather than only evaluation-setting question answering.
3 System and Deployment Setting
The system extends VideoPoints with chapter summaries and a course-isolated, citation-first retrieval-augmented chatbot. Summaries support both student navigation and transcript ranking while preserving access to all active-course material.
- Platform: VideoPoints supplies thumbnails, captions, search, transcripts, keywords, visual highlights, instructor chapter editing, and topic-based lecture segmentation.The chatbot deployment leverages this existing production infrastructure.
- Chapter summaries: Each chapter receives an LLM-generated title and summary displayed in the chapter index so students can judge relevance before watching.This summary interface was tested during Phase 1 before chatbot deployment.
- Chapter summaries: Chapter summaries provide a cleaner relevance signal than noisy transcript spans and are stored for query-time ranking without regenerating them.The system uses query-summary similarity as a soft prior for transcript and slide-text chunks.
- Chapter summaries: All active-course transcript and slide-text chunks remain eligible, while better-matching chapter summaries raise their retrieval scores.This preserves recall rather than imposing a hard summary-based filter.
- Deployment constraints: The chatbot enforces course-only retrieval, clickable timestamp citations, privacy-preserving text handling, and response speed suitable for routine study.These constraints motivated a retrieval-only design instead of a general chatbot.
- Citation-First RAG Pipeline: The pipeline encodes questions, restricts retrieval to the active course, combines dense, BM25, and summary-prior scores, then generates timestamp-grounded answers.Chapter summaries act as ranking anchors while transcript and slide-text chunks remain eligible.
- Deployment phases: Phase 1 measured chapter-summary feedback before release, whereas Phase 2 collected chatbot logs and student perception during deployment.The two phases ran in Fall 2025 and Spring 2026, respectively.
4 Evaluation Methodology
The evaluation combines surveys, production and retrieval data, privacy-preserving logs, diagnostic auditing, and an offline benchmark. Because production traffic lacks gold retrieval labels and the study has no randomized control group, claims focus on deployment behavior and perceived usefulness.
- Evaluation overview: The study analyzes survey data, system logs, retrieval traces, de-identified cases, diagnostic audits, and controlled offline retrieval evaluation.It does not measure learning outcomes or include a randomized control group.
- Survey measures: Surveys measure chapter-summary accuracy and value, chatbot experience, trust, citation and exam usefulness, and interest in practice questions and textbook integration.All items use a 1–5 Likert scale, with response counts reported per item.
- Privacy and log measures: Student privacy is protected through salted and hashed session identifiers, local raw-log storage, de-identification, and screening before external-model auditing.Stored records contain questions, responses, and returned video matches.
- Diagnostic audit: The diagnostic audit covers 224 unique messages and evaluates citation support, evidence relevance, refusal appropriateness, and missed implicit refusals.Sampling includes all four citation-refusal outcome groups, with smaller groups and imperative requests audited in full.
- Offline retrieval evaluation: The offline benchmark uses 269 questions from 99 lectures across a corpus of 139 videos, 10 courses, 1,062 chapters, and 5,924 chunks.EduVidQA is used because production questions lack annotated correct segments for scoring retrieval quality.
5 Results
The deployment shows concentrated, exam-aligned student use, strong demand for citation-supported course answers, and clear limits for agentic study tasks. Controlled evaluation and audits indicate that course isolation and retrieval design improved grounding, while citation presence did not guarantee support.
- Usage and study behavior: 328 messages, or 39.4% of all messages, occurred during a five-day midterm window, while another high-use period overlapped finals.The longest session contained 68 turns reviewing an exam outline, but temporal alignment does not establish that exams caused increased use or improved learning.
- Interaction patterns and task expectations: Imperative messages had higher odds of refusal (odds ratio 2.74, 95% CI 1.55–4.86) and no citation (odds ratio 3.74, 95% CI 2.10–6.68) than non-imperative messages.Direct questions and keyword searches were handled more successfully than commands requesting tasks such as practice-quiz generation.
- Course isolation and citation coverage: 70.5% of 833 messages included citations, and none of the cited messages pointed outside the active course.The system displayed exactly seven citations for each cited message, producing 4,109 citation events.
- Citation support and refusal quality: 65.0% of cited non-refusal answers were fully supported, while 86.3% were fully or partially supported and 11.3% were unsupported.The audit also estimated that 61.9% of refusals were appropriate, showing that citation presence and refusal frequency are not correctness measures by themselves.
- Controlled offline retrieval evaluation: Removing course isolation reduced video hit from 0.747 to 0.569, a decrease of 17.8 percentage points, the largest decrease among tested changes.The decrease was statistically supported for video hit (p ≤0.0005) and timestamp hit (p ≤0.034); the unrestricted system often retrieved evidence from the wrong course.
- Student perceptions and unmet needs: Among 22 self-reported chatbot users, citation usefulness and exam usefulness both averaged 4.55/5, while practice-question generation and grading was the strongest unmet need at 4.59/5.Every respondent rated citation links 4 or 5, whereas survey responses measured perceptions rather than controlled learning outcomes.
6 Discussion
The discussion identifies course isolation, citation support, and clear task boundaries as central deployment lessons. Chapter summaries help refine retrieval, but students also need instructor-controlled tools beyond direct question answering.
- Retrieval scope: Course isolation matters because unrestricted retrieval can return high-confidence evidence from the wrong course.Removing course isolation degraded retrieval more than changing either scoring component.
- Retrieval refinement: Chapter summaries provide a useful but limited ranking signal rather than the main source of retrieval quality.They offer cleaner descriptions than noisy conversational transcripts and produce small, consistent improvements.
- Citation evaluation: Citation coverage and citation support are distinct properties that require separate evaluation.The automated audit should be followed by human evaluation, and refusal frequency does not establish refusal appropriateness.
- Study needs: Students used the chatbot for direct questions and exam review but also expected practice generation, grading, study planning, and memory.The retrieval-based design was not designed for task-oriented requests, motivating instructor-controlled tools and clear boundaries.
7 Conclusion
The study deployed a course-isolated lecture-video chatbot with chapter summaries and timestamped citations in STEM courses. Usage and perception results support citation-first design while identifying study tasks requiring additional instructor-controlled support.
- Conclusion: 833 messages included short information requests and sustained exam-review sessions, while self-reported users rated timestamped citations positively.246 messages received no citation, and some cited answers were not fully supported.
- Conclusion: All returned citation events stayed within the active course.The study separately examined course isolation, citation presence, citation support, and refusal appropriateness.
- Conclusion: The study measures usage and perception rather than learning gains.It does not establish that the chatbot improved learning outcomes or exam performance.
- Conclusion: The findings provide practical evidence for citation-first educational chatbots and study tasks requiring instructor-controlled support beyond question answering.The conclusion links deployment evidence to design needs outside grounded question answering.
Limitations
The study’s evidence is bounded by its institutional setting, concentrated usage, session-level analysis, survey design, automated labels, diagnostic audit, and controlled offline evaluation. These limits constrain generalization, learning-outcome claims, and validated factuality estimates.
- Scope and sampling: One institution and concentrated usage limit generalization across departments and instructors.One large introductory course produced 656 of 833 messages.
- Scope and sampling: Session-level analysis without user identifiers or enrollment denominators prevents computing unique-user counts, repeat-use curves, and adoption rates.The study therefore cannot characterize individual-level engagement or adoption.
- Outcome measurement: Survey results measure perception rather than learning outcomes, so the study cannot claim improved exam performance.This limitation applies directly to interpretation of the survey evidence.
- Automated labeling: Intent labels are exploratory because primary-intent agreement was low, and automated-method consistency is not accuracy against human annotation.Agreement was high for interaction form and refusal, moderate for agentic expectation, and low for primary intent.
- Citation and factuality: Citation generation does not prove citation use, factual correctness, or video watching, and the audit lacks a human-validated factuality rate.The logs include at least one confirmed grounded-but-incorrect answer.
- Offline evaluation: The offline retrieval evaluation compares design choices but does not reproduce production courses, questions, or retrieval conditions.It is a controlled reconstruction on a public benchmark rather than a replay of production traffic.
Ethical Considerations
The deployment treated privacy and academic integrity as central concerns through anonymized session-level analysis and optional, consented surveys. Assessment-related chatbot use also motivated instructor controls and assessment-mode policies.
- Privacy: Session identifiers were salted and hashed before export, while the dataset contained no student identifiers.Survey responses were anonymous and not joinable to logs.
- Privacy: The diagnostic audit used anonymized production questions and responses with privacy scanning before external-model requests.Raw session identifiers were not transmitted, though automated scanning cannot guarantee that free-form text contains no identifying information.
- Academic integrity: Some students pasted multiple-choice or true-false items into the chatbot, but this was reported as deployment reality rather than evidence of misconduct.The authors note that such items may come from review materials or ungraded practice.
- Academic integrity: Educational RAG systems should include instructor controls, assessment-mode policies, and explain-without-answer options alongside graded work.These design implications follow from assessment-related chatbot behavior in deployment.
A.1 Additional Figures and Interface
The study combines course-isolated retrieval, lexical and dense matching, and chapter-summary priors to support citation-grounded lecture-video answers. Evaluations examine message behavior, citation and refusal quality, retrieval metrics, and sensitivity to course isolation and summary weighting.
- Scoring function: The scoring function combines normalized dense similarity, BM25 lexical relevance, and a chapter-summary prior while retaining all active-course chunks as candidates.BM25 received a higher weight than the dense channel, and summary priors were computed from the best query-to-summary-bullet similarity.
- Citation and refusal audit: The diagnostic audit found that 54% of uncited messages were clearly unanswerable from course evidence, while 58% of refusals were judged appropriate.The dominant labeled failure reasons were insufficient evidence to judge and missing course content, each at 40%.
- Scoring function: Chapter-summary priors provided small, consistent ranking improvements but were treated as refinements rather than the main source of retrieval quality.Performance was strongest at γ = 0.1 and generally declined as the prior received more weight.