Source-linked AI summary
Potential for Enhanced Learning in Machine Learning Classes by Using Wiki LLM Indexing
Brian Wright
TL;DR
Course-specific tutors need grounding in vetted materials, but prior course RAG work found that fixed retrieval strategies behaved differently across question types. This paper compares vector RAG with an ingest-time compiled wiki on an authentic course corpus; the wiki scored higher overall and was more often grounded. The authors present the findings as directional and call for further testing.
Problem
Prior course RAG work found fixed retrieval strategies suboptimal across question types, motivating a test of whether ingest-time knowledge structure matters beyond query-time retrieval.
Source: §1.1 · p. 1Method
The study compares vector RAG with an LLM-compiled linked wiki over the same course corpus under matched generator, prompt, context budget, and question conditions.
Source: §3 · p. 4Results
9.95 vs. 9.05 out of 10 overall, with 98% vs. 81% grounded, for the compiled wiki and vector RAG respectively.
Source: §6 · p. 7Takeaways & Limitations
The authors suggest courses teaching dynamic material consider the linked-wiki approach for generative AI assistants, while noting further testing is needed.
Source: §6 · p. 7The study does not test generalization beyond one course corpus, and its vector-RAG configuration was inherited rather than retuned for this corpus.
Source: §5.4 · p. 6
Abstract
from arXiv · showhide
Large language models are increasingly deployed as course-specific tutors, but their usefulness depends on grounding in vetted instructional materials that are often revised mid-semester. Our prior work built a multimodal retrieval-augmented generation (RAG) system over an authentic machine learning course corpus (Foundations of Machine Learning) and found that retrieval improved contextual grounding, but that fixed retrieval strategies were suboptimal. That motivates a different question: whether how a corpus is structured at ingest time matters more than how much is retrieved at query time. We present a controlled head-to-head comparison of two knowledge representations over an identical classroom corpus: (A) vector RAG, replicating the best-performing configuration from our prior study, and (B) an LLM-compiled wiki (Karpathy framework), in which the corpus is synthesized at ingest into linked concept pages with explicit cross-references and citations back to source materials. We evaluate 59 questions spanning single-fact recall, cross-unit concept linking, synthesis and explanation, and currency after a syllabus revision, scored by an LLM judge against a human-authored rubric. Both representations answered single-fact questions about equally well (9.33 vs. 9.96 of 10), but diverged sharply on questions requiring links across course units. The compiled wiki remained accurate and grounded (9.93; 100% grounded in cited sources), while retrieval scored lower and was markedly less grounded (8.14; 64%). The wiki's citations let students and instructors trace any claim back to the lecture that introduced it, adding a layer of dynamic retrieval that machine learning courses require. While further testing is needed, instructors using AI to support learning in ML courses should consider wiki-based structure for its potential to support foundational elements of best practice.
1 Introduction
Course-specific tutors need answers grounded in vetted materials, yet fixed retrieval strategies can behave differently across question types. This study tests whether structuring course knowledge at ingest addresses that gap and whether the choice matters for questions that connect ideas across a course.
1.1 Motivation
- Fixed retrieval strategies in prior course RAG work performed differently across question types, motivating a comparison of ingest-time structure with query-time chunk retrieval.Replacing text with top-ranked images improved context recall for specific questions but degraded faithfulness and factual correctness for generic questions.Source: §1.1 · p. 1
1.2 Effective Teaching Practices with Chatbots
- Course tutors need to reflect course structure, sequencing, and cross-references, while supporting student discernment rather than simply resolving questions.Prior classroom work describes mismatches between benchmark assumptions and real student interactions, and emphasizes that tool outcomes depend on instructional embedding.Source: §1.2 · p. 2
1.3 The Gap
- No prior study had isolated knowledge representation on an authentic course corpus, despite evidence that structure can help relational questions and hurt simple lookup.The cited comparison was on a small multi-domain research corpus, not authentic classroom data.Source: §1.3 · p. 2
1.4 Research Questions
- The study asks whether a compiled wiki improves classroom answer quality, how effects vary by question type, and whether either approach is better for learning in an ML class..Source: §1.4 · p. 2
1.5 Contributions
- The paper contributes an authentic-course wiki-versus-RAG comparison, an evaluation protocol with a five-part taxonomy, and directional evidence about cross-topic synthesis and groundedness.The groundedness finding complicates retrieval-centered accounts of RAG shortcomings.Source: §1.5 · p. 2
2 Background
Prior work motivates evaluating course-grounded systems not only by answer quality, but also by question type, provenance, and educational fit. This background frames the study’s comparison of retrieval and structured representations.
2.1 Learning Outcomes from AI Chatbots
- In prior studies, AI chatbots and course RAG systems have been associated with learning benefits, while outcomes vary with instructional mode and tool design.The cited meta-analyses report stronger effects in higher education and for shorter interventions in one analysis, and instructional mode as a significant moderator in another.Source: §2.1 · pp. 2–3
2.2 Retrieval Augmented Generation
- RAG grounds generated answers in external documents, but its benefits depend on retrieval relevance, task type, and evaluation design.The cited literature frames RAG as a reliability layer that can support knowledge updates without retraining.Source: §2.2 · p. 3
2.3 Structured and Graph Based Retrieval
- Graph-enhanced RAG supports multi-hop reasoning and explainability, but prior results show trade-offs between relational queries and simple lookup.Reported findings range from gains on multi-hop benchmarks to underperformance against naïve RAG on other question sets.Source: §2.3 · p. 3
2.4 RAG in Education
- Educational RAG should align answers with vetted course materials and the local curriculum, while overly rigid grounding can reduce helpfulness.Prior work reports educational graph-RAG retrieval accuracy of 91.4% and improved learner satisfaction, but representation itself had not been isolated on an authentic course corpus.Source: §2.4 · p. 3
2.5 Prior Work: Multimodal RAG on DS3001
- Prior multimodal course RAG work found that context composition mattered: adding images helped some questions but hurt faithfulness and correctness on generic questions.This motivated comparing ingest-time structure with query-time chunk retrieval.Source: §2.5 · p. 3
2.6 Evaluating RAG Systems
- The evaluation retains RAGAS metrics for continuity and reports accuracy, synthesis quality, citation alignment, and cost separately to avoid a single aggregate winner.The cited rationale notes that lexical overlap metrics can be unstable under LLM output stochasticity.Source: §2.6 · p. 3
3 Study Design
The study compares vector RAG with an LLM-compiled wiki while holding key answering conditions constant. Its design covers multiple course-question types and evaluates answer quality, grounding, and other endpoints.
3 Study Design
- The controlled head-to-head comparison holds the corpus, base LLM, prompts, and question set constant so knowledge representation differs between arms.The design combines a preregistered comparison model with a classroom-focused evaluation approach.Source: §3 · p. 3
- Vector RAG reuses the prior study’s best-performing configuration, while the wiki compiles linked concept pages with source citations at ingest and navigates pages at query time.The extraction pipeline was reused unchanged to avoid attributing arm differences to preprocessing.Source: §3 · pp. 3–4
- The wiki-versus-RAG comparison matches context budget and holds the generator, system prompt, question set, and decoding parameters constant; a hybrid arm is deferred.Matching context budget avoids conflating representation with simply giving the model more context.Source: §3 · p. 4
- Questions cover single-fact lookup, concept linking, explanation and synthesis, contradiction handling, and curriculum updates.Questions were human-generated with reference answers and word limits, and balanced across course topics.Source: §3 · p. 4
- Evaluation adds citation alignment, blinded rubric-based synthesis scoring, knowledge-shift robustness, and cost to the retained RAGAS metrics.Cost includes ingest compute, query tokens, and latency.Source: §3 · p. 4
- Both indexes use a frozen corpus snapshot, with 20,000 bootstrap rounds for metric stability and blinded judging for synthesis and preference.The judge scores correctness from 1 to 10 and assigns a binary groundedness flag based on whether claims trace to the answerer’s context.Source: §3 · p. 4
- The preregistered hypotheses predict vector RAG at least matching the wiki on single-fact lookup, wiki gains on multi-hop and synthesis questions, and endpoint trade-offs..Source: §3 · p. 4
- The evaluated wiki corpus has 12 pages, and the 59 retained questions comprise 45 answerable from one page and 14 requiring cross-page connections.The 45 topic-page questions are evenly distributed across three pages; question labels also distinguish factual, conceptual, synthesis, and update demands.Source: §3 · p. 4
4 Results
The wiki led vector RAG on both question classes, with a larger advantage on cross-page questions. Retrieval often found relevant material, but many ungrounded answers still added unsupported details; the hypotheses therefore support a wiki advantage, qualified by the study’s scale and design.
4 Results
- The wiki led vector RAG on both measures; the groundedness gap was 17 points (95% CI [7, 27]) and the raw-score gap was 0.90 points (95% CI [0.42, 1.44]).Both confidence intervals excluded zero; Table 1 covers 59 questions, with both arms graded 59/59 and zero errors.Source: §4 · pp. 4–5, Table 1 · p. 5
- 9.96 vs. 8.14 on cross-page questions and 9.93 vs. 9.33 on topic-page questions; the wiki’s advantage was roughly three times larger cross-page in score and groundedness.Cross-page groundedness was 100% for the wiki versus 64% for RAG; topic-page groundedness was 98% versus 87%.Source: Table 2 · p. 5, §4 · p. 5
4.1 Where Vector RAG Lost Ground
- 43 of 45 topic-page questions (96%) retrieved a chunk from the question’s source page in the top five; the two misses retrieved short navigational pages instead.Both misses concerned questions about ML Bias interventions and the canonical k-means slide deck, and both answers scored low and were marked ungrounded.Source: §4.1 · p. 5
- Only 2 of vector RAG’s 11 ungrounded answers were retrieval misses; the other 9 had relevant material but added unsupported details.The authors suggest chunked context may invite fabrication independently of retrieval success.Source: §4.1 · p. 5
4.2 Relation to the Hypotheses
- H1 was not well supported: the topic-page score gap was statistically distinguishable, while the 87% versus 98% groundedness gap was the sole comparison whose confidence interval touched zero.The groundedness shortfall was directionally consistent with a gap but not statistically established at n = 45.Source: §4.2 · p. 5
- H2 was supported: the wiki’s cross-page score advantage was roughly three times its topic-page advantage, and every cross-page comparison’s confidence interval excluded zero.The cross-page cell contained only n = 14 questions, the study’s smallest cell.Source: §4.2 · p. 5
- H3 was not supported at this corpus scale: the wiki led both tested question classes, but the authors treat this as provisional given a possible information advantage for the wiki arm.They also suggest the structure may fit courses organized week by week, while framing that implication as potential.Source: §4.2 · p. 5
5 Discussion
The wiki improved response quality over vector RAG under matched conditions, with a larger advantage on cross-page synthesis than on single-topic lookup. The discussion links explicit structure to grounded answers and student source-checking, while treating the findings as directional and limited in scope.
5.1 RQ1/RQ2: Structure at Ingest vs. Retrieval at Query
- The wiki improved response quality over vector RAG under matched conditions, with an advantage roughly three times larger on cross-page synthesis than on single-topic lookup.The authors describe the finding as directional rather than definitive because it comes from a single run on a 21-vector index.Source: §5.1 · pp. 5–6
5.2 Structure Also Reduces Fabrication, Not Just Retrieval Misses
- Only 2 of vector RAG’s 11 ungrounded answers were retrieval misses; the other 9 retrieved relevant material but added unsupported detail.The authors suggest that linked concept pages may constrain generator elaboration as well as provide access to relevant content.Source: §5.2 · p. 6
5.3 RQ3: Impacts on Learning
- Citation traceability lets students check claims against source materials, keeping some evaluative responsibility with students rather than the tutor.The discussion frames ungrounded but fluent answers as a greater classroom risk than wrong answers students can check against sources.Source: §5.3 · p. 6
- A 9+ judge score cannot distinguish durable understanding from answers a rubric rewards, so high scores alone do not establish pedagogical quality.The authors recommend evaluating whether students can follow the reasoning path, not only whether the terminal answer earns a high score.Source: §5.3 · p. 6
- Explicit links can surface recurring concepts across units, such as squared-error loss in linear regression and regression trees, for retrieval in a new context.The discussion connects delayed retrieval and interleaving to durable learning and transfer.Source: §5.3 · p. 6
5.4 Limitations and Threats to Validity
- Generalization beyond DS3001 is untested, and re-tuning Arm A could narrow the reported gap because its inherited configuration may confound the comparison.Each arm was answered and graded once per question; the study also did not measure cost, latency, or ingest compute, and describes the results as directional evidence for further exploration.Source: §5.4 · pp. 6–7
6 Conclusions
The compiled wiki outperformed vector RAG overall, with larger advantages on cross-page questions, though the comparison gives the wiki preselected context. The authors call for classroom validation and broader comparisons.
6 Conclusions
- 9.95 vs. 9.05 out of 10, and 98% vs. 81% groundedness, favored the compiled wiki over vector RAG overall.Both 95% confidence intervals for the differences exclude zero.Source: §6 · p. 7
- On cross-page questions, score and grounded-rate gaps widened to 1.79 and 36 points, with every interval excluding zero.The respective gaps were 0.63 and 11 points on topic-page questions; the topic-page score gap was 9.33 vs. 9.96, while groundedness was the only endpoint not significant at 95%.Source: §6 · p. 7
- The wiki’s advantage is not a general result: its fixed indexed sections covered all 59 questions, avoiding a context-selection step that could fail.Vector RAG retrieved five chunks from a 21-vector index and missed on 2 of 45 topic-page questions.Source: §6 · p. 7
- Only 2 of vector RAG’s 11 ungrounded answers were retrieval failures; in the other 9, the answerer added details unsupported by retrieved excerpts.The authors infer that a substantial share of the groundedness gap originates in generation, so perfect retrieval would not close it.Source: §6 · p. 7
- The study establishes an end-to-end pipeline and discriminating endpoints, and suggests linked wikis may merit consideration in courses delivering dynamic material.The authors say further testing is needed.Source: §6 · p. 7
6.1 Future Work
- Classroom testing with actual students, larger-corpus and cost comparisons, harder questions, and additional hybrid, graph-RAG, and zero-shot arms remain future work.The zero-shot comparison is described as likely belonging to a different study focused on model quality rather than course-content retrieval.Source: §6.1 · p. 7