PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Premises Multi-Course RAG Tutoring for Business Education: Hardware-Software Trade-offs in a Campus AI Tutor

Sidney Shapiro, Joshua Lindemann

arXiv:2610.02510v1cs.AIcs.IR

TL;DR

CourseChat addresses how to ground campus AI tutoring in assigned materials while keeping textbooks and student dialogue on institutional infrastructure. It combines local multi-course tutoring with model trials and software changes, retaining the 8B production model because alternatives have not demonstrated an overall advantage.

  • Problem

    Campus AI tutoring must ground help in assigned course materials while protecting textbooks and student dialogue from external APIs and respecting hardware, concurrency, and course-isolation constraints.

    Source: §1 · p. 1

  • Method

    CourseChat combines a replicated local RAG architecture and six course-scoped offerings with model comparisons and changes to retrieval-topic resolution and prebuilt practice.

    Source: §1 · p. 2, §9.4 · p. 18, §10 · p. 20

  • Results

    The production system retains llama3.1:8b because evaluated alternatives have not demonstrated an overall advantage across grounded explanation, continuity, latency, and serving compatibility.

    Source: §10 · p. 20

  • Takeaways & Limitations

    Model choice, evidence preparation, dialogue handling, serving compatibility, and product design require joint consideration in this local business-school deployment.

    Source: §10 · p. 20

    Engineering regressions are not representative of student use, faculty usefulness and factual-support ratings remain pending, and no learning-outcome or controlled classroom study is reported.

    Source: §9.8 · p. 20

Abstract

from arXiv · show

Campus AI tutors based on retrieval-augmented generation (RAG) must ground answers in assigned course materials while keeping textbooks and student dialogue on institutional infrastructure. We present CourseChat, an on-premises, multi-course RAG tutor for undergraduate business education, deployed behind a campus web gateway and intended for use embedded in Moodle. Six isolated course offerings, each keyed by its own course reference number (CRN), share twin-edge AI hosts running a FastAPI service, a local vector database, and a local large language model (LLM) served by Ollama. We report two generation-model bake-off rounds, a separate fixed-evidence source-fidelity comparison, and conversation and quiz audits. Several larger models failed the classroom speed gate, but a 12B model and a 7B alternative passed. A separate mixture-of-experts candidate improved some corrections while introducing new factual and continuity errors. We therefore retain the 8B production model pending a demonstrated overall improvement, rather than claiming that 8B is universally optimal. Software changes improved follow-up topic resolution while preserving course scope; 435 prebuilt questions across 65 modules decouple practice from live generation. The results support treating model choice, evidence selection, serving compatibility, and product design as a joint engineering decision. They do not establish learning gains: faculty ratings, peak-load capacity, and complete public-gateway acceptance remain separate evaluation needs.

1 Introduction

CourseChat addresses the challenge of providing course-grounded tutoring on institutional infrastructure while managing limited hardware, classroom concurrency, and strict course isolation. The paper presents a multi-course RAG system and frames its design around diagnosing whether poor answers stem from dialogue context, retrieval, filtering, or generation.

System and contribution

  • CourseChat serves six offerings through one application binary while isolating their data and routing, with conversation grounded in course materials and module-scoped practice quizzes.Quiz support includes answer keys, hints, and source excerpts.Source: §1 · p. 1

Problem framing

  • Poor answers can arise from lost context, missing evidence, rejected relevant passages, or a generator misrepresenting retrieved evidence, so a larger model addresses only some failures.Model benefits must also be weighed against response time and serving compatibility.Source: §1 · p. 2

Contributions

  • The paper contributes a replicated local RAG architecture, a generation-model comparison with fixed retrieval and token limits, and a separate model-and-response-contract experiment.It also connects observed failures to retrieval-topic resolution, bounded evidence selection, and prebuilt practice banks.Source: §1 · p. 2

Pilot motivation

  • CourseChat was designed to complement existing AI services with support tailored to local courses, teaching materials, and learning expectations, encouraging students to engage with assigned material and work independently.These are intended educational aims, not reported learning outcomes.Source: §1 · p. 2

2 Related work and positioning

The related work positions CourseChat at the intersection of instructional interaction, evidence-grounded explanations, and reliable institutional serving. It motivates evaluating these dimensions separately rather than treating fluent dialogue as proof of tutoring effectiveness.

2.1 Tutoring, practice, and self-regulated learning

  • Tutoring research distinguishes feedback by interaction granularity, while evidence on learning effects varies with comparison conditions, assessments, and implementation.Retrieval practice and formative feedback motivate attempts, optional hints, and explanatory feedback, but do not establish the effectiveness of CourseChat’s multiple-choice banks.Source: §2.1 · pp. 2–3

2.2 Generative AI assistance and independent learning

  • Studies of generative AI assistance report different outcomes for assisted performance and unassisted learning, making instructional design and help policy part of the intervention being evaluated.CourseChat’s connection to assigned materials and a path from attempts to limited assistance is a design rationale, not a claim of reproduced learning outcomes.Source: §2.2 · p. 3

2.3 Course grounding and source fidelity

  • Retrieval can control which course materials support generation, but citation quality and relevant evidence do not guarantee that an explanation interprets its conditions correctly.Longer context is not necessarily a complete fix; prior work motivates testing evidence selection and organization.Source: §2.3 · pp. 3–4

2.4 Evaluation and serving constraints

  • RAG evaluation frameworks separate context, faithfulness, and answer quality, while serving research highlights memory demands during concurrent generation beyond model weights.Neither automated retrieval or citation checks nor CourseChat’s serial trials establish student understanding or classroom capacity.Source: §2.4 · p. 4

2.5 Position of the CourseChat pilot

  • CourseChat integrates course-specific evidence, conversation continuity, prebuilt practice, and a shared local generator across six separately scoped offerings in a business-school setting.Its contribution is deployment evidence and observed trade-offs, not a new retrieval algorithm, superiority over available AI services, or a controlled learning-outcome comparison.Source: §2.5 · pp. 4–5
  • Assessing whether a local tutor supports independent effort requires measures such as unaided problem solving, retention, source use, and hint-request patterns.The study supplies implementation and diagnostic evidence for designing that evaluation.Source: §2.5 · p. 5

3 System overview

The system organizes student interactions around the selected course offering, combining conversation, practice quizzes, and scoped retrieval with a replicated local deployment. Its interface and architecture distinguish live generation from stored practice and keep server-local audits separate from gateway acceptance.

3.1 Student experience

  • Moodle course context establishes the full-course, module, or named-activity scope, while the header displays the course, instructor, CRN, and selected material.Complete public-gateway acceptance remains a separate deployment task.Source: §3.1 · p. 5
  • Conversation answers cite passages from the selected offering, and source cards expose excerpts and available chapter, section, or page information.The relevance score measures passage-query match, not the probability that an explanation is correct; students may raise but not lower the course-configured relevance floor.Source: §3.1 · p. 5
  • Follow-ups use recent dialogue to resolve references before retrieving current course evidence; ambiguous references prompt clarification, and earlier assistant text is not authoritative evidence.Fallback may broaden an explanation using the same course offering, never another segregated course; the system reports an evidence gap if qualifying evidence remains unavailable.Source: §3.1 · p. 5

3.2 Logical modules

  • The application routes course, mixed, and dependent follow-up requests through scoped retrieval before generation, while social and clarification routes do not retrieve course content.Quiz practice follows stored-item and deterministic-grading paths; responses return through the same API and UI.Source: Figure 1 · p. 6, §3.2 · p. 6

3.1 Student experience

  • Quiz practice uses stored items and deterministic grading, with 435 prepared questions across 65 modules and no live generation.The banks await instructor review, and technical audits establish reproducible behavior rather than educational validity.Source: §3.1 · pp. 5–6

3.3 Deployment topology

  • Two local nodes each run the full stack, while index snapshots transfer between them; reported node audits bypass the gateway and do not establish complete public course-slug acceptance.Each host has on the order of 128 GB unified memory shared across the application, vector database, embedding/rerank models, and Ollama; worker count alone does not establish classroom capacity.Source: §3.3 · p. 7, Figure 2 · p. 7

4 Hardware–software trade-off strategy

The trade-off strategy treats model selection as an operating decision constrained by answer usefulness, classroom wait, and serving requirements. It separates generator comparisons from changes to retrieval, practice delivery, and deployment workflows.

Operating objective

  • Candidate promotion requires median complete-answer latency no more than 30% above the same-round production 8B baseline, usable overlapping requests, preserved course scope, and meaningful improvement in reviewed answers.These are promotion requirements, not evidence that every corresponding experiment was completed.Source: §4 · p. 7

Component choices

  • Model family and size affect generation, while retrieval depth, context limits, and reranking affect evidence selection and when generation can begin.Server-wide application profiles package performance settings but do not select a course or change collection boundaries.Source: §4 · p. 8

Controlled comparisons and workflow

  • The model bake-off keeps the serving retrieval stack fixed, while the structured-answer experiment remains separate; quiz authoring is moved out of the request path.Document extraction and review occur on a workstation, leaving serving hosts responsible for indexing and inference.Source: §4 · p. 8

Intervention overview

  • Table 1 distinguishes generator comparisons from retrieval and practice-delivery changes and summarizes observed outcomes and resulting design decisions.Section 6 supplies model-by-model measurements.Source: §4 · p. 8

5 Design choices

CourseChat separates course-scoped evidence, conversational generation, and prepared practice while sharing local infrastructure across offerings. The design also distinguishes server-local completion time from gateway-visible responsiveness and keeps corpus preparation off the classroom hosts.

5.1 Isolation without six fine-tuned LLMs

  • Each CRN has its own course collection and session scope, while offerings share the generator; content updates change indexed evidence rather than model weights.Request handling and session validation help preserve scope, including when two offerings share a catalog code.Source: §5.1 · p. 8

5.2 RAG for content; prompts for pedagogy

  • Course materials are indexed as evidence, while prompts and optional profiles set tutoring tone; revisions are re-ingested and replicated rather than used to retrain the LLM.Chapter scope is applied through request validation and retrieval metadata filters.Source: §5.2 · pp. 8, 10

5.3 Prebuilt quizzes

  • The six course banks contain 435 questions across 65 modules, with structural and API grading checks passed but instructor pedagogical review still pending.Prepared banks support inspectable hints, keys, explanations, and excerpts, at the cost of a finite question pool and less spontaneous adaptation.Source: §5.3 · p. 10

5.4 Conversational turns over shared course evidence

  • A shared resolver raised intended-topic follow-up rates to 90–100% across five courses without changing the generator, while preserving course-matched, bounded browser-supplied history.The router uses deterministic rules first, then a short classification call for ambiguity; content questions retrieve fresh evidence.Source: §5.4 · pp. 10–11

5.5 Streaming vs. raw latency

  • Server-local complete-answer time and gateway-visible responsiveness are distinct: incremental SSE delivery can show text earlier without reducing total inference work.The reported bake-off measures completion time on the node, not browser-perceived waiting time through IIS.Source: §5.5 · p. 11

5.6 Workstation / Node A / Node B pipeline

  • OCR and page review run on a workstation; reviewed corpora are ingested on Node A and snapshot-replicated to Node B to maintain parity.The classroom hosts do not run the OCR engines.Source: §5.6 · p. 11

6 Model bake-off

The bake-off compares generators under fixed retrieval and serving constraints, then separately tests a model-and-response configuration with fixed evidence. Round 2 yielded faster alternatives for further review, but the evidence does not establish an overall quality win or justify replacing the production 8B model.

6.1 Protocol

  • The bake-off held the profile, 8,192-token context, 768-token output cap, retrieval settings, and probe set fixed, with the embedding encoder and reranker unchanged.The twelve probes checked keyword presence and citations; they were not a complete pedagogical rubric.Source: §6.1 · pp. 11–12

6.2 Results

  • Round 2’s Qwen2.5 7B ran at 1.01× baseline with a 0.93 completeness proxy versus 0.91, while Mistral-Nemo 12B ran at 1.24×; production remains 8B pending faculty review.No larger round-1 model cleared the speed gate, so these are shortlist candidates, not a parameter-count rule or confirmed promotion.Source: Table 3 · p. 12, Table 2 · p. 12, §6.2 · pp. 12–13

6.1 Protocol

  • Twelve automated passes did not establish the 95% supported-claim target or paired-review improvement, and reported point estimates lack run uncertainty intervals and a concurrency study.A speed pass is only one prerequisite for promotion.Source: §6.1 · p. 12

6.2 Results

  • Some zero-pass results reflected serving incompatibility: the tested Qwen MoE supplied no student-visible answer, while Granite returned empty content through the application despite answering through raw Ollama.These outcomes characterize evaluated configurations, not the models’ underlying ability to answer.Source: §6.2 · p. 13

6.3 Separate fixed-evidence fidelity comparison

  • In the fixed-evidence comparison, nine answers passed structural checks—including one materially wrong claim—and three were withheld, including one answerable follow-up.Validation checked structure, current source identity, quotations, and cited source numbers; rejected drafts received a fixed gap response.Source: §6.3 · p. 13
  • The fixed-evidence candidate produced three substantial improvements, two modest improvements, three ties, one mixed result, and three regressions; it was not activated.Warm medians were 3.469 s for production and 2.997 s for the candidate, but new factual and continuity errors prevented an overall quality win.Source: §6.3 · p. 13

7 Content and RAG pipeline

The content pipeline combines scoped retrieval with source-provenance checks and distinct quiz publication paths. Automatic regeneration did not pass its complete gate for the four textbook courses, while independently reviewed additions remained separate from chat evaluation and faculty approval.

Quiz publication paths

  • Figure 3 separates automatic quiz building, which requires corpus quality gates, from independently reviewed additions; both paths require artifact validation, and faculty pedagogical review remains pending.The reviewed-additions path does not clear failed chat evaluations.Source: Figure 3 · p. 14

Retrieval and scope

  • Retrieval uses hybrid course search, metadata filters, cross-encoder reranking, and evidence tiers that prefer original lectures and readings over generated study aids.When a locked module or chapter yields no results, a counted fallback may widen scope within the same course offering and report that it did so.Source: §7 · pp. 13–14

Content and corpus scope

  • The four textbook-course corpora contain 15,730 prepared passages, and course metadata distinguishes teaching modules from textbook chapters.This is a prepared-corpus count, not the total number of Qdrant points across all six offerings.Source: §7 · p. 14

Provenance and review

  • Original-file hashes, reviewed page metadata, stable chunk IDs, and ingestion receipts support checking quotations against current sources, while scanned tables and formulas still need numerical review.Generated study aids are not treated as primary evidence for reviewed quiz items.Source: §7 · p. 14

Quiz publication paths

  • None of the four textbook courses passed complete automatic regeneration; 45 added items instead followed a primary-source authoring and independent item-review path.That path did not clear failed chat evaluations or imply faculty approval, and banks and review histories require separate deployment to both nodes.Source: §7 · pp. 14–15

8 Evaluation

The conversation audits show improved follow-up topic resolution while preserving course scope, but topic hits do not establish explanation correctness. Separate audits found substantive source-fidelity errors despite valid citations and verified consistent quiz behavior, while routing checks and generative tutoring latency remain distinct.

8.1 Multi-turn conversation

  • All five courses reached the 90% follow-up target with 400 scope checks passed and zero API errors; Course 1 improved from 31 to 60 successful follow-ups as median latency rose from 4.92 to 5.81 s.Five-course median per-turn latency ranged from 3.24 to 6.19 s; the Course 1 replay was separate. Topic-hit scores measure subject retention, not whether explanations are correct.Source: §8.1 · p. 15

8.2 Source fidelity and routing checks

  • The ten-request textbook audit found three supported replies, four retrieval misses, two source contradictions, and one partly grounded answer, although every citation resolved to its source.The selected diagnostic requests are not a representative sample for estimating a population error rate.Source: §8.2 · p. 16
  • The later router extension passed 306 focused tests and documented live probes on both nodes, but those checks do not retroactively validate the earlier conversation runs.Its frozen contract included 32 crossover cases.Source: §8.2 · p. 16

8.3 Quizzes

  • Audits exercised all 435 questions across 65 modules, checking source bindings, grading, hints, and cross-course quiz-state isolation.Instructor review of correctness, distractors, difficulty, and instructional coverage remains incomplete.Source: §8.3 · p. 16
  • Warm quiz-operation medians were 4.8 ms on Node A and 5.8 ms on Node B, measurements that do not represent generative tutoring latency or browser response time.These were server-local timings from a separate 166-request live API audit.Source: §8.3 · p. 16

9 Discussion

The discussion retains the 8B generator because no evaluated replacement has demonstrated the full combination needed for promotion, despite alternatives that passed speed checks or corrected some errors. It treats retrieval, serving compatibility, abstention, course boundaries, and maintenance as distinct engineering concerns, with quality and operational limits still unresolved.

9.1 Why retain the 8B classroom generator?

  • The 8B generator remains in production because no replacement has demonstrated a clear grounded-tutoring benefit together with acceptable speed, continuity, integration, and operational readiness.Qwen 7B and Mistral Nemo 12B met the prescribed speed allowance, but passing twelve keyword-and-citation checks did not establish the required benefit; faculty comparison of actual explanations remains pending.Source: §9.1 · p. 16

9.2 A model is evaluated through its serving contract

  • The 35B-A3B candidate corrected some baseline misconceptions but also introduced factual and continuity errors, and its comparison changed the model, prompt, schema, validation, and transport together.The serving stack did not consistently apply structured output to thinking-capable chat requests with thinking disabled; a direct generation request resolved the observed JSON-formatting problem, while factual and continuity failures remained.Source: §9.2 · p. 17

9.3 Grounding requires useful answers and appropriate abstention

  • All twelve final-candidate outputs were valid JSON and nine passed quotation and citation checks, yet an accepted answer strengthened a conditional benefit into a guarantee.Two of three withheld answers lacked sufficient evidence, while one was an answerable follow-up; exact quotation does not verify that an explanation preserves source meaning.Source: §9.3 · p. 17

9.4 Retrieval and conversation control are complementary to generation

  • Resolving follow-ups improves retrieval topic alignment without relaxing course or module boundaries, but correct topics and retrieved passages do not ensure faithful explanations.Clean-concept probes found evidence for 19 of 24 affected topics; bounded expansion recovered 11 original queries without lowering the relevance floor. The later router uses deterministic rules for obvious cases and the resident 8B model for ambiguous ones.Source: §9.4 · pp. 17–18

9.5 On-premises control carries operational obligations

  • CRN-specific collections and routing keep offerings scoped while sharing one generator, but the recorded probes do not establish general security, privacy compliance, or enrollment enforcement.Two local stacks provide an alternative during node maintenance but require synchronized indexes, quiz banks, evidence ledgers, code, configuration, and artifacts; deployment costs were not measured.Source: §9.5 · p. 18

9.6 Prebuilt practice moves uncertainty into a reviewable build stage

Prebuilt practice makes quiz behavior reviewable and stable while limiting variation and requiring renewed review when course materials change. The paper proposes faculty and student evaluation workflows, but its current engineering audits do not establish instructional quality, classroom readiness, or learning gains.

9.6 Prebuilt practice moves uncertainty into a reviewable build stage

  • Prebuilt questions avoid a generation request on each practice turn and make hints and explanations stable, but finite banks require rebuilding or re-review when materials change.Six or seven questions per module provide availability across the module map, not exhaustive coverage of learning objectives.Source: §9.6 · pp. 18–19

9.7 Faculty review, student uptake, and educational value

  • The proposed next stage pairs faculty review of questions and supporting excerpts with student-uptake, feedback, and learning assessments that distinguish assisted performance from independent learning.Suggested measures include return visits, usefulness and trust feedback, attempts before revealing answers, delayed assessments, new problems without the tutor, and faculty workload; streamlined review and approval remain future work.Source: §9.7 · p. 19

9.8 Limits of the evidence and next evaluation priorities

  • The reported engineering regressions are not representative student-use or load studies, and faculty ratings, learning outcomes, concurrent browser-to-gateway timing, and complete gateway readiness remain unresolved.Latency measurements were predominantly serial and server-local; the two-request overlap gate remains a promotion requirement, and public gateway readiness and verification for every course slug remained open.Source: §9.8 · pp. 19–20

10 Conclusion

CourseChat’s contribution is a shared tutoring system that preserves course-specific instructional context while supporting conversation and inspectable practice. The authors retain the 8B model conditionally and identify instructor-reviewed content and further evaluation as next steps.

10 Conclusion

  • CourseChat uses CRN-scoped collections, traceable ingestion, conversation controls, and prebuilt quizzes so one generator can serve offerings while course content and interaction are developed and reviewed separately.The design also brings preparation effort, limited question-bank variety, and replicated-artifact consistency trade-offs.Source: §10 · p. 20
  • The 8B model remains in production because evaluated alternatives have not shown an overall advantage across grounded explanation, continuity, latency, and serving compatibility.Credible 7B and 12B candidates remain available, and the choice can change without rebuilding the course architecture.Source: §10 · p. 20
  • Future ingestion should add instructor-reviewed cases, worked examples, lecture explanations, and context; adding documents alone does not ensure better answers without evaluating extraction, mapping, retrieval, and fidelity.A shared, versioned corpus could support multiple interaction modes without separate knowledge bases for each.Source: §10 · p. 20
  • Instructor review of current quiz banks is still pending, and the next pilot should assess sustained uptake, interaction-mode acceptance, faculty workload, and independent learning through delayed, unassisted assessments.Comparisons with existing resources and general-purpose assistants are also proposed.Source: §10 · pp. 20–21

A Campus request path (deployment contract)

The intended request path begins in an LMS embed and passes through a campus gateway to a course-configured API. The selected course and turn mode determine the response path, and course-answer citations remain scoped to that course.

A Campus request path (deployment contract)

Related papers

Loading 2610.02510v1…