PaperlayerView exact revision on arXiv

Source-linked AI summary

TeachMateGPT: A Multi-Agent Knowledge-Grounded Framework for Pedagogical Assessment Generation from Science Curriculum Materials

Fatema Tuj Johora Faria, Mukaffi Bin Moin, M. F. Mridha, Jubayer Al Mahmud

arXiv:2608.13708v1cs.CLcs.AI

TL;DR

Existing RAG systems struggle to generate reliable, curriculum-aligned science assessments beyond isolated questions and weak-evidence safeguards. TeachMateGPT addresses this with hierarchical retrieval, fail-closed multi-agent generation, and source-attributed verification, improving faithfulness from 0.68 to 0.96 and answer relevancy from 0.60 to 0.89 over vanilla RAG.

  • Problem

    Existing assessment-generation systems mainly address isolated pipeline components and lack reliable, pedagogically constrained curriculum grounding for multi-format science items.

  • Method

    TeachMateGPT combines COPE’s hierarchical curriculum index with a fail-closed multi-agent pipeline and SAVER teacher-reviewed source-attributed verification.

  • Results

    0.54 to 0.92 context precision and 0.58 to 0.91 context recall were achieved, while ablations showed retrieval and verification had complementary roles.

  • Takeaways & Limitations

    The findings support combining reliable retrieval, evidence-grounded verification, and refusal to generate when curriculum evidence is insufficient.

  • Takeaways & Limitations

    The text-only framework cannot author assessment items whose stimuli or stems depend on textbook figures, diagrams, or other visual objects.

Abstract

from arXiv · show

Automatically generating textbook-grounded assessment items can reduce science teachers' workload, but existing retrieval-augmented generation (RAG) systems rely on flat retrieval, support only single-question generation, lack safeguards against weak evidence, and are ill-suited to low-resource, board-exam-structured curricula. We address these limitations with TeachMateGPT, a multi-agent system contributing four advances to curriculum-grounded science-assessment authoring. (i) COPE, a hierarchical knowledge base replacing token-window chunking with a multi-resolution index that segments documents along syllabus structure and links them at three granularities via a traversable graph-based lineage, matching evidence to each topic's instructional level. (ii) A staged, fail-closed agent pipeline replacing one-shot retrieve-then-generate: routing gates search, retrieval fuses dense and lexical evidence under a coverage gate that withholds generation on insufficient evidence, and specialist agents draft objective and constructed-response items. (iii) SAVER, a source-attributed verification protocol scoring faithfulness, relevance, and hallucination risk against retrieved evidence, applying stricter grounding checks across each creative question's four sub-parts, paired with teacher-in-the-loop evaluation rather than automatic filtering. (iv) NCTB-SciGen8, a curriculum-grounded dataset of 198 items (143 multiple-choice, 55 creative questions) spanning all 14 chapters of the NCTB Class 8 science textbook, produced by the pipeline and rated by three practicing teachers. TeachMateGPT raises faithfulness (0.68 $\rightarrow$ 0.96) and answer relevancy (0.60 $\rightarrow$ 0.89) over a vanilla RAG baseline.

1 Introduction

TeachMateGPT addresses the limitations of vanilla retrieve-then-generate assessment authoring with a curriculum-grounded, multi-agent workflow for Bangla Class 8 NCTB science. It combines hierarchical curriculum indexing, staged evidence-aware generation, and verification to support teacher review before classroom use.

  • Motivation: Unconstrained LLM assessment generation can produce unsupported facts, chapter drift, weak distractors, and invalid board structures.These risks motivate knowledge-grounded, multi-format generation under pedagogical constraints.
  • Problem: Vanilla retrieve-then-generate pipelines remain insufficient because requests need clarification, retrieval must handle curriculum context and noisy sources, formats impose distinct constraints, and items require validation.Educational systems therefore coordinate routing, retrieval, generation, and verification through agentic workflows.
  • Contributions: COPE organizes textbook content at multiple instructional resolutions while preserving ingest-time lineage and neighbor links for traversable retrieval.Removing COPE causes the largest ablation drop in answer relevancy and substantially reduces stimulus realism.
  • Contributions: A staged, fail-closed multi-agent pipeline routes queries, performs hybrid retrieval, refines evidence, and generates items by assessment format.It incorporates CCI for evidence restoration and CCR for redundancy reduction.
  • Contributions: The framework frames trustworthy assessment authoring as source-attributed verification combined with teacher-in-the-loop review before classroom use.The research questions explicitly target acceptance, editing, or cautioning of generated assessments.

2 Related Work

Related work applies retrieval-augmented generation and agent-based architectures to educational assessment tasks, but existing studies largely advance isolated pipeline components rather than complete systems. Representative approaches cover evidence retrieval, item creation, document analysis, question construction, and evaluation, with capabilities still missing across systems.

  • Retrieval-Augmented Generation: RAG connects language models to external knowledge sources and supports educational assessment generation from course documents and examination archives.Surveys emphasize RAG for knowledge-intensive educational tasks requiring reliable access to instructional materials, including MCQ generation with answer keys.
  • Agent-Based Architectures: Agent-based educational systems divide assessment workflows among specialized components for item creation, retrieval, question construction, document analysis, and evaluation.CODE-GEN uses human-in-the-loop RAG agents for coding-comprehension MCQs, with separate modules for creation and quality assessment.
  • Limitations of Existing Approaches: Recent assessment-generation studies primarily advance isolated pipeline components, leaving capabilities missing across representative systems.A summary table highlights these gaps among existing approaches.

3 The TEACHMATEGPT Framework

TeachMateGPT is a staged, multi-agent framework that converts teacher requests into curriculum-grounded assessments through hierarchical knowledge construction, routed retrieval, evidence validation, generation, and source-attributed verification. Its fail-closed workflow preserves flagged assessments and reports for teacher decisions rather than automatically regenerating or discarding them.

  • Assessment generation and verification: The end-to-end pipeline generates textbook-grounded MCQs and board-style creative questions, then verifies them with source-attributed evidence support, faithfulness, and relevance checks.The framework maps teacher queries through COPE, intent analysis, hybrid retrieval, coverage checks, generation, verification, and auditable output packaging.
  • COPE curriculum repository: COPE builds a hierarchical curriculum repository through structural segmentation, multi-resolution pedagogical chunking, graph-aware linking, and embedding-based indexing.It preserves pedagogical structure and supports retrieval at chapter, concept, and fine-grained knowledge resolutions.
  • Intent routing: IAC filters unrelated requests, detects underspecified topics, and seeks clarification before curriculum retrieval proceeds.The Intent, Ambiguity Detection, and Clarification agents ensure that only well-specified curriculum-related requests continue through the pipeline.
  • Evidence retrieval and validation: The retrieval workflow combines dense and lexical search, restores context, reduces overlapping evidence, and validates curriculum coverage before generation.The Retrieval Agent and Concept Detector coordinate hierarchical hybrid retrieval, CCI, CCR, and evidence validation so generation proceeds only with sufficient evidence.
  • Teacher-in-the-loop review: Failed verification does not trigger automatic regeneration or discarding; teachers decide whether to accept, edit, or discard the assessment and its report.SAVER retains both the assessment and verification report for teacher-in-the-loop review.

4 Dataset Construction

TeachMateGPT produces NCTB-SciGen8, a provenance-preserving dataset of curriculum-grounded Class 8 science assessments spanning the complete official textbook. The dataset contains 198 Bangla assessments distributed across multiple-choice and creative-question formats and all 14 chapters.

  • Construction pipeline: Every assessment that passes SAVER verification and output packaging becomes an NCTB-SciGen8 dataset instance.TeachMateGPT therefore serves as the dataset-creation pipeline for reusable curriculum-grounded assessments assembled from verified outputs.
  • Dataset scope: 198 Bangla Class 8 science assessments span all 14 chapters and 156 pages of the official NCTB Class 8 Science textbook.Each assessment preserves its full generation provenance.
  • Dataset composition: The dataset includes 143 MCQs and 55 CQs, with per-chapter distributions covering pages, subject areas, and assessment instances.Its export schema, provenance format, teacher-reviewed subset, and coverage-based adequacy argument are detailed in Appendix A.

5 Experimental Setup

The paper states that complete implementation details and evaluation settings are provided in Appendix D.

  • Complete implementation details and evaluation settings are provided in Appendix D.

6 Results Analysis

Results show that TeachMateGPT preserves curriculum structure, improves routing and retrieval safety, and generates grounded assessments with strong validation and teacher-evaluated gains. SAVER found no fabricated facts in the audited sample, while the complete pipeline outperformed Vanilla RAG on key metrics.

  • COPE and curriculum knowledge base: COPE preserves the NCTB curriculum structure across all 14 chapters, with four of six deterministic validity gates achieving a 100% pass rate.Remaining errors concerned formatting constraints rather than missing curriculum evidence.
  • Intent and clarification routing: 30% of evaluation queries are removed before retrieval, while the Bangla specificity guard resolves 81% of ambiguity cases without model intervention.Explicit teacher prompts trigger no unnecessary clarification.
  • Hybrid retrieval and coverage gating: 12.5% fail-closed rate with 0.724 coverage ratio balances safety and coverage, whereas dense-only retrieval reaches 31.3% refusal and 0.618 coverage.Gate removal reduces safety despite fewer refusals, supporting complementary dense and lexical signals for OCR-affected Bangla text.
  • Curriculum-grounded assessment generation: 85.7% of MCQs pass first-attempt validation, while CQ validation rises from 7.1% →100% after narrative adjustment.Initial CQ errors mainly reflected narrative-style mismatch rather than weak curriculum grounding, and stem length remains nearly unchanged across difficulty levels.
  • Verification and teacher review: SAVER detects no fabricated facts in 55 CQ clues and a 15-item MCQ spot check, while TeachMateGPT improves faithfulness (0.68 ↑0.96), context precision (0.54 ↑0.92), and teacher utility (2.00 ↑4.80) over Vanilla RAG.Ablations indicate that COPE chiefly affects retrieval quality and stimulus realism.

7 Conclusion

TeachMateGPT is presented as a curriculum-grounded multi-agent framework combining hierarchical retrieval, fail-closed generation, and source-attributed verification for Bangla science assessment generation. Its evaluations show substantial gains in retrieval quality, while ablations demonstrate complementary contributions from COPE and SAVER.

  • Framework contributions: TeachMateGPT contributes COPE, a hierarchical graph-aware curriculum index with ingest-time lineage and neighbor links that extends retrieval beyond static parent-child chunks.COPE is one of the framework’s four stated contributions.
  • Framework contributions: The framework uses a staged, fail-closed multi-agent pipeline that withholds generation when evidence is insufficient, avoiding fabricated assessments.This pipeline is identified as a core contribution alongside COPE and SAVER.
  • Framework contributions: TeachMateGPT includes a teacher-rated NCTB Class 8 science assessment dataset reviewed by three teachers.The conclusion reports the dataset and its teacher-reviewed subset as a contribution.
  • Evaluation results: 0.54 to 0.92 context precision and 0.58 to 0.91 context recall are achieved by TeachMateGPT across automatic evaluations.These results quantify improvements in retrieval quality.
  • Ablation results: Removing COPE reduces answer relevancy from 0.89 to 0.76 and pedagogical alignment from 4.90 to 4.05.The ablation supports COPE’s complementary role in retrieval and verification.
  • Ablation results: Removing SAVER causes faithfulness to fall from 0.96 to 0.79 and pedagogical alignment from 4.90 to 3.25.The conclusion identifies this as the largest ablation drop in faithfulness and pedagogical alignment.

Limitations

TeachMateGPT’s limitations concern its narrow NCTB Class 8 Bangla science scope, text-only handling of diagram-dependent content, and dependence on error-prone textbook transcription. Its fail-closed behavior can reduce recall, while limited evaluation scale constrains the strength and generalizability of reported findings.

  • Scope and Transferability: The framework is scoped to Bangla Class 8 science assessment generation from authorized NCTB textbooks and does not claim direct transfer to other educational settings.New grades, subjects, languages, or curricula would require index reconstruction and pipeline adaptation.
  • Indexing and Curriculum Representation: COPE’s native extraction or vision-based transcription can introduce OCR errors, incomplete page coverage, and corrupted mathematical notation.Under fail-closed retrieval, these errors cause refusal or limited evidence rather than unsupported generation, but recall loss is only partially quantified.
  • Text-Only Modality and Diagram-Dependent Items: Text-only retrieval, generation, and verification collapse or discard curriculum figures, limiting support for diagram-dependent assessment items.Examples include circuit and ray diagrams, microscopic illustrations, atomic-structure schematics, and labeled graphs.
  • Retrieval, Generation, and Verification: Fail-closed retrieval improves evidence quality but reduces recall under paraphrases, synonymy, or OCR-induced lexical mismatch.Refusal precision, refusal recall, and false-refusal rate against gold refusal decisions remain unmeasured, so reported coverage figures are descriptive rather than calibrated detection metrics.
  • Evaluation Scale: Human evaluation by three practicing teachers and small component query sets limit statistical power and inter-rater generalizability.The reported ratings and agreement are indicative rather than definitive, motivating larger teacher panels and evaluation banks.

Ethical Considerations … A.2 Dataset Schema and Teacher Review

TeachMateGPT is positioned as a teacher-supervised drafting aid with safeguards for reliability, misuse, bias, and copyright-conscious data use. NCTB-SciGen8 assessments preserve provenance and structured schema fields for curriculum evidence, generated outputs, and review annotations.

  • Ethical Considerations: Every generated item includes supporting evidence and a verification report, while teachers decide whether to accept, edit, or discard it before classroom use.The framework warns rather than silently rewriting flagged items and is not intended for fully automated deployment.
  • Ethical Considerations: The system uses only the officially authorized NCTB Class 8 science textbook for curriculum content, excluding third-party notes, commercial question banks, and unrestricted web material.The textbook is used for non-commercial research without redistributing it or full textbook pages.
  • Ethical Considerations: NCTB-SciGen8 was reviewed voluntarily by three practicing secondary-school science teachers who gave informed consent without monetary compensation.No students or other minors participated, and the evaluation collected only pedagogical quality judgments without personal, sensitive, or identifying data.
  • Ethical Considerations: Reliability safeguards combine curriculum-grounded retrieval, a fail-closed coverage gate, and post-generation scoring of faithfulness, relevance, and hallucination risk.These safeguards reduce, but do not eliminate, the risk of incorrect items misleading learners or reinforcing misconceptions.
  • Ethical Considerations: An intent-routing stage screens every input and substitutes a fixed safe response for unsafe or inappropriate requests.Examples include violence, self-harm, weapons, sexual content involving minors, harassment, and cheating assistance; teacher oversight remains necessary.
  • Ethical Considerations: The framework acknowledges model bias, Bangla fluency and terminology errors, and the lack of psychometric validation for prompt-based difficulty labels.These limitations are especially relevant in low-resource language settings and mean difficulty labels should not be treated as validated measures.
  • A.1 Export Format and Provenance: Each exported assessment records the original query, detected chapter and concept, retrieved textbook sources, generated content, and verification results.MCQs use Bangla stems with four options labeled k, x, g, and G, while CQs use an Uddipok stimulus and four cognitive-level components.
  • A.2 Dataset Schema and Teacher Review: Every assessment instance is stored in JSON with a unique question_id and fields for teacher requests, curriculum metadata, retrieved evidence, generated outputs, and evaluation annotations.The schema overview is provided in Table 8.

A.3 Adequacy as an Evaluation Dataset

NCTB-SciGen8 is presented as a curated evaluation dataset whose adequacy rests on curricular coverage and item quality rather than raw size. It supports evaluation of curriculum-grounded generation within the NCTB Class 8 science domain, while broader datasets and evaluation panels are needed for generalizable conclusions.

  • Dataset positioning: NCTB-SciGen8 is positioned as a curated evaluation dataset rather than a large item bank.Its adequacy is framed around coverage and quality, not raw dataset size.
  • Coverage and quality: 198 assessment items span all 14 chapters of the NCTB Class 8 science syllabus, providing complete coverage of this bounded domain.This coverage is distinguished from evaluating only a partial curricular sample.
  • Coverage and quality: Every item passes validation gates before release, and a subset is independently reviewed by three practicing science teachers.These procedures provide quality evidence beyond the dataset’s curricular coverage.
  • Scope and limitation: NCTB-SciGen8 is considered adequate for evaluating curriculum-grounded generation within this domain, but broader conclusions require larger datasets and evaluation panels.The stated limitation concerns generalizability beyond the bounded NCTB Class 8 science setting.

B NCTB-SciGen8 Dataset Details … D.2.1 Automatic Evaluation

TeachMateGPT is evaluated on a 198-item, 14-chapter NCTB Class 8 science dataset using automatic and human comparisons, ablations, retrieval diagnostics, and efficiency measurements. Results attribute major gains to COPE and SAVER, while routing, hybrid retrieval, and the coverage gate improve safety, coverage, or cost.

  • B NCTB-SciGen8 Dataset Details: COPE recursively indexes 14 chapters at macro, meso, and micro resolutions, supporting broad conceptual and localized fact retrieval while disabling token-aligned micro chunks by default because of OCR noise.The paper illustrates chapter-grounded creative questions and MCQs produced from retrieved curriculum evidence.
  • C.1 Curriculum Indexing (COPE): Removing COPE causes the largest answer-relevancy decline, 0.89 → 0.76, and context-precision decline, 0.92 → 0.73, alongside lower stimulus realism, 4.70 → 3.90.This supports COPE’s role in supplying well-scoped, curriculum-aligned evidence.
  • C.2 Source-Attributed Verification (SAVER): Removing SAVER causes the largest faithfulness drop, 0.96 → 0.79, and pedagogical-alignment drop, 4.90 → 3.25, while retrieval metrics remain comparatively high.SAVER functions as the final faithfulness check before teacher presentation.
  • C.3 Redundancy Reduction and Context Restoration (CCR, CCI): CCR and CCI ablations produce smaller automatic degradations, while both yield nearly identical fail-closed rates of 18.8% and coverage ratios of 0.701 and 0.705.No query in the 16-query bank triggered a missing-parent or near-duplicate case.
  • C.4 Query Routing: Removing the IAC lowers all four automatic metrics to 0.81/0.77/0.89/0.88, while also more than doubling latency, 11.3s → 26.3s, and tripling generation-reaching queries, 3/16 → 9/16.Routing filters unsafe, off-topic, and underspecified queries before costlier stages; science queries comprise 70% of the routing bank.
  • C.6 Comparison Against Prior RAG Baselines: TeachMateGPT outperforms RAPTOR, GraphRAG, CRAG, and Adaptive RAG on every automatic and human-rated criterion, while requiring zero LLM calls and zero seconds of one-time indexing.CRAG is strongest among baselines on faithfulness, 0.86, and coverage, 0.708, but still trails TeachMateGPT on every reported metric.
  • B NCTB-SciGen8 Dataset Details: The NCTB-SciGen8 corpus contains 198 evaluation items across 14 textbook chapters: 143 multiple-choice and 55 creative questions.The corpus spans 156 pages and includes 100% Bangla-dominant MCQ text and 100% board-style creative-question framing.

D.2.2 Teacher-in-the-Loop Evaluation

Three practicing secondary-school science teachers independently evaluated a configuration-blind subset of generated assessments on four pedagogical dimensions using a 5-point Likert scale. Human evaluation complements automatic metrics by assessing classroom-oriented quality, and the full system consistently outperformed its ablations.

  • Evaluation Design: Three practicing secondary-school science teachers independently assessed a shared, configuration-blind subset of generated assessments.The evaluation uses human judgment because automatic metrics alone cannot fully assess educational quality.
  • Evaluation Dimensions: Assessments were rated on Pedagogical Alignment, Stimulus Realism for creative questions, Linguistic Fluency, and Overall Utility.These four dimensions target curriculum fit, stimulus quality, language quality, and classroom usefulness.
  • Rating Scale: Ratings used a 5-point Likert scale ranging from Poor to Excellent.A score of 1 indicates major errors requiring complete revision, whereas 5 indicates a factually accurate, pedagogically sound, classroom-ready assessment.
  • Evaluation Results: Human evaluation showed that the full system consistently outperformed all ablations across the four assessment-quality criteria.This comparison is reported in the qualitative and structural analysis of TeachMateGPT.

E Detailed Analysis of Research Questions · F Detailed Pseudocode for the TEACHMATEGPT Framework · G Agent Prompt Specifications Used in TEACHMATEGPT

TeachMateGPT’s analyses show that hierarchical curriculum structure, gated routing and retrieval, validated generation, and source-attributed teacher review jointly support cautious assessment authoring. Its pseudocode and agent specifications operationalize these safeguards through staged abstention, format-specific generation, and teacher-facing outputs.

  • E Detailed Analysis of Research Questions: COPE spans all 14 chapters and four subject areas, while four of six deterministic validity gates achieve 100% pass rates.The remaining gates reach 95.8% for MCQ structural well-formedness and 94.5% for CQ Bangla-script purity.
  • E Detailed Analysis of Research Questions: Intent routing filters 30% of evaluation queries before retrieval, while the Bangla specificity guard resolves 81% of ambiguity decisions without model overrides.The filtered queries are evenly divided among greetings, harmful requests, and off-topic queries.
  • E Detailed Analysis of Research Questions: The full hybrid pipeline balances refusal and evidence with a 12.5% fail-closed rate and 0.724 coverage ratio.Dense-only retrieval raises refusal to 31.3% and lowers coverage to 0.618, whereas removing the coverage gate produces 0.649 coverage with no refusals.
  • E Detailed Analysis of Research Questions: Accepted evidence produces every requested 5 MCQs and 1 CQ, while MCQs pass strict first-attempt validation on 12 of 14 turns.This corresponds to 85.7% first-attempt reliability, with two turns recovered through relaxed processing.
  • E Detailed Analysis of Research Questions: Manual review finds zero fabricated facts across 55 CQ clues and a 15-item MCQ spot check, with all 143 MCQs and 55 CQs judged on-topic.SAVER and deterministic gates instead surface structural or notation-level issues for review.
  • F Detailed Pseudocode for the TEACHMATEGPT Framework: The pseudocode constructs COPE from pedagogical segmentation, multi-resolution chunks, and a linked graph, then routes queries, performs hybrid retrieval, and abstains when evidence is insufficient.The workflow proceeds through six stages and packages provenance, assessments, verification, caution status, and execution traces for teacher presentation.
  • G Agent Prompt Specifications Used in TEACHMATEGPT: The Intent and Ambiguity agents gate downstream processing, classifying messages and requesting clarification when queries lack sufficient specificity for reliable retrieval.The Intent Agent favors science_query under uncertainty, while the Ambiguity Agent identifies semantic, scope, referential, and under-specified exam ambiguities.
  • G Agent Prompt Specifications Used in TEACHMATEGPT: Specialist prompts enforce textbook grounding and format constraints: MCQs require four options and a valid answer index, while creative items require realistic stimuli and four cognitive-level parts.Both specialists must verify requested counts and trace factual claims to retrieved context; creative items additionally require 5–8 Bangla-script sentences.
Loading 2608.13708v1…