PaperlayerView exact revision on arXiv

Source-linked AI summary

CHSR-RRF: A curriculum-gated hybrid retrieval framework with reciprocal rank fusion and leakage-aware benchmarking for educational RAG

Terence Ateya, Zavier Ndum Ndum, Jicheng Fu, Kelly Tendongkeng

arXiv:2609.02913v1cs.IR

TL;DR

Standard educational retrieval optimizes relevance without enforcing curriculum admissibility, creating leakage from valid subject, level, or examination scope. CHSR-RRF combines curriculum gating, hybrid retrieval, and CERB evaluation; gating reduces leakage, while exact-scope recovery remains limited by corpus gaps.

  • Problem

    Standard RAG systems optimize semantic relevance without enforcing admissibility, allowing retrieved documents outside the valid curriculum scope.

  • Method

    CHSR-RRF uses structured curriculum metadata to gate candidates before retrieval, while CERB provides machine-parseable, deterministically evaluable benchmark cases.

  • Results

    4.6× leakage reduction is the largest ablation effect, while the full pipeline achieves 0.0656 exact-scope success and 58% of zero-recall failures are data gaps.

  • Takeaways & Limitations

    Curriculum-aware retrieval improves admissibility, but exact-scope recovery requires both corpus expansion and method refinement.

  • Takeaways & Limitations

    The benchmark is single-annotator, subject coverage is uneven, and transfer beyond structured secondary-exam corpora remains untested.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) is increasingly used in educational question answering, but standard retrievers optimize topical relevance without enforcing curriculum validity. In school settings, a passage can be relevant yet inappropriate if it comes from the wrong subject, level, or examination context; we call this failure mode curriculum leakage. We present CHSR-RRF, a curriculum-gated hybrid retrieval framework that applies metadata constraints before retrieval, then combines sparse and dense search with reciprocal rank fusion and deterministic reranking. We also introduce CERB, a 126-case benchmark for curriculum-constrained retrieval with hierarchy-aware relevance labels and explicit leakage annotations. On a 61-case pilot, pre-retrieval gating reduces leakage by 4.6x ($p<0.001$) while preserving ranked recall, whereas applying the same constraints after retrieval collapses recall and exact-scope success to zero ($p=0.039$). A full-benchmark lower-bound analysis further shows that many remaining failures arise from corpus and metadata gaps rather than retrieval design alone. These results show that retrieval in structured educational domains should be treated as constrained selection, with validity enforced when the candidate pool is formed rather than after ranking.

1. Introduction

Educational RAG requires curriculum-valid evidence, not merely topical relevance, because out-of-scope material can confuse learners. CHSR-RRF reframes retrieval as constrained selection and evaluates pre-retrieval gating against post-hoc filtering using CERB.

  • Problem: Topical relevance alone can fail in education because evidence from the wrong subject, level, examination paper, or session may be inappropriate.The paper calls this failure mode curriculum leakage.
  • Problem: Curriculum admissibility should constrain the candidate pool before ranking rather than filter an unconstrained retrieval output afterward.This reframing treats retrieval as constrained selection subject to curriculum-validity constraints.
  • Contributions: CERB is a 126-case benchmark designed to measure curriculum-constrained retrieval with structured, machine-parseable records and explicit admissibility evaluation.The benchmark supports hierarchy-aware relevance and leakage analysis.
  • Contributions: CHSR-RRF instantiates curriculum-constrained retrieval by enforcing admissibility before ranking and combining retrieval components within the gated candidate pool.The contribution is the constrained framing and enforcement point, not any individual retrieval component.
  • Contributions: Pre-retrieval gating reduces leakage by 4.6× while preserving recall, whereas post-retrieval filtering collapses recall and exact-scope success to zero.The pilot comparison reports p < 0.001 for the leakage reduction and p = 0.039 for the recall contrast.

2. Related Work and Background

Prior retrieval and RAG research improves relevance, recall, reasoning, or critique within unconstrained candidate pools. This paper adds curriculum admissibility as an explicit retrieval-time constraint and introduces CERB to measure scope fidelity.

  • Retrieval Foundations: BM25 remains a strong lexical baseline, while dense, late-interaction, and learned sparse retrievers replace or augment term-overlap ranking.The related work positions BM25, ColBERT, Contriever, SPLADE, and ANCE as major retrieval approaches.
  • Hybrid Retrieval: Hybrid retrieval combines sparse and dense signals, reciprocal rank fusion combines ranked lists, and cross-encoders rerank retrieved candidates.These components are established techniques rather than CHSR-RRF’s sole contribution.
  • RAG Architectures: Advanced RAG systems add joint training, iterative reasoning, self-correction, hierarchical summaries, or corrective retrieval, but generally retain unconstrained candidate selection.CHSR-RRF places a deterministic curriculum gate before these retrieval and reasoning operations.
  • Benchmarking Gaps: Standard retrieval and RAG benchmarks measure topical relevance, answer quality, faithfulness, or context precision without hierarchical curriculum admissibility labels.The paper identifies this as a measurement gap for educational retrieval.
  • Benchmarking Gaps: CERB measures retrieval-time curriculum fidelity through curriculum tuples, structurally admissible evidence labels, leakage annotations, slice decomposition, and failure taxonomy.Its protocol is designed to transfer to national examination systems with structured metadata.

3. Materials and Methods

The study uses a frozen, quality-audited corpus of 74,018 evidence chunks from 11 GCE subjects, with corpus composition and data-quality conditions shaping evaluation coverage. The methodology documents both the source collection and limitations affecting dense retrieval and curriculum gating.

  • Corpus and source documents: The evaluation corpus contains 74,018 evidence chunks indexed from over 2,000 source documents across 11 GCE subjects at Ordinary and Advanced Levels.Sources include examination papers, official syllabi, solutions, mock examinations, textbooks, workbooks, and teaching materials.
  • Snapshot provenance: The evaluation uses a canonical frozen snapshot because a later truncation-and-reingest cycle reduced the live database to a narrower KB v2 slice.The frozen snapshot is the reproducible corpus for the paper’s figures, tables, and 61-case pilot ablation.
  • Data-quality audit: The methodology treats embedding coverage and metadata completeness as distinct data-quality conditions that affect dense retrieval and curriculum gating.These facets are used to interpret the ablation results.
  • Corpus composition: Physics and Chemistry dominate the indexed corpus, contributing 35.9% and 26.8% of chunks, respectively.Their prominence reflects the focus of CERB cases on these subjects.
  • Data-quality audit: Dense retrieval has 89.48% embedding coverage, with residual NULL embeddings concentrated in Physics and Chemistry at 83.22% and 83.23% coverage.The other nine subjects have 100% embedding coverage.

3.2. The CERB Benchmark

CERB is a machine-readable, curriculum-constrained benchmark designed to separate topical relevance, scope correctness, and controlled widening across diagnostic exam-retrieval cases. Its 126 queries span seven evidence and query-type slices and support deterministic, auditable evaluation.

  • Annotation schema: CERB labels relevance, curriculum leakage, ambiguity, and task type so semantic usefulness can be evaluated separately from scope correctness.Task types include exact retrieval, scope relaxation, synthesis, and adversarial evaluation.
  • Case structure: Each CERB case records its query, curriculum scope, slice, expected evidence types, exact metadata fields, optional question number, and annotation notes.The benchmark is distributed as a machine-readable specification that can be validated and re-executed deterministically.
  • Evaluation design: The evaluation pipeline compares five retrieval variants using nDCG@K, Recall@K, leakage rate, and exact-scope success, with results decomposed by slice.Paired bootstrap significance tests are applied to consecutive variant transitions.
  • Labeling policy: Exact-scope success requires at least one returned document to match the expected metadata fields and, when specified, the expected question number.Relevant documents address the query topic, while scope-correct documents match all non-null expected fields.
  • Benchmark composition: CERB contains 126 distinct student queries partitioned into seven nonoverlapping slices: MCQ, structured, formula-heavy, cross-subject, bilingual, mark-scheme, and table-heavy.The distribution is intentionally imbalanced, with MCQ and structured cases predominating and the remaining slices probing edge conditions.
  • Auditable widening: CERB makes scope widening explicit, ordered, and auditable rather than hiding it inside retrieval heuristics.Each case record exposes whether retrieval ended in an exact hit, a leakage-flagged recovery, or a contaminated or empty failure.

3.3. Problem Formulation and Metadata Hierarchy

The paper formulates educational retrieval as relevance optimization over documents that satisfy a query-induced curriculum scope. This reframing distinguishes semantic relevance from admissibility and places curriculum validity in candidate-pool construction.

  • Metadata hierarchy: Each evidence document carries a curriculum tuple containing subject, level, year, session, paper, and content type.The query induces a fully or partially specified constraint tuple for matching.
  • Constrained formulation: The retrieval objective maximizes relevance over the admissible subset of corpus documents satisfying the query’s curriculum constraints.The admissible set is defined by the document and query constraint tuples.
  • Evaluation dimensions: Relevance is measured with nDCG@K and Recall@K, whereas admissibility is measured with leakage rate and exact-scope success.Standard RAG optimizes relevance without enforcing admissibility.
  • Design contribution: CHSR-RRF restricts the candidate pool before ranking and explicitly models, orders, and audits scope relaxation.Its contribution is the enforcement point and relaxation policy rather than any individual retrieval component.

3.4. The CHSR-RRF Retrieval Pipeline

CHSR-RRF first extracts curriculum scope and gates the candidate pool, then performs hybrid retrieval, reciprocal rank fusion, deterministic reranking, and controlled relaxation when necessary. Relaxation follows explicit decision rules and preserves leakage visibility for auditing.

  • Stage 1: Hard Curriculum Gate: The hard curriculum gate excludes documents that fail exact subject and level matching, plus exact paper matching when the query specifies a paper.The gate is implemented as database-level SQL filtering before embedding computation.
  • Stage 2: Dual-Path Retrieval: Within the gated pool, BM25 and dense cosine-similarity retrieval run in parallel, with deterministic curriculum-aware query expansion adding further candidate lists.BM25 uses PostgreSQL full-text search, while dense retrieval uses pgvector HNSW.
  • Stage 3: Reciprocal Rank Fusion: Reciprocal rank fusion merges all non-empty sparse, dense, and expanded candidate lists using κ = 60.The paper uses the standard default constant and describes RRF as robust to its choice.
  • Stage 4: Deterministic Reranker: The deterministic reranker combines the fused score with curriculum alignment and source reliability while penalizing redundant results.The production implementation realizes the same policy through multiplicative boost and penalty factors without LLM inference.
  • Stage 5: Scope Relaxation: Scope relaxation is triggered by insufficient or mismatched results and follows a fixed sequence that distinguishes corpus sparsity from a genuine topic miss.When on-topic results exist but the pool is below kmin, widening is skipped to avoid replacing relevant results with off-topic material.
  • Stage 5: Scope Relaxation: W1 drops topic constraints, W2 can drop level and cycle constraints unless strict-scope keys are present, and W3 permits an explicit unfiltered lexical fallback.Each widening step reruns the full retrieval pipeline and labels relaxed results for joint quality and leakage analysis.

3.5. Ablation Variants

The ablation isolates the marginal contributions of six retrieval variants, progressing from scoped dense retrieval to the full CHSR-RRF pipeline. All variants retrieve eight documents from the same indexed educational corpus, while adding gating, fusion, reranking, expansion, and relaxation capabilities.

  • Variant progression: Six variants isolate the marginal contributions of curriculum gating, hybrid retrieval, post-filtering, reranking, query expansion, and scope relaxation.The sequence runs from A, dense-only scoped retrieval, through B/B+, C, D, and E, the full CHSR-RRF configuration.
  • Variant progression: Variant A uses cosine similarity over 1536-dimensional embeddings with a hard curriculum metadata gate.It excludes lexical retrieval and rank fusion.
  • Variant progression: Variant B combines BM25 and dense retrieval with reciprocal rank fusion but no metadata gate, serving as the unconstrained baseline.Variant B+ applies the same curriculum metadata filters only after retrieval.
  • Variant progression: Variant C applies the hard curriculum gate before retrieval, whereas D adds deterministic metadata-aware reranking and content-based deduplication.Deduplication uses a threshold of 0.75.
  • Variant progression: Variant E adds curriculum-aware query expansion and progressive scope relaxation with leakage tracking to the gated, reranked pipeline.This configuration is the full CHSR-RRF pipeline.
  • Shared setup: All variants retrieve K = 8 documents from a corpus of 74,018 evidence chunks across 11 subjects, using PostgreSQL pgvector HNSW and BM25 search.All runs completed with zero errors.

3.6. Metrics

The evaluation separates retrieval quality, curriculum fidelity, and efficiency. This metric design distinguishes whether retrieved documents are relevant, curriculum-admissible, and operationally efficient.

  • Metric families: Retrieval quality is measured with Precision@K, Recall@K, and normalized discounted cumulative gain (nDCG@K).
  • Metric families: Curriculum fidelity is measured with leakage rate, contamination rate, and exact-scope success (ESS).ESS requires at least one result matching all expected metadata fields and the question number.
  • Metric families: Efficiency is measured as end-to-end latency per query, including network round-trips.

3.7. Experimental Setup and Statistical Method

The study evaluates consecutive ablation pairs on a frozen 61-case CERB pilot and supplements the analysis with an LLM-judge consistency check. Statistical reporting uses paired bootstrap intervals and cautious p-values for sparse binary outcomes.

  • Evaluation design: All ablation results come from one frozen evaluation run on the 61-case CERB pilot subset at K = 8 against the canonical corpus snapshot.The CERB specification contains 126 cases.
  • Statistical method: Consecutive ablation pairs are evaluated with paired bootstrap confidence intervals using N = 1000 resamples and seed = 42.
  • Statistical method: Exact-scope success is binary and sparse, so the study reports p-values while cautioning that the pilot has limited power for small intermediate-variant differences.
  • Annotation consistency: CERB cases were authored by one annotator, so true inter-annotator agreement is deferred; a deterministic LLM-judge consistency check is used as a conservative substitute.The judge rederives subject, level, and paper number without seeing annotator-assigned metadata.

4. Results and Discussion

Pre-retrieval curriculum gating is the central determinant of leakage and recall in the pilot, while later components provide narrower gains and trade-offs. Remaining failures largely reflect corpus, preprocessing, and metadata limitations, with one deliberate cross-subject gating boundary.

  • 4.1. Main Ablation Results: 4.6×: Variant C cuts leakage from 0.7596 to 0.1653 relative to ungated hybrid B while recovering nDCG@8 = 0.5529; post-filtered B+ scores zero on R@8, nDCG@8, and ESS.The B+ → C recall comparison is significant at p = 0.039, and leakage at p < 0.001.
  • 4.1. Main Ablation Results: Removing the curriculum gate increases leakage from 0.0969 to 0.7596 and lowers nDCG@8 from 0.4874 to 0.2577, the largest ablation transition.The bootstrap test reports p < 0.001.
  • 4.1. Main Ablation Results: Adding FTS and RRF raises nDCG@8 from 0.4874 to 0.5529 while preserving ESS at 0.0492, but leakage rises from 0.0969 to 0.1653.The deterministic reranker adds no retrieval-quality improvement at this sample size (p = 1.000).
  • 4.1. Main Ablation Results: Variant E uniquely recovers cross-subject Recall@8 of 0.083 and raises ESS from 0.0492 to 0.0656, while leakage rises from 0.1646 to 0.2593 and latency from 1754 ms to 2645 ms.Three slices remain at zero recall across all five variants, indicating unresolved data and metadata gaps.
  • 4.2. Uncertainty and Slice-Level Analysis: The pilot’s zero-recall slices expose structured failures: bilingual, mark-scheme, and table-heavy cases remain at zero across variants.The analysis attributes the primary bottleneck to corpus coverage and metadata completeness rather than retrieval algorithm design.
  • 4.3. Annotator-Consistency Findings: Exam-level labels are rarely explicit in queries, with 35% coverage, while subject identification reaches κ = 0.59 and 72% raw agreement over 122 covered cases.The protocol therefore under-specifies level context when the retriever must match it.
  • 4.4. Root-Cause Analysis of Zero-Recall Slices: The best variant achieves exact-scope success of 0.0656, and 21 of 36 low-performing-slice failures arise from near-zero target-subject content.Ten additional failures reflect preprocessing gaps, while five expose a genuine CHSR-RRF limitation.
  • 4.4. Root-Cause Analysis of Zero-Recall Slices: The hard gate blocks legitimate multi-subject queries because query understanding extracts a single subject, creating a deliberate leakage-versus-exploration trade-off.This affects about 5 of 11 cross-subject cases.

5. Conclusion

The study shows that curriculum-gating enforcement point materially affects retrieval reliability, while scope widening improves exact-scope recovery at measurable costs. Remaining failures reflect both method limitations and corpus or metadata gaps.

  • Removing the curriculum gate increases leakage from 0.0969 to 0.7596, while restoring it reduces leakage to 0.1653.The 4.6× reduction is the largest and most statistically significant ablation effect.
  • Scope relaxation raises exact-scope success from 0.0492 to 0.0656, but also increases leakage from 0.1646 to 0.2593 and latency from 1,754 ms to 2,645 ms.The reported improvement is 33%, and uniquely recovers recall in the cross-subject slice.
  • The full CHSR-RRF pipeline achieves the highest exact-scope success at 0.0656, but absolute success remains low.Per-slice analysis attributes 58% of zero-recall failures to data gaps rather than method limitations.
  • The benchmark is single-annotator with uneven subject coverage, and the headline ablation remains based on a 61-case pilot pending a full rerun.The study identifies corpus expansion and further validation as necessary boundaries on interpretation.
  • Future work targets broader CERB coverage, lower full-pipeline latency, and transfer to other structured examination systems.The proposed latency target is below 1.5 s, compared with the stated current latency of 2.6 s.

Appendix A. Illustrative Failure Cases

The appendix presents representative retrieval failure cases to show how aggregate ablation statistics appear in individual retrieval traces.

  • Representative failure cases illustrate how aggregate leakage and zero-recall statistics manifest in individual retrieval traces.

Highlights

The paper highlights curriculum leakage as a measurable educational-RAG failure mode and presents CHSR-RRF and CERB as tools for constrained retrieval analysis.

  • Curriculum leakage is introduced as a measurable failure mode in educational RAG retrieval.
  • Curriculum gating cuts cross-scope leakage 4.6× with no LLM cost.The reported significance is p < 0.001.
  • CHSR-RRF fuses sparse, dense, and rerank signals under curriculum constraints.
  • CERB is a 126-case sliced benchmark that diagnoses retrieval failures by failure type.
  • 86% of zero-recall cases trace to data or preprocessing gaps rather than method design.
Loading 2609.02913v1…