Source-linked AI summary
GRASP: Graph-Retrieval Automated Scoring Pipeline for Label-Free Multi-Topic Essay Grading
Aafreen Husain, Samar Shailendra, Saad Sajid Hashmi
TL;DR
Label-free multi-topic grading requires scoring several related answers merged into one unmarked response, a setting less explored than single-topic grading. GRASP combines topic detection, graph-augmented reference retrieval, one-to-one assignment, and segment grading; across one to four topics, it retrieves complete gold reference sets more often than flat RAG and improves grading agreement at low and moderate topic counts.
Problem
Multi-topic answers are merged without labels or boundaries, and systems must infer topic count, segment by meaning, and match answers to references.
Source: §1 · p. 2Method
GRASP detects topic count, segments the essay, retrieves reference nodes with graph-augmented expansion, assigns distinct references, and grades each segment.
Source: §1 · p. 2Results
Across one to four topics, GRAG retrieved the full gold reference set more often than RAG and yielded higher grading agreement at low and moderate topic counts.
Source: §6 · p. 13Takeaways & Limitations
The retrieval mechanism can use examiner-chosen connections, including co-occurrence links, rather than only similarity-based graph edges.
Source: §6 · p. 13Because essays concatenate single-topic answers in fixed source order and questions derive from the same references, reported detection, segmentation, and retrieval results are upper bounds for real responses.
Source: §5.5 · pp. 12–13
Abstract
from arXiv · showhide
Automated short-answer grading research has historically focused on exams consisting solely of questions pertaining to a single topic. Automatic grading of exams containing questions about more than one topic remains less explored. In this work, a Graph-Retrieval Automated Scoring Pipeline (GRASP) is introduced for grading label-free multi-topic science exams. Label-free exams are short-answer exams in which a student's responses to several distinct topics are merged into a single paragraph, with no markup labels or segmentation indicating which span answers which question. Reference answers for each question are encoded into a FAISS vector index via Sentence-BERT, and a semantic similarity graph is constructed over this set of reference answers. At grading time, sentence count heuristics, with a large language model used to resolve ambiguous cases, are first applied to predict how many distinct topics were answered in the student essay. This process is performed without training data or domain-specific example essays. Candidate reference nodes, each storing one (question, reference answer, concatenation of both) from the reference index, are then retrieved through cosine similarity based Retrieval-Augmented Generation (RAG) and Graph Retrieval-Augmented Generation (GRAG). GRAG operates by taking the top cosine matches as seed nodes and then performing a graph traversal over strong edges to find additional reference nodes that may have been missed by RAG. The Hungarian algorithm is then used to optimally assign one reference node per question segment such that no reference is duplicated. Each segment is then graded against its assigned reference independently using GPT-4.1-mini. This experiment is performed to show the effect of retrieval quality on grading accuracy and the benefit of graph-augmented retrieval versus strict cosine similarity methods at various levels of essay complexity.
1 Introduction
Label-free multi-topic grading asks a system to score several related answers merged into one unmarked paragraph, without explicit topic counts or sentence-to-topic labels. GRASP combines topic detection, graph-based retrieval, and one-to-one matching to address this setting.
1 Introduction
- Students answer several related topics in one continuous paragraph, with no labels or boundaries showing which span answers each question.The topics may share concepts, wording, and logic, making their reference answers close in vector space.Source: §1 · p. 2
- The system must infer the number of topics, segment the response by meaning, and match each segment to an unannotated reference node.The number of topics is not stated, and responses may contain multiple sentences without sentence-to-topic annotations.Source: §1 · p. 2
- Existing AES systems assume one topic per response, while flat-index RAG systems lack awareness of relationships among references.The paper states that flat retrieval accuracy collapses when several related reference nodes must be retrieved simultaneously.Source: §1 · p. 2
- GRASP detects topic count, segments the response, retrieves related reference nodes through a graph, and globally assigns distinct references to segments.Its main contribution is an end-to-end formulation and evaluation methodology separating retrieval quality from grading quality; graph expansion adapts existing GraphRAG-style methods.Source: §1 · p. 2
2 Literature Survey
The survey spans automated essay scoring, semantic similarity, retrieval-augmented grading, graph-enhanced retrieval, and LLM-based feedback. It reviews how these strands inform automated grading and the GRASP setting.
2.1 Automated Essay Scoring (AES) Models
- Transformer-based transfer learning dominates recent holistic and multidimensional AES, with reported gains and strong benchmark performance across several approaches.Examples include hierarchical BERT scoring improvements on ASAP and CELA, few-shot QWK results on ASAP, and T5-based multi-trait scoring.Source: §2.1 · p. 3
2.2 Semantic Similarity Approaches
- SBERT produces fixed-length sentence embeddings for cosine search, while scalar similarity alone does not explain how passages are related.The survey also covers siamese Bi-LSTM grading, SimCSE, and ColBERT.Source: §2.2 · pp. 3–4
2.3 RAG-Based Grading Systems
- RAG-based grading grounds LLM scores in retrieved evidence, but flat retrieval can return duplicate passages and does not model reference relationships.SteLLA applies reference-grounded RAG to short-answer grading; a review reports improved factual grounding and less hallucination from retrieving evidence first.Source: §2.3 · p. 4
2.4 Graph-Enhanced Retrieval (GRAG)
- Graph-enhanced retrieval structures evidence into relations for multi-hop reasoning, and GRASP expands from FAISS seeds along strong similarity edges.The survey describes GraphRAG and GRAG as graph-based approaches to organizing related evidence.Source: §2.4 · pp. 4–5
2.5 LLM-Based Feedback Generation
- LLM-based feedback research explores instruction tuning, prompting, adaptive feedback, engineered features, and human-AI scoring, while fairness and adaptation remain open issues.The survey notes that GRASP is designed to be training-data free.Source: §2.5 · p. 5
3 Problem Definition
The paper defines multi-topic grading as scoring an unlabeled continuous response to related topics and specifies the detection, segmentation, retrieval, assignment, and grading tasks. The setup constrains topics to related reference nodes and uses an oracle to distinguish retrieval from grading performance.
3 Problem Definition
- Essays contain one to four related topics without structural markers, and answers may appear in an order different from the exam question.For essays with multiple topics, the corresponding reference nodes must form a connected similarity-graph subgraph, with pairwise cosine similarity of at least 0.42.Source: §3 · p. 5
- The reference index contains 79 nodes, each storing an exam question, its correct answer, and their concatenation.The nodes come from SciEntsBank and cover physics, chemistry, biology, and earth science.Source: §3 · p. 6
- The system must infer topic count from the essay and paraphrased question, then partition the essay into n contiguous, non-overlapping answer segments.Topic count must be inferred from linguistic content without structural labels.Source: §3 · p. 6
- Retrieval queries use paraphrased questions against an SBERT index of reference answers, returning k = n + 2 candidates for both configurations.The equal candidate budget is intended to make downstream grading differences reflect retrieval quality rather than candidate-set size.Source: §3 · p. 6
- The Hungarian algorithm assigns each question segment to a distinct candidate, and GPT grading gives each segment a score of 1 or 2.A TRUE Oracle bypasses detection, segmentation, and retrieval, providing perfect retrieval by construction to isolate the grader's upper bound.Source: §3 · p. 6
4 Methodology
GRASP separates reusable offline resource construction from per-essay grading. Its workflow prepares reference nodes and a similarity graph, then detects and segments topics, retrieves and assigns references, and grades the matched segments.
4 Methodology
- The methodology has an offline phase that builds the dataset, FAISS index, and similarity graph, followed by per-essay detection, segmentation, retrieval, assignment, and grading.The offline resources are reused for student essays.Source: §4 · p. 7
4.1 Phase A: Offline steps for building the dataset
- Phase A creates one reference node per unique question-answer pair and indexes answer embeddings with SBERT for cosine-based retrieval.The reference-answer index uses frozen SBERT embeddings, while graph edges connect answer embeddings meeting the similarity threshold and stronger edges support later expansion.Source: §4.1 · pp. 7–8
- Topic groups are selected as similarity-constrained cliques, favoring groups with higher minimum edge weights, and synthetic essays concatenate sampled answers without structural separators.The experiment constructs 160 essays for each topic count n ∈ {1, 2, 3, 4}.Source: §4.1 · p. 8
4.2 Phase B: Per-Essay Processing Steps
- Phase B uses agreeing essay and question sentence counts to infer n, calls an LLM when they disagree, and splits essay sentences into topic segments when the segment count matches n.The topic-count prompt is designed to guard against over-detection when a student elaborates on one topic across sentences.Source: §4.2 · p. 9
- RAG retrieves cosine matches, while GRAG expands from its top three seeds along one-hop strong graph edges and combines query similarity with edge weight.Both methods use the same candidate budget, k = n_detected + 2.Source: §4.2 · p. 9
- The Hungarian algorithm assigns distinct references, after which GPT-4.1-mini grades each segment; an Oracle path and repeated grading support comparison and uncertainty estimates.Grading is repeated 11 times at temperature 0.7 while deterministic retrieval and assignment remain fixed, and the Oracle bypasses detection, segmentation, retrieval, and assignment.Source: §4.2 · pp. 9–10
5 Results
GRAG retrieved complete gold reference sets more frequently than flat RAG, with the largest gap at three topics. Grading agreement was stronger for GRAG at lower topic counts, but four-topic grading remained difficult, and the synthetic evaluation limits how directly these results transfer to genuine student essays.
5 Results
- Essays were evaluated at n ∈ {1, 2, 3, 4}; grading was repeated 11 times at temperature 0.7, and summed scores ranged from n to 2n.The evaluation reports N-Hit for retrieval and QWK for grading agreement; Oracle receives gold reference nodes directly, making its N-Hit 1 by construction.Source: §5 · p. 10
5.1 Evaluation Metrics
- N-Hit is the fraction of essays whose assigned references include all gold nodes, while QWK measures ordinal-rater agreement and penalizes larger disagreements more heavily.QWK ranges from −1 to 1, with 1 indicating perfect agreement and 0 indicating no agreement better than chance.Source: §5.1 · p. 10
5.2 Retrieval.
- At n = 3, GRAG recovered the complete gold set over twice as often as RAG (62.5% vs 25.6%); N-Hit was higher at n = 1 (88.8% vs 80.0%), n = 2 (71.2% vs 55.0%), and n = 4 (59.4% vs 38.1%).Source: §5.2 · p. 10
5.3 Grading and false positives.
- At n = 4, QWK was 0.02 for GRAG and 0.05 for RAG, versus Oracle 0.50; GRAG’s false-positive rate was 10.5–15.8% versus RAG’s 17.7–30.1%.GRAG’s QWK exceeded RAG’s at n = 1 (0.52 vs 0.46), n = 2 (0.29 vs 0.26), and n = 3 (0.23 vs 0.15); the authors characterize n = 4 as an open problem.Source: §5.3 · pp. 11–12, Figure 3 · p. 12
5.4 Why we report per-n.
- Pooled QWK was 0.978 for Oracle, 0.892 for RAG, and 0.901 for GRAG at α = 0.8, β = 0.2; pooled scores reflect end-to-end grading, while per-n scores isolate retrieval.Topic count was recovered correctly for 90.8% of essays, and the authors report pooled and per-n results because they answer different questions.Source: §5.4 · p. 12
5.5 Limitations
- Because essays concatenate single-topic reference answers in source order and questions reuse source questions, detection, segmentation, and retrieval results are upper bounds for genuine responses.The construction does not capture students blending topics within a sentence, using cross-referential language, or answering out of order.Source: §5.5 · pp. 12–13
6 Conclusion and Future Work
GRAG retrieves clustered gold references more reliably than flat RAG, with its largest gains when references are close in embedding space. The conclusion also identifies grading limits and alternative graph constructions for future study.
6 Conclusion and Future Work
- At n = 3, GRAG retrieved the full gold reference set more than twice as often as RAG, and beat RAG at every n from one to four.The margins widened as the gold set became more tightly clustered.Source: §6 · p. 13
- QWK fell to 0.50 at n = 4 even with TRUE Oracle retrieval, indicating that grading and retrieval errors are at least partly separable.Grading agreement was higher with GRAG at low and moderate topic counts.Source: §6 · p. 13
- The grading evaluation used only GPT-4.1-mini, leaving open whether a stronger or differently prompted grader would narrow the gap.The authors leave comparison across multiple graders to future work.Source: §6 · p. 13
- GRAG’s largest gains occurred when relevant references were close in embedding space and flat similarity retrieval struggled to distinguish them.The authors suggest exploring graphs based on examiner-chosen connections, including reference co-occurrence on exams or in curricula.Source: §6 · p. 13