Source-linked AI summary
Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
Aditya Pathak, Rachit Gandhi, Vaibhav Uttam, Arnav Ramamoorthy, Pratyush Ghosh, Aaryan Raj Jindal, Shreyash Verma, Aditya Mittal, Aashna Ased, Chirag Khatri, Yashwanth Nakka, Devansh, Jagat Sesh Challa, Dhruv Kumar
TL;DR
LLM-based code evaluation and grading remain under-explored, particularly for assessing student code beyond generic correctness and syntax. The paper introduces question-specific rubric methods, supporting datasets, and the Leniency metric, and finds that these rubrics improve logical assessment, feedback relevance, and educational alignment over question-agnostic approaches. The study is bounded by its focus on GPT-4o, Java, two intermediary courses, and single-file assignments.
Problem
LLM-based code grading remains under-explored, while existing programming-evaluation work commonly relies on question-agnostic criteria despite instructors using question-specific rubrics.
Method
The paper introduces CRE, PRE, and EME for question-specific rubric evaluation, two OOP and DSA datasets, and the Leniency metric alongside existing evaluation measures.
Results
Question-specific rubrics substantially outperform question-agnostic rubrics, improving accuracy, feedback relevance, educational alignment, and logical code assessment.
Takeaways & Limitations
Question-specific rubrics provide a basis for more accurate and contextually relevant automated code evaluation in the studied educational settings.
Takeaways & Limitations
The study mainly uses GPT-4o and Java assignments from two intermediary courses with single-file code, without comparing other models, languages, or multi-file advanced-course tasks.
Abstract
from arXiv · showhide
Since the emergence of Large Language Models (LLMs) popularized by the release of GPT-3 and ChatGPT, LLMs have shown remarkable promise in programming-related tasks. While code generation using LLMs has become a popular field of research, code evaluation using LLMs remains under-explored. In this paper, we focus on LLM-based code evaluation and attempt to fill in the existing gaps. We propose multi-agentic novel approaches using \emph{question-specific rubrics} tailored to the problem statement, arguing that these perform better for logical assessment than the existing approaches that use \emph{question-agnostic rubrics}. To address the lack of suitable evaluation datasets, we introduce two datasets: a Data Structures and Algorithms dataset containing 150 student submissions from a popular Data Structures and Algorithms practice website, and an Object Oriented Programming dataset comprising 80 student submissions from undergraduate computer science courses. In addition to using standard metrics (Spearman Correlation, Cohen's Kappa), we additionally propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment. Our comprehensive analysis demonstrates that \emph{question-specific rubrics} significantly enhance logical assessment of code in educational settings, providing better feedback aligned with instructional goals beyond mere syntactic correctness.
1 Introduction
LLM-based code grading remains under-explored despite the importance and labor intensity of evaluating students’ programming logic. This paper studies question-specific rubrics, proposes three evaluation techniques and a leniency metric, and reports improved accuracy, feedback relevance, and educational alignment over question-agnostic rubrics.
- Motivation: LLM-based grading has received less attention than feedback generation, although code evaluation helps assess programming understanding and consumes substantial instructor time.Existing studies examine code quality, syntax, logic, and feedback more often than grading itself.
- Motivation: Existing programming-grading approaches often use rigid tests, simplistic criteria, or question-agnostic rubrics emphasizing generic correctness and syntax.The paper contrasts these approaches with instructors’ use of rubrics tailored to individual questions.
- Research question: The paper asks whether question-specific rubrics improve LLM-based grading quality and feedback specificity compared with question-agnostic rubrics in advanced computing courses.The study focuses on Object-Oriented Programming and Data Structures and Algorithms.
- Contributions: The proposed methods are Complete Rubric Evaluation, Pointwise Rubric Evaluation, and Ensembling Method Evaluation, complemented by the Leniency metric.CRE emphasizes logical correctness, PRE evaluates rubric criteria individually, EME aggregates evaluations, and Leniency compares automated strictness with expert assessment.
- Findings: Question-specific rubrics substantially outperform question-agnostic rubrics in accuracy, feedback relevance, and alignment with educational objectives.The paper presents these findings as evidence that rubric tailoring improves LLM-based code evaluation.
- Contributions: The authors introduce a dataset spanning OOP and DSA submissions with problem descriptions, student code, model solutions, rubrics, and qualitative feedback.The dataset was created because no suitable public benchmark containing these components was available.
- Implications: The resulting automated tools could reduce instructors’ grading effort and provide students with more detailed, contextually relevant feedback.The paper links these potential benefits to identifying improvement areas and deepening programming-concept understanding.
2 Related Work
Prior programming assessment systems often rely on rigid test cases or generic rubrics, limiting semantic understanding, feedback quality, and question-specific grading. Related work motivates rubric-based LLM evaluation while identifying reliability concerns and a gap in question-specific code assessment.
- Limitations of Traditional Auto-grading Approaches: Traditional autograders compare outputs against predefined test cases, which limits semantic diagnosis and nuanced explanations.They also require instructors to create comprehensive test suites and may overlook style, efficiency, and best practices.
- Rubric-based Assessment: Rubrics provide structured criteria and feedback beyond binary judgments, although creating them remains manually intensive for instructors.The paper describes rubrics as supporting consistency, objectivity, clear expectations, and constructive feedback.
- LLM-based Techniques for Grading and Feedback: Existing LLM-based assessment methods include semantic code similarity, direct evaluation, and step-by-step semantic analysis, but each has documented limitations.CodeBERTScore can confuse contextual similarity with semantic similarity, while ICE-Score has limited human correlation and remains vulnerable to hallucinations.
- LLM-based Techniques for Grading and Feedback: Question-agnostic rubrics can miss problem-specific nuances and produce misaligned scores or feedback compared with human tutors.Prior findings report incorrect identification of code issues and point assignment for general correctness and edge cases.
- LLM-based Techniques for Grading and Feedback: The paper positions question-specific rubric grading of student code as an underexplored direction and separates logical reasoning assessment from syntactic correctness.This approach is presented as distinct from short-answer rubric systems and as mirroring the preference for conceptual understanding over syntax.
3 Dataset
The paper constructs OOP and DSA datasets containing student submissions, problem materials, rubrics, model solutions, grades, and feedback. The datasets are designed to represent varied undergraduate programming assessment settings and solution outcomes.
- OOP Dataset Construction: The OOP dataset uses one Java exam question with seven methods covering file I/O, data filtering, and data updates.Students received scaffold code with class declarations, helper functions, and predefined method scopes.
- OOP Dataset Construction: OOP submissions were sampled as 20 solutions from each of four score ranges, producing 80 undergraduate submissions from 350 total.The score ranges were 0–10, 10–20, 20–30, and 30–35 marks.
- OOP Dataset Construction: The OOP sample illustrates rubric-level grading by assigning 8 out of 9 points despite one incorrect Java interface instantiation.The example records zero for the first step and one for each subsequent step.
- DSA Dataset Construction: The DSA dataset spans nine topics and three difficulty levels, with six submissions per problem representing correct, wrong, TLE, and compilation-error outcomes.Rubrics were based on multiple solution approaches, and two graders collaboratively assigned marks and implementation-step feedback.
4 Proposed Techniques
The paper proposes rubric-based techniques that evaluate student code against problem-specific steps, prioritizing logical understanding while separating syntax checking. CRE evaluates the full rubric, PRE evaluates one criterion at a time, and EME aggregates multiple LLM evaluations.
- Motivation: Question-specific rubrics break problem statements into implementation steps and award binary marks for each step.This design aims to emulate university-level grading with grading specificity and controlled leniency.
- Complete Rubric Evaluation (CRE): CRE evaluates the complete rubric against the entire student code file and returns nested JSON marks organized by methods and finer rubric points.A deterministic compiler-equipped agent separately checks syntax, allowing CRE to focus on logical evaluation.
- Pointwise Rubric Evaluation (PRE): PRE evaluates each rubric point separately using the problem statement, student code, and one criterion, producing granular JSON marks.Its point-by-point design requires multiple API calls per submission, increasing token and time costs.
- Ensembling Method Evaluation (EME): EME uses sampling, majority voting, and rounded-mean fallback to aggregate evaluations from multiple LLMs.It also selects feedback whose total score is closest to the final ensemble score.
5 Metrics and Evaluation
The evaluation aligns experiment scores with question-specific ground truth through linear scaling and compares graders using complementary measures of ordering, agreement, and strictness. Leniency supplements correlation-based metrics by exposing systematic generosity or strictness.
- Scaling: Experiment scores are linearly scaled to the question-specific rubric before comparison when the evaluation and base rubrics use different scales.For OOP, the question-specific rubric reaches 35 points, while the question-agnostic rubric uses a 5-point holistic scale.
- Scaling: Method-level PRE scores are aggregated across methods and scaled to produce a one-dimensional student-level dataset.Each method's marks are weighted by its maximum marks in the question-specific rubric.
- Binning: Continuous scores are converted into five rank-based bins before Cohen-Kappa is calculated.Sensitivity checks with four and six bins changed κ by no more than 0.02.
- Evaluation Metrics: Seven complementary statistics assess student ordering, grader agreement, and absolute bias or strictness.The measures include Pearson, Spearman, and Kendall correlations; ICC variants and Cohen-Kappa; and Leniency.
- Evaluation Metrics: Leniency measures relative grading strictness against base scores, with positive values indicating generosity and negative values indicating strictness.The paper identifies ±0.1 as a target range when average deviation is 10%.
6 Results
The results compare rubric-based and rubric-free code evaluation techniques across OOP and DSA datasets, showing stronger performance for LLM graders and question-specific rubrics in key settings. They also show that prompt granularity, model size, and ensemble size affect evaluation strictness and effectiveness.
- Overall Observations: LLM-based techniques significantly outperform CodeBERTScore across evaluation metrics, while both question-agnostic and question-specific rubrics outperform the no-rubric CodeJudge technique.The authors attribute rubric gains to giving the grader an anchor for evaluating code.
- DSA Results: On DSA, switching from EME-QA to EME-QS raises ICC3 from 0.560 to 0.819 and Pearson r from 0.562 to 0.825.The comparison is between question-agnostic and question-specific rubrics within the EME approach.
- DSA Results: EME with a question-agnostic rubric achieves significantly better correlation than Five Point Marking, which uses five predefined marking points.The paper reports that direct assessment of the whole code performs better than grading from only a problem statement and code snippet against fixed points.
- OOP Results: Question-specific and question-agnostic EME approaches achieve comparable performance across all metrics on the OOP dataset.The authors relate this result to the dataset’s homogeneous, implementation-oriented questions.
- Prompt Granularity: PRE lowers average scores by 11.5/35 marks with leniency = −0.329, whereas CRE has leniency = 0.081 and remains near human strictness.Single-criterion prompts tend to assign zero when exact logic is absent, while whole-rubric prompts allow partial credit.
- EME Performance: EME performance improves with larger model parameter sizes, while ensemble gains typically plateau around three to four high-parameter models.The reported high-parameter examples include GPT-4 and Claude 3.
7 Discussion
The discussion describes practical ways to combine CRE, PRE, and EME in instructional workflows and use their outputs for criterion-level learning analytics. It emphasizes staged grading, stricter checking for high-stakes cases, and feedback that supports targeted instructional responses.
- Instructional Workflow: A staged workflow can use CRE for rapid triage and reserve PRE or human review for borderline submissions.The proposed workflow uses lightweight screening before stricter evaluation.
- Instructional Workflow: PRE may provide additional reassurance in summative examinations when false positives carry high costs, while trading away some overall agreement.The discussion characterizes PRE as strict and suitable for high-stakes settings.
- Learning Analytics: Aggregated rubric-criterion misses can reveal common misconceptions and guide targeted recitations, micro-lectures, or discussion-board posts.The proposed analytics use JSON traces from the graders.
- Student Feedback: Rubric-aligned comments anchored to specific steps can help learners understand why marks were lost instead of receiving only a generic wrong-output notice.The discussion suggests organizing missed criteria, probable causes, and planned fixes in a worksheet.
8 Limitations and Future Work
The study’s scope is limited by its primary use of GPT-4o, Java programming questions, and single-file assignments from two intermediary courses. The authors identify comparisons across models, languages, course levels, assignment structures, and rubric granularities as future work.
- Model Scope: The study mainly uses GPT-4o and does not compare variability across different LLMs or open- versus closed-source models.The authors state that training methodologies may influence evaluation effectiveness and accuracy.
- Dataset Scope: The analysis covers Java questions from two intermediary courses with single-file assignments, excluding other languages and multi-file advanced-course settings.The authors specifically mention Python, C++, and advanced courses requiring multiple files as future directions.
- Rubric Scope: The effectiveness of low-, medium-, and fine-grained rubrics is not compared systematically.The authors identify rubric-granularity analysis as another future research avenue.
9 Conclusion
The paper evaluates question-specific versus question-agnostic rubrics using new DSA and OOP datasets and compares proposed techniques with existing approaches using Leniency alongside established metrics. Its findings indicate potential benefits for improving the logical assessment of code and motivate further work on evaluation methods and dataset availability.
- Conclusion: The study introduces DSA and OOP datasets containing solutions with varying correctness levels to evaluate LLM-based code grading.These datasets support comparison between question-specific and question-agnostic rubric approaches.
- Conclusion: The proposed techniques are assessed against existing approaches using established metrics and the new Leniency metric.Leniency is used alongside existing evaluation measures to compare grading behavior.
- Conclusion: The findings demonstrate potential benefits of question-specific rubrics for enhancing logical assessment of code.The conclusion also identifies refinement of evaluation methods and expanded dataset availability as future research directions.
A Prompts for Techniques
The prompts define three code-evaluation techniques: holistic rubric scoring, rubric-point decisions, and approach-specific assessment. Across them, evaluators infer logical intent, follow rubric criteria exactly, and return structured JSON feedback.
- A.1 CRE: The CRE prompt evaluates university-level Java object-oriented submissions using question-specific rubrics, compiler output, and logical correctness rather than syntax.Code formatting and placement relative to instructor comments are treated as nonessential.
- A.1 CRE: CRE instructs evaluators to infer student intent despite syntax errors and return marks and feedback for every rubric line in a JSON dictionary.The required output contains rubric lines as keys with awarded marks and feedback, without additional text.
- A.2 PRE: The PRE prompt narrows evaluation to one specified rubric point and requires a YES or NO decision based on whether its logic is correctly implemented.Because partial marking is excluded, the prompt permits liberal judgments about code quality while retaining logical correctness as the basis.
- A.2 PRE: PRE requires a dictionary containing only the decision and feedback, while accepting flexible code formatting and code placed before or after instructor comments.The requested response uses the keys DECISION and FEEDBACK and forbids additional text.
- A.3 EME: The EME approach-identification stage selects the exact rubric-named algorithmic approach by matching code patterns, variable names, and algorithm structures.It returns the approach name, a confidence value, and a brief explanation tied to matching code patterns.
- A.3 EME: After identifying an approach, EME evaluates only that approach’s exact criteria and point allocations, preserving rubric wording and prohibiting newly invented criteria.The evaluation includes criterion scores, maximum scores, feedback, totals, and ratings for approach correctness, code correctness, efficiency, and readability.
- A.3 EME: The EME rubric distinguishes multiple solution approaches, each with its own criteria and point allocations, so code analysis must first determine the student’s approach.Key patterns, variable names, and algorithm structures guide this classification before scoring.
- A.3 EME: EME uses valid JSON-only outputs for both approach identification and code evaluation, including structured criterion-level feedback and summary ratings.The approach must be named exactly as it appears in the rubric, such as Brute Force, Dynamic Programming, or Kadane’s Algorithm.