Source-linked AI summary
Automated Grading and Feedback Tools for Programming Education: A Systematic Review
Marcus Messer, Neil C. C. Brown, Michael Kölling, Miaojing Shi
TL;DR
This systematic review examines how automated grading and feedback tools assess programming skills, generate feedback, automate assessment, and evaluate performance. Across 121 papers from 2017–2021, correctness assessment dominated, feedback was often limited, and evaluation datasets were typically unavailable.
Problem
Automated assessment must provide timely feedback for growing programming courses, but grading time is short and feedback quality can be inconsistent.
Method
The review categorised 121 research papers from 2017–2021 by assessed skills, grading and feedback techniques, programming paradigms, automation degree, and evaluation methods.
Results
Most tools assessed correctness dynamically, usually with unit testing, while static analysis commonly compared submissions with reference solutions and assessed maintainability, readability, or documentation less often.
Takeaways & Limitations
Fully automated assessment supports near-instantaneous feedback and multiple resubmissions, while limited feedback and unavailable evaluation datasets constrain progress and reproducibility.
Takeaways & Limitations
The review may have missed relevant studies outside IEEE, ACM, or Scopus-indexed publications and may have mistakenly excluded papers during title and abstract screening.
Abstract
from arXiv · showhide
We conducted a systematic literature review on automated grading and feedback tools for programming education. We analysed 121 research papers from 2017 to 2021 inclusive and categorised them based on skills assessed, approach, language paradigm, degree of automation and evaluation techniques. Most papers assess the correctness of assignments in object-oriented languages. Typically, these tools use a dynamic technique, primarily unit testing, to provide grades and feedback to the students or static analysis techniques to compare a submission with a reference solution or with a set of correct student submissions. However, these techniques' feedback is often limited to whether the unit tests have passed or failed, the expected and actual output, or how they differ from the reference solution. Furthermore, few tools assess the maintainability, readability or documentation of the source code, with most using static analysis techniques, such as code quality metrics, in conjunction with grading correctness. Additionally, we found that most tools offered fully automated assessment to allow for near-instantaneous feedback and multiple resubmissions, which can increase student satisfaction and provide them with more opportunities to succeed. In terms of techniques used to evaluate the tools' performance, most papers primarily use student surveys or compare the automatic assessment tools to grades or feedback provided by human graders. However, because the evaluation dataset is frequently unavailable, it is more difficult to reproduce results and compare tools to a collection of common assignments.
1 INTRODUCTION
Growing course enrolments and assignment loads make timely, consistent grading difficult, motivating automated assessment tools. This review examines recent automated grading and feedback research and categorises its tools, techniques, and evidence.
- Motivation: Larger computer science courses create more assignments to grade within short evaluation windows, increasing risks of inconsistent grading and low feedback quality.Prompt and timely feedback is associated with greater student satisfaction.
- Motivation: Automatic assessment tools can fully automate marking and feedback or identify potential issues for instructors conducting manual assessment.They are used to provide consistent grades and feedback for large cohorts.
- Motivation: Assessment encompasses grading and/or feedback, spanning formative support for learning and summative judgements against standards.Some tools provide only formative feedback, while many provide grades and feedback for summative assessment.
- Research gap: Traditional tools focus on program correctness through unit tests and pattern matching but often struggle with design quality and identifying misunderstandings.Research has only recently begun addressing these additional areas.
- Research gap: Unit-test-based tools require comprehensive test suites and often constrain solution structure, encouraging short, well-defined coursework rather than open-ended projects.Large-scale projects may be less well-defined to support student choice and creativity.
- Review scope: The review systematically investigates recent automated grading and feedback research, categorising tools by programming skills and summarising techniques, paradigms, evaluation methods, and research gaps.Its stated contributions include a state-of-the-art summary and detailed statistics.
2 PROGRAMMING SKILLS
The review frames programming-assignment assessment around correctness, maintainability, readability, and documentation. These criteria cover functional and methodological compliance, code quality, understandability, and documentation quality.
- Assessment criteria: The review uses four core criteria for assessing programming assignments: correctness, maintainability, readability, and documentation.The criteria can be graded manually or automatically through static or dynamic evaluation.
- Correctness: Correctness measures whether a student implemented the assignment according to its specification, including functionality and required programming methodology.Examples include testing features and verifying recursion, loops, modulo, polymorphism, or inheritance.
- Maintainability: Maintainability concerns elegant implementation, including reducing duplication, avoiding unnecessary complexity, and lowering class coupling.Functions, polymorphism, and inheritance are examples of relevant design choices.
- Readability: Readability concerns whether code is easy to understand through style compliance, meaningful naming, constants instead of magic numbers, and whitespace.Maintainable code contributes to readability but does not exhaust its indicators.
- Documentation: Documentation assessment examines the existence and quality of inline comments and docstrings explaining functionality, implementation, purpose, and interactions.The review notes that comments can be useless or poor quality.
3 CATEGORIES OF AUTOMATIC ASSESSMENT TOOLS
Automatic assessment tools combine dynamic and static techniques, with machine-learning approaches spanning both categories. Their methods range from execution-based testing and efficiency checks to source comparison, quality analysis, and automated hints.
- Overview: Automatic assessment tools reduce assessment time and can provide real-time feedback through unit testing or comparison with model solutions and correct submissions.Multiple methods may be combined to assess different learning outcomes.
- Dynamic analysis: Dynamic analysis evaluates running programs through executions, commonly using unit tests that compare printed output or method return values.It can also assess coding efficiency or test-suite completeness.
- Static analysis: Static analysis evaluates software without running it or considering inputs, using tools such as linters to identify issues and assess code quality.These tools commonly address maintainability, readability, and documentation.
- Static analysis: Software metrics assess maintainability using measures such as program length, comprehension difficulty, programming effort, and cyclomatic complexity.Halstead metrics and McCabe’s Cyclomatic Complexity are cited examples.
- Static analysis: Static comparison methods evaluate similarity between student code and model or applicable solutions, including representations such as abstract syntax trees.They target correctness despite syntactic differences between functionally equivalent programs.
- Machine learning: Machine-learning tools use dynamic or static approaches for grading and feedback, while comparative methods can generate next-step hints from student traces.Some ML approaches require large ground-truth datasets, whereas zero-shot learning addresses tasks without prior historical data.
- Degree of automation: Fully automated assessment supports smaller assignments and reduces instructor workload, whereas semi-automated assessment combines automation with manual grading for large-scale work and design quality.Fully automated grading has difficulty with large-scale assignments.
4 RELATED WORK
Earlier reviews examined automated assessment tools, feedback, techniques, and historical trends, but many were narrow, non-systematic, or outdated. This review differentiates itself through systematic searching, detailed extraction, pedagogical analysis, and expanded categorisation.
- Prior reviews: Previous reviews covered automated grading and feedback, including assessed domains, testing, security, feedback generation, usability, supported languages, and verification.Several reviews also classified tools by automation degree, orientation, specialisation, or platform.
- Prior reviews: Feedback-focused research reviewed feedback types, teacher inputs, adaptability, and evaluation quality, commonly identifying mistakes and next steps as feedback categories.Technical evaluations sometimes compared generated grades or feedback with existing graded-work datasets.
- Review distinction: Several reviews were small, non-systematic, and hand-picked tools, whereas this review systematically searches the literature and extracts more detail from each system.This distinction defines a methodological difference from those reviews.
- Review distinction: Earlier grading reviews covering 2006–2010 are outdated in the context of recent growth in automated assessment systems.The comparison establishes a need to examine newer research.
- Review distinction: Compared with Paiva et al.’s concurrent review, this work examines evaluation against human graders in greater detail and adopts a more pedagogical focus.Its focus includes how well tools grade, what they grade, and what they should be grading.
- Contribution: The review introduces skill-based categories covering correctness, maintainability, readability, and documentation, adds machine learning as a static-and-dynamic subcategory, and analyses evaluation and data availability.It also compares its findings with previous literature reviews.
5 METHODOLOGY
The review used a preregistered, systematic process to identify, screen, assess, and synthesise research on automated grading and feedback tools. Searches across three databases were validated with benchmark papers and applied explicit eligibility criteria with two-screeners screening.
- Review design: A systematic literature review locates, assesses, and interprets accessible evidence to summarise findings, identify gaps, and support further research.The review process includes protocol development, study identification, selection, quality assessment, data extraction, and synthesis.
- Review design: The authors preregistered their research questions, search terms, screening procedures, data extraction, and data synthesis with the Open Science Foundation.
- Research questions: The review asked which techniques, languages, programming skills, feedback methods, and evaluation approaches are most common in automated assessment research.It also examined performance and feedback comparability with human graders.
- Search process: Search strings for grading and feedback were built from research-question keywords, expanded with synonyms, and refined to avoid irrelevant results.The queries excluded sources focused on robotics, source-code vulnerability, and information and communication technology.
- Search process: The authors searched ACM Digital Library, IEEE Xplore, and Scopus because these databases cover relevant computer-science education literature and additional peer-reviewed sources.They rejected other databases because of excessive results or limitations handling their Boolean search strings.
- Selection and screening: Known benchmark papers were used to validate the search strings, after which explicit eligibility criteria and staged screening determined the included studies.Two screeners reviewed sources at every stage, resolved conflicts through discussion, and extracted data from full texts.
6 RESULTS
The review found that programming assessment tools predominantly target correctness, especially in object-oriented languages, using dynamic testing and static comparison techniques. Most tools are fully automated, while broader code-quality assessment and advanced feedback approaches remain less common.
- Approaches: Most tools assessed correctness with dynamic or static techniques, while readability and maintainability were usually assessed with static analysis.Dynamic tools primarily used unit testing; static tools commonly compared submissions with model or existing correct solutions.
- Degree of automation: 81% of tools were fully automated, compared with 14% using semi-automated assessment.Full automation supported near-instantaneous feedback and multiple resubmissions, while semi-automation assisted manual grading or verified automated grades.
- Skills assessed: 66% of reviewed research focused only on assessing correctness.Correctness-focused tools commonly assess functionality through unit testing or methodology through language-feature checks.
- Code quality: Only four tools focused exclusively on readability, and no papers focused entirely on documentation.AppGrader assessed documentation presence and best-practice adherence but did not analyse documentation quality.
- Languages assessed: 69% of tools graded object-oriented languages, making this the most developed language-paradigm category.Java and Python were cited as examples of object-oriented languages assessed by these tools.
7 DISCUSSION
The review finds that automated assessment enables rapid, scalable feedback but remains concentrated on correctness and constrained by limited feedback, assignment structure, and evaluation practices. It identifies broader assessment of code quality, shared benchmarks, and complementary human involvement as important areas for improvement.
- Feedback: Unit-test-based tools often provide insufficient feedback by reporting test outcomes, output differences, exceptions, or compiler messages without explaining how students should fix problems.Such feedback can frustrate students, including when failures arise from subtle differences such as unnoticed trailing spaces.
- Assessment scope: Most automated tools cannot award partial grades for incomplete or uncompilable programs or distinguish qualitatively different incorrect solutions, often assigning zero where human graders would award partial marks.Code-repair techniques can restore compilability before testing and support partial grades, typically by deducting marks from repaired-code test results.
- Assignment design: Strict code structures and small-scale closed-ended assignments can reduce students’ opportunities for creativity and for learning maintainable, readable, and well-designed code.Open-ended assignments are difficult and time-consuming to assess automatically at scale, while multiple human graders can introduce consistency issues and cannot provide near-instantaneous feedback.
- Assessment scope: Correctness dominates automated assessment, while few tools assess maintainability, readability, or documentation and those that do typically use minimal static analysis.Static-analysis outputs often report a number or identify missing elements without adapting guidance for novice programmers.
- Degree of automation: Fully automated assessment provides near-instantaneous feedback and can encourage early submission and misconception correction, but typically limits assignment scope.Most fully automated tools use unit testing, which favors closed-ended or structured assignments and often provides limited code-quality assessment.
- Evaluation: Evaluation commonly relies on student surveys or comparisons with human grading, but unavailable and rarely reused datasets hinder reproducibility and cross-tool comparison.Public shared datasets would support validation and informed tool selection, especially for less commonly assessed skills.
8 CONCLUSION
The review finds that automated assessment tools primarily grade correctness in object-oriented programming through dynamic testing or static comparison, while offering rapid feedback and resubmission opportunities. However, assessment and evaluation remain limited for design-oriented skills, open-ended work, and reproducible comparison across tools.
- Findings: Most tools assess correctness dynamically, primarily using unit tests, but feedback often reports only test status or expected versus actual outcomes.This limited feedback may leave students without enough detail to progress.
- Findings: Static analysis commonly compares submissions with reference solutions or correct student submissions, while less often assessing maintainability, readability, and documentation.These design-oriented skills are typically assessed alongside correctness grading.
- Automation: Most tools provide fully automated assessment, enabling near-instantaneous feedback and multiple resubmissions without increasing grading workload.Fully automated tools typically constrain assignments to smaller scales and limit opportunities for creativity.
- Automation: Semi-automated tools combine automatic correctness assessment with human grading of maintainability, readability, and documentation, but do not typically remove assignment-scope limitations.They are faster than manual assessment but do not provide near-instantaneous feedback.
- Coverage: Object-oriented languages such as Java, Python, and C++ are assessed most often, while functional and logic languages receive less research attention.The review attributes this focus to the prominence of object-oriented languages in education and industry.
- Evaluation: Evaluations most often use student surveys or comparisons with human graders, while unpublished evaluation datasets hinder reproducibility and comparison across common assignments.The review recommends combining benchmark accuracy with user surveys and publishing annotated datasets.