PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks

Sahil Bhatia, Rishabh Singh

arXiv:1603.06129v1cs.PLcs.AIcs.LGcs.SE

TL;DR

AST-dependent feedback techniques cannot analyze student programs with syntax errors, a substantial class of submissions. This paper trains assignment-specific RNNs on syntactically correct submissions to suggest token-level repairs, completely correcting 31.69% of the evaluated syntax-error submissions and partially correcting another 6.39%.

  • Problem

    Syntax errors account for more than 34% of submissions, but AST-dependent feedback methods cannot analyze programs containing them.

    Source: §5 · p. 10

  • Method

    For each assignment, the system trains an RNN on syntactically correct student submissions and uses predicted token sequences to suggest syntax repairs.

    Source: §1 · p. 2

  • Results

    31.69% (4501/14203) of submissions were completely corrected, and an additional 6.39% (908/14203) received fixes for their first syntax error.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The system provides potential syntax fixes for student submissions, including cases AST-dependent feedback techniques cannot analyze.

    Source: §1 · p. 2, §5 · p. 10

    The algorithm currently handles only one syntax error in a student program.

    Source: §6 · p. 10

Abstract

from arXiv · show

We present a method for automatically generating repair feedback for syntax errors for introductory programming problems. Syntax errors constitute one of the largest classes of errors (34%) in our dataset of student submissions obtained from a MOOC course on edX. The previous techniques for generating automated feed- back on programming assignments have focused on functional correctness and style considerations of student programs. These techniques analyze the program AST of the program and then perform some dynamic and symbolic analyses to compute repair feedback. Unfortunately, it is not possible to generate ASTs for student pro- grams with syntax errors and therefore the previous feedback techniques are not applicable in repairing syntax errors. We present a technique for providing feedback on syntax errors that uses Recurrent neural networks (RNNs) to model syntactically valid token sequences. Our approach is inspired from the recent work on learning language models from Big Code (large code corpus). For a given programming assignment, we first learn an RNN to model all valid token sequences using the set of syntactically correct student submissions. Then, for a student submission with syntax errors, we query the learnt RNN model with the prefix to- ken sequence to predict token sequences that can fix the error by either replacing or inserting the predicted token sequence at the error location. We evaluate our technique on over 14, 000 student submissions with syntax errors. Our technique can completely re- pair 31.69% (4501/14203) of submissions with syntax errors and in addition partially correct 6.39% (908/14203) of the submissions.

1. INTRODUCTION

As MOOC enrollment grows, providing individual assignment feedback at scale is difficult. This section presents automated syntax-error repair using models trained on correct student submissions.

1. INTRODUCTION

  • Syntax errors comprise more than 34% of submissions, while individual MOOC feedback is costly and AST-based systems cannot analyze programs with syntax errors.Compiler messages can locate and describe errors but do not generally suggest how to fix them.Source: §1 · pp. 1–2
  • The system trains an RNN on syntactically correct submissions for each assignment, then uses predicted token sequences to suggest fixes to syntax-error submissions.The training submissions may be functionally incorrect, provided they have no syntax errors.Source: §1 · p. 2
  • Unlike corpus-wide language models, the authors train assignment-specific RNNs to capture longer token dependencies for syntax repair.Earlier code models addressed tasks such as next-token suggestions, code completion, and style consistency.Source: §1 · p. 2
  • 31.69% (4501/14203) of submissions were completely corrected, and an additional 6.39% (908/14203) received fixes for their first syntax error.The evaluation covered student solutions for five programming problems in edX’s Introduction to Programming class.Source: §1 · p. 2
  • The paper formalizes repair as RNN token-sequence learning and introduces SYNFIX to rank insertions and replacements of predicted sequences.These contributions target syntax errors that AST-dependent feedback techniques cannot analyze.Source: §1 · p. 2

2. MOTIVATING EXAMPLES

Examples from the recPower Python assignment illustrate how the system uses token predictions to repair syntax and indentation errors. Its repairs may pass syntax checks without producing the intended behavior.

2. MOTIVATING EXAMPLES

  • The algorithm uses the compiler’s error offset to form a prefix, predicts likely next tokens, and tries inserting or replacing them at the error location.When the offset is inaccurate, it uses the prefix ending at the previous line.Source: §2 · p. 2
  • For insertion-type errors, the system tries predicted sequences in increasing length until the syntax error is fixed.Examples include unbalanced parentheses, incomplete expressions, and missing colons after if or else expressions.Source: §2 · p. 2
  • For replacement-type errors, the system substitutes predicted tokens for original ones, addressing cases such as incorrect operators or mismatched parentheses.It constructs prefixes similarly to the insertion strategy.Source: §2 · p. 2
  • When the compiler cannot locate the error accurately, the algorithm ignores the error line and replaces it using a prediction based on the preceding line.This fallback applies to cases such as misspelled keywords and incorrect return expressions.Source: §2 · pp. 2–3
  • Indentation errors, including incorrect indentation after function definitions, conditionals, or loops, use the same insertion-or-replacement strategy.Source: §2 · p. 3
  • A parser-valid repair may still compute the wrong result or raise a runtime exception, although some suggested repairs also correct the program’s semantics.For example, a predicted expression can be syntactically valid without computing the intended power.Source: §2 · p. 3

3. APPROACH

The approach trains a problem-specific RNN on correct submissions, then SYNFIX uses predicted tokens to propose parser-checked edits to syntactically invalid programs.

3.1 Neural Network Model

  • At each step, the RNN computes a hidden state from the current input and previous hidden state, then applies softmax to produce output probabilities.The recurrent connection retains information from earlier steps for processing subsequent tokens.Source: §3.1 · pp. 3, 5
  • Training pairs each token sequence with itself shifted left by one token, minimizing the negative log likelihood of the next-token prediction.At prediction time, the model uses a supplied prefix and feeds each predicted token back as input to generate the next.Source: §3.1 · p. 5, Figure 3 · p. 5
  • For each assignment, the RNN learns valid token sequences from syntactically correct submissions, relabeling rare tokens as a general IDENT token when needed.The frequency threshold controls which tokens receive the IDENT label.Source: §3.1 · p. 5

3.2 The SYNFIX Algorithm

  • SYNFIX predicts a fixed-length token sequence after the error prefix, then tests increasingly long insertions or replacements until parsing succeeds or no fix is found.If those edits fail, it predicts from a prefix that omits tokens on the error line and tests replacing that line with a sequence ending at a newline.Source: §3.2 · p. 6
  • When token-level edits cannot repair a misspelled return statement, SYNFIX drops the error-line tokens from the prefix and predicts a replacement statement, return base.The predicted sequence ends with a newline and fixes the syntax error in the example.Source: §3.2 · p. 6

4. EVALUATION

The evaluation tests syntax-error repair across five edX programming problems, using syntactically correct submissions to train problem-specific token models. It measures repair coverage and examines algorithm choices, network configurations, and vocabulary thresholds.

4.1 Benchmarks

  • The benchmark spans five problems and 40,835 submissions; 34.78% had syntax errors, while evalPoly and compDeriv had fewer attempts because data collection was ongoing.The five problems are recurPower, iterPower, oddTuples, evalPoly, and compDeriv.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 Training Phase

  • All syntax-correct submissions trained each problem-specific token model, with tokens occurring fewer than four times mapped to IDENT.Training used sequence length 10, batch size 50, RMSProp, and at most 40 epochs; average training time was 1.5 hours.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Number of Corrected Submissions

  • 31.69% of syntax-error submissions were completely fixed, 6.39% were partially fixed, and 61.92% received no fix.The system fixes the first syntax error in partially fixed submissions but does not resolve later errors.Source: Table 3 · p. 7, §4.3 · p. 7
  • No single repair choice dominated: SYNFIX tries insertion and replacement variants before replacing the error line using the previous-line prefix, favoring smaller changes first.Offset-1 prefixes performed better than prefixes ending at the parser-reported error token; Previous Line performed best for complete fixes but could make undesirable semantic changes.Source: §4.3 · p. 8, Table 4 · p. 8
  • About 28% of submissions otherwise receiving no repair could get PrevLine feedback, though replacing the error line could introduce new syntax errors.These cases were not counted as partially fixed programs, but the authors considered the suggestions potentially useful as hints.Source: §4.3 · p. 8

4.4 Different Neural Network Baselines

  • RNN-(1,128) fixed the most recurPower submissions completely (1078) and performed best including partial fixes; larger networks degraded, possibly through overfitting.RNNs performed slightly better than LSTMs in this setting.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 Effect of Different Threshold values

  • Threshold t=1 fixed the fewest recurPower errors, while t=4 outperformed t=8; the authors suggest higher thresholds can overgeneralize tokens as IDENT.Without thresholding, rare-token prefixes such as infrequent variable names were difficult for the model; excessive generalization can erase token-specific information needed for repair.Source: §4.5 · pp. 8–9, Table 6 · p. 9

5. RELATED WORK

Related work covers code language models, automated programming feedback, and machine-learning grading in other domains. The paper distinguishes its syntax-repair task from prior applications and AST-dependent methods.

Language Models for Big Code

  • Earlier Big Code work used n-gram models for token suggestions, extended them with semantic annotations and topic modeling, and reported 18–68% accuracy gains over syntactic n-grams.Larger code corpora were also reported to improve n-gram predictive capabilities.Source: §5 · p. 9
  • NATURALIZE learns project coding conventions for stylistic revisions and reported 94% accuracy for identifier-name suggestions; JSNice predicts names and types using CRFs.Other cited neural language models suggest method and class names.Source: §5 · p. 9

5. RELATED WORK

  • This work applies code language modeling to syntax repair, uses RNNs for long-context token relationships, and trains separate models for each programming assignment.Unlike prior AST-dependent techniques, its repair setting cannot use ASTs for submissions with syntax errors.Source: §5 · p. 9

Automated Code Grading and Feedback

  • Prior programming feedback systems used synthesis, dynamic analysis, AST-based clustering or search, and learned embeddings; syntax errors prevent AST-based analysis in these approaches.AutoProf targets functional correctness, while another approach analyzes performance problems; Codewebs propagates teacher feedback through AST queries, and another method uses program embeddings.Source: §5 · pp. 9–10

Machine learning for Grading in Other domains

  • Other-domain grading methods cluster mathematical or short-answer responses and propagate instructor judgments, unlike this paper's RNN token-sequence modeling for syntax errors.Source: §5 · p. 10

6. LIMITATIONS AND FUTURE WORK

The algorithm has several limits: it handles one syntax error at a time, relies on prefix context, and checks syntax rather than semantic correctness. Future work includes better context and repairs, plus feedback that distinguishes typos from deeper misconceptions.

6. LIMITATIONS AND FUTURE WORK

  • The algorithm fixes only one syntax error per submission, so another error can remain and prevent the updated program from passing the compiler check.The example fixes an indentation error on line 3, but a second indentation error remains on line 5.Source: §6 · p. 10
  • Using only the prefix can favor a weaker repair: the example suggests exp == 0, while following context could support exp == 1.The passage proposes investigating bidirectional RNNs for using past and future context, and whether they can improve fix coverage.Source: §6 · p. 10, Figure 8
  • Because repair candidates are checked only for syntactic correctness, a syntax-fixing suggestion can still be semantically incorrect.Future work proposes adding a semantic check and querying multiple predictions until a semantically correct fix is found.Source: §6 · p. 10
  • The paper leaves open how to turn repairs into pedagogical feedback that distinguishes simple typos from errors reflecting deeper misconceptions.Examples of deeper misconceptions include assigning to the return keyword or placing an assignment inside a function-call parameter.Source: §6 · p. 10

7. CONCLUSION

The conclusion presents an RNN-based method that learns token-sequence models from syntactically correct student submissions and uses them to repair syntax errors. On the edX dataset, it generates repairs for about 32% of submissions, and the authors see it as a basis for automated feedback at large scale.

7. CONCLUSION

  • The method trains an RNN on syntactically correct submissions for an assignment, then predicts token sequences to repair submissions with syntax errors.The paper draws on learning language models from big code and efficient RNN learning techniques.Source: §7 · p. 10
  • About 32% of submissions in the edX dataset receive generated repairs.Source: §7 · p. 10
  • The authors present the technique as a potential basis for automated syntax-error feedback to hundreds of thousands of students in online introductory programming courses.Source: §7 · p. 10
Loading 1603.06129v1…