PaperlayerView exact revision on arXiv

Source-linked AI summary

Repairing Bugs in Python Assignments Using Large Language Models

Jialu Zhang, José Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, Gust Verbruggen

arXiv:2209.14876v1cs.SEcs.AI

TL;DR

Custom repair support for novice programming assignments is difficult to scale, and prior symbolic and neural systems have engineering or data demands. MMAPR uses a code-trained language model with multimodal prompts, iterative querying, few-shot selection, and program chunking for syntax and semantic repair. On its benchmark, it repairs more programs than the combined baseline and produces smaller patches.

  • Problem

    Prior student-assignment repair systems use symbolic or neural approaches that can require substantial engineering or data, and typically focus on syntax or semantics alone.

    Source: §I · p. 1

  • Method

    MMAPR uses a code-trained language model with multimodal prompts, iterative querying, test-case-based few-shot selection, and program chunking to repair syntax and semantic mistakes.

    Source: §I · pp. 1–2

  • Results

    MMAPR repairs 86.71% without few-shots and 96.5% with few-shots versus baseline 67.13%, with average token edit distances 31.4 and 31.29 versus 42.50.

    Source: §I · p. 2

  • Takeaways & Limitations

    Using a code-trained language model lets MMAPR address both error types without custom symbolic repair logic or retraining a new neural model.

    Source: §I · p. 1

    Evaluation covers one set of 286 student programs and Python programs only.

    Source: §VI · p. 10

Abstract

from arXiv · show

Students often make mistakes on their introductory programming assignments as part of their learning process. Unfortunately, providing custom repairs for these mistakes can require a substantial amount of time and effort from class instructors. Automated program repair (APR) techniques can be used to synthesize such fixes. Prior work has explored the use of symbolic and neural techniques for APR in the education domain. Both types of approaches require either substantial engineering efforts or large amounts of data and training. We propose to use a large language model trained on code, such as Codex, to build an APR system -- MMAPR -- for introductory Python programming assignments. Our system can fix both syntactic and semantic mistakes by combining multi-modal prompts, iterative querying, test-case-based selection of few-shots, and program chunking. We evaluate MMAPR on 286 real student programs and compare to a baseline built by combining a state-of-the-art Python syntax repair engine, BIFI, and state-of-the-art Python semantic repair engine for student assignments, Refactory. We find that MMAPR can fix more programs and produce smaller patches on average.

I. INTRODUCTION

MMAPR targets scalable feedback for novice programmers by using a code-trained language model to unify syntax and semantic repair. Its multimodal, iterative, few-shot, and chunking techniques outperform the combined baseline on the evaluated programs and produce smaller patches.

I. INTRODUCTION

  • The repair objective includes minimizing changes to support student learning.Source: §I · p. 1
  • Prior student-repair systems use symbolic or neural methods, often requiring substantial engineering or data, and typically target syntax or semantics exclusively.Source: §I · p. 1
  • Classroom repair can draw on test cases, task descriptions, compiler messages, and other submissions, while LLMC-generated unnecessary edits and prompt design also require attention.Source: §I · pp. 1–2
  • MMAPR repairs 86.71% without few-shots and 96.5% with few-shots versus baseline 67.13%, with average token edit distances 31.4 and 31.29 versus 42.50.Source: §I · p. 2
  • Iterative querying raises repair success from 82.87% to 86.71%, few-shots raise it to 96.5%, and removing the chunker increases average token edit distance from 5.46 to 9.38.Combining candidates from different multimodal prompts gives the best performance.Source: §I · p. 2
  • MMAPR combines multimodal prompts, iterative querying, test-case-based few-shot selection, and structure-based program chunking to repair syntax and semantic mistakes with one code-trained language model.Source: §I · pp. 1–2
  • On 286 real student Python programs, MMAPR uses Codex and is compared with a baseline combining BIFI for syntax repair and Refactory for semantic repair.Source: §I · p. 2

II. MOTIVATING EXAMPLE

MMAPR repairs student programs containing syntax and semantic mistakes, sometimes refactoring code while preserving the student’s structure. In the examples, its repairs are closer to the original submissions than the instructor’s reference solutions.

II. MOTIVATING EXAMPLE

  • MMAPR fixes the example’s input-processing and comparison errors, and refactors repeated prime-checking code into a function.The student’s task was to read two numbers and print based on whether they were prime.Source: §II · p. 2, Figure 1 · p. 3
  • The complex palindrome repair addresses syntax and semantic mistakes while preserving the student program’s overall structure.It corrects the lower method, division operators, stray output, and missing even-length checks and output.Source: Figure 2 · p. 3, §II · p. 3
  • 52 versus 97 edit-distance units separate the MMAPR repair and instructor reference from the original student program, respectively.The MMAPR repair is closer to the student submission.Source: §II · p. 3

III. BACKGROUND

The background introduces code-trained language models and prompt-based learning as context for MMAPR. Prompts may combine code, natural language, and data, and their generated completions can vary with settings such as temperature.

III. BACKGROUND

  • A large language model trained on substantial amounts of code is termed an LLMC in this paper.The background describes language models as learned probability distributions over sequences.Source: §III · pp. 3–4
  • Prompt-based learning supplies a textual query and optionally task examples, or shots, to guide iterative token generation.The example prompt asks for a syntax fix and includes buggy code.Source: §III · p. 4
  • Multi-modal prompts combine textual inputs from different modalities, such as natural language, code, and data.Source: §III · p. 4
  • Temperature controls how much the model samples less likely completions, and prompt or sampling choices can change generations.Source: §III · p. 4
  • The paper uses OpenAI’s Codex standard completion model, reporting better performance than its Codex-Edit alternative.CodeGen and BLOOM are also named as possible alternatives.Source: §III · p. 4

IV. METHODOLOGY

MMAPR repairs programs in successive syntax and semantic phases, using syntax checks and assignment tests to validate candidates. It narrows syntax prompts to error-centered code chunks and selects among passing semantic candidates by edit distance.

IV. METHODOLOGY

  • MMAPR first generates syntax candidates and validates them with a syntax oracle, then uses assignment-oriented prompts and tests to select semantic repairs.When several candidates pass all tests, it returns the one with the smallest token edit distance from the original.Source: §IV · p. 4

A. Syntax Phase

  • The syntax phase resolves syntax errors before a program can proceed to semantic repair.This order mirrors students’ typical progression from syntax errors to semantic errors.Source: A. Syntax Phase · p. 4

1) Program Chunking:

  • The program chunker limits syntax prompts to code around the reported error because including the full program can prompt unnecessary edits.It extracts lines containing the error location and nearest encompassing control-flow statement; this approximates a basic block.Source: A. Syntax Phase · p. 4, 1) Program Chunking · p. 4
  • The chunking procedure uses indentation and control-flow structure to include adjacent code and relevant matching branches.The algorithm locates the error, slices nearby code, and adjusts the slice when control-flow keywords occur.Source: 1) Program Chunking · pp. 4–5

2) Syntax Prompt Generator:

The syntax prompt generator queries the LLMC with alternative prompts and uses syntax checks to filter repairs. Valid candidates proceed to semantic repair, where assignment test cases serve as the correctness oracle.

2) Syntax Prompt Generator:

B. Semantic Phase

  • Syntactically valid candidates move from syntax repair into semantic repair.Source: B. Semantic Phase · p. 5
  • The syntax prompt examples contrast prompts without an error message with prompts that include the compiler message.The figure caption says code fragments are elided for brevity.Source: Figure 4 · p. 5, B. Semantic Phase · p. 5
  • The semantic phase uses the instructor’s input-output test suite as its oracle for whether a program is repaired.A repaired program produces the expected outputs for the given inputs.Source: B. Semantic Phase · p. 5

1) Semantic Prompt Generator:

MMAPR’s semantic prompt generator combines assignment context with a syntactically valid program to prompt the language model for repairs. It generates multiple prompt variants and uses the semantic oracle to assess candidates.

1) Semantic Prompt Generator:

  • The generator combines a syntactically valid program, the natural-language task description, and instructor-provided test cases in semantic prompts.When peer solution histories are available, prompts can also include few-shot examples.Source: 1) Semantic Prompt Generator · p. 6
  • The generator produces prompts with different combinations of the available information rather than relying on a single prompt template.The assignment context includes the task description, test cases, and peers’ programming solutions.Source: 1) Semantic Prompt Generator · p. 6
  • Each prompt can generate up to ten candidates, which are checked by executing them on the test suite.The authors heuristically set K to ten to balance candidate exploration with search-space growth.Source: 1) Semantic Prompt Generator · p. 6
  • Among multiple valid candidates after the semantic phase, MMAPR returns the one with the smallest token-based edit distance to the student program.Source: 1) Semantic Prompt Generator · p. 6

2) Few-Shot Learning:

MMAPR’s few-shot approach retrieves paired peer program versions with similar test-suite outcomes, while a zero-shot option remains available without peer programs. Its evaluation uses a filtered benchmark of student programs and a baseline combining syntax and semantic repair tools.

2) Few-Shot Learning:

  • MMAPR’s few-shot bank pairs an incorrect program with its corrected historical version, and retrieves the three most similar pairs for semantic prompts.The incorrect version passes the syntax oracle but fails the semantic oracle; its corrected version satisfies both.Source: 2) Few-Shot Learning · p. 6
  • Program similarity is measured by one minus the normalized Hamming distance between vectors recording which test cases fail.The illustrated shot and target buggy program have the same [pass, fail] test-suite outcomes.Source: 2) Few-Shot Learning · p. 6, Figure 6
  • Without peer programs, MMAPR can use zero-shot querying, which the authors report still performs competitively.Source: 2) Few-Shot Learning · p. 6

V. EVALUATION

  • The evaluation benchmark contains 286 buggy–correct program pairs from student histories across 18 assignments, and the baseline combines BIFI with Refactory.The benchmark retains students with eventually correct programs and selects their latest syntactically erroneous version; three assignments were excluded.Source: §V · p. 7

A. RQ1: Overall Repair Performance

MMAPR repaired more programs than the composed baseline, with few-shot learning increasing its repair rate while keeping edits comparable. It also repaired all benchmark syntax errors, though BIFI performed better on small syntax mistakes in a lower-difficulty assignment.

A. RQ1: Overall Repair Performance

  • 96.50% repair rate with few-shots beat the baseline by 29.37 percentage points and reduced average token edit distance by 26.38%.Without few-shots, MMAPR repaired 86.71% of programs versus 67.13% for the baseline; few-shots kept mean token edit distance comparable at 31.29.Source: A. RQ1: Overall Repair Performance · p. 7
  • 100% syntax repair rate for MMAPR exceeded BIFI’s 80.07%, with mean token edit distance of 5.46 versus 25.07.MMAPR repaired syntax bugs in all 286 benchmark programs.Source: A. RQ1: Overall Repair Performance · p. 8, Table II · p. 8
  • BIFI failed to handle 17 of 286 inputs, potentially because of lexer issues.MMAPR’s prompt-based strategy has no constraints over the input, according to the authors.Source: A. RQ1: Overall Repair Performance · p. 8
  • On assignment 2865, BIFI repaired all syntax errors with lower mean token edit distance than MMAPR, 1.82 versus 2.18.The authors identify combining BIFI with MMAPR as a future direction because their repairs can be complementary.Source: A. RQ1: Overall Repair Performance · p. 8

B. RQ2: Ablation Study

The ablation study examines the effects of MMAPR’s multimodal prompts, iterative querying, test-case-based few-shot selection, and structure-based program chunking.

B. RQ2: Ablation Study

  • The study tests three design choices beyond few-shot selection: multimodal prompts, iterative querying, and structure-based program chunking.The authors state that the few-shot selection effect was already shown in Table I.Source: B. RQ2: Ablation Study · p. 8

1) Program Chunking:

Program chunking reduces the size of syntax-stage edits, while iterative querying improves overall repair rate compared with addressing syntax and semantic bugs in one round.

1) Program Chunking:

  • Chunking selects syntax-error subprograms with surrounding context while excluding irrelevant lines, reducing the code surface presented to the LLMC.The evaluation compares average syntax-phase token edit distance with and without chunking.Source: 1) Program Chunking · p. 8
  • 9.38 to 5.46 tokens (41.79%): program chunking reduces average syntax-phase token edit distance across assignments.The reduction reaches 56.32% on assignment 2878.Source: 1) Program Chunking · pp. 8–9, Table III · p. 9

2) Iterative Querying:

  • 82.87% to 86.71%: splitting syntax and semantic repair into two phases raises the overall repair rate without few-shots.Average token edit distance increases slightly, from 30.29 to 31.40.Source: 2) Iterative Querying · p. 9

3) Multimodal Prompts:

MMAPR combines multimodal prompts and iterative repair, using syntax and semantic oracles to filter candidates. Its prompt candidates are complementary, while syntax fixes that delete code can complicate later semantic repair.

3) Multimodal Prompts:

  • MMAPR combines code, natural-language descriptions, and tests in prompts, then uses syntax or semantic oracles to filter generated candidates.The prompt structures are queried together to test whether their candidates complement one another.Source: 3) Multimodal Prompts · p. 9

4) Combining BIFI and Refactory to Repair Programs:

  • BIFI sometimes deletes code containing syntax errors, which can remove logic needed for subsequent semantic repair.In the example, BIFI deletes the second loop, leaving Refactory with a more difficult semantic repair.Source: 4) Combining BIFI and Refactory to… · p. 9
  • 82.87% to 86.71%: iterative querying raises the overall repair rate by splitting repair into syntax and semantic phases, without few-shots.The table caption identifies the comparison as iterative querying across the two phases.Source: Table IV · p. 9

3) Multimodal Prompts:

  • Ensembling prompt candidates outperforms any individual prompt structure because the candidates are complementary.Among single structures in this experiment, Program + Diagnostics + Description + Tests is the most effective.Source: Figure 7 · p. 9, 3) Multimodal Prompts · p. 9

4) Combining BIFI and Refactory to Repair Programs:

  • In the example, the student used an equals sign where a plus operator was needed, while BIFI removed the loop containing the syntax error.The original snippet shows the incorrect equals sign in the final print statement.Source: 4) Combining BIFI and Refactory to… · p. 9
  • Although Refactory eventually repairs the example, its result is effectively rewritten relative to the original incorrect program.The repair is syntactically equivalent to the reference solution.Source: 4) Combining BIFI and Refactory to… · p. 10
  • Combining specialized syntax and semantic tools does not guarantee the best overall repair and required non-trivial engineering effort.BIFI and Refactory focus on syntax and semantic bugs, respectively.Source: 4) Combining BIFI and Refactory to… · p. 10

VI. THREATS TO VALIDITY

MMAPR’s evaluation is limited to one set of 286 Python programs, and test-based validation is weaker than formal verification. The paper positions its unified LLMC approach as a way to handle student repairs without custom symbolic logic or neural retraining.

VI. THREATS TO VALIDITY

  • MMAPR was evaluated on one set of 286 student programs and only on Python, limiting the evaluated dataset and language scope.The authors identify a larger evaluation dataset as future work and say the prompt-based approach can be applied to other languages.Source: §VI · p. 10

VII. RELATED WORK

  • MMAPR targets student submissions and seeks to minimize edits so students can learn from repaired programs.The paper distinguishes this focus from tools for professional developers’ production bugs.Source: §VII · p. 10
  • Prior educational APR work includes symbolic and neural approaches; MMAPR presents LLMC-based repair as complementary and potentially lower-effort to develop and maintain.The paper describes prior work on feedback generation and program repair in programming education.Source: §VII · p. 10
  • MMAPR uses Codex for student programming tasks, and its design choices such as reducing edit distance may not apply to professional developers.The paper situates Codex among models used for other code-intelligence and repair tasks.Source: §VII · p. 10

VIII. CONCLUSION

  • MMAPR unifies syntax and semantic repair with an LLMC, multimodal prompts, iterative querying, test-case-based few-shot selection, and program chunking.On real student programs, it outperforms the combined syntax-and-semantic baseline while producing smaller patches.Source: §VIII · p. 10

Related papers

Loading 2209.14876v1…