PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors

Yun Peng, Shuzheng Gao, Cuiyun Gao, Yintong Huo, Michael R. Lyu

arXiv:2306.01394v1cs.SE

TL;DR

Python’s dynamic typing brings run-time type errors, while existing repair prompts lack type-specific guidance. TypeFix mines generalized fix templates and uses them to guide code prompts; it repairs more errors than the strongest baselines on both benchmarks.

  • Problem

    Existing prompt-based repair uses manually defined prompts without type-specific clues, limiting its effectiveness on complex Python type-error patterns.

    Source: §1 · pp. 1–2

  • Method

    TypeFix mines generalized fix templates with hierarchical clustering and applies them to buggy code to guide prompts for a code pre-trained model.

    Source: §1 · p. 2

  • Results

    TypeFix fixes 26 of 54 BugsInPy errors, 9 more than Codex, and 55 of 109 TypeBugs bugs, 14 more than PyTER.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Mined templates cover about 75% of type errors in both benchmarks, compared with PyTER’s 40%.

    Source: §1 · p. 2

    Mined templates do not match about 25% of cases whose patterns are not represented in the training set.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

Although the dynamic type system of Python facilitates the developers in writing Python programs, it also brings type errors at run-time. There exist rule-based approaches for automatically repairing Python type errors. The approaches can generate accurate patches but they require domain experts to design patch synthesis rules and suffer from low template coverage of real-world type errors. Learning-based approaches alleviate the manual efforts in designing patch synthesis rules. Among the learning-based approaches, the prompt-based approach which leverages the knowledge base of code pre-trained models via pre-defined prompts, obtains state-of-the-art performance in general program repair tasks. However, such prompts are manually defined and do not involve any specific clues for repairing Python type errors, resulting in limited effectiveness. How to automatically improve prompts with the domain knowledge for type error repair is challenging yet under-explored. In this paper, we present TypeFix, a novel prompt-based approach with fix templates incorporated for repairing Python type errors. TypeFix first mines generalized fix templates via a novel hierarchical clustering algorithm. The identified fix templates indicate the common edit patterns and contexts of existing type error fixes. TypeFix then generates code prompts for code pre-trained models by employing the generalized fix templates as domain knowledge, in which the masks are adaptively located for each type error instead of being pre-determined. Experiments on two benchmarks, including BugsInPy and TypeBugs, show that TypeFix successfully repairs 26 and 55 type errors, outperforming the best baseline approach by 9 and 14, respectively. Besides, the proposed fix template mining approach can cover 75% of developers' patches in both benchmarks, increasing the best rule-based approach PyTER by more than 30%.

1 INTRODUCTION

Python’s dynamic typing supports rapid development but leaves type errors to be repaired at run time. TypeFix mines fix templates to guide prompts and repairs more errors than the strongest reported baselines on both benchmarks.

1 INTRODUCTION

  • Rule-based repair can generate accurate patches, but manual templates have low real-world coverage and require substantial domain-expert effort.Static type checkers and type inference can reduce potential errors but provide limited help repairing existing ones.Source: §1 · p. 1
  • Predefined prompts lack type-specific clues, limiting their ability to repair type errors with complex patterns.Automatically incorporating domain knowledge is challenging because type errors and their fixes occur at different levels and have varied patterns.Source: §1 · p. 2
  • TypeFix mines and generalizes fix templates from existing fixes, then applies ranked templates to buggy code to prompt CodeT5.The approach is fully automated and does not require manually defined templates or patch synthesis rules.Source: §1 · p. 2
  • TypeFix fixes 26 of 54 BugsInPy errors, 9 more than Codex, and 55 of 109 TypeBugs bugs, 14 more than PyTER.Its mined templates cover about 75% of type errors in both benchmarks, compared with PyTER’s 40%.Source: §1 · p. 2
  • TypeFix is presented as the first domain-aware prompt-based approach for repairing Python type errors.The paper also proposes a fix-template design for different error levels and a hierarchical clustering approach to mine templates.Source: §1 · p. 2

2 MOTIVATION

The scrapy example requires converting the entire string to bytes with a user-defined `to_bytes` function. TypeFix generalizes a conversion template and uses it to guide where its prompt masks are placed.

2 MOTIVATION

  • The correct repair wraps the entire string in the user-defined `to_bytes` conversion function.The example comes from the scrapy project in BugsInPy.Source: §2 · pp. 2–3
  • The baselines miss the required whole-string conversion in different ways, from editing string contents to adding checks around individual variables.AlphaRepair misses the mask location without domain knowledge; PyTER also fails to introduce the required user-defined conversion function.Source: §2 · p. 2
  • TypeFix generalizes a variable-level type-conversion pattern to an expression-level template, despite the latter’s low frequency.It uses the template to wrap the buggy string and mask the new function name rather than masking arbitrary tokens.Source: §2 · p. 3

3 APPROACH

TypeFix mines generalized fix templates from existing type-error fixes and uses selected templates in adaptive prompts for CodeT5 to generate patches. It validates the patches with syntax checks, test-suite execution, and manual review of plausible patches for semantic correctness.

3 APPROACH

  • TypeFix’s approach has two phases: mining domain-aware fix templates and generating patches by selecting templates, building prompts, and invoking CodeT5 to fill their masks.The mining phase parses specific templates and clusters them into general templates; the generation phase applies mined templates to buggy code.Source: §3 · p. 3

3.1 Mining Phase

  • Mining converts type-error fixes into specific templates, then abstracts and merges them into general templates using hierarchical clustering.It organizes specific-to-general templates into clustering trees.Source: §3.1 · p. 3, §3 · p. 3

3.1.1 Definition of Fix Template.

  • A fix template combines a fix pattern describing the edit with internal context locating it and external context locating that context in the program.Template trees use abstract-syntax-tree structure with eight higher-level base types, enabling different AST node types to share a generalized representation.Source: §3.1.1 · p. 3

3.1.2 Fix Parsing.

  • Fix parsing extracts edited AST subtrees, prunes shared structure to isolate changes, and derives internal context for expression-level edits and external context from statements sharing variables with the fix.Statement-level edits have empty internal context; external context can also locate new statements that do not modify existing buggy code.Source: §3.1.1 · p. 3, §3.1.2 · p. 4

3.1.3 Fix Template Mining.

  • Template mining clusters the most similar templates while minimizing lost domain knowledge, using structural and value similarity and abstracting patterns top-down and contexts bottom-up.Abstraction preserves identical nodes, replaces differing values or types with holes, or removes nodes without common attributes; the process prioritizes external context, then internal context, then fix patterns.Source: §3.1.3 · p. 5

3.2 Patch Generation Phase

  • Patch generation has two processes: matching and ranking suitable fix templates, then applying them to create prompts for a pretrained code model to predict masks.The approach stores specific-to-general templates in clustering trees for matching and prompt construction.Source: §3 · p. 3, §3.2 · pp. 6–7

3.2.1 Fix Template Matching.

  • Template matching uses breadth-first search on clustering trees to find the deepest matching templates, then ranks solutions by training frequency and groups by lower abstraction ratio.Lower abstraction in the replacement pattern leaves more domain knowledge in the prompt for the model.Source: §3.2.1 · p. 7

3.2.2 Prompt-based Patch Generation.

  • For each selected template, TypeFix replaces the matching buggy-code subtree with its completed replacement pattern, converts holes to masks, and uses CodeT5 to predict their values.CodeT5’s predicted mask values are filled into the prompts to produce candidate patches.Source: §3.2.2 · pp. 7–8
  • TypeFix filters generated patches with syntax errors, then runs the test suite to identify plausible patches that pass all test cases.Patches that pass both stages are considered plausible.Source: §3.2.2 · p. 8
  • The authors manually classify plausible patches as correct when they are semantically identical to the developer patch, ignoring I/O side effects such as print messages.Source: §3.2.2 · p. 8

4 EXPERIMENTAL DESIGN

The experiments use two Python type-error benchmarks, compare TypeFix with rule-based and learning-based baselines, and evaluate repair performance and template coverage. The implementation uses CodeT5-base for mask prediction and patch generation.

4.2 Baselines

  • The four baselines are PyTER, AlphaRepair, CoCoNuT, and Codex.PyTER is rule-based; AlphaRepair and CoCoNuT are general-purpose prompt-based and NMT-based approaches, respectively, while Codex is a large GPT model fine-tuned on public code.Source: §4.2 · p. 8

4.3 Metrics

  • Evaluation uses Correct and Plausible metrics, plus Template Coverage, the ratio of bugs whose developer patch matches an approach’s fix template.Table 1 reports results in Correct/Plausible format; fix rate is the ratio of correct patches.Source: §4.3 · p. 8, Table 1 · p. 8

4.4 Implementation

  • The Python implementation uses CodeT5-base to predict prompt masks and generate candidate patches.Codex was queried through the public code-davinci-002 API; its prompt used three examples and masked only the buggy line.Source: §4.4 · pp. 8–9

5 EVALUATION

Evaluation examines TypeFix’s repair effectiveness, fix-template mining, and failure cases. Across two benchmarks, it compares repair results with rule-based and learning-based approaches, measures template coverage, and analyzes limitations.

5.1 RQ1: Effectiveness of TypeFix

  • TypeFix has more unique fixes than PyTER and the learning-based approaches; Table 2 compares unique fixes and template coverage between TypeFix and PyTER.TypeFix obtains 24 and 16 unique fixes in TypeBugs and BugsInPy, respectively, versus PyTER’s 10 and 5; it also finds 16 and 10 unique fixes in TypeBugs and BugsInPy, while other learning approaches find 0 ∼3.Source: §5.1 · p. 9, Table 2 · p. 9
  • TypeFix achieves a 1× ∼4× larger fix rate than AlphaRepair and fixes 16 and 9 more bugs than Codex in TypeBugs and BugsInPy, respectively.Source: §5.1 · p. 9
  • TypeFix fixes 55 and 26 bugs in the two benchmarks, outperforming state-of-the-art approaches by at least 14 and 9 bugs, respectively.It also obtains the most unique type error fixes in both benchmarks.Source: §5.1 · pp. 9–10

5.2 RQ2: Capability of TypeFix to Mine Fix Templates

  • Mined templates cover about 75% of type errors in both benchmarks, compared with PyTER’s 30% ∼40%; mining yields 5 ∼70 trees and takes under a minute to nine hours.Source: §5.2 · p. 10
  • All four template categories contribute to correct patches, with Insert and Add contributing most; developers often add guards or convert input types when fixing type errors.Source: §5.2 · p. 10

5.3 RQ3: Limitations of TypeFix

  • TypeFix fails when no mined template matches a bug’s pattern and when CodeT5 cannot generate a correct patch despite a matching template.About 25% of cases lack similar training-set patterns; in TypeBugs, templates cover 83 bugs but only 55 are correctly fixed.Source: §5.3 · p. 10

6 RELATED WORK

Related work contrasts rule-based APR, which relies on manually specified templates, with learning-based repair, including prompt-based methods. TypeFix adds mined, domain-aware fix templates to prompts for Python type errors.

6.1 Automatic Program Repair

  • TypeFix automatically mines type-error fix templates from real-world fixes, addressing the limited real-world coverage of traditional rule-based approaches.PyTER uses nine predefined templates; TypeFix also designs templates for different error levels.Source: §6.1 · p. 10
  • AlphaRepair reformulates repair as fill-in-the-blank prediction, but its general domain-unaware prompts can struggle with complicated type errors.TypeFix incorporates domain knowledge through fix templates mined from existing type-error fixes.Source: §6.1 · p. 11

6.2 Pre-trained Language Models

  • Code pre-trained models include CodeBERT, GraphCodeBERT, CodeGPT, Codex, and CodeT5, and prompt learning applies them to repair by predicting code tokens.The passage describes models with differing architectures and pre-training setups.Source: §6.2 · p. 11

7 CONCLUSION

TypeFix repairs Python type errors by incorporating mined fix templates into prompts for code pre-trained models. Its template design spans error levels, and experiments demonstrate the approach's effectiveness and the usefulness of the mined templates.

7 CONCLUSION

  • TypeFix incorporates domain-aware fix templates into prompts to repair Python type errors.The approach is named TypeFix and is prompt-based.Source: §7 · p. 11
  • TypeFix designs templates for different type-error levels and mines them using hierarchical clustering.It applies the mined templates to buggy code before invoking code pre-trained models to generate candidate patches.Source: §7 · p. 11
  • Experiments demonstrate TypeFix's effectiveness and the usefulness of its mined fix templates.Source: §7 · p. 11

Related papers

Loading 2306.01394v1…