Source-linked AI summary
PyTy: Repairing Static Type Errors in Python
Yiu Wai Chow, Luca Di Grazia, Michael Pradel
TL;DR
Python type annotations can expose errors that developers may lack time to fix, and existing automated repair did not target static Python type errors. PyTy builds a repair dataset and fine-tunes a JavaScript repair model for Python; it removes errors in 85.4% of 281 real-world cases, with 20 of 30 suggested-fix pull requests merged.
Problem
Newly added Python annotations can reveal type errors that developers commonly lack time to fix, while no automated repair approach targeted static Python type errors.
Source: §1 · p. 1Method
Results
PyTy found fixes removing type errors for 85.4% of 281 real-world errors, and 20 of 30 pull requests with suggested fixes were merged.
Source: §1 · p. 2Takeaways & Limitations
PyTy’s suggested fixes were useful in practice, with developers merging 20 of 30 pull requests containing them.
Source: §1 · p. 2, §9 · p. 11PyTyDefects includes only errors fixable with a single-hunk change, so the study does not support conclusions about more complex fixes.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Gradual typing enables developers to annotate types of their own choosing, offering a flexible middle ground between no type annotations and a fully statically typed language. As more and more code bases get type-annotated, static type checkers detect an increasingly large number of type errors. Unfortunately, fixing these errors requires manual effort, hampering the adoption of gradual typing in practice. This paper presents PyTy, an automated program repair approach targeted at statically detectable type errors in Python. The problem of repairing type errors deserves specific attention because it exposes particular repair patterns, offers a warning message with hints about where and how to apply a fix, and because gradual type checking serves as an automatic way to validate fixes. We addresses this problem through three contributions: (i) an empirical study that investigates how developers fix Python type errors, showing a diverse set of fixing strategies with some recurring patterns; (ii) an approach to automatically extract type error fixes, which enables us to create a dataset of 2,766 error-fix pairs from 176 GitHub repositories, named PyTyDefects; (iii) the first learning-based repair technique for fixing type errors in Python. Motivated by the relative data scarcity of the problem, the neural model at the core of PyTy is trained via cross-lingual transfer learning. Our evaluation shows that PyTy offers fixes for ten frequent categories of type errors, successfully addressing 85.4% of 281 real-world errors. This effectiveness outperforms state-of-the-art large language models asked to repair type errors (by 2.1x) and complements a previous technique aimed at type errors that manifest at runtime. Finally, 20 out of 30 pull requests with PyTy-suggested fixes have been merged by developers, showing the usefulness of PyTy in practice.
1 INTRODUCTION
Python’s optional annotations can expose static type errors, but developers often lack time to fix them and choosing a repair is non-trivial. PyTy combines a study of developer fixes, a repair dataset, and cross-language transfer learning to address these errors.
1 INTRODUCTION
- Type annotations can reveal previously unnoticed errors, but developers commonly lack time to fix them, hampering gradual typing’s usefulness.Different errors can require different fixes, and finding the right one is non-trivial.Source: §1 · p. 1
- The authors’ preliminary study found recurring repair patterns but ambiguous rules for choosing among them, while type-checker locations and messages provide useful information.The paper argues that type-specific patterns, diagnostic hints, and automatic fix validation make this repair problem amenable to automated repair.Source: §1 · p. 2
- PyTyDefects contains 2,766 real-world type-error/single-hunk-fix pairs from 176 GitHub repositories, gathered through gradual type checking and delta debugging.The paper presents the dataset as the first of its kind.Source: §1 · p. 2
- PyTy fine-tunes TFix, a JavaScript repair model, with PyTyDefects to transfer learned repair knowledge to Python type errors.The paper motivates transfer learning by the relative data scarcity of the repair problem.Source: §1 · p. 2
- PyTy finds fixes removing type errors for 85.4% of 281 real-world errors, and 20 of 30 pull requests with suggested fixes were merged.It outperformed three state-of-the-art large language models in exact matches, 54.4% versus 26.4%, and complements a technique for runtime type errors.Source: §1 · p. 2
2 BACKGROUND ON PYTHON TYPE CHECKERS
Python leaves static type checking to third-party tools, whose implementations can detect different type errors. PyTy’s implementation uses Pyre under its default configuration.
2 BACKGROUND ON PYTHON TYPE CHECKERS
- Python’s optional type annotations are not checked at runtime, and the language leaves static type checking to third-party tools.The paper describes gradual type checking as checking provided annotations while allowing parts of a program to remain unannotated.Source: §2 · p. 2
- Different type checkers implement different type systems and may therefore reveal different type errors.The approach is conceptually independent of a specific checker and could be adapted to popular checkers.Source: §2 · p. 2
- PyTy’s implementation uses Pyre and its default configuration, which runs only on functions that are at least partially type-annotated.The paper selects Pyre because it is widely used, open-source, backed by a major technology company, and used in recent annotation-practice research.Source: §2 · p. 2
3 PRELIMINARY STUDY
The preliminary study examines repair patterns, fix locations, and the usefulness of type-checker messages. Its findings motivate PyTy’s data-driven design, single-hunk focus, and use of diagnostic messages.
3 PRELIMINARY STUDY
- The preliminary study investigates repair-pattern recurrence, fix-location difficulty, and the usefulness of type-checker messages.The questions inform the choice of repair technique, how it locates edits, and whether it can benefit from diagnostic messages.Source: §3 · p. 3
3.1 Data Collection
- The study collected 125 type-error fixes from 14 GitHub repositories after reviewing commits and excluding irrelevant or warning-suppressing changes.Commit sources included help-seeking issues, GitHub keyword searches, and a dataset of popular Python repositories.Source: §3.1 · p. 3
3.2.1 PQ1: Repetitiveness of Type Errors and Fixes.
- 64.8% of fixed errors were incompatible return, variable, or parameter types; fixes showed recurring patterns but only an ambiguous mapping from error classes to patterns.Incompatible return type/Modify function return type accounted for 14.4%, and incompatible parameter type/Modify function parameter type for 13.6%.Source: §3.2.1 · p. 3, Figure 2 · p. 3
3.2.2 PQ2: Difficulty of Identifying the Fix Location.
- Most fixes changed one line, and more than half changed the exact line where the type checker reported the error.Other fix locations included function parameters, return annotations, and called functions.Source: §3.2.2 · p. 3
3.2.3 PQ3: Usefulness of Error Messages and Locations.
- 71.2% of type fixes were correctly hinted by the checker; among those hints, developers used the suggested type exactly in 71.9% of cases.Most of the remaining messages without correct hints gave no hint at all: 24 out of 36.Source: §3.2.3 · p. 4
3.3 Implications
- The study’s findings lead PyTy to use a data-driven model, focus on single-hunk edits near the reported location, and provide the error message as input.The authors reject rules and heuristics because the mapping from error types to repair patterns is ambiguous.Source: §3.3 · p. 4
4 APPROACH
PyTy repairs static Python type errors by learning from automatically isolated fixes and using the type checker to validate its suggestions. Its approach combines delta-debugging-based data gathering with cross-lingual transfer learning.
4.1 Automated Data Gathering
- Delta debugging reduces commits to hunks that fix a particular type error, rejecting unparsable changes and changes that introduce new errors.It checks that other warnings remain unchanged while the targeted error disappears.Source: §4.1 · pp. 5–6
- The data-gathering pipeline produced 2,766 entries across ten frequent type-error categories after filtering fixes for size, suppression, deletion-only changes, and location.The extracted commits contained 32,330 removed type errors from 4,515 commits in 176 repositories.Source: §4.1 · pp. 5–6
4.2.1 Base Model.
- PyTy fine-tunes TFix, a model trained on JavaScript linter repairs, to transfer learned repair knowledge to Python type errors.TFix was trained on 104,804 bug-fix samples; the paper motivates transfer by the relatively small PyTyDefects dataset.Source: §4.2.1 · p. 6
4.2 Neural Type Error Fixing
- The model receives an error class, message, location, and buggy code hunk, then outputs replacement code for that hunk.The task prefix supplies error context while the buggy code is the model input.Source: §4.2.2 · p. 6
4.2.2 Fine-Tuning for Python Type Error Fixing.
- PyTy preserves Python indentation by adding special indentation and dedentation tokens before modeling, then converts them back into code afterward.This addresses the tokenizer’s treatment of whitespace, which carries semantics in Python.Source: §4.2.3 · p. 6
4.2.4 Validating Fixes via Type Checking
- PyTy validates ranked fix candidates with the type checker, reporting a suggestion only when the target error disappears and no new errors appear.The model can supply up to 50 candidates for validation.Source: §4.2.4 · pp. 6–7
5 IMPLEMENTATION
The implementation fine-tunes a T5-based model and validates generated candidates with the type checker, with most evaluation time spent on validation.
5 IMPLEMENTATION
- 281 test errors across 174,586 lines of code took six hours and 44 minutes to process, averaging 86.2 seconds per error fix.Most of the time was spent running the type checker to validate candidate fixes.Source: §5 · p. 7
- The implementation fine-tunes the 220M-parameter t5-base model for 30 epochs with batch size 32, selecting the checkpoint with the lowest validation loss.The model converged at epoch 17.Source: §5 · p. 7
- PyTy samples up to 50 predictions for type-checker validation and presents only the first candidate that successfully removes the targeted error.Experiments used a server with 48 CPU cores, 250GB RAM, and one Tesla V100 GPU.Source: §5 · p. 7
6 EVALUATION
The evaluation examines the quality of automatically gathered fixes through independent annotation. Annotators judged 97 of 100 sampled entries correct, including 94 that were minimal under the study’s criteria.
6.1 RQ1: Effectiveness of Automatic Data Gathering
- Two authors independently labeled a random sample of 100 dataset entries as minimal, correct but not minimal, or wrong.The sample included at least one error from each class except call errors.Source: §6.1.1 · p. 7
6.1.2 Results.
- Three labels remained divergent because some hunks fixed two errors at once, making them irreducible by hunk-based reduction despite possible finer-grained reduction.The reported inter-rater agreement was Cohen’s kappa of 0.651.Source: §6.1.2 · p. 7
- 97/100 sampled PyTyDefects entries were correct and 94/100 were minimal, supporting the dataset as a basis for training and validating PyTy.The remaining three entries were correct but not minimal.Source: §6.1.2 · p. 7
6.2 RQ2: Effectiveness of PyTy
PyTy removes many evaluated type errors, though exact matches capture only part of its repair ability. Examples show that validation filters fixes that leave errors but cannot alone prevent unintended semantic changes; developer feedback provides evidence of practical usefulness.
6.2.1 Metrics.
- The evaluation measures error removal without new type errors and exact match against the developer’s fix.Exact match can miss reasonable alternatives; Table 1 reports sample counts and both rates by error kind.Source: §6.2.1 · p. 7
- 85.4% of evaluated errors were removed, and 54.4% of predictions exactly matched the developer fixes; Weakened postcondition errors were an exception requiring non-local, project-specific information.The exact-match rate underapproximates PyTy’s abilities because reasonable fixes may differ from the original fix.Source: §6.2.2 · p. 8, §6.2.1 · p. 8
6.2.2 Quantitative Results.
- PyTy exactly reproduced the developer’s fix by replacing the undefined name constrained with constraint.Figure 8 presents the exact match.Source: §6.2.3 · p. 8, Figure 8 · p. 8
6.2.3 Examples.
- For a string-to-bytes error, PyTy introduced byte_string rather than using the developer’s direct argument fix, while still matching the developer’s intent.Figure 9 shows this alternative fix.Source: §6.2.3 · p. 8, Figure 9 · p. 8
- A predicted replacement for an imported identifier failed to remove the error, so PyTy’s validation kept it from users.The available context did not identify the needed imported name.Source: §6.2.3 · p. 8, Figure 10 · p. 8
- A fix that resolved an incompatible variable type changed behavior unintentionally; PyTy’s type-checker-only validation did not catch the semantic change.The proposed fix created new_Bot but did not update the global Bot variable; the authors suggest tests as further validation.Source: §6.2.3 · pp. 8–9, Figure 11 · p. 8
6.2.4 Type Fixes in the Wild.
- 20 of 30 pull requests with PyTy-suggested fixes had been merged, while six remained open and four were closed.The closed requests reflected developers’ preferences or a false-positive warning; overall, the authors describe the feedback as confirming usefulness in practice.Source: §6.2.4 · p. 9
6.3 RQ3: Ablation Study of PyTy
The ablation results show that the full PyTy outperforms simpler variants. Removing pretraining, preprocessing, or model capacity reduces effectiveness, although considering more suggestions partly offsets the smaller model’s impact.
No pre-training.
- 37.7% to 30.2%: top-1 exact match rate drops without JavaScript pretraining.The no-pretraining model was trained on PyTyDefects for up to 100 epochs, with the lowest validation loss at epoch 32.Source: §6.3 · p. 9
Vanilla TFix.
- 1.8%: vanilla TFix achieved this exact-match rate within the top-50 suggestions without Python-specific fine-tuning.The authors cite Python–JavaScript syntax differences and the model’s lack of type-error training as reasons.Source: §6.3 · p. 9
No preprocessing.
- 50.9% to 17.8%: top-1 error removal rate falls without indentation and dedentation preprocessing.The authors caution that exact-match comparisons ignoring newline and whitespace differences may not reflect the actual developer fix.Source: §6.3 · p. 9
Small TFix model.
- 43.1% instead of 50.9%: top-1 error removal rate is lower with the 60M-parameter model; top-50 exact match rate changes from 54.4% to 53.0%.Considering more suggestions partly compensates for the smaller model’s negative impact.Source: §6.3 · p. 9
6.3 RQ3: Ablation Study of PyTy
- The full PyTy outperforms simpler variants, with each component contributing to its effectiveness.The ablation study evaluates configurations of the approach.Source: §6.3 · p. 9
6.4 RQ4: Comparison with Prior Work
PyTy is more effective than prompted general-purpose LLMs on the reported comparison, while PyTy and PyTER address different kinds of type errors and fixes. The paper compares their scope rather than reporting a direct end-to-end evaluation.
6.4.1 RQ4a: PyTy vs. Large Language Models.
- The LLM comparison used text-davinci-003, gpt-3.5-turbo, and gpt-4, with prompts containing buggy code, the type-checker message, the error line, and expected output format.PyTy is fine-tuned for type-error repair, unlike task-independent LLMs, which are typically trained on more data.Source: §6.4.1 · pp. 9–10
- 54.4% vs. 26.4%: PyTy is more effective than prompted general-purpose LLMs on the reported comparison.The authors report that PyTy exceeds all three evaluated LLMs in error removal and exact developer-fix matching; LLM successes mostly involved single-token edits, while PyTy handled more complex errors.Source: §6.4.1 · p. 10
6.4.2 RQ4b: PyTy vs. PyTER.
- 15 of 281 PyTyDefects errors fell within PyTER’s repair-template scope, while 266 were not covered by any template.PyTER’s templates cover adding an instanceof check, a type conversion, or code to catch and handle a TypeError.Source: §6.4.2 · p. 10
- Only 11 of 93 errors in PyTER’s dataset matched PyTy’s single-hunk and error-location assumptions.The type checker checked 16 of those 93 errors in annotated functions, but did not find the errors fixed by PyTER.Source: §6.4.2 · p. 10
- PyTy and PyTER are complementary: they address errors that manifest differently and use different kinds of fixes.The authors state that a direct end-to-end comparison is neither possible nor meaningful because one detects static errors and the other runtime exceptions.Source: §6.4.2 · p. 10
7 DISCUSSION AND THREATS TO VALIDITY
The discussion bounds PyTy’s evidence and operation by repository selection, static-checker behavior, annotation coverage, and single-hunk fixes. The authors identify improved localization, model comparisons, broader error coverage, and IDE integration as future work.
Python repositories.
- The study’s selected popular repositories may not represent other repositories, particularly for the preliminary study.The authors note that a different repository set could yield different results.Source: §7 · p. 10
Limitations of static type checking.
- False positives from Pyre may prompt unnecessary changes, while false negatives may leave errors unfixed or lead to new problems.Other type checkers may report different errors and provide different hints; PyTy treats the checker as a black box.Source: §7 · p. 10
Type annotations.
- PyTy cannot fix errors in completely unannotated code because the type checker reports errors only in at least partially annotated functions.The authors cite growing annotation adoption as evidence that more code may fall within PyTy’s scope.Source: §7 · p. 10
Type errors.
- PyTyDefects includes only errors fixable with a single-hunk change, so its results do not establish performance on more complex fixes.The dataset’s error-class distribution does not cover every class a type checker may find; the data-driven approach could address further classes with corresponding training data.Source: §7 · pp. 10–11
Future work.
- Future work includes improving error localization, testing different LLM prompts and models, covering more type-error classes, and integrating PyTy into an IDE.The authors also plan to fine-tune models beyond TFix.Source: §7 · p. 11
8 RELATED WORK
Related work spans learning-based repair, type-error-specific tools, type annotation research, and code-model transfer learning. PyTy targets statically detectable Python type errors and differs from approaches that rely on tests or address other repair settings.
Automated program repair
- PyTy is learning-based, uses a static type checker rather than tests to validate fixes, and learns to fix multiple error types in one model.Earlier heuristic and constraint-based APR approaches rely on test suites, while PyTy uses a text-to-text transformer rather than a GNN-based model.Source: §8 · p. 11
Type-error repair
- Rite repairs OCaml type errors with AST-based templates, while PyTER targets runtime type errors in Python and complements PyTy's static-error scope.PyTER requires tests that trigger a runtime error; the two approaches address different settings.Source: §8 · p. 11
Type annotations and type errors
- Type-prediction techniques predict correct types, whereas PyTy addresses the complementary task of fixing type-related errors.Prior studies report that adding annotations can reveal static errors developers may not have time to fix, and type-annotated repositories rarely type-check.Source: §8 · p. 11
Transfer learning on code
- PyTy builds on TFix and shows transfer across both repair tasks and programming languages, extending prior code-model transfer work.VRepair transfers from C bug fixes to C vulnerability fixes.Source: §8 · p. 11
Delta debugging
- PyTy treats type-error-fixing hunks as failure-inducing for delta debugging, adapting the technique to type errors.Source: §8 · p. 11
9 CONCLUSION
PyTy repairs statically detectable Python type errors using a dataset built with delta debugging and type checking, and cross-lingual transfer learning. It removes the targeted error in 85.4% of studied cases, with 20 of 30 pull requests based on its fixes merged.
9 CONCLUSION
- 85.4% of studied errors received a PyTy fix that removes the targeted type error.Source: §9 · p. 11
- 20 out of 30 GitHub pull requests based on PyTy-generated fixes had been merged by developers as of the paper's writing.Source: §9 · p. 11
- PyTyDefects contains 2,766 Python type errors and fixes, generated by combining delta debugging with type checking.Source: §9 · p. 11
- PyTy uses cross-lingual transfer learning, fine-tuning an existing automated repair model trained for another task and language to address limited data.Source: §9 · p. 11