Source-linked AI summary
Quality Estimation & Interpretability for Code Translation
Mayank Agarwal, Kartik Talamadupula, Stephanie Houde, Fernando Martinez, Michael Muller, John Richards, Steven Ross, Justin D. Weisz
TL;DR
Automated code translation lacks established ways to estimate translation quality and interpret model choices. This paper studies TransCoder confidence alongside lint errors, drawing motivation from software professionals’ feedback. The authors report no significant correlation between token probabilities and lint violations.
Problem
Automated code translation lacks established methods for estimating translation quality and interpreting model choices, while users may not understand when or why the model is confident about a token.
Source: §1 · p. 1, §2.1.2 · p. 2Method
The authors study TransCoder translations by relating model confidence values to lint errors, using user-study insights to motivate lint as a familiar indicator.
Source: §1 · p. 1, §2.1.3 · p. 2Results
Takeaways & Limitations
The analysis underscored the need to incorporate coding standards into the translation process.
Source: §5 · p. 4The study examines 400 Java-to-Python translations produced by TransCoder.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Recently, the automated translation of source code from one programming language to another by using automatic approaches inspired by Neural Machine Translation (NMT) methods for natural languages has come under study. However, such approaches suffer from the same problem as previous NMT approaches on natural languages, viz. the lack of an ability to estimate and evaluate the quality of the translations; and consequently ascribe some measure of interpretability to the model's choices. In this paper, we attempt to estimate the quality of source code translations built on top of the TransCoder model. We consider the code translation task as an analog of machine translation (MT) for natural languages, with some added caveats. We present our main motivation from a user study built around code translation; and present a technique that correlates the confidences generated by that model to lint errors in the translated code. We conclude with some observations on these correlations, and some ideas for future work.
1. Introduction
The introduction frames automated quality estimation and interpretability as open needs in neural translation, then extends that concern to code translation. The paper studies TransCoder confidence against lint errors, motivated by software professionals’ user-study feedback.
1. Introduction
- Automated quality estimation became a pressing need as neural machine-translation systems improved translation fluency.Source: §1 · p. 1
- Prior work treated neural machine translation as a glassbox and proposed unsupervised quality-estimation measures correlated to varying degrees with human judgments.Source: §1 · p. 1
- Code translation resembles natural-language machine translation but has stricter grammar rules and a stricter notion of correctness.Source: §1 · p. 1
- The paper studies TransCoder confidence values in relation to lint errors, motivated by a user study of software professionals.The authors aim to support further work on code-translation quality metrics and interpretations familiar to software professionals.Source: §1 · p. 1
2. User Study
The user study gathered software engineers’ reactions to Java-to-Python translation and its confidence displays. Participants emphasized target-language conventions and found token-level confidence difficult to interpret; the paper uses lint errors as a familiar frame for studying that gap.
2. User Study
- The study interviewed 11 software-engineering professionals, the paper’s likely target users, to motivate evaluating TransCoder translations.Source: §2 · p. 1
2.1.1. SYNTAX & STYLING
- Participants said successful translation must meet target-language conventions, not merely produce executable code.They connected shared style conventions with easier code reading.Source: §2.1.1 · p. 2
2.1.2. CONFIDENCE OF THE MODEL
- Participants were confused when confidence appeared low for lines they considered syntactically correct, and questioned uncertainty on punctuation and comments.Source: §2.1.2 · p. 2
- The authors describe a gap between human users’ understanding of translations and their understanding of when or why the model is confident about tokens.Source: §2.1.2 · p. 2
2.1.3. MAPPING TO INTERMEDIATE REPRESENTATIONS
- The study motivates using familiar lint errors as an intermediate representation to relate model confidence to translation issues.The authors contrast human use of larger, more global context with the model’s next-token probabilities over a restricted neighborhood.Source: §2.1.3 · p. 2
3. Experiments
The experiments translate complete Java programs with TransCoder, inspect the generated Python with Pylint, and compare line-level lint flags with model uncertainty. They use point-biserial correlation to quantify that relationship.
3. Experiments
- The experiments translate complete programs to examine how imports, class definitions, and multiple functions affect translation quality.Source: §3 · p. 2
3.1. Method
- The analysis relates continuous TransCoder token probabilities to categorical Pylint error categories flagged for translated lines.Source: §3.1 · p. 3
3.1.1. DATA
- The dataset comprises 400 Java algorithm implementations translated to Python 3 with a pretrained TransCoder model using beam size 5.The authors record output probabilities for each translated token.Source: §3.1.1 · p. 3
3.1.2. LINT ERRORS
- Pylint checks the 400 translations for programming errors, coding-standard violations, and refactoring suggestions.The study validates all but three of the 311 violations in Pylint’s default configuration.Source: §3.1.2 · p. 3
3.2. Calculating Correlation
- Point-biserial correlation compares binary line-level lint flags with continuous line uncertainty, measured using joint token-distribution uncertainty or minimum token confidence.Table 1 reports PBCC values and sample sizes for Pylint error, refactor, warning, and convention messages.Source: §3.2 · p. 3, Table 1 · p. 3
4. Results & Discussion
The results profile lint violations in TransCoder’s Java-to-Python3 output, examine whether model confidence tracks those violations, and illustrate the relationship in a specific translation. Across the quantitative analysis, confidence did not correlate with lint violations.
4. Results & Discussion
- The results comprise a lint-error profile, a quantitative confidence-correlation analysis, and a qualitative translation example.Source: §4 · p. 3
4.1. Translation Error Analysis
- Among 66 observed violations, undefined-variable occurred in 67% of translations, unused-variable in 39%, redefined-builtin in 32%, and line-too-long in about 20%; invalid-name was most frequent.Figure 1 displays violations occurring in more than 5% of translations.Source: Figure 1 · p. 3, §4.1 · p. 4
- Function-level TransCoder translation achieved around 60% accuracy with beam size 5, while translating whole classes requires accounting for auxiliary code and interfunction dependencies.Source: §4.1 · p. 4
4.2. Quantitative Analysis
- The quantitative analysis found no correlation between TransCoder token confidence and tokens resulting in lint violations.The qualitative example also shows both low-confidence violations and violations with high confidence.Source: §4.2 · p. 4, §4.3 · p. 4
4.3. Qualitative Example
- In the example, TransCoder conflated Java’s Math.max operator and max variable, while confidence was low for some violations but high for others.It translated i>=0 in Java to -1 in the Python3 loop condition.Source: §4.3 · p. 4
5. Conclusion & Future Work
The conclusion reports that syntax and styling matter to users alongside executability and correctness, while the TransCoder analysis found no significant correlation between token probabilities and lint violations. The authors are working on metrics better aligned with programmers’ perceptions of interpretability.
5. Conclusion & Future Work
- The authors found no significant correlation between token probabilities and lint violations used as a surrogate for code interpretability.They are working on metrics that correlate better with programmers’ perceptions.Source: §5 · p. 4
A.1. User Study
The appendix presents a design prototype used to demonstrate the translation model to user-study participants.
A.1. User Study
- The prototype presents Java source code on the left and TransCoder’s Python source output on the right.Source: Figure 3 · p. 6
- Tokens with model confidence below 95% are highlighted in red, with a tooltip displaying the confidence level.Source: Figure 3 · p. 6
- The user-study participants were shown this interface to demonstrate operation of the NMT model.Source: Figure 3 · p. 6
1. Pylint URL: http://pylint.pycqa.org/en/latest/
The passage provides a URL for the full list of Pylint validations.
1. Pylint URL: http://pylint.pycqa.org/en/latest/
- The full list of Pylint validations is available at pylint-messages.wikidot.com/all-codes.Source: §1 · p. 6
3. Human readable messages for Pylint codes can be found at https://git.io/JTIma
This appendix specifies three Pylint checks excluded from evaluation and illustrates TransCoder’s Java-to-Python output alongside per-line lint errors and model confidence.
3. Human readable messages for Pylint codes can be found at https://git.io/JTIma
- C0111, C0326, and R0201 are ignored because translation omits comments, detokenization inserts token spaces, and algorithmic-problem data do not require self.The corresponding reasons are missing docstrings, bad-whitespace, and no-self-use.Source: §3 · p. 6
A.3. Qualitative Example Source and Target Code
- Figure 4 pairs Java source code on the left with TransCoder’s translated Python target code.The text points to this as the qualitative example.Source: Figure 4 · p. 7, §A.3 · p. 7
- Figure 5 shows little correlation between per-line model confidence and Pylint violations; TransCoder also uses max as both a variable and the maximum operator.The appendix identifies Figure 5 as an enlarged version of Figure 2.Source: Figure 5 · p. 7, §A.3 · p. 7