Better Together? An Evaluation of AI-Supported Code Translation

1 INTRODUCTION

• The study addresses uncertainty about whether imperfect AI provides a productivity boost and how its use affects work practices.The introduction notes that state-of-the-art code models do not always produce fully correct output and that human effort is needed to improve generated code.Source: §1 · pp. 1–2
• The study compares engineers’ Java-to-Python translation work alone with work supported by model translations, varying the translations’ quality and quantity.It examines professional software engineers using a state-of-the-art model to translate an object-oriented data structure.Source: §1 · p. 2
• The paper provides quantitative evidence that imperfect generative code models can improve software engineers’ code-translation quality across varying output quality and quantity.The authors also identify a shift from code production to review and call for interfaces that help people work effectively with generative code models.Source: §1 · p. 2
• AI support transforms code translation from production into review, with secondary benefits beyond producing code.The paper identifies a need for intelligent interfaces to help people work effectively with generative code models and achieve superior outcomes.Source: §1 · p. 2

2 RELATED WORK

• Few human-centered AI studies quantify effects on human performance or outcome quality; this paper evaluates source-code translation quality and completeness.The review also notes that AI support does not guarantee superior outcomes, motivating work on interfaces that help people work effectively with generative AI.Source: §2 · pp. 2–3

2.2 Human-centered Examinations of AI-supported Tasks

• Human-centered examinations span decision making and artifact creation, with prior work reporting both improved and unchanged or worse outcomes.The review frames whether AI augments task performance or outcome quality as an open question.Source: §2 · p. 3, §2.2 · p. 3

2.1 Enabling Technologies for AI-supported Code Work

• Code models support translation, documentation, completion, duplication detection, and test generation, while evaluations tend to emphasize output accuracy or correctness over developers’ effectiveness.The review situates these capabilities in modern NLP techniques applied to source code.Source: §2.1 · p. 3

2.2.1 AI-supported Decision Making.

• Studies of AI-supported decision making report improved decision quality or speed in some cases, but unchanged or worse outcomes in others.The review also describes difficulty calibrating reliance on AI recommendations and cases where explanation methods were ineffective for people with limited domain expertise.Source: §2.2 · p. 3, §2.2.1 · p. 4

2.2.2 AI-supported Artifact Creation.

• In artifact creation, AI has supported faster or more numerous outputs in some tasks, while other studies found no quality difference or lower creativity.The review notes that participants could value the process without showing improved artifact quality.Source: §2.2.2 · pp. 4–5

2.2.3 AI-supported Code Work.

• Findings for AI-supported code work are mixed: some tools improved task completion or speed, while other studies found no significant improvement or inconclusive quantitative results.Prior work also reports faster documentation while preserving the quality achieved by participants working alone.Source: §2.2.3 · p. 5

3 AI-SUPPORTED CODE TRANSLATION

• TransCoder produces correct translations that pass unit tests 30%-70% of the time, depending on the source and target languages.The section notes that additional human effort is needed to identify and fix errors in AI translations.Source: §3 · p. 5
• The section asks whether multiple translation options are desirable, noting that reviewing 25 or 100 alternatives is not feasible, especially without unit tests.It raises the question of whether engineers should instead focus on a single translation.Source: §3 · p. 6
• The section asks whether engineers translate code more effectively with AI-produced translations and how translation quality and quantity affect their work.It also asks whether engineers can improve imperfect translations.Source: §3 · p. 6
• The study also asks how AI translations affect the translation process, whether they make the task easier, and how engineers perceive their benefits and drawbacks.These questions cover process impact and participant perceptions of working with AI-produced translations.Source: §3 · p. 6

4.1 Code Translation Tasks

• Participants translated object-oriented Trie and Priority Queue data structures from Java to Python; more complete translations were observed when participants worked with AI support.Source: §4.1 · pp. 6–7

4.2 Participants

• The researchers recruited 32 of 182 screened technologists who met criteria for Java, Python, VSCode, and recent data-structure experience.Eligibility required more than a year of familiarity with both languages, recent Java and Python coding, VSCode familiarity, and no recent implementation of the study structures.Source: §4.2 · p. 7

4.3 Procedure

• The remote, two-hour sessions counterbalanced task order and data structure; each participant completed two 30-minute tasks, one without AI and one with one or five translations of better or worse quality.AI quantity and quality assignments were randomized and counterbalanced; participants received the source Java and an empty Python file, plus AI translations in the supported condition.Source: Figure 1 · p. 8, §4.3 · p. 8

4.4 Measures

• The study collected code-quality measures and post-task and post-study survey data to assess AI efficacy, work process, and participants’ views of its helpfulness.Participants completed a survey after each translation task and another at the end of the study.Source: §4.4 · p. 9

4.4.1 Code quality measures.

• To assess translation quality, the researchers built an error taxonomy from Ko and Myers [50] and analyzed errors across AI and participant code artifacts.They corrected 20 AI translations and 64 participant translations using unit tests and corrected files before identifying errors.Source: §4.4.1 · p. 9
• Translation quality was measured by error rate (specified error types per SLOC) and secondary PCM, the proportion of methods passing unit tests without modification.Researchers identified errors by comparing original and corrected files; two authors manually labeled code-diff errors and resolved disagreements.Source: §4.4.1 · pp. 9–10
• PCM ranged from 11% to 67% and error rates from .16 to .60; worse-quality translations had higher error rates than better-quality translations (.47 vs .27).The error-rate difference was statistically significant, F(1, 18) = 34.0, p < .001, η^2_p = 0.65.Source: §4.4.1 · pp. 10–11, Table 3 · p. 11

4.4.2 Survey measures.

• After each task, participants completed NASA TLX ratings for demand, effort, performance, and frustration, described their work process, and rated and discussed the AI translations.The AI-condition questions asked how the provided translations affected their work.Source: §4.4.2 · p. 11
• After both tasks, participants compared their experiences with and without AI support and provided demographic and work-background information.This included job role, workplace AI or machine-learning use, and prior code-translation experience.Source: §4.4.2 · p. 11

5.1.1 Outliers.

• The two participants who struggled to translate without AI made progress with AI and improved their translations, so their incomplete No-AI sessions were excluded from statistical analyses.Both cited insufficient Python familiarity; with AI, one reviewed and fixed translation errors, while the other used tests to find and fix them incrementally.Source: §5.1.1 · p. 12

5.2.1 Effect of AI support on code quality.

• With AI support, error rate (M (SD)) was .31 (.16) versus .63 (.42) without AI, while PCM (M (SD)) was 50.9% (16.0%) versus 34.1% (29.3%).These differences corresponded to a 50.8% reduction in error rate and a 62.1% increase in PCM.Source: §5.2.1 · pp. 12–13, Table 4 · p. 13
• Among participants working with AI translations, better-quality translations had lower error rates than worse-quality translations (.22 (.04) vs. .40 (.16); F(1,25)=15.8, p=.001, ηp²=.39); quantity had no error-rate effect, and neither factor significantly affected PCM.Source: §5.2.1 · p. 13

5.2.2 Individual-level improvements to AI translation quality.

• Participants’ error-rate improvements over their own AI translations were statistically significant overall, but PCM showed no net improvement.The AI translations’ overall error rates ranged from .16 to .60, and participants could not be compared against a single pooled AI baseline because each received a different set.Source: §5.2.2 · pp. 13–14
• Among 32 participants, 17 (53%) improved their AI translations’ quality by reducing error rates by an average of .11 (SD = .10); 6 (19%) fell within their AI translations’ error-rate range, and 9 (28%) did worse.Source: §5.2.2 · pp. 14–15

5.2.3 Effect of AI support on kinds of errors.

• Across 64 participant translations, the researchers identified 1,709 errors, and error distributions differed between work with and without AI support.The analysis categorized errors as translation, language, spurious, code omission, documentation omission, or correctness errors.Source: Table 5 · p. 15, §5.2.3 · p. 15
• Participants made 946 errors without AI support versus 763 with it, consistent with significantly lower error rates with AI (.31 vs. .63).Source: §5.2.3 · p. 15
• Without AI, a smaller proportion of errors were translation errors (M = 11.8% vs. 36.8% with AI) and language errors were more common (M = 35.0% vs. 23.7%).Spurious code was also less common without AI support (M = 2.5% vs. 8.4% with AI).Source: §5.2.3 · pp. 15–16
• Without AI, participants made more omission errors; even with AI, omissions occurred when participants copied selectively across translations and failed to carry over code or documentation.The AI translations were near-complete, but participants’ grazing copy-paste strategy could leave out some of their content.Source: §5.2.3 · p. 16
• Comparing errors in participants’ improvements with those in AI translations, distributions differed overall (χ2(5, N = 901) = 11.3, p = .04), yet no specific error type was reduced.Documentation omission errors were more common in participants’ improvements than in AI translations (3.9% vs. 1.7%); improvements were otherwise evenly distributed across error types.Source: §5.2.3 · p. 16

5.3.1 Effect of AI support on the code translation experience.

• Participants rated AI-supported work as easier, faster, and better in quality, although they reported similar task demand, effort, and frustration with and without AI.TLX ratings showed no differences between participants given worse- versus better-quality translations; the reported AI-effect ratings were positive and significantly different from zero.Source: §5.3.1 · pp. 16–17, Table 7 · p. 17
• Participants provided 5 translations reported significantly higher frustration (M (SD) = 9.06 (4.57) of 20 vs. 4.94 (3.26) with one) and mental demand (14.1 (3.59) of 20 vs. 10.4 (4.63) of 20) than those provided one translation.Source: §5.3.1 · p. 16

5.3.2 Effect of AI support on work practices.

• Participants spent substantial time reviewing and reasoning about AI-produced code, either choosing a whole translation as a starting point or selectively combining code from several.AI support shifted the task from greenfield translation toward code review: participants checked, tested, and corrected an already-provided starting point.Source: §5.3.2 · p. 18
• Participants said AI support could reduce their attention to the Java source, and one questioned whether this aided their code understanding.One participant said they referred to the AI-generated code significantly more than the source when translations were available.Source: §5.3.2 · p. 18

5.4.1 Benefits & utility of AI translations.

• 50% rated the translations useful a great deal and 37.5% somewhat; 53.1% said they helped a great deal with task completion and 28.1% somewhat.Participants also rated the translations as having errors a great deal (21.9%) or somewhat (43.75%).Source: §5.4.1 · p. 19
• Roughly 69% felt the translations taught them something new about Python, including syntax and APIs.The translations also provided a starting structure and completed boilerplate, shifting some work toward reviewing code rather than writing it.Source: §5.4.1 · p. 19

5.4.2 Drawbacks & hindrances of AI translations.

• Participants described bugs and unreadable code as frustrating, with subtle errors difficult and time-consuming to find.Some reported that incorrect code made them doubt their own work or the model, while others criticized formatting and non-idiomatic Python choices.Source: §5.4.2 · p. 20
• Participants given five translations reported higher NASA TLX frustration and mental demand, and described stress comparing options and managing multiple code windows.One participant suggested a GUI for comparing code patterns across translations.Source: §5.4.2 · p. 20

6.1 Imperfect, but Useful

• With AI-produced translations, participants implemented a greater proportion of methods correctly and made fewer errors per SLOC than when working alone.Participants also shifted from producing code to reviewing it and found value in the imperfect translations.Source: §6.1 · p. 20

6.1.1 Generative models as performance amplifiers.

• Baseline AI translations had error rate M (SD) = .37 (.12) and PCM M (SD) = 47.1% (16.9%), versus solo results of .63 (.42) and 31.4% (29.3%), respectively.About half of participants could not effectively leverage the AI head start, while the other half improved its quality. For two participants, AI support enabled completion of a task they could not complete alone and had given up on.Source: §6.1.1 · p. 21

6.1.2 Generative models as coaches, colleagues, or teachers.

• 69% of participants said AI translations taught them something new about Python, and their presence shifted the task from production to review.Participants described the model in human-role terms such as coach, teacher, or colleague, suggesting its value may include helping users learn as well as producing code.Source: §6.1.2 · p. 21

6.1.3 Generative models as scaffolds.

• For two strong engineers whose Python skills were insufficient to translate alone, AI translations scaffolded the task by filling knowledge gaps and helping them improve the output.The support let them focus on identifying problems in the translation instead of spending time refreshing their Python knowledge.Source: §6.1.3 · p. 22

6.2 Limitations.

• The study examined code translation in its simplest setup, with no interactive features or way to query the model, and cautions that self-estimated programming experience may not indicate skill in a given language.AI-produced translation error rates ranged from .16 to .60, and were generally superior to human-only translations.Source: §6.2 · p. 22

6.3 Future Opportunities

• The study points to transparency and usability as opportunities for generative-code interfaces to help users work more effectively and establish trust.The authors also provide a reproducibility package to support further research in AI-supported code translation.Source: §6.3 · p. 23

6.3.1 Intelligent presentation of multiple alternatives.

• Multiple translation alternatives were associated with increased frustration and mental demand, and comparing similar outputs may be too effortful for users.Some participants preferred constructing a translation by combining code from alternatives rather than selecting one; the discussion describes a single-translation interface that uses highlighting to indicate differences.Source: §6.3.1 · p. 23
• The authors call for tools that help users browse, compare, and manage many model outputs, while noting that their effects on user experience and effectiveness need evaluation.They argue that some level of intelligent in-IDE support is needed.Source: §6.3.1 · p. 23

6.3.2 Building appropriate trust with a generative model.

• Participants examined translations to understand their relation to the Java source and assess correctness, and the authors argue that experience and further research can help calibrate appropriate trust.One participant said they would feel more comfortable working with AI-translated code after the study experience.Source: §6.3.2 · p. 23

6.3.3 Explainable generative AI.

• Participants’ need to understand model operations and outputs motivates formal study of explainability for generative models, including source-to-translation correspondences as a possible aid.The authors also call for further study of the questions people ask about generative models.Source: §6.3.3 · pp. 23–24

6.4 Societal Implications

• AI for Code may democratize high-quality software translation by reducing the skills and education required, but automating one task component could lead to full automation and societal harms.The authors frame this as a potential benefit alongside a risk of radical automation.Source: §6.4 · p. 24
• The authors adopt a human-centered framing and argue that researchers and designers should build and evaluate tools for effective work with imperfect model outputs, which people must review, test, and approve.They do not ask the ML community simply to improve model quality.Source: §6.4 · p. 24

• Participants produced higher-quality code with AI translations, and 59% made improvements to their output despite errors in the AI translations.Source: §7 · p. 24
• Two participants who struggled to translate on their own were highly successful with AI support.Source: §7 · p. 24
• Working with AI translations shifted the task from writing to reviewing code, without reducing task demands or effort.Source: §7 · p. 24
• AI translations provided a starting point that participants felt saved time and effort, though time spent finding and fixing errors may have offset those perceived savings.Source: §7 · p. 24
• The authors describe AI translations as potential scaffolds for people with weaker skills and as a source of programming knowledge or insights.Source: §7 · p. 24
• The study motivates further work on interfaces that help people understand and evaluate model outputs and support outcomes better than either party working alone.Source: §7 · p. 24

• The Java Trie implementation comprised 170 lines, including 93 SLOC, and two classes: TrieNode and Trie.Source: §A.1 · p. 29
• TrieNode was a helper class for storing Trie data, with four one-line methods.Source: §A.1 · p. 29

Trie

• Java’s overloaded delete() and enumerate() methods required a design choice because Python does not allow duplicate method names.Options included using default arguments for recursion or giving each recursive helper a different name.Source: Trie · p. 29
• Across Trie translations, the number of methods ranged from 10 to 12, depending on whether delete() and enumerate() used one method or a primary method plus a recursive helper.The count mattered when computing the proportion of correctly implemented methods.Source: Trie · p. 29

A.2 Priority Queue

• The Priority Queue implementation had nine methods covering queue construction, insertion and removal, peeking, size and emptiness checks, enumeration, and heap restoration.Source: §A.2 · p. 29

• TransCoder could not translate whole classes, so the authors translated Java classes method by method and manually cleaned up signatures and instance-data access.Source: B PRODUCING CODE TRANSLATIONS WITH… · p. 29
• To vary translation quality, the authors introduced a beam-offset parameter that altered which tokens beam search considered.Source: B PRODUCING CODE TRANSLATIONS WITH… · p. 29
• Default top-k token selection produced the better-quality translations; an offset of 10 discarded those most-likely tokens in favor of less-likely ones for the worse-quality set.Source: B PRODUCING CODE TRANSLATIONS WITH… · p. 29

B.1 Spurious Code Examples

• The Trie example invokes unnecessary os-package methods with no equivalent use in the Java source.Source: §B.1 · p. 30
• The examples span a worse-quality Priority Queue translation and a better-quality Trie translation, with both containing irrelevant or nonsensical code.The examples include references to undefined or task-irrelevant classes and methods.Source: §B.1 · pp. 29–30, Figure 3 · p. 30
• The Priority Queue example references an undefined Value class and constructs string representations absent from the Java source.It uses "".join(...) to represent list elements.Source: §B.1 · p. 30
• Other spurious output included unnecessary imports and return statements, extra call arguments, duplicate initializers, and calls to undefined methods.Source: §B.1 · p. 30

C.1 Code

• The authors published Java data structures, AI translations, and error-analysis codebooks to support reproduction and invite community extensions.The codebooks document how programming errors were classified.Source: §C.1 · p. 30

C.2 Participant Instructions

• The participant-instruction scripts mark experimenter notes and condition-dependent wording for the study introduction and tasks.Source: §C.2 · p. 31

C.2.1 Initial Instructions.

• Participants completed two Java-to-Python data-structure tasks, one with AI support and one without, in randomized order.Each task allowed 30 minutes, and participants were asked to think aloud.Source: §C.2.1 · p. 31

C.2.2 Task Instructions: AI Support.

• For the AI-supported task, participants received one or five pre-generated translations and were told the output could contain errors.They could decide how to use the translations.Source: §C.2.2 · p. 32

C.2.3 Task Instructions: No AI Support.

• The no-AI task asked participants to translate the Java data structure into Python without AI support.Source: §C.2.3 · p. 32

C.3 Surveys

• The package includes surveys administered before the study, after each task, and after the second task.Source: §C.3.1 · p. 32, §C.3.2 · p. 32

C.3.1 Pre-Screen Survey.

• The pre-screen questions assessed participants’ Java and Python experience, data-structure familiarity, and familiarity with code-editing tools.Source: §C.3.1 · p. 32

C.3.2 Post-Task Survey.

• Participants completed the post-task survey immediately after each translation task; it took approximately 5–10 minutes.Source: §C.3.2 · p. 32

C.3.3 Post-Study Survey.

• The post-study survey followed the second post-task survey and included questions about participants’ overall experience.Source: §C.3.3 · p. 32, Table 10 · p. 35

Effort (NASA TLX [40])

• NASA TLX items asked about mental demand, pace, success, effort, and negative feelings such as stress or irritation.Source: Effort (NASA TLX [40]) · p. 34

C.3.3 Post-Study Survey.

• Participants described their overall task approach and time allocation, and rated the provided translations’ quality and perceived usefulness.Source: Effort (NASA TLX [40]) · p. 34
• AI-support questions covered perceived errors, task help, Python learning, speed, difficulty, work quality, and whether more translations were wanted.Source: Effort (NASA TLX [40]) · p. 34
• Participants were also asked what they found useful or least useful and what extra information could make translations more helpful.Source: Effort (NASA TLX [40]) · p. 34