Source-linked AI summary
Perfection Not Required? Human-AI Partnerships in Code Translation
Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, Kartik Talamadupula
TL;DR
Generative NMT can translate code across languages, but its outputs may contain compilation or logical errors that challenge software engineers’ confidence and verification practices. This paper uses scenario-based interviews with 11 engineers to examine imperfect AI in application modernization. Engineers viewed imperfect output as useful when integrated with review and testing, while confidence highlighting helped direct attention toward potential problems.
Problem
Code translation models can produce imperfect outputs, but evidence is limited about whether software engineers will accept and use them when correctness is essential.
Method
The authors used a three-stage design scenario and interviewed 11 software engineers about NMT-assisted Java-to-Python translation in application modernization.
Results
Engineers considered imperfect generative code a useful aid when subjected to the same review and testing procedures as human-produced code, with confidence highlighting steering attention toward potential errors.
Takeaways & Limitations
Human review and testing remain part of the partnership, while interface features can help engineers understand model output and inspect questionable translations.
Takeaways & Limitations
The findings may not generalize beyond the examined code-translation task, such as to generative models used for molecular discovery.
Abstract
from arXiv · showhide
Generative models have become adept at producing artifacts such as images, videos, and prose at human-like levels of proficiency. New generative techniques, such as unsupervised neural machine translation (NMT), have recently been applied to the task of generating source code, translating it from one programming language to another. The artifacts produced in this way may contain imperfections, such as compilation or logical errors. We examine the extent to which software engineers would tolerate such imperfections and explore ways to aid the detection and correction of those errors. Using a design scenario approach, we interviewed 11 software engineers to understand their reactions to the use of an NMT model in the context of application modernization, focusing on the task of translating source code from one language to another. Our three-stage scenario sparked discussions about the utility and desirability of working with an imperfect AI system, how acceptance of that system's outputs would be established, and future opportunities for generative AI in application modernization. Our study highlights how UI features such as confidence highlighting and alternate translations help software engineers work with and better understand generative NMT models.
1 INTRODUCTION
Generative models can translate source code across programming languages, but code’s brittleness and correctness requirements make imperfect outputs consequential. This study examines how software engineers might use and accept imperfect NMT during application modernization.
- Code is brittle: changing a few characters or tokens can alter its meaning, while correctness requires compilation and freedom from bugs.
- Generative code models may deviate from programmers’ intentions and produce translations containing errors despite high-quality outputs.
- The study asks whether engineers would use imperfect generative AI, how model state could support error detection and correction, and which UX features facilitate acceptance.
- The authors investigate NMT for modernizing legacy applications, including translation from languages such as COBOL and J2EE into Python or Go.
- Interviews with 11 professional software engineers identified four themes: verification-based acceptance, human-AI interaction, imperfect-AI utility, and modernization opportunities.
- Engineers’ concerns about NMT quality were tempered by code review and testing, while confidence highlighting and alternate translations supported understanding and bug discovery.
- The paper argues that humans and AI have distinct roles in modernization and motivates further study of partnerships that may outperform either alone.
2 RELATED WORK
Prior work covers AI for software engineering, imperfect-AI trust and explanations, and human-AI co-creation. This paper extends that literature to code translation, where objective correctness creates a lower tolerance for error.
- AI research in software engineering spans code completion, classification, recommendation, naming, type inference, repair, documentation, summarization, and clone detection.
- TransCoder is an unsupervised NMT model that transforms source code from one programming language to another.
- Prior imperfect-AI studies examine controllable errors, confidence scores, and explanations that influence trust, reliance, and acceptance.
- Explanations can restore trust after observed errors but may also increase reliance when that reliance is unwarranted.
- The paper examines whether confidence information and translation alternatives affect the utility and acceptance of generative AI in code co-creation.
- Human-AI co-creation research reports favorable subjective outcomes, including self-efficacy, ownership, and preferences for AI-assisted restoration.
- This work addresses a less-studied setting with objective quality criteria, where correct code is essential and probabilistic models may produce errors.
3 DESIGN SCENARIO
The design scenario presented progressively enhanced interfaces for translating Java into Python with a real, flawed TransCoder output. Participants could inspect confidence cues and alternative translations.
- The scenario used three progressively enhanced UX variants to show how engineers might interact with NMT when translating source code.
- The interface placed source code in a left pane and NMT-generated output in a right pane.
- The base variant translated a Java function into Python using a pretrained TransCoder model.
- The first enhancement highlighted low-confidence tokens in red and exposed token-level confidence through hover tooltips.
- The second enhancement offered alternate translations for low-confidence regions and supported downstream edits when alternatives affected other code.
- The scenario used syntactically correct output containing a loop logic error, an exception-message discrepancy, and flaws discovered by several participants.
4 METHOD
The authors conducted interviews in which software engineers explored the three UX variants, discussed broader modernization uses, and reflected on imperfect AI outputs.
- The interview study used the scenario to prompt discussion of generative AI’s role in application modernization.
- Each one-hour interview presented the variants in the same A-to-C order and included background questions, think-aloud exploration, and design feedback.
- Participants were told that the AI-produced translations might be imperfect and then brainstormed modernization applications after reviewing all variants.
- Brainstorm prompts covered translating complex algorithms, translating large codebases, and converting third-party library usage across languages.
- At least three authors attended each interview, with one leading and two taking notes, probing topics, and explaining the algorithm when needed.
- The study recruited 11 full-time engineers with Java and Python experience across varied technical roles and backgrounds.
- Questions examined participants’ understanding of the model, highlighted tokens, confidence information, and the origins of alternate translations.
5 RESULTS
The study analyzed interviews with software engineers about application modernization and found strong interest in AI assistance for code translation. Its results center on legacy-code difficulties and four themes: verification-based acceptance, human-AI interaction, imperfect AI, and future modernization opportunities.
- Major themes: The analysis identified four major themes: acceptance through verification, human-AI interaction, imperfect AI’s utility, and future generative-AI opportunities.These themes organize the paper’s discussion of how engineers might work with code-translation systems.
- Modernization challenges: Participants described legacy applications as difficult to modernize because documentation, architecture, tests, and institutional knowledge were fragmented or obsolete.These difficulties included undocumented systems, dead tests, inconsistent additions, and knowledge held by former employees.
- Modernization challenges: All participants involved in modernization wanted help, and the code-translation use case resonated positively with them.The strongest interest came from participants who had direct application-modernization experience.
5.1 Acceptance Through Verification, Not Understanding
Participants generally accepted AI-generated code through review and testing rather than requiring an understanding of the model’s internal mechanics. However, confidence displays could create confusion when they conflicted with engineers’ own judgments, making explanations and alternate translations valuable.
- Acceptance through verification: Participants wanted to establish trust in translated code through review and testing, as they do with human developers’ code.One participant described review by another person as the existing basis for production-code trust and requested a similar standard for the AI system.
- Acceptance through verification: Understanding how the NMT model operates was not considered a prerequisite for using its output, although architects overseeing projects were expected to understand it.Individual contributors could focus on conversion tasks, while project-responsible architects needed operational understanding.
- Understanding AI mechanics: Model understanding could still help engineers prepare code for translation and reduce confusion about confidence highlights.Participants saw value in inspecting the model’s operation even while accepting that productive use did not require such knowledge.
- Understanding AI mechanics: Participants’ mental models often conflicted with the NMT system’s behavior, especially when syntactically correct code received low confidence.Some assumed the translator was rule-based and questioned confidence assigned to comments, punctuation, function signatures, and correct assignments.
- Understanding AI mechanics: Alternate translations acted as implicit explanations by showing why tokens were highlighted and how different code constructs could express the same functionality.Participants compared alternatives to understand the model’s choices and the underlying code.
- Feedback and learning: Participants valued feedback mechanisms, including unflagging tokens and editing translations, because review could improve both the code and the NMT model.Several participants imagined iterative learning from corrections and edits.
5.2 Human-AI Patterns of Interaction
Participants viewed confidence highlighting and alternate translations as useful supports for human-AI collaboration. These features directed attention, clarified uncertainty, exposed logical errors, and enabled humans to select and refine machine-proposed solutions.
- Human-AI workflow: Participants responded positively to confidence highlights and alternate translations as supports for human-AI workflows.They also saw human actions and edits as potential feedback for improving the NMT model.
- Human-AI workflow: Participants favored asymmetric task allocation in which the machine handled easier work and humans assisted with more difficult parts.This division was presented as an ideal interplay between human and machine.
- Confidence highlighting: Confidence highlighting helped developers prioritize review by directing attention toward code locations more likely to contain errors.Participants described the highlights as a checklist or guide for locating problems in large codebases.
- Alternate translations: Alternate translations explained confidence highlights because participants could see alternative ways of translating low-confidence tokens.Once alternatives were shown, participants better understood why particular tokens had been flagged.
- Alternate translations: Reviewing alternate translations helped participants identify a logical error in the translated code.One participant recognized that d[i] = i was wrong after comparing it with a correct alternative.
- Human-AI workflow: The interaction pattern let the model propose multiple solutions while the human selected one and triggered consistent downstream edits.Participants connected this collaborative selection process to a feedback loop for improving the NMT model.
5.3 Utility of Imperfect AI
Participants generally considered imperfect translations useful when they provided a head start over rewriting code manually. Acceptance depended on the number and nature of errors, especially whether they were easy to detect and fix without distracting from the central task.
- Error detection: Participants found concrete translation errors, including a string mismatch and a logic error.These examples show that participants could detect both textual and functional problems in the generated code.
- Utility of imperfect AI: Many participants preferred imperfect translations with small mistakes to rewriting or retyping code from scratch.They described even distorted output as useful when it reduced manual work.
- Tolerance boundaries: Acceptance depended on error severity, error count, and whether errors were easy to spot and fix.Participants tolerated a useful head start but rejected output that was egregiously wrong or repeatedly distracting.
- Tolerance boundaries: Tolerance for minor errors varied across participants and could depend on whether corrections interrupted work on the central problem.Some accepted nitpicks, while others found repeated unrelated fixes annoying.
- Utility of imperfect AI: Participants viewed imperfect translation as potentially worthwhile when it saved developers time, even if the result was incomplete.The value came from providing a starting point rather than guaranteeing a finished translation.
5.4 Beyond our Design Scenario: Opportunities for Generative AI Across the Modernization Lifecycle
Participants identified opportunities for generative AI across all three modernization phases: understanding legacy systems, performing migration, and reviewing and testing migrated code. Suggested support included documentation, architecture comprehension, translation, coding guidance, and automated tests.
- Application modernization comprises understanding legacy code and architecture, performing migration, and reviewing and testing migrated code.
- Creating Understanding: Generative methods could generate documentation and help engineers understand poorly documented code, architecture, and dead codebases.
- Supporting Activities: Participants also expected AI-generated code and documentation to follow team coding and formatting conventions.
- Performing Migration: Translation could support language, version, API, and framework migration while exposing language-specific features and alternative implementation choices.
- Reviewing and Testing: Automated unit tests and realistic test values were requested to probe translation errors and verify expected behavior.
6 DISCUSSION
Participants generally viewed imperfect AI-translated code as useful when it accelerated work and could be reviewed and corrected. Confidence highlights and alternate translations supported prioritization, error detection, and understanding, while the study points toward broader human-AI interaction designs.
- Human-AI Partnerships: Participants were eager to use imperfect AI assistance because translated code provided a useful starting framework they could adjust and correct.
- Human-AI Partnerships: Confidence highlights and alternate translations helped engineers prioritize review, understand translation intent, and identify potential logical errors.
- Limitations and Future Work: The system examined only human-initiated translation, leaving mixed-initiative and tightly coupled interaction patterns for future study.
- Acceptance Through Verification: Reviewing and testing output mattered more for acceptance than understanding the NMT model’s mechanics, although probabilistic-model knowledge reduced confusion about confidence highlights.
- Future Opportunities: Future opportunities include stronger source-target visual correspondence, review checklists, engineer feedback for training, and fine-grained translation control.
- Broader Opportunities: Generative AI may support understanding as an intermediate output, not only artifact production, including documentation and architecture specifications for legacy modernization.
7 LIMITATIONS
The study is preliminary and its findings may not generalize beyond the examined code-translation context. The authors also distinguish verification-based acceptance from the continuing importance of explanations and accurate mental models.
- The study is a preliminary examination of probabilistic generative models for a task requiring objective quality.
- The findings may not generalize to domains or use cases beyond the studied code-translation task.
- Acceptance through verification does not diminish the importance of research on explaining generative models’ operation and limitations.
- Accurate mental models helped participants understand confidence levels, and salient operational details may help people form them.
8 CONCLUSION
The conclusion presents imperfect generative code translation as a useful aid when engineers review and test its output. Confidence highlighting and alternate translations support attention, explanation, and error detection, while future work targets broader human-AI partnerships.
- Engineers viewed code containing compilation or logical errors as a useful aid, but subject to review and testing like colleague-produced code.
- Confidence highlighting directed human attention toward potentially problematic translated-code areas.
- Alternate translations provided explanatory power for why translated code was rated low confidence and helped reveal its semantic meaning and logical errors.
- Future work should examine interaction patterns, initiative, and capabilities beyond code translation to enable effective human-AI partnerships.
- The intended partnerships aim to produce better outcomes than either humans or AI could accomplish alone.