Source-linked AI summary
Investigating Explainability of Generative AI for Code through Scenario-based Design
Jiao Sun, Q. Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, Kartik Talamadupula, Justin D. Weisz
TL;DR
Generative AI systems that produce code artifacts raise explainability questions that have received less attention than those for discriminative models. This paper uses scenario-based and question-driven workshops with software engineers to elicit needs across three code-generation use cases, identifying explainability categories and proposing feature types and design ideas. It concludes that human-centered methods can guide XAI development for GenAI for code, while noting limitations in current technical support and model readiness.
Problem
The paper asks what users need to understand about generative AI systems that produce code, addressing limited explainability research for generative models.
Method
The study combines scenario-based design, question-driven XAI elicitation, and participatory workshops involving software engineers across three GenAI-for-code use cases.
Results
The study identifies 11 categories of explainability needs and proposes four XAI feature types for GenAI for code, supplemented by participant design ideas.
Takeaways & Limitations
Human-centered approaches can inform the technical development of explainability for generative AI in novel application domains.
Takeaways & Limitations
Generative code models remain prone to errors and require human post-generation improvement, raising questions about the technology’s readiness.
Abstract
from arXiv · showhide
What does it mean for a generative AI model to be explainable? The emergent discipline of explainable AI (XAI) has made great strides in helping people understand discriminative models. Less attention has been paid to generative models that produce artifacts, rather than decisions, as output. Meanwhile, generative AI (GenAI) technologies are maturing and being applied to application domains such as software engineering. Using scenario-based design and question-driven XAI design approaches, we explore users' explainability needs for GenAI in three software engineering use cases: natural language to code, code translation, and code auto-completion. We conducted 9 workshops with 43 software engineers in which real examples from state-of-the-art generative AI models were used to elicit users' explainability needs. Drawing from prior work, we also propose 4 types of XAI features for GenAI for code and gathered additional design ideas from participants. Our work explores explainability needs for GenAI for code and demonstrates how human-centered approaches can drive the technical development of XAI in novel domains.
1 INTRODUCTION
This paper addresses the limited understanding of what users need to know about generative AI systems that produce code artifacts. It studies these needs through human-centered methods and proposes XAI features and design recommendations for GenAI for code.
- Motivation: Explainability for GenAI concerns helping users understand the system well enough to achieve their goals, including capabilities, limitations, provenance, and model behavior.Prior HCXAI work argues that explanations should extend beyond model internals to support actionable understanding.
- Motivation: Users’ explainability needs emerge from usage context, shaped by their goals, backgrounds, expectations, and social, organizational, and cultural contexts.The paper therefore emphasizes involving target users through methods that examine realistic use.
- Approach: The study examines natural language to code, code translation, and code auto-completion through scenario-based design and question-driven XAI elicitation.The researchers conducted 9 workshops with 43 software engineers and gathered explainability needs and design ideas around these scenarios.
- Contributions: The authors identify 11 categories of explainability needs for GenAI for code and provide definitions, examples, and contrasts with prior XAI techniques.These categories include needs that may be distinctive to generative code systems and their use cases.
- Contributions: The paper proposes four XAI feature types for GenAI for code: AI documentation, model-uncertainty indications, attention visualizations, and social transparency.Participant responses were used to develop concrete design recommendations for operationalizing these features.
- Contributions: Its methodological contribution combines scenario-based design, participatory workshops, and a question-driven approach to study explainability in a novel domain.The authors also reflect on the method’s values and limitations for future GenAI research.
2 RELATED WORK
Prior research has developed GenAI-for-code systems and XAI techniques, but explainability for generative artifacts remains less defined and less studied than explainability for discriminative models. Human-centered approaches, including scenario-based design, provide ways to investigate users’ needs in realistic future-use contexts.
- Generative AI for Code: GenAI-for-code research includes code translation, documentation generation, auto-completion, unit-test generation, duplicate-code detection, and multi-use-case models.TransCoder applies neural machine translation to code, while Codex powers GitHub Copilot for code completion.
- Generative AI for Code: GenAI-for-code outputs still have quality limitations; TransCoder produces correct translations 30%-70% of the time depending on source and target language.The passage also reports that software engineers may nevertheless tolerate such models in their work.
- Explainable AI: XAI research has largely focused on discriminative models through interpretable models and posthoc explanations such as global and local explanations.These approaches commonly explain model logic or particular outputs.
- Explainable AI: Compared with discriminative models, substantially less attention has been devoted to XAI techniques for generative models and their learned representations.Existing work includes disentanglement, interpretability measures, and interactive modification of latent representations.
- Explainable AI: Generative-model explainability also includes visualization and interaction approaches that let users modify inputs or guide output generation while observing feedback.These approaches help users make sense of model behavior through interaction rather than only inspecting internals.
- Human-Centered AI: The paper adopts a human-centered definition of explainability as enabling understanding of AI for users’ goals, including information about performance, limitations, training data, and development.This framing supports more actionable understanding than focusing only on model internals.
- Human-Centered AI: Scenario-based design addresses the challenge that AI systems may not yet exist for users to experience by using narratives of future use and task accomplishment.The broader human-centered AI tradition aims to develop systems that serve stakeholder needs and align with their values.
3 METHODOLOGY: SCENARIO-BASED DESIGN WORKSHOPS
The study used scenario-based, question-driven workshops to investigate explainability needs for generative AI in code translation, code autocompletion, and natural language to code. Participants examined realistic model outputs and co-designed ideas for four XAI feature types.
- Study design: 9 remote workshops with 3–6 participants each covered code translation, code autocompletion, and natural language to code.The workshops lasted 60–70 minutes and used Mural for virtual collaboration.
- Use cases and scenarios: Scenarios used state-of-the-art models, including TransCoder for code translation and Copilot for the other use cases.The translation scenario used an integer-to-Roman-numeral problem containing a subtle TransCoder bug identified during scenario introduction.
- Workshop activities: Each workshop combined open-ended question elicitation with ideation around four XAI feature types.The question-driven exercise asked participants what Alex would want to know, followed by discussion of AI documentation, uncertainty, attention visualization, and social transparency.
- XAI feature ideation: The study adapted existing discriminative-AI XAI approaches into UI probes to elicit questions and design ideas for generative code models.The probes addressed AI documentation, uncertainty indicators, attention visualization, and social transparency.
- Participants: 43 software engineers participated after recruitment and screening for relevant programming experience.Selection required at least one year of Python experience for two use cases and both Python and Java experience for code translation.
4 EXPLANABILTY NEEDS FOR GENAI FOR CODE
Across the three generative-code use cases, participants expressed 11 categories of explainability needs, including four categories not prominent in prior decision-support work. Input, output, global how, performance, and how-to questions were the most frequent.
- Input: Input questions were the most prominent, comprising about 16% of all questions.Participants asked what programming languages, data types, algorithms, and language versions the model could process, and how to optimize inputs.
- Output: Output questions addressed generated code characteristics, output scope, quality assessment, shortcomings, errors, and alternative candidates.Participants also wanted to know whether the system could generate test cases or multiple alternatives.
- Performance: Performance questions concerned overall performance, the quality of specific generated code, and run-time efficiency.Examples included inference time and support for multi-threading.
- How to and Control: How-to questions focused on changing or improving inputs to obtain better outputs, while control questions concerned customizing model behavior.Control was absent from Liao et al.’s decision-support taxonomy, indicating interest in specifying how the generative model should work.
- Additional needs: Participants also asked why specific outputs occurred, where training data came from, what system requirements and impacts applied, what limitations existed, and what would happen under hypothetical input changes.These questions covered local explanations, data provenance, usage conditions, model boundaries, and counterfactual behavior.
- Overview: 11 categories of explainability needs emerged from the workshops, with four categories not prominent in Liao et al.’s prior work.The four were Input, Control, System Requirements & Impact, and Limitations.
5 XAI FEATURES FOR GENAI FOR CODE
Participants wanted explainability features that clarify both model behavior and generated code, extending beyond uncertainty indicators to interactive, actionable support. Their suggestions included documentation, alternative outputs, uncertainty explanations, attention visualizations, and human feedback mechanisms.
- AI documentation: Participants identified code-focused documentation categories, including examples, software-engineering capabilities, code quality and utility, supported languages and frameworks, and deployment requirements.These categories extend prior AI documentation approaches toward generated artifacts and software-engineering contexts.
- Model uncertainty: Participants wanted alternative outputs and explanations of why particular code regions were uncertain, including the model’s competing options, goals, or rationale.They also wanted reminders of the natural-language prompt from which code was derived.
- Model uncertainty: Participants proposed human-initiated interactions in which the AI asks for confirmation, clarification, inspection, preferences, or immediate modifications to resolve uncertainty.Interactive testing of uncertain regions was another requested way to support decisions about retaining or changing generated code.
- Model uncertainty: Line-level uncertainty indicators could guide attention to potentially low-quality code, but participants considered uncertainty alone insufficient for understanding model behavior.They requested interactive testing, alternative outputs, and explanations to build a more appropriate mental model.
- Attention distribution: Attention visualizations could show how prior content contributed to selected generated spans, helping users modify inputs and obtain better results.Participants suggested line-level or syntax-tree selections, natural-language explanations, and immediate edits after identifying suspicious regions.
6 DISCUSSION
The discussion argues that explainability for GenAI for code must be grounded in generated artifacts, software-engineering workflows, and users’ practical goals. The study also identifies technical gaps, user differences, and limits to treating explainability as sufficient for adoption.
- Informing XAI approaches for GenAI for code: Participants’ needs centered on input, output, and global explanations, indicating a mismatch with technical attention to representation learning and representation visualization.They sought understanding of supported languages, frameworks, data structures, and generated-code characteristics.
- Informing XAI approaches for GenAI for code: Current techniques do not cover all identified categories, including artifact characteristics, runtime efficiency, programming-language-grounded prompt effects, and incorporating human feedback into training.The authors describe these areas as under-explored or varying in technical feasibility.
- Design implications for GenAI for code: Explainability needs were actionable and utility-oriented, supporting strategies for improving generated outputs and programming productivity through follow-up actions.The relevant questions included Input, Output, How, and How-to XAI questions.
- Design implications for GenAI for code: Software-engineering context shaped requested information about system requirements, artifact impacts, team practices, and different stages of the development lifecycle.The discussion of social transparency extended explainability beyond a single decision point toward software-development stages.
- Design implications for GenAI for code: Participants responded positively to natural-language explanations and interactions, suggesting conversational interfaces may fit code assistants and co-programming tools.This observation is specifically reported for programming- or natural-language-based inputs.
- Design implications for GenAI for code: Less experienced participants asked fewer questions, so future work should examine whether novices need more proactive explanations or dedicated training interactions.The paper frames this as an observed difference and a future research direction, not as a demonstrated intervention effect.
- Design implications for GenAI for code: Generative code models remain prone to errors and require human post-generation improvement, so explainability alone may not resolve questions about technology readiness and appropriate use.The paper calls for evaluating intended uses and avoiding unvalidated or harmful use cases.
- Study methodology: Scenario-based design, realistic personas and prototypes, real model outputs, and participatory question clustering helped elicit explainability needs for a novel technology.The authors recommend using examples that reflect both current capabilities and limitations.
7 CONCLUSION
The paper studies explainability needs for generative AI for code through participatory, human-centered research. It identifies user needs and proposes XAI feature areas to inform future software-engineering AI work.
- The study is among the first to examine users’ explainability needs for generative AI for code.
- 9 participatory workshops with 43 software engineers covered natural language to code, code translation, and code auto-completion.
- The authors identified 11 explainability-need categories and proposed four XAI feature areas with design recommendations.
A THE OTHER TWO USE CASES
The supplementary use-case materials adapt the natural-language-to-code workshop example for code translation and code auto-completion while retaining the other workshop elements with minimal edits.
- Natural language to code: Figure 1 provides the base code example for the natural-language-to-code workshops.
- Code translation and code auto-completion: Figure 3 replaces Figure 1(c) with other code examples for code translation and code auto-completion.
- Cross-use-case materials: The remaining Figure 1 sub-figures stay almost the same across use cases with minimum edits.