Source-linked AI summary
Using an LLM to Help With Code Understanding
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, Brad Myers
TL;DR
Developers often lack accessible, current information for understanding unfamiliar code, motivating support beyond conventional search. The paper introduces GILT, an in-IDE LLM assistant with contextualized prompt-based and prompt-less interactions, and evaluates it in a 32-participant study. GILT improved task completion compared with web search, while benefits and usage differed between students and professionals.
Problem
Developers struggle to locate scattered, inadequately documented information about unfamiliar code, APIs, and domain concepts, while LLM-generated code can be difficult to understand.
Method
The study developed and evaluated GILT, an in-IDE LLM information-support tool that uses local code context and offers preset prompt-less requests alongside open-ended prompts.
Results
GILT produced statistically significant gains in task completion rate compared with web search, but not in completion time or understanding level.
Takeaways & Limitations
In-IDE prompt-less interaction with LLMs is a promising direction, although developers’ benefits and interaction patterns differ between students and professionals.
Takeaways & Limitations
The study used only tasks with visible outputs, which may make errors easier to detect than in optimization or parallel-programming tasks.
Abstract
from arXiv · showhide
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
1 INTRODUCTION
Understanding unfamiliar code requires developers to find scattered, incomplete information about semantics, APIs, and domain concepts. GILT investigates in-IDE, context-aware LLM support with prompt-less interactions and evaluates it against web search.
- Developers spend substantial time searching for, organizing, and digesting information needed to understand unfamiliar codebases.
- Information about code semantics, APIs, and domain concepts is often scattered, inadequately documented, distributed across formats, or outdated.
- LLM code-generation tools can leave developers with code they do not understand and make debugging more difficult and frustrating.
- GILT provides on-demand, in-IDE information support using local code context and introduces prompt-less interaction to reduce prompt-writing demands.
- A user study with 32 participants compared GILT with web search for unfamiliar-code tasks while examining completion, understanding, interaction, and perceived usefulness.
- Statistically significant task-completion gains occurred with GILT over web search, but utility gains in completion time and understanding level were not found.
2 RELATED WORK
Prior work has examined developer information seeking and LLM programming assistance, but provides limited evidence about LLM tools as user-oriented information-support systems for code understanding.
- Developers need varied knowledge to learn unfamiliar code and domains, including information about APIs and software documentation.
- Research on documentation has cataloged developers’ information-seeking problems and identified knowledge types critical for learning new APIs and libraries.
- Empirical studies have evaluated LLM-generated code and explanations, their feasibility in development tools, and their use in computer-science education.
- Studies of LLM programming tools have more often focused on code generation, including telemetry and user research involving GitHub Copilot.
- Existing work does not establish the opportunities and challenges of LLM-based tools specifically as information-support tools, despite exceptions involving explanations and conversational programming assistance.
3 THE GILT PROTOTYPE TOOL
GILT is a VS Code plugin that uses selected code as LLM context and supports both preset, prompt-less explanations and open-ended conversational queries. Its design emphasizes on-demand code understanding rather than code generation.
- GILT considers user-selected code context when querying an LLM for information-support applications in VS Code.
- Selecting code and triggering AI Explanation produces an overview summary of the highlighted code.
- Preset API, Concept, and Usage actions provide explanations of API calls, domain-specific concepts, and usage examples.
- Users can submit direct questions with either the entire source file or highlighted code as context and continue through conversational follow-up queries.
- The prototype deliberately omitted code generation to focus the investigation on developers’ understanding of code.
- Prompt-less buttons use pre-generated prompts intended to reduce the difficulty of constructing effective queries, especially for novices or users unfamiliar with the APIs or domains.
- GILT generates explanations only when requested, aiming to reduce information overload and distraction.
4 HUMAN STUDY DESIGN
The study used a counterbalanced within-subjects comparison of GILT and web search on unfamiliar Python tasks in data visualization and 3D rendering. Participants modified runnable starter code toward specified visual goals while researchers collected performance, survey, and interaction data.
- Each task supplied a high-level goal, start and goal outputs, and runnable starter code that participants modified to meet the requirements.
- The tasks covered Bokeh data visualization and Open3D 3D rendering, using visible outputs so participants could compare progress with target results.
- Less common libraries were selected to reduce prior familiarity and the risk of giving the LLM-powered tool an outsized advantage.
- The Bokeh starter code had 101 LOC with 11 Bokeh API calls, while the Open3D starter code had 43 LOC with 18 Open3D API calls.
- The within-subjects design had participants use GILT and a search engine on different code-understanding tasks, enabling comparative ratings and feedback.
- Task order and assignments were counterbalanced and randomized while balancing programming and domain experience across groups.
- Researchers collected task timing, interaction logs, post-task understanding measures, post-study perception measures, and open-ended feedback.
- The study analyzed 32 participants after excluding one participant’s data from 33 total studies.
5 RQ1: EFFECTS OF GILT
GILT increased task progress relative to search engines, but did not significantly change completion time or code-understanding scores. The benefit was significant for professionals but not students.
- Measures: The study evaluated task completion time, task progress, and understanding level as three outcome measures for unfamiliar-code tasks.Understanding level was assessed with API-focused quiz questions about the starter code.
- Analysis: The regression analysis used quasi-Poisson models for count outcomes and linear regression for task completion time.Models included task experience, programming experience, and LLM knowledge as controls, with uses_GILT indicating condition.
- 0.47 more sub-tasks were completed with GILT than with search engines, controlling for experience and AI-tool familiarity.The difference was statistically significant (p < 0.01).
- Completion time and code-understanding quiz scores did not differ significantly between GILT and search conditions.The experiment’s statistical power limits the interpretation to no sufficiently detectable difference.
- GILT may improve task progress without changing task speed or code-understanding ability.
- Additional analysis: Professionals completed 0.57 more sub-tasks with GILT than with search engines, while students showed no significant gain.Both groups were more successful with the tool, but the significant effect appeared only in the professionals model.
6 RQ2: GILT USAGE
GILT usage centered on overview and context-aware prompting, but interaction patterns differed between professionals and students. Familiarity with AI tools was associated with greater use of natural-language prompting, while prompt formulation remained consequential.
- Feature usage: Participants interacted with GILT 15.34 times on average, ranging from 5 to 23 interactions.
- Feature usage: Overview was the most frequently activated feature, followed by Prompt-context and Prompt-followup.Average activations were 4.76 for Overview, 4.12 for Prompt-context, and 2.88 for Prompt-followup.
- Feature usage: The overview feature was rated extremely or very important by 53% of participants, despite being used more frequently than its reported importance suggested.This mismatch suggests Overview may often have been used as context for further prompting or to activate buttons rather than for its summary description.
- Professionals vs. Students: 86% of professionals’ successful sub-tasks originated from prompt-based interaction, compared with 62% for students.Prompt-based interaction succeeded in 71% of professionals’ cases and 58% of students’ cases, while overview and buttons succeeded in 56% and 85%, respectively.
- Professionals vs. Students: Professionals used more effective prompt keywords, whereas students sometimes omitted critical terms such as “annular wedge.”In the bokeh-3 sub-task, students instead tended to use more general or library-mismatched keywords and had difficulty revising prompts.
- Other factors associated with feature use: Greater familiarity with AI developer tools was associated with more natural-language prompting and prompt-followup, while information-processing style had no significant impact on GILT use.
7 RQ3: USER PERCEPTIONS
Participants perceived GILT as more useful and usable than web search and valued its incorporation of local code context. However, prompt formulation and occasional inaccurate responses limited its productivity benefits for some users.
- Comparison with Web Search: Participants rated GILT significantly higher than search engines for perceived usefulness and perceived ease of use.TAM scores were [27.3, 29.75] for control and [33.49, 34.2] for treatment, with p< 0.001 for both dimensions.
- Comparison with Web Search: GILT differed from search engines significantly in temporal demand and perceived performance, but not in the other TLX items.Temporal-demand averages were 4.0 for control and 2.6 for treatment; performance averages were 1.6 and 3.3, respectively.
- User feedback: Participants valued GILT’s ability to incorporate code as context, which tailored suggestions to their specific programming tasks.
- User feedback: Many participants said GILT sped information seeking by reducing the need to forage across sources.
- User feedback: Some participants struggled to formulate effective prompts, and occasional inaccurate responses reduced productivity.
- User feedback: Seventeen of 19 respondents reported advantages of GILT over ChatGPT, while participants viewed GILT, search engines, documentation, and CoPilot as potentially complementary.
8 THREATS TO VALIDITY
The study’s validity is constrained by task and library choices, design differences between GILT and search, laboratory conditions, sample composition, regression threats, and rapid technological change.
- Task and library selection: The visible-output tasks may make errors easier to detect than optimization or parallel-programming tasks, limiting task-setting generalizability.Participants nevertheless reported that the data visualization and 3D rendering tasks closely resembled real-world tasks at rates of 82% and 73%.
- Task and library selection: Different libraries or larger codebases could produce different results, despite the authors’ effort to select medium-sized, non-popular libraries.
- Study design: The in-IDE design may have contributed benefits independently of the LLM by potentially reducing context switching.Further studies are needed to separate the effects of the prototype’s interface from access to LLMs.
- Study design: The laboratory setting, time pressure, novelty effects, and small academia-skewed sample may limit generalizability to professional programmers and real-world work.The study included 32 participants.
- Statistical analysis: Regression analyses retain standard threats to statistical conclusion validity, although outliers were removed and all variance inflation factors were below 2.5.
- Technological change: Rapid technological advances in LLM tools may render these findings obsolete relatively quickly.
9 DISCUSSION AND IMPLICATIONS
The discussion identifies comprehension outsourcing and uneven benefits across users, while proposing richer contextualization and real-world studies to guide LLM-based developer tools.
- Comprehension outsourcing: Some participants outsourced code comprehension to the LLM, prompting at a high level without fully reading the starter code before making changes.The authors raise concerns about potential effects on code quality and over-reliance on LLMs.
- Need for more research in UI: Professionals benefited more than students, which the authors associate with students’ difficulty constructing or revising effective queries.Students used prompt-less interaction more often and succeeded more with interface buttons than prompts.
- Utilize more context: GILT’s code-context prompting could be extended with project, system, and personal context to provide more relevant information with less prompt engineering.
- Need further studies in real-world settings: Real-world studies should examine larger systems, broader goals, and longer time frames because the lab setting encouraged narrowly task-focused behavior.The authors describe such research as necessary for understanding deployment in general developer pipelines.
- Implications: The findings are presented as a timely first step for researchers and tool builders designing LLM-based developer assistants.
10 CONCLUSION
The paper evaluates GILT, an in-IDE LLM information-support prototype, and finds improved task completion over search while identifying differences between students and professionals and varied interaction patterns.
- Conclusion: The user study found that GILT significantly improved developers’ task completion compared with traditional search-based information seeking.
- Conclusion: Benefits differed between students and professionals, and tool interaction varied with learning styles and familiarity with other AI tools.
- Conclusion: The supplementary material includes the study protocol, tasks, data, analysis scripts, and the GILT prototype in an online replication package.