Source-linked AI summary
Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book
Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, Juho Leinonen
TL;DR
Students’ engagement with and perceptions of LLM-generated code explanations in classrooms were unclear. The paper deployed multiple explanation types in a web-development e-book and found preliminary evidence that students used them and generally considered them useful, although engagement was limited and varied across contexts.
Problem
It was unclear whether and how students would engage with LLM-generated code explanations in practice.
Method
The study generated multiple explanation types with GPT-3 and Codex, integrated them into an online web-development e-book, and collected usage and feedback data.
Results
Students viewed the explanations and generally rated them useful for learning, while engagement varied with code complexity, explanation type, and snippet length.
Takeaways & Limitations
LLM-generated explanations provide preliminary evidence of benefit for students in computer science classrooms, while their design space warrants systematic investigation.
Takeaways & Limitations
The pilot may be affected by selection bias, course language and self-directed context, and students’ existing access to explanations in the e-book.
Abstract
from arXiv · showhide
Advances in natural language processing have resulted in large language models (LLMs) that are capable of generating understandable and sensible written text. Recent versions of these models, such as OpenAI Codex and GPT-3, can generate code and code explanations. However, it is unclear whether and how students might engage with such explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. We modified the e-book to make LLM-generated code explanations accessible through buttons next to code snippets in the materials, which allowed us to track the use of the explanations as well as to ask for feedback on their utility. Three different types of explanations were available for students for each explainable code snippet; a line-by-line explanation, a list of important concepts, and a high-level summary of the code. Our preliminary results show that all varieties of explanations were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement appeared to vary by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into existing computer science classrooms.
1 INTRODUCTION
LLMs can generate code explanations at multiple levels, potentially reducing the instructor workload required to support diverse learners. This study explores that potential by deploying generated explanations in a web-development e-book.
- Code explanations can support understanding from particular lines and syntax to the higher-level purpose of code fragments.
- Generating many high-quality, appropriately targeted explanations is time-consuming and requires substantial instructor expertise.
- Students with different levels of web-development and JavaScript knowledge may prefer different explanation levels.More experienced students may prefer succinct summaries, while less familiar students may prefer line-by-line explanations addressing syntax and terminology.
- The study generated explanations for web-software code listings and deployed them to students through an online e-book.
- The work contributes an in-class deployment and evaluation of LLM-generated explanations and an in-class comparison of multiple explanation types.
2 RELATED WORK
Prior work shows that explanations, tracing, feedback, and intelligent tutoring systems can support code comprehension while reducing some instructional workload. LLMs extend this line of work by generating code explanations, but their classroom use remains an emerging area.
- Code explanations can help students connect code execution to their understanding, reduce debugging effort, and improve reasoning when writing code.
- Self-explanation activities and code tracing have been used to develop students’ understanding of code and its execution.
- Explaining code to peers and tracing execution are cognitively demanding, while instructors often lack time to provide personalized explanations to every student.
- Intelligent tutoring systems provide automated grading, hints, error-specific feedback, and execution tracing in computer science classrooms.
- LLMs are being explored for generating assignments, helping students write code, and producing code explanations to facilitate learning.
3 STUDY CONTEXT
The study took place in an English-language, fully online web-software-development course at Aalto University in Finland. Students learned server-side development through an instructor-authored e-book combining theory, practice, code, exercises, and quizzes.
- The in-situ study was conducted in Aalto University’s English-language web-software-development course in Finland.
- Students from other programs could take the course, whose workload was 5 ECTS.
- The course focuses on server-side web development, including APIs, databases, and server-rendered pages, using JavaScript and Deno.
- The fully online course had no explicit deadlines and supported flexible self-paced work through online support channels.
- The instructor-authored e-book interleaves theory and practice with code examples, programming exercises, and quizzes.
4 METHODOLOGY
The methodology investigated how students used LLM-generated explanations and which explanation characteristics they found helpful. Explanations from GPT-3 and Codex were integrated into selected e-book code snippets and evaluated through usage logs and student ratings.
- Research questions: The study asked which explanation types students used, which code snippets prompted requests, and which explanation characteristics students rated as most helpful.
- Prompt engineering and model selection: The team compared GPT-3 and Codex through iterative prompt engineering to generate natural-language explanations for course code snippets.Codex was more verbose and less helpful, often adding unwanted code, whereas GPT-3 more consistently produced high-quality explanations.
- Explanation generation: The study generated 195 explanations for 13 snippets: five versions of each of three types—line-by-line, summarization, and concepts.
- Deployment and feedback: Students accessed explanations through buttons beside code snippets and could rate their code match, prior knowledge, usefulness for learning, and personal usefulness.
- Measures: Student engagement was measured using explanation views, timestamps, and calculated view time for each code snippet.
5 RESULTS
Students viewed and rated LLM-generated explanations, with engagement varying across snippets and explanation types; ratings generally indicated code alignment and moderate usefulness, while line-by-line explanations were rated least helpful.
- 176 explanations were viewed by 58 students, who spent an average of 51.5 seconds viewing each explanation.Students viewed 3.0 explanations on average; the maximum view time was 268 seconds.
- A strong positive correlation linked explanation viewing time with the length of the corresponding code snippet.The correlation was r(11) = 0.92, p< .05.
- 42.1% of Chapter 3 viewers opened the first explanation, while Chapter 11 explanation view rates ranged from 32.3% to 39.3%.For later Chapter 3 snippets, view rates fell to 9.4%, 10.7%, and 10.3%; Chapter 11 contained more challenging code snippets.
- Students viewed line-by-line explanations most often, accounting for 58.5% of explanation views.Line-by-line explanations were viewed 103 times, compared with 39 summary and 34 concept explanations.
- Students rated explanations as matching the code, somewhat useful for learning, and somewhat useful personally.Mean ratings were 4.5 for code matching, 3.8 for learning usefulness, and 3.9 for personal usefulness.
- Line-by-line explanations were perceived as less useful than summary and concept explanations, but differences were not statistically significant.The Kruskal-Wallis test yielded p> .05, with independence partially violated and small sample sizes.
6 DISCUSSION
GPT-3 explanations were judged higher quality than Codex explanations in the authors’ comparison, with Codex more often departing from the requested content and structure.
- GPT-3 consistently generated more useful, better-structured explanations than Codex.Codex explanations tended to go off-topic, include randomly generated code snippets, and sometimes ask rhetorical questions.
6.2 Engagement with code explanations
Student engagement with explanations varied by snippet and appeared related to curiosity and code complexity, while the e-book context may have constrained overall use.
- Around half of the students who opened the e-book engaged with explanations during the study.Engagement varied across code snippets and explanation types.
- Students engaged most with the first code snippet, likely reflecting curiosity about the new functionality.Later snippets with more views were more complex.
- Students did not use the explanations as extensively as the research team expected, possibly because of the e-book’s format and contents.The e-book was self-contained, previously used for teaching, and already contained instructor-written explanations.
6.3 Code explanation usefulness and quality
Students generally found the generated explanations relevant and useful, although usefulness differed by explanation type and the evaluation did not fully examine explanation quality dimensions.
- Students rated the explanations as relevant and useful for learning, and were satisfied with how they represented the code snippets.The findings were preliminary and did not evaluate explanation length, clarity, or completeness in depth.
- Line-by-line explanations were requested most often but rated less helpful, while students appeared to prefer summary explanations.The higher number of line-by-line views may have been partly due to the user interface.
- The authors’ cursory analysis found the generated explanations generally correct, though they sometimes omitted details.The authors relate this outcome partly to prompt engineering and plan to collect more subjective and qualitative data.
6.4 Future directions
The authors identify future directions for making LLM-generated code explanations more useful, engaging, and applicable beyond online e-books. These include prompt engineering, interactive learner participation, on-demand support, and explanations for code encountered outside course materials.
- Future research: Future work should investigate how LLM-generated explanations can increase utility in online e-books and computer science education.The authors describe multiple directions rather than a single established design.
- Prompt engineering: Prompt engineering should receive further attention when applying LLMs to produce code explanations.The study used trial-and-error prompt engineering to evaluate explanations from GPT-3 and Codex.
- Learnersourcing and engagement: Students could engage more deeply by asking questions, adapting explanations, or creating their own explanations.These activities are framed as learnersourcing that may improve explanation quality while encouraging engagement.
- On-demand explanations: LLMs may be especially useful for providing on-demand explanations when students write their own code, receive debugging feedback, or lack existing explanations.Student-generated explanations and peer feedback could contribute to continuously improving learning resources.
- Explanations beyond e-books: A browser extension could let students request LLM-generated explanations for code snippets on external web pages.This would extend code explanation beyond constrained e-books and course materials to resources such as online forums.
6.5 Limitations of work
The study's findings are preliminary because participation, language, interface design, rating procedures, and sample size constrain interpretation. These limitations affect who engaged, how explanations were understood and rated, and how confidently the results can be generalized.
- Participant selection: Selection bias may limit interpretation because students who engaged with the explanations may not represent the average course student.They may have been more engaged than average, or struggling students may have been more likely to seek additional support.
- Course context: English as a second language for many Finnish students may have affected understanding of explanations and willingness to rate them.The course materials and study were conducted in English at Aalto University in Finland.
- Course context: Existing explanations in the self-directed e-book may have limited students' need for additional LLM-generated explanations.This course context may reduce the observable value of supplementary explanations.
- Data collection: The feedback form's default Likert value of 3 was logged when students closed the form, creating uncertainty about whether some ratings were intentional.Some submissions were created very quickly after opening an explanation and may not reflect deliberate ratings.
- Data analysis: 61 ratings with the default value selected for both responses were removed, potentially excluding intentional ratings and affecting results.The removal was intended to address ambiguity in the feedback data.
- Study scope: The study used a relatively small number of students in an online classroom, so the reported results remain preliminary.The authors call for further research in this area.
7 CONCLUSION
The study reports preliminary classroom experiences using LLMs to create code explanations. Students generally rated the explanations as useful, but systematic research is still needed to investigate explanation design, especially where students lack strong prior understanding of the code.
- Contribution: The study reports classroom experiences using LLMs to create code explanations.The explanations were used in a web software development course setting.
- Findings: Students tended to rate the explanations as useful for learning, although not all students used them.The conclusion characterizes this as preliminary evidence rather than a definitive evaluation.
- Future work: LLM-generated explanations might be more helpful when students do not already understand the code well.The authors identify this as a future-work direction.
- Future work: More research is needed to systematically investigate the design space of LLM-generated explanations in computer science classrooms.The conclusion frames the study as preliminary evidence of potential benefit.