Source-linked AI summary
Comparing Code Explanations Created by Students and Large Language Models
Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, Arto Hellas
TL;DR
Students often struggle to explain code, while producing example explanations for large classes is difficult to scale. This study compares student- and LLM-generated explanations and finds that students perceive the LLM versions as more accurate and easier to understand, with equivalent ideal length. The authors conclude that LLM explanations can serve as examples in early learn-by-example contexts, though learning effects remain untested.
Problem
Students struggle to develop accurate and succinct code explanations, and instructor-created examples are difficult to provide at scale in large classrooms.
Method
The study compares student- and GPT-3-generated explanations for three functions using student ratings of accuracy, understandability, and length.
Results
LLM-generated explanations were rated more accurate and easier to understand than student-generated explanations, while their perceived and actual lengths did not differ.
Takeaways & Limitations
LLM-generated explanations could provide useful examples for students practicing code reading and explanation, particularly in early learn-by-example contexts.
Takeaways & Limitations
The study evaluated student perceptions rather than whether student- or LLM-created explanations produce different learning outcomes.
Abstract
from arXiv · showhide
Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course ($n \approx 1000$) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.
1 INTRODUCTION
The study addresses the difficulty of developing accurate code explanations at scale by comparing student- and LLM-generated explanations. It finds that LLM explanations are perceived as more accurate and easier to understand while matching students’ ideal length.
- Students struggle to explain code, although this skill supports later code writing and professional communication.The paper also notes that code explanation is not always an explicit learning objective in CS courses.
- Generating example code explanations is time-consuming for instructors, while student-sourced content can raise quality concerns.
- The study compares explanations of three functions created by students and LLMs, measuring perceived quality and the characteristics students value.
- LLM-generated explanations are perceived as more accurate and easier to understand than student-generated explanations, with equivalent ideal length.
2 RELATED WORK
Prior work presents code explanations as useful learning resources, but students have limited access to examples and instructors cannot easily provide them at scale. This study therefore examines whether AI-generated explanations can address that gap and directly compares them with student-generated explanations.
- Code comprehension strategies include tracing execution, explanations, and notional machines, but they vary in effectiveness and can leave students facing logical or syntactical errors.
- Explanations help students understand code execution, reason about writing code, and reduce stress by breaking down complex concepts.
- Prior approaches, including explain-in-plain-English activities and peer explanation, provided learning benefits but did not ensure broad access to example explanations.
- Instructor-created explanations are difficult to provide personally in large classrooms, motivating study of LLMs as a scalable source of examples.
- Earlier AI-code-generation research evaluated programming performance and explored generating learning resources such as explanations and exercises.
- This study directly compares student- and AI-generated explanations using students’ evaluations, extending prior work that relied on expert assessment.
3 METHOD
The study collected student-written explanations and ratings of student- and GPT-3-generated explanations in a first-year C programming course. It compared sources across understandability, accuracy, and length, and analyzed students’ stated preferences.
- Data collection: Approximately 1000 students were enrolled in the first-year programming course where the study data were collected in 2022.
- Data collection: Students first explained three C functions involving arithmetic, types, functions, loops, and arrays, then later rated sampled explanations.
- Data collection: The functions each contained one loop processing an input array, and students inferred intended purpose from names, variables, and implemented algorithms.
- Data sampling: 963 student explanations per function were stratified by word length, yielding 27 student and 27 GPT-3 explanations for evaluation.
- Data sampling: Students rated four randomly selected explanations using five-point scales for understandability, accuracy as a code summary, and ideal length.
- Analysis: The analysis compared Likert responses between sources and used Mann–Whitney U tests with Bonferroni correction and effect sizes.
- Analysis: A thematic analysis of 100 randomly selected responses examined which aspects of code explanations students found useful.
4 RESULTS
Students rated LLM-generated code explanations as more accurate and easier to understand than student-generated explanations, while judging their ideal and actual lengths similarly. Students valued explanations that efficiently combine purpose, operation, inputs, outputs, and sufficient detail.
- Evaluation: 954 students assessed code explanations using ratings of understandability, accuracy, and ideal length.The study also compared the actual character lengths of student- and LLM-generated explanations.
- Quality differences: LLM-generated explanations were significantly easier to understand and more accurate summaries than student-generated explanations.The comparison used two-sided Mann-Whitney U tests with Bonferroni correction.
- Length: There was no statistically significant difference in perceived ideal length or actual explanation length between the two sources.The result was reported after Bonferroni correction.
- Quality differences: Approximately 60% of explanation pairs favored LLMs for understandability, compared with approximately 40% favoring student explanations.The corresponding accuracy comparison favored LLM-generated explanations in approximately 56% of pairs versus 44% for student-generated explanations.
- Explanation qualities: Students preferred explanations combining a function’s purpose with line-by-line operational details and its inputs and outputs.Students described useful explanations as connecting coding terms with basic-English descriptions of what the code achieves.
- Explanation qualities: Students characterized poor explanations as omitting code details while being either too long or too short.Requested details included data structures, function inputs, variable names, parameters, definitions, examples, templates, and the thought process behind the code.
5 DISCUSSION
The discussion argues that LLM-generated code explanations can scaffold novice code learning because they are rated more accurate and understandable without being longer. It also identifies preferred explanation formats and boundaries on generalizability, learning effects, and student reliance.
- 5.1 Differences Between Student- and LLM-Created Code Explanations: LLM-created explanations were rated better on understandability and accuracy than student-created explanations, with no perceived or actual length difference.The authors therefore suggest they could serve as examples for students learning to read code.
- 5.1 Differences Between Student- and LLM-Created Code Explanations: LLM explanations may scaffold students who can understand them but cannot yet create accurate explanations themselves.They may also provide examples that help students craft their own explanations.
- 5.1 Differences Between Student- and LLM-Created Code Explanations: Potential over-reliance on LLM support is a downside that the authors propose addressing by monitoring or limiting explanation requests.One suggested mechanism is token budgets that students spend requesting explanations and regain by writing their own.
- 5.2 What Do Students Value in Code Explanations?: Students preferred line-by-line explanations, which matched the format LLMs appeared best at producing.This contrasts with prior work that rated short, abstract relational explanations more highly and suggests instructor and student preferences may differ.
- 5.2 What Do Students Value in Code Explanations?: LLM explanations closely followed a standard format that could help students structure their own explanations.The discussion connects this possibility to prior findings that templates can improve problem framing and email writing.
- 5.3 Limitations: Novice participants limit generalizability because advanced students or instructors might produce higher-rated explanations and value shorter, more abstract explanations.The authors note that expertise could affect both explanation production and ratings.
- 5.3 Limitations: The study used only correct, relatively simple code and measured perceived explanation quality rather than differences in student learning.The authors call for work on buggy, more varied code and learning outcomes.
- 5.3 Limitations: The aggregate analysis included students who may have seen only LLM-created or only student-created explanations.A brief analysis of students exposed to both sources showed similar effects, supporting the authors’ view that aggregation was methodologically valid.
6 CONCLUSION
The conclusion reports a study comparing student- and GPT-3-created code explanations. Students rated GPT-3 explanations as more accurate and understandable despite equal length, and preferred detailed explanations.
- 6 CONCLUSION: Students rated GPT-3 explanations higher in accuracy and understandability than student-created explanations, despite no difference in perceived or actual length.The study also found that students preferred detailed explanations over concise, high-level explanations.
- 6 CONCLUSION: LLM-created explanations could help students practice reading and explaining code, particularly as AI code generators increase the importance of evaluating generated source code.The authors frame this as a supported use of LLM explanations rather than a replacement for students’ active involvement.