Source-linked AI summary
Self-Explanation Tutor for Active Study of CS1 Worked Examples
Arun-Balajiee Lekshmi-Narayanan, Mohammad Hassany, Kamil Akhuseyinoglu, Rully Hendrawan, Peter Brusilovsky
TL;DR
The paper addresses the difficulty of scaling timely, reliable assessment of free-text self-explanations during worked-example study. It builds ESSE, an LLM-powered tutor, and evaluates both its judgments against expert and crowd standards and its effects in a CS1 deployment. Correctness judgments agree strongly with both standards, while feedback is associated with revision and increasingly complete, conceptually richer explanations; the evidence is cautiously positive but limited by the small pilot and narrow evaluation scope.
Problem
Free-text self-explanations can make worked-example study constructive, but existing approaches lack scalable, reliable, and timely assessment with feedback for revision.
Method
The paper develops ESSE, which uses an LLM to assess line-level explanations for correctness and completeness, and evaluates it with a CS1 pilot plus expert and reliability-filtered crowd standards.
Results
Correctness judgments achieved F1 = .91 against the expert and .93 against the crowd, while students revised flagged explanations and added required concepts rather than merely more words.
Takeaways & Limitations
LLM-assisted assessment can support self-explanation tutoring, with concept-focused completeness feedback serving as an actionable target for revision and improvement.
Takeaways & Limitations
The evidence comes from a single CS1 pilot with N = 8 students, only 3 valid posttests, one expert and crowd standard, and one grader configuration.
Abstract
from arXiv · showhide
Worked examples are a important part of introductory programming, but reading their expert explanations is passive. Self explanation, students explaining the problem and its solution to themselves with subgoal level analysis, turns that study into an active task, yet it is hard to scale because assessing free-text explanations and returning timely feedback has had no easy automated solution. We investigate whether a large language model (LLM) can fill that gap. We build a self-explanation tutor for introductory programming, ESSE, in which students explain lines of worked examples and receive immediate LLM feedback on the correctness and completeness of each explanation, and we pursue two goals. First, we ask whether the LLM judges student explanations well enough to serve as the engine of the tutor; we assess its judgments against two independent human reference standards of different kinds, a single domain expert and a crowd of non-expert raters, each with its own strengths and weaknesses, characterizing both where the LLM is reliable and the systematic tendencies in how it diverges. Second, we ask whether the LLM-based tutoring benefits students; deploying it in an introductory Java course, we find that its feedback leads students to persist and revise rather than abandon a line, that their explanations grow more complete and conceptually richer across attempts, and that students show evidence of learning. These indicate that LLM-based assessment is good enough to power a self-explanation tutor, and that the tutor positively shapes how students study worked examples.
1 Introduction
The paper addresses the lack of scalable, reliable, timely assessment for free-text self-explanations in worked-example study. It presents an LLM-powered tutor and evaluates both the quality of its judgments and its effects on student explanation behavior.
- Motivation: Worked examples help novices acquire problem-solving schemas, but reading expert explanations is passive rather than constructive.Self-explanation asks students to articulate each step’s purpose and behavior in their own words.
- Motivation: Existing self-explanation implementations lack a scalable, reliable, and timely way to assess free-text responses and return feedback for revision.Correct explanations can vary greatly in wording, making single-reference semantic similarity unreliable.
- Contribution: ESSE converts worked-example study into an active writing task with layered, immediate feedback intended to sustain productive engagement.For each selected line, it reports binary correctness, continuous completeness, and counts of present and absent concepts.
- Research questions: The study asks whether an LLM can judge correctness and completeness well enough to support tutor interaction and whether students’ explanations improve across attempts.These questions respectively target assessment quality and changes in self-explanation behavior during worked-example study.
- Contributions: The paper contributes an interactive tutor, analyses LLM disagreement with expert and crowd standards, and identifies conceptual completeness as the signal tracking revision and improvement.It also argues that raw agreement alone can mislead under skewed label distributions.
2 Related Work
Prior programming-education research supports self-explanation as constructive active study, but reliable assessment of free-form explanations remains the central technical challenge. ESSE extends LLM-based judging by directly assessing line-level code explanations for correctness and completeness.
- Self-explanation and active study: Self-explanation is a constructive activity that outperforms passive review within the ICAP framework and has been associated with comprehension gains in programming.Prior systems have applied self-explanation to SQL, Python, and worked examples.
- Self-explanation and active study: Programming self-explanation studies have addressed concepts, code fragments, and preparatory materials, with reported positive results.One cited study found guided self-explanation more effective than free self-explanation.
- Assessment challenge: Reliable assessment is the main technical challenge for deploying self-explanation in computer-science courses.Earlier worked-example tools used menu-based evaluation, while later work explored semantic-similarity approaches for dialogue.
- Assessment challenge: Traditional automated response scoring compares student answers with instructor references using semantic-similarity methods, including LSA, SEMILAR, and sentence embeddings.These approaches share a structural limitation when correct answers differ substantially in wording from the reference.
- LLM-based assessment: Prompted LLMs can judge explanations directly by reasoning about the concepts they should contain rather than matching a reference answer.Recent work reports performance at least comparable to fine-tuned similarity models.
- LLM-based assessment: ESSE takes a direct route by prompting an LLM to assess each line-level code explanation’s correctness and completeness.Related systems have instead fine-tuned open-source models or generated code from explanations for behavioral comparison.
3 ESSE: An LLM-Powered Self-Explanation Tutor
ESSE replaces passive expert-explanation reading with line-by-line student writing and immediate LLM feedback. Its prompt scores correctness and completeness while layered scaffolding progressively supports revision without immediately revealing the solution.
- Tutor design: ESSE is an interactive smart-content tutor that replaces reading an expert explanation with writing one’s own explanation.Its design targets constructive articulation while keeping the cost of getting unstuck low.
- Tutor design: Students explain why each highlighted code line is used, submit an initial unscaffolded response, and may revise and resubmit before advancing.The workflow organizes worked-example study one line at a time.
- LLM assessment: A single GPT-4o mini prompt returns binary correctness, continuous completeness in [0, 1], and counts of expected concepts present and absent.An explanation is considered sufficiently complete at ≥0.5; the prompt receives the problem, example, target line, and expert explanation as context.
- Layered feedback: Feedback is delivered through four progressively unlocked levels of increasing scaffolding, with each level requiring engagement and revision.The levels include flags, peer-written samples, concept-focused feedback, and an expert answer.
- Layered feedback: Flags appear after every submission, while feedback and the expert answer remain gated until students consult earlier levels and resubmit.Advancing requires a correct explanation, whereas completeness is encouraged through the indicators.
4 Study Design and Data
The study combines a deployed CS1 pilot with an independent crowd re-rating to evaluate ESSE’s judgments and student effects. It uses mixed-effects behavioral models and prevalence-aware agreement analyses for correctness and completeness.
- Datasets: The study uses two datasets: a deployed CS1 pilot and an independent crowd re-rating of its explanations.Together they support the paper’s behavioral and assessment research questions.
- Deployed CS1 study: Eight introductory Java students worked through 30 code lines across four examples and completed pretest and isomorphic posttest assessments.The deployment was conducted as a recitation activity with all enrolled students participating under consent.
- Independent crowd study: 124 Mechanical Turk workers produced 1,696 ratings for 216 explanations using the same correctness and completeness rubric.Workers were filtered using chance-corrected agreement against the majority before aggregation.
- Analysis: Behavioral analyses use mixed-effects models with student random intercepts, while correctness agreement uses precision, recall, F1, and McNemar’s test.Completeness comparisons use point-biserial correlation and AUC, and worker reliability uses κ-vs-majority.
5 Evaluation
ESSE’s evaluation compares LLM judgments with expert and crowd standards, then examines revision behavior, explanation quality, learning gains, and student perceptions. The LLM aligned more strongly with the lenient crowd than the expert, while feedback prompted targeted revision and concept-rich improvements; learning evidence remained suggestive because few students completed the posttest.
- 5.1.1 LLM vs. the Expert: Against the expert, the LLM achieved precision .95, recall .87, F1 = .91, and accuracy 83.8% on 358 rated attempts.It produced 42 false negatives versus 16 false positives, indicating systematic under-scoring of correctness; κ = .32.
- 5.1 LLM Evaluation: The completeness score separated expert judgments significantly (r = .39, p < .001; AUC = .67) but had mean absolute error = .45.Against the crowd, it reached AUC = .76 with mean absolute error = .42, indicating moderate ordering but poor absolute calibration.
- 5.1.2 LLM vs. the Crowd: Against the crowd, the LLM achieved precision .90, recall .95, and F1 = .93 across 216 explanations.Its errors favored false positives (19) over false negatives (9), reversing the expert comparison.
- 5.2.1 Intrinsic: Change in Explanation Quality: Students averaged 1.66 attempts per explanation, revising 63% of the time after incorrect feedback versus 35% after correct feedback.Revision increased as completeness fell, from 0% for near-complete explanations to about two-thirds for the least complete.
- 5.2.1 Intrinsic: Change in Explanation Quality: Across successive attempts, completeness rose substantially (r = .84, p < .001), with conceptual content predicting completeness more strongly than word count.The mixed-effects coefficients were βconcept = .50*** and βvolume = .33***; completeness also negatively correlated with word-count change (r = −.49, p < .001).
- 5.2.2 Extrinsic: Learning Gain from Pretest to Posttest: Only 3 of 8 students completed the posttest, with normalized gains of .67, −.25, and .78, so the learning evidence is suggestive.The authors exclude the five non-completers rather than scoring them zero and caution that the pilot is too small for a stronger conclusion.
6 Discussion
The tutor’s correctness judgments align strongly with both human standards, while completeness and its conceptual content are more actionable targets for feedback and persistence. The study also argues for evaluation metrics suited to skewed labels and two-part judgments.
- LLM judgment quality: F1 = .91 against the expert and .93 against the crowd, indicating strong agreement for the LLM’s correctness judgments.Residual errors differ by standard: the LLM produces more false negatives against the stricter expert and more false positives against the more lenient crowd.
- Evaluation guidance: Raw agreement or Cohen’s κ can mislead under skewed labels, while precision, recall, AUC, and MAE separate distinct judgment properties.The paper recommends reporting both ordering and absolute calibration for continuous completeness scores, and filtering crowd raters by chance-corrected reliability.
- Actionable feedback: Completeness-driven conceptual content tracks revision and improvement better than explanation length.The authors therefore recommend foregrounding missing concepts rather than rewarding more words.
- Interface design: Faded scaffolding coexisted with productive persistence, supporting a progression from flags to samples to feedback to the answer.The authors recommend retaining this fade rather than revealing the expert answer early.
7 Limitations
The study is a small pilot with limited posttest coverage, preliminary perception data, one expert standard, and one LLM prompt configuration.
- Pilot scope: N= 8 students participated in one CS1 section, so RQ2 findings, especially learning gains, are suggestive rather than confirmatory.Only 3 of 8 students have valid posttest data, and the perception survey is preliminary and small.
- Evaluation scope: The human evaluation uses one expert plus an independent crowd standard but cannot rule out a shared blind spot.The study also evaluates only a single LLM grader under a single prompt configuration.
8 Future Work
Future work will expand ESSE from a small pilot to broader confirmatory studies and integrate its concept-level signals into learner modeling and adaptive support.
- Larger evaluation: A larger, semester-long study across multiple CS1 sections is planned to provide confirmatory evidence for learning gains.The study will be powered for the pretest–posttest analysis that the pilot could only suggest.
- Learner modeling: ESSE’s detected present, missing, and expected concepts could update the same open learner model used by other practice activities.The proposed integration is with Mastery Grids, allowing self-explanation evidence to accompany problem-solving evidence.
- Adaptive support: Concept-level signals could drive adaptive recommendations for worked examples or targeted feedback based on concepts a learner repeatedly omits.The proposed support is intended to use omissions detected across self-explanations.
9 Conclusion
The pilot provides cautiously positive evidence that LLM-assisted self-explanation can support introductory programming, while identifying completeness and evaluative blind spots as important boundaries.
- Conclusion: The LLM’s correctness judgments agree strongly with two independent human standards, while completeness remains harder to score.The standards are a single expert and a reliability-filtered crowd.
- Conclusion: Students revise when feedback flags a problem and stop when their explanations are good.Their explanations become more complete by adding required concepts rather than merely increasing word count.
- Conclusion: Together, the results provide cautiously positive evidence for LLM-assisted self-explanation in introductory programming.The conclusion explicitly keeps the tutor’s evaluative blind spots visible.