Source-linked AI summary
Flawed but Memorable: Student Critical Reception of Interest-Personalized GenAI Analogies in Computing Education
Seth Bernstein, Naaz Sibia
TL;DR
GenAI analogies may obscure structural mismatches and assumptions about learners, yet students’ critical reception of them remains underexamined. This study examines students’ evaluations of personalized analogies and generic explanations, finding greater engagement with personalized analogies but mixed trust and expertise-linked scrutiny.
Problem
Students’ critical reception of GenAI analogies remains underexamined, despite risks of structural mismatch and assumptions about whose experiences count as familiar.
Source: §1 · p. 1Method
Ten students who completed CS2 evaluated linked-list and recursion explanations through think-aloud annotation and semi-structured interviews guided by Paul-Elder standards.
Source: §1 · p. 2, §3.1 · p. 4Results
Most participants found personalized analogies more engaging, trust varied, and scrutiny tracked domain expertise more consistently than personal interest.
Source: §5 · pp. 8–9, §6 · p. 10Takeaways & Limitations
The findings support tools that verify structural fit, place personalization under student control, and treat analogy critique as a learning activity.
Source: §6 · p. 10Because personalized explanations were analogies and generic explanations were technical, engagement differences cannot be attributed to personalization alone.
Source: §5.5 · p. 10
Abstract
from arXiv · showhide
Motivation: Undergraduate computing students increasingly turn to generative AI (GenAI) tools to understand abstract concepts through analogies. Analogies compare an unfamiliar concept to something familiar, but judging whether the comparison holds requires knowledge of both. GenAI may also embed assumptions about who the learner is. GenAI education research centers on output correctness, leaving students' critical reception of analogies largely unexamined. Method: We investigate how students evaluate the accuracy, appropriateness, and assumptions in GenAI-generated analogies, and their perceptions of interest-personalized versus generic technical explanations. Ten students with CS2 experience participated in a pre-survey, a think-aloud task with linked-list and recursion explanations, and a semi-structured interview grounded in the Paul-Elder framework. They judged accuracy, clarity, engagement, and trust separately. Results: Most participants described interest-personalized analogies as more engaging or memorable than generic technical explanations, while trust was mixed. Some trusted the tailored analogies more; others scrutinized them more closely or distrusted the tailoring. Participants with deep source-domain knowledge identified structural flaws requiring that knowledge to recognize. Because personalization and explanation format differed together, these findings do not isolate an effect of personalization alone. Implications: A familiar source flips the student's role. On the concept they are still learners, but on the familiar source they are the expert, and that is the position from which an analogy can be judged. We call this two-sided analogy auditing. GenAI systems should ask what students know, not just what interests them, and treat a flawed analogy as something to inspect and fix rather than accept.
1 Introduction
GenAI analogies can make abstract computing concepts familiar, but their structural and social assumptions may be difficult for students to evaluate. This paper examines how students judge these explanations and introduces two-sided analogy auditing as a way to understand that evaluation.
1 Introduction
- Analogies can misrepresent computing structures or assume a learner’s experiences, creating risks for students who cannot check the mapping or do not identify with its implied audience.A traversal analogy may still misrepresent insertion or deletion.Source: §1 · p. 1
- GenAI research often emphasizes technical accuracy, while how students evaluate analogy accuracy, appropriateness, and embedded assumptions remains underexamined.The comparison between interest-personalized analogies and generic technical explanations also remains open.Source: §1 · pp. 1–2
- The study uses think-aloud annotations and interviews with undergraduate students who completed CS2, guided by Paul-Elder standards including accuracy, clarity, assumptions, and point of view.Interest-personalized means the participant’s self-reported interest was supplied to the model as the requested analogy source domain.Source: §1 · p. 2
- The paper frames analogy evaluation as two-sided: students use computing knowledge to assess the target and source-domain knowledge to check whether the mapping holds.The paper names this joint evaluation process two-sided analogy auditing.Source: §1 · p. 2
2 Related Work
Prior work describes structural risks and uneven effects of analogies in computing, while GenAI explanation research considers quality and engagement. The paper situates its study at the intersection of critical evaluation and interest-based personalization.
2.1 Analogies in Computing Education
- Computing analogies can mislead when their structural relations fail to match the target, and their benefits vary by learner, topic, and transfer context.The “variable as box” metaphor can imply that a variable holds multiple values simultaneously; brief analogy gains in one study were not sustained on later exams.Source: §2.1 · pp. 2–3
2.2 GenAI-Generated Explanations
- GenAI explanations have shown uneven accuracy and context-sensitive performance, while AI-generated stories and analogies can be more motivating and memorable than expository text.In chemistry, over 80% of students reported that AI-generated narratives improved conceptual clarity.Source: §2.2 · p. 3
2.3 Critical Evaluation and Personalization
- Critical evaluation research foregrounds assumptions and point of view, and prior work finds that models may affirm weak responses rather than elicit deeper student reasoning.The paper applies this concern to whether an analogy’s framing signals whose world it assumes and who it positions as a competent learner.Source: §2.3 · p. 3
- Interest personalization has been studied in computing practice materials but not conceptual analogies, leaving open how it affects evaluation of source-domain fit and target correspondence.Prior work also raises the possibility that personally relevant material may reduce critical scrutiny.Source: §2.3 · p. 3
3 Method
Ten students who had completed CS2 compared interest-based analogies with generic explanations for linked lists and recursion. The study combined pre-survey information, think-aloud annotations, interviews, and reflexive thematic analysis.
3.1 Study Design
- Ten CS2-complete students evaluated personalized analogies and generic explanations for linked lists and recursion, with concept order fixed and explanation order counterbalanced within each concept.The generic explanations were identical across participants, while personalized analogies drew on individual interests.Source: §3.1 · p. 4
3.2 Participants
- Researchers recruited ten volunteers from 667 students who had completed CS2 at a large public research university in North America.Individual Zoom sessions lasted approximately 60 to 90 minutes.Source: §3.2 · p. 4
3.3 Pre-Survey
- The pre-survey collected coursework, interests, prior analogy and GenAI experience, and computing belonging; researchers used participants’ specific interest responses to generate analogies.Belonging was measured with three items adapted from established scales.Source: §3.3 · p. 4
3.4 Analogy Generation
- Researchers generated personalized explanations using each participant’s interest as a source domain and prompts specifying concept learning goals, while generic explanations used the same structure without a theme.Stimuli included stronger and weaker structural mappings; researchers checked clarity and goal coverage but did not require perfect structural fidelity.Source: §3.4 · pp. 4–5
3.5 Think-Aloud Task
- Participants annotated and edited each explanation alongside code while thinking aloud, with personalized-versus-generic presentation order counterbalanced within each concept.Researchers preserved the annotation artifacts for analysis.Source: §3.5 · p. 5
3.6 Interview
- Semi-structured interviews moved from participants’ general explanation experiences to retrospective probes on accuracy, clarity, assumptions, point of view, and comparison of explanation types.The probes drew on Paul-Elder standards.Source: §3.6 · p. 5
3.7 Analysis
- Researchers independently developed interpretive readings, then collaboratively refined themes against transcripts and annotations; the Paul-Elder framework informed interpretation, not coding.Patterns described as common refer to spread within this ten-person sample, not prevalence in a wider population.Source: §3.7 · pp. 5–6
4 Results
Participants evaluated GenAI analogies as arguments to inspect, drawing on knowledge of both computing concepts and source domains. Personalized analogies could be engaging, but reception and trust varied with fit, completeness, and learners’ knowledge.
4 Results
- Participants used analogies for initial big-picture understanding, then examined their accuracy, assumed audience, and personalization rather than treating them as simply true or false.They drew on knowledge of both the computing concept and the source domain, a pattern termed two-sided analogy auditing.Source: §4 · p. 6
4.1 Evaluation Criteria (RQ1)
- Students with enough domain knowledge identified structural mismatches, questioned vague or under-explained wording, and tested AI claims against course materials or worked examples.They checked whether source-domain mechanics preserved the target structure, and used follow-up questions to probe for inconsistency.Source: §4.1 · p. 7
4.2 Affective and Social Dimensions (RQ2)
- Students flagged explanations that assumed unfamiliar coursework or cultural references; excessive explanation could feel condescending, while some students attributed confusion to their own effort.Participants described length and repetition as signals that the tool underestimated their competence.Source: §4.2 · pp. 7–8
4.3 Personalization and Reception (RQ3)
- Personalized analogies sometimes felt memorable and intuitive, but poor fit or lost completeness prompted rejection; trust varied, and participants favored learner control over imposed interests.Accounts linked scrutiny more consistently to domain expertise than to personal interest, without establishing a causal effect.Source: §4.3 · p. 8
5 Discussion
Students judged explanations across structural fit, assumed knowledge, and engagement, with trust varying across participants. The discussion frames source- and target-domain knowledge as central to analogy auditing and outlines design, instructional, and research implications alongside study limitations.
5 Discussion
- Students judged analogies by structural correspondence, evaluated assumptions about whose knowledge was expected, and generally found personalized analogies more engaging while trust varied.Scrutiny aligned more consistently with domain expertise than with personal interest.Source: §5 · pp. 8–9
5.1 Two-Sided Analogy Auditing
- Two-sided analogy auditing requires target knowledge to identify relations an analogy must preserve and source knowledge to check whether those relations hold.The discussion proposes that a familiar source may make an analogy both more engaging and more inspectable, but the design cannot isolate personalization as the cause of scrutiny.Source: §5.1 · p. 9
- A single explanation-quality score can obscure cases where students value an analogy’s memorability but doubt its accuracy, or accept its accuracy but reject its tone.Participants assessed the concept, source domain, and correspondence as separate layers.Source: §5.1 · p. 9
5.2 Implications for Design and Instruction
- The authors recommend personalizing through source-domain structure, checking structural fit before delivery, and letting students choose whether to view generic or personalized explanations.They also propose using flawed analogies as objects for students to inspect and repair, making critique a check on understanding.Source: §5.2 · p. 9
5.3 Motivation or Knowledge
- The discussion locates personalization’s possible role in the knowledge learners bring to evaluation, distinguishing source-domain expertise from computing expertise and motivation.This reframes earlier accounts of analogy performance and scrutiny without treating expertise as uniformly helpful or harmful.Source: §5.3 · pp. 9–10
5.4 Future Work
- Future work should test whether critiquing flawed analogies predicts independent learning outcomes, whether novices audit differently, and whether structural checks improve reception.The proposed classroom study would distinguish failures in the target concept, source domain, and correspondence.Source: §5.4 · p. 10
5.5 Limitations
- The study’s findings are bounded by confounded explanation format and personalization, curated stimuli, fixed concept order, ten participants, and a CS2-complete sample with a narrow interest profile.Several patterns rested on one or two accounts, and the authors caution that the findings may not transfer to true beginners or students whose interests fall outside the sampled reference space.Source: §5.5 · p. 10
6 Conclusion
The conclusion reports how ten undergraduate computing students evaluated personalized analogies and generic technical explanations for linked lists and recursion. Students identified structural fit and assumptions about the imagined reader; most found personalized analogies engaging, while trust varied and scrutiny aligned more consistently with domain knowledge than personal interest.
6 Conclusion
- Ten undergraduate computing students evaluating linked-list and recursion explanations found personalized analogies engaging, while trust varied and scrutiny aligned more with domain knowledge than personal interest.The authors support verifying structural fit, giving students control over personalization, and using analogy critique as a learning activity.Source: §6 · p. 10