PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z. Henley, Paul Denny, Michelle Craig, Tovi Grossman

arXiv:2401.11314v2cs.HCcs.AI

TL;DR

Programming courses need timely help, while LLM coding assistants can provide direct solutions that raise educational concerns. The paper presents CodeAid, deployed in a 700-student programming class, and reports correctness and helpfulness findings from analyzed responses.

  • Problem

    Programming courses need on-the-spot assistance, while LLM coding assistants’ ability to generate direct solutions raises educational concerns.

    Source: §1 · pp. 1–2

  • Method

    The authors iteratively designed CodeAid to assist without directly revealing solutions, then deployed it in a 700-student class and analyzed usage, surveys, and interviews.

    Source: §1 · p. 2, §10 · p. 17

  • Results

    79% of 1,749 sampled responses were correct, and 86% of correct instances were rated helpful; both measures increased after CodeAid was updated.

    Source: §6.2.1 · p. 11

  • Takeaways & Limitations

    The paper’s design implications favor scaffolded assistance without direct solutions and ways for students to refine and test AI responses.

    Source: §8.4 · p. 16, §10 · p. 17

    Evidence comes from a second-year C course at one university and may not generalize directly to other courses or contexts.

    Source: §9 · p. 16

Abstract

from arXiv · show

Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI's unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.

1 INTRODUCTION

Growing programming-course enrollments strain support resources, while traditional help can be underused and inequitable. CodeAid offers real-time programming assistance without revealing code solutions, and its classroom deployment informs design considerations for educational AI assistants.

Motivation

  • Growing participation in programming across disciplines is increasing institutional resource pressure, while scheduled office hours may be underused and in-person help inequitable.Initiatives to broaden computing participation include pairing computing with disciplines with traditionally greater gender diversity; the resulting pressure concerns administrators and educators.Source: §1 · pp. 1–2
  • LLMs can generate direct solutions to introductory programming problems, raising academic-integrity and over-reliance concerns and motivating guardrails for constructive use, [12], [14].The passage also notes broader educational concerns and that some institutions have banned access to tools like ChatGPT.Source: §1 · p. 2

CodeAid

  • CodeAid was designed to help students and educators with technically correct programming assistance without revealing code solutions, through five question and code-support features.Students could ask general programming questions, ask about or request explanations of supplied code, get help fixing it, or get help writing code.Source: §1 · p. 2

Deployment

  • A 12-week deployment in an introductory C and Systems Programming course with about 700 students combined over 8,000 interactions with surveys and student and educator interviews.The study included ten weekly surveys, 22 structured student interviews, a final survey comparing CodeAid with other AI coding tools, and interviews with eight educators from six countries; thematic analysis covered 2,100 randomly sampled usages.Source: §1 · p. 2

Contribution

  • The paper synthesizes iterative development and deployment insights into four design considerations, highlighting trade-offs and offering generalizable guidelines for pedagogical LLM-powered coding assistants.Its research questions address student usage patterns, response effectiveness, student perspectives, and educators’ perspectives on integration, improvement, and pedagogy.Source: §1 · p. 2

2 RELATED WORK

Related work frames LLMs as promising educational tools and a source of debate, then situates CodeAid within research on computing education, question answering, code explanation, and writing and debugging support.

2.1 LLMs in Computer Science Education

  • Instructors differ between restricting and integrating AI, while prior work studies LLM programming performance [54] and exercise generation [53]; CodeAid helps with exercises without direct solutions.Earlier work found Codex outperformed most students on programming exam questions, GPT-4 passed programming-course assessments [54], and Codex could generate programming exercises [53].Source: §2.1 · pp. 2–3

2.2 Question Answering

  • CodeHelp [37] offers free-form, on-demand programming help, while CodeAid adds input templates and interactive responses and evaluates response quality with a larger student cohort and instructors.CodeHelp’s semester-long study involved 52 students, who mostly valued help with specific code-related tasks such as fixing errors; CodeAid’s deployment included 700 students.Source: §2.2 · p. 3

2.3 Explaining Code

  • LLM explanations can help students, but engagement varies with code length, complexity, and explanation type [39]; CodeAid adds code-linked questions and interactive line-by-line explanations.Expert explanations can help students build code-reasoning skills, but producing many varied explanations burdens instructors [39]; plain-English explanations can yield short- and long-term learning benefits.Source: §2.3 · p. 3

2.4 Writing and Debugging Code

  • Prior LLM debugging work supports clearer error messages and syntax fixes [34] [49], while CodeAid annotates erroneous code and restricts direct solutions to scaffold independent writing and debugging.Coding Steps users often copied exercise prompts and used generated code unchanged, suggesting over-dependency [27] [10]; CodeAid instead offers interactive pseudo-code and annotations.Source: §2.4 · p. 3

3 INITIAL SYSTEM DESIGN AND ARCHITECTURE

CodeAid’s initial design used five assistance features to support coding and conceptual understanding without directly showing code solutions. Its interface, prompts, and architecture were designed to deliver and display tailored LLM responses to students.

3.1 Primary CodeAid Assistance Features

  • CodeAid offered five assistance features: conceptual questions, code-based exploration, questions about code, debugging, code explanation, and code-writing guidance without displaying solutions.The features used distinct response formats, including natural-language explanations, line-by-line annotations, bullet-point fix suggestions, and pseudo-code with sub-goals.Source: §3 · p. 3, §3.1.1 · p. 4, §3.1.2 · p. 4

3.1.1 General Question.

  • General Question generated short, explanatory answers to conceptual C questions in natural language, sometimes with inline code but no multi-line code.Selecting it disabled the code editor so the model considered only the question input.Source: §3.1.1 · p. 4

3.1.2 Inline Code Exploration.

  • Inline Code Exploration let students hover over C programming keywords to generate sample code, view documentation, or ask a question about the keyword.Source: §3.1.2 · p. 4

3.1.3 Question from Code.

  • Question from Code let students ask debugging or conceptual questions in the context of supplied code, with both the code editor and question input enabled.Source: §3.1.3 · p. 4

3.1.4 Help Fix Code.

  • Help Fix Code used buggy code and intended behavior to generate corrected code and explanations, but displayed only bullet-point changes to avoid revealing the solution.Source: §3.1.4 · p. 4

3.1.5 Explain Code.

  • Explain Code presented students’ code with hoverable, line-specific explanations of each line and its interaction with the rest of the code.Few-shot prompting formatted explanations as comments appended to matching lines.Source: §3.1.5 · p. 4

3.1.6 Help Write Code.

  • Help Write Code turned intended behavior into a high-level structure with sub-goals and natural-language pseudo-code, including relevant C library functions but no code.Source: §3.1.6 · pp. 4–5

3.2 System Architecture

  • The TypeScript client-server system used React, Node.js, and MongoDB to authenticate users, store responses and feedback, and stream parsed OpenAI output to the interface.The interface selectively enabled a text box or C syntax-highlighting editor based on the chosen feature.Source: §3.2 · p. 5

4 SEMESTER-LONG CLASS DEPLOYMENT

CodeAid was offered as an optional resource in a 700-student programming course, with voluntary, confidential research participation and multiple data sources used to study students’ interactions with the assistant.

4 SEMESTER-LONG CLASS DEPLOYMENT

  • CodeAid was available throughout a semester to more than 700 students in a large North American university programming course, with participation in the research voluntary and ethics-approved.Students could use CodeAid as an additional resource; engagement, surveys, and interviews were confidential from course instructors.Source: §4 · p. 6, §4.2 · p. 6

4.1 Course Structure

  • The second-year C-focused course combined weekly preparation, ten labs, four programming assignments, a midterm, and a final, while permitting CodeAid but forbidding other AI tools for coursework.Students also had lecture materials, a moderated Q&A board, and office hours; survey completion could earn up to 1% of the course mark.Source: §4.1 · p. 6

4.2 Deployment and Participants

  • Of the 700-student class, 563 (80%) consented to share their data; most were Computer Science majors (74%), and 62% disagreed or strongly disagreed that they were competent in C before the course.Students selected whether to share research data, complete surveys without sharing data, or opt out; research participation was confidential from instructors.Source: §4.2 · pp. 6–7

4.3 Data Sources

  • The study combined CodeAid interaction logs, ten weekly surveys, an anonymous post-course survey, and semi-structured interviews with 22 randomly sampled students.Logs captured student questions and assistant responses, while students rated each response’s usefulness on a five-point scale and could explain their rating.Source: §4.3 · p. 7, §4.3.1 · p. 7

4.3.1 CodeAid Usage and In-situ Feedback.

  • Weekly surveys asked students to compare CodeAid use with course resources and report why they used or avoided it, its usefulness, and feedback about the tool.The surveys also asked about use of lecture videos, notes, discussion boards, and office hours.Source: §4.3.2 · p. 7

4.3.2 Weekly Research Surveys.

  • Interviews with 22 randomly selected students explored productivity, workflow, reliability checks, usability, learning, and comparisons with other resources, then invited students to co-design future features.Eight interviews took place mid-course and 14 after the course; each lasted about an hour, and participants received a $25 gift card.Source: §4.3.3 · p. 7

4.3.4 Post-Course Anonymous Survey.

  • An anonymous post-course survey explored whether and why students used ChatGPT despite the course policy, and compared their engagement with ChatGPT and CodeAid.It was administered after final grade submissions to capture students’ perceptions of AI assistants in large classes.Source: §4.3.4 · p. 7

4.4 Thematic Analysis

  • The thematic analysis used 1,750 usages sampled from 8,132 logged interactions after exclusions for non-consent, irrelevant or excessively long queries, and technical errors.Two researchers inductively developed and refined codes for user queries and CodeAid responses, then independently coded the remaining 1,430 sampled usages after assessing reliability.Source: §4.4 · pp. 7–8

5 MIDTERM FEEDBACK AND SYSTEM ITERATION

Midterm feedback identified both valued support and gaps in CodeAid, prompting updates to make responses more informative, responsive, and steerable while withholding complete code solutions. The iteration added pseudo-code, function documentation, streaming, refined prompts, and redesigned code-fix and follow-up features.

5 MIDTERM FEEDBACK AND SYSTEM ITERATION

  • Midterm feedback showed students valued clear explanations, error help, availability, personalized interaction, and question flexibility, but wanted deeper answers, more reliable fixes, faster responses, line-specific guidance, follow-ups, and documentation.Feedback came from eight student interviews and five weekly surveys conducted midway through the course.Source: §5 · p. 8

5.1 System Updates and Enhancements

  • CodeAid added pseudo-code with line-by-line explanations as scaffolding between a program outline and coding, without showing generated code.Prompts asked the model to generate code, which a separate LLM function converted to pseudo-code; hovering over each line revealed its explanation.Source: §5.1 · p. 8, §5.1.1 · p. 8

5.1.1 Pseudo-Code Integration.

  • CodeAid replaced underused Inline Code Exploration with static documentation for query-relevant functions, including usage descriptions and code examples.The system retrieved documentation from a local key-value database populated with Standard C Library documentation.Source: §5.1.2 · p. 8

5.1.3 Stream Generation.

  • CodeAid added streaming to show responses immediately, using feature-specific markup to parse partial outputs and display progress during sequential LLM calls.For sequential functions, the interface displayed and updated the number of generated code lines without showing the code itself.Source: §5.1.3 · p. 8

5.1.4 Updated Prompts and OpenAI Model.

  • CodeAid refined examples for more detailed answers, upgraded from code-davinci-002 to gpt-3.5-turbo, and restricted responses to questions relevant to C programming.Source: §5.1.4 · pp. 8–9

5.1.5 Redesigned Help Fix Code.

  • The redesigned Help Fix Code highlighted lines to change, remove, or add, while follow-up questions and suggested prompts expanded response steering.The workflow generated fixed code, matched its lines to buggy code, annotated differences, and added explanations; follow-ups were enabled in Question from Code, Explain Code, and Help Write Code.Source: §5.1.5 · p. 9, §5.1.6 · p. 9

6 RESULTS · 6.1 RQ1: Students’ Usage Patterns and Feedback

Across a 12-week course, students used CodeAid for programming questions, debugging, writing, and code explanation, with activity rising near major deadlines.

6 RESULTS

  • The results address RQ1 usage patterns, RQ2 response effectiveness, and RQ3 student perceptions using interactions, thematic analyses, interviews, and surveys.The study drew on 22 student interviews and weekly and end-of-semester anonymous surveys.Source: §6 · pp. 9–10

6.1 RQ1: Students’ Usage Patterns and Feedback

  • 372 students made 8,132 original and 1,986 follow-up queries; 300 consented to analysis of 7,003 original queries and responses.Consented students averaged 23.3 questions each (SD=41.2; range 1–333), and activity surged near assignment deadlines or exams.Source: §6.1 · p. 10, Figure 6 · p. 10
  • Women averaged 33.8 CodeAid uses versus 18.4 for men (p=.004, d=.34), despite comprising 30% of the course.The comparison used an independent samples t-test.Source: §6.1 · p. 10
  • General Question was the most-used feature (38%, n=2,682), followed by Question from Code (28%, n=1,959) and Help Fix Code (23%, n=1,611).Explain Code accounted for 5.5% (n=388) and Help Write Code for 4% (n=283); survey respondents reported similar weekly average usage of CodeAid and course office hours.Source: §6.1 · p. 10, Figure 6 · p. 10

6.1.1 Asking Programming Questions.

  • Programming questions were the largest analyzed inquiry type (36%, n=643), covering code or conceptual clarification (26%, n=453), function specifications (5%, n=96), and execution probes (5%, n=94).Students mainly used General Question for conceptual clarification and function queries, and Question from Code to probe execution as they would with a compiler.Source: §6.1.1 · p. 11
  • Debugging accounted for 32% of analyzed inquiries, including buggy-code resolution (22%, n=385), problem-source identification (7%, n=130), and error-message interpretation (3%, n=50).Students primarily used Help Fix Code for buggy-code resolution and Question from Code to identify why code produced errors or unexpected behavior.Source: §6.1.2 · p. 11

6.1.2 Debugging Code.

  • Writing-code inquiries made up 24% of analyzed usage: students sought high-level guidance (13%, n=237) or direct code solutions (10%, n=185).They mostly used General Question for high-level guidance; direct-solution requests also used Help Write Code and Question from Code.Source: §6.1.3 · p. 11

6.1.3 Writing Code.

  • Students used Explain Code to understand course-provided starter code in 6% of analyzed instances (n=97).Source: §6.1.4 · p. 11

6.2 RQ2: CodeAid’s Response Quality

CodeAid’s response quality varied by feature: overall correctness and helpfulness improved after an update, while students valued clear, accurate assistance and reported shortcomings such as incomplete answers and difficulty with complex code. It generally avoided showing solutions, though some features still generated behavior-specific code.

6.2 RQ2: CodeAid’s Response Quality

  • 79% of sampled responses were correct (1,386 instances), and 86% of correct responses were helpful (1,196 of 1,386).After the update, correctness rose from 74% (781 of 1,057) to 87% (603 of 692), and helpfulness rose from 83% (646 of 781) to 91% (550 of 603).Source: §6.2.1 · p. 11

6.2.1 Overall Correctness and Helpfulness.

  • CodeAid gave conceptual natural-language answers to 43% of queries and pseudo-code to 24%, while debugging responses never displayed fixed code.Of all queries, 16% received generic high-level example code and 6% behavior-specific pseudo-code; the initial General Question and Question from Code features also produced short solutions in 2% of instances.Source: §6.2.2 · p. 11

6.2.3 General Question.

  • 91% of General Question responses were correct and 84% helpful; students rated usefulness 4.04 out of 5.High ratings cited correctness, helpfulness, concision, and clarity; low ratings cited superficial explanations, absent example code, irrelevant or unclear answers, and misleading information on complex requests.Source: §6.2.3 · p. 11

6.2.4 Question from Code.

  • Question from Code responses were 66% correct and 55% helpful, with average usefulness rated 3.28 out of 5.Students valued precise error identification, detailed answers, and compile checks; low ratings cited incorrect or redundant changes, vague explanations, and difficulty understanding submitted code.Source: §6.2.4 · pp. 11–12

6.2.5 Help Fix Code.

  • Help Fix Code responses were 63% correct and 42% helpful, with average usefulness rated 2.67 out of 5.Students valued fix suggestions and visual annotations, but reported inaccurate or irrelevant suggestions, difficulty with complex tasks, and trouble handling longer code or interactions across files.Source: §6.2.5 · p. 12

6.2.6 Explain Code.

  • Explain Code explanations were 95% accurate, and students rated their usefulness 4.17 out of 5.Students valued accurate code breakdowns and using explanations for review; lower ratings cited inaccuracies, brevity, or lack of new information.Source: §6.2.6 · p. 12

6.2.7 Help Write Code.

  • Help Write Code responses were 92% correct and 82% helpful; 53% gave the exact requested solution and 32% supplied high-level example pseudo-code.Usefulness averaged 3.29 out of 5; students valued help breaking coding tasks into smaller steps, while poorly rated answers were incomplete or irrelevant.Source: §6.2.7 · p. 12

6.3 RQ3: Students Perspectives and Concerns

Students valued CodeAid’s convenient, course-relevant support and described ways it could aid learning, while raising concerns about accuracy, trust, and reliance on AI. They also compared CodeAid with ChatGPT and considered whether and how they would use AI coding tools in the future.

6.3 RQ3: Students Perspectives and Concerns

  • Students valued CodeAid’s 24/7 availability, private help, and ability to resolve coding issues they could not solve alone.They described it as a supplementary resource where they could ask questions without fear of human judgment.Source: §6.3.1 · p. 12

6.3.1 Accessibility and Convenience.

  • Students appreciated CodeAid’s concise, course-relevant help and flexible question wording, but cited misunderstood queries and input-length limits.They contrasted pasting code and asking questions naturally with searching for relevant Stack Overflow posts.Source: §6.3.2 · p. 12

6.3.2 Contextual Assistance.

  • Some students said CodeAid deepened their understanding, while others preferred hints and many self-regulated by avoiding solutions or limiting their queries.One student worried about over-reliance, but others avoided the fix-code feature or asked general questions to support their learning.Source: §6.3.3 · p. 12

6.3.3 Learning and Dependency.

  • Students likened CodeAid’s accuracy to teaching assistants and found it more accurate on simple questions, but warned that wrong answers could sound confident.One student described it as “80% accurate answers, similar to TA.”Source: §6.3.4 · p. 12

6.3.4 Trust and Reliability.

  • Students’ trust varied: some trusted CodeAid or its course endorsement, while others doubted AI accuracy or avoided it for lack of need, self-reliance, or competing resources.Other reasons for not using it included being unaware or forgetting, skepticism after negative experiences, preferring friends, and mandatory feedback.Source: §6.3.4 · p. 13, §6.3.5 · p. 13

6.3.5 Reasons for Not Using CodeAid.

  • Students who did not use CodeAid cited sufficient course materials or easy coursework, preference for other resources, self-reliance, and skepticism about AI.Some were unaware of or forgot the tool, preferred friends, or objected to mandatory feedback.Source: §6.3.5 · p. 13

6.3.6 Comparing CodeAid with ChatGPT.

  • Among 200 survey respondents, 38% used both tools, 23% only CodeAid, and 19% only ChatGPT; ChatGPT was favored for complex inputs and code examples.90 students used ChatGPT sometimes, often, or a lot, compared with 66 for CodeAid; some preferred CodeAid’s technical explanations because they made them think.Source: §6.3.6 · p. 13

6.3.7 Future Integration of AI Coding Tools.

  • Most students expected to keep using AI coding tools for efficiency and coding support, but some rejected near-term adoption over learning and debugging limitations.Students envisioned using AI for skeleton code, optimization, and tedious tasks; others wanted coding-environment integration or prompt-writing instruction, while some opposed AI for learning because answers could be confidently incorrect and discourage figuring things out independently.Source: §6.3.7 · p. 13

7 EDUCATOR INTERVIEWS

Eight computing educators interviewed about CodeAid generally valued its pedagogical approach but raised concerns about accuracy, student reliance on AI, and how the tool should be controlled. They also proposed ways to keep students engaged with CodeAid and customize it for courses while balancing instructional insight with student privacy.

7.1.1 General Impressions.

  • Educators viewed CodeAid as a sensible, integrity-conscious alternative to direct-answer tools and as an assistant that can prompt critical thinking and support flipped-classroom preparation.One educator compared it to an excellent TA, while another saw it helping students prepare for class and reflect on lecture material.Source: §7.1.1 · p. 13

7.1.2 Perceptions on Pseudo-Code Usage.

  • Educators valued pseudo-code with line-by-line explanations for structuring solutions without exposing syntax, though one worried that revealing algorithms could harm learning in advanced courses.They said hiding syntax could reduce cognitive load and support metacognitive skills; one also considered pseudo-code preferable to readily available precise solutions for some queries.Source: §7.1.2 · pp. 13–14
  • Educators warned that inaccurate responses could mislead students with weak fundamentals, urged teaching students to critique AI, and suggested tutorials, accuracy transparency, reflective prompts, and usage limits.T3 specifically worried that Help Fix Code might highlight the wrong lines and proposed labeling potentially incorrect lines while asking students to reflect.Source: §7.1.3 · p. 14

7.1.3 Concerns about Incorrect Responses and Misuses.

  • To keep students from switching to ChatGPT, educators proposed an all-in-one tool with code editing and execution, and some suggested revealing solutions after repeated failed attempts with learning-oriented safeguards.T3 opposed revealing solutions, while another educator noted that disabling a CodeAid feature might push students toward ChatGPT.Source: §7.1.4 · p. 14

7.1.4 Keeping Students from Switching to ChatGPT.

  • Educators wanted instructors to control response types and when pseudo-code appears, and to constrain answers to topics already taught in class.Suggestions included using pseudo-code for implementation questions but hints for problem-solving, and enabling pseudo-code only early in a course.Source: §7.1.5 · p. 14

7.1.5 Pedagogical Customization.

  • Educators proposed dashboards to identify common questions and instructional gaps, while cautioning that individual monitoring may conflict with privacy rules and student comfort.Aggregated data could inform teaching, but educators raised GDPR-related limits on individual data access and emphasized that students should feel free to ask questions without feeling watched.Source: §7.1.6 · p. 14

8 BEYOND CODEAID: IMPLICATIONS FOR PEDAGOGICAL LLM-POWERED CODING ASSISTANTS

The paper frames educational AI-assistant design around four stages of help-seeking: choosing a tool, formulating a query, receiving a response, and acting on it. It proposes balancing AI’s distinctive benefits, query usability and reflection, response directness and learning, and trust, transparency, and control.

8.1 D1: Exploiting Unique Advantages of AI

  • Course-specific assistants can combine natural-language, contextual, tailored help and pseudo-code or documentation with accurate, course-aligned responses and links to relevant resources.The paper suggests retrieval augmented generation [35], curriculum access to align responses with students’ zone of proximal development, and instructor-verified practice exercises and worked examples.Source: §8.1 · p. 15, Figure 7 · p. 15

8.2 D2: Designing the AI Querying Interface

  • Query interfaces should balance free-form ease with structured reflection: automatic context, free-form input, and proactive help may improve usability but reduce metacognitive engagement.Future research should assess minimally distracting proactive assistance, since students in prior work disliked suggestions when they did not need help [51].Source: §8.2 · p. 15

8.3 D3: Balancing the Directness of AI Responses

  • Students wanted different levels of directness, from explicit example code to subtle hints or direct solutions, while educators wanted scaffolding they could customize to course goals.The design challenge is to balance critical learning and learner autonomy against frustration; overly direct assistance may miss learning opportunities, while indirect help may leave students overwhelmed or unsupported.Source: §8.3 · p. 16
  • To support learning without simply giving solutions, assistants could gate generated code behind comprehension activities, use Socratic problem-solving, and distinguish example code from assessment solutions.One proposal reveals code segments after students answer related questions; example code could be offered for queries unrelated to course assessments.Source: §8.3 · p. 16

8.4 D4: Supporting Trust, Transparency and Control

  • CodeAid’s sometimes incorrect or unhelpful indirect responses made verification harder, risking student trust compared with directly runnable code from tools such as ChatGPT.Students often had to write and test a small program to check CodeAid’s suggestions.Source: §8.4 · p. 16
  • Assistants could make assumptions visible, invite clarification and feedback, and help users test scaffolded responses through code execution without revealing the underlying code.The paper proposes a sufficiency check to elicit missing context [37] and a line-by-line execution interface, potentially showing pseudo-code, similar to PythonTutor; response validation remains an open research question.Source: §8.4 · p. 16

9 LIMITATIONS AND FUTURE WORK

The findings are bounded by a single course and university and by the models used; future research should test learning outcomes, demographic patterns, and design trade-offs across evolving educational contexts.

9 LIMITATIONS AND FUTURE WORK

  • Findings come from one second-year C course, and model version and language/task variability may shape perceived accuracy, utility, trust, engagement, and usage.The initial code-davinci-002 release may have contributed to inaccurate responses and declining usage compared with the later gpt-3.5-turbo model.Source: §9 · p. 16
  • Longitudinal studies should examine assistants’ effects on learning, competency, self-regulation, and frustration, with controlled comparisons against unrestricted LLMs such as ChatGPT.Source: §9 · p. 16
  • Women used CodeAid more often than men, motivating research on other demographic factors and links to resource use such as TA office hours and online discussion forums.Source: §9 · p. 16
  • The educational AI design considerations need further exploration because educational contexts and AI technologies vary and may reveal new dimensions and trade-offs.Source: §9 · p. 16

10 CONCLUSION

CodeAid pairs guardrails against direct solutions with scaffolding to support programming learning. A semester-long classroom deployment informed four design considerations for educational AI tools.

10 CONCLUSION

  • The study synthesizes deployment data, student feedback, surveys, and interviews with students and educators into four design considerations for educational AI tools.The deployment involved a programming class of 700 students; data included approximately 8,000 usages, weekly surveys, 22 student interviews, and interviews with eight educators.Source: §10 · p. 17
  • CodeAid uses guardrails to avoid direct solutions and instead offers scaffolding, including interactive pseudo-code, to support engaging learning experiences.Source: §10 · p. 17
  • The study analyzes 1,749 usages for correctness, helpfulness, and whether responses reveal direct solutions, alongside feedback on usefulness and a final survey about ChatGPT.Source: §10 · p. 17

A THEMATIC ANALYSIS CODEBOOK

The appendix presents the final codebook for analyzing CodeAid usage across two primary dimensions: students’ queries and CodeAid’s responses.

A THEMATIC ANALYSIS CODEBOOK

  • The final codebook organizes analysis of each CodeAid usage around Query and Response.The appendix identifies these as its two primary dimensions.Source: A THEMATIC ANALYSIS CODEBOOK · p. 19
  • The Query codebook asks what students are asking CodeAid.Table A1 is titled “CodeAid Thematic Analysis Codebook: Query.”Source: Table A1 · p. 19
  • The Response codebook examines how much CodeAid directly reveals the solution.Table A2 covers the degree of direct solution disclosure.Source: Table A2 · p. 19
  • The Response codebook separately assesses how technically correct CodeAid’s answers are.This dimension is covered in Table A3.Source: Table A3 · p. 20
  • The Response codebook also assesses how helpful an answer is if it is correct.This conditional helpfulness dimension is covered in Table A4.Source: Table A4 · p. 20

Related papers

Loading 2401.11314v2…