PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z. Henley, Paul Denny, Michelle Craig, Tovi Grossman

arXiv:2401.11314v2cs.HCcs.AI

TL;DR

Programming courses need scalable, equitable assistance, while direct code-generating tools raise concerns about academic integrity and over-reliance. The paper develops and deploys CodeAid, a guarded LLM assistant that supports programming without directly revealing code solutions, and evaluates it through a semester-long classroom study. The study identifies four design considerations and reports measured response correctness and helpfulness alongside limitations tied to context and model choice.

  • Problem

    Growing programming enrollments and inequitable, poorly utilized in-person support motivate scalable assistance, while direct LLM code generation raises educational concerns.

  • Method

    The authors iteratively designed CodeAid’s guarded programming features and deployed it in a 12-week course with about 700 students, analyzing usage, surveys, and interviews.

  • Results

    CodeAid’s analyzed responses were 79% correct and 86% helpful among correct instances; after updating the system, correctness rose from 74% to 87% and helpfulness from 83% to 91%.

  • Takeaways & Limitations

    The findings identify four design considerations for educational AI assistants, including exploiting AI benefits, supporting cognitive engagement, avoiding direct responses, and preserving transparency and control.

  • Takeaways & Limitations

    The findings come from one second-year C programming course, and perceived utility and trust were strongly tied to the OpenAI models used.

Abstract

from arXiv · show

Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI's unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.

1 INTRODUCTION

Programming education faces growing demand, limited scalable support, and inequitable access to instructors. CodeAid addresses these challenges with an LLM assistant designed to help students without directly revealing code solutions, evaluated through a semester-long deployment.

  • Growing enrollment across disciplines is increasing pressure on institutional programming-education resources.
  • Scheduled office hours are often poorly utilized, limiting timely assistance when students need programming help.
  • In-person support can be inequitable because students differ in comfort approaching instructors and in how repeatedly they receive help.
  • LLMs can provide instant coding assistance but raise concerns about academic integrity, over-reliance, and direct solution generation.
  • CodeAid was iteratively designed to provide technically correct, helpful assistance without directly revealing solutions through five programming-support features.
  • The study combined more than 8,000 interactions, weekly surveys, and student and educator perspectives to examine CodeAid’s educational use.

2 RELATED WORK

Prior work establishes both the educational potential and risks of LLMs in computing education. CodeAid extends this work by studying scaffolded, non-solution assistance across conceptual questions, code explanations, and programming tasks.

  • LLM research in education examines content generation, engagement, customized learning, and the need to adapt computing instruction.
  • Earlier studies primarily assessed LLM capabilities on programming tasks or their ability to generate instructional materials.
  • CodeAid differs from direct code generators by helping students complete programming exercises without providing direct code solutions.
  • CodeHelp demonstrated that students valued on-demand assistance, especially for specific code-related tasks such as fixing errors.
  • CodeAid broadens this approach with multiple input templates and interactive response formats for diverse student needs.
  • Accurate code explanations can improve learners’ reasoning, while producing high-quality explanations for varied code fragments burdens instructors.
  • LLM-generated explanations are generally perceived as helpful, with engagement varying by code length, complexity, and explanation type.
  • CodeAid adds direct questioning from student code and interactive line-by-line explanations to existing LLM explanation approaches.

3 INITIAL SYSTEM DESIGN AND ARCHITECTURE

CodeAid combines five guarded programming-assistance features with interfaces and prompts tailored to conceptual understanding, debugging, explanation, and code planning. Few-shot examples constrain outputs while supporting structured, interactive responses.

  • CodeAid provides five main features: Help Write Code, Help Fix Code, General Question, Explain Code, and Question from Code.
  • The system’s guardrails aim to produce helpful responses and reinforce concepts without directly generating code solutions.
  • The interface enables feature selection, then selectively activates question and code inputs before displaying a feature-specific response.
  • General Question generates short natural-language answers with informative explanations for conceptual C-programming questions.
  • Question from Code combines student code and questions to support debugging or conceptual questions in a specific context.
  • Help Fix Code generates a corrected version internally and explains what changed and why through suggested fixes.
  • Explain Code presents students’ code with hover-enabled, line-by-line explanations of each line and its orchestration with the rest of the code.
  • Help Write Code generates high-level structures, sub-goals, and natural-language pseudo-code while including relevant C-library information without code.

4 SEMESTER-LONG CLASS DEPLOYMENT

CodeAid was studied as an optional resource in a large second-year C and Systems Programming course using logs, surveys, interviews, and thematic analysis. The study combined quantitative usage data with qualitative examination of queries and responses.

  • The deployment ran for a 12-week semester in a large North American university course with over 700 students.
  • The second-year course covered C programming, shell programming, systems topics, and basic network programming across a large multi-section class.
  • Course policy permitted CodeAid but prohibited using other AI tools, including ChatGPT, to complete coursework.
  • Participation in CodeAid use, surveys, and interviews was voluntary and confidential from course instructors.
  • The study collected interaction logs, ten weekly surveys, an anonymous post-course survey, and semi-structured interviews with 22 randomly selected students.
  • Researchers analyzed students’ queries and CodeAid responses separately to study usage patterns and response quality.
  • From 8,132 usages, filtering and sampling produced 1,750 usages for thematic analysis after excluding nonparticipants, unrelated or excessively long usages, and technical errors.
  • Thematic coding used inductive codebook development, independent coding, disagreement resolution, and inter-rater reliability assessment.

5 MIDTERM FEEDBACK AND SYSTEM ITERATION

Student feedback prompted iterative updates to CodeAid that expanded explanations, improved debugging guidance, and gave students more ways to steer responses without directly revealing solutions.

  • Feedback: Midcourse feedback praised CodeAid’s clear explanations, error identification, availability, personalization, and flexibility, but found responses often too brief.These findings motivated revisions after discussions with the course instructor.
  • Response design: Pseudo-code was added as scaffolding, providing structured guidance between natural language and executable code without displaying the generated code.Updated prompts generated code internally, then passed it to another function that produced interactive pseudo-code and line-by-line explanations.
  • Supporting resources: The updated system replaced underused Inline Code Exploration with static documentation for relevant functions, including usage descriptions and code examples.An LLM identified relevant functions, while a local key-value database supplied their documentation.
  • Debugging support: Help Fix Code was redesigned to highlight lines requiring modifications, deletions, or additions after generating a fixed-code version and explained changes.The annotated changes were streamed to the client for highlighted lines and hover interactions.
  • Student control: Prompt revisions added follow-up questions and suggested follow-ups to improve students’ ability to steer interactions.The redesign also refined prompts and upgraded the model from code-davinci-002 to gpt-3.5-turbo.

6 RESULTS

Across the semester, CodeAid was used by hundreds of students for conceptual questions, debugging, code-writing assistance, and code explanation, with usage varying across features and student groups.

  • Overall usage: 372 students submitted 8,132 original and 1,986 follow-up queries, while 300 consenting students contributed 7,003 original queries for analysis.Students averaged 23.3 queries each, with usage ranging from 1 to 333 questions.
  • Overall usage: Women averaged 33.8 CodeAid uses versus 18.4 among men, a significant difference (p=.004, d=.34).Women comprised 30% of the course according to self-reported gender categories.
  • Feature usage: 38% (n=2682) of usages involved General Question, followed by Question from Code at 28% (n=1959) and Help Fix Code at 23% (n=1611).Students reported similar weekly average usage of CodeAid and course office hours.
  • Inquiry types: Among 1,749 thematically analyzed usages, programming questions comprised 36% (n=643), debugging 32%, code writing 24%, and code explanation 6%.Programming questions included conceptual clarification, function specifications, and code-execution probes.
  • Inquiry types: Students used CodeAid for high-level coding guidance as well as direct code requests, with writing-code inquiries accounting for 24% of analyzed usages.Debugging inquiries included buggy-code resolution, source identification, and error-message interpretation.

6.2 RQ2: CodeAid’s Response Quality

CodeAid generally produced correct and helpful responses while avoiding direct solutions, but response quality differed substantially across features and could decline for complex code or requests.

  • Overall quality: 79% (1,386 correct instances) of 1,749 sampled responses were correct, and 86% (1,196 out of 1,386) of correct responses were helpful.After the update, correctness rose from 74% (781 out of 1,057) to 87% (603 out of 692), while helpfulness rose from 83% (646 out of 781) to 91% (550 out of 603).
  • Solution avoidance: CodeAid avoided displaying direct code solutions: 43% of responses were natural-language explanations, 24% pseudo-code, and debugging responses provided suggestions rather than fixed code.However, 2% of responses from two features generated short code solutions for specific behaviors.
  • Feature quality: General Question responses were 91% (n=668) correct and 84% (n=613) helpful across 733 samples, with an average usefulness rating of 4.04/5.Low ratings commonly reflected superficial, incomplete, irrelevant, unclear, or misleading answers, especially for complex requests.
  • Feature quality: Question from Code responses were 66% (n=310) correct and 55% (n=258) helpful across 467 samples, with an average usefulness rating of 3.28/5.Students valued precise error localization, while lower ratings cited incorrect, incomplete, vague, redundant, or poorly understood responses.
  • Feature quality: Help Fix Code achieved 63% (n=214) correctness and 42% (n=142) helpfulness across 340 samples, receiving an average usefulness rating of 2.67/5.Students appreciated visual annotations but reported challenges with longer and more complex code.
  • Feature quality: Explain Code produced accurate and beneficial explanations in 95% (90 out of 95) of sampled usages and received a 4.17/5 average usefulness rating.Help Write Code produced 92% correct and 82% helpful responses across 77 samples, but generated exact solutions in 53% of cases.

6.3 RQ3: Students Perspectives and Concerns

Students valued CodeAid’s availability, privacy, contextual assistance, and learning support, while reporting concerns about over-reliance, incorrect confident answers, and differences from ChatGPT.

  • Student experiences: Students valued CodeAid’s 24/7 availability, private questioning environment, and supplementary help with coding problems.Students described it as an always-available resource that helped resolve issues they could not solve independently.
  • Student experiences: Students appreciated CodeAid’s concise, contextually relevant responses and ability to accept questions phrased in their own terms.They contrasted this with search engines and valued assistance tailored to course context.
  • Learning and self-regulation: Students reported deeper understanding from CodeAid, but some preferred hints over answers and deliberately avoided features that could reduce independent thinking.Several students described self-regulation, while one reported over-reliance.
  • Accuracy and trust: Students judged CodeAid roughly comparable to teaching assistants in accuracy, while warning that confident incorrect answers could undermine trust.Trust varied, with course endorsement increasing confidence for some students.
  • Adoption and comparison: Students chose not to use CodeAid when existing resources seemed sufficient, they preferred tools such as debuggers or ChatGPT, or they sought self-reliance.ChatGPT users valued its interface, larger character limit, flexible input, code examples, and comprehensive reviews, though some found direct solutions poor for learning.
  • Future use: Most students expected to continue using AI coding tools for efficiency and routine programming, but some questioned their value for learning and debugging incomplete programs.Students specifically cited confident errors and the need to see an entire program for effective debugging.

7 EDUCATOR INTERVIEWS

Educators generally viewed CodeAid as a safer, more pedagogical alternative to unrestricted coding assistants, but emphasized risks from incorrect answers, misuse, limited course fit, and privacy-sensitive monitoring.

  • Participants: Eight educators from six countries, primarily experienced undergraduate programming instructors, participated in semi-structured interviews.Six educators had more than ten years of teaching experience.
  • Perceived risks: Educators saw unrestricted tools such as ChatGPT as especially threatening in introductory programming courses and preferred assistants that recognize learners’ needs.Concern was less pronounced for advanced Computer Science courses.
  • Perceived benefits: Educators generally favored CodeAid as an academically safer tool that encourages critical thinking instead of supplying direct answers.One educator compared its approach to an excellent teaching assistant.
  • Pseudo-code: Educators valued pseudo-code with line-by-line explanations because it provides structure and focuses attention on logic without revealing syntax.One educator cautioned that exposing algorithms through pseudo-code could harm learning in upper-level courses.
  • Concerns and safeguards: Educators worried that incorrect responses and excessive trust could harm students with weak fundamentals, and proposed tutorials and quizzes to build critique skills.The proposed preparation would occur before students use the assistant.
  • Retention and directness: Educators proposed making CodeAid comprehensive enough to compete with ChatGPT, while disagreeing about whether failed attempts should eventually reveal code solutions.Suggestions included an editor, execution capabilities, gamification, locking, and learning activities after disclosure.
  • Pedagogical customization: Educators wanted control over response types, pseudo-code availability, and taught-topic boundaries so assistance could match course content and instructional timing.Examples included hints for problem-solving, pseudo-code for implementation, and restricting advanced topics.
  • Monitoring and privacy: Educator dashboards could reveal recurring instructional gaps, but individual monitoring raises privacy and anonymity concerns under regulations such as GDPR.Aggregated data was viewed as more compatible with feedback than unrestricted access to individual interactions.

8 BEYOND CODEAID: IMPLICATIONS FOR PEDAGOGICAL LLM-POWERED CODING ASSISTANTS

The paper frames pedagogical coding assistants around four stages of help-seeking: choosing the tool, formulating queries, controlling response directness, and evaluating responses. Each stage involves usability, learning, trust, and privacy trade-offs.

  • Design framework: The proposed design space follows four help-seeking stages: choosing an AI tool, formulating a query, receiving assistance, and acting on the response.These stages organize four corresponding design considerations.
  • D1: Exploiting Unique Advantages of AI: D1 recommends exploiting AI’s distinctive advantages, including natural-language interaction, contextual assistance, and tailored responses relative to existing learning resources.Course-specific assistants and contextual methods such as retrieval augmented generation are suggested.
  • D1: Exploiting Unique Advantages of AI: Course-specific assistants should provide accurate, technically correct, informative responses while helping retain students within a learner-centered tool.The paper identifies course context as a way to differentiate educational assistants from productivity-focused tools.
  • D2: Designing the AI Querying Interface: D2 concerns interfaces that balance easy querying with meta-cognitive engagement, contrasting students’ preference for free-form input with educators’ support for structured inquiry.Structured interaction was associated with active engagement, critical thinking, and thoughtful inquiry.
  • D2: Designing the AI Querying Interface: Designers must choose how context is gathered, how freely students enter questions, and how proactively the assistant identifies problems.Automation and proactivity may improve usability but reduce self-reflection and well-formed questioning.
  • D2: Designing the AI Querying Interface: Proactive assistance remains an open evaluation area because students may dislike unsolicited suggestions when they do not perceive a need for help.Future work should focus on minimally distracting designs.
  • D3: Balancing the Directness of AI Responses: D3 addresses how response directness and scaffolding should be controlled, since students and educators differ in whether they prefer examples, hints, pseudo-code, or direct solutions.Customization may vary by query type or course segment.
  • D3: Balancing the Directness of AI Responses: Overly direct assistance may remove learning opportunities, whereas overly indirect responses may frustrate students who feel unsupported or unable to progress.The design challenge is balancing critical scaffolding, frustration, and learner autonomy.

9 LIMITATIONS AND FUTURE WORK

The deployment’s findings are bounded by a single-university C-programming context and evolving model performance, while future work should test broader settings and longer-term educational outcomes. The paper also reports that women used CodeAid more often than men and calls for further exploration of its design considerations.

  • CodeAid’s perceived accuracy, utility, trust, and engagement were closely tied to the OpenAI models used.The authors note differences between code-davinci-002 and gpt-3.5-turbo, including possible effects on inaccurate responses and declining usage.
  • Future studies should examine long-term effects on learning outcomes, competency, self-regulation, and frustration, alongside controlled comparisons.
  • Women used CodeAid more often than men, motivating research on additional demographic factors and related resource-use patterns.
  • The proposed educational-AI design considerations require further exploration because educational contexts and AI technologies vary.

10 CONCLUSION

CodeAid uses guardrails to prevent direct coding solutions and instead provides scaffolding such as interactive pseudo-code. Its semester-long deployment examined student use and the implications of scaling instructional expertise through an AI-powered tutor.

  • CodeAid prevents direct solutions and provides scaffolding, including interactive pseudo-code, to support engaging learning experiences.
  • The paper studies how students use an AI-powered tutor through a semester-long programming-class deployment.
  • The deployment examines broader implications of using AI to scale instructional expertise.

A THEMATIC ANALYSIS CODEBOOK

The thematic-analysis codebook organizes CodeAid usage around what students ask and how the assistant responds. Response analysis covers solution revelation, technical correctness, and helpfulness when correct.

  • The analysis codes each CodeAid usage along two primary dimensions: Query and Response.
  • The Query dimension captures what students ask CodeAid.
  • The Response dimension assesses how directly CodeAid reveals a solution.
  • Response analysis evaluates how technically correct CodeAid’s output is.
  • Response analysis evaluates how helpful an answer is when it is correct.
Loading 2401.11314v2…