Source-linked AI summary
CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Mark Liffiton, Brad Sheese, Jaromir Savelka, Paul Denny
TL;DR
Large programming classes need scalable, timely support, but direct LLM assistance can encourage student over-reliance and reveal solutions. CodeHelp combines an LLM-powered support tool with guardrails and prompting strategies, then evaluates it in a 52-student, 12-week course deployment. The tool was well-received, easy and inexpensive to deploy, and appeared to complement existing instructor and teaching-assistant support.
Problem
Large programming classes need scalable, immediate support, while direct LLM use raises concerns about over-reliance and solution disclosure.
Method
CodeHelp uses guardrails and prompting strategies to provide real-time programming help without directly revealing solutions, with instructor-oriented interaction-review features.
Results
CodeHelp was well-received, easy and inexpensive to deploy, and appeared to complement and expand support from instructors and teaching assistants.
Takeaways & Limitations
In a 52-student, 12-week pilot, CodeHelp provided an additional avenue for immediate support alongside professors and teaching assistants.
Takeaways & Limitations
Each query is one-shot, with no follow-up dialogue, because additional interaction rounds increase risks of incorrect, harmful, or off-topic responses.
Abstract
from arXiv · showhide
Computing educators face significant challenges in providing timely support to students, especially in large class settings. Large language models (LLMs) have emerged recently and show great promise for providing on-demand help at a large scale, but there are concerns that students may over-rely on the outputs produced by these models. In this paper, we introduce CodeHelp, a novel LLM-powered tool designed with guardrails to provide on-demand assistance to programming students without directly revealing solutions. We detail the design of the tool, which incorporates a number of useful features for instructors, and elaborate on the pipeline of prompting strategies we use to ensure generated outputs are suitable for students. To evaluate CodeHelp, we deployed it in a first-year computer and data science course with 52 students and collected student interactions over a 12-week period. We examine students' usage patterns and perceptions of the tool, and we report reflections from the course instructor and a series of recommendations for classroom use. Our findings suggest that CodeHelp is well-received by students who especially value its availability and help with resolving errors, and that for instructors it is easy to deploy and complements, rather than replaces, the support that they provide to students.
1 INTRODUCTION AND MOTIVATION
Expanding programming enrollments make timely, scalable support difficult, while LLMs offer promising on-demand assistance but raise concerns about student over-reliance. CodeHelp addresses this tension with guardrails that guide students without directly revealing solutions and was evaluated in a 52-student, 12-week course deployment.
- Large classes make timely expert support difficult to scale, and some students are uncomfortable asking instructors or teaching assistants for help.
- Static hints for common programming issues are time-intensive to author and may not be exhaustive, creating demand for immediate, high-quality scalable support.
- LLMs can generate programming exercises, code explanations, model solutions, and potentially real-time on-demand help for computing education.
- CodeHelp uses guardrails and prompting strategies to guide students toward solutions without directly revealing answers.
- 52 students used CodeHelp in a first-year computer- and data-science course over 12 weeks while researchers examined engagement, help requests, and perceived usefulness.
- CodeHelp was well-received, easy and inexpensive to deploy, and appeared to complement and expand instructor and teaching-assistant support.
2 RELATED WORK
Prior programming-support systems include intelligent tutors, rule-based chatbots, and code-generation tools, but recent evidence highlights both LLM capability and risks of direct solution provision. CodeHelp differs by explicitly using guardrails and evaluating always-available classroom support over an extended period.
- Intelligent programming tutors have primarily focused on generating effective hints and feedback for novice programmers.
- Existing programming chatbots provide definitions, error guidance, exercise assistance, topic practice, assessment, or help-seeking support through more predetermined interactions.
- LLMs can perform strongly on programming tasks, but studies also report unreliable code generation and weaker tutoring outcomes than human-generated hints.
- Coding Steps users submitted AI-generated code without modification 49% of the time across 45 Python tasks, raising concerns about student over-reliance.
- Prior work describes language models as unreliable tutors that may reveal model solutions even when explicitly instructed not to.
- CodeHelp explicitly designs guardrails against over-reliance, evaluates use over 12 weeks beyond scheduled classes, and provides immediate feedback at any time.
3 CODEHELP DESIGN AND IMPLEMENTATION
CodeHelp combines a structured student interface, instructor-facing monitoring, and a multi-prompt LLM workflow with guardrails to provide guidance without directly revealing solutions.
- 3.1 Student Interfaces: CodeHelp structures help requests through separate language, code, error, and issue fields, guiding students while adding context to the LLM prompt.Code and error fields are optional, and provided error messages are prompted for explanation.
- 3.1 Student Interfaces: Students receive the generated response alongside their query, an incorrectness warning, and a feedback form indicating whether the answer was helpful.The query, response, and feedback are stored for instructor review.
- 3.1 Student Interfaces: When requests lack sufficient context, CodeHelp presents a clarification request while still providing a response to prevent students from becoming stuck.The clarification and response are generated through the LLM workflow, and the retry form is pre-filled with the same query.
- 3.2 Generating Responses: The response goals are to provide explanations and guidance, avoid complete copyable solutions, identify ambiguous queries, and restrict responses to course-relevant questions.These goals are implemented through multiple prompts because current LLMs do not consistently follow all instructions in one prompt.
- 3.2 Generating Responses: CodeHelp uses parallel sufficiency and main-response prompts, scores two main completions, and removes code blocks from the selected response when necessary.The code-removal prompt rewrites remaining text so it remains clear without the removed code.
- 3.2 Generating Responses: The tool costs roughly $0.002 per query on average, with estimated unrestricted use for 50 students over a semester costing under $10.Increasing completion lengths could raise the estimated cost to roughly $0.05 per query.
4 LIMITATIONS AND RISKS
CodeHelp retains several known LLM risks, including inaccurate or biased outputs, while restricting interactions to one-shot requests to reduce risks from extended dialogue.
- CodeHelp can produce factually incorrect statements that may confuse or mislead users.The authors describe these inaccuracies as hallucination or confabulation and warn users through a prominent notice.
- Inaccuracies often affect particular details while preserving correct high-level guidance or pointing users in the right direction.The authors report that the observed inaccuracy rate remained low enough for the tool to be valuable to students.
- CodeHelp may reflect harmful gender or racial stereotypes learned from LLM training data, although this is considered highly unlikely for specific programming-help requests.The models were trained to reduce such biases, but the authors note that some bias remains possible.
- Each CodeHelp query is a one-shot request without follow-up dialogue because additional interaction rounds increase the likelihood of incorrect, harmful, or off-topic responses.This design mitigates risks but limits usefulness because follow-up questions could provide valuable context or clarification.
5 EXPERIENCES AND RESULTS
Students used CodeHelp consistently and generally valued its availability, error assistance, and guidance, while noting occasional confusion, unfamiliar concepts, and potential over-reliance. The instructor found it easy to deploy and complementary to human support.
- Student Use: Roughly half the class used CodeHelp each week, about 70% used it in four or more weeks, and 80% submitted at least 10 queries.Usage occurred across nearly all hours, including times when instructors or TAs were unavailable.
- Student Survey: 95% of survey respondents expressed interest in using CodeHelp in future computer science courses.Forty-five students, representing 87% of the class, completed the optional survey.
- Student Survey: Students most often valued CodeHelp’s availability and help with fixing errors, with 19 and 11 responses respectively.Students also described support for independence, speed, and learning or understanding.
- Student Survey: Students reported occasional misunderstandings, confusing responses, unfamiliar concepts, and concerns about becoming dependent on CodeHelp.Some students nevertheless viewed the tool as guidance rather than an answer source, and 24 reported no concerns.
- Instructor Reflections: The instructor found CodeHelp easy to introduce, useful for common concerns and sophisticated introductory problems, and inexpensive to deploy.The instructor encouraged students to seek CodeHelp before instructor or TA assistance without enforcing that sequence.
- Instructor Reflections: The instructor reported that CodeHelp usually provided accurate, helpful responses without giving away answers or replacing active problem solving.It was viewed as balancing enough information to move students forward with preserving assignment intent.
- Instructor Reflections: CodeHelp supported students who could not attend office hours or hesitated to seek help from instructors and TAs.The instructor described it as an additional support avenue for students reluctant to ask humans for help.
- Instructor Reflections: The avoid set largely resolved cases where CodeHelp suggested methods inconsistent with course content or students’ knowledge levels.Such suggestions had confused and frustrated some students before the functionality was added.
6 RECOMMENDED PRACTICES
The authors recommend introducing CodeHelp transparently, monitoring student interactions and responses, and adapting course configuration when outputs do not fit instructional goals.
- Initial introduction: Instructors should explain CodeHelp’s course-specific strengths and limitations and teach students how to formulate effective help requests.Recommended request details include relevant code and important information about the problem.
- During Use: Instructors should regularly review student queries to identify learning difficulties, ineffective use, and possible over-reliance.These observations can inform course-plan adaptations and direct guidance for students who need it.
- During Use: Instructors and TAs should sample responses in each course section and add unsuitable techniques, functions, or concepts to the avoid set.The avoid set prevents specified content from appearing in future responses.
7 CONCLUSION AND FUTURE WORK
The authors conclude that properly integrated LLM assistance can expand programming support while complementing instructors and TAs, and they identify directions for tailoring CodeHelp further. They also emphasize that stronger evaluation and broader deployments are needed.
- Conclusion: CodeHelp provided immediate support while using guardrails to mitigate over-reliance, and students viewed it as a welcome addition to instructor and TA support.The authors connect its availability and reduced help-seeking anxiety with increased support during the course.
- Future Work: The authors identify the avoid set as critical for obtaining course-appropriate responses and plan to add more course-context controls.Planned improvements also include interventions tailored to task complexity, student experience, and learning objectives.
- Limitations: The study’s initial scope leaves efficacy measures such as learning outcomes and comparisons with classrooms without CodeHelp for future work.The authors characterize the current evidence as an initial exploration despite positive student ratings and instructor impressions.
- Future Work: Future studies should include larger, more diverse student populations and settings such as distance learning or self-paced programming courses.These deployments would evaluate CodeHelp’s flexibility and adaptability across educational contexts.
- Future Work: The authors suggest that CodeHelp’s approach could potentially be adapted to learning in domains beyond computing education.They frame this as an opportunity for further exploration rather than an established result.