Source-linked AI summary
Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming
Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, Amanpreet Kapoor
TL;DR
Guardrailed AI TAs may create friction when help feels overly restrictive or poorly contextualized, and prior evaluations often did not isolate configuration effects. This study randomized 132 introductory Python students across four combinations of guidance style and context. Socratic guidance with full context received the least favorable task-completion support ratings, underscoring the need to balance guidance and context.
Problem
Overly restrictive or poorly contextualized guardrails may prompt students to disengage or turn to external LLMs, while the effects of different AI TA configurations on student experience remain a design question.
Source: §1 · p. 1Method
A randomized experiment with 132 introductory Python students compared four AI TAs varying Socratic versus Direct guidance and full versus no context, measuring perceptions, behavior, and post-task comprehension.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
AI TA effectiveness depends on balancing guidance and context so the support feels useful and keeps students engaged with course-approved tools.
Source: §5 · p. 7, §6 · p. 7The study sampled one introductory course at a large research university and assessed immediate comprehension for only one task, not long-term retention.
Source: §5 · p. 7
Abstract
from arXiv · showhide
AI teaching assistants (AI TAs) backed by large language models (LLMs) and pedagogical guardrails are increasingly being integrated into programming courses, providing students with scalable access to hints, conceptual explanations, and code-level feedback. However, guardrails may also create friction. If students feel that the support provided is overly restrictive or poorly contextualized to their current progress, they may bypass approved tools for general-purpose LLMs. To investigate how AI TA design affects students' learning experiences, we conducted a randomized controlled trial with 132 students in an introductory programming course. Students completed three tasks related to code-writing and debugging and were randomly assigned to one of four AI TAs varied across two dimensions: pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (no context vs. full context of the problem and student solution). We examined students' perceptions, interaction behaviors, and evidence of post-task comprehension. Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion. Descriptively, this condition also showed the highest observed interaction stress, the highest rate of external LLM use, and the lowest proportion of post-task explanations demonstrating full comprehension, though these differences were not statistically significant. These findings suggest that guardrailed AI TAs are not automatically better for learning. Instead, their effectiveness depends on how pedagogical guidance and contextual awareness are balanced in ways that students experience as useful, supportive, and worth continuing to use.
1 Introduction
Guardrailed AI TAs aim to support learning without completing students’ work, but overly restrictive or poorly contextualized help may prompt disengagement or use of external LLMs. This study tests how guidance style and context awareness shape students’ experiences across four AI TA configurations.
1 Introduction
- The design challenge is to balance pedagogical soundness with support students find useful and worth continuing to use.Students who migrate to external systems may reduce educators’ access to learning-related data and become more susceptible to over-reliance on generated solutions.Source: §1 · p. 1
- The study addresses a gap in understanding how design choices within the same AI TA shape students’ perceptions, behaviors, and learning-related outcomes.Prior evaluations generally treated fixed AI TA systems as complete interventions.Source: §1 · p. 1
- A 2×2 randomized experiment with 132 introductory Python students compared Socratic versus Direct guidance and full versus no problem context.Students completed one code-writing and two debugging tasks; outcomes included perceptions, interaction behaviors, and post-task comprehension.Source: §1 · pp. 1–2
- Socratic guidance with full context was rated least favorably for supporting task completion, showing that pedagogically motivated designs are not automatically experienced as more helpful.The study frames its findings as guidance for deploying AI TAs effectively in classroom settings.Source: §1 · p. 2
2 Related Work
Related work spans LLMs’ pedagogical capabilities and students’ reception of scaffolded systems. Prior findings motivate examining how guidance style and available context jointly shape AI TA experiences.
2 Related Work
- Research on LLMs in computing education includes work on pedagogical capabilities and limitations, as well as student reception of scaffolded, pedagogically grounded systems.Examples of the latter include CodeHelp [19] and CodeAid [15].Source: §2 · p. 2
- Prior Socratic-guidance findings are mixed: slower-paced interactions produced greater learning gains for some students, while students have also rated Socratic hints as less helpful and more time-consuming.The paper extends this line of work to guidance style, context availability, perceptions, interaction behavior, and immediate post-task comprehension.Source: §2 · p. 2
- Context may include the problem statement, current code, and interaction history; prior work generally evaluated fixed configurations rather than isolating configuration effects.The study compares guidance style and context availability experimentally.Source: §2 · p. 2
3 Methods1
The study used a 2×2 randomized design to compare Socratic versus Direct guidance and full versus no context in an introductory Python course. It measured student perceptions, interaction behavior, and immediate comprehension of a debugging task.
3.1 Study Design
- A between-subjects 2×2 randomized design varied guidance style and context awareness, operationalizing learning experiences through perceptions, interaction behaviors, and task comprehension.Full Context included the problem statement and current solution; No Context provided chat history only.Source: §3.1 · p. 2
3.2 Coding Tool and AI TA design
- The web platform placed the problem statement, submission history, and AI assistant beside an embedded editor, where students could run tests or submit code.The four AI TAs used the same interface and differed in system prompts and contextual information; Direct gave brief responses without full solutions, while Socratic used targeted questions.Source: §3.2 · pp. 2–3, Figure 1 · p. 3
3.3 Participants
- The final analytic dataset contained 132 consenting students recruited from a large public university’s introductory Python course.Students were excluded for non-consent, incomplete responses, not using the chatbot, or missing or inconsistent logs.Source: §3.3 · p. 3
3.4 Study Tasks
- Students completed one code-writing task and two debugging tasks; the debugging set included a matrix-summing function with a boundary-checking bug.Function names were anonymized to reduce the chatbot’s ability to infer intended solutions; 128 students completed all three tasks and passed all test cases.Source: §3.4 · pp. 3–4
3.5 Data Collection and Analysis
- Post-survey items and tool logs measured perceptions, external LLM use, message volume and length, and comprehension of the matrix-debugging solution.Quantitative comparisons used Kruskal–Wallis tests; open-ended responses were inductively coded and reviewed by two researchers.Source: §3.5 · p. 4
4 Results
Across four conditions, Socratic guidance with full context received the lowest task-completion support ratings. Behavioral and comprehension patterns varied descriptively, while some condition differences were not statistically significant.
4.1 Perception Measures
- Mean task-completion support was 3.53 for Socratic + Full Context, versus 4.27 for Direct + Full Context, 4.16 for Direct + No Context, and 4.12 for Socratic + No Context.The condition effect was significant (χ2(3) = 12.14, p = .007); differences in perceived task-comprehension support, correct information, and stressful interaction were not significant.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
4.2 Behavior
- External LLM use was reported by 15% of students overall and was highest in Socratic + Full Context at 23%, though condition differences were not significant.Students most often cited inadequate guidance; other reasons included going in circles and forgetting context.Source: §4.2.1 · p. 5
4.2.2 Interaction volume.
- Socratic + No Context students sent 11.1 queries per problem, compared with 5.1 for Socratic + Full Context, 4.1 for Direct + Full Context, and 4.7 for Direct + No Context.The query-count difference across conditions was significant (χ2(3) = 13.28, p = .004).Source: §4.2.2 · p. 5
- Direct + No Context students wrote the longest messages, averaging 126 characters, followed by Direct + Full Context at 81, Socratic + No Context at 69, and Socratic + Full Context at 54.Average message length differed significantly across conditions (χ2(3) = 29.96, p < .001).Source: §4.2.2 · p. 5
4.3 Task Comprehension Analysis
- The post-task assessment asked students to re-identify and explain the matrix-debugging error, with full comprehension requiring identification of the out-of-bounds bug and a correct fix.Responses were categorized as full, partial/impure, minimal, or ungaugable comprehension.Source: §4.3 · pp. 5–6, Figure 3 · p. 6
- Among 128 non-blank comprehension responses, 57% showed full comprehension; by condition, the proportion ranged from 48% in Socratic + Full Context to 67% in Direct + Full Context.The Socratic + Full Context group also had the highest minimal-comprehension proportion (14%); patterns were descriptive.Source: Figure 4 · p. 6, §4.3 · p. 6
5 Discussion
The discussion shows that perceived usefulness depends on how guidance style and context are paired: more context reduced query-writing effort but did not ensure stronger perceived support or comprehension. The authors frame AI TA configuration as a task-sensitive balance rather than assuming more scaffolding or context is always better.
Role of Pedagogical Guidance Style
- Socratic + Full Context was rated least favorably for task completion and had the lowest proportion of full-comprehension post-task explanations.The authors note that these results caution against assuming that a pedagogically motivated approach will feel useful to students focused on completing a programming task.Source: §5 · p. 6
Role of Contextual Awareness
- Students in No Context conditions tended to write longer queries, while Socratic + Full Context produced the shortest messages on average.The authors suggest students without context supplied more detail themselves, while full context reduced query-formulation effort without guaranteeing useful-feeling support or stronger understanding.Source: §5 · p. 6
Interaction Between Pedagogical Guidance Style and Context Awareness
- With Direct guidance, task-completion ratings were similar across context levels, whereas Full Context reduced perceived support when guidance was Socratic.The authors conclude that context awareness was not uniformly beneficial and its effect depended on pedagogical style.Source: §5 · p. 6
- Socratic + No Context produced the highest query volume, while Direct + Full Context produced the lowest.The discussion suggests that Socratic guidance without task knowledge can require more back-and-forth, whereas direct guidance with full context can help students get help quickly.Source: §5 · p. 7
Discussion takeaway
- The authors argue that guardrails can become roadblocks when guidance and context are poorly balanced, and recommend configurable support fitted to task and student needs.They propose future work on adaptive AI TAs that shift between Socratic and more direct guidance based on student progress, confusion, or help-seeking behavior.Source: §5 · p. 7
Limitations
- The study was conducted in one introductory programming course at a large research university, and measured immediate comprehension for only one task rather than retention.The authors also note that qualitative coding involves subjectivity, which they sought to mitigate through a codebook, transparent reporting, and a detailed flowchart.Source: §5 · p. 7
6 Conclusion
The conclusion reports that students did not experience all guardrailed AI TA designs as helpful, with the Socratic + Full Context condition showing the least favorable pattern. It argues that AI TA effectiveness depends on balancing guidance style and context in ways that support learning and keep students engaged with course-approved tools.
6 Conclusion
- Socratic + Full Context had significantly lower perceived support for task completion and less favorable descriptive patterns on several other measures.The other patterns included perceived comprehension support, interaction stress, external LLM use, and post-task comprehension.Source: §6 · p. 7
- The experiment compared four configurations of the same AI TA by varying guidance style and context awareness.The conclusion contrasts this approach with evaluating a single AI TA system as a complete intervention.Source: §6 · p. 7
- The authors conclude that effective AI TA design requires balancing pedagogical guidance with contextual awareness so support remains useful and students stay engaged with course-approved tools.They caution that pedagogically motivated designs can become roadblocks if students experience them as too rigid or insufficiently useful.Source: §6 · p. 7