Will It Teach as Intended? How Teachers Configure Educational AI Chatbots

1 Introduction

• Teachers need classroom chatbots to account for instructional objectives, student characteristics, curricular constraints, and expected learner interactions—not just subject matter.Purpose-specific agents may offer reusable support across learning activities, but teachers must also define the bot’s role, communication, and behavioral boundaries.Source: §1 · p. 2
• The authoring environment exposes purpose, character and personality, communication tone, and rules, plus adjustable traits, to shape instructional roles and behavioral constraints.Traits include confidence, transparency, formality, and assertiveness.Source: §1 · p. 2
• The study investigates how teachers envision chatbot roles, operationalize instructional goals, and align those goals with configurations and generated responses.The workshops involved 27 middle school teachers; interviews were analyzed alongside configuration and interaction logs.Source: §1 · p. 2
• The paper contributes an empirical account of teachers’ pedagogical intentions and examines how alignment or misalignment can emerge across goals, authored configurations, and chatbot behavior.The findings inform authoring-tool design to help teachers express, inspect, and assess their intentions.Source: §1 · p. 2

2.1 Generative AI and Chatbots in K–12 Teaching

• K–12 chatbots can provide immediate, adaptive support, but hallucinations, cognitive bypassing, privacy concerns, and curricular-fit gaps complicate classroom integration.Teachers’ needs also vary with subject, instructional goals, and students’ developmental needs; generic chatbots may lack the contextual boundaries required for specific pedagogical roles.Source: §2.1 · pp. 2–3

2.2 Teacher Agency and Pedagogical Control over Classroom AI

• Teacher agency requires authority to configure, adjust, or override chatbot behavior so responses remain aligned with pedagogical goals and classroom norms.Authoring mechanisms must translate teachers’ professional judgment into operational system parameters.Source: §2.2 · p. 3

2.3 Teacher Customization and Authoring of LLM-Based Chatbots

• Structured authoring controls can make customization more manageable than open-ended prompt engineering and let teachers specify behavioral constraints and scaffolds that preserve student agency.Prior work describes progressive guidance, Socratic questioning, and timed hints as ways to support productive struggle rather than provide immediate solutions.Source: §2.3 · pp. 3–4

2.4 Aligning Pedagogical Intent, AI Configuration, and Generated Behavior

• Probabilistic LLM behavior can drift from configured intent, including by over-cooperating or reverting to generic assistant behavior under conversational pressure.Empirical measurement of alignment during direct interaction testing remains limited; this study traces intentions through configurations to generated responses.Source: §2.4 · p. 4

3 The chatbot Platform

• The web-based platform lets teachers create and customize classroom chatbots without programming expertise.Teachers can test bots by submitting prompts and observing generated responses.Source: §3 · p. 4

3.1 Configuring Pedagogical Intentions

• Purpose captures instructional goals, intended role, and content focus; Rules captures behavioral expectations, pedagogical strategies, constraints, and other interaction instructions.Both fields are open-ended, so teachers can describe intentions in their own language; teachers may also select an underlying model and adjust behavioral traits.Source: §3.1 · p. 4

3.1.1 Testing and Iterative Refinement.

• An interactive testing environment lets teachers simulate student interactions, inspect responses, and revise configurations; Fact Check and Compare Models provide additional review tools.Fact Check sends a response to a separate model for review, while Compare Models presents two models’ responses side by side.Source: §3.1.1 · p. 5

4 Research Methods and Analysis

• For 27 teachers, the workshops combined chatbot exploration and testing with focus-group interviews, producing configuration and interaction logs alongside transcripts.RQ1 primarily used interviews, RQ2 combined interviews and configurations, and RQ3 examined configurations, testing messages, responses, and relevant interview data.Source: §4 · p. 5

4.1 Participants and Background Measures

• Participants represented varied disciplines and two workshop sites; prior computational-thinking and AI familiarity measures were used descriptively.The study included 20 teachers at Site 1 and seven at Site 2; surveys were reviewed for face and content validity.Source: §4.1 · p. 5

4.2 Log-Based Alignment Analysis

• Linked log records paired each human message and AI response with the associated configuration; the rubric assessed responsiveness, purpose, rules, and persona alignment.Four-point ratings were converted to pass/fail for bot-level analysis, with scores 3–4 passing and 1–2 failing.Source: §4.2 · p. 5

4.3 Human–AI Agreement and Rubric Calibration

• After rubric calibration, overall human–AI agreement rose from κw = 0.806 to 0.883, and exact agreement rose from 76.7% to 82.8%.The largest criterion-level gain was Persona alignment, from κw = 0.604 to 0.832; the refined procedure was then applied to the full interaction-log dataset.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Thematic Analysis of Logs

• Two analysts used a hybrid deductive–inductive codebook to analyze Purpose and Rules configurations, resolving disagreements through discussion and consensus.The initial codebook drew on prior chatbot customization categories and expanded when configurations expressed dimensions not adequately captured.Source: §4.4 · pp. 6–7, Table 3 · p. 7

4.5 Focus-Group Interview Analysis

• Four analysts used hybrid thematic analysis of focus-group transcripts, comparing paired analyses and resolving differences through discussion and consensus.The process retained research-question grounding while allowing unanticipated patterns in teachers’ experiences to emerge.Source: §4.5 · p. 7

5.1 RQ1: Envisioned Roles, Affordances, and Considerations for Classroom Use

• Teachers envisioned adaptive chatbots that tailor scaffolding to learner needs, help students organize ideas, and extend assistance when teachers cannot provide individual support.They also saw chatbots as potentially redistributing planning and feedback work.Source: §5.1.1 · p. 8, §5.1.2 · pp. 8–9

5.1.3 Theme 3 : Balancing Teacher Control, Student Agency, and Trustworthy AI Use.

• Teachers wanted scaffolding that preserves student reasoning and authorship, with boundaries on answer-giving, content, and information reliability.Some described adjusting support from tutoring toward coaching as students gained experience.Source: §5.1.3 · p. 9

5.1.4 Theme 4 : Negotiating Classroom, Institutional, and Practical Fit.

• Adoption depended on institutional approval and privacy review, reliable technology access, classroom time, teacher readiness, and fit with existing routines.Teachers also wanted chatbot tools integrated into systems such as Schoology.Source: §5.1.4 · p. 9

5.2.1 Theme 5: Translating Pedagogical Intent into Chatbot Con-

• Purpose mainly captured instructional goals and content focus, while Rules more often specified pedagogical behavior, guardrails, and learner-specific adaptations.Task/Objective appeared in Purpose for 21 teachers and in Rules for 19.Source: §5.2.1 · p. 9

5.2.2 Theme 6: Configuration as an Interpretive and Iterative Author-

• Configuration required interpretation and iteration: teachers requested clearer guidance, adapted to character limits, and tested whether boundaries worked as intended.Some teachers proposed examples and tooltips to explain configuration fields.Source: §5.2.2 · pp. 9–10

5.3.1 Theme 7: Alignment and Gaps between Configured Intentions and Generated Behavior.

• Teachers reported both successful boundary-setting and mismatches, including inconsistent enactment of configured open-ended reasoning and model-dependent answer-giving.One teacher reported that a water-cycle chatbot answered on-topic questions and refused an off-topic question.Source: §5.3.1 · p. 10

5.3.2 Log-Based Evaluation of Configuration Alignment in Chat-

• 88.9% responsiveness (Avg = 3.67) and 81.5% persona alignment (Avg = 3.48) exceeded rules adherence (70.4%, Avg = 3.26) and purpose alignment (59.3%, Avg = 3.00).The bot-level analysis retained 27 unique chatbots.Source: Table 5 · p. 10, §5.3.2 · p. 10

5.4 Theme 8 : Exploratory Findings: Making Student–AI Activity Visible and Actionable for Teachers

• Teachers wanted dashboards to show students’ learning progress, identify struggling students, and support timely intervention through high-priority flags and summaries, while avoiding excessive monitoring and information overload.Personalization appeared in Purpose for 9 teachers and Rules for 13; age- or grade-level targeting was expressed through Purpose by six of nine teachers.Source: §5.4 · pp. 10–11, §5.6 · p. 12

6.1 Contribution and Consistency with Prior Work

• Extending Hou et al. [21], the analysis maps customization intentions to interface fields and traces them from configuration to generated chatbot behavior.Task/Objective appeared mainly in Purpose, whereas Pedagogical Strategy and Course Material appeared more often in Rules and Guidelines.Source: §6.1 · p. 12

6.2 Teacher Control Through Configurable AI Authoring

• Purpose mainly carried instructional goals and content, while Rules more often encoded pedagogical behavior, guardrails, and learner-specific adaptations.Teachers also requested clearer labels, examples, templates, and in-interface guidance, and refined configurations through testing.Source: §6.2 · p. 12

6.3 Bridging Pedagogical Intent and AI Behavior

• Conversationally appropriate responses did not necessarily enact pedagogical intent; alignment was stronger for responsiveness and persona than for purpose and rules, with Purpose weakest.The pedagogical Gulf of Execution concerns expressing a goal through configuration; the Gulf of Evaluation concerns judging whether generated behavior reflects that goal.Source: §6.3 · pp. 12–13

6.4 Implications for Classroom AI Design

• Teachers envisioned chatbots that provide differentiated task support and extend assistance when teachers are unavailable, while preserving student thinking and authorship within boundaries.They emphasized teacher-defined instructional boundaries and the need to examine how intent carries from goals to configuration and generated behavior.Source: §6.4 · p. 13

• Alignment was stronger for responsiveness and persona than for teacher-defined purpose and rules, motivating evaluation of instructional fidelity alongside response quality.Source: §7 · pp. 13–14
• Authoring tools should help teachers translate goals into settings and test, diagnose, and refine chatbot behavior before classroom deployment, preserving teacher agency in K–12.Source: §7 · p. 14
• Future systems could flag configuration elements weakly reflected in responses, recommend revisions, and compare how language models enact the same configuration.Source: §7 · p. 14
• Behavior was evaluated mainly through teachers’ testing during short workshops, not sustained student use in authentic classrooms.Observed alignment may not capture failures or adaptations during longer, more varied, or unexpected student interactions.Source: §7 · p. 14

• The survey asked about computational-thinking teaching practice, including improvement, knowledge of effective teaching steps, and confidence in explaining and teaching concepts.Source: Pre-Survey · p. 17
• Items asked teachers about confidence answering computational-thinking questions, helping students understand concepts, welcoming questions, and increasing student interest.Source: Pre-Survey · p. 17
• A separate section asked respondents about their familiarity with and experience using artificial intelligence.Source: Pre-Survey · p. 17