PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Pedagogical Behavior in AI Tutoring Interactions

Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin

arXiv:2608.22993v1cs.CLcs.HC

TL;DR

Students increasingly use LLMs as tutors, but the assistance they receive in authentic interactions is not well characterized. This paper develops and validates a five-level scaffolding scale and applies it to 14,637 responses from 203 students. More than 95% of responses are Explaining or Solving, while scaffolding relates to subsequent dialogue behavior but adds little predictive information about three exam outcomes beyond prior achievement and dialogue behavior.

  • Problem

    The assistance students actually receive across authentic LLM tutoring interactions is less well characterized than student behavior or deliberately designed tutoring behavior.

  • Method

    The study develops a five-level scale for direct assistance, validates it against human annotations, and applies it to 14,637 responses from 203 students.

  • Results

    More than 95% of responses are Explaining or Solving; scaffolding is associated with subsequent conversational behavior but adds little predictive information about three exams beyond prior achievement and dialogue behavior.

  • Takeaways & Limitations

    The findings establish an empirical baseline and measurement framework for evaluating whether alternative tutoring designs change the assistance students receive.

  • Takeaways & Limitations

    The observed distribution characterizes one introductory AI course and one model configuration rather than general-purpose LLMs universally.

Abstract

from arXiv · show

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

1 Introduction

The paper addresses limited knowledge about how directly LLMs assist students in authentic tutoring interactions by defining scaffolding level as a measurable dimension. It develops and applies a five-level scale, finding that responses are concentrated at high assistance levels and relate to subsequent dialogue behavior and exam-performance prediction.

  • LLM tutoring responses vary in how directly they help students complete learning tasks, but assistance levels in authentic interactions remain insufficiently characterized.Students may request clarification, explanations, hints, or complete solutions.
  • Scaffolding level captures the degree of direct assistance in a tutoring response, reflecting how much responsibility remains with the learner.The concept connects to instructional-support theories emphasizing productive effort and gradual transfer of responsibility.
  • The study develops a five-level scaffolding scale, validates it against human annotations, and applies it to 14,637 responses from 203 students.The responses come from the StudyChat dataset collected in an introductory university AI course without additional pedagogical instructions.
  • More than 95% of responses are classified as Explaining or Solving, indicating that the assistant predominantly provides high levels of assistance.The study also examines whether scaffolding varies across requests and assignments.
  • Scaffolding level is associated with students’ subsequent conversational behavior but adds little predictive information about performance on three subsequent exams.The exam-performance analysis considers information beyond prior achievement and observed dialogue behavior.

2 Related Work

Prior research has increasingly characterized student behavior and designed pedagogical LLM tutors, while this paper focuses on the assistance contained in LLM responses. It positions the observed behavior of a general-purpose assistant as a baseline for evaluating pedagogical designs.

  • Research on LLM-assisted learning has largely examined what students ask, how they use LLMs, and how usage relates to course outcomes.This work provides an increasingly detailed account of the student side of LLM-assisted learning.
  • Dialogue-act schemes characterize an utterance’s communicative function, whereas scaffolding measures how much assistance the response provides.Scaffolding ranges from little substantive assistance to directly solving the student’s task.
  • Pedagogically designed systems such as SocraticLM and LearnLM explicitly shape tutoring behavior to engage reasoning or specify teaching behaviors.These approaches use tutoring paradigms, instruction following, or pedagogical examples to influence model behavior.
  • The paper characterizes assistance emerging from a general-purpose LLM without a tutoring strategy, providing a reference point for evaluating alternative designs.The baseline is logically prior to assessing whether pedagogical designs change the assistance students receive.

3 Data

The study uses StudyChat data from an introductory university AI course, where students interacted with a GPT-4o-mini interface without explicit teaching instructions. The dataset includes assignment interactions, dialogue-act annotations, and assessment information, with exams completed without LLM access.

  • The StudyChat dataset was collected at a U.S. university introductory AI course across Fall 2024 and Spring 2025.
  • Assignments allow LLM use, whereas exams do not, separating supported coursework interactions from assessment conditions.
  • Students interacted with GPT-4o-mini through a ChatGPT-like interface using only the system prompt “You are a helpful assistant.”No additional instruction specified how the model should teach or how much assistance it should provide.
  • The dataset contains 16,851 turns from 203 students across 2,214 conversations and seven programming assignments in three course modules.
  • Student utterances include broad dialogue-act labels validated against human annotations, with κ = .74 agreement at the broad level.Assessment data are available for 181 students, and students could use the LLM during assignments but not during the three exams.

4 Methods

The methods pair each LLM response with its preceding student context and the next student dialogue act, then classify responses on a five-level assistance scale. Analyses describe scaffolding distributions, test associations with follow-up behavior, and assess incremental prediction of exam performance beyond prior achievement and dialogue behavior.

  • 4.1 Paired Turn Construction: 14,637 LLM responses followed by student turns form paired interaction units linking prompts, dialogue acts, responses, and subsequent student behavior.Of these paired turns, 12,692 come from students with assessment data.
  • 4.2 Measuring Scaffolding Level: The five-level scale ranges from Minimal and Prompting through Hinting and Explaining to Solving, ordered by directness of assistance.Solving provides a complete task-specific solution, while Explaining gives an approach the student must still apply.
  • 4.2 Measuring Scaffolding Level: GPT-4o-mini classifies responses at temperature 0 using category definitions, the target response, and preceding context.The study summarizes levels across dialogue acts and assignments and uses a Kruskal–Wallis test for assignment-level differences.
  • 4.2 Measuring Scaffolding Level: Human validation uses 300 stratified responses, yielding κ = .901 human agreement and 75.3% classifier accuracy on the held-out test split.Weighted F1 is .751 on n = 150 held-out responses.
  • 4.3 Scaffolding and Student Follow-up: For each scaffolding level, the study computes the conditional distribution of the next student dialogue act and tests independence with a chi-squared test.The analysis is repeated within broad current dialogue-act categories to assess whether associations persist beyond request-type differences.
  • 4.4 Scaffolding and Assessment Performance: The assessment analysis compares a baseline using prior performance and dialogue-act counts with a full model that additionally includes mean scaffolding level.The comparison measures scaffolding’s incremental predictive information for normalized scores on exams e1–e3.

5 Results

LLM responses were concentrated at high assistance levels across request types and assignments, and scaffolding level was associated with students’ subsequent dialogue acts. However, it added little predictive information about exam performance beyond prior achievement and dialogue behavior.

  • Scaffolding distribution: More than 95% of responses were Explaining or Solving, with a mean scaffolding level of L̄ = 3.43 (SD = 0.84).More than half were complete, task-specific solutions, while 40.3% provided detailed explanations; Prompting, Hinting, and Minimal responses together accounted for fewer than 5%.
  • Variation across requests: Writing requests had the highest mean scaffolding level at L̄ = 3.72, while off-topic turns had the lowest at L̄ = 0.87.Editing, provide-context, conceptual, verification, and contextual requests had means of 3.57, 3.53, 3.38, 3.34, and 3.27, respectively.
  • Variation across assignments: Scaffolding varied across assignments (Kruskal–Wallis H = 336.9, p < .001), but high-assistance responses remained dominant.The pandas assignment had a mean of 3.58 versus 2.91 for the search-algorithm assignment.
  • Subsequent dialogue: Students’ next dialogue acts differed by scaffolding level (χ2 = 1,092.6, df = 28, p < .001, V = .137).After Solving, another writing request was most common (29.7%); after Explaining, conceptual questions were most common (40.5%); after Prompting, providing context was most common (34.0%).
  • Subsequent dialogue: The association persisted within all four broad student dialogue-act categories and after combining Explaining and Solving, although V decreased from .137 to .116.This indicates the pattern was not explained solely by broad differences in the preceding request type.
  • Exam prediction: Adding mean scaffolding increased explained exam-score variance by only 1–2 percentage points, and none of three comparisons was statistically significant.Baseline models explained approximately 12–30% of exam-score variance; alternative specifications produced the same overall pattern.

6 Discussion

A general-purpose assistant without explicit pedagogical prompting overwhelmingly provides substantial assistance, and scaffolding level is reflected in subsequent interaction but adds little predictive information about later exam performance.

  • Default LLM Assistance Is Highly Direct: More than 95% of responses are classified as either Explaining or Solving, while Prompting and Hinting are rare.This distribution establishes a behavioral baseline rather than identifying an optimal assistance level.
  • Scaffolding Is Reflected in Subsequent Interaction: Scaffolding level is systematically associated with students’ subsequent conversational behavior.Explaining responses are followed relatively often by conceptual questions, whereas Solving responses are followed more often by additional writing requests.
  • Scaffolding Is Reflected in Subsequent Interaction: These interaction patterns persist within broad categories of initial student requests and after collapsing the L3/L4 distinction.
  • Interpretation: The observational design does not identify whether scaffolding itself causes differences in subsequent student behavior.Prompt content, task difficulty, conversation history, and student characteristics may influence both the response and the behavior that follows.
  • Interactional Relevance Does Not Imply Later Performance Differences: Scaffolding adds little predictive information about subsequent exam performance beyond prior achievement and observed dialogue behavior.This distinguishes immediate interactional associations from later academic outcomes.
  • Implications for Evaluating LLM Tutors: Evaluations should distinguish whether an intervention changes received assistance from whether that change improves learning.The scaffolding measure addresses the behavioral question, whereas educational effects remain a separate empirical question.

Limitations

The study’s conclusions are bounded by its setting, imperfect automated classification, observational follow-up analyses, and limited variation and sample size in assessment analyses.

  • Scope: The data come from one introductory AI course and one model configuration, so the observed distribution characterizes this setting rather than LLMs universally.“Default” means use without explicit pedagogical prompting.
  • Measurement: Automated scaffolding classification is imperfect despite high human agreement on the underlying scale.Most errors occur at the L3/L4 boundary, and fine-grained estimates for L0–L2 should be interpreted cautiously.
  • Observational Analyses: The follow-up analyses are observational and do not account for all differences in prompt content, task difficulty, student characteristics, or conversation history.Turn-level chi-squared analyses also do not explicitly model within-student dependence despite repeated student turns.
  • Assessment Analysis: Assessment analysis evaluates incremental prediction rather than the causal effect of scaffolding on learning.The assessment sample is modest, and responses are concentrated at L3 and L4, limiting variation for detecting performance differences.

Ethics Statement

The study uses a publicly released, de-identified dataset under informed consent and IRB approval, with author-run validation and documented classification procedures.

  • Data Governance: The StudyChat dataset was collected under IRB approval with informed consent, and personally identifiable information was removed before release.The study’s use is described as consistent with the dataset’s stated research purpose.
  • Human Validation: Two authors independently annotated all 300 validation responses while blind to each other’s labels and classifier outputs.Disagreements were resolved through discussion to form the human reference set.
  • Scaffolding Classification: The codebook classifies responses by assistance amount and the task completion left to the student.L3 requires student adaptation or implementation, whereas L4 applies an approach directly to the student’s specific task.
  • Human Validation: Human-human agreement was high, with Cohen’s κ = .901 and 95.0% raw agreement.Agreement remained similarly high at the L3/L4 boundary, with κ = .894.
  • Automated Classification: Against human reference labels, the GPT-4o-mini classifier achieved 75.3% accuracy and weighted F1 = .751 on the held-out test split (n = 150).Across all 300 annotated responses, accuracy was 75.7%, weighted F1 = .759, and κ = .552.
  • Validation Scope: The validation sample oversampled rare response types, so its marginal label frequencies should not estimate the population scaffolding distribution.

A.3 Classification Error Structure

Classification errors are concentrated at the boundary between Explaining and Solving, so the analysis tests whether findings persist when those levels are collapsed. The classifier’s broader high-assistance pattern remains stable, although sparse cells make chi-squared p-values approximate.

  • Classification errors: 57 of 73 disagreements were human-labeled L4 responses classified as L3, compared with 9 L3 responses classified as L4.The classifier therefore more often shifts responses from Solving to Explaining than in the reverse direction.
  • Boundary robustness: The L3/L4 error pattern affects their precise distinction but does not alter the broader concentration of responses at high assistance.This motivates treating the collapsed specification as a robustness check rather than a replacement for the five-level analysis.
  • Boundary robustness: Collapsing Explaining and Solving preserves a significant association between scaffolding and the next student dialogue act.The association remains significant with χ2 = 592.2, df = 21, p < .001, while Cramér’s V decreases from .137 to .116.
  • Interpretive caveat: Sparse cells involving L2 and uncommon student dialogue acts make the chi-squared p-values approximate.Observed transition patterns and effect sizes therefore warrant greater emphasis than the p-values alone.
  • Transition patterns: Within four common request categories, scaffolding remains significantly associated with the next student dialogue act.For writing requests, provide-context turns follow 16.4% of L4 responses versus 7.3% of L3 responses.

B.4 Provide-Context Responses

Provide-context responses vary by scaffolding level overall and within conceptual questions, but the relationship is not monotonic. Because the measure does not establish independent student effort, it is interpreted descriptively.

  • Measure interpretation: Provide-context turns include assignment information, error messages, code, or other task-relevant context supplied by students.The label identifies additional context but not whether it is newly produced or adapted from an earlier LLM response.
  • Overall and conceptual-question rates: Across all interactions, provide-context rates are 14.0% after L0, 34.0% after L1, 23.3% after L2, 13.6% after L3, and 20.6% after L4.Within conceptual questions, the corresponding rates are 4.8%, 12.5%, 11.8%, 8.2%, and 12.6%.
  • Overall and conceptual-question rates: Within conceptual questions, provide-context turns occur at similar rates after Prompting and Solving responses.The passage characterizes this relationship as non-monotonic.
  • Measure interpretation: The provide-context measure is treated descriptively rather than as a measure of independent student effort.The data do not distinguish newly produced work from material adapted or copied from an earlier LLM response.
  • Outcome context: Exam scores, rather than assignment scores, are used as the main RQ3 outcomes because assignments average approximately .95 and show substantial negative skew.Exam scores show greater variation and were obtained without concurrent LLM access.

C Alternative Exam-Prediction Specifications

The study tests whether exam prediction improves when scaffolding is represented by counts of Solving or low-assistance responses instead of mean scaffolding. Neither alternative adds meaningful predictive information.

  • Alternative predictors: The alternative specifications replace mean scaffolding with counts of L4 Solving responses or L0–L2 low-assistance responses before each exam.All other predictors remain unchanged.
  • Alternative predictors: Neither the Solving-response count nor the low-assistance-response count reaches conventional statistical significance for any exam.Both alternative representations add little explained variance, consistent with the main analysis using mean scaffolding level.
Loading 2608.22993v1…