PaperlayerView exact revision on arXiv

Source-linked AI summary

TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education

David Barron, Xiaohang Tang, Rezky Dwisantika, Minsun Kim, David H. Smith, Jiaming Cui, Yan Chen

arXiv:2608.26184v1cs.AIcs.HC

TL;DR

AI tutors lack the behavioral context human tutors use to adapt support, because they typically see learners’ questions rather than the preceding process. TutorTrace turns real-time IDE telemetry into a dataset and behavioral abstraction pipeline, then derives a three-window taxonomy and evaluates downstream utility. Across 480 learners, it reports classroom-associated behavioral changes and held-out prediction performance, while remaining limited by single-institution data and a nonrandomized preliminary comparison.

  • Problem

    AI tutoring systems typically respond to learners’ explicit information while missing the behavioral context preceding help-seeking.

  • Method

    TutorTrace combines classroom-scale IDE telemetry with an automated real-time classifier, observable metrics, behavioral sequences, and a three-window taxonomy.

  • Results

    TutorTrace captured approximately 180K telemetry events, 13,633 behavioral segments, and 27 metrics; behavior-aware prompting was associated with a reduction from 50.0% to 20.7%, with AUROC=.726 and AUROC=.717 on two held-out tasks.

  • Takeaways & Limitations

    The released dataset, taxonomy, classifier, and code make learner behavior surrounding AI interactions programmatically accessible for adaptive AI tutoring research and design.

  • Takeaways & Limitations

    The data come from short introductory programming tasks at one institution, and the preliminary evaluation used nonrandomized morning and afternoon deployments.

Abstract

from arXiv · show

AI programming tutors provide scalable support, yet lack the behavioral context human tutors rely on to adapt support to learners' needs. We present TutorTrace, a dataset and behavioral abstraction pipeline that makes learners' behavioral context visible and computable in real time from low-level IDE telemetry. Across four deployments in two introductory Python courses (N=480), TutorTrace captures approximately 180K telemetry events, 13,633 behavioral segments, and 27 continuously computed metrics. From this foundation, we derive a taxonomy of learner activity before the first AI query, between consecutive queries, and across the full session, enabling systems to respond not just to what learners say, but to what they have done leading up to the help-seeking moment. In a preliminary classroom evaluation, behavior-aware prompts were associated with a decrease in intervals between queries with no independent work from 50.0% to 20.7%. As an additional demonstration of downstream utility, we evaluate TutorTrace on two held-out prediction tasks: whether a learner will query within the next 60 seconds (AUROC=.726) and whether an upcoming query reflects guided or dependent help-seeking (AUROC=.717). Together, these findings show how behavioral context can enable adaptive AI tutoring at scale.

1 Introduction

TutorTrace addresses the lack of classroom-scale, fine-grained behavioral context in AI tutoring by transforming IDE telemetry into computable behavioral representations. It introduces a dataset, classifier, taxonomy, and preliminary demonstrations of adaptive-tutoring utility.

  • Motivation: Existing AI tutors typically respond to learners’ questions while missing the behavioral process preceding help-seeking.Human tutors use observable behavior to distinguish learners’ needs, but current datasets and systems rarely provide this context at classroom scale.
  • Dataset and infrastructure: TutorTrace captures fine-grained IDE telemetry from 480 learners across four deployments in two introductory Python courses.The dataset includes approximately 180K events, 13,633 behavioral segments, and 27 observable metrics surrounding 1,386 AI interactions.
  • Dataset and infrastructure: The automated classifier transforms raw IDE telemetry into real-time labeled behavioral sequences, with 78%–87% overall pairwise agreement.It is grounded in expert-developed rules and validated against expert labels and student self-reports.
  • Taxonomy and utility: TutorTrace organizes behavior into ten profiles across three windows: before the first query, between consecutive queries, and across recurring re-querying.These representations characterize behavioral context surrounding AI interactions across the session.
  • Taxonomy and utility: 50.0% to 20.7%: behavior-aware prompting was associated with fewer Passive inter-query windows lacking independent work.The preliminary classroom comparison also found greater observable activity between queries.
  • Taxonomy and utility: AUROC=.726 and AUROC=.717: observable metrics supported prediction of 60-second query imminence and guided versus dependent help-seeking.These were held-out prediction tasks demonstrating downstream predictive utility.

2 Related Work

Related work shows that LLM-assisted programming changes student interaction and raises concerns about over-reliance, while existing interventions mainly use query content. TutorTrace contributes a taxonomy focused on behavioral context preceding AI requests.

  • LLM-assisted programming: LLM-based programming assistants reshape students’ interaction patterns, problem-solving processes, and reliance on AI assistance.This makes behavioral analysis of programming interaction more important and complex.
  • Research gap: Taxonomies specifically characterizing student behavior in LLM-assisted programming contexts remain rare.TutorTrace introduces learner behavioral profiles to make student–AI interaction patterns more visible and interpretable.
  • LLM-assisted programming: Students increasingly rely on AI to complete assignments, raising concerns about over-reliance and reduced independent problem-solving.Related tools use scaffolded hints and guardrails to limit solution delegation.
  • Existing interventions: CodeHelp and CodeAid scaffold requests using structured fields, templates, or sufficiency checks based on query content.These approaches shape what students submit before the request reaches the underlying language model.
  • TutorTrace’s position: TutorTrace makes the behavioral context preceding each query computable, complementing interventions based only on what students say.The system is designed to distinguish learners who submit similar questions but arrive through different behavioral processes.

3 System

TutorTrace is a task-based IDE that continuously captures fine-grained interactions and transforms them into behavioral representations for adaptive LLM tutoring. Its tutor prompt uses escalating scaffolding, from Socratic questions to concrete guidance.

  • TutorTrace provides a fixed-layout programming environment with an editor, terminal, task materials, tests, and an AI chat window.
  • Raw telemetry is translated in real time into behavioral sequences and observable metrics that expose learners’ behavioral context to the tutor.
  • The system records 37 event types across six interface regions, including timestamps, source regions, and payloads.Events are captured continuously and batched to the back end every five seconds.
  • The LLM prompt uses three escalating scaffolding levels: Socratic questioning, conceptual hints, and concrete scaffolding.Concrete scaffolding provides direct guidance with pseudocode and blanks after sustained effort without progress.

4 Behavioral Classification

TutorTrace develops expert-grounded behavioral labels and converts them into automated segmentation and classification rules. Validation compares learner reports, expert annotations, and classifier outputs, while telemetry alone remains insufficient for some semantic distinctions.

  • Codebook Development: Four domain experts labeled learner-session replays using a timeline-based interface and an iteratively refined behavioral codebook.
  • Codebook Development: The finalized codebook defined behaviors, subtypes, and corresponding automated segmentation and classification rules.
  • Behavioral Segmentation: 2.3 seconds was the mean discrepancy in boundary placement across annotators during pilot sessions.
  • Validation: 78% to 87% was the overall raw agreement range across learner self-reports, expert annotations, and the automated classifier.Validation triangulated agreement across three sources using six user-study sessions.
  • Validation: Raw IDE telemetry could not alone distinguish some cases, such as implementing versus debugging or thinking about code versus thinking about error.These cases require semantic understanding to differentiate the behavior.

5 Observable Metrics

Observable metrics complement behavioral sequences by summarizing aggregate activity within time windows. TutorTrace retains 27 metrics after pruning sparse measures and measures outside the instrumented workspace.

  • Observable metrics summarize the amount, frequency, and distribution of learner activity within a specific temporal window.They capture aggregate activity such as code edits, terminal runs, and character deletions.
  • Behavioral sequences preserve temporal progression, whereas observable metrics represent aggregate activity within temporal windows.
  • 35 candidate observable metrics were reduced to 27 after excluding sparse metrics and measures of activity outside the instrumented workspace.The candidates cover code activity, terminal activity, error recovery, time distribution, chat behavior, and interface events.

6 Deployments and Dataset

TutorTrace was deployed across four sessions in two introductory Python courses, producing a foundation dataset of telemetry events, observable metrics, and automatically classified behavioral sequences. Deployment tasks and time limits varied across courses.

  • Deployment Context: Four deployments covered two introductory Python courses with tasks on list manipulation or nested lists and iteration.
  • Deployment Context: Deployments 1 and 2 used a list-indexing, slicing, and in-place-manipulation task with a 15-minute limit.
  • Deployment Context: Deployments 3 and 4 used a nested-lists-and-iteration task with a 10-minute limit because of scheduling constraints.The shorter limit may have contributed to lower completion rates in those sessions.
  • Dataset: 180K raw telemetry events, 27 continuously computed observable metrics, and 13,633 automatically classified behavioral sequences formed the foundation dataset.

7 The TutorTrace Taxonomy

TutorTrace’s taxonomy organizes help-seeking context into three behavioral windows and recurring profiles, linking observable activity with descriptive completion and help-seeking patterns. A preliminary comparison and held-out tasks demonstrate associations with changed activity and predictive utility.

  • Taxonomy structure: The three-window taxonomy covers activity before the first query, between consecutive queries, and across session-wide re-querying patterns.
  • Window 3: Session-Wide Re-querying Patterns: Passive Re-querying, Active Testing, and Untested Editing profiles distinguish session-wide patterns of inactivity, execution, and editing without execution.Active Testing sessions had an 81% completion rate, while Untested Editing sessions had an 88% completion rate.
  • Window 1: Before the First Query: Cold Start, Oriented, and Struggling profiles distinguish no attempt, moderate activity, and extended unsuccessful effort before the first query.Cold Start learners made no edits or terminal runs, while Oriented learners showed moderate activity and Struggling learners spent longer before querying.
  • Window 2: Between Queries: Passive, Iterating, Debugging, and Spinning profiles distinguish no activity, brief iteration, active error recovery, and prolonged activity with limited execution between queries.Passive windows accounted for 46% of valid inter-query windows; Iterating was the most common active profile at 42%.
  • Preliminary evaluation: Passive inter-query windows decreased from 50.0% in the baseline deployment to 20.7% in the intervention deployment, a 29.3-percentage-point difference.The intervention deployment also showed more edits, terminal runs, and activity between queries, while completion differences were not statistically significant.
  • Help-seeking patterns: 96.7% of labeled Cold Start first queries were dependent, compared with 69.1% for Oriented and 71.4% for Struggling learners.
  • Predictive utility: Observable metrics achieved AUROC 0.726 for query-imminence prediction, compared with 0.689 for raw telemetry.The held-out task evaluated whether a learner would initiate an AI interaction within the next 60 seconds.

8 Discussion

The discussion argues that adaptive tutors should interpret queries through preceding behavior and tailor scaffolding to demonstrated effort. TutorTrace supports this direction by making behavioral process context computable for real-time, behaviorally aware systems.

  • Interpret behavior before the query: Queries provide an incomplete account of learner needs when they are interpreted without prior editing, execution, thinking, and error-recovery evidence.First queries from Cold Start learners were 96.7% dependent, compared with 30.9% for Oriented and 28.6% for Struggling learners.
  • Scaffold learner behavior: Learners who repeatedly query without editing or executing may need a concrete-action prompt, while untested editors may need prompts to run and inspect code.
  • Scaffold learner behavior: Behavior-aware prompting was associated with fewer Passive windows and greater observable activity between queries.The preliminary comparison reported a reduction from 50.0% to 20.7% in Passive windows.
  • Match support to effort: The taxonomy supports matching assistance to demonstrated effort, including orientation for Cold Start learners, independence prompts for Passive learners, and direct support after extended unsuccessful effort.
  • Toward behaviorally aware learning systems: Behaviorally aware systems can incorporate process-level context to preserve productive struggle, support learner agency, and provide scaffolding appropriate to recent behavior.

9 Limitations and Future Work

TutorTrace’s evidence is constrained by its setting, exploratory profile selection, labeling approach, model choice, and rule-based segmentation. The authors identify broader validation, cross-model testing, semantic analysis, and learning-outcome studies as future work.

  • Scope and evaluation: The data come from short introductory programming tasks at one institution, limiting generalizability across courses, tasks, populations, and learning environments.
  • Scope and evaluation: The preliminary evaluation used nonrandomized morning-versus-afternoon deployments, so cohort composition, time of day, or unmeasured factors may explain observed differences.
  • Taxonomy and labels: Window 1 and Window 2 completion rates are descriptive because profile selection was partly informed by task-completion differences.
  • Taxonomy and labels: GPT-4o generated guided and dependent help-seeking labels validated against two human raters on a subset, so the labels remain approximations.The raters achieved substantial agreement, but the labels are not direct measurements of learners’ cognitive states.
  • Model scope: Using a single tutor model leaves open whether profiles and intervention effects remain stable across models with different capabilities, styles, or scaffolding strategies.
  • Pipeline limitations: Rule-based segmentation lacks semantic understanding of code and intent, which can misclassify edits when learners work on unrelated features.Future work proposes combining telemetry with lightweight semantic code analysis and testing knowledge-retention and transfer outcomes.

10 Conclusion

TutorTrace makes learner–AI interaction context computable through a dataset, behavioral abstraction pipeline, and three-window taxonomy. Its preliminary evaluation and held-out predictions support using observable behavior alongside queries, while the released resources support further system design.

  • Contribution: TutorTrace is a real-time behavioral abstraction pipeline that makes context surrounding learner–AI interactions computable.
  • Dataset: The dataset spans 480 learners in four introductory Python deployments and captures approximately 180K telemetry events, 13,633 behavioral segments, and 27 observable metrics.
  • Taxonomy: The three-window taxonomy characterizes activity before the first query, between consecutive queries, and across the session.
  • Results: Passive inter-query windows decreased from 50.0% to 20.7% in the preliminary between-deployment evaluation.
  • Results: Observable metrics reached AUROC 0.726 for query imminence and 0.717 for help-seeking type in held-out tasks.
  • Implications: The released dataset, taxonomy, classifier, and code support tutors that respond to what learners do as well as what they say.

E Foundation Dataset Structure

TutorTrace structures learner telemetry into behavioral segments and profiles that can be used to contextualize AI help-seeking. Its intervention prompts distinguish first-time, passive, testing, debugging, and other learner states using activity before and between queries.

  • Dataset and taxonomy: The foundation dataset organizes raw IDE telemetry into abstraction layers, while the taxonomy derives higher-level behavioral profiles from that foundation.Layer D is derived in the TutorTrace Taxonomy; the released foundation comprises Layers A–C.
  • Behavioral abstraction: The auto-segmentation pipeline classifies events, groups consecutive same-category events, splits long-separated code activity, and assigns initial behavior labels.Initial labels map Code to Implementing, Terminal to Testing, ChatInput to Seeking Help, and ChatResponse to Thinking.
  • Behavioral abstraction: Gaps of at least 3 seconds between segments become Thinking segments, making pauses part of the behavioral sequence.The pipeline inserts a Thinking segment spanning each qualifying gap.
  • Behavior-aware intervention: Deployment-time prompts use Cold Start for a learner’s first query and select Passive, Iterating, Debugging, or Spinning for later queries.These rules use the same profile names as the later taxonomy but represent precursor heuristic states.
  • Help-seeking labels: Help-seeking labels classify queries as guided or dependent using the query, code at submission, and preceding chat history.These semantic inputs generate offline labels and are excluded from the prediction models.

I Task Descriptions

The study uses two short Python list-manipulation tasks across four deployments: Playlist in Deployments 1–2 and Grade Book in Deployments 3–4. The held-out analyses evaluate prediction performance and feature contributions across observation windows and feature layers.

  • Playlist task: Students completed a four-step Playlist task within 15 minutes in Deployments 1–2.The task covers replacement, slicing removal, in-place reversal, and stepped slicing.
  • Playlist task: The Playlist task’s expected outputs include “Purple Rain” at the first position, a five-item length, and every other song selected by slicing.The expected stepped-slice output is ["Purple Rain", "Imagine", "Bohemian Rhapsody"].
  • Grade Book task: Students completed a three-step nested-list Grade Book task within 10 minutes in Deployments 3–4.The task asks students to add a student, correct an existing score, and print all scores with a nested loop.
  • Prediction tasks: The held-out evaluation reports AUROC across observation-window sizes and three feature layers for query imminence and help-seeking type.The reported rows use a 30-second window for query imminence and a 15-second window for help-seeking type.
  • Prediction tasks: Additional tables list Random-Forest feature importances and behavioral-context distributions with positive or guided rates for the reported configurations.Query imminence uses a 30-second window, while help-seeking type uses a 15-second window.
Loading 2608.26184v1…