PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Performance, Different Process: Epistemic Ownership in AI-Mediated Education

Jorge Fábrega

arXiv:2610.02731v1cs.CY

TL;DR

Generative AI weakens the inference from assessed performance to the cognitive process that produced it, making the distribution of cognitive work difficult to infer from the final product alone. This study analyzes observable interaction traces through Direction, Integration, and Evaluation, finding that similar assessed performance can coexist with different observable participation patterns.

  • Problem

    Generative AI weakens the familiar inference from students’ assessed work to the learning process it reflects, limiting what product quality alone indicates about cognitive work.

    Source: §1 · p. 1, §5 · p. 13

  • Method

    The study treats recorded student–AI interaction as a partial process window and analyzes observable Direction, Integration, and Evaluation using an exploratory coding scheme.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Similar assessed performance can accompany substantially different observable patterns of Direction, Integration, and Evaluation.

    Source: §4 · p. 9, §5 · pp. 13–14

  • Takeaways & Limitations

    Assignment scores characterize assessed products, while interaction traces provide evidence about how students participated in producing them.

    Source: §5 · pp. 13–14

    Interaction records capture only activity within the conversation, and the descriptive analysis does not estimate a causal relationship between observable participation and performance.

    Source: §2.1 · p. 4, §5 · p. 14

Abstract

from arXiv · show

Generative AI weakens the link between assessed performance and the cognitive processes that educational work is expected to reflect. This paper examines observable traces of those processes in a sample of 150 student-AI conversations. The analysis uses the concept of epistemic ownership as an interpretive framework, focusing on three dimensions that can become visible in interaction: Direction, Integration, and Evaluation. The results show that similar assessed performance can coexist with substantially different observable patterns of cognitive participation. Assignment scores therefore provide information about assessed outcomes but do not recover how cognitive work was distributed between student and AI during the interaction. The study provides an empirical basis for asking how assessment should incorporate evidence about cognitive process, rather than relying exclusively on the quality of the resulting product.

1 Introduction

Generative AI weakens the familiar inference from a successful product to the learning process that produced it. The paper frames the resulting concern as a question of how cognitive work is distributed between students and AI, and what interaction records can reveal about that process.

1 Introduction

  • AI can produce polished work while contributing cognitive operations that assignments were expected to elicit from students.Its educational effects depend on how students use the system and on the learning activity.Source: §1 · p. 1
  • Delegating operations central to developing competence may create a gap between performance and learning, while productive difficulty can support later learning.Formulating problems and choosing between explanations differ from tasks such as correcting typos or reorganizing references.Source: §1 · pp. 1–2
  • AI may remove low-value effort or provide explanations, but it can also perform cognitive work learners would otherwise need to formulate, inspect, revise, or connect.Prior experimental work reports that stronger task performance can coexist with weaker knowledge or transfer gains when AI use reduces metacognitive engagement.Source: §1 · p. 2
  • The study treats recorded student–AI interaction as a partial window into whether learners direct activity, incorporate AI contributions, and participate in judging the work.These records do not reveal everything students think, understand, or do outside the conversation.Source: §1 · p. 2

2 Theoretical framework: epistemic ownership under cognitive delegation

Epistemic ownership concerns a learner’s justificatory command over work produced through distributed cognition. The paper examines observable interactional evidence of direction, integration, and evaluation, while treating answerability as requiring evidence beyond the interaction record.

2 Theoretical framework: epistemic ownership under cognitive delegation

  • Epistemic ownership is the learner’s command to reconstruct and defend reasoning in distributed work; strong AI assistance can coexist with that command, while unexamined adoption can constitute justificatory dispossession.The framework identifies direction, integration, evaluation, and answerability as interdependent functions.Source: §2 · pp. 2–3
  • Direction concerns control of an activity’s purpose and trajectory, integration concerns carrying AI contributions into later reasoning or action, and evaluation concerns judging whether contributions merit acceptance, revision, or rejection.Topical continuity alone does not establish integration, and asking AI to make a judgment does not itself constitute learner evaluation.Source: §2 · p. 3
  • Answerability concerns whether learners can reconstruct and defend the reasoning embodied in final work, which interaction records may not elicit.Assessing it requires additional elicitation beyond the empirical analysis.Source: §2 · p. 3

2.1 From epistemic ownership to observable interactional evidence

  • Interaction histories preserve sequences of delegation, uptake, modification, testing, or redirection that can help distinguish students’ participation even when submitted work is similarly successful.Human–AI chat is sequentially analyzable, though it does not reproduce the social conditions of ordinary conversation.Source: §2 · p. 3, §2.1 · p. 4
  • The analysis codes student re-entry for task-relevant control, specific uptake of prior AI contributions, and participation in judging contributions, procedures, artifacts, or results.A learner move may evidence multiple analytically independent dimensions; interpretation depends on the preceding sequence and the provenance of constraints or content.Source: §2.1 · p. 4
  • The codes describe observable participation rather than an additive ownership score, and an unobserved behavior does not establish that the underlying capacity is absent.Relevant activity may occur outside the chat, so broader claims about epistemic ownership require evidence beyond these interaction records.Source: §2.1 · p. 4

3 Data and Method

The methods use a stratified sample of StudyChat conversations and an exploratory coding scheme to trace observable Direction, Integration, and Evaluation. The analysis also considers how available interaction length and score linkage shape what can be compared.

3.1 StudyChat data

  • The StudyChat corpus contains 2,214 conversations from 203 students across seven AI and computer science assignments, with 16,851 student utterances and a mean of 7.6 turns per conversation.Conversations were the sampling unit; student turns supplied within-conversation evidence.Source: §3.1 · p. 5

3.2 Coding observable cognitive participation

  • The exploratory coding scheme treats Direction, Integration, and Evaluation as independent, multilabel dimensions rather than combining them into an overall epistemic-ownership score.The protocol was developed iteratively and frozen before coding the main 150-conversation sample; evidence was classified as explicit or inferable, with provenance recorded separately.Source: §3.2 · pp. 5–6
  • Coding used full conversation transcripts to produce turn-level codes, evidence records, episode segments, and trajectories, with semantic validation and manual supervision of a random subsample.A learner move could evidence multiple dimensions, while their absence in a transcript—especially a short one—only indicates that participation was not observed there.Source: §3.2 · p. 6

3.3 Observable process in a task environment with immediate feedback

  • Computer science and data-analysis tasks supplied rapid external feedback, making actions such as implementing AI suggestions or diagnosing errors especially observable in this setting.Activities with less immediate consequences may leave fewer traces in chat.Source: §3.3 · p. 7
  • 97.3% of conversations showed Direction, while Integration and Evaluation rose from 37.3% and 46.0% overall to 90.5% each among conversations with at least 15 student turns.Table 2 reports observable dimensions by available conversation length; the pattern is consistent with Integration and Evaluation needing opportunities for student re-entry.Source: Table 2 · p. 7, §3.3 · p. 7

3.4 Linking observable process to assessed performance

  • Assignment scores were treated as assessed performance, not direct measures of learning, cognitive effort, or epistemic ownership; exact linkage was available for 134 of 150 conversations.Unmatched cases were excluded without approximate linkage or imputation, and minimum-turn thresholds were descriptive rather than substantively meaningful cutoffs.Source: §3.4 · pp. 7–8
  • Across alternative opportunity thresholds, score contrasts between conversations with all three dimensions observed and those without showed no stable performance difference.Table 3 presents the analytical sample across thresholds; at lower thresholds, the longest eligible conversation per student-assignment pair was retained, with ties broken by case ID.Source: Table 3 · p. 8, §3.4 · p. 8

4 Results

Across the descriptive comparisons, similar assessed performance accompanies different observable patterns of Direction, Integration, and Evaluation. Matched conversations show how these differences can persist even when scores and interaction length are alike.

4 Results

  • Across opportunity thresholds, score contrasts between conversations with all three observable dimensions and those missing at least one are small and unstable; scores do not recover process.These groupings describe observable D/I/E coverage, not an overall measure of epistemic ownership or the underlying cognitive process.Source: §4 · pp. 8–9, Table 4 · p. 9

4.1 Same performance signal, different observable process

  • Within matched assignment-score pairs, comparable performance coexists with different observable patterns of participation in the epistemic process.The selected cases contrast scores while requiring sufficient interaction to interpret differences in Direction, Integration, and Evaluation.Source: §4.1 · p. 10, Table 5 · p. 10

4.2 Qualitative reconstruction of two matched pairs

  • At the same 1.00 score in S25 Assignment 4, both conversations showed Direction and Integration, but only one visibly evaluated AI work, revising verbose=True.The other conversation showed re-entry, including model results, oriented toward another transformation or explanation rather than Evaluation.Source: §4.2 · pp. 10–11, Table 6 · p. 11
  • At normalized score 0.8727 and six student turns, one F24 interaction showed Direction, Integration, and Evaluation while the other showed Direction alone.The contrast persisted with score and conversation length matched; one conversation showed uptake and inspection, while the other centered on successive directed requests.Source: §4.2 · pp. 11–13, Table 7 · p. 12
  • Assignment scores capture how final work was evaluated, while interaction traces add evidence about how cognitive work was distributed during AI-mediated activity.The distinction motivates considering evidence about process in educational assessment.Source: §4.2 · p. 13

5 Discussion and final remarks

Similar assessed performance can coexist with substantially different observable participation in cognitive work, so scores and interaction traces provide different kinds of information. The discussion frames epistemic ownership as a way to study that distinction while emphasizing the limits of interaction records and the need for assessment designs that make participation observable.

5 Discussion and final remarks

  • Similar performance can coexist with different observable processes, and scores characterize product quality rather than how cognitive work was distributed.The analysis establishes variation in observable participation alongside performance, not a causal relationship between them.Source: §5 · pp. 13–14
  • In StudyChat, programming and data-analysis tasks create re-entry opportunities that make Integration and Evaluation more observable than tasks without intermediate consequences.Learners can execute code, inspect results, identify discrepancies, and revise procedures.Source: §5 · p. 13
  • Interaction records reveal some participation but cannot establish epistemic ownership as a whole; Answerability requires learners to reconstruct and defend their reasoning.StudyChat does not systematically elicit Answerability, so the analysis cannot identify individual cases of justificatory dispossession.Source: §5 · p. 13
  • Assignments and assessment environments could be designed to make participation observable by requiring students to apply, compare, inspect, diagnose, or justify AI contributions.The aim is not to prescribe a single interaction sequence, but to create conditions where cognitively relevant participation leaves examinable evidence.Source: §5 · p. 14
  • A fuller assessment could combine product evidence, process evidence, and later elicitation of whether learners can reconstruct, defend, or transfer reasoning.How to combine these sources and match elicitation to learning objectives remains open.Source: §5 · p. 14
  • The descriptive findings concern observable cognitive participation and course assignment scores, not learning, epistemic ownership, or a causal relationship between them.StudyChat records only within-interaction activity, and its computational setting may make some participation easier to observe than other domains.Source: §5 · p. 14
Loading 2610.02731v1…