PaperlayerView exact revision on arXiv

Source-linked AI summary

Fluency Without Evidence: Constraint-First Design and the Limits of Self-Report in AI-Assisted Learning

Fatima T. Zahra, Wei Wang, Frances Harper, Jiangen He

arXiv:2609.37880v1cs.HC

TL;DR

Prior learner-outcome evidence leaves open whether AI teaching partners support learning in authentic courses. This study examines a first cycle in two graduate research methods courses using pre- and post-measures alongside interaction records. AI literacy increased, while critical thinking, metacognitive awareness, and objectively scored knowledge did not change.

  • Problem

    Prior learner-outcome evidence has focused on almost everything except learning, leaving the question of learning outcomes in AI-supported education open.

    Source: §2.2 · p. 7

  • Method

    The first design cycle in two graduate research methods courses collected pre- and post-measures of student outcomes alongside interaction records.

    Source: §1 · p. 5

  • Results

    AI literacy increased, concentrating in conceptual understanding of AI, while critical thinking disposition, metacognitive awareness, and objectively scored knowledge did not change.

    Source: §6 · p. 32

  • Takeaways & Limitations

    Design research on AI-supported learning requires interaction evidence over self-report to provide a more defensible basis for designs.

    Source: §6 · p. 33

    Every outcome that changed was self-reported, while only three objectively scored items were used and those did not change.

    Source: §5.4 · p. 32

Abstract

from arXiv · show

A generative AI teaching partner should support reasoning over supplying conclusions; however, this has not been tested against learning in an authentic course. Drawing on design-based research, we specify the position as a conjecture map and report a first design cycle in two graduate-level research methods courses. Students used an AI teaching partner employing a constraint-first sequence requiring them to state and justify positions before receiving questions. Pre- and post-measures of AI literacy, critical thinking, and metacognitive awareness were collected alongside interaction records. AI literacy increased, concentrating in understanding AI, whereas critical thinking, awareness, and knowledge did not change. Since changes were limited to self-report measures, they may reflect growth in confidence instead of capacity. Interaction records, meanwhile, showed brief exchanges, uneven enactment of the constraint-first sequence, and missing records. These findings show why AI-supported learning requires interaction records to provide a more defensible basis for AI-supported designs than self-reports.

1 Introduction

The paper addresses an untested proposition that positioning AI as a reasoning partner rather than an authority improves reasoning, then tests it in authentic coursework. It specifies the proposition as a conjecture map and pairs outcome measures with interaction records.

1 Introduction

  • Prior work had not directly tested whether positioning AI as a partner rather than an authority improves reasoning through measured learning or higher-order thinking in authentic coursework.The literature had designed for and theorized learner outcomes without directly measuring them.Source: §1 · pp. 3–4
  • The conjecture map proposes that three design features shift students from confirmation-seeking toward reasoning, producing gains in higher-order thinking and AI literacy.The features require students to state and justify a position before the model responds, restrict the partner to questions, and frame it as a statistical model rather than an authority.Source: §1 · p. 4
  • The first empirical cycle implemented the design across a semester in two graduate research methods courses and collected pre- and post-measures alongside students’ interaction records.Measures covered AI literacy, critical-thinking disposition, and metacognitive awareness.Source: §1 · p. 5
  • The paper treats interaction records as evidence of student engagement and whether the constraint-first sequence operated as intended, rather than relying on self-report for engagement.It also develops a measurement argument prompted by the fact that changed outcomes were self-reported while unchanged outcomes were measured differently.Source: §1 · p. 5

2 Related Work

Prior work frames AI-supported learning through motivation, scaffolding, and learner outcomes, while this paper centers the learner’s epistemic positioning and uses design-based research to test an explicit conjecture.

2.1 From answer engine to thinking partner

  • Epistemic positioning asks what role learners assign the model and how that role affects their own authority over material, unlike Dual-ZPD’s focus on support and motivation.The framing addresses students who may treat model output as authoritative even when they are motivated.Source: §2.1 · p. 6

2.2 Learner outcomes in the CHI education literature

  • The CHI education literature largely studies perceptions, artifacts, and theory, leaving whether learner thinking changes largely unaddressed.The authors describe this as a mismatch between claims about learning and evidence that often does not measure learning.Source: §2.2 · p. 7

2.3 Design-based research as a bridge

  • Conjecture mapping specifies how design features may lead through mediating processes to outcomes, making claims open to disconfirmation.The paper brings design-based research into contact with another research tradition that asks different questions of the same designs.Source: §2.3 · p. 8
  • The study’s conjecture is that co-thinker positioning supports richer reasoning and stronger gains in higher-order thinking and AI literacy.The authors treat this as one cycle in a multi-cycle program, with unsupported conjectures serving as grounds for refinement.Source: §2.3 · p. 8

3 Methods

The design was embedded in graduate research-methods coursework, with a teaching partner structured to elicit students’ reasoning before offering reflective questions.

3.1 Study context

  • Four graduate research-methods courses implemented the design, but only two returned data; departures from the protocol left those other courses out of the analyses.The authors treat this as site-level attrition; 26 students were enrolled across the two courses and 20 consented to research use.Source: §3.1 · p. 9
  • Research-methods coursework provides an authentic setting for higher-order thinking because students must defend methodological choices under competing constraints.The courses were regular instruction rather than a stand-alone AI workshop.Source: §3.1 · pp. 9–10

3.2 The teaching partner

  • The constraint-first sequence requires students to state a position and reasoning before consulting the partner, which responds with reflective questions rather than answers.The partner is also instructed to return judgment to the student when deference becomes apparent.Source: §3.2 · p. 10

3.3 Design cycles

This design cycle extended earlier work into graduate courses and added systematic interaction records and measures of AI literacy and metacognitive awareness.

3.3 Design cycles

  • The earlier cycle in one undergraduate research-methods course identified validator and co-thinker positions; this cycle expanded to graduate coursework across two disciplines.It also treated interaction records as data and added AI-literacy and metacognitive-awareness measures.Source: §3.3 · p. 11

3.4 Measures

  • Critical-thinking and metacognition scales were adapted rather than administered in previously validated form, and critical-thinking reliability was marginal at pretest (alpha = .64).Metacognition reliability was alpha = .82 and .94 across administrations; the authors note that marginal pretest reliability bears on interpreting small pre-post differences.Source: §3.4 · p. 12
  • Course B’s three objectively scored knowledge items were the only performance measures, while transcripts alone documented engagement as enacted rather than reported.The other instruments asked students to appraise their own capabilities.Source: §3.4 · pp. 12–13

3.5 Interaction corpus

The interaction corpus was incomplete and uneven: some submissions were unretrievable or unsegmentable, while the analyzable transcripts came from a subset of students.

3.5 Interaction corpus

  • Nine submissions had no retrievable content; of 95 retrievable files, 20 lacked speaker markup and could not be segmented into turns; these averaged 3,688 characters, versus 22,730 for segmentable files.Source: §3.5 · p. 13
  • The analyzable record comprised 75 transcripts from 16 students, yielding 529 analytic student turns after 65 pasted prompt-opening turns were removed.Course A contributed 408 turns (77%) and Course B 121 (23%); Course B contributed eight of thirteen students to the record.Source: §3.5 · p. 13

3.6 Coding of student prompts

The authors coded individual student prompts for epistemic positioning, cognitive process, and conversational uptake, then analyzed matched measures and interaction patterns descriptively.

3.6 Coding of student prompts

  • Each student prompt—one message to the AI assistant—was the unit of analysis for coding.The coding scheme was fixed before coding began.Source: §3.6 · p. 13
  • Prompts received a cognitive-process code for analyze, evaluate, create, or procedural, plus a separate metacognitive-regulation flag.Nonprocedural prompts also received an epistemic-positioning code.Source: §3.6 · p. 14
  • Epistemic positioning distinguished validator, co-thinker, and generator roles, while an open category captured turns outside the three-position model.The open category allows the adequacy of the model to be assessed rather than assumed.Source: §3.6 · p. 14
  • Conversational uptake was defined as building on the AI’s contributions, with high uptake elaborating and low uptake not moving the topic forward.A trained educational researcher coded student prompts for uptake of the AI assistant’s prompts.Source: §3.6 · p. 14
  • Inductive uptake codes captured responding, asking, revising, or challenging the AI, as well as ignoring a request or redirecting the conversation.Each prompt received one cognitive-process code and, unless procedural, one epistemic-positioning code.Source: §3.6 · pp. 14–15

3.7 Analysis

  • Matched analytic samples were 18 for MAILS, 19 for critical thinking and metacognition, 11 for Course B’s learning assessment, and 20 for demographics.Analyses used paired t tests with Cohen’s dz, Holm adjustment across outcome families, and robustness checks including Wilcoxon tests and bootstrap intervals.Source: §3.7 · p. 15
  • With small analytic samples, nonsignificant findings were treated as uninformative about change rather than as evidence of no change.Sensitivity power analysis was reported alongside results.Source: §3.7 · p. 15
  • Interaction analyses described the corpus and examined within-semester patterns only across Course A’s sequenced assignments, the study’s sole ordered series.Displays examined codes by course, assignment, and student, along with code co-occurrence and uptake strategies.Source: §3.7 · pp. 15–16

3.8 Ethics

The study reports its ethics procedures and the research team’s relationships with course instructors, including how those relationships shaped the account of implementation.

3.8 Ethics

  • The study received university IRB approval; students consented to research use of course data, participation did not affect standing, and records were de-identified and stored separately from linking information.Instructional activities continued for all enrolled students regardless of consent.Source: §3.8 · p. 16

3.9 Positionality

  • The first author designed the study but taught neither course, while team members also brought prior implementation, analysis, liaison, or research roles.The first author’s earlier involvement informed interpretation of the data.Source: §3.9 · pp. 16–17
  • Because instructors were colleagues and needed for implementation and data return, the account of enactment relied more on artifacts than instructor self-report.The authors recognized that this dependence informed what was reported to them about implementation.Source: §3.9 · p. 16

4 Findings

Findings cover learning measures and student interaction records, including participation patterns and record completeness. Self-reported AI literacy increased, especially understanding AI, while critical thinking, metacognitive awareness, and objectively scored knowledge showed no reliable gains; interaction records showed brief turns, more questions than stated positions, uneven uptake of AI prompts, occasional reports of reasoning revision, and incomplete records.

4.1 Sample

  • Among 20 students across two courses, the sample was predominantly graduate or professional (95%) and women (75%), and enrolled in a college of education, health, and human sciences (80%).The authors described the group as an older, largely graduate sample with strong digital access alongside heterogeneous educational, family, and economic circumstances.Source: §4.1 · p. 17

4.2 Self-reported capability increased

  • In Course B (n = 11), five self-rated competencies survived Holm adjustment: reproducible workflows (dz = 1.09), external or API data (1.56), code-based visualization (1.26), geographic and spatial data (1.36), and evaluating AI-generated code or explanations (1.56).Source: §4.2 · p. 19, Table 4

4.3 Demonstrated performance did not increase

  • Critical-thinking disposition rose 1.95 points (95% CI [-0.70, 4.59], Holm-adjusted p > .999; dz = .35); metacognitive awareness changed 0.47 (dz = .05), and Course B's objectively scored knowledge items were unchanged.Interpretation is limited because students were near the ceiling and pretest reliability was marginal.Source: §4.3 · pp. 22–23

4.4 The interaction record

  • Student turns were brief: median word counts were 38.5 in Course A and 49 in Course B versus a model median of 338; transcripts had a median of five student turns (mean 7.6, range 1 to 45).Source: §4.4 · p. 24
  • Questions appeared in 48% of student turns, versus explicit position statements in 25.5%; across courses, the respective rates were 47% and 51% for questions and 24.8% and 28.1% for positions.Across the corpus, students asked more often than they asserted.Source: §4.4 · pp. 24–25
  • In Course A, median turn length rose from 37 words on the first assignment to 44 on the last, while turns containing a question rose from 54% to 78%; neither trend was monotonic.The final assignment included six students and 45 turns, and turns per student showed no trend.Source: Table 5 · p. 25, §4.4 · p. 25
  • Illustrative passages, mostly from Course A, showed students revising positions and linking changes to AI questions, but lexical identification could miss reasoning changes across turns.These passages were illustrative rather than estimates of revision frequency, and they were the study’s only evidence of students noticing and revising reasoning as it happened.Source: §4.4 · pp. 25–26
  • Among 448 turns in which the AI directly sought a response, students mostly took up its questions or suggestions but ignored them approximately 25% of the time; questions were the most common response strategy.About half of student responses included one of the identified strategies.Source: §4.4 · pp. 26–27
  • Uptake varied across assignments and courses: around three-fourths of response turns came from Course A, while uptake remained even and low across Course B assignments.In Course A, low uptake could involve redirecting the conversation or entering instructor prompts without engaging the AI’s responses.Source: §4.4 · p. 27

4.5 What the corpus indicates about enactment

  • Twenty of 95 submissions lacked speaker markup or preserved only model output, and another nine had no retrievable content, together accounting for roughly 28% of submissions.The artifacts do not establish why student contributions were absent, but the design presumed a dialogic record that was missing in a substantial minority of cases.Source: §4.5 · pp. 28–29

5 Discussion

The first cycle partially supports the conjecture: AI literacy increased, but the link from positioning to higher-order thinking remains untested. The discussion calls for interaction evidence and design changes while bounding claims through measurement and data limitations.

5.1 The conjecture, partially tested

  • AI literacy increased, especially conceptual understanding of AI, but the conjecture’s central link to higher-order thinking remains untested.The authors consider the literacy increase consistent with epistemic framing, while noting that the evidence does not establish the full conjectured chain.Source: §5.1 · p. 29

5.2 A measurement problem in this literature

  • Survey findings permit two readings: gains in rehearsed practices with little broader transfer, or increased self-appraisal without evidence that underlying thinking changed.The measures cannot distinguish these interpretations; the authors say the link from positioning to higher-order thinking remains untested.Source: §5.2 · pp. 29–30
  • Interaction records warrant treatment as primary evidence about mechanism because they observe engagement, unlike reports that ask learners about it.A design study reporting learner views without what learners did has not tested its conjecture.Source: §5.2 · p. 30

5.3 Implications for design

  • Position-stating occurred in roughly a quarter of turns; a median of five turns and low conversational uptake point to structural enforcement and longer exchanges as design targets.The authors suggest these revisions may matter more than prompt wording.Source: §5.3 · pp. 30–31
  • More than a quarter of submissions were unusable or partial, supporting system-side record capture alongside attention to exchange length and uptake.The unusable submissions clustered in one course, and the authors argue system-side capture could address both the data problem and a signal about how students understood the activity.Source: §5.3 · pp. 30–31

5.4 Limitations

  • Results came from two of four courses that followed protocol, with 77% of analyzable turns from one course, limiting representation across sites.The other course contributed only eight of its thirteen students to the turn-level record.Source: §5.4 · p. 31
  • Only self-reported outcomes changed; three objectively scored items did not, and samples of 11–20 had power to detect only large standardized changes.The authors caution that null results are uninformative, and survey evidence cannot distinguish capability growth from confidence growth.Source: §5.4 · p. 32
  • The study was single-arm, pre–post, at one institution, while adapted instruments and reliance on colleagues to implement the intervention also bound interpretation.The first author designed the intervention, which may have influenced what colleagues reported about implementation.Source: §5.4 · p. 32

6 Conclusion

The paper tests a conjecture about positioning generative AI as a thinking partner in two graduate research methods courses. AI literacy increased, but other measured outcomes did not; the conclusion emphasizes interaction evidence for evaluating such designs.

6 Conclusion

  • The proposition that generative AI should be a thinking partner rather than an answer engine had seldom faced empirical test.The paper specified it as a conjecture and examined a first cycle in two graduate research methods courses.Source: §6 · p. 32
  • AI literacy increased, concentrated in conceptual understanding of AI, while critical thinking disposition, metacognitive awareness, and objectively scored knowledge did not change.Self-appraisal moved while demonstrated performance did not.Source: §6 · p. 32
  • The conclusion argues that design research needs interaction evidence to determine whether its conjectures hold, though collecting that evidence is itself a design challenge.The reported corpus suggests evidence collection is more difficult than it appears.Source: §6 · p. 33

A1 Prompt Template

The prompt frames the AI as a learning partner that facilitates engagement rather than giving direct answers. It asks the partner to encourage reflection, inquiry, and collaborative exploration.

A1 Prompt Template

  • The prompt asks the assistant to cultivate critical thinking and deep engagement by prompting reflection, data analysis, and exploration of perspectives instead of giving direct answers.Source: A1 Prompt Template · p. 37
  • It frames the student as a dialogic partner and calls for open-ended questions about interpreting data and factors influencing outcomes.Source: A1 Prompt Template · p. 37
  • The template encourages alternative statistical approaches, assumption-checking, inquiry using credible sources, peer comparison, and supportive iterative exploration.Source: A1 Prompt Template · p. 37
Loading 2609.37880v1…