PaperlayerView exact revision on arXiv

Source-linked AI summary

Critical Thinking with Generative AI: A Constraint-First Design Pilot of a Thinking-Partner Intervention

Fatima Tuz Zahra, Jiangen He, David M. Bowers, Wei Wang

arXiv:2609.38029v1cs.HC

TL;DR

The paper addresses the risks and design conditions of GenAI in education through a Design-Based Research pilot of a critical-thinking framework. AASCDM and MAILS gains coexisted with a flat WGCTA result, and the pilot lacked a dedicated assessment of students’ critique of LLM outputs.

  • Problem

    Overreliance risks intellectual apathy, and trust in apparently helpful outputs can make occasional failures harder to catch.

    Source: §2.3 · p. 8

  • Method

    This Design-Based Research pilot examines a GenAI-assisted critical-thinking framework.

    Source: §6 · p. 39

  • Results

    Reported gains covered AASCDM dimensions and eight of nine MAILS dimensions, while WGCTA was flat.

    Source: §5 · pp. 35–36

  • Takeaways & Limitations

    The next iteration will add explicit critique-of-AI tasks and matched assessments.

    Source: §5 · p. 36

    The pilot did not include a dedicated assessment of how students critique LLM outputs.

    Source: §5 · p. 36

Abstract

from arXiv · show

Generative AI (GenAI) tools entered higher education classrooms faster than the field was able to study their effects on learning. One concern is that GenAI may displace the critical thinking and AI literacy that students will need after graduation. This paper reports a Design-Based Research pilot of a GenAI-assisted critical thinking framework, in which ChatGPT was used as a thinking partner in an undergraduate research methods and statistics course during Spring 2025 (N = 14). The mixed-methods design combined pre- and post-intervention measures of statistical learning (AASCDM), AI literacy (MAILS), and critical thinking (WGCTA) with instructor field notes, student artifacts, and student-AI interaction logs. Pre-post tests showed gains on every AASCDM dimension and on eight of nine MAILS dimensions, while WGCTA percentiles did not change. Qualitative analysis identified four themes: the ways students positioned the LLM (as answer generator, validator, or co-thinker); the depth of student engagement (procedural vs. conceptual); occasional humanizing of the tool; and the role of curriculum design in shaping each of the prior three. Read together, the findings indicate that one semester of GenAI-assisted instruction can move domain learning and self-reported AI literacy but does not move standardized critical thinking, and that the modal student-LLM relationship is one of validation instead of dialogue. We end with design principles for the next iteration of the framework and implications for research on adaptive and personalized learning.

1 Introduction

The introduction frames GenAI in higher education as both a potential support for critical thinking and a risk of replacing student analysis, with outcomes dependent on pedagogy and context. It presents a Design-Based Research pilot of ChatGPT as a thinking partner and asks about learning, student engagement, and design refinements.

1 Introduction

  • GenAI implementations carry both opportunity and risk, with outcomes resting on pedagogical and contextual factors and overreliance potentially substituting for analysis.The introduction also links overreliance with intellectual complacency.Source: §1 · pp. 3–4
  • The paper identifies a gap between students’ critical-thinking abilities and workplace demands, alongside concerns about inadequate development of those skills.It presents AI literacy as necessary for using AI to build on students’ own thinking.Source: §1 · pp. 3–4
  • The paper pilots a GenAI-assisted critical-thinking framework, using ChatGPT as a personalized, adaptive thinking partner in an undergraduate research methods and statistics course.The pilot took place during Spring 2025.Source: §1 · p. 4
  • The study asks about pre-post gains in statistical learning, AI literacy, and critical thinking, students’ engagement with GenAI, and lessons for learning outcomes and design refinements.These are framed as three research questions.Source: §1 · p. 4

2 Literature Review

The literature review connects critical-thinking development, AI literacy, and technology-mediated learning to the pilot’s design and analytic choices, emphasizing that GenAI’s educational effects depend on how students engage with it and how instruction structures that engagement. It also highlights the risk that GenAI may simulate understanding without supporting knowledge construction and the need for integrated, observed, iterative studies of critical thinking and AI literacy.

2.1 Critical Thinking in Higher Education

  • Critical thinking is purposeful, self-regulating judgment, but its definition and measurement remain contested, and cognitive skills only moderately correlate with dispositions.The review also describes critical thinking as involving interpretation, analysis, evaluation, and inference.Source: §2.1 · p. 5
  • Meta-analytic evidence indicates that direct, explicit critical-thinking instruction can produce gains, but transfer of those gains across domains requires deliberate design of the instructional sequence (Abrami et al., 2008, 2015; Halpern, 1998).Source: §2.1 · p. 6

2.2 AI Literacy: Emerging Frameworks

  • AI literacy entails knowledge, skills, and dispositions for responsible AI use, and frameworks treat evaluating AI as a critical-thinking task; the study’s MAILS scale measures nine dimensions.The review treats AI literacy and critical thinking together because AI evaluation depends on a contested conception and measurement of critical thinking.Source: §2.2 · pp. 6–7

2.3 GenAI and Critical Thinking: Empirical Evidence

  • The review warns that accepting AI-generated recommendations without questioning them increases performance errors in decision-making contexts; inappropriate trust, rooted in systems’ frequent correctness and helpfulness, can make occasional failures harder to catch and discourage in-depth engagement.Source: §2.3 · p. 8

2.4 Constructivist Theory and Technology-Mediated Learning

  • Constructivist learning theory offers a frame for knowledge construction, while applying it to AI-assisted learning raises tensions of its own.Source: §2.4 · p. 9
  • AI-generated explanations, analyses, or solutions may create cognitive engagement without learners constructing knowledge themselves, and behavioral engagement metrics need not indicate meaningful learning.The review notes that time on task, interaction frequency, and completion rates do not necessarily map onto meaningful learning.Source: §2.4 · p. 9
  • AI can provide sophisticated scaffolds but lacks a model of learners’ needs, misconceptions, or developmental trajectories, offering assistance without comprehension.Source: §2.4 · p. 10

2.5 Conceptual and Procedural Knowledge in AI-Assisted Learning

  • Procedural knowledge concerns action sequences, while conceptual knowledge concerns principles and relationships; conceptual knowledge should anchor procedural fluency.Source: §2.5 · p. 10
  • AI-generated explanations can lead students to mistake polished articulation for understanding they have internalized, making metacognitive instruction relevant to AI literacy.Source: §2.5 · pp. 10–11

2.6 Design-Based Research as Methodology

  • Design-based research suits complex interventions with emerging technologies because it combines empirical questions about learning with design-oriented questions about subsequent iterations.It accommodates quantitative and qualitative evidence while supporting iterative refinement.Source: §2.6 · p. 11

2.7 Synthesis and Gap

  • Prior research often separates critical thinking from AI literacy, emphasizes short-term behavioral indicators, and offers few iterative studies linking outcomes to design moves.The review also notes that behavioral indicators are often not paired with validated learning measures and direct observation of AI interaction.Source: §2.7 · p. 11
  • The study addresses three gaps by pairing validated measures of statistical learning, AI literacy, and critical thinking with student–AI interaction analysis and qualitative observation in a design-based research cycle.The cycle is intended to yield design principles for subsequent work.Source: §2.7 · pp. 11–12

3 Methodology

The methodology describes a mixed-methods, design-based pilot of a constraint-first ChatGPT thinking-partner intervention in a one-semester undergraduate statistics course. It combines pre/post measures and analysis of student–AI prompts and conversation sequences with iterative qualitative classroom analysis to assess implementation and guide design refinements.

3.1 Design-Based Research

  • The study adopted mixed-methods design-based research to examine GenAI’s effects on critical thinking, AI literacy, and statistical learning while informing intervention refinement.The approach combined diverse assessment indices with qualitative evidence to study the intervention in an educational setting.Source: §3.1 · p. 12

3.2.1 Study Setting

  • The pilot ran in Spring 2025 in a one-semester undergraduate applied statistics course at an R1 public university, with 14 students completing both pre- and post-measures.The university provided reliable institutional access to GenAI tools, which the authors said removed access as a confound.Source: §3.2.1 · pp. 12–13

3.2.2 The Thinking-Partner Implementation: A Constraint-First Design

  • The first semester-long empirical implementation used a three-step protocol—think independently, engage with ChatGPT, then report shifts—to frame the LLM as a thinking partner rather than an answer authority.Weekly homework and practicum activities asked students to test and refine their own arguments; instructors used a Socratic prompt and emphasized that responsibility for reasoning remained with learners.Source: §3.2.2 · p. 13
  • Two midsemester design revisions added practicum rehearsal before graded tasks and more explicit reminders that students’ reasoning—not LLM output—should guide decisions.The changes were made in response to qualitative analysis; the LLM’s output was reframed as one input to student reasoning rather than its destination.Source: §3.2.2 · p. 14

3.3 Quantitative Methods

  • The pre-post pilot assessed critical thinking, AI literacy, and statistical learning using WGCTA, MAILS, and AASCDM; paired-samples t-tests examined AASCDM and MAILS dimensions, with results described as descriptive rather than confirmatory.Source: §3.3 · pp. 15–16

3.4 Human Behavior Analysis

  • Interaction analysis examined 442 student-authored prompts from seven assignments, averaging 197.4 words (1,252 characters) per prompt, across three analyses of discussion topics, requested AI actions, and request sequences.Topic analysis embedded each prompt and clustered the vectors with BERTopic, using HDBSCAN.Source: §3.4 · p. 16
  • Cluster labels drew on representative documents and keywords, and UMAP projected embeddings into two dimensions for visual inspection.Source: §3.4 · p. 16
  • Two researchers iteratively developed a prompt-coding schema from 50 random prompts; two coders independently coded another 50, and a third adjudicated disagreements to create a consensus gold standard.Source: §3.4 · p. 17
  • GPT-5 classified the full corpus using category definitions, examples, and decision criteria, returning JSON labels and confidence scores; classifier agreement was reported in Section 4.2.The codebook, including definitions and anonymized student examples, was provided in Appendix C.Source: §3.4 · p. 17
  • Conversation sequences were analyzed with bigram, trigram, and fourgram frequencies and a first-order Markov chain to characterize request order and transitions.Markov transition probabilities were calculated as the frequency of a category-to-category transition divided by the total frequency of the starting category.Source: §3.4 · p. 17

3.5 Qualitative Methods

  • The study treated quantitative methods as summative assessments of implementation efficacy and qualitative methods as formative evidence to characterize implementation and guide refinements.A more substantial retrospective qualitative analysis was planned beyond the pilot analysis reported here.Source: §3.5 · pp. 17–18
  • Across the semester, continuous DIER-cycle iterations produced two major revisions informed by thematic analysis of assignments, homework, and classroom observations through critical landscapes of education.Source: §3.5 · p. 18
  • The reported finer-grained analysis evaluated content and implementation through conceptual and procedural knowledge and described how findings guided ongoing design revisions.The analysis was intended to make visible what revisions were made, when, and what prompted them.Source: §3.5 · p. 18

4 Findings and Discussion

The findings examine changes in statistical learning and AI literacy, stability in standardized critical-thinking percentiles, and patterns in students’ prompts and interaction modes with AI.

4.1.1 Sample Demographics

  • The sample comprised 14 undergraduates, mostly early-stage students from the education-focused college, and all reported access to a computer and reliable internet.Source: §4.1.1 · pp. 18–19

4.1.2 AASCDM (Statistical Learning) Pre-Post Comparison

  • AASCDM scores improved significantly across all ten dimensions; the largest gains were selecting statistical tests (M = 1.57 to M = 2.79), using statistical software (M = 1.07 to M = 2.36), and explaining probability distributions (M = 1.64 to M = 2.93).Source: §4.1.2 · p. 20

4.1.3 MAILS (AI Literacy) Pre-Post Comparison

  • MAILS improved significantly on eight of nine dimensions; AI Persuasion Literacy was the exception, changing from M = 25.43 to M = 25.86 (p > .05).Source: §4.1.3 · pp. 22–23

4.1.4 WGCTA (Critical Thinking) Pre-Post Comparison

  • All three WGCTA subscales showed no significant pre-post percentile differences (all p > .05), with substantial participant variability and no standardized critical-thinking gains.Source: §4.1.4 · pp. 24, 26

4.2 Patterns in Student-AI Interaction

  • Topic modeling of 442 prompts identified nine distinct, cleanly separated clusters; most drew messages from multiple assignments, and feedback requests came from all seven.Source: §4.2 · pp. 26–27
  • GPT-5 classified prompts against a 50-prompt gold standard with 82% accuracy (41/50, 95% CI [.69, .91]).The coding schema comprised eight categories grouped as Information Requests/Needs and Context Description.Source: §4.2 · pp. 27–28
  • Across the full corpus, assignment description was the most common category (36.9%, n = 163), followed by general improvement request (22.0%, n = 97); reflection analysis was next (14.3%, n = 63).The remaining categories were higher-order thinking (11.5%, n = 51), appreciation (5.4%, n = 24), applicability of method (5.0%, n = 22), specific improvement request (3.2%, n = 14), and course description (1.8%, n = 8); context description accounted for 44.1% of messages.Source: §4.2 · p. 28
  • Category proportions were largely stable across seven assignments, with assignment description and general improvement request dominant and no consistent trend or phase shift in other categories.The other categories fluctuated within narrow bands.Source: §4.2 · p. 28
  • The Markov chain showed strong self-loops for assignment description (.58), higher-order thinking (.47), and general improvement request (.46), while cross-category moves included context-to-help and reflection–conceptual transitions.Assignment description to general improvement request was the most frequent cross-category move (6.8%); reflection analysis to higher-order thinking and the reverse had transition probabilities of .22 and .21, respectively.Source: §4.2 · pp. 28–29

Ten most frequent bigram transitions between prompt categories

The ten most frequent category transitions account for 65.8% of observed turn-to-turn transitions. Across the first five conversation positions, category flows show persistence, context-setting followed by feedback requests, and some movement toward analytical engagement.

Ten most frequent bigram transitions between prompt categories

Flow of student prompt categories across the first five conversation positions

The qualitative analysis found distinct generator, validator, and co-thinker relationships, alongside differences in engagement depth and occasional humanization of the LLM. Across the pilot, statistical learning and AI literacy gains did not coincide with gains on standardized critical thinking, and validator-style engagement was most common.

Flow of student prompt categories across the first five conversation positions

  • The flow visualization encodes category frequency through node size and flow width, uses consistent colors for categories, and shows only patterns occurring at least twice across P0–P4.Source: Flow of student prompt categories… · p. 30

4.3 Qualitative Reflections from the Design Process

  • Students positioned the LLM as an answer generator, validator, or co-thinker; generator use displaced students’ critical thought, while co-thinkers actively practiced it.Source: §4.3 · p. 31
  • Students varied between reflective, conceptual exchanges and shallower, procedural ones; co-thinker positioning was associated with potential for deeper AI literacy than generator or validator use.Source: §4.3 · p. 31
  • Students sometimes humanized the LLM with thanks or personal disclosures, but such language does not necessarily mean they believe it is sentient.The authors flag a presumption of sentience as a concern for critical AI literacy.Source: §4.3 · p. 32
  • The curriculum shaped these interaction patterns; proposed revisions include rubrics that encourage co-thinking and activities that surface the model’s statistical rather than sentient behavior.Source: §4.3 · p. 32

4.3.1 Interaction Patterns Behind the Positionings

  • Validator positioning was most common: students often submitted drafted answers for checking, yet frequently did not integrate the feedback into their work.Some copied the LLM response after their own answer without visible integration.Source: §4.3.1 · pp. 32–33
  • Closed yes-or-no questions from the LLM often drew brief affirmative replies, ending possible inquiry and suggesting the response format steered interactions toward validation.Open-ended follow-up questions are under consideration for the next iteration.Source: §4.3.1 · p. 34

4.4 Synthesis Across Quantitative and Qualitative Findings

  • Students gained in self-assessed statistical learning and AI literacy, but not on standardized critical thinking; the most common student–LLM relationship was validation rather than co-thinking.The synthesis also notes that AI Persuasion Literacy, the MAILS dimension that did not significantly move, taps evaluative reasoning similar to WGCTA.Source: §4.4 · pp. 34–35

5 Discussion

The pilot showed gains in statistical learning and most AI-literacy dimensions, but not standardized critical thinking; students most often used the LLM to validate rather than extend their thinking. The discussion links that pattern to curriculum design and outlines design revisions and scope limits for the next iteration.

5 Discussion

  • Students gained on every AASCDM dimension and eight of nine MAILS dimensions, but showed no gains on a standardized critical-thinking measure.The authors interpret the learning and AI-literacy gains as strongest in dimensions tied directly to course tasks.Source: §5 · p. 35
  • The authors offer three nonexclusive explanations for the flat critical-thinking result: limited transfer sensitivity and time, validator behavior, and no dedicated assessment of critique of AI outputs.They note that the standardized measure is intentionally insensitive to course-specific content, while one semester is a short window for measurable transfer.Source: §5 · p. 36
  • The most common student-LLM relationship was validation: students confirmed their thinking without challenging or extending it.The discussion suggests this default reflects the design more than a deficit among students.Source: §5 · p. 36
  • The discussion identifies direct instruction and authentic practice evaluating AI outputs for bias, framing, and rhetorical influence as priorities for the next iteration.The planned framework revision also adds explicit critique-of-AI tasks and matched assessments.Source: §5 · pp. 36–37

5.1 Design Principles for Constraint-First GenAI Instruction

  • Design principles call for course- and assignment-specific prompts, reflection across assignments, and rubrics that reward the quality of students’ AI engagement.The pilot’s co-thinker engagement was most consistent in assignments that built reflection into the task.Source: §5.1 · pp. 37–38
  • The authors identify advance instructor preparation, ongoing support, and student reflections as diagnostic evidence for revising the design.The pilot’s mid-semester revisions drew on analysis of what students produced and wrote about their LLM engagement.Source: §5.1 · p. 38
  • The six design principles are being tested in a larger implementation to assess which hold across instructors and disciplinary contexts.The authors say that study will also identify principles needing further refinement.Source: §5.1 · p. 39
  • The pilot’s quantitative findings are descriptive, based on 14 self-selected students from one course and institution, limiting claims about population effects.The sample was demographically skewed, and the authors say the results inform the next design-based research iteration rather than estimate population effects.Source: §5.1 · p. 39

6 Conclusion and Implications

This pilot paired ChatGPT with constraint-first, Socratic instruction to support critical thinking. Students gained in domain learning and most AI-literacy dimensions, but not standardized critical thinking; most used the LLM as a validator rather than a co-thinker.

6 Conclusion and Implications

  • The framework used a structured, Socratic deployment of ChatGPT in a research methods course rather than content delivery or recommendations.The intended interaction was closer to dialogic learning, while fostering a co-thinker orientation depended on instructional design.Source: §6 · pp. 39–40
  • The course improved domain learning and most AI-literacy dimensions, but not standardized critical thinking; most students treated the LLM as validator, not co-thinker.Source: §6 · pp. 39–40
  • Here, personalization means adapting the model’s register and depth to a student in a particular conversation, with instructors shaping conditions for productive adaptation.It is not algorithmic content matching.Source: §6 · p. 40
  • Instructors should explicitly teach AI persuasion literacy and critique of AI outputs, and use rubrics that make co-thinker engagement visible and gradeable.Source: §6 · p. 40
  • Pair self-reported AI-literacy measures with observations of actual AI interactions, because the two are related but distinct.Source: §6 · p. 40
  • The pilot suggests higher-education outcomes depend on pedagogical scaffolding more than model choice; without faculty development and curriculum design, validator-style use may predominate.The authors associate that pattern with displacement of reflective learning.Source: §6 · pp. 40–41
  • The constraint-first pilot analyzed activity, work products, and implementation evidence from 14 undergraduates, and its design principles are informing a larger multi-course implementation.Source: §6 · p. 41
  • Future iterations will add direct AI-persuasion-literacy tasks, larger samples across disciplines, and objective higher-order-thinking measures; the pilot also offers a case for adaptive and personalized learning research.Source: §6 · pp. 40–41

Data Availability Statement

The de-identified interaction-log corpus and analysis code are available from the corresponding author on reasonable request, subject to IRB approval terms. Raw logs cannot be shared publicly because consent covered research use only.

Data Availability Statement

Declaration of Generative AI and AI-Assisted Technologies

The authors used GPT-4.1 and GPT-5 for research topic modeling and classification, and ChatGPT and Claude for manuscript-preparation assistance. They reviewed and edited the content and retained responsibility for it.

Declaration of Generative AI and AI-Assisted Technologies

Loading 2609.38029v1…