PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring undergraduate students' reliance on generative AI during problem-solving: Scale development and validation

Chenyu Hou, Gaoxia Zhu, Vidya Sudarshan, Fun Siong Lim, Yew Soon Ong

arXiv:2610.30195v1cs.HC

TL;DR

The paper addresses the need to measure undergraduate students’ reliance behaviors during Generative AI problem-solving without tying them directly to outcomes. It develops and validates a four-factor scale, with fit evidence from two problem-solving tasks, for measuring such behaviors across activities.

  • Problem

    The study addresses the need to measure undergraduates’ use of and reliance on Generative AI during problem-solving activities.

    Source: Discussion · p. 30

  • Method

    The authors used EFA and CFAs to develop and validate a scale of reliance behaviors in human-AI problem-solving.

    Source: Discussion · p. 30

  • Results

    The scale comprises four factors—reflective, cautious, thoughtless, and collaborative use—and showed good model fit across programming-game and policy-drafting tasks.

    Source: Discussion · pp. 30, 33

  • Takeaways & Limitations

    The scale measures reliance behaviors across activities without directly linking them to problem-solving outcomes.

    Source: Discussion · p. 30

    The scale was developed and validated with a relatively homogeneous undergraduate sample from one Asian university, which may limit generalizability.

    Source: Limitations · p. 34

Abstract

from arXiv · show

Reliance on AI describes the behavioral patterns of when and how individuals depend on AI suggestions, and appropriate reliance patterns are necessary to achieve effective human-AI collaboration. Traditional measures often link reliance to decision-making outcomes, which may not be suitable for complex problem-solving tasks where outcomes are not binary (i.e., correct or incorrect) or immediately clear. Therefore, this study aims to develop a scale to measure undergraduate students' behaviors of using Generative AI during problem-solving tasks without directly linking them to specific outcomes. We conducted an exploratory factor analysis on 800 responses collected after students finished one problem-solving activity, which revealed four distinct factors: reflective use, cautious use, thoughtless use, and collaborative use. The overall scale has reached sufficient internal reliability (Cronbach's alpha = .84). Two confirmatory factor analyses (CFAs) were conducted to validate the factors using the remaining 730 responses from this activity and 1,173 responses from another problem-solving activity. CFA indices showed adequate model fit for data from both problem-solving tasks, suggesting that the scale can be applied to various human-AI problem-solving tasks. This study offers a validated scale to measure students' reliance behaviors in different human-AI problem-solving activities and provides implications for educators to responsively integrate Generative AI in higher education.

Introduction

The introduction motivates a user-centered self-report scale to capture undergraduates’ reliance on Generative AI during complex problem-solving. It explains the limitations of outcome-based measures and introduces the measure and development of its questionnaire items.

Measures for Reliance on AI Systems

  • Existing reliance measures often use decision outcomes or advice adoption, but can miss complex problem-solving behaviors and may not distinguish sound reliance from following erroneous AI advice.Some approaches also depend on binary decisions and immediate correctness feedback, which may not be available in real-world or non-binary collaboration tasks.Source: Introduction · p. 4, Measures for Reliance on AI Systems · pp. 9–10

Introduction

  • The study proposes a nuanced self-report scale that maps Generative AI use across problem-solving stages to reveal reliance patterns that objective measures may not capture.The stages are problem identification, information exploration, and solution generation and presentation.Source: Introduction · p. 4

Problem-Based Learning and Problem-Solving

  • Problem-based learning centers on ill-structured, collaborative problem-solving that proceeds through problem identification, information exploration, and solution generation and presentation.These stages involve defining problems and knowledge gaps, iteratively researching and analyzing information, then developing and sharing solutions.Source: Problem-Based Learning and… · pp. 5–6

Human-AI Problem-Solving

  • Generative AI can support learning and problem-solving, yet inaccurate or biased outputs and passive acceptance can threaten critical thinking, so educators need to ensure it supplements rather than replaces students’ effort.Hamid et al. (2023) found enhanced collaboration, engagement, and motivation in a PBL setting, while noting that AI-generated information required human verification.Source: Human-AI Problem-Solving · p. 8

Measures for Reliance on AI Systems

  • Existing measures may miss implicit, context-dependent reliance in complex human–AI problem-solving because they assume performance can be estimated and humans and AI share an understanding.These measures may remain useful in structured decision-making, but do not fully address problem-solving complexity.Source: Measures for Reliance on AI Systems · pp. 10–11

The Current Study

  • The study developed and validated a self-report questionnaire to capture implicit, evolving reliance during exploratory problem-solving and help educators understand students’ balance of reasoning and AI assistance.Problem-solving involves exploration, iterative adjustments, and creative reasoning, and reliance may be difficult to observe directly.Source: The Current Study · p. 11

Questionnaire Items Development

  • Researchers developed 23 behavior-specific items from student reflections, mapping reliance on AI, self, or peers across problem-identification, information-exploration, and solution-generation and presentation stages.The item development drew on reflective responses from a similar problem-solving context (Authors, 2024).Source: Questionnaire Items Development · pp. 11–12
  • Items used a five-point frequency scale and underwent expert review for clarity; two items were flagged as nonspecific, but all 23 were administered after both activities.The review included two human–AI collaboration professors and three graduate student observers; wording changes emphasized student autonomy.Source: Questionnaire Items Development · p. 13

Questionnaire items mapped to the problem-solving stages and reliance categories

The questionnaire maps reliance behaviors across problem-solving stages and distinguishes AI reliance, self-reliance, and peer reliance. Its items cover students’ use and evaluation of AI, collaboration with peers, and the study’s undergraduate course setting.

Questionnaire items mapped to the problem-solving stages and reliance categories

Participants

  • Participants were first- and second-year students in a digital literacy course at an Asian university, grouped across tutorials for in-class activities.Source: Participants · p. 16
  • The study collected 1,818 and 1,406 weekly responses; ethics approval waived consent forms, participation was voluntary, and only student IDs were collected.Source: Participants · p. 16

Problem-solving Activities

Students completed two distinct Generative AI problem-solving activities: co-designing and solving 3D mazes, and drafting a policy proposal. The activities took place in a flipped course with weekly in-class work and post-activity reliance questionnaires.

Problem-solving Activities

  • In the 3D Maze activity, students created mazes with environmental assets and solved them using pseudocode; AI use was encouraged for both tasks.Source: Problem-solving Activities · pp. 16–17
  • In Policy Drafting, students proposed fair and ethical AI-use policies by reviewing university guidelines, brainstorming misuse cases, and evaluating proposals across disciplines.Students could use Generative AI freely throughout the activity.Source: Problem-solving Activities · p. 17

Procedures

  • The flipped course assigned preparatory videos and quizzes, followed by two hours weekly for in-class AI problem-solving activities.Source: Procedures · p. 17
  • In class, instructors reviewed key concepts, distributed activity handouts, emphasized interdisciplinary collaboration, and reminded students to complete the questionnaire afterward.Some students missed activities, ran out of time, experienced nonstandard activity designs, or did not complete the questionnaire.Source: Procedures · p. 17

Data Analyses

The analyses combined response screening, exploratory and confirmatory factor analyses, qualitative checks against chat histories, and cross-task comparisons. Careless responses were screened using longstring-index distributions and outlier exclusions before analysis.

Data Analyses

  • The analysis plan used EFA on part of the 3D Maze responses, CFAs across 3D Maze and Policy Drafting, chat-history checks, and descriptive analysis with a one-tailed t-test across tasks.The EFA explored factor dimensions; CFAs tested the structure and its reliability and validity across tasks; chat histories were used for qualitative validation.Source: Data Analyses · p. 18
  • Qualitative validation examined eight group chat histories, purposefully selected when a group’s average score exceeded 4.5 on one factor in either week.The researchers selected high-scoring groups because their behaviors were expected to show factor distinctions clearly.Source: Data Analyses · p. 18

Careless Responses

  • For careless-response screening, researchers calculated each response’s longstring index—the maximum run of identical answers—and plotted its distribution separately by activity.They calculated the index across all survey questions, not only the reliance items.Source: Careless Responses · p. 19
  • Because no unified longstring cutoff was recommended, the researchers excluded average-index outliers above 1.5 times the interquartile range above the third quartile.Source: Careless Responses · p. 19
  • After screening, 1,540 3D Maze surveys and 1,173 Policy Drafting surveys remained for analysis.Source: Careless Responses · p. 19

The Boxplots of the Average Longstring Index of Survey Responses

The section describes the survey-response screening and factor-analysis procedures used to develop and validate the reliance scale. It also specifies the fit and reliability criteria applied to the models.

The Boxplots of the Average Longstring Index of Survey Responses

The Exploratory Factor Analysis and Confirmatory Factor Analysis

  • Weakly correlated items R6 and R8 were excluded before exploratory factor analysis, which used maximum-likelihood factoring and oblique rotation.Cross-loading items were removed when both loadings exceeded .30 and differed by less than .15.Source: The Exploratory Factor Analysis and… · p. 20
  • Two confirmatory factor analyses used the remaining 740 3D Maze responses and 1,173 Policy Drafting responses.Source: The Exploratory Factor Analysis and… · p. 20
  • Acceptable fit was defined as CFI above .90, RMSEA below .08, and SRMR below .08; reliability was assessed with Cronbach’s alpha and mean inter-item correlations.The stated optimal range for mean inter-item correlations was .20 to .40.Source: The Exploratory Factor Analysis and… · p. 21

Results

The exploratory analysis identified four factors in students’ reliance behaviors, yielding a 16-item scale. The analysis also reported the screening criteria and the items removed.

Question Items?

  • Polychoric correlations were used for the 1-to-5 ordinal items, and Bartlett’s test and the overall KMO measure supported proceeding with factor analysis.Bartlett’s test: X2 (df = 190) = 6760.16, p < .05; overall KMO: .92.Source: Question Items? · p. 21
  • Parallel analysis suggested four reliance-behavior factors, each represented by four items in the resulting 16-item scale.Source: Question Items? · p. 22
  • R5 was removed for loading below .30 on all factors, while R7, R17, and R20 were removed for cross-loading; the final scale retained 16 items.Source: Question Items? · p. 22

The Correlation Matrix of Questionnaire Items

The scale’s four-factor structure showed adequate fit across two problem-solving activities, and the factors were interpreted as reflective, cautious, thoughtless, and collaborative use. Chat-history comparisons provided qualitative support, with collaborative use harder to verify because peer interactions often occurred offline.

The Correlation Matrix of Questionnaire Items

Types of Problem-solving Tasks Involving Generative AI?

  • The 3D Maze CFA had CFI = .89, RMSEA = .06, and SRMR = .06; the Policy Drafting CFA had CFI = .91, RMSEA = .06, and SRMR = .06, indicating adequate fit in both activities.Source: Types of Problem-solving Tasks… · p. 23
  • Reflective use involved revising prompts or solutions (α = .74), while cautious use involved recognizing and reacting to AI errors (α = .65).Cautious use’s mean inter-item correlation was in the stated optimal range, and removing an item did not improve its alpha.Source: Types of Problem-solving Tasks… · p. 24
  • Thoughtless use covered copying or pasting and relying on AI for problem-solving (α = .72); collaborative use covered peer communication and collaboration (α = .76), and overall scale alpha was .84.Source: Types of Problem-solving Tasks… · p. 24
  • Table 3 presents the finalized scale and Spearman correlations among its variables.Source: Types of Problem-solving Tasks… · p. 24
  • Chat histories showed reflective users adapting AI outputs, cautious users identifying errors or using AI minimally, and thoughtless users copying prompts, sometimes with placeholders unchanged.Source: Types of Problem-solving Tasks… · p. 24
  • Collaborative use was harder to validate from chat histories because such behavior often involved offline peer interactions; the qualitative checks supported the scale’s measurement of reliance behaviors.Source: Types of Problem-solving Tasks… · p. 25

RQ3: How Do Undergraduate Students’ Reliance Behaviors on Generative AI Differ Across

Reliance behavior scores differed between the 3D Maze and Policy Drafting activities for reflective, thoughtless, and collaborative use. Reflective use was most frequent overall, while cautious use was least frequent; the factors also showed reported associations.

Problem-solving Tasks?

  • Reflective use was the most frequent factor and cautious use the least frequent across students’ responses.Source: Problem-solving Tasks? · p. 25
  • Collaborative and reflective use were strongly correlated, while thoughtless use was moderately related to reflective use.Source: Problem-solving Tasks? · p. 25
  • Reflective use was higher in 3D Maze than Policy Drafting (M = 4.01, SD = .59 vs. M = 3.76, SD = .67; t(2284.3) = -9.95, p < .001).Source: Problem-solving Tasks? · p. 25
  • Thoughtless use was higher in 3D Maze than Policy Drafting (M = 3.61, SD = .69 vs. M = 3.15, SD = .82; t(2220.2) = -15.26, p < .001).Source: Problem-solving Tasks? · p. 25
  • Collaborative use was higher in 3D Maze than Policy Drafting (M = 3.92, SD = .62 vs. M = 3.74, SD = .67; t(2387.4) = -7.28, p < .001).Source: Problem-solving Tasks? · p. 25

The Finalized Scale of Reliance Behaviors during Human-AI Problem-Solving with Descriptive Results

The scale distinguishes four reliance behaviors, and reflective, thoughtless, and collaborative use differed between the two problem-solving tasks. The discussion presents these behaviors as context-dependent and notes limits in sample generalizability, item coverage, and validation.

The Finalized Scale of Reliance Behaviors during Human-AI Problem-Solving with Descriptive Results

  • The scale distinguishes reflective prompting and output critique, cautious error recognition, thoughtless copying, and peer collaboration.Its item examples include revising prompts, spotting AI errors, copying AI output into solutions, and asking peers for support.Source: The Finalized Scale of Reliance… · pp. 27–28

Discussion

  • The authors avoid labeling behaviors inherently good or bad because their value may differ across problem-solving stages and task demands.They contrast reflective behavior for deep understanding and critical thinking with thoughtless use that may not significantly affect outcomes in routine tasks.Source: Discussion · p. 33

The Finalized Scale of Reliance Behaviors during Human-AI Problem-Solving with Descriptive Results

  • Reflective use was most frequent and cautious use least frequent; collaborative use was strongly correlated with reflective use.Thoughtless use was also moderately related to reflective use, while thoughtless and cautious use were conceptually distinct.Source: Discussion · p. 33

The Difference in Undergraduate Students’ Reliance Behaviors in the Two Problem-Solving

  • Reflective, thoughtless, and collaborative use were higher in 3D Maze than in Policy Drafting, showing task-linked variation.The authors mention confidence, prior AI exposure, perceived ease, and attitudes toward collaboration as possible explanations, while noting that examining them was beyond the study's scope.Source: Discussion · p. 33

Limitations

  • The scale was validated with a relatively homogeneous undergraduate sample from one Asian university, which may limit generalizability.The authors call for more diverse samples across educational levels, cultural backgrounds, and learning environments.Source: Limitations · p. 34
  • Some problem-solving stages are represented by a single item, and Cautious Use reliability is .65, below the commonly used .70 benchmark.The authors recommend expanding subcategory items to better cover these behaviors and refine the instrument.Source: Limitations · p. 35
  • Chat-history validation compared self-reports with observable behavior only for high-scoring groups, leaving the full response range unexamined.The authors recommend broader sampling and analysis to further validate and refine the scale.Source: Limitations · p. 35

Implications

  • Educators can use scale results to identify high thoughtless use and tailor tasks to encourage critical, balanced reliance on AI and peers.The scale may also support research linking reliance behaviors with learning outcomes and skills such as problem-solving and critical thinking.Source: Implications · pp. 35–36

Problem-solving Activity Requirement and Sample Prompts for Students

Students worked on two problem-solving activities: drafting university guidance for GenAI use and designing and solving a 3D maze. Sample prompts and requirements guided the tasks while leaving room for students to use GenAI.

Problem-solving Activity Requirement and Sample Prompts for Students

  • For the maze-design task, groups created a solvable maze at least 10 units wide and long, using required environmental assets and optionally collectible gems.The prompt also specified start, end, path, obstacle, monster, gem, and heart symbols.Source: Problem-solving Activity Requirement… · p. 46
  • For maze solving, groups could choose how and when to use GenAI, including generating or correcting pseudocode built from specified actions, conditions, and control blocks.The listed actions include turning, moving forward, and attacking; the control blocks include while and if/else loops.Source: Problem-solving Activity Requirement… · p. 47
  • The policy-drafting activity asked student representatives to suggest ways to promote ethical GenAI use and minimize misuse across campus.The prompt framed this around balancing students’ critical thinking, creativity, and agency with exploration of AI innovations.Source: Problem-solving Activity Requirement… · pp. 48, 50
  • A sample policy prompt asked students to use three Socratic questions to help a team examine and critically evaluate its preliminary suggestion.The questions were described as helping learners distinguish what they know from what they do not understand.Source: Problem-solving Activity Requirement… · p. 48

Appendix B

Appendix B is identified as an illustration of the 3D Maze platform.

Appendix B

  • Appendix B labels its visual as an illustration of the 3D Maze platform.The supplied text does not specify what the illustration depicts beyond the platform.Source: Appendix B · pp. 50–51

Related papers

Loading 2610.30195v1…