Source-linked AI summary
Metacognitive Scaffolds Create Productive Friction for AI-Assisted Knowledge Work
Viktor Kewenig, Lev Tankelevitch, Sean Rintel, Advait Sarkar, Abigail Sellen
TL;DR
Knowledge workers increasingly steer AI rather than produce artifacts, creating metacognitive demands that current interfaces poorly support. This study tests scaffolds for goal specification and task decomposition in AI-assisted writing, finding better writing and prompting, transfer and recall benefits, alongside increased perceived effort and reduced helpfulness.
Problem
As AI shifts work toward steering and orchestrating model behavior, workers face metacognitive demands that current interfaces poorly support.
Source: §1 · p. 2Method
The study tests dynamic scaffolds for goal specification and task decomposition in AI-assisted writing against a standard chat-and-canvas interface.
Source: §3.2.2 · p. 9, §6 · p. 23Results
Scaffolding improved writing quality and prompting, with writing gains persisting after removal and evidence of improved topic recall, while participants reported greater cognitive load and lower helpfulness.
Source: §1 · p. 3, §5 · pp. 19–20Takeaways & Limitations
Designers should make the value of productive friction visible while supporting users’ judgment and meaningful contribution when working with AI.
Source: §6 · p. 23Recall findings are imprecise, use a lenient topic-identification rule, and are limited by substantial follow-up attrition that may bias the sample.
Source: §5.5 · p. 23
Abstract
from arXiv · showhide
Generative AI imposes new metacognitive demands on knowledge workers as their work shifts from production to planning, evaluation, and orchestration. In a randomized, between-subjects experiment on AI-assisted writing (297 workers, 594 essays), we test dynamically-generated metacognitive scaffolds supporting goal specification and task decomposition. We find that scaffolds improve writing quality, with this boost persisting in short-lag transfer when scaffolds are removed. We also find evidence of improved topic recall three days later. These effects are consistent with observed changes in how people learn to prompt AI systems after exposure to scaffolding. However, scaffolded participants rate the AI as less helpful and more cognitively demanding even as they perform better, suggesting that subjective ease can understate the value of support. Our findings make the case for metacognitive scaffolds in AI-assisted knowledge work, revealing a productive friction worth designing for as workers increasingly delegate work to agentic AI.
1 Introduction
As AI shifts knowledge work from producing artifacts to steering and evaluating them, workers face metacognitive demands that current interfaces poorly support. This study tests dynamic scaffolds for goal specification and task decomposition, finding improved writing quality and transfer alongside greater perceived effort and lower helpfulness.
1 Introduction
- As AI takes over more material production, people increasingly steer and orchestrate its work, making goal setting, task decomposition, and evaluation central human contributions.Insufficient support for these processes risks poorer AI steering and reduced cognitive engagement with the work.Source: §1 · p. 2
- Prior work on scaffolds has largely measured proximal usability and steerability, leaving output quality, persistence after support removal, and cognitive engagement less examined.This study addresses those more distal outcomes.Source: §1 · p. 2
- In a randomized between-subjects AI-assisted writing experiment, 297 participants completed 594 essays while using dynamic scaffolds for goal specification and task decomposition.The study examined output quality, prompting, cognitive load, and topic recall.Source: §1 · p. 2
- Scaffolding improves written output quality, with gains persisting after support is removed and accompanied by clearer prompting and evidence of improved topic recall.Participants also rated the AI as less helpful and reported greater cognitive load.Source: §1 · p. 3
- The paper contributes evidence on dynamic scaffolding for goal specification and task decomposition, including writing and prompt quality, short-lag transfer, topic recall, and perceived trade-offs.The reported trade-offs include increased cognitive load and reduced perceived helpfulness.Source: §1 · p. 3
2 Related Work
Generative AI shifts some knowledge work from producing artifacts toward directing and evaluating model output, while research on offloading and scaffolding leaves important questions about engagement and distal work outcomes. Prior work informs the study’s focus on early metacognitive support, its possible effects, and the limits of what a single-session evaluation can establish.
2.1 Generative AI’s Reshaping of Knowledge Workflows
- When AI produces whole artifacts, human contribution shifts toward specifying objectives, planning, selecting outputs, and critically evaluating them.This raises questions about which cognitive tasks remain essential and how people engage with their work.Source: §2.1 · p. 4
2.2 Cognitive Offloading and Engagement
- AI offloading has been linked to both weakened recall and learning in some settings and improved later unaided performance with structured engagement, making cognitive task type and timing consequential.Topic recall is presented as a simple indicator of individual cognitive engagement, and early-stage scaffolding may support engagement.Source: §2.2 · pp. 4–5
2.3 Metacognitive Demands of Generative AI
- Delegating production shifts effort toward planning, verification, and the metacognitive work of deciding what to delegate and how to evaluate AI output.Longer autonomous task horizons leave fewer opportunities for intervention and increase the consequences of initial specifications and later verification.Source: §2.3 · p. 5
2.4.1 Goal and Intent specification.
- Goal-support systems have helped users articulate, explore, clarify, and manage intentions, with dynamic, activity-contingent support a recurring design pattern.Examples include Luminate, Intent Tagging, OnGoal, Dynamic Prompt Middleware, and IntentFlow.Source: §2.4.1 · pp. 5–6
2.4.2 Task Decomposition.
- Task-decomposition systems externalize complex structure through editable workflows, phasewise or stepwise plans, and visual tokens that support steering and verification.The current scaffold applies a related principle to goals and task structure in writing.Source: §2.4.2 · p. 6
2.4 Metacognitive Scaffolds for Goal and Task Specification in AI-Assisted Workflows
- Prior scaffold research reports gains in interaction outcomes and some task performance, but mixed effects on cognitive load; this study asks about more distal work outcomes.The study targets objective output quality, subsequent interaction patterns, and topic recall.Source: §2.5 · p. 6
2.5 Synthesis and Current Study
- The study evaluates goal-specification and task-decomposition scaffolds against standard chat-and-canvas interaction, asking about writing quality and transfer, prompting, recall, and user perceptions.Its between-subjects design aims to isolate scaffold effects, while durable skill change and in-depth engagement exceed what one session can establish.Source: §2.5 · p. 6
3 Methods
The study compared dynamically generated metacognitive scaffolds with a familiar chat-and-canvas interface in a two-task AI-assisted writing experiment. Measures covered writing quality, prompting, self-reports, subjective experience, and later topic recall.
3 Methods
- 297 participants completed two professional-writing tasks; scaffold access was available in Task 1, while both groups used the unscaffolded interface in Task 2 to test short-lag transfer.The final analytic sample included 144 controls and 153 scaffolded participants, producing 594 essays.Source: §3 · p. 7, §3.1 · p. 7
3.2.1 Dynamic Metacognitive Scaffold.
- The dynamic scaffold first elicited editable goal specifications, then generated task-specific controls for analytical approaches, structure, and critical considerations.Controls updated from the participant’s request, selections, and work phase; selected controls informed Task 1 only.Source: §3.2.1 · pp. 8–9
3.2.2 Baseline Interface.
- The control group used the same chat-and-canvas workflow without scaffold panels, but could still clarify goals and decompose tasks through ordinary prompting.This naturalistic baseline tested the total effect of adding the scaffold while preserving user agency.Source: §3.2.2 · p. 9
3.2 Research Probe and Interface Design
- Participants wrote a city-planning proposal and a hybrid-work consulting brief in a browser environment connected to GPT-5.3, with free prompting, manual editing, and version histories.Tasks were self-paced up to 80 minutes, and participants could manually review documents before submission.Source: §3.2 · p. 8, §3.3 · p. 9
3.4.1 Essay quality.
- Writing quality was the primary outcome: GPT-5.4 rated de-formatted essays on a seven-item rubric, with ratings compared against a human benchmark.De-formatting removed visual cues while preserving lexical content and sentence order; the benchmark comprised two human ratings for each of 117 essays.Source: §3.4.1 · pp. 9–10
3.4.2 Prompt Behavior.
- Task 2 prompt sequences were coded for goal clarity and task decomposition; coders rated the full sequence rather than isolated messages.The transfer analysis used ordinal logistic regression on the two component scores and their combined score, adjusted for six baseline covariates.Source: §3.4.2 · p. 10, §3.4.6 · p. 11, §3.5.4 · p. 12
3.4.3 Metacognitive Self-Report.
- Self-report, AI helpfulness and control, NASA-TLX workload, and free recall of both writing topics three days later provided complementary outcome measures.Recall responses were manually coded as binary correct/incorrect, with lenient credit for identifying either assigned topic; follow-up attrition was high, prompting multiple-imputation sensitivity analysis.Source: §3.4.3 · p. 10, §3.4.4 · p. 10, §3.4.5 · p. 11
3.5.1 Writing quality.
- Analyses adjusted for six baseline indicators and included outcome-specific models; writing-quality models accounted for repeated tasks, order, condition, and scenario.The primary writing-quality model omitted Task 1 time because planning time could be part of the intervention; adding time left contrasts essentially unchanged.Source: §3.4.6 · p. 11, §3.5.1 · p. 11, §3.5.2 · pp. 11–12
4 Results
Scaffolding improved essay quality during use and after removal, alongside stronger later prompting and topic recall. Participants also reported lower helpfulness and control, and higher cognitive load while using the scaffold.
4.1.1 Overall Essay Quality.
- b=0.089 (95% CI [0.023, 0.155], P=.008) for essay quality during scaffold use, and b=0.070 (95% CI [0.004, 0.136], P=.037) after removal.Both adjusted contrasts were statistically reliable; Figure 3 shows essay-level scores and condition means across the two tasks.Source: §4.1.1 · p. 13, Figure 3 · p. 14
4.1.2 Individual Rubric Analysis.
- Human ratings supported the quality advantage in both tasks, while highlighting feasibility and organization differently from the full-corpus judge’s emphasis on argument quality, evidence credibility, and impact.The rubric-level analyses were exploratory and used uncorrected P values; the human benchmark included 107 essays.Source: §4.1.2 · pp. 15–16, Table 2 · p. 15
4.2 The scaffold improved later prompting
- 14.43 versus 11.07 of 39: previously scaffolded participants scored higher on combined goal clarity and task decomposition in unscaffolded Task 2.The figure reports reliable condition differences for the combined score (P=.0067), goal clarity (P=.0036), and task decomposition (P=.0431).Source: §4.2 · p. 16, Figure 4 · p. 17
4.3 The scaffold was associated with improved short-lag topic recall
- In the complete-case follow-up, topic recall was higher for scaffolded participants in Task 1 (OR=5.07, 95% CI [1.21, 21.25], P=.026) and Task 2 (OR=4.24, 95% CI [1.02, 17.57], P=.046).The voluntary survey occurred approximately three days after the session, and 159 participants supplied scored responses for both tasks.Source: §4.3.1 · p. 17, §4.3 · p. 17, Figure 5 · p. 18
4.3.2 Multiple imputation analysis.
- After multiple imputation, recall effects remained positive for Task 1 (OR=3.24, 95% CI [1.11, 9.49], P=.032) and Task 2 (OR=3.06, 95% CI [1.04, 9.02], P=.042).The positive pattern therefore extended to recall of the task completed after scaffold removal.Source: §4.3.2 · p. 17
4.4.1 Metacognitive self-report.
- Scaffolded participants more strongly reported breaking the task into smaller steps (b=0.753, 95% CI [0.546, 0.959], PBH<.001); no other item-level effect survived correction.The staged composing-process item’s positive association did not survive correction.Source: §4.4.1 · p. 18
4.4.2 Helpfulness and perceived control.
- Scaffolded participants rated the AI as less helpful (b=-0.342, 95% CI [-0.510, -0.174], P<.001) and reported lower perceived control (b=-0.251, 95% CI [-0.486, -0.017], P=.036).Both ratings moved opposite to the objective Task 1 quality effect.Source: §4.4.2 · p. 18
4.4.3 Cognitive load.
- During Task 1, scaffolded participants reported higher load than controls (M=4.71 vs 3.96; b=0.745, 95% CI [0.381, 1.110], P<.001), while the Task 2 difference was not statistically reliable.The additional load was localized to scaffold use, with burden including managing the AI, stress or frustration, and rushed pace.Source: §4.4.3 · pp. 18–19
5 Discussion
The discussion argues that dynamic metacognitive scaffolds improve AI-assisted writing and prompting, with gains extending to an unscaffolded task and topic recall, despite greater perceived effort and lower helpfulness. It frames this effort as potentially productive friction while emphasizing limits on the evidence and the need to make the value of scaffolding visible.
5.1.1 How people approach the task.
- Scaffolded participants specified goals and decomposed tasks more in their prompts, and writing quality improved across tasks, although they did not report setting clearer goals.Task decomposition was the most reliable effect, including after scaffold removal; participants reported more task structuring but not clearer goals.Source: §5.1.1 · p. 20
5.1.2 How people work with the AI.
- Participants exposed to scaffolds later wrote better essays and clearer, better-structured prompts on their own, indicating short-lag transfer.The authors do not argue that support must be removed, but note that it cannot be guaranteed across knowledge-work contexts.Source: §5.1.2 · p. 20
5.1.3 How people understand and engage with the work itself.
- Scaffolded participants were more likely to recall writing topics three days later, but this comparison does not show that scaffolds reverse AI-related retention effects versus unaided work.The recall contrasts were imprecise, based on the returning half of the cohort, and topic recall is only a narrow measure of cognitive engagement.Source: §5.1.3 · pp. 20–21
5.2 Divergence Between Subjective and Objective Measures
- Participants performed better while rating the AI as less helpful; they reported task decomposition but not clearer goals, despite prompts showing gains in both.The authors suggest decomposition may be more visible because it leaves inspectable traces, while clearer goal-setting may be less noticeable to the person doing it.Source: §5.2 · p. 21
5.3 Designing productive friction with visible value into AI assistance
- The authors argue that friction should be designed as a material rather than treated only as a cost, with its value made legible during work, across workflow phases, and in outcomes.Participants’ comments suggest that front-loaded specification can be costly under time pressure, and that scaffolds should stage choices and pair planning support with revision support.Source: §5.3 · pp. 21–22
5.4 Supporting Human Metacognition as Agentic Autonomy Grows
- As agentic systems take on longer task chains, the authors argue that people will still need to direct and verify work, making metacognitive regulation important.They propose scaffolds that fade as relevant skills are internalized, helping workers specify goals and break work into checkable parts.Source: §5.4 · p. 22
5.5 Limitations
- Essay-quality estimates rely partly on an LLM judge whose agreement with human ratings was moderate and whose scores compressed the human range.The study also lacked a static-scaffold condition and tested goal specification and task decomposition together, so their distinct effects cannot be separated.Source: §5.5 · pp. 22–23
- Recall was scored leniently and had substantial follow-up attrition; the transfer findings cover a subsequent unscaffolded task, not persistence over weeks or months.The study used one-shot online writing tasks with U.S.-based Prolific adults and could not separate effects by experience.Source: §5.5 · p. 23
6 Conclusion
The conclusion argues that dynamic scaffolds for goal specification and task decomposition can improve AI-assisted writing while increasing perceived effort and reducing perceived helpfulness. It calls for designing around this productive friction and users’ ability to calibrate their work with and without AI.
6 Conclusion
- The scaffold improved writing and prompting despite users perceiving more effort and less helpfulness, motivating design for productive friction.The conclusion also calls for attention to users’ capacity to calibrate work practices and outcomes with and without AI.Source: §6 · p. 23
A.1 Dynamic scaffold interface
The scaffold staged support in two phases: first making the writing goal explicit, then offering editable, task-specific planning controls. This dynamic design responded to the user’s request and stage of work rather than presenting a static checklist.
A.1 Dynamic scaffold interface
- The interface staged support to make the goal explicit before helping users plan how to break down the work.Its sequencing was intended to mirror a simple logic of knowledge work: first specify the goal, then decide how to decompose the task.Source: §A.1 · p. 30
- Goal Clarity asked users to specify their message, audience, intended reader outcome, writing objective, and desired scope or depth.Editable fields made these goal parameters inspectable before substantial text generation began.Source: §A.1 · p. 30, Figure A1 · p. 30
- After goal specification, the system generated scenario-specific planning controls, including analytical approaches, essay structure, and critical considerations.Participants could accept or revise controls, or request alternatives in a natural-language box.Source: §A.1 · pp. 30–31, Figure A2 · p. 31
- The two phases made metacognitive support dynamic and editable, using goal clarity to constrain the target and decomposition to organize execution and evaluation.The scaffold addressed how users represent and manage the task, not only what they ask the model.Source: §A.1 · p. 31
A.2 Scenario briefs
The appendix describes two persuasive knowledge-work scenarios and the materials used to evaluate essays. The scenarios require participants to consider trade-offs, audience, and the justification of recommendations.
A.2 Scenario briefs
- The scenario briefs asked participants to write a city-planning proposal about adopting a four-day work week or a consulting brief on launching hybrid work.The tasks called for persuasive knowledge work involving trade-off analysis, audience awareness, and justified recommendations or client outcomes.Source: Figure A3 · p. 32, Figure A4 · p. 33
B.1 Essay-scoring rubric
- The essay-scoring rubric comprised seven shared criteria, labeled A–G, each rated on a 1–4 scale by human raters and supplied to the LLM judge.Table A1 presents the rubric criteria.Source: §B.1 · p. 34, Table A1 · p. 34
B.2 Human-rating interface
- The human-rating interface supported scoring essays and citing the evidence in them that supported those scores.The interface is shown in Figure A5.Source: Figure A5 · p. 35
C Prompt-coding codebook and coder agreement
The appendix defines a conversation-level codebook for rating goal clarity and task decomposition in participant prompts, with indicators and decision rules for each construct. It also documents the coding workflow, examples, edge cases, and reliability materials.
C.1 Prompt-coding codebook
- The prompt-coding codebook was designed to capture how explicitly conversations state writing goals and decompose tasks into manageable sub-goals.It drew on writing-process accounts, task-analysis approaches, and the 4D AI Fluency framework.Source: §C.1 · p. 36
- A 20% subset of prompt sequences—60 conversations—was independently coded by three researchers and an LLM coder to assess reliability and agreement.Table A2 reports original categorical-codebook ratings, human–essay-judge agreement, and agreement across five essay-judge runs; the graded analysis scoring is described in Methods.Source: §C.1 · p. 36, Table A2 · p. 36
C.2 A. Goal clarity
- Goal clarity measures how explicitly a full prompt sequence specifies the writing purpose, audience, and constraints.Indicators include purpose, audience, constraints, evaluative criteria, genre, rhetorical situation, and process goal; ratings range from implicit goal (GC-1) to explicit purpose plus at least two additional indicators (GC-3).Source: §C.2 · p. 37, Table A3 · p. 37
C.3 B. Task decomposition
- Task decomposition measures whether prompts divide a writing task into identifiable sub-tasks, steps, or structured parts.Indicators include sub-goals, sequencing, granularity, dependencies, delegation, and iteration; the highest rating requires sub-goals and sequencing plus at least two additional indicators.Source: §C.3 · p. 37, Table A4 · p. 38
C.4 C. Worked examples
- Worked examples distinguish an underspecified prompt, a goal-rich but undecomposed prompt, and an iterative prompt sequence with explicit goals and hierarchical decomposition.The examples are coded GC-0/TD-0, GC-3/TD-0, and GC-3/TD-3, respectively.Source: §C.4 · p. 38
C.5 D. Coding procedure and edge cases
- Coders rated the full prompt sequence and counted information across turns, including goals stated early and constraints added later.They capped inferred goals at GC-1, excluded bullets used only for readability from task decomposition, and counted AI-proposed structure only when users actively took it up.Source: §C.5 · p. 38
C.6 E. Blank coding sheet
- The appendix includes a blank coding sheet for the prompt-coding procedure.Table A5 is identified as the blank coding sheet.Source: Table A5 · p. 39
D Survey items
The supplementary survey materials document the timing and wording of measures administered across the study, from baseline and post-task surveys through a three-day follow-up.
D Survey items
- Table A6 lists survey items in administration order across baseline, post-Task 1, post-Task 2, and three-day follow-up surveys.Item numbers align with those used in Tables A11 and A12.Source: Table A6 · p. 40, D Survey items · p. 40
- Task 1 measures include self-reported metacognition, perceived AI helpfulness and control, and workload items about writing and AI-related demands.The workload block includes mental demand, effort, AI burden, prompting, and integration of AI suggestions.Source: D Survey items · p. 41
- The post-Task 2 survey asks about workflow and future AI use and repeats workload items referring to the second essay.It also includes self-reported prompting ability covering stepwise prompting, prompt formats, constraints, and tailored examples.Source: D Survey items · pp. 41–42
- The three-day follow-up measures topic recall, free recall of proposal content, recognition of topic-specific benefits, and a reported surprising insight.The topic prompts ask participants to recall their essay topics and proposal details.Source: D Survey items · p. 42
E Supplementary results: essay quality
The supplementary essay-quality materials report robustness models, judge–human agreement, and standardized essay-quality effect sizes.
E Supplementary results: essay quality
- Table A7 presents adjusted mixed-effects models for primary and time-on-task robustness analyses, controlling for six common baseline covariates.The caption does not provide model estimates.Source: Table A7 · p. 43
- Table A8 compares LLM-judge and human scores on 107 human-rated essays not used as calibration examples across seven shared rubric dimensions.It reports essay-level mean agreement statistics and a calibration slope, where a slope below 1 indicates range compression by the judge.Source: Table A8 · p. 43
- Table A9 is titled as a summary of standardized essay-quality effect sizes using the de-formatted judge for 297 participants.The supplied caption does not specify effect-size estimates.Source: Table A9 · p. 43
F Supplementary results: topic recall and follow-up missingness
The supplementary materials describe models for topic recall and missing recall records, alongside item-level self-reported metacognition.
F Supplementary results: topic recall and follow-up missingness
- Table A10 summarizes topic-recall odds ratios for scaffold versus control in Task 1 and previously scaffolded versus control in Task 2.The caption specifies logistic mixed models with a participant random intercept, task position, condition interaction, scenario, and six common covariates.Source: Table A10 · p. 44
- Table A11 describes a joint Firth-penalized model of missing recall records using complete predictor data from 234 participants.Its caption defines odds ratios as changes in missingness odds per standard deviation for continuous predictors or by coded category for binary predictors.Source: Table A11 · p. 44
- Task 2 prompt count had OR 1.06 [0.76, 1.49] (p=.712), Task 1 essay length had OR 0.93 [0.66, 1.31] (p=.673), and Task 2 essay length had OR 0.98 [0.71, 1.37] (p=.920) in the missing-record model.The model is Firth-penalized logistic regression of missing scored recall records.Source: F Supplementary results: topic recall… · p. 45
- Table A12 reports item-level self-reported metacognition before and after Task 1.The supplied caption does not provide item-level results.Source: Table A12 · p. 46
G Supplementary results: subjective outcomes
The supplied supplementary result identifies a table of subjective-outcome summaries after Task 1.
G Supplementary results: subjective outcomes
- Table A13 summarizes subjective outcomes measured after Task 1.The supplied caption does not specify the outcomes or their estimates.Source: Table A13 · p. 46