PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-AI Conversational Behaviors Predict Unassisted Task Performance

Li Siyan, Federico Bianchi, James Zou, Kaitlyn Zhou

arXiv:2610.09547v1cs.HC

TL;DR

The paper examines how conversational behavior during AI-assisted problem-solving relates to what people can do without AI. Across its studies, greater cognitive participation tracks better independent performance, while direct requests for next moves track poorer transfer; the authors limit their claims to immediate near-transfer on related puzzles.

  • Problem

    The paper asks how conversational behaviors relate to unassisted performance after assistance ends, amid concerns that AI’s immediate benefits may entail future costs.

    Source: §1 · p. 1

  • Method

    The paper connects conversational behaviors during AI-supported problem-solving with later performance and examines how interaction patterns relate to assistance received.

    Source: §1 · p. 3, §8 · p. 20

  • Results

    Participants’ unassisted performance tracked reasoning behaviors: maintaining cognitive participation accompanied better independent performance, while more direct requests for next-best moves accompanied poorer transfer.

    Source: §8 · p. 20, §11 · p. 24

  • Takeaways & Limitations

    The findings suggest that preserving users’ cognitive participation may support performance beyond AI-assisted interaction.

    Source: §11 · p. 24

    The studies measure immediate near-transfer to related puzzles in the same game type, not delayed retention or far transfer, and do not claim durable learning outcomes.

    Source: §10 · p. 24

Abstract

from arXiv · show

Assistance from AI tools has supported and improved human performance across domains. However, recent research suggests that these immediate benefits may entail future costs, including diminished performance when AI assistance is no longer available. We study how human-AI interaction behaviors correlate with immediate and future unassisted task performance across two game-based problem-solving user studies ($n=139$ and $n=111$), using a \textit{dialogue act} framework adopted from a tutor-student dialogue taxonomy. In our studies, verbalizing thought processes correlates with higher unassisted outcomes, whereas directly requesting solutions correlates negatively. Similar to these participant-side patterns, assistant explanations of the current problem state are associated with better subsequent unassisted performance, whereas directly providing the next action is associated with worse performance. Qualitative and subtype analyses further show that ostensibly similar reasoning turns can elicit different assistance. Our findings suggest that preserving users' cognitive participation in problem-solving may support performance beyond AI-assisted interaction.

1 Introduction

The introduction asks which freely emerging conversational behaviors track unassisted performance after AI support ends. Across two studies, it relates user and assistant dialogue behaviors to later performance and presents cognitive participation as a potentially important part of effective AI use.

1 Introduction

  • The paper asks which conversational behaviors track performance retention when users interact freely with an assistant that permits multiple forms of help.It frames this as complementary to prior work on system-imposed scaffolding.Source: §1 · p. 3
  • Using a dialogue-act framework, the authors examine conversational actions as recurring patterns that distribute reasoning work between users and assistants.The framework treats utterances as communicative actions as well as information carriers.Source: §1 · p. 3
  • The studies measure near-transfer on unseen, similar tasks, such as a new board position with related motifs from the same game.This distinguishes near-transfer from applying skills in radically different domains.Source: §1 · p. 3
  • Articulating reasoning correlated positively with unassisted performance (ρ = +0.36, p < .001), while directly requesting solutions correlated negatively (ρ = −0.22, p = .019).Assistant explanations of the current problem state were positively related to subsequent unassisted performance (ρ = +0.227, p = .017), whereas giving the next action was negatively related (ρ = −0.243, p = .010).Source: §1 · p. 3
  • The work connects conversational behavior during AI-supported problem-solving to later near-transfer and to theories of self-explanation and help-seeking.It reports that these distinctions also hold for adults using a general-purpose assistant that can supply answers on request.Source: §1 · p. 3
  • The authors report that users’ conversational behavior is related to the form of assistance they receive, and that thinking-aloud turns are not homogeneous.Qualitative and quantitative subtype analyses find that turns with the same label can prompt meaningfully different assistant behaviors.Source: §1 · p. 3

2 Related Work

Prior work links conversational behavior and cognitive engagement with collaboration and learning outcomes, but much conversation analysis focuses on immediate outcomes or educational settings. The paper builds on these literatures to examine naturally emerging interaction strategies and subsequent unassisted performance.

2.1 Conversational Behavior and Reliance in Human-LLM Interaction

  • Prior work connects human-LLM conversational behavior with outcomes including satisfaction, task success, and divisions of responsibility between users and models.Some studies also link conceptually oriented engagement with better outcomes than solution-oriented use.Source: §2.1 · p. 4
  • Most conversational analyses focus on immediate outcomes such as task success, reliance, and helpfulness; transfer studies have largely examined educational settings.This leaves open how naturally emerging strategies relate to retained performance among broader user populations.Source: §2.1 · p. 4
  • The paper examines how conversational behaviors during AI-assisted problem-solving relate to later unaided accomplishment and shape the assistance users receive.Source: §2.1 · p. 4

2.2 Cognitive Engagement and Help-Seeking in AI-Assisted Learning

  • Tutoring research often preserves learners’ cognitive engagement through prompts, hints, and scaffolding; prompting explanations has been linked to greater understanding and transfer.Self-explanation research also reports moderate learning improvements across instructional contexts, with learner-generated explanations sometimes outperforming instructional ones.Source: §2.2 · p. 4
  • Research on general-purpose LLMs reports that unrestricted access can improve performance during assistance yet impair later unassisted performance, while pedagogical assistance preserving engagement avoids much of this harm.Help-seeking research distinguishes requests for enough information to continue independently from requests to have the problem or next step completed.Source: §2.2 · pp. 4–5

3 Methods: User Study Design

The methods use controlled game puzzles to relate conversations during AI-assisted play to subsequent performance on similar puzzles without assistance. The study design includes a prompting-proficiency screener, an assistant with fixed underlying solutions, and a common procedure across the game-based studies.

3 Methods: User Study Design

  • The studies pair an AI-assisted game puzzle with subsequent independent puzzles to measure near-transfer, while controlling assistant-response correctness as a potential confound.The design also includes surveys about engagement and participant experience.Source: §3 · p. 5

3.1 Task Design Desiderata

  • Connect Four and Othello puzzles were chosen for objective move scoring and similar but unseen variants that require related reasoning without sharing the same solution.End-game puzzles standardize duration; optimal opponent strategies were precomputed, and gameplay simulations scored optimal moves and Othello disc counts.Source: §3.1 · pp. 5–6

3.2 Prompt Creation Screener

  • Participants completed a fictional-scenario prompt-writing task as a proxy for AI interaction competence, and those meeting the screener threshold qualified.LLM judges’ scores were validated against human annotations, with substantial agreement reported (ICC(2,1) = 0.787).Source: §3.2 · p. 6

3.3 AI Assistant Design

  • The assistant was prompted to act as a coach but could provide direct answers on request; it received the participant’s next optimal move and board image to reduce response-error variation.Holding the underlying solution constant allowed the study to examine how conversation strategies shaped the form of assistance.Source: §3.3 · p. 6

3.4 Study Procedure

  • Participants completed eligibility checks and the prompt screener, attempted an initial puzzle, used assistance according to condition, and then solved near-transfer puzzles without AI.They completed workload and experience questionnaires, with feedback on move optimality delayed until after the corresponding workload questionnaire.Source: §3.4 · p. 7

4 Discovery Study Results: Connect Four

In Connect Four, Solvers more often verbalized their thinking and coupled solution requests with reasoning. Think Aloud prevalence was associated with higher unassisted CFU scores, including after adjustment for measured pre-assistance factors.

4.1 Recruitment

  • The Connect Four analysis included 139 participants who had produced conversational turns; 270 were recruited, 165 completed the study, and 26 were excluded for no turns.Participants were recruited from Prolific’s “Qualified AI Taskers” pool.Source: §4.1 · p. 7

4.2 Participant Partition

  • Solvers were participants who made at least two optimal moves (n=79); all others were Strugglers (n=60).The authors used this binary partition because the outcome ranged only from 0 to 3, to establish preliminary trends for later studies.Source: §4.2 · p. 8

4.4 Thinking Aloud Is Associated with Better Unassisted Transfer

  • Solvers more frequently verbalized their thought processes than Strugglers (Mann-Whitney U=2946.5, p=.011; Figure 2).The study compared dialogue acts exploratorily, without prespecified hypotheses or correction for multiple comparisons.Source: §4.4 · p. 9, Figure 2 · p. 9
  • 8.6% of Solvers’ solution requests, versus 1.4% of Strugglers’ requests, included thinking aloud (Mann-Whitney U=2742.0, p=.005).This suggests the group difference may concern whether participants contribute their own reasoning while asking for help, not simply whether they request assistance.Source: §4.4 · p. 9
  • β=+0.380, p=.018 for Think Aloud DAP after adjustment for pre-assistance measures, with the association unchanged by those measures.Think Aloud DAP was unrelated to the measured unaided first-move correctness, confidence, and self-rated skill; the adjustment does not rule out a broader disposition to engage with the board.Source: §4.4 · p. 10, Table 3 · p. 10

5 Replication Study Results: Othello

The Othello replication addressed prior limitations with broader outcome measurement, minimum assisted-turn requirements, two unassisted puzzles, and an all-unassisted baseline. It examined how standard AI assistance and dialogue behaviors related to assisted performance and unassisted transfer.

5.1 Relationships between Conversational Behaviors and Unassisted Performance Replicate in Othello

  • Think Aloud correlated positively with transfer (ρ= +0.36, bootstrapped 95% CI [+0.20, +0.50], Holm-adjusted p< .001), while Solution Request correlated negatively (ρ= −0.22, 95% CI [−0.40, −0.03], Holm-adjusted p= .019).Both prespecified behavioral relationships replicated in Othello (n=111).Source: Figure 3 · p. 11, §5.1 · p. 11

5.2 Solution Request is Associated with Stronger Assisted Performance

  • Solution Request correlated with higher assisted disc counts (ρ= +0.375, p< .001), while Think Aloud was unrelated to assisted performance (ρ= −0.052, p= .60).These patterns contrasted with the behaviors’ opposite associations with transfer.Source: §5.2 · p. 12

5.3 Assistant-side Behavior Links to Unassisted Performance

  • Assistant Board Report correlated positively with transfer (ρ= +0.227, p= .017), whereas Provide Correct Answer correlated negatively (ρ= −0.243, p= .010).These exploratory correlations were not corrected across five assistant acts.Source: §5.3 · p. 12
  • Knowledge Deficit Questions increased the likelihood of Board Report, but most often elicited General Principle; retrospective questions were more likely to elicit Board Report (OR= 3.13, p< 10−6).Human-coder agreement was κ= 0.83 for retrospective orientation, but only κ= 0.51 for prospective orientation and κ= 0.22 for current-state orientation.Source: §5.3 · pp. 12–13
  • Counterfactual rewrites increased direct answers for next-move requests and Board Reports for retrospective questions across all four assistant models.This supports a narrow causal claim about immediate assistant responses, not participants’ later unassisted performance.Source: §5.3 · p. 13

5.4 AI Assistance Improves Assisted Performance but Does Not Reliably Improve Transfer

  • Only participants who thought aloud on at least 40% of turns exceeded baseline, by +1.78 discs per unassisted game (95% bootstrap CI [+0.69, +2.84], d= 0.59).This was the sole prevalence group whose difference from baseline survived Holm correction across six comparisons.Source: Figure 4 · p. 14, §5.4 · p. 14
  • Assisted participants finished with 27.26 versus 21.52 discs, but their transfer advantage over baseline was nonsignificant (+0.25 discs per game).They also rated their performance higher on the NASA-TLX (3.14 versus 2.01).Source: §5.4 · p. 14
  • After the unassisted round, assisted participants rated task success higher than baseline (2.95 vs. 2.51, d=0.32, p=.007), while no other NASA-TLX items differed significantly.Perceived success correlated with actual disc count (r=0.41, p<.001).Source: §5.4 · pp. 14–15

5.5 Additional Validation Under an External Taxonomy

  • Task management, mapped to asking the assistant for a move, was the strongest assisted-performance covariate and a significant negative covariate of unassisted performance; reasoned agreement reversed direction across outcomes.Information requests correlated negatively with assisted performance and were unrelated to transfer.Source: Table 7 · p. 15, §5.5 · p. 15

5.4 AI Assistance Improves Assisted Performance but Does Not Reliably Improve Transfer

  • Standard AI assistance improved performance while available but showed no reliable unassisted advantage; assisted and unassisted performance were negatively correlated (ρ=−0.307, p=.002, n=97).The authors conclude that access to AI alone does not reliably produce transfer.Source: §5.4 · p. 15

5.5 Additional Validation Under an External Taxonomy

  • TurnStyle, an external taxonomy developed for other tasks, was applied to the Othello conversations as a convergent check, but its labels were not revalidated against human labels on this dataset.The authors used TurnStyle’s published annotation prompts and treated this as a convergent check, not an independent measurement.Source: §5.5 · p. 15
  • Information-request loops were associated with lower odds of winning the assisted round (OR=0.12, 95% CI [0.01, 0.91], p=.020, n=101).Other prior findings tied more to open-ended collaboration did not appear; the authors attributed this to the fixed-puzzle task structure rather than either framework.Source: §5.5 · p. 15
  • Among assisted-round successes, participants also successful unassisted used more thinking aloud than unsuccessful peers (42% vs 17%); among assisted-round failures, the corresponding figures were 33% vs 26%.Solution-request usage was flipped across these comparisons.Source: Table 8 · p. 16

6 Case Study: Qualitative Look at Exemplar Conversations

Four Othello case studies illustrate how assisted and unassisted outcomes can diverge, and how participants’ reasoning and requests shaped their conversations with the assistant.

6 Case Study: Qualitative Look at Exemplar Conversations

  • The four anonymized cases were selected to represent distinct combinations of assisted success and unassisted transfer.They include two who succeeded with assistance but failed without it, one who succeeded in both, and one who transferred well after an assisted loss.Source: §6 · p. 16

6.1 P1: Efficient Assistant Consultation without Intuition Building; Label: 1-0

  • P1 won with assistance at 33 discs but averaged 24.0 without it, below the 28.5-disc median; every prompt was a direct solution request.The conversation lasted 68 seconds, and the participant did not peruse the provided rationales.Source: §6.1 · pp. 17–18

6.2 P2: Stating Reasoning and Being Corrected; Label 1-1

  • P2 stated move rationales, received corrections to two misconceptions, and tied for the study’s highest unassisted transfer score at 36.0 discs.P2 won the assisted round with 41 discs and used Think Aloud in 78% of turns.Source: §6.2 · p. 18

6.3 P3: Scrutinizing the Assistant, Not the Board; Label: 1-0

  • P3 succeeded with assistance at 41 discs but scored 26.0 without it, while scrutinizing the assistant’s confidence more than the board.Think Aloud accounted for 30% of speech acts and Solution Request for 50%.Source: §6.3 · p. 18

6.4 P4: Extracting a Principle from a Loss; Label: 0-1

  • P4 lost the assisted round with 19 discs but tied for the highest unassisted transfer score at 36.0, asking about general principles and never requesting a solution.The case describes P4 retaining generalizable Othello principles despite poor assisted performance.Source: §6.4 · p. 19

7 Think-Aloud Subtypes Elicit Assistant Acts Differently

The subtype analysis separates Think Aloud turns by their conversational function and shows that these subtypes can elicit different assistant responses, though they do not robustly predict transfer.

7 Think-Aloud Subtypes Elicit Assistant Acts Differently

  • Of 193 Think Aloud utterances, 139 received a subtype assignment using an unsupervised clustering tool; the authors report four clusters for readability.The remaining utterances carried no usable dependency information.Source: §7 · p. 19
  • The clusters distinguish diagnostic turns about why an outcome occurred or what follows from a move, verification turns asking for candidate-move evaluation, narration, and evaluative turns.The reported cluster counts are 44 diagnostic, 39 verification, 41 narration, and 15 evaluative turns.Source: §7 · p. 19
  • Solution Requests were followed by Provide Correct Answer (OR=6.71, 95% CI [3.55, 12.65], p<.001), while diagnostic turns suppressed it (OR=0.23, 95% CI [0.07, 0.69], p=.009).ORs are turn-level, with standard errors clustered on participant. Solution Request is compared with all other participant turns; other subtypes are compared with remaining Think Aloud turns.Source: §7 · p. 19, Table 10 · p. 20
  • The subtype findings support differences in which assistant acts turns elicit, not a claim that Think Aloud subtype predicts transfer.The authors note that the result does not depend on the number of clusters, but do not interpret subtypes as intrinsically better or worse for transfer.Source: §7 · p. 20

8 Discussion

Across the studies, conversational behaviors are associated with unassisted performance and with the forms of assistance users receive, while the broad Think Aloud label covers distinct strategies.

8.1 Thinking Aloud as Cognitive Participation during AI Assistance

  • Think Aloud tracks better unassisted performance across both studies, while direct Solution Requests track poorer transfer.The discussion interprets these patterns as consistent with users retaining more of the reasoning process when they remain involved in constructing and monitoring their problem representation.Source: §8.1 · pp. 20–21
  • The authors treat these results as associations, not effects of conversational behavior, and note that Think Aloud may mark users already inclined to engage.They say stronger evidence requires testing whether interventions that increase Think Aloud also improve unassisted performance.Source: §8.1 · p. 21

8.2 Users Shape the Pedagogical Behavior of Less Constrained Assistants

  • Board Report is positively correlated with transfer in Othello, whereas Provide Correct Answer is negatively correlated; Think Aloud tends to evoke tutor-like responses and Solution Request direct answers.The discussion also reports counterfactual rewriting experiments in which minimally changing a request causally altered assistant response behavior.Source: §8.2 · p. 22

8.3 Think Aloud Collapses Multiple Interaction Strategies

  • Think Aloud includes interactionally distinct subtypes, and verification turns have higher odds of eliciting direct answers, but the subtypes do not robustly predict transfer.The authors therefore caution that frequency alone does not fully characterize how reasoning work is distributed.Source: §8.3 · p. 22

9 Design Implications

The paper proposes user-facing interventions to support interaction behaviors associated with retained performance, while emphasizing that these design ideas are hypotheses to test rather than prescriptions.

9 Design Implications

  • The proposed design implications aim to help users develop interaction patterns they can carry into future AI use when scaffolds are absent.The authors explicitly frame the implications as hypotheses to test, not prescriptions.Source: §9 · p. 22

9.1 Redirecting Solution Request Toward Active Reasoning

  • Systems could detect repeated solution-seeking and nudge users toward articulating reasoning, uncertainty, or misconceptions rather than merely increasing Think Aloud frequency.The authors note that Think Aloud alone may not indicate sufficient cognitive involvement.Source: §9.1 · pp. 22–23

9.2 Teaching Interaction Strategies That Persist Beyond Scaffolding

  • Short trainings could contrast asking for the next action with proposing and justifying a solution and seeking critique, emphasizing strategies for retaining more reasoning.The section also suggests interactive practice or lighter-weight contrasting examples and videos.Source: §9.2 · p. 23

9.3 Making Reasoning Delegation Visible Through Peer Comparison

  • Interfaces could compare users’ dialogue-act distributions with those of peers who show stronger subsequent unassisted performance, making reasoning delegation more visible.The proposal avoids prescribing a fixed optimal conversational composition.Source: §9.3 · p. 23

10 Limitations and Future Work

The study measures immediate near-transfer on related puzzles within the same game, not delayed retention or far transfer. Its controlled game setting, participant screening, and observational measures constrain how broadly the findings can be generalized.

10 Limitations and Future Work

  • The outcomes cover immediate near-transfer to related puzzles of the same game, so the study does not establish delayed retention or far transfer.The authors therefore do not claim durable learning outcomes.Source: §10 · p. 24
  • Connect Four and Othello offer controlled tasks with objectively optimal solutions, but may not generalize to open-ended writing or programming.In open-ended domains, assistant answers may not have guaranteed optimality.Source: §10 · p. 24
  • The assistant was given the optimal next move to reduce response-correctness variation, improving experimental control but differing from realistic AI interactions.Participants were also filtered for the ability to produce high-quality prompts, constraining generalization to all AI users.Source: §10 · p. 24
  • Dialogue acts capture observable behavior, and links between participant behavior and transfer are observational rather than causal.Available pre-assistance measures were coarse and weakly predictive; a validated pre-test or randomized Think Aloud manipulation is a natural next step.Source: §10 · p. 24
  • Future work should test the patterns over longer timescales, in less constrained tasks, and under different assistant policies.Interventions based on these patterns should be evaluated for effectiveness in improving unassisted performance.Source: §10 · p. 24

11 Conclusion

Across two controlled problem-solving studies, conversational behavior was associated with subsequent independent performance, and users’ choices shaped the assistance they received. Participants who maintained cognitive participation performed better independently afterward, although the design cannot establish that participation caused the difference.

11 Conclusion

  • Participants who maintained cognitive participation during AI assistance performed better independently afterward, though the design cannot establish that participation produced the difference.The authors conclude that users should learn to retain their ability to reason and perform independently without AI support.Source: §11 · p. 24

B Puzzle Configurations

C Prompt Creation Screener Details

D.1 Demographic Differences

  • Frequent generative-AI users were disproportionately classified as Strugglers, a significant but exploratory pattern that does not establish why prior use may affect transfer.The reported group difference was χ2 = 9.49, p < .01.Source: §D.1 · p. 31

D.2 Conversation Statistics

  • Solvers averaged 2.5 assistant turns versus 2.2 for Strugglers and used 22 versus 17 unique words, but these differences were not statistically significant.Source: §D.2 · p. 31

D.3 Self-report Survey Results

Across the reported surveys, participants generally rated the assistant favorably, while self-perceived success during assistance did not always align with unassisted performance. The interface required an unaided first move before assistance, and exploratory Othello comparisons describe differences in conversation and survey responses.

D.3 Self-report Survey Results

  • Connect Four participants reported high mental load and effort across both rounds; Strugglers rated their assisted performance higher, while the pattern reversed for the unassisted round.Solvers and Strugglers did not significantly differ in assistant intent-consistency or ease-of-use ratings.Source: §D.3 · p. 32

E Study Interfaces

F.1 Conversation Statistics

  • Othello participants at or above the unassisted-performance median used more distinct words and longer turns, while turn counts were similar under the five-turn engagement requirement.The groups averaged 41.5 versus 33.6 distinct words and 10.2 versus 8.1 words per turn; the five-turn comparison was 5.54 versus 5.56.Source: §F.1 · p. 33

F.2 Self-report Survey Results

  • Othello participants who later transferred better rated the assistant as less easy to use, while intent-consistency ratings did not differ.The ease-of-use ratings were 4.03 versus 4.39 (p = .013).Source: §F.2 · p. 34
  • In Othello, participants who later transferred better reported more effort and lower self-rated performance during assistance; the self-rated performance pattern reversed in the final unassisted round.Assisted-round effort was 4.90 versus 4.34, and self-rated performance was 2.72 versus 3.66; unassisted self-rated performance was 3.23 versus 2.64.Source: §F.2 · p. 34

F.3 Robustness Check for Othello’s Participant-side Dialogue Act Results

In Othello, Think Aloud prevalence remained associated with unassisted performance after adjustment for measured skill and other covariates. Analyses also indicate that this association is not reducible to message volume, although the authors distinguish reasoning talk from talk in general.

F.3 Robustness Check for Othello’s Participant-side Dialogue Act Results

  • Solution Request showed no significant relationship with transfer once Think Aloud was included, and the two dialogue-act measures were negatively correlated (r = −0.26).Source: §F.3 · p. 34
  • Across both games, participants who more frequently articulated their reasoning while working with the assistant performed better on subsequent unassisted tasks.In Othello, the Think Aloud association remained stable after adding AI usage, Solution Request prevalence, and assisted-round performance.Source: §F.3 · p. 34
  • The Othello Think Aloud coefficient stayed stable when first-move correctness, confidence, or self-rated skill was added, while none of those measures correlated significantly with Think Aloud or transfer.The partial Spearman correlations were +0.358, +0.353, and +0.344, compared with +0.358 unadjusted.Source: §F.3 · p. 35, Table 17 · p. 35

G Is Think Aloud Prevalence Just Verbosity?

  • Think Aloud prevalence remained associated with transfer after controlling for turn count, total words, or mean words per turn; the volume measures no longer did so when Think Aloud was held.With mean words per turn added to the full model, Think Aloud DAP was +0.329 (p < .01), while the new term was +0.011.Source: G Is Think Aloud Prevalence Just… · p. 35
  • In Connect Four, Think Aloud DAP correlated with unassisted score (ρ = +0.182, p = .032), while turn count, total words, and mean words per turn were null.Source: G Is Think Aloud Prevalence Just… · p. 35
  • Othello turn count tracked assisted performance but not transfer, whereas Think Aloud was unrelated to assisted performance and strongest on transfer; however, Think Aloud turn count also tracked transfer.Thus, the data distinguish reasoning talk from talk in general, not a rate from an amount.Source: G Is Think Aloud Prevalence Just… · p. 36

H.1 Agentic Simulation of Full Participant Trajectories

This section presents controlled replay and agentic simulations of full participant–assistant trajectories, with Othello examples illustrating how participant utterances and assistant coaching around future replies, corner access, legal moves, mobility, and final disc count feature in the exchanges, including a participant who won with assistance but showed poor unassisted transfer. It also reports annotation-method details and panel-model agreement for several dialogue acts.

H.1 Agentic Simulation of Full Participant Trajectories

  • Full-trajectory simulations reproduced Solution Request eliciting Provide Correct Answer and the temporal-orientation link to Board Report (OR=6.940, p<.001), but not the human Think Aloud pattern.Across simulated interactions, Think Aloud co-occurred with more Provide Correct Answer rather than suppressing it.Source: §H.1 · p. 36
  • Simulated-human discrepancies included Knowledge-Deficit Questions being tied to Board Report, likely because simulated participants used more board coordinates and fewer single-act turns.These differences qualify the simulated reproduction of human conversational patterns.Source: §H.1 · p. 36

H.2 Controlled Replay of Participant Utterances through LLM Simulations

  • Rewriting participant utterances shifted assistant acts: Board Report rose from 16% to 94% (+78 pp), Provide Correct Answer from 46% to 94% (+48 pp), and the reverse rewrite lowered it from 94% to 74% (−20 pp; all p<.001).The replay changed one utterance while holding the conversation history, board state, and assistant fixed; significance was tested with exact McNemar tests on paired outcomes.Source: §H.2 · p. 37, Table 20 · p. 38
  • The replayed causal patterns replicated across four assistant models from three providers, although effect magnitudes depended on each model’s baseline tendency to produce the target act.The same stimuli were replayed across models; baseline rates constrained how much a rewrite could shift responses.Source: §H.2 · p. 37

I Full Text of the Exemplar Conversations

  • The appendix reproduces four assisted-round conversations verbatim, preserving participant spelling and assistant formatting, with dialogue-act annotations on participant turns.The supplied exemplars include the poor-transfer P1 and good-transfer P2 conversations.Source: I Full Text of the Exemplar… · p. 39, §I.1 · p. 39, §I.2 · p. 40

I.1 P1 (p258)

I.2 P2 (p322)

  • P2 won with assistance and transferred well, averaging 36.0 discs; the participant used Think Aloud on 78% of turns and Solution Requests on 11%.The transcript presents the participant considering a move and the assistant explaining its strategic risk.Source: §I.2 · p. 40, Assistant Turn #1 · p. 40

Assistant Turn #2

  • P2’s assistant checked the participant’s claim, explained that White could take the corner diagonally, and encouraged comparing legal moves by their consequences.After the participant acknowledged not realizing diagonals counted, the assistant reiterated the point and prompted further comparison.Source: Assistant Turn #1 · p. 40, Assistant Turn #2 · p. 40, Assistant Turn #3 · p. 41

Assistant Turn #3

I.3 P3 (p161)

  • 41 of 64 discs were assisted; transfer mean was 26.0 (Puzzle #2 26, Puzzle #3 26), despite an assisted win; the 10-turn trajectory had Think Aloud prevalence 0.30 and Solution Request prevalence 0.50.Source: §I.3 · p. 42

Assistant Turn #1

Assistant Turn #8

Assistant Turn #9

I.4 P4 (p786)

  • Assisted play scored 19 of 64 discs; transfer mean was 36.0 (Puzzle #2 38, Puzzle #3 34), with Think Aloud prevalence 0.67 and Solution Request prevalence 0.00 across 6 turns.The participant lost the assisted game but showed good transfer.Source: §I.4 · p. 45

Assistant Turn #1

  • The assistant cautioned against treating edges as automatically safe, comparing moves by corner access and how many replies they leave the opponent.In the example, it flagged a7 and b8 as adjacent to the empty a8 corner and urged checking White’s replies after c8.Source: Assistant Turn #1 · p. 45

Assistant Turn #2

  • When comparing b6 with g2, the assistant favored considering whether a move avoids the corner fight and constrains White’s mobility over whether it is an edge move.It also asked the participant to weigh corner-taking replies and the possibility of forcing a pass.Source: Assistant Turn #2 · p. 46, Assistant Turn #3 · p. 46

Assistant Turn #5

  • The assistant explained that corners are permanent, can stabilize nearby edges, and matter because the goal is the final disc count; it then confirmed the full-board game was over.It advised checking whether a move lets White take a corner rather than focusing only on immediate flips.Source: Assistant Turn #5 · pp. 46–47, Assistant Turn #6 · p. 47

J Prompts

  • The two assistant prompts ran live during the study; the remaining prompts were used offline for labeling or screening by a three-model panel with per-item majority voting.Source: J Prompts · p. 47

K Annotation Reliability

  • Panel-model agreement was strongest for targeted acts: Think Aloud and Solution Request reached κ≥0.75 in both games, and Provide Correct Answer reached 0.82–0.85.Board Report agreement was 0.70 in Othello, where results are reported, and 0.43 in Connect Four.Source: K Annotation Reliability · p. 49

K.1 Agreement with Human Coders

The section evaluates agreement between human coding and the annotation pipeline across participant and assistant turns. Think Aloud had the least stable coding boundary, and its agreement varied after a coder threshold shift.

K.1 Agreement with Human Coders

  • After the 60-turn Connect Four comparison, validation added 50 Connect Four and 100 Othello participant turns, plus 50 Othello assistant replies annotated by an author.Source: §K.1 · pp. 49–50
  • Think Aloud agreement shifted from κ=0.40 on the first 65 turns to 0.67 on the remaining 85, prompting blind recoding of the first 65.Source: §K.1 · p. 50
  • Think Aloud had the weakest coder test–retest agreement (κ=0.40), versus κ=0.84–0.96 for every other act, making its boundary hardest to apply consistently.Source: §K.1 · p. 50
  • For Board Report, the coder agreed on 18 of 19 replies all three models labeled and 13 of 15 none labeled; 9 of 12 disagreements fell where the panel disagreed.Source: §K.1 · p. 50, Table 24 · p. 51
Loading 2610.09547v1…