TraceMind: Predicting User Information Uptake from Low-Cost Interaction Traces during Human-LLM Content Co-Generation

• AI-generated information can enter final artifacts without adequate processing, and overlooked information can lead to consequential errors or miscoordination.The concern is especially relevant when artifacts are shared or acted upon in public settings.Source: §1 · p. 2
• The paper operationalizes uptake as immediate correct recognition of a single-information statement in the final draft, rather than comprehension or later application.An atomic information unit is a self-contained statement expressing one piece of information.Source: §1 · p. 2
• Open-ended assessment must derive units dynamically, while trace alignment must handle information moving or changing across Chat, Draft, and on-screen layouts.The study generated post-task recognition labels for 1,187 units from 62 participants across three co-generation tasks.Source: §1 · p. 2
• TraceMind finds semantic occurrences of each unit across Chat and Draft histories, recovers their layouts and associated interactions, then models spatial, temporal, and workflow-informed evidence.The framework predicts uptake from aligned traces tied to each information unit.Source: §1 · p. 2
• 81.17% AUROC, 80.58% balanced accuracy, 49.34% AUPRCnon, and 73.05% macro-F1; TraceMind outperformed all learned baselines across these four metrics.The findings also characterize uptake as unfolding through interaction, with sustained active engagement more informative than isolated signals.Source: §1 · p. 2

2 Related Work

• Human-LLM co-generation research examines users’ collaboration with AI, including accepting suggestions, editing generated text, and dividing writing work.Prior work spans creative, educational, professional, and personal writing contexts.Source: §2.1 · pp. 2–3

2.1 Human-LLM Content Co-Generation and Information Uptake

• Prior work studies adoption, reliance, and scrutiny of AI-generated content, while this paper measures immediate recognition of atomic information in the final co-generated draft.The focus shifts from what enters the artifact to what users take up from it.Source: §2.1 · p. 3

2.2 Assessing User States in HCI Tasks

• Subjective measures are easy to collect but may diverge from behavioral or performance outcomes; fixed assessment units make existing performance tests difficult to apply to open-ended co-generation.Each user produces a different artifact whose information units emerge during interaction.Source: §2.2 · p. 3

2.3 Modeling User States from Interaction Traces

• Interaction traces can model engagement and related user states, but remain indirect proxies; this work follows interactions with the same information as they evolve over time.Prior work often summarizes traces within fixed scopes such as pages, documents, sessions, or tasks.Source: §2.3 · p. 3

3 Data Collection Study

• The study paired unit-level recognition responses with interaction traces recorded throughout open-ended co-generation.Atomic units were dynamically derived from each final draft, and post-task recognition questions supplied the uptake labels.Source: §3 · p. 4

3.1.1 Participants.

• 62 participants, aged 18–30, took part; they reported frequent conversational LLM use and writing-related LLM use.Participants’ mean age was 21.48 years; average familiarity with conversational LLMs was 4.10 on a five-point scale.Source: §3.1.1 · p. 4

3.1.2 Task Design.

• Participants were randomly assigned to a club announcement, roommate recruitment, or lecture announcement task, selected as flexible everyday writing scenarios with assessable details.The task settings were chosen to support varied co-generation behaviors and unit-level recognition questions.Source: §3.1.2 · p. 4

3.1.3 Procedure.

• Each session lasted about 30 minutes, and participants used a laptop with an external mouse and produced a draft of at least 1,500 characters.They reviewed study guidance before the task and uploaded screen recordings afterward.Source: §3.1.3 · p. 4

3.2.1 User Interface.

• Participants worked in a side-by-side workspace with LLM Chat and an editable Draft, then answered four-option recognition questions and rated confidence.Questions appeared one at a time without backward navigation; answer correctness was the primary uptake label.Source: §3.2.1 · p. 4

3.2.2 Interaction Event Logging.

• The platform logged timestamped pointer coordinates and user operations, plus content, layout, and visibility snapshots as surfaces changed or scrolled.These records preserved histories and on-screen states for aligning interactions with displayed content.Source: §3.2.2 · pp. 4–6

3.2.3 Post-task Question Generation.

• The question pipeline filtered draft-derived atomic units, generated one correct answer and three distractors, validated and repaired items, then sampled and shuffled qualified questions.Units had to be well-formed, self-contained, single-information statements supported by the draft; invalid questions were discarded after a failed repair attempt.Source: Figure 3 · p. 6, §3.2.3 · p. 6

3.3 Dataset Construction

• The 62 sessions yielded 1,187 validly labeled units: 881 correctly recognized and 306 incorrectly recognized, for 74.22% overall accuracy.The dataset also included Chat and Draft histories, event logs, snapshots, questions, responses, and unit occurrences and exposure intervals.Source: §3.3 · p. 7

4 TraceMind Framework

• TraceMind predicts unit-level uptake from session records by tracking information across Chat and Draft, aligning traces to layouts, and combining spatial, temporal, and workflow-informed signals.Source: §4 · p. 7, Figure 4 · p. 7

4.1 Identifying Semantic Occurrences of Each Information Unit

• Starting from a target unit’s final-draft span, TraceMind traces retained or revised occurrences backward through Draft history, then matches corresponding Chat spans deterministically or with an LLM judge.Source: §4.1 · pp. 7–8

4.2 Reconstructing Layout-Aware Occurrence States for Each Semantic Occurrence

• TraceMind groups observations into occurrence states when interface context, text, visibility, and bounding-box shape match; scrolling alone is merged, while reflow or other changes split states.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 Aligning and Encoding Interaction Traces

• It aligns pointer events to occurrence states and operations to target units, producing state-specific pointer trajectories and a target-linked operation sequence for three complementary representations.Source: §4.3 · p. 8

4.3.1 State-Specific Pointer-Trajectory Heatmaps.

• For each occurrence state, pointer points within a soft-expanded target box become 128 × 128 six-channel heatmaps, retaining nearby activity beyond the hard target boundary.Source: §4.3.1 · pp. 8–9, §4.2 · p. 8

4.3.2 Time-Binned Activity Sequence.

• Target-linked events are placed in 512 equal-width session bins, each represented by 44 features spanning interaction context, event-target relations, and their conjunctions.Source: §4.3.2 · p. 9

4.3.3 Workflow-Informed Interaction Features.

• A 40-dimensional workflow vector captures eight recurrent behavior families, identified through inspection of 15 pilot replays and retained when objectively computable from logs.Source: §4.3.3 · p. 10

4.4 Information-Uptake Prediction Model

• Three branches model heatmaps, temporal sequences, and workflow features; positive weighted-logit late fusion combines their probabilities, with the decision threshold selected using training data.Source: §4.4 · p. 10

5.1 Evaluation Setup

• TraceMind was tested with participant-grouped 5-fold cross-validation and compared with content/context, conventional interaction, workflow, sequence, trajectory, and early-fusion baselines using four class-sensitive metrics.The folds kept each participant’s units together; metrics included balanced accuracy, macro-F1, AUPRCnon, and AUROC.Source: §5.1.1 · p. 10, §5.1.2 · pp. 10–11, §5.1.3 · p. 11

5.3.1 Contribution of Three Input Branches.

• All three input branches predicted uptake independently, while removing the temporal branch caused the largest AUROC and balanced-accuracy drops, the workflow branch the largest macro-F1 drop, and the spatial branch the largest AUPRCnon drop.Independent branch AUROCs ranged from 74.69% to 78.00%.Source: §5.3.1 · pp. 11–12, Table 4 · p. 12

5.2 Overall Performance

• 4.49 percentage points in AUROC, 12.38 in balanced accuracy, 3.98 in AUPRCnon, and 8.21 in macro-F1: TraceMind exceeded the strongest learned baseline for each metric.Its advantage also persisted across decision thresholds.Source: Table 3 · p. 11, §5.2 · p. 11

5.3.2 Contribution of Chat and Draft Interaction Traces.

• Draft-only traces retained more predictive signal than Chat-only traces, but combining both interfaces improved all four metrics over either alone.Against Chat-only and Draft-only, respectively, the full model gained 5.68 and 5.21 AUROC points, 12.64 and 6.71 balanced-accuracy points, 9.50 and 8.98 AUPRCnon points, and 16.75 and 8.19 macro-F1 points.Source: §5.3.2 · p. 12, Table 4 · p. 12

5.3.3 Contribution of Semantic and Layout-state Alignment.

• Using only final-draft occurrences reduced AUROC by 8.26 points, balanced accuracy by 10.22, AUPRCnon by 16.13, and macro-F1 by 7.96; layout-state alignment also improved fused performance.Without layout-state alignment, balanced accuracy fell 2.57 points, AUPRCnon 4.09, and macro-F1 2.35; spatial-branch AUROC fell from 77.91% to 74.12%.Source: §5.3.3 · p. 12, Table 4 · p. 12

5.3.4 Contribution of TCN Pretraining on Temporal Patterns Branch.

• Masked-bin pretraining improved the temporal branch’s AUROC by 1.93 percentage points and AUPRCnon by 7.76, with gains persisting after fusion across all four metrics.The full model’s gains over training the TCN from scratch were 1.23 AUROC points, 8.31 balanced-accuracy points, 5.74 AUPRCnon points, and 6.54 macro-F1 points.Source: §5.3.4 · pp. 12–13, Table 4 · p. 12

5.3.5 Effect of Soft Boundary.

• Restricting trajectories to the target bounding box weakened the spatial branch to 52.77% AUROC; a ρ=250 boundary gave the strongest overall fusion results, while broader regions did not consistently help.The results indicate that nearby pointer movements carry useful evidence beyond the unit’s bounding box.Source: §5.3.5 · p. 13, Figure 8 · p. 13

6.1 Information Uptake Across Exposure Duration and Active Engagement

• Greater exposure and active engagement were each positively associated with uptake, with stronger evidence for active-engagement signals including dwell, clicks, revisits, and manual edits.Estimated uptake was significantly higher for high exposure (p<0.05) and for target-near dwell, clicks, revisits, and manual edits.Source: §6.1 · p. 13, Figure 9 · p. 14
• Within high-exposure units, High Engagement yielded 81.8% vs. 75.3% uptake; revisiting, Chat–Draft coordination, Draft revision, and occurrence inspection showed the largest standardized differences.The engagement difference was significant in the high-exposure group but not the low-exposure group.Source: §6.1 · p. 13, Figure 9 · p. 14

6.2 Information Uptake Across Content Adoption Completeness and Post-Adoption Engagement

• Complete and incomplete Chat adoption had nearly identical uptake probabilities, 73.7% and 74.5%, while uptake rose from 67.7% with no target edit to 79.4% after repeated revision.Initial Draft insertion timing was not reliably associated with uptake; final-stage review after re-engagement was associated with higher uptake than earlier re-engagement (p<0.05).Source: §6.2 · p. 14, Figure 10 · p. 14

6.3 Self-reported Confidence and Metacognitive Awareness of Information Uptake

• Recognition accuracy rose from 10.2% at confidence 1 to 92.0% at confidence 5; within participants, mean confidence was 4.22 for correct units versus 2.75 for incorrect units.This pattern held among the 58 participants with both correct and incorrect units.Source: §6.3 · p. 15, Figure 11 · p. 15
• There were 102 high-confidence misses, 12.3% of high-confidence units and 33.3% of incorrect units; five workflow signals showed weaker engagement for misses than for high-confidence correct units.Signals included fewer source checks, fewer Draft revisits, fewer visited occurrences, and fewer unit-linked engagement events.Source: §6.3 · p. 15, Figure 11 · p. 15

7.1 Information Uptake Unfolds Across the Interaction Process

• Uptake evidence is distributed across an information unit’s evolving interaction history, and sustained engagement such as revisiting, revising, and checking it is informative for modeling uptake.The same acceptance event may carry different evidence depending on later engagement with that information.Source: §7.1 · pp. 15–16

7.2 Extending Performance-Based Uptake Assessment to Open-Ended Artifacts

• Deriving recognition questions from each user’s final artifact makes performance-based assessment possible in open-ended tasks and supports content-level diagnosis of which information that user took up.This approach uses immediate recognition; the authors suggest extending it to delayed recognition or recall, comprehension, and transfer.Source: §7.2 · p. 16

7.3 From Specialized Sensing to Low-Cost Interaction Traces in Everyday Workflows

• Mouse, keyboard, and interface traces are available in ordinary browser and editor use without dedicated sensing hardware, making them a potential basis for modeling user states in everyday workflows.The discussion suggests this approach could extend beyond uptake to other states in dynamic tasks.Source: §7.3 · p. 16
• Restricting pointer traces to target bounding boxes weakened prediction, while moderate nearby context helped and further expansion added little benefit.The discussion also argues that semantic identity and changing screen layout should be tracked separately, as both alignments contribute to prediction.Source: §7.3 · p. 16

7.4 Toward Uptake-Aware Human-LLM Co-Generation Systems

• Uptake-aware systems could flag content with weak evidence of uptake for review in co-generation, public communication, learning, or AI-assisted decision making.The authors recommend keeping support probabilistic, inspectable, and privacy-conscious, with user control over recorded and retained data.Source: §7.4 · p. 17

7.5 Limitations and Future Work

• The study measures immediate recognition rather than comprehension, application, or longer-term retention, and its participants and tasks limit the tested scope.Participants were predominantly experienced LLM users in China, the study covered three writing tasks, and capture used laptops with external mice.Source: §7.5 · p. 17

• The paper argues that uptake is an unfolding process and presents performance-based assessment for open-ended artifacts alongside low-cost trace alignment to atomic information units.It calls for future HCI research to better assess and support what users actually take up.Source: §8 · p. 17

• The section contains participant demographics and the complete participant-facing instructions for the three co-generation tasks.Source: A DATA COLLECTION STUDY DETAILS · p. 21

• Participant demographics and LLM usage background are reported in Table 5.Source: §A.1 · p. 21

A.2 Participant-Facing Task Instructions

• The instructions cover a club team-building announcement, a roommate recruitment post, and a university lecture announcement.The English instructions were originally presented in Chinese, and identifying proper nouns were replaced with consistent placeholders.Source: §A.2 · p. 21, §A.2.1 · p. 21, §A.2.2 · p. 21

A.2.1 Club Team-Building Announcement

• The team-building scenario fixes a coach trip to Historic Town H, visits to Temple T, the South Gate, and the historic district, and an evening meal before returning by coach.The meal is at Restaurant R and includes the town’s “Eight Bowls” cuisine.Source: §A.2.1 · p. 21
• Participants must draft a club announcement introducing the activity’s purpose, itinerary, attractions, restaurant, and practical guidance for members.Suggested practical information includes transportation, attire, cuisine, preparation, and precautions.Source: §A.2.1 · p. 21
• The team-building announcement should be complete and ready for public release.Source: §A.2.1 · p. 21

A.2.2 Roommate Recruitment Post

• The roommate scenario is a two-month summer stay in a shared apartment, with one additional roommate to recruit after three students’ places are confirmed.The apartment is in District D and is described as unit type D2–4×4.Source: §A.2.2 · p. 21
• The recruitment post may describe the apartment, amenities, neighborhood, transport, move-in arrangements, and expectations for shared living.Examples of house rules include expense-sharing, cleaning, visitors, and daily schedules.Source: §A.2.2 · p. 21
• The roommate post should be complete and suitable for public publication.Source: §A.2.2 · p. 21

A.2.3 Lecture Announcement

• The lecture announcement concerns a Friday 2:00 p.m. AI Lecture Series with three academicians, individual talks, and a Q&A.It should introduce each scholar and cover the event theme, value of attending, preparation, and possible discussion topics.Source: §A.2.3 · p. 21

B.1 Workflow-informed Interaction Features

• TraceMind uses 40 workflow-informed interaction features organized by eight behavior types.Source: §B.1 · p. 21
• Table 5 records participant demographics and LLM-use background, including familiarity and writing-reliance categories.Source: Table 5 · p. 22
• Table 6 lists the workflow-informed operation features used in TraceMind.Source: Table 6 · p. 23