Source-linked AI summary
ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions
Chuanyang Jin, Binze Li, Haopeng Xie, Cathy Mengying Fang, Tianjian Li, Shayne Longpre, Hongxiang Gu, Maximillian Chen, Tianmin Shu
TL;DR
Existing human–AI datasets capture users’ messages but not their underlying thoughts, leaving a largely unexplored dimension of interaction. ThoughtTrace pairs real-world multi-turn conversations with self-reported reasons and reactions, finding that thoughts add useful signals for predicting behavior and aligning assistants.
Problem
Existing human–AI interaction datasets largely omit users’ unspoken thoughts, including the motivations, expectations, and reactions behind their messages.
Method
ThoughtTrace collects naturalistic multi-turn conversations paired with users’ self-reported reasons for messages and reactions to assistant responses.
Results
+41.7% relative gain in user-behavior prediction and +25.6% win rate for fine-grained alignment, while thoughts remain distinct, diverse, and difficult for frontier LLMs to infer.
Takeaways & Limitations
User thoughts constitute a distinct signal for studying cognitive dynamics and modeling users beyond surface dialogue in human–AI interaction.
Takeaways & Limitations
Externalizing thoughts may shape interactions, and the dataset captures only consciously accessible reasoning rather than subconscious judgments.
Abstract
from arXiv · showhide
Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale dataset that pairs real-world multi-turn human--AI conversations with users' self-reported thoughts: their reasons for sending prompts and reactions to assistant responses. ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20 language models. Our analysis shows that ThoughtTrace captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as inference-time context. Second, thought-guided rewrites provide fine-grained alignment signals for training personalized assistants. Together, ThoughtTrace establishes user thoughts as a new data modality for studying the cognitive dynamics behind human--AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.
1. Introduction
ThoughtTrace addresses the largely unexplored problem of understanding users’ unspoken thoughts during real-world human–AI conversations. It introduces a large-scale dataset and shows that thoughts reveal cognitive context, exhibit distinctive properties, and support user modeling and assistant alignment.
- Motivation: User thoughts encode the motivations, goals, constraints, response expectations, interpretations, and reactions hidden behind observable messages.They provide cognitive context that ordinary dialogue transcripts do not expose.
- Contribution: ThoughtTrace introduces a framework and dataset pairing naturalistic real-world conversations with users’ contextually grounded, first-person thought annotations.The dataset is designed to illuminate users’ lived experience of interacting with AI systems.
- Dataset: 1,058 users, 2,155 timestamped conversations, 17,058 interaction turns, and 10,174 thought annotations span open-ended tasks and 20 language models.The corpus contains long-horizon, multi-turn interactions from a diverse user base, with thoughts aligned to individual user and assistant messages.
- Analysis: ThoughtTrace shows that conversations are long-horizon and topically broad, while thoughts differ from messages, resist frontier-LLM inference, vary in content, and depend on conversation stage.These analyses characterize both the conversational structure of the dataset and the cognitive properties of its annotations.
- Utility: Thoughts improve user-behavior prediction and provide supervisory signals for training and aligning personalized assistants.They enable models to predict, learn from, and adapt to users’ latent goals, preferences, and needs.
2. Related Work
Prior work has assembled real-world human–AI conversation corpora, studied latent mental-state inference, and developed user simulators to address data gaps. ThoughtTrace builds on these directions by focusing on users’ self-reported thoughts in multi-turn interactions.
- Real-World Human-AI Conversations: Real-world conversation datasets span general chat, domain-specific interactions, and corpora pairing conversation logs with sociodemographic surveys or stated preferences.Examples include WildChat, LMSYS-Chat-1M, SWE-Chat, and PRISM.
- User Thoughts: Machine Theory of Mind research increasingly seeks to infer latent mental states from behavior, but much work restricts inference to structured variables such as goals, beliefs, and desires.These inferences are typically grounded in simple contexts.
- User Simulations: User-simulation research addresses data gaps for training and evaluating assistants, relying heavily on prompted or fine-tuned language models.Fine-tuning commonly targets ground-truth responses or persona-consistent behavior.
3. Data Collection
ThoughtTrace collects users’ latent thoughts alongside their messages and assistant responses, distinguishing reasons for sending messages from reactions to responses. Participants completed open-ended tasks through natural multi-turn interactions with one of 20 models, annotating thoughts privately at each turn.
- Thought annotation: Thought annotations comprise reasons for user messages and reactions to assistant responses, capturing users’ needs, preferences, and perceptions across turns.Both thought types are collected because they jointly shape how users proceed in the next turn.
- Collection procedure: The study used a consent-and-training pipeline requiring informed consent, a guided tutorial, and a comprehension quiz before participation.Participants also completed post-task descriptions and a demographic survey covering age, gender, education, occupation, AI usage frequency, and primary purposes.
- Collection procedure: Participants completed two open-ended, self-defined tasks within separate 10-minute windows while chatting naturally and privately annotating reasons and reactions.They could start new conversations or end tasks at any time, mirroring real-world conversational-AI use.
- Model coverage: Each participant interacted with one of 20 language models, including frontier and smaller open-weight models, without knowing the model identity.Detailed per-model counts of users, conversations, messages, and thoughts are provided in Appendix A.
- Dataset structure: Each conversation record stores model and timing metadata, task summaries and expectations, survey responses, ordered messages, and timestamped reason or reaction annotations.Thought labels are drawn from seven reason types or five reaction types.
4. Data Properties
ThoughtTrace contains representative, long-horizon, topically diverse conversations dominated by extensions of prior tasks, alongside thoughts that add latent information beyond messages and vary by category and conversation stage. These properties establish conversations and thoughts as complementary data dimensions for understanding real-world human–AI interaction.
- Conversation-level properties: ThoughtTrace pairs conversations with rich demographic and usage metadata, covering diverse AI users and everyday purposes while skewing toward ages 18–34 and participants with at least an undergraduate degree.Common uses include learning and working, followed by brainstorming, research, and coding.
- Conversation-level properties: 8 turns is ThoughtTrace’s median conversation length, compared with 2 in both WildChat and LMSYS-Chat-1M, and its conversations span seven broad categories and 36 fine-grained subtopics without a dominant category.WildChat and LMSYS-Chat-1M short 2-turn exchanges account for over 60% and 67% of their conversations, respectively.
- Conversation-level properties: 57.0% of user turns extend, deepen, or build on a prior task, far exceeding first requests, completely new requests, re-attempts, and new variations, with this pattern strengthening over conversation progress.The other relationship labels account for 25.2%, 12.5%, 2.9%, and 2.3%, respectively.
- Thought-level properties: Thoughts differ semantically from messages: message–reason pairs show larger embedding distances, reaction–next-message pairs show the widest dispersion, and three frontier models consistently struggle to infer thoughts from context.These findings support explicit thought annotations as a complementary signal rather than a restatement of conversation messages.
- Thought-level properties: ThoughtTrace thoughts span seven reason categories and five reaction categories, including Task Motivation & Goal (36.9%), Task Continuation (21.4%), and Explicit Affirmation (72.2%), alongside targeted dissatisfaction categories.Other reported categories include Context Grounding & Constraints (13.1%), Content Expectation (11.5%), Content Relevance (11.9%), Presentation Style (6.4%), and Scope Fit (6.1%).
- Thought-level properties: Thought dynamics depend on conversation stage and multi-turn relationships rather than topics or length: Task Motivation & Goal dominates early, Task Continuation becomes primary later, and Explicit Affirmation rises from 67% to 79%.Context Grounding & Constraints and expectation-related reasons remain substantial through middle stages, while Presentation Style and Scope Fit decline later.
5. Utility of Thoughts
ThoughtTrace thoughts provide actionable signals beyond conversation transcripts in two validated applications: predicting users’ next messages and improving model alignment. They substantially improve prediction performance and produce stronger response-revision supervision than messages.
- 5.1 Predicting User Behavior: Thought annotations improve next-user-message prediction across three frontier models, raising average semantic similarity from 21.6 to 30.6.Predictions use conversation history alone or history augmented with annotated reasons and reactions, scored on a 0–100 semantic-similarity scale.
- 5.1 Predicting User Behavior: The average prediction improvement is 41.7% relative, with gains of 14.2 for Claude Opus 4.6, 6.0 for GPT-5.4, and 6.8 for Gemini 3.1 Pro Preview.Thoughts capture latent reasons and reactions that help predict future user behavior.
- 5.2 Improving Model Alignment: Thought-guided rewrites outperform message-guided rewrites by +4.5% on Arena-Hard, exceed the base model by +25.6%, and exceed the WildChat baseline by +6.6%.The experiment fine-tunes Qwen3.5-4B using revisions prompted by thought content and dissatisfaction reaction labels.
- 5.2 Improving Model Alignment: Thoughts surface 1,000 dissatisfaction instances versus 450 in messages, providing 2.2× more supervision for response revision.Thoughts provide ground-truth reactions and combine identifying an unsatisfactory response with specifying how to revise it.
6. Conclusion
ThoughtTrace introduces a large-scale dataset pairing real-world human–AI conversations with users’ self-reported thoughts and establishes thoughts as a distinct modality with downstream value. The conclusion also identifies limitations from in-situ collection, conscious-reasoning coverage, and Prolific recruitment.
- Contributions: ThoughtTrace pairs real-world human–AI conversations with users’ self-reported thoughts as a large-scale dataset.The paper presents it as the first dataset of this kind.
- Contributions: Thoughts capture latent information beyond surface messages, challenge frontier LLM inference, span diverse content, and vary across conversation stages.The analysis characterizes thoughts as a distinct data modality.
- Contributions: Thoughts improve user behavior prediction when provided as inference-time context.This demonstrates downstream utility for modeling user behavior.
- Limitations and Future Work: In-situ thought collection may shape interactions, omit subconscious judgments, and introduce modest selection effects through Prolific recruitment.The dataset captures only consciously accessible reasoning, while demographic analysis suggests the recruitment effect is modest.
Author Contribution Statement
The authors divided contributions across project conception, data collection, analysis, experiments, advising, and manuscript preparation. Chuanyang led conception and data collection design, while all authors contributed to editing and feedback.
- Research and Analysis: Chuanyang and Tianmin conceived the project, and Chuanyang designed data collection and processed metadata.Chuanyang also contributed to conversation-property and thought-property analyses.
- Research and Analysis: Conversation- and thought-property analyses involved Chuanyang, Binze, and Cathy, while thought-utility experiments involved Chuanyang, Haopeng, and Tianjian.Advising was provided by Tianmin, Maximillian, Hongxiang, and Shayne.
- Manuscript Preparation: Chuanyang and Binze wrote the manuscript, and everyone contributed to manuscript editing and feedback.
A. Details of Models Used in ThoughtTrace · B. Additional Results
ThoughtTrace draws on interactions from 1,058 users across 2,155 conversations and 17,058 turns, with 10,174 thought annotations collected through a chatbot service powered by 20 language models. All models use inference temperature 0.7, and thinking-mode models display only final responses rather than intermediate reasoning traces.
- A. Details of Models Used in ThoughtTrace: 1,058 users contributed 2,155 conversations, 17,058 interaction turns, and 10,174 thought annotations across 20 language models.These model-wise statistics are summarized in Table A1.
- A. Details of Models Used in ThoughtTrace: All models use an inference temperature of 0.7.
- A. Details of Models Used in ThoughtTrace: For models with a thinking mode, the chatbot displays only the final response and hides intermediate reasoning traces enclosed in <think> and </think>.A loading indicator is shown during the thinking process.
B.1. Qualitative Examples of Frontier Model Failures in Thought Inference
Four qualitative cases show frontier models failing to infer users’ Reasons and Reactions, with all four models scoring 1/5 against the ground-truth thought. Reason failures follow surface topics instead of motivations, while Reaction failures confuse follow-up content with affective response.
- Overview: 1/5: GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro Preview each received 1/5 on representative Reason or Reaction thought-inference cases.Examples 1 and 3 use GPT-5.4, Example 2 uses Claude Opus 4.6, and Example 4 uses Gemini 3.1 Pro Preview.
- Reason prediction: Reason failures arise when models follow the latest surface topic instead of the user’s broader motivation or metacognitive intent.GPT-5.4 linked “it” to motorcycle switches rather than licensing and safety concerns, while Claude Opus 4.6 treated a newly raised problem as a standalone question rather than part of an ongoing solution process.
- Reaction prediction: Reaction failures occur when models mistake the content of a follow-up message for the user’s affective response.GPT-5.4 inferred dissatisfaction from the absence of a microwave despite genuine appreciation, while Gemini 3.1 Pro Preview inferred approval despite frustration with an over-scoped answer.
B.2. Qualitative Examples of User Behavior Prediction
Qualitative examples show that thought annotations can redirect next-message predictions toward users’ actual frustrations and preferences, but accurate thoughts do not always improve prediction. Their benefits are clearest when reactions reveal dissatisfaction with the assistant’s response pattern or level of detail.
- Success examples: Across three examples, thought-aware predictions improved when annotations exposed dissatisfaction with excessive detail or repeated questioning, shifting outputs toward the users’ actual requests and complaints.In the Paris case, all three models shifted toward concise options; in the anxiety case, GPT-5.4 and Gemini directly identified frustration with questioning.
- Success Example 1: Paris itinerary: In the Paris itinerary example, “too much data, narrow it down” flipped all three predictions toward a quick request for only the top few activities.The thought-aware outputs matched the ground-truth request for a shorter, narrower answer, whereas context-only outputs assumed cooperation with scoping questions.
- Success Example 2: Anxiety chat: In the anxiety chat, “It always ask me questions” shifted GPT-5.4 and Gemini toward direct complaints about questioning instead of generic answers.The actual next message asked why the assistant kept asking questions; Claude remained similar across conditions, though its thought-aware version was judged marginally closer in tone.
- Failure example: Piano learning: In the piano-learning failure example, all three thought-aware predictions lost because the user accepted realistic timing advice while redirecting the conversation toward class.The annotations accurately captured formatting preferences and growing openness to realism, but the actual next message ignored formatting, showing that high-quality thoughts do not always predict chosen wording.
B.3. Conversation, Message, and Thought Lengths · B.4. Full Topic Distribution · B.5. Task Descriptions and AI Expectations
ThoughtTrace contains longer, denser conversations than comparison datasets, with highly variable assistant responses and generally brief thoughts. Its topics span practical everyday concerns, while users describe structured, goal-oriented tasks involving planning, problem solving, and daily routines.
- B.3. Conversation, Message, and Thought Lengths: ThoughtTrace conversations spread more evenly across 2k–5k tokens and retain a non-trivial tail beyond 10k, unlike shorter comparison conversations.Nearly 60% of WildChat and over 90% of LMSYS-Chat-1M conversations fall below 1k tokens.
- B.3. Conversation, Message, and Thought Lengths: Assistant responses center around 561 tokens versus 13-token user prompts, with occasional responses exceeding 2,000 tokens.User prompt length remains roughly stable across turns, while assistant responses fluctuate between approximately 480 and 810 tokens per turn.
- B.3. Conversation, Message, and Thought Lengths: 27.1% of thoughts peak at 8–12 tokens, roughly three quarters fall between 4 and 20 tokens, and fewer than 3% exceed 40 tokens.Thoughts are unimodal and concentrated within a narrow range.
- B.3. Conversation, Message, and Thought Lengths: Average thought length is highest in opening turns at 15–18 tokens, then stabilizes at 11–13 tokens from turn 4 onward.The opening-turn pattern is associated with initial goal setting and exploration.
- B.4. Full Topic Distribution: Travel & Tourism (9.0%), Lifestyle (8.9%), and Food & Dining (8.4%) are the three most prevalent subtopics within Culture & Lifestyle.Culture & Lifestyle is identified as the largest parent category, with conversations concentrated on practical everyday concerns.
- B.4. Full Topic Distribution: The full topic distribution covers 36 fine-grained subtopics organized under seven parent categories.Figure A4 provides the detailed breakdown underlying the seven parent categories summarized in the main text.
- B.5. Task Descriptions and AI Expectations: Task summaries prominently feature planning, trip, problem solving, and daily routine, indicating structured, goal-oriented requests.Users provide both a task summary and a task expectation for each interaction in multi-turn conversations.
B.6. Embedding Differences Between Messages and Thoughts
Embedding analyses show that consecutive user messages are distributionally similar, while reasons and especially reactions add distinguishable semantic information. Message–reason pairs remain semantically aligned but differ moderately in distribution, whereas reaction–next-message pairs exhibit the largest shift.
- Method: The analysis compares embeddings for consecutive messages, message–reason pairs, and reaction–next-message pairs using shared UMAP projections and three distributional metrics.The metrics are Centroid Distance, RBF-kernel MMD, and Linear Probe AUC.
- Distributional differences: Current and next messages show the smallest separation, with Centroid = 0.120, MMD = 0.096, and AUC = 0.721.Their embeddings largely overlap, indicating that consecutive user messages are drawn from similar distributions.
- Distributional differences: Message–reason pairs show moderate separation, with Centroid = 0.225, MMD = 0.182, and AUC = 0.977.The embeddings substantially overlap, reflecting shared semantic grounding, while remaining distinguishable at the distribution level.
- Distributional differences: Reaction–next-message pairs show the largest shift, with Centroid = 0.320, MMD = 0.257, and AUC = 0.988.This contrast indicates that reactions to LLM responses introduce more distributional change than reasons for the original message.
B.7. Relationships Between Thought Types and Conversation Properties … C.6. Safeguards
Thought types shift with conversation dynamics but remain largely independent of topic and length, while explicit affirmation dominates reactions across models. The data collection used consented, tutorial-guided annotation procedures, limited post-task surveys and cleaning, and safeguards for cognitive self-report data.
- B.7. Relationships Between Thought Types and Conversation Properties: Task Motivation dominates opening turns, while continuation-, context-, and expectation-related reasons prevail during re-attempts, variations, and extensions.This indicates a shift from goal-setting toward refinement as conversations progress.
- B.7. Relationships Between Thought Types and Conversation Properties: Thought types are largely independent of conversation topics and lengths; explicit affirmation is linked to slightly shorter remaining conversations, without a significant effect.The topic and length analyses report broad independence, with explicit affirmation as a minor exception for remaining length.
- C. Details of Data Collection Methodology; C.1. User Consent: Participants were recruited through Prolific, compensated above the applicable minimum wage, required to self-report English fluency, and enrolled voluntarily under prior IRB approval.They received information about the study, duration, anonymity, voluntary participation, and withdrawal rights before providing informed consent.
- C.2. Tutorial: A step-by-step tutorial used plain-language instructions and interface demonstrations to teach natural chatbot interaction and contextually grounded thought recording.Participants recorded reasons for each message and reactions to each AI response, including satisfaction or dissatisfaction.
- C.3. Chat Interface: The web chat interface used HTML, CSS, JavaScript, and Firebase Firestore, with a 10:00 timer and private reason and reaction annotation controls.Thoughts were not sent to the AI, and participants could continue annotating after timeout.
- C.4. Post-Chat Surveys: After each task, participants answered open-ended questions about the completed task and their expectations, then completed a six-question demographic and AI-use survey after both tasks.The background survey covered age, gender, education, occupation, AI-chat usage frequency, and main AI-use purposes.
- C.5. Data Cleaning: Most collected data were retained, with small removals for unusually fast completion, chatbot-response failures causing low-quality conversations, or extremely low-quality conversations lacking annotations and complete surveys.These cleaning rules targeted engagement, system-failure artifacts, and incomplete data while preserving the original dataset characteristics.
C.7. Limitations … D.2. Conversation Property 2: ThoughtTrace Features Long-horizon Diverse Conversations
ThoughtTrace’s in-situ thought collection entails reactivity and recruitment constraints, although demographic analysis suggests limited distribution shift. Its analyses characterize conversation length, message length, and multi-label topic diversity using standardized aggregation and labeling procedures.
- C.7. Limitations: Thought elicitation can reshape interactions because participants may adjust stated preferences or intentions to make later annotations easier.This reactivity is inherent to in-situ collection: unsolicited thought reporting would require invasive instrumentation that is unethical and infeasible at scale.
- C.7. Limitations: Recruitment introduces a modest selection effect, but demographic analysis indicates that ThoughtTrace spans diverse AI users and everyday use cases.The reported distribution shift is described as small relative to the value of obtaining rich, consented thought annotations.
- D.1. Conversation Property 1: ThoughtTrace Captures a Representative Spectrum of Users: Participant profiles are aggregated across six axes: age, gender, education, occupation, LLM-use frequency, and free-text usage purposes.Responses come from post-task surveys completed alongside conversations, retaining the first survey response for each conversation.
- D.1. Conversation Property 1: ThoughtTrace Captures a Representative Spectrum of Users: Demographic responses are standardized into age brackets, Likert-based usage anchors, fixed gender and education categories, and canonicalized open-ended fields.Age uses six brackets from 18–24 through 65+, while usage frequency ranges from “Never” to “Many times a day.”
- D.2. Conversation Property 2: ThoughtTrace Features Long-horizon Diverse Conversations: Conversation and message-length analyses use a shared message-level frame that records role, one-indexed turn position, and GPT-4o token counts.Conversation lengths are aggregated per conversation in tokens or turns, with ThoughtTrace compared against WildChat using corpus-level distributions.
- D.2. Conversation Property 2: ThoughtTrace Features Long-horizon Diverse Conversations: Prompt and response lengths are analyzed with role-specific token histograms and median markers, while turn-position analyses average lengths across turns 1–20.User means are retained at odd positions and assistant means at even positions to respect dialogue alternation.
- D.2. Conversation Property 2: ThoughtTrace Features Long-horizon Diverse Conversations: Conversation topics are assigned by GPT-5.4 using a predefined taxonomy, allowing multiple labels when a conversation spans domains.Topic counts are aggregated into a two-level hierarchy with broader manually defined categories such as Technology, Business & Society, and Arts & Entertainment.
D.3. Conversation Property 3: ThoughtTrace Conversations are Dominated by Task Extension … D.9. Thought Utility 2: Thoughts Improve Model Alignment
The paper characterizes ThoughtTrace conversations, thoughts, and their utility through turn-relationship labeling, semantic analyses, stage-based categorization, next-message prediction, and thought-guided alignment training. These analyses treat thoughts as distinct, difficult-to-infer, diverse, stage-dependent signals that improve behavioral prediction and support preference optimization.
- D.3. Conversation Property 3: ThoughtTrace Conversations are Dominated by Task Extension: Users’ successive turns are deterministically classified as first requests, task extensions, revisions, new variations, or completely new requests.GPT-5.4 compares each current user prompt with its predecessor and stores the relationship labels at both message and conversation levels.
- D.4. Thought Property 1: Thoughts Are Different from Messages: Thoughts provide information that is largely absent from users’ surface utterances, as indicated by low semantic coverage between messages and associated reasons or reactions.Coverage is scored separately for reasons and reactions on a 1-to-5 scale.
- D.5. Thought Property 2: Thoughts Are Difficult for LLMs to Infer: Three frontier models struggle to infer users’ reasons and reactions from dialogue context, with low similarity to human-written thoughts indicating that these signals are not reliably recoverable.GPT-5.4, Gemini 3.1 Pro Preview, and Claude Opus 4.6 are evaluated against human annotations using a fixed five-point semantic-similarity rubric.
- D.6. Thought Property 3: Thoughts Are Diverse in Content: Thoughts are categorized into diverse content types, including task motivations and continuations, content and style expectations, explicit affirmations, satisfaction, presentation style, scope fit, and content relevance.Reasons are interpreted with conversation history, whereas reactions are classified from the specific assistant response they evaluate.
- D.7. Thought Property 4: Thought Dynamics Depend on Conversation Stages: Thought dynamics are examined across Early, Mid-Early, Mid-Late, and Late conversation stages, using normalized category distributions, inter-stage flows, and positional statistics.The analysis also relates thought labels to conversation topics, multi-turn relationship types, total conversation length, and remaining conversation length.
- D.8. Thought Utility 1: Thoughts Predict User Behavior: Across all three predictor models, thought-augmented next-message prediction consistently outperforms the history-only baseline, showing that thoughts add actionable information about subsequent user behavior.The experiment compares raw dialogue history with contexts interleaving annotated reasons and reactions, scoring semantic similarity against the actual next message.