Building Machines that Learn and Think with People

• Language models catalyze a shift toward thought partnership by letting users engage with computers through natural language.Source: §1 · p. 1
• Effective thought partners should understand people, be understandable to them, and know enough about the world to engage on common ground.Source: §1 · p. 2
• Scaling foundation models with demonstrations, feedback, and traces can mimic human behavior but does not robustly simulate cognition such as reasoning about the world or other minds.Source: §1 · p. 2
• A promising design path builds explicit, structured models of the task, world, and human, drawing on formal cognitive-psychology frameworks for human thought.The paper argues that effective thought partners build models of both the human and the world.Source: §1 · p. 2
• The authors propose applying a toolkit spanning foundation models, probabilistic programming, goal-directed search, and structured representations of agents to collaborative cognition.Source: §1 · p. 2

2.1 Modes of Collaborative Thought

• The proposed modes, inspired partly by psychology, are illustrative rather than comprehensive and are presented as promising directions for AI collaboration across domains.The domain examples highlight computational challenges that motivate the proposed desiderata.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Example Domains

• Programming partners should infer users’ intentions and knowledge, answer why code works or fails, and go beyond inline suggestions; current tools can miss intentions and generate risky bugs [46–50].A good partner adapts to what programmers do and do not know, rather than specializing only in in-line code suggestions.Source: §2.2 · p. 3
• Embodied partners should interpret actions and instructions as goals grounded in physical possibilities, account for mutual limitations, and make intentions legible; current models struggle with novel-goal planning [57–69].Reliable assistance may range from cooking with a friend to search-and-rescue, and requires correcting for mistakes in perception, planning, and world modeling.Source: §2.2 · p. 3
• Storytelling partners can help iteratively ideate and revise, but must model both authors’ intentions and audience expectations and likely interpretations [70–77].The audience’s social world matters alongside the writer’s intentions and dispositions.Source: §2.2 · p. 3
• Doctors must integrate evidence under uncertainty despite limited time and burnout [78,79]; medical foundation models may assist [80–84], but their user/world understanding and explainability remain uncertain [85–88].Thought partners could free doctors to spend more time communicating with patients, but safety, reliability, and transparency remain concerns.Source: §2.2 · pp. 3–4

2.3 Desiderata

• Thought partners should understand users’ goals and limits [90,91], communicate legibly [68,92–95], and share a reality-tethered world model to enable synergy beyond either partner alone [96–99].The proposed criteria center on the relationship to the user, alongside conventional tool criteria such as efficiency, accuracy, robustness, fairness, cost, and scalability.Source: §2.3 · p. 4

3 Engineering Human-Centered Thought Partners

• The authors propose explicitly engineering cooperative thought partners from computational cognitive science, rather than leaving the desiderata to emerge from systems optimized for other ends.They argue that models of human cognition and choice as approximately optimal given goals and constraints provide a starting point, with Bayesian formalism as a shared language across disciplines.Source: §3 · p. 4

3.1 Implementing Our Desiderata

• Thought partners should model collaborators’ beliefs, knowledge, goals, and resource limits, adapt to their strengths and constraints, and ground shared goals and communication in the task environment.They should also reason about the computational demands placed on people to predict their behavior appropriately [18, 157].Source: §3.1 · pp. 4–5

3.2 Computational Cognitive Science Motifs

• Bayesian models of cognition capture human belief updating, integration of commonsense knowledge with evidence, and flexible hierarchical planning over varied timescales.These approaches model cognition as inference over structured probabilistic generative world models and have been applied across diverse cognitive domains [21, 131, 137, 158, 159].Source: §3.2 · p. 6
• Bayesian theory-of-mind models explain how people infer others’ mental states, while Rational Speech Act models capture recursive reasoning by speakers and listeners.Mental models of partners can support communication, teaching, reliance decisions, and adaptation to new conversation partners [33, 147, 183–185].Source: §3.2 · pp. 6–7
• Resource-rational accounts explain biased judgments through finite time, memory, and attention, while tractable theory-building models people as constructing compressed world representations, including programs.Approximate inference and bounded planning can capture resource limits; program synthesis can help explore suboptimal behavior [17, 142, 143, 182, 196, 201, 202].Source: §3.2 · p. 7

3.3 Scaling Thought Partners via Probabilistic Programming

• Probabilistic programming offers a cognitively grounded approach for evolving, learning, and inspecting models of people and the world, with programmable inference combining optimization, dynamic programming, and Monte Carlo methods.Programs can be learned from data or synthesized with LLMs, and their models can be modularly inspected and edited by humans.Source: §3.3 · p. 7

3.4 Infrastructure around Thought Partners

• Effective thought partnership requires designing the surrounding workflow and sociotechnical infrastructure, including when people engage AI and which partner starts a task.The workflow should account for partners’ strengths, weaknesses, and task costs, and may be shaped by regulation, organizational practice, and studies of human behavior [73, 212–215].Source: §3.4 · p. 7

4.1 Thought Partners for Programming

• Chandra et al.’s WatChat [106] uses Bayesian mental-state inference to explain surprising program behavior by inferring and debugging programmers’ misconceptions.It represents possible mental models as programs with bugs, enabling Bayesian program synthesis and the generation of questions that help teachers or learners identify misconceptions.Source: §4.1 · pp. 8–9

4.2 Thought Partners for Embodied Assistance

• CLIPS [35] infers collaborators’ shared goals and plans from actions and instructions, enabling context-sensitive instruction following and proactive assistance.It models humans as cooperative planners [65] and combines Bayesian planning and alignment tools with LLMs to infer intentions from natural-language instructions.Source: §4.2 · p. 9

4.3 Thought Partners for Storytelling

• Recent work grounded in Bayesian ToM predicts and designs interventions on a story audience’s experience; Chandra et al. frame storytelling as inverse inverse planning, optimizing narrative events to shape model inferences over time [224,225].Plot twists and stage mime are expressed in this framework to create animations with desired cognitive effects on viewers.Source: §4.3 · p. 9

4.4 Thought Partners for Medicine

• No medical system yet meets the desiderata, but Bayesian thought partners could support collaborative diagnosis and treatment by modeling care agents, updating medical knowledge, and coordinating human-AI query routing.Early work uses probabilistic programming to learn generative oncology models and support user queries; routing systems based on probabilistic modeling have shown success in simulation.Source: §4.4 · pp. 9–10

5.1 Non-Dyad Settings

• Many humans and machines may collaborate across roles and specialties, extending thought partnership beyond dyads into complex social systems.Examples include group learning with an AI partner, medical robots accounting for multiple humans, and citizen-science networks that combine human and machine classifications and support scientific discovery [235,236,238].Source: §5.1 · p. 10

5.2 Evaluation

• Evaluating thought partners requires interactive, cross-disciplinary methods that assess interaction, first-person experience, and preference alongside output quality.Games offer a controlled yet rich setting for studying repeated interactions and social collaboration among multiple agents [241–244].Source: §5.2 · p. 10

5.3 Risks and Important Considerations

• Thought partners risk both over-reliance that impairs critical thinking and under-reliance that excludes people with less AI experience [219,245–247,251–253].Cognitive forcing functions may help address over-reliance [249,250], while AI literacy training and access are important so benefits are not limited to an exclusive group; over-reliance has also been likened to “steroids” for the mind [248].Source: §5.3 · p. 11
• Human-centric partners should be understandable without being mistaken for humans, since human-like presentation can invite undue intention attribution and devalue human intelligence [254,6,255].Maintaining categorical distinctions between humans and machines can help prevent over-reliance and promote human dignity [245,256,257]; terms such as “teammate” or “tool” shape assumptions about machine capabilities and human-likeness.Source: §5.3 · p. 11
• Inaccurate, non-robust, or cognitively ungrounded human models can misalign partners, while Bayesian uncertainty-awareness may reduce overconfidence but cannot remove dual-use risks [223,258–263].Misalignment may lead to wrong goals, misleading information, or safety violations; models of people can also enable surveillance or manipulation, making alignment with society—not just users—important [264].Source: §5.3 · p. 11

• Design for helpful, reliable human–AI thought partnerships should recognize and engage with the richness and diversity of human thought in an unpredictable world.Source: §6 · p. 11
• Case studies support drawing on computational cognitive science and cognitive-AI to engineer thought partners and the infrastructure around their use.Source: §6 · p. 11
• The authors describe collaborative cognition’s future as bright but risky, and call for continual knowledge sharing among behavioral scientists, AI practitioners, domain experts, and related disciplines.Source: §6 · p. 11