PaperlayerView exact revision on arXiv

Source-linked AI summary

Creating General User Models from Computer Use

Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, Michael S. Bernstein

arXiv:2505.10831v3cs.HCcs.AIcs.CL

TL;DR

Current user models are narrow and lack the broad, cross-context grounding needed for computers to understand users and act helpfully. This paper presents GUMs, which transform unstructured computer observations into confidence-weighted, revisable propositions that applications can query. The evaluations report rapid, calibrated, accurate inferences and proactive assistance that discovers and attempts useful actions.

  • Problem

    Current user models are narrow, while AI systems lack the shared context needed to reason about users across contexts.

  • Method

    GUMs use multimodal observations to construct confidence-weighted propositions, continuously refine them, and expose them through a queryable interface.

  • Results

    Evaluations found that GUMs produce rapid, calibrated, and accurate inferences, while Gumbo surfaces helpful suggestions and attempts to complete them.

  • Takeaways & Limitations

    GUMs provide a framework for grounding language models in observed behavior and enabling proactive systems that act on a user’s behalf.

  • Takeaways & Limitations

    The evaluation generalizes most strongly to technical users familiar with AI, and the authors identify privacy and overly candid propositions as boundaries.

Abstract

from arXiv · show

Human-computer interaction has long imagined technology that understands us-from our preferences and habits, to the timing and purpose of our everyday actions. Yet current user models remain fragmented, narrowly tailored to specific apps, and incapable of the flexible reasoning required to fulfill these visions. This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer. The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted propositions that capture user knowledge and preferences. GUMs can infer that a user is preparing for a wedding they're attending from messages with a friend. Or recognize that a user is struggling with a collaborator's feedback on a draft by observing multiple stalled edits and a switch to reading related work. GUMs introduce an architecture that infers new propositions about a user from multimodal observations, retrieves related propositions for context, and continuously revises existing propositions. To illustrate the breadth of applications that GUMs enable, we demonstrate how they augment chat-based assistants with context, manage OS notifications to selectively surface important information, and enable interactive agents that adapt to preferences across apps. We also instantiate proactive assistants (GUMBOs) that discover and execute useful suggestions on a user's behalf using their GUM. In our evaluations, we find that GUMs make calibrated and accurate inferences about users, and that assistants built on GUMs proactively identify and perform actions that users wouldn't think to request explicitly. Altogether, GUMs introduce methods that leverage multimodal models to understand unstructured context, enabling long-standing visions of HCI and entirely new interactive systems that anticipate user needs.

1 Introduction

The paper introduces General User Models (GUMs), which build computational representations of users from unstructured observations and support applications that reason across contexts. It demonstrates GUM-based assistants, including Gumbo, while reporting accurate, calibrated inferences and identifying privacy and candor boundaries.

  • The architecture lets applications query user representations in real time to add context and adapt experiences across applications.
  • Gumbo continuously observes a computer, discovers context-conditioned suggestions, and decides whether and when to intervene or execute them.
  • Examples include finding a suit rental after a wedding invitation, proposing system bug fixes, and suggesting paper revisions.
  • N=18 participants judged GUM propositions overall accurate and well-calibrated, with confidence tracking correctness.
  • Evaluations identify privacy considerations and overly candid propositions as limitations and boundary conditions.
  • GUMs represent a user’s behavior, knowledge, beliefs, and preferences from unstructured observations.

2 Related Work

Related work frames GUMs as a response to AI systems’ limited shared context and narrow, interaction-bound grounding. The paper instead proposes proactively building common ground from unconstrained multimodal observation with a flexible context structure.

  • Current AI systems often lack shared context, producing over-informative, overconfident, and ambiguity-prone responses and potentially unfavorable actions.
  • Existing grounding approaches rely on clarification, dialogue history, or lengthy inputs, tying grounding to specific interactions or domains.
  • GUMs proactively build common ground by observing users rather than grounding only when users interact with a model.
  • The paper uses unconstrained input as a multimodal grounding engine instead of constraining model inputs, outputs, and tasks.
  • Unlike context-aware frameworks tied to fixed data structures or sensors, GUMs use a flexible structure with few constraints.

3 General User Models

GUMs convert everyday observations into confidence-weighted propositions about users, refine those propositions over time, and expose them to applications through an interface. The representation supports uncertainty, staleness, grounding, and permissioned observation streams.

  • GUMs learn stable user inferences and moment-to-moment context from low-level inputs such as screenshots.
  • Propositions and confidences: A GUM is a collection of natural-language propositions paired with confidence scores from 0 to 1.
  • Propositions and confidences: Observations are factual, metadata-bearing records, while propositions are inferred textual assumptions constructed from those observations.
  • Propositions and confidences: Confidence reflects evidence quality and quantity, enabling decisions that prioritize reliable information while representing uncertainty.
  • Propositions and confidences: Propositions also carry decay scores, supporting applications that distinguish slowly changing facts from transient activities.
  • Interfacing with the GUM: Observers convert permitted screen, notification, filesystem, and manual-input events into new propositions, which users and applications can query naturally.

4 Applications of General User Models

GUMs let applications use cross-application context to personalize interactions, manage attention, and support proactive assistance. Gumbo demonstrates discovering, evaluating, and executing suggestions based on continuously observed user context.

  • 4.1 Prompting Models: GUMs address underspecified prompts by retrieving propositions about the user’s current work, relevant materials, and prior considerations.This context lets language models produce responses tailored to the user’s ongoing task rather than request clarification or rely on a narrow chat history.
  • 4.2 Any Application Can Query A GUM: Applications can query a GUM to adapt experiences using context gathered across applications and time.The architecture supplies a shared context layer for systems including chat assistants, operating systems, and interface agents.
  • 4.2 Any Application Can Query A GUM: GUM-enhanced operating systems can selectively surface notifications by combining user priorities, deadlines, and notification-ignore patterns.In the example, contextual propositions allow only the CHI registration notification through while filtering routine billing messages.
  • 4.2 Any Application Can Query A GUM: GUM-augmented interface agents adapt recommendations to the user’s interests and current priorities, including whether a recommendation is timely.The movie-agent example uses draft comments and a paper deadline to avoid an inappropriate recommendation context, while an ablation without deadline information still recommends relevant movies.
  • 4.3 Gumbo: an Assistant Built On GUMs: Gumbo ingests screenshots, builds an internal GUM, discovers suggestions, decides whether to show or execute them, and performs sandboxed work before sharing results.It can complete reversible tasks on the user’s behalf while avoiding irreversible side effects such as ordering a suit.
  • 4.3 Gumbo: an Assistant Built On GUMs: Gumbo’s evaluations include personalized planning, research brainstorming, email organization, paper revisions, and proactive bug-fix suggestions.Computer Use was disabled during evaluation because it was considered too slow and buggy, although the authors expect future advances to benefit GUMs.

5 Constructing a General User Model

GUM constructs a private user model from unstructured observations by filtering sensitive data, generating confidence-weighted propositions, retrieving relevant context, and revising propositions over time. Its retrieval process prioritizes recent, diverse, and semantically related information before reranking it.

  • 5.1 Observing Users: GUMs ingest unstructured observations and aim to operate locally without sending private user data to third-party AI platforms.The implementation uses open-source models for screen observation and proposition generation, supporting the authors’ local-deployment principle.
  • 5.2 Auditing Observations: The Audit module filters listener updates for privacy violations before sensitive observations reach the proposition collection.It applies contextual-integrity questions to distinguish ordinary professional communication from disclosures such as banking credentials.
  • 5.3 Constructing Propositions: The Propose module converts observations into propositions with reasoning traces and confidence scores that represent evidential support.Confidence distinguishes direct observations, such as a wedding invitation, from speculative inferences, such as needing to rent a suit.
  • 5.3 Constructing Propositions: The pipeline observes behavior, audits updates, constructs confidence-weighted propositions, retrieves similar context, and revises the model iteratively.This sequence is designed to refine the GUM as new observations reinforce or contradict prior propositions.
  • 5.3 Constructing Propositions: GUM assigns proposition-specific decay scores so stable facts remain relevant longer than transient activities.The score is generated from the proposition, its confidence, reasoning, and observations rather than applying one global decay rate.
  • 5.4 Retrieving Context-Sensitive Propositions: Recency-adjusted retrieval combines relevance with proposition age using γ_i = exp(−α_i·k·age(d_i)), where k = 2 and age is measured in days.Retrieved propositions are then reranked as identical, similar, or unrelated, while MMR balances relevance and diversity.

6 Evaluating General User Models: Accuracy and Calibration

The evaluation tests whether GUMs can synthesize accurate, calibrated user propositions from noisy email histories and whether retrieval and revision improve proposition quality. Across 18 participants, the full architecture was preferred over ablations, while email-only propositions were generally accurate but limited as a proxy for broader context.

  • Evaluation setup: Across 18 participants, GUM propositions were judged accurate and well calibrated despite irrelevant content in email histories.Participants trained GUMs on sequentially processed emails containing metadata, attachments, links, replies, and substantial irrelevant material.
  • Comparisons with ablations: Retrieval and revision were interdependent: without relevant retrieval, revision failed to capture long-range dependencies between spaced-apart emails.The evaluation compared no-retrieval/no-revision, no-retrieval, and full-architecture conditions using participant rankings and pairwise win rates.
  • Comparisons with ablations: The full GUM had a significantly higher pairwise win rate than both ablations: 0.618 ± 0.07 versus 0.463 ± 0.07 and 0.420 ± 0.07.The difference was significant after Holm correction; no significant difference appeared between the two ablations without full retrieval and revision.
  • Accuracy and calibration: Participants rated email-derived propositions 6.47 on a 1–7 accuracy scale, while overall proposition accuracy was 76.15 ± 5.85%.The full model’s accuracy was not significantly higher than the ablations when propositions were evaluated in isolation.
  • Accuracy and calibration: GUMs achieved a Brier score of 0.17 ± 0.03, with most calibration error arising from underconfidence rather than overconfidence.Underconfidence was considered preferable in this setting because it supports more cautious adaptation and avoids wrongly assuming knowledge of user preferences.
  • Scope and qualitative findings: Email-based GUMs sometimes over-indexed on one life facet, showing that email is a constrained proxy for users’ broader context.The paper addresses this boundary in a later end-to-end evaluation using computer screenshots collected over five days.

7 Evaluating General User Models: Privacy Audit Module

The Audit module uses a GUM's inferred contextual privacy preferences to assess whether observed information flows fit social-context norms. In evaluation, the full GUM generally respected contextual integrity, but privacy violations and overly candid social inferences revealed important trust boundaries.

  • Privacy Audit Module: The Audit module infers user-specific contextual privacy preferences while processing interaction data.It instantiates contextual integrity theory, which evaluates whether information flows match social-context norms.
  • Procedure: Participants evaluated propositions for privacy violations and selected the best contextual-integrity response across GUM variants.The evaluation used the same email setting and 18 participants as the accuracy and calibration evaluation.
  • Results: 7 of 180 propositions from the full GUM were flagged as contextual-integrity violations.Participants agreed or strongly agreed with contextual-integrity responses at μ=6.06 on a 1-7 Likert scale.
  • Results: 61.1% of participants selected the full GUM as producing the best contextual-integrity response, versus 22.2% without retrieval and 16.6% without revision.The full GUM was compared with ablations removing retrieval or revision.
  • Limitations: Unexpectedly explicit social inferences, such as naming friends from email, marked a potential privacy boundary for some users.The authors note that users may explicitly dictate desired contextual privacy norms, but maintaining trust remains delicate.
  • Limitations: Phishing emails, advertisements, and spam can rewrite GUM propositions, exposing the system to injection attacks.The authors believe such attacks can be mitigated, though not entirely eliminated without further research.

8 Evaluating General User Models: End-to-End via Gumbo

The end-to-end Gumbo study found accurate, calibrated propositions and useful suggestions spanning immediate assistance, unexpected needs, and personalized support. It also exposed concerns about agency, privacy, candid judgments, execution failures, and the limits of screen-based context.

  • Evaluation: GUM propositions were accurate and calibrated in the end-to-end evaluation, with errors arising from underconfident predictions.Participants rated propositions 0.79 ± 0.07 for accuracy and Brier = 0.28±0.04 for calibration.
  • Agency and interpretation: Participants differed in their reactions to candid judgments about stress, priorities, abilities, and dissatisfaction, ranging from validation to feeling judged.Some propositions supported self-reflection, while ability-related propositions were considered unconstructive.
  • Suggestions: 25% of suggestions were rated strong or excellent, and some participants wanted to keep Gumbo running after the study.Highly rated suggestions often made extensive use of the underlying GUM.
  • Suggestions: Participants found useful suggestions for both immediate tasks and needs they had not explicitly signaled.Examples included Gmail notification setup, presentation advice, research brainstorming, and personalized moving plans.
  • Personalization: Gumbo adapted suggestions to user preferences, such as offering travel ideas rather than taking over travel planning.The system used the user’s preference for planning travel themselves to limit its intervention.
  • Errors and boundaries: Gumbo’s agentic execution remained unreliable: 20.69% of suggestions were on-track but failed during execution.Failures reflected underlying model and tool-access limitations, including absent notification APIs and lack of internet ingestion.
  • Errors and boundaries: The study raised privacy concerns about sensitive inferences and the generalizability of a GUM trained only on computer activity.Participants noted that events outside the computer would not be captured, while recruitment and self-hosted models shaped trust in the study.

9 Discussion

The discussion frames GUMs as promising but socially and technically constrained systems whose benefits increase with access to personal data. It emphasizes privacy, manipulation, bias, sampling, hallucination, and incomplete context as central deployment boundaries.

  • Implications: GUMs showed promise across technical evaluations and end-to-end deployment, while raising questions about how users should encounter their propositions.The discussion considers raw, sanitized, hidden, or context-adapted representations.
  • Reflecting on Propositions: Accurate propositions can support downstream assistance but may also expose value judgments and encourage actions that users believe are helpful without being helpful.The authors connect candid assessments to both useful suggestions and risks such as canceling advising meetings to make time for ice cream.
  • The Privacy Paradox: The privacy paradox is intensified because GUM precision and downstream recommendations improve as users provide more data.Two of five participants wanted to continue using a version of Gumbo after the study.
  • Ethical and Societal Risks: The authors recommend local or personal hosting and controlled, audited access to reduce risks from persuasion, advertising, surveillance, and unauthorized training.Their implementation used open-source models and infrastructure managed by the research team.
  • Ethical and Societal Risks: GUMs can be manipulated by malicious inputs such as spam email, potentially affecting every application that relies on their propositions.The paper suggests asking users to confirm inputs that appear out of character as one possible mitigation.
  • Ethical and Societal Risks: Deployment requires broader testing because GUMs inherit model bias, may hallucinate or misattribute information, and were evaluated with a small institution-centered sample.The authors say results likely generalize most strongly to technical users familiar with AI.
  • Technical Boundaries: Screen activity remains an incomplete proxy for context, so GUMs may fail to generalize when important actions occur outside the observed computer environment.The discussion also notes that current models struggle with long contexts and generic propositions, motivating retrieval and explicit confidence verbalization.

10 Conclusion

The paper presents GUMs as models that convert everyday behavior into rich, confidence-weighted representations of user context. Gumbo demonstrates applications ranging from contextual grounding to proactive suggestions, while evaluations report accurate and calibrated inferences.

  • Conclusion: GUMs transform unstructured everyday interactions into confidence-weighted propositions and continuously refine those inferences as new evidence accumulates.They are intended to move beyond siloed, application-specific user models toward representations spanning many life contexts.
  • Conclusion: GUMs can ground language models in observed user behavior and support proactive systems that surface and attempt to complete helpful suggestions.Gumbo is the paper’s interactive assistant demonstration of this capability.
  • Conclusion: The evaluations report rapid, calibrated, and accurate inferences about users.This is presented as evidence for the utility of GUMs across the paper’s demonstrated applications.

B Abridged Prompts

The appendix summarizes prompts for confidence calibration, proposition retrieval and revision, mixed-initiative suggestion filtering, and tool selection in Gumbo.

  • B.1 GUM prompts: GUM elicits conservative verbalized support scores from 1–10 alongside generated propositions.High scores require explicit, direct evidence rather than mere application visibility; the resulting calibration was overall good.
  • B.1.2 Reranker: The retrieval reranker classifies proposition pairs as highly related, somewhat related, or different.Based on these classifications, the system skips, revises, or adds propositions.
  • B.2.1 Mixed Initiative Interaction: Gumbo evaluates each suggestion by benefit, false-positive cost, false-negative cost, and decay over time.The prompt supplies the suggestion, propositions, and underlying observations as context.
  • B.2.1 Mixed Initiative Interaction: Suggestion evaluation also considers how quickly usefulness diminishes, distinguishing immediately obsolete assistance from help that remains useful long-term.This is represented by the prompt’s decay scale from 1 to 10.
  • B.2.2 Tool Use: Gumbo selects external tools using the suggestion and observation as context, with available options including language-model responses, search, filesystem access, reasoning, and image generation.The appendix shows a prompt that asks for a JSON list of useful tools and parameters.
  • B.2.2 Tool Use: The tool inventory includes filesystem access only when a file certainly exists on the user’s computer.The listed tools also include reasoning for challenging coding or math problems and image generation from a prompt.
  • B.2.2 Tool Use: The tool-selection prompt returns a JSON-formatted list of useful tools with parameters.This provides the interface between Gumbo’s suggestion context and downstream execution.

C Survey Questions

The email study survey asked participants about their demographics, email use, inbox content, proposition accuracy and relevance, data-sharing preferences, and privacy perceptions.

  • Participants reported demographic information, email-checking frequency, number of regularly used accounts, primary use cases, and inbox content.
  • Participants rated how accurately propositions reflected their email context, including the accuracy of high-confidence propositions.
  • Participants identified specific propositions and data-sharing generations they considered most accurate.
  • Participants rated agreement with their best data-sharing response and explained whether propositions respected their context and privacy.

D Interview Questions

The end-to-end interview examined participants’ expectations and experiences with Gumbo, the accuracy and personal impact of propositions, the usefulness of suggestions, and desired future uses.

  • Interviews began by asking about participants’ expectations, overall experience, and unmet expectations when using Gumbo.
  • Participants evaluated how well propositions understood them, how the propositions felt, which they liked, whether they edited them, and which they disputed.
  • Participants were asked whether propositions revealed something they had not previously recognized.
  • Participants assessed whether suggestions understood their goals and concerns, provided useful or unusable recommendations, and influenced their actions.
  • Participants described desired long-term uses, improvement features, recommendations to others, and additional feedback.

E Screen Observer Outputs

The screen observer produces text transcriptions of the current screen and descriptions of user actions across recent frames, which the GUM uses to create propositions.

  • The screen observer transcribes the active application, window title, open tabs, open file, and function being edited.
  • A second transcription component describes user actions across the past 10 unique frames.
  • The anonymized example captures Figma design work on a user-model interface with confidence values and recommendation cards.
  • The example also captures Overleaf editing of the General User Models paper and reference to its Figure 1.
  • The Xcode example records development activity, threshold-based suggestion notifications, and backend or symbol warnings.
Loading 2505.10831v3…