PaperlayerView exact revision on arXiv

Source-linked AI summary

nanoMuse: An Open-Source Personal Agent for Every Device You Own

Guangyi Liu, Yong Liu, Jiangning Zhang

arXiv:2610.08699v1cs.AI

TL;DR

Personal agents are expected to act for one person across accounts and devices, remember them over time, and answer for their actions, yet many everyday phone apps remain beyond cloud agents’ reach. The report presents nanoMuse as an open, device-based counterpart with shared conversation, governed actions, and person-readable memory; it also points to physical devices as a later extension.

Abstract

from arXiv · show

Assistants from 2011 answered and waited, and agents from 2023 did a task and stopped. In September 2026 Meta's Muse showed an agent for one person, with accounts, devices, memory and a conversation that lasts, closed, in a vendor's cloud, in one country. Such an agent is expected to act on a person's accounts and devices, remember them across weeks, speak first when it is worth it, and answer for what it did. It is a kind of software, not a model, and until now had no open counterpart. This report defines the personal agent in five questions and three horizons. It reads how Muse is built from Meta's public record and a copy of its production prompt, each statement marked by its source. It then presents nanoMuse, the open-source counterpart under the GPL-3.0, one agent on every device a person owns, with hands on the phone's screen and the computer's. They share one conversation over a relay anyone can run; every action goes through a Sentinel, memory is files the person can read, and the model is their choice. Its size and cost are given as estimates. What is open, memory with provenance, an evaluation suite for the hands and an open model for them, is set out as a roadmap.

1 Introduction

A personal agent acts for one person across their accounts, devices, and files over time, then answers to them for its actions. nanoMuse is presented as an open counterpart that should be inspectable and run on a person’s own devices where possible.

1 Introduction

  • A personal agent acts for one person across accounts, devices, and files, works for weeks, and answers afterward for what it did.Its record includes a log, a ledger, and the permissions it holds.Source: §1 · p. 2
  • Muse showed the personal-agent product shape, but its agent runs in Meta’s cloud and is largely closed.The SDK for gadgets was the stated exception to what was open.Source: §1 · p. 2
  • nanoMuse is an attempt at a complete open counterpart, inspectable by the person or someone they trust and running on their hardware where possible.The shared meeting point for devices should also be code anyone can read and run.Source: §1 · p. 2
  • The report moves from a definition and history through a reading of Muse, five questions and three horizons, nanoMuse, and a roadmap.The roadmap includes work the authors would not yet trust the system with.Source: Contributions · p. 2

2 Personal Agents So Far

The account traces the shift from assistants that wait and task agents that stop to products designed to act for one person. In September 2026, the remaining gap identified is access to phone applications that lack an API or web interface.

Assistants, then agents.

  • From assistants to screen-operating agents, the history moves through tool use, phone and desktop control, open screen models, and hosted cloud agents.Earlier open personal-agent projects remembered across sessions but lacked a screen; OpenMinis put a complete agent inside an Android app without a server.Source: Assistants, then agents · p. 2

September 2026.

  • Muse was described as the first product to assemble the pieces for one person, while September releases and announcements included other cloud-based personal agents and OpenMuse.The figure places these developments on a five-week September 2026 timeline, distinguishing open-source cards from closed ones.Source: September 2026 · pp. 2–3, Figure 2 · p. 3
  • Cloud computers can reach APIs and web pages but not many everyday apps, including banking and government services, which is the access gap nanoMuse’s hands address.The account links the possibility of acting to tool and screen-reading models, and acting for one person to memory and background execution.Source: What changed · p. 3

3 How Muse Is Built

The report reconstructs Muse from public documents and a copy of its production prompt, marking whether claims are documented, prompted, observed, or inferred. It emphasizes legible operation and safety boundaries, while distinguishing those transferable design choices from Muse’s cloud and in-house model.

3 How Muse Is Built

  • The analysis combines Meta’s public documents with a production-prompt copy and labels claims by their source status.The prompt is treated as evidence of what the agent is told, not as identical to the agent itself; Figure 3 distinguishes prompt-derived from publicly documented details.Source: §3 · p. 3, Figure 3 · p. 5

Why it worked.

  • Muse’s design was not presented as a benchmark win; its visible interface decisions made a background agent legible to the person.The account highlights a named agent, a persistent conversation, tool-call activity, and controls or approvals.Source: Why it worked · pp. 3–4

Who it is told it is.

  • Muse’s prompt gives the agent a persona for one person, limits its authority to that person’s requests, and directs it to use only what a task needs.The prompt also states hard stops and the person’s authority over their household, devices, and children’s care.Source: Who it is told it is · p. 4

Where it runs.

  • Muse runs its agent and tools in a container inside a dedicated Linux VM, with persistent files and a browser that retains cookies and tabs.The prompt copy describes scheduled jobs and sub-agents; the authors infer that the per-person VM contributes to the economics of a cloud service.Source: Where it runs · p. 4

Memory, self-improvement and proactivity.

  • Muse’s memory uses curated agent and person files plus an indexed memory bank and relationship map, with instructions to search memory before answering about prior context.Background jobs maintain memories and relationships, and a tool can explain provenance and replacements; the authors characterize distrust of recollection and provenance as a strong memory-confidence policy.Source: Memory, self-improvement and proactivity · pp. 4–5

Sentinel, injection and the browser.

  • Muse places permission authority outside the agent, scopes approvals, and combines probabilistic defenses with deterministic boundaries on what an agent can do.The browser sub-agent sees accessibility-tree snapshots rather than the DOM, limiting its ability to operate interfaces without such a tree.Source: Sentinel, injection and the browser · pp. 5–6

Devices, voice and the gadgets.

  • Muse’s paired-phone tool exposes data and commands, but does not let the agent operate the phone’s screen; its gadget SDK supports a limited set of commands.The SDK requires a Meta token under its own terms, and the account notes that the phone itself is not a screen-control surface for Muse.Source: Devices, voice and the gadgets · p. 6

What we take from it.

  • The authors identify Muse’s standard tool-calling agent architecture and externally placed safety mechanisms as transferable, but not its per-person VM or in-house model.They characterize the design budget as focused on legibility, much of it expressed in the prompt.Source: What we take from it · p. 6

4 What a Personal Agent Should Be

The section frames a personal agent around five questions and three horizons, with the person at the centre and voice in the inner ring. It considers what exists in 2026, what an open counterpart should add, and what an agent meant to last years should become.

4 What a Personal Agent Should Be

  • Memory should be readable, editable, exportable plain files with provenance, reconciliation against new information, and priority for lines written by the person.Provenance records the writing model, date, and confidence; reconciliation records whether an old line is confirmed, updated, or doubted.Source: What it knows: memory the person can… · p. 6

Where it acts: one agent, many hands.

  • One agent should represent a person across their accounts and devices, with screens, microphones, and motors serving as its hands.The proposed horizons move from cloud tools and a few gadgets, to the person’s existing screens, then to robots and appliances.Source: Where it acts: one agent, many hands · p. 6

Whom it answers to: approvals as the grammar of trust.

  • Trust should be expressed through scoped, readable, revocable grants, with the action shown before approval and payments and passwords never remembered.Grants may apply once, for a conversation, or always for a named recipient, site, folder, or other target.Source: Whom it answers to: approvals as the… · p. 6

How it learns: in the open, with consent.

  • Because speaking first can drive a person away, interruptions should have a visible, adjustable threshold and an away-time feed that reads like a colleague’s note.The harder goal is to learn when to interrupt well enough to ask first.Source: How it learns: in the open, with consent · p. 6
  • The open counterpart should train the hands model on traces people choose to share, evaluate it with a runnable suite, and eventually keep learning on the person’s devices.The proposed local-learning horizon improves the agent without sending the person’s life elsewhere; the cited example describes demonstrations teaching a phone agent an unfamiliar app [12].Source: How it learns: in the open, with consent · p. 7

How it is paid for and governed: small enough to run at home.

  • The proposed long-term model is a commons of skills, evaluations, and an open model governed by the people whose data it is.The person should be able to pay a provider directly or run the device meeting point for the price of the smallest server.Source: How it is paid for and governed: small… · p. 7

5 nanoMuse Today

nanoMuse is presented as a minimal, complete, GPL-3.0 personal agent spanning mobile, desktop, and web apps plus a relay. The section compares its on-device, open design with Muse and other projects, and describes its implementation.

5 nanoMuse Today

  • nanoMuse is a GPL-3.0 personal agent with Android and iOS apps, desktop apps for macOS, Windows, and Linux, a web app, and a relay with a console.The first release was September 25, 2026, and this report describes version 0.1.40 from October 2026; “minimal” means every piece works end to end.Source: §5 · p. 7
  • Each device is a whole agent with its own Sentinel; the optional, open relay is the only shared piece, and a device with its own key can contact the provider directly.Figure 5 depicts the construction as of version 0.1.40.Source: Figure 5 · p. 8

Where it differs.

  • Compared with Muse, nanoMuse adds on-device phone-screen control, open components including the relay, and model choice, but lacks Muse’s model, VM, wallet, artifacts, and large-team polish.Compared with OpenMinis, it adds the components that make an agent personal; compared with server-based OpenMuse, it runs on-device and can work without a server.Source: Where it differs · p. 8
  • The Android app modifies OpenMinis with a sandboxed Alpine Linux environment and agent tools, while the desktop app wraps the DeepSeek harness and carries its Python runtime.The same runtime serves the web app from the person’s own computer; nanoMuse additions sit beside upstream code to allow future merging.Source: Devices · p. 8

Devices.

The Devices section places every tool call behind a Sentinel policy gate and describes how the agent reaches apps, preferring the lowest rung that works before resorting to screen control.

Devices.

  • Every tool call passes the Sentinel’s fixed decision order, from denied tools and personal rules through lists, taint, risk mode, and warnings.After private data has been read, taint turns calls sending data outside the allow-list into approval requests and never reverses that condition.Source: Sentinel · p. 8
  • Approvals are scoped grants, warnings can force once-only approval, and every decision is recorded in a user-openable log.The Sentinel is a policy boundary in the same process family, not Muse’s privilege boundary.Source: Sentinel · p. 8

Sentinel.

  • To reach an app, the agent uses the lowest working option, progressing from skills or command-line and MCP tools through authenticated pages and browsers to the device screen.This combines shortcuts and screen control, the hybrid measured by MAS-Bench for phone agents [46].Source: Hands · p. 8

Hands.

Hands provides screen-based control, while memory is stored in files and devices operate as peers that can delegate work under each device’s own permissions.

Hands.

  • Hands is an off-by-default, one-action-per-screenshot loop that uses a screenshot, goal, step history, and available accessibility elements to produce an intended action and gesture.Its sentence describing the impending action lets deterministic policy check what the model plans to press; committing labels and submit-in-one-go typing require approval, and passwords and codes are forbidden.Source: Hands · p. 9

Memory as files.

  • Phone memory uses Markdown files for agent and user identity, cross-conversation memory, diary, and routines; memory lines have change logs and undo, with recall by rare words or optional embeddings.A tidy-up pass proposes merges and deletions but never drops what the person wrote; Feed, Ideas, and Goals are written in hidden conversations and read by the person.Source: Memory as files · p. 9

Every device.

  • Devices are peers with their own agents, and cross-device jobs require a grant on the device that will act and run under that device’s Sentinel.Any device’s agent can use the others as tools, allowing a task such as finding an order number on a phone and entering it in a Mac spreadsheet within one conversation.Source: Every device · p. 9

The model is the person’s choice.

nanoMuse can use a person’s own model key, a ChatGPT plan, or a model server on their machine. For cross-device use, the relay stores selected account and sync data, with controls for sync, deletion, and model-training data.

The model is the person’s choice.

  • One device needs no server: the person supplies an API key, and only model calls leave the device.A model server on the person’s own machine also works.Source: The model is the person’s choice · p. 9
  • The apps and relay share a catalogue of eighteen providers; a ChatGPT plan covers chat and the hands, while a key remains the dependable route.The ChatGPT option uses OpenAI’s Codex authorisation flow and rests on OpenAI’s terms; it does not cover images or video.Source: The model is the person’s choice · p. 9

What the relay keeps.

  • The relay stores account and device data, model-call metadata, profile and presence, plus conversation text chosen for sync—not files, images, or tool returns.Synced conversations remain associated with their account, including when two people share a device.Source: What the relay keeps · p. 9
  • Sync can be turned off to delete stored data; deleting an account removes it, and the community relay’s model-training text setting is disclosed and can be switched off.The community relay enables that setting for new accounts; a self-hosted relay chooses its own default.Source: What the relay keeps · p. 9

Running it yourself.

nanoMuse offers single-device, community-relay, and self-hosted-relay ways to run. The reported installation sizes and costs are estimates, and the cost figures do not measure actual use.

Running it yourself.

  • Three deployment modes are described: one device with a personal key or ChatGPT plan, several devices on the community relay, or a relay of one’s own.The single-device option needs no account or server; the self-hosted relay can be deployed with Docker on a small server tier.Source: Running it yourself · pp. 9–10
  • Table 2 reports release-file and installation sizes, memory on one Linux workstation, and estimated costs; it does not measure anyone’s use.Source: Table 2 · p. 10
  • A self-hosted relay can be brought up with TLS using one script, with everyone who signs in billed to the provider key.Source: Running it yourself · p. 10

6 A Roadmap With the Community

The roadmap lays out community work across three distances and five lanes. It includes near-term parity, evaluation and an open model for the hands, longer-term physical devices and persistence, and ways to contribute.

6 A Roadmap With the Community

  • The roadmap spans three distances and five lanes, with a first task on each Now card.Source: §6 · p. 10

Now: catching up with Muse.

  • Near-term work targets timely memory, between-schedule proactivity, a Muse-like conversation, more reliable hands, and one-command self-hosting.Source: Now: catching up with Muse · p. 10
  • The project proposes an evaluation suite built from failed runs and a small open model for the hands trained on contributed traces.The suite would run alongside AndroidWorld for phones, OSWorld for desktops, MemGUI-Bench for cross-app hand memory, and OS-Harm for refusals.Source: Next · p. 10

Later: looking up.

  • The hub’s look, act, and ask frames could bring cameras, appliances, and robot arms under the same approvals and log; a longer horizon is memory and identity that persist across models, devices, and vendors.Source: Later: looking up · p. 10

How to join.

  • People can contribute skills, translations, devices, failing hand traces, or corrections to the report through the repository and project chat server.Source: How to join · p. 10

7 Limitations

The limitations concern trust boundaries, unmeasured screen interaction, uncertain memory provenance, and reliance on a shared relay and an unverified, changeable prompt copy.

A policy boundary is not a privilege boundary.

  • A compromised device compromises the agent: the Sentinel shares the agent’s trust domain, and its rules mitigate but do not solve that exposure.Unlike Muse, nanoMuse has no host outside the agent’s reach holding the real credentials.Source: A policy boundary is not a privilege… · p. 10

The hands are judged by their words, and not yet measured.

  • Screen interaction has no reported success rate or human-takeover count; unlabeled icon buttons and dense professional screens remain difficult, and Linux support is limited to X11.The person watching the ring is the last line of defense.Source: The hands are judged by their words,… · p. 10
  • Roadmap milestones have no dates; a feature ships when it works on a real phone and a real computer.Figure 6 distinguishes current work closing the gap to Muse from later work not yet present in a shipped product.Source: Figure 6 · p. 11

Memory does not yet know who wrote it.

  • Memory can turn a weak model’s guess into fact without recording its source model, date, or confidence, and old entries have no re-check rule.People can undo a changed line, but the model cannot; until re-checking exists, the memory page is likened to notes from a colleague not yet fully trusted.Source: Memory does not yet know who wrote it · p. 11

The relay is shared, and it keeps text.

  • Conversation sync stores text on a server the person does not run, while the project’s community relay remains a single process on one project-operated machine.The shared component is small and runnable at home, but not unnecessary.Source: The relay is shared, and it keeps text · p. 11
  • The Muse account relies partly on a production-prompt copy whose provenance cannot be verified and whose contents may change between deployments.The report says prompt- or inference-marked statements should be weighed accordingly.Source: Where the record is thin · p. 11
Loading 2610.08699v1…