Source-linked AI summary
Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World
Yusong Lin, Xinyuan Liang, Haiyang Wang, Qipeng Gu, Siqi Cheng, Jiangui Chen, Shuzhe Wu, Feiyang Pan, Lue Fan, Sanyuan Zhao, Dandan Tu
TL;DR
Existing personal-assistant systems and benchmarks expose only limited portions of users’ digital worlds, leaving broad, always-on reasoning insufficiently evaluated. Claw-Anything simulates long-horizon, noisy environments spanning services and devices, and tests reactive and proactive assistance. GPT-5.5 reaches only 34.5% pass@1, while generated training data improves Qwen3.5-27B by 23.7%.
Problem
Existing systems and benchmarks provide narrow slices of user state, omitting broad temporal, service, and device context needed to evaluate always-on personal assistance.
Method
Claw-Anything simulates persona-grounded, noisy digital worlds with long histories, interdependent services, heterogeneous GUI and CLI devices, and reactive or proactive tasks.
Results
34.5% pass@1 is achieved by GPT-5.5, while fine-tuning Qwen3.5-27B on generated trajectories improves pass@1 by 23.7%.
Takeaways & Limitations
Claw-Anything exposes a gap between current frontier-model performance and real-world assistance requirements while providing scalable supervision for post-training.
Takeaways & Limitations
Many backend services are controllable mock environments, and the benchmark covers only a limited subset of real-world personal devices.
Abstract
from arXiv · showhide
Large language model agents are increasingly envisioned as always-on personal assistants with access to anything relevant in the user's digital world. Yet current systems operate over only narrow slices of that world, limiting context-sensitive reasoning and effective assistance. Existing benchmarks similarly provide only partial user state and therefore fail to capture performance in such a broad, always-on setting. To address this gap, we introduce Claw-Anything, a benchmark that expands agent context along three dimensions: long-horizon activity histories, interdependent backend services, and integrated GUI and CLI interaction across multiple devices. To instantiate this setting, we simulate months of user activity through multi-round event injection, producing complex world states and realistic noise, including irrelevant events and conflicting signals. Agents must reason over rich contextual environments while remaining robust to such noise. This expanded scope also enables the evaluation of proactive assistance, requiring agents to anticipate user needs and deliver timely recommendations. Experiments show that GPT-5.5 achieves only 34.5% pass@1, substantially below prior benchmarks, underscoring a gap between current agent capabilities and the demands of always-on personal assistance. Alongside the benchmark, we release an automated data-generation pipeline that yields 2,000 training environments and improves the base model by 23.7%, demonstrating its utility of scalable data infrastructure.
1 Introduction
Claw-Anything addresses the mismatch between always-on personal-assistant goals and benchmarks that expose only narrow, static user-state slices. It broadens context across time, services, and devices while also providing scalable benchmark and training data.
- Existing benchmarks omit long-horizon activity, cross-service dependencies, and multi-device interaction, limiting evidence about agents in realistic digital environments.
- Claw-Anything integrates months of event streams, interdependent backend services, and heterogeneous GUI and CLI interfaces.This setting requires agents to coordinate information and actions across time, services, and devices.
- Proactive assistance is evaluated alongside reactive task completion, requiring agents to anticipate user needs and provide timely recommendations.
- 34.5% pass@1 is achieved by GPT-5.5 on Claw-Anything, while fine-tuning Qwen3.5-27B improves success by 23.7%.The results indicate both a substantial capability gap and the utility of generated training data.
- 200 human-verified evaluation tasks and 2,000 training environments are constructed through an automated pipeline.
2 Related Work
Prior personal-assistant benchmarks broaden task coverage and interaction realism, but generally evaluate localized tasks in isolated, short-horizon settings. Claw-Anything targets always-on assistance over noisy, accumulated context spanning services and devices.
- Existing benchmarks advance planning, tool use, and grounded interaction across standardized, open, productivity, professional, CLI, and rubric-based tasks.
- Most prior evaluations remain isolated, short-horizon, and relatively clean rather than modeling an assistant embedded in the user’s broader digital world.
- Claw-Anything evaluates reasoning over noisy event streams, interconnected backend systems, heterogeneous devices, and accumulated personal context.
- Scalable environments have been studied mainly for code-centric and terminal-centric agents, leaving personal-assistant settings comparatively underexplored.
3 Methodolgy
Claw-Anything places reactive and proactive tasks in noisy, context-rich environments spanning long histories, backend services, and heterogeneous device interfaces. An automated multi-stage pipeline synthesizes environments, tasks, verifiers, and filtered instances for evaluation and training.
- Task Formulation: Each task combines a persona, long-horizon history, diverse backend services, and coordinated GUI and CLI interactions across multiple devices.
- Construction Pipeline: The pipeline iteratively samples task or noise templates, adapts events to the current world, updates state, and extracts task snapshots.
- Task Formulation: Environments include over three months of logs, more than forty backend services, persistent states, and irrelevant events requiring closed-loop reasoning.
- Task Formulation: Queries may be underspecified and require integrating task-relevant signals across event streams, services, and CLI-based Linux and GUI-based Android environments.
- Task Formulation: Evaluation combines rule-based checks with LLM judgments, emphasizing final outcomes because tasks can have multiple valid solution paths.
- Construction Pipeline: For each selected snapshot, the pipeline generates a user query, executable verifier, and reference solution, then applies automated filtering before optional human verification.
- Construction Pipeline: 2,000 task environments are generated for training, with 1,500 successful trajectories collected from personas disjoint from the evaluation set.
4 Experiment
Experiments evaluate Claw-Anything across frontier models, context-scaling factors, data-generation choices, skill loading, and failure modes. Results show that broader context increases task difficulty, pipeline-generated trajectories improve training, and execution remains a central bottleneck.
- Main results: 23.7% improvement on pass@1 results from fine-tuning Qwen3.5-27B with 1,500 successful trajectories generated by the automated pipeline.The study constructs 2,000 training tasks, trains for 10 epochs, and reports improved performance with more collected trajectories.
- Scaling Context: Removing event streams substantially lowers success, while longer histories degrade performance even when event streams remain available.The ablation attributes many tasks’ solvability to event-history information and identifies difficulty leveraging long-horizon context.
- Scaling Context: Success rates collapse to nearly zero when tools for multi-service tasks are removed, and performance declines as the number of involved services increases.These results indicate that many tasks require retrieval and action across backend services, while coordination across more services remains difficult.
- Scaling Context: Removing GUI access makes CLI–GUI collaborative tasks nearly unsolvable, while restored joint access makes them tractable but still less successful than pure CLI tasks.The ablation isolates cross-interface coordination as both necessary for some tasks and challenging even when available.
- Data pipeline: Higher noise ratios, more simulation rounds, and greater fixture-level conflict each reduce task success, making generated environments progressively harder.The pipeline ablations associate irrelevant events, richer personas, and cross-service inconsistency with benchmark difficulty.
- Additional analyses: Proactive tasks are substantially harder than reactive tasks, while lazy skill loading degrades both success rate and stability.Failure analysis identifies the investigation–execution gap as dominant; hallucination-related errors are comparatively rare.
5 Conclusion
Claw-Anything evaluates personal-assistant agents with broader operational scope by combining long-horizon events, diverse backend services, multi-device interaction, and proactive tasks. Results show a pronounced gap between current frontier models and real-world assistance requirements, while the accompanying data-generation pipeline supports scalable environment construction.
- Claw-Anything combines long-horizon event streams, diverse backend services, multi-device interaction, and proactive tasks to capture challenges absent from existing evaluations.
- Proactive settings remain especially difficult for current personal-assistant agents.
- The automated data-generation pipeline provides a practical foundation for scalable environment construction and future personal-assistant research.
A Details of Task Generation Pipeline
The task-generation pipeline begins with a coarse persona and progressively enriches it through simulated events. It uses conflict seed tasks and distracting noise events to create personalized, diversified, and realistic environments.
- The pipeline starts with a coarse persona describing basic user information, then enriches it alongside sampled tasks or events.The initial persona can be generated by prompting a modern LLM.
- Persona enrichment adds concrete user preferences and activities by instantiating persona-specific events with antecedents and consequences.
- Seed tasks model realistic conflicts such as time-slot conflicts, information contradictions, and financial limitations.These conflicts require agents to identify and appropriately handle inconsistent information.
- Noise events introduce irrelevant routine activities, including inbox scanning, discarded drafts, deleted notes, and browsing feeds or dashboards.These activities may appear in logs or as concrete dated sessions across the persona’s working days.
A.2 Task Query and Verifier Generation
The second pipeline stage adapts a sampled conflict seed task to the generated persona and produces the target task and verifier. An end-to-end agent execution then validates that the task is solvable and well-formed.
- Stage II samples a conflict seed task and adapts it to the Stage I persona to create a persona-specific target event.
- An LLM generates the event’s antecedents and articulates the problem that the agent must solve.
- The generated task and reference solution are executed end-to-end by an agent to verify solvability and conformity with the grader’s recommended solution path.
B Claw-Anything Evaluation
Claw-Anything evaluation modifies prompts and tool interfaces, simulates app backends, and uses task-specific dates and tailored scoring. It supports skill and tool modes while scoring both pass rates and solution trajectories.
- Evaluation modifies the agent’s system prompt and tool interface, provides a simulated app-backend service, and adopts a tailored scoring scheme.
- Table 5 reports pass@1 performance separately for 150 CLI-only tasks and 50 CLI+GUI tasks.
- Task-specific current dates are injected into system prompts so repeated evaluations remain comparable across calendar dates.
- Agents can access app databases through two evaluation modes: skill mode uses a meta-tool for specifications, while tool mode exposes tools directly.
- The backend service parses agent tool calls and returns requested data or performs corresponding write and delete operations.
- The grader assigns failed tasks continuous trajectory scores because tasks contain noise, alternative cues, and multi-threaded clues rather than one canonical solution path.
B.4 Detailed Evaluation Setting and Result
Claw-Anything evaluates 200 tasks across CLI and CLI+GUI settings, separating skill-mode and tool-mode CLI tasks.
- 200 tasks comprise 150 CLI tasks and 50 CLI+GUI tasks.
- The 150 CLI tasks include 100 skill mode tasks and 50 tool mode tasks.
- Detailed model performance is reported separately for the 150 CLI tasks and 50 CLI+GUI tasks.
C Training Details
Training uses cosine-decayed learning rates with linear warmup, 10 epochs, batch size 16, the qwen3-coder agent template, and 100k-token sequences.
- 2 × 10^-5 is the initial learning rate, reduced with cosine decay after warmup.
- 5% of training steps use linear warmup from 1 × 10^-6 to 2 × 10^-5.
- 10 epochs are trained with batch size 16.
- The qwen3-coder agent template is used with a maximum sequence length of 100k tokens.
D Limitations
The benchmark remains limited by mock backend services and incomplete device coverage, while its broader-access setting raises privacy and autonomy risks.
- Many backend services remain controllable mock environments rather than fully real-world systems.
- Device coverage includes cross-device interaction but omits much of the ecosystem spanning phones, laptops, tablets, wearables, and smart-home devices.
- Broader access may amplify privacy concerns and increase risks of erroneous actions, overreach, or misuse in high-stakes settings.
- The benchmark is intended for evaluation rather than direct deployment, motivating safeguards such as permission boundaries, transparency, auditability, and user control.