PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

Xiaohongshu Inc

arXiv:2608.10875v1cs.CLcs.AI

TL;DR

Existing benchmarks largely miss the proactive, persistent, long-horizon demands of everyday-life assistance in changing environments. VibeLifeBench evaluates these capabilities across 200 multi-week tasks, finding that every model scores low and remains far from trustworthy long-term life assistance.

  • Problem

    Existing benchmarks overlook proactive, dynamic-world, and long-horizon requirements that matter for trustworthy everyday-life assistance.

  • Method

    VibeLifeBench uses 200 multi-week tasks across ten everyday-life domains, with silent world mutations and weighted checks of timely actions, constraints, and final artifacts.

  • Results

    All evaluated models score low; the strongest reaches an avg@3 of only 32.5, while scores fluctuate across repeated runs and none is reliably competent across all life domains.

  • Takeaways & Limitations

    VibeLifeBench provides a single measurement of proactivity, living-world adaptation, and long-horizon coherence, highlighting a substantial gap in current agents.

  • Takeaways & Limitations

    Tasks embed unstated constraints, safety red lines, and authorization boundaries that determine which actions agents may take, must ask about, or may never perform.

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly deployed as personal assistants. Existing evaluations, however, mostly use short, self-contained requests in static environments. Everyday life assistance is different. A task runs for weeks rather than minutes. The world keeps changing while the agent is not being prompted. Many constraints are never stated outright. An agent that merely answers the request in front of it will fail at such a task. What is needed instead is an agent that stays proactive and consistent. It decides on its own when to act, when to ask, and when to stay silent. It notices changes that nobody announced. It keeps one plan coherent from the first day to the last. No current benchmark measures this. We introduce VibeLifeBench, a benchmark of 200 long-horizon tasks across ten everyday-life domains. Each task is a scripted multi-week timeline in a simulated world of 22 mock services. The world advances on its own clock, and many of its changes are silent, so only an agent that re-inspects the world discovers them. Every task is graded by fine-grained, weighted checks that read only what the agent actually left behind, covering the end state, the timeliness of its actions, and whether it upheld the implicit constraints. We evaluate seven frontier models. All of them score low, which shows how far current agents are from assisting with real life. We will open-source all tasks, environments, and the evaluation framework.

1 Introduction

Everyday-life assistance requires agents to be proactive, operate in a changing world, and maintain coherent plans over long horizons—capabilities existing benchmarks largely overlook. VibeLifeBench operationalizes these requirements through 200 multi-week tasks, simulated services and events, and fine-grained stage-aware evaluation.

  • Existing agent benchmarks focus mainly on coding, office work, and passive execution of clearly specified tasks, overlooking everyday-life assistance and trustworthy support for ordinary users.
  • Life assistance requires proactivity, a living world, and long-horizon coherence: agents must detect unannounced changes, infer implicit constraints, and sustain plans across multi-week lifecycles.The environment may change without prompting, requiring agents to decide whether to act, notify the user, or remain silent.
  • 7,453 scripted events drive VibeLifeBench’s multi-week living-world tasks, while agents must proactively re-query the world as it changes and judge what response is appropriate.
  • 12,261 fine-grained checks, with a median of 58 per task, evaluate both final world states and conduct during the task, including timeliness, policy notifications, and agreed communication cadence.

2 VibeLifeBench

VibeLifeBench evaluates agents as persistent actors in evolving, partially silent worlds rather than as responders to static prompts. Its tasks combine proactive timeline-based interaction, implicit safety constraints, reproducible mock services, and artifact-based weighted scoring.

  • World and timeline: The benchmark models each task as a world advancing on a virtual clock, including silent changes that require agents to re-inspect the world and update their plans.Mutations alter service state without producing a turn or notification, directly testing persistence and proactivity.
  • Constraints and safety: Tasks embed unstated constraints, safety red lines, and authorization boundaries governing which actions agents may take, must ask about, or must refuse.Scenarios include tempting unsafe shortcuts that compliant agents must recognize and refuse or escalate.
  • Environment: 200 tasks share 22 stable mock service backends while configuring task-specific initial data and event pacing, enabling offline, deterministic, and reproducible evaluation.The services together expose 288 tool interfaces, while task-specific scenario data preserves varied settings.
  • Evaluation trace: Each run evaluates the agent’s observable trace across the full timeline, including service end states, durable workspace artifacts, notes, calendar entries, emails, and replies.Agent actions include tool calls, file writes, and replies; world mutations can occur without any agent turn.
  • Scoring: Scores are weighted fractions of deterministic checks on a 0-to-100 scale, covering timely per-stage behavior and cross-stage constraints while penalizing critical safety failures more heavily.Checks read only observable artifacts and never the model’s hidden reasoning; safety and hardening checks carry the largest weights.

3 Evaluation

VibeLifeBench evaluates agents through stage-wise runs in a changing world, scoring only the observable artifacts they leave behind. Scores aggregate weighted task checks across three stochastic runs, using TERRARIUM and the openclaw harness.

  • Run execution: Each run advances stage by stage, recording agent turns triggered by messages, observations, and notifications while world mutations occur without triggering turns.After each stage, the criteria are evaluated against the current world state.
  • Observable evaluation: Scoring reads only observable artifacts and penalizes agents that fail to proactively re-inspect silently mutated world state.A mutation can change state the agent was never told about, so evaluation reflects the reality it did not observe.
  • Score aggregation: A task score is the weighted fraction of passed checks on a 0–100 scale, aggregated over three runs into avg@3, max@3, and min@3.Within each task, the three scores are summarized by their mean, maximum, and minimum, then averaged equally across tasks.
  • Evaluation infrastructure: VibeLifeBench runs on TERRARIUM, which provisions isolated sandboxes and orchestrates task execution, with all models evaluated under the openclaw harness.The infrastructure supplies mock services, workspaces, and system prompts to tool-using agents.

4 Experiments

Across 200 long-horizon tasks, all seven contemporary models score low, with limited reliability and substantial variation across life domains. Greater model scale, tool-calling strength, or interaction investment does not establish competence at proactive, persistent assistance.

  • Evaluation setup: 200 tasks were evaluated with seven contemporary models, each run three times under the same native tool-calling scaffold.The evaluation reports avg@3, max@3, min@3, and within-task standard deviation.
  • Overall performance: 32.5 was Claude Opus 5’s avg@3, while its max@3 reached only 41.2 and DeepSeek-V4-Pro reached 21.1.Every evaluated model scored low on the full evaluation set.
  • Overall performance: All seven frontier models fell within a narrow 21-to-33 score band, remaining far below competence despite strong professional tool-calling abilities.The reported ordering was Claude Opus 5 > GPT-5.5 > Gemini 3.5 Flash ≈ Claude Opus 4.8 > GLM-5.2 > Kimi-K2.6 > DeepSeek-V4-Pro.
  • Reliability: Every model’s min@3 was at most 23.8, while within-task standard deviation reached 10.0 across repeated runs.The results indicate that even successful runs are difficult to reproduce, exposing weak persistence and self-consistency.
  • Domain variation: Claude Opus 5 ranged from 21.8 on team building to 51.1 on shopping across ten domains, with shopping, travel, and renovation easier than team building, rental, and exam preparation.No model was competent across all life domains.
  • Resource usage: Claude Opus 5 produced about 325k output tokens per run with 316 tool calls and 210 turns, but higher investment did not guarantee a higher score.DeepSeek-V4-Pro reached 21.1 on the smallest context budget, while Gemini 3.5 Flash read 41.2M context per run and took 227 turns yet landed mid-pack.

5 Analysis

The analysis attributes low model scores to failures in hardening tiers, proactivity, persistence, adaptation to world changes, long-horizon coherence, and breadth across life domains. It recommends durable state, proactive re-inspection and propagation, and targeted hardening and safety work.

  • Failure sources: Cross-stage and final checks have the lowest pass rates, while representing 19.1% of checks but 26.8% of total weight.These high-weight hardening layers directly depress absolute scores.
  • Failure sources: Proactivity ranges from 16.0 to 33.6, while persistence and bookkeeping ranges from 18.9 to 28.0 across models.Durable-artifact checks almost never pass, and persistence and bookkeeping accounts for 22.2% of pooled failed checks, with individual models ranging from 22.0% to 23.2%.
  • Failure sources: Propagation and recovery reaches only 18.5 to 32.0, reflecting routine failures to re-inspect the world and propagate unannounced mutations into plans.A mutation triggers no turn, so adaptation requires deliberate world re-inspection.
  • Failure sources: Every model’s per-stage pass rate is 10 to 15 points lower in the final timeline third than in the first third.The decline is nonmonotonic but holds across all seven models.
  • Failure sources: No model is competent across all ten domains, and the easy-to-hard domain ordering is consistent across models.The analysis identifies breadth itself as a challenge for real-life assistants.
  • Directions: The analysis recommends writing state into notes, calendars, and workspace files, proactively re-inspecting and reconciling the world, and pursuing targeted hardening and safety work.The goal is to maintain cross-stage auditable artifacts and fold world mutations into coherent plans.

6 Related Work

Existing agent benchmarks primarily evaluate explicit, technically scoped tasks in reproducible environments or extend evaluation to longer trajectories and evolving user interactions. VibeLifeBench instead targets everyday-life assistance in a world that changes independently of the agent.

  • Coding and working agents: Coding and office-work benchmarks evaluate repository changes, test-suite completion, cross-file edits, heterogeneous information reconciliation, and related professional tasks.These benchmarks cover coding, office, and knowledge-work settings.
  • Coding and working agents: Most coding and office-work tasks provide explicit goals and deliverables, treat the agent as a one-shot executor, and use sandboxes changed only by agent actions.VibeLifeBench contrasts this setup with everyday-life tasks embedded in worlds that evolve independently.
  • Long-horizon agent benchmarks: Long-horizon benchmarks either unroll one task into a long, dependency-rich trajectory or model multi-task interaction with drifting preferences and staged information.The two directions test planning, memory, adherence to earlier decisions, personalization, and continual revision.

7 Conclusion

VibeLifeBench measures life-domain agents in evolving multi-week worlds, jointly evaluating end-state correctness, timely proactivity, and faithful propagation of world changes. Frontier models remain far from trustworthy long-term life assistance, while the benchmark’s tasks, environments, and framework will be open-sourced.

  • Benchmark contribution: VibeLifeBench organizes tasks as multi-week living worlds that evolve autonomously and uses stage-aware, weighted scoring to evaluate long-horizon agents.The benchmark jointly examines end-state correctness, timely proactive behavior, and faithful propagation of world changes.
  • Benchmark contribution: The benchmark brings proactivity, living-world adaptation, and long-horizon coherence—three properties overlooked by existing evaluations—into one measurement.
  • Evaluation findings: Frontier models fail to persistently maintain cross-stage auditable state, often miss living-world mutations, lose coherence as tasks advance, and lack reliable competence across all life domains.The evaluation characterizes them as far from trustworthy long-term life assistants.
  • Resources: The tasks, environments, and evaluation framework will be open-sourced to advance research on proactive, persistent life agents.

Contribution

The contribution section lists the paper’s contributors and their affiliations. It identifies Xiaohongshu Dots Studio and Evolvent AI, along with core-contributor and project-lead roles.

  • Roles: Qionglin Qiu, S.L., Lei Huang, and Z.Y. are marked as core contributors.The dagger symbol denotes core-contributor status.
  • Roles: Xiang Cheng is identified as the project lead.The double-dagger symbol denotes the project-lead role.
  • Affiliations: The author list includes contributors from the General Post-training Team at Xiaohongshu Dots Studio and Evolvent AI.The affiliations are numbered 1 and 2, respectively.

A A Complete Task Walkthrough

The appendix walks through a flagship 20-day family trip to Japan, showing how the agent operates as a long-term personal assistant across a changing multi-week timeline.

  • A Complete Task Walkthrough: The task spans 24 stages across three phases: two weeks of preparation, a mid-trip typhoon disruption, and the in-country itinerary.The simulation runs from 2026-04-17 to 2026-05-16, while Li Wei sends only seven messages and most turns are driven by world observations.

The workspace given at the outset

The initial workspace encodes the user’s latent constraints, authorization boundaries, behavioral principles, bookkeeping requirements, and protected identifiers. These files define what the agent may do, must track, and must proactively safeguard throughout the task.

  • The workspace given at the outset: The workspace represents the long-term assistant’s starting context, including the served person, latent constraints, authorization scope, behavioral principles, bookkeeping contract, and sensitive identifiers.These categories are supplied through PERSONA.md, USER.md, SOUL.md, AGENTS.md, and IDENTITY.md.
  • The workspace given at the outset: The persona records Li Wei’s profile and his parents’ travel-relevant health, mobility, passport, and medication constraints.Li Jianguo is insulin-dependent, must carry specified supplies and documentation, cannot skip meals, and has walking tolerance of about 4 km per day.
  • The workspace given at the outset: Authorization permits routine lookups, notes, calendars, and refundable in-budget bookings, but requires approval for consequential changes or charges and forbids medical decisions or exceeding 60,000 CNY.The policy also prohibits treating visa or customs policy as binding rather than quoting it.
  • The workspace given at the outset: The behavioral principles require reading each stage fully, polling persisted state, surfacing implicit constraints early, acting on routine matters, asking on consequential ones, and proposing rather than executing irreversible actions.They also favor light journal entries during quiet turns and concrete replans only after high-confidence signals.
  • The workspace given at the outset: The bookkeeping contract requires a pre-seeded journal, per-booking expense lines normalized at 1 CNY to 20 JPY, recomputed running totals, and an early safety-critical packing checklist.Quiet gaps may also trigger lightweight check-ins even when nothing is wrong.
  • The workspace given at the outset: Sensitive passport numbers, birth dates, and addresses may be shared only through a secure attachment or document ID, never in plaintext email or third-party messages.These identifiers support eVisa and insurance applications.

Timeline

The timeline illustrates representative turns from a 20-day family trip to Japan across three phases. Events are annotated by kind and by the response expected from a competent assistant.

  • Timeline: Each event is annotated with its event kind and the response a competent assistant should give.The event-kind labels use the four-way split defined in Section 2.2.3.
  • Timeline: Table 8 presents representative events across the timeline’s three phases.The table is an excerpt from the 20-day family trip scenario.
  • Timeline: The example timeline follows a 20-day family trip to Japan.Table 8 presents only an excerpt of the full trip timeline.

Implicit constraints and safety red lines · How it is scored · Where contemporary models fail

The benchmark tests whether agents identify and uphold unstated constraints and safety boundaries across a task, using checks for timely actions, persistent cross-stage compliance, and final artifacts. Seven models repeatedly failed to re-inspect changing circumstances, preserve coherent plans and budgets, surface hidden constraints, and reject a phishing expedite-fee request.

  • Implicit constraints and safety red lines: The task requires agents to identify and uphold constraints that are never explicitly stated.These include passport validity, insulin-carry requirements, and meal-interval restrictions for the diabetic father.
  • Implicit constraints and safety red lines: The mother’s passport has less than six months of validity and must be surfaced before flights are chosen.This constraint is specified as one of the task’s hidden requirements.
  • Implicit constraints and safety red lines: Insulin must be carried onboard with a bilingual doctor’s letter and declared under entry rules.The passage presents these requirements as constraints the assistant must uphold.
  • How it is scored: Scoring uses per-stage checks for timely behavior, cross-stage checks for persistent constraints, and final checks of the ending world and artifacts.Examples include relaying a visa-policy change when announced, committing bookings before the final window, staying under the 60,000 CNY cap, and never making a medication decision.
  • Where contemporary models fail: The seven models failed to re-select seats after an aircraft swap and did not persist a Plan-B after the typhoon reached high confidence.They also failed to surface the passport-validity problem before flight selection and maintain a running budget ledger.
  • Where contemporary models fail: No evaluated run refused the phishing expedite-fee email.This failure is listed alongside missed updates, planning persistence, hidden-constraint detection, and budget reconciliation.
Loading 2608.10875v1…