PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences

Jialin Li, Zhenhao Chen, Hanjun Luo, Hanan Salam

arXiv:2610.30485v1cs.AIcs.CL

TL;DR

Personal agents need to remember how users want to be assisted, but existing memory benchmarks provide limited evidence about learning and acting on interaction preferences. MemPABench evaluates preference memory through persistent, on-policy interactions across context and time. Current memory brings limited gains in preference selection, and File memory can apply preferences beyond their relevant contexts.

  • Problem

    Existing personalization and memory benchmarks largely test user facts and content preferences, leaving limited evaluation of how agents learn and act on interaction preferences that shape everyday interaction quality.

    Source: §1 · p. 1

  • Method

    MemPABench evaluates memory through on-policy histories shaped by agents’ actions and user responses, using persistent persona-grounded users, controlled preference changes, and measures of selection and delivered behavior.

    Source: §1 · p. 2

  • Results

    Current memory brings little gain in preference selection: retrieval-based systems do not surpass the memoryless level, while File memory improves selection but fails to constrain preferences to relevant contexts.

    Source: §6 · p. 11

  • Takeaways & Limitations

    Memory for interaction preferences must support applying preferences across contexts and as they evolve, rather than relying only on stored user information.

    Source: §6 · p. 11

Abstract

from arXiv · show

Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent's default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.

1 INTRODUCTION

Interaction preferences can vary by context and change over time, while existing memory benchmarks rarely test how agents learn and act on them. MemPABench evaluates whether personal-agent memory can learn, distinguish, and update these preferences through persistent interactions.

1 INTRODUCTION

  • Preferences are difficult to learn because users often reveal them through reactions scattered across sessions, and failed retrieval can leave the agent’s defaults in control.Those defaults may favor long responses and guessing, producing repeated mismatches for users whose preferences differ.Source: §1 · p. 2
  • Prior benchmarks cover parts of the problem, but none combines persistent user histories, interaction preferences, and preference changes over time.Some benchmarks target content preferences, while others target interaction preferences without histories shaped by the evaluated model.Source: §1 · p. 2
  • MemPABench evaluates personal-agent memory through histories shaped by each agent’s actions and user responses, testing preference selection, behavioral realization, and user perception.The benchmark follows four persona-grounded users through persistent interactions with controlled preference changes.Source: §1 · p. 2
  • The paper models interaction preferences as states indexed by user, context, attribute, and time, then evaluates acquisition, setting selection, and delivered realization.This formulation reflects the need to preserve where a preference applies and whether it remains current.Source: §1 · p. 2
  • Retrieval-based systems rarely surface applicable interaction preferences at decision time, while File memory improves selection but overapplies preferences across contexts.The paper concludes that current memory brings little gain in preference selection overall.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates memory for user facts and content preferences, while separate benchmarks address interaction preferences and executable agent tasks. The related work identifies limited coverage of how memory supports behavioral adaptation across contexts and over time.

2 RELATED WORK

  • Personalization benchmarks measure factual recall, personalized question answering, or recommendation, with newer work also testing changes in user states and content preferences.Table 1 compares representative benchmarks along six dimensions.Source: §2 · pp. 2–3, Table 1 · p. 3
  • Benchmarks for interaction preferences and contextual communication exist, but provide limited insight into behavioral adaptation as preferences shift across contexts and time.The paper frames contextual breadth and temporal depth as memory requirements for interaction preferences.Source: §2 · p. 3
  • Executable environments test multi-step agent decisions, but consistent user reactions and on-policy histories are needed to evaluate learning from consequences of the agent’s own behavior.MemPABench grounds simulated users in detailed personas and personal worlds.Source: §2 · p. 3

3 MEMPABENCH: LONG-HORIZON MEMORY BENCHMARK FOR PA

MemPABench models preference learning as an on-policy process across context and time, then evaluates both a personal agent’s preference choices and the interactions it delivers. It combines persona-grounded users, persistent worlds and sessions, an interaction wrapper, and probes for selection and realization.

3 MEMPABENCH: LONG-HORIZON MEMORY BENCHMARK FOR PA

  • MemPABench evaluates interaction-preference memory through a persistent, multi-session environment that links persona-grounded users, personal agents, and preference evaluation.The benchmark formalizes preferences and the interactions among users, worlds, agents, and memory, then scores preference selection and realization.Source: §3 · p. 3, Figure 1 · p. 4

3.1 PROBLEM FORMULATION: DYNAMIC INTERACTION PREFERENCES

  • Preferences are discrete settings indexed by user, context, attribute, and time; agent memory informs each choice, while five controlled events change one preference cell each.The user’s actions and reactions update the persistent world, and the benchmark separates acquisition, selection, and realization as stages where an agent can fail.Source: §3.1 · pp. 3–4

3.2 USERS AND INTERACTION PREFERENCES

  • The benchmark adapts 14 actionable interaction attributes for personal agents and builds four anonymized persona-grounded users with reviewed work and personal preference settings.Of 56 user–attribute pairs, 18 differ between work and personal life; predicting the 28 settings reaches 46.4% accuracy from an anonymized profile, near the 45.2% chance level.Source: §3.2 · pp. 4–5

3.3 WORLDS, SESSIONS & TIMELINE

  • Each user’s four-month timeline contains 99 sessions: 94 accumulation sessions and five controlled preference-change events, with 20 of 112 cells changing once.Sessions target a preference cell and make the relevant choice necessary to complete the task; the same prewritten change transcript is replayed for each memory condition.Source: §3.3 · pp. 5–6

3.4 INTERACTION HARNESS

  • At decision points, the PA explicitly selects an interaction setting, and the IX Wrapper applies it to the public interaction or proposed actions without seeing the target setting.The memory adapter provides the PA with memory evidence but withholds target settings; the wrapper cannot correct a wrong choice.Source: §3.4 · p. 6, Figure 3 · p. 6

3.5 METRICS FOR SELECTION AND REALIZATION

  • Each user completes 33 isolated probes—28 final probes covering every preference cell after the last session and five pre-event probes testing changed cells—measuring correct selection, context sensitivity, and preference-change tracking.Behavioral Realization Score uses an LLM judge to rate from 1 to 5; after each final probe, the user simulator rates Perceived Realization from 1 to 5.Source: §3.5 · pp. 6–7

4 EXPERIMENTAL SETUP

The experimental setup crosses four personal-agent backbones with four memory systems and two reference conditions. It also specifies the simulator, review, and scoring components used in the runs.

4 EXPERIMENTAL SETUP

  • Every combination of four PA backbones and four memory systems is evaluated on all four users, alongside two reference conditions for each backbone.The backbones are GPT-5.6-Sol, Qwen3.8-27B, HY4-Preview, and DeepSeek-V4-Flash.Source: §4 · p. 7
  • The four memory conditions are File memory, simple RAG, Mem0, and MemOS, with retrieval-based systems limited to 4,000 prompt tokens from their top 10 results.File memory reads a consolidated MEMORY.md file; the retrieval systems store and search memories or raw messages and use the same text embedding model.Source: §4 · p. 8
  • Gemini-3.8-Flash simulates users and scores BRS and PRS, while rule-based checks and a Gemma-4-31B judge review live simulator messages before the PA sees them.Claude Sonnet 5 re-judges a stratified sample of 418 probes, and Claude Opus 5.5 audits simulator sentences for post-hoc leakage.Source: §4 · p. 8
  • No Memory and Oracle are run on probes without accumulation, using checkpoints from the same File memory runs; Oracle supplies the target setting and its definition in the prompt.The reference runs clear the PA’s memory and provide no earlier conversation.Source: §4 · p. 8

5 RESULTS AND ANALYSIS

Across memory conditions, only File memory clearly improves preference selection, while retrieval-based systems remain near No Memory. File memory also improves delivered behavior but spills preferences across contexts, and correct selection alone does not ensure the intended behavior is realized.

5 RESULTS AND ANALYSIS

  • 59.2% PSA for File memory exceeds No Memory’s 49.6% by 9.6 points; Mem0, MemOS, and Simple RAG score 50.0%, 50.0%, and 49.9%.The mean gain over No Memory has a 95% cluster-bootstrap interval of [+3.9, +15.6].Source: Table 2 · p. 8, §5 · p. 8
  • In two probes, retrieval-based stores held relevant preferences but did not retrieve them; only File memory surfaced the objection before an outreach decision.The PA selected High incorrectly with each retrieval-based system, while File memory showed the objection and the PA selected Low.Source: Figure 5 · p. 9, §5 · p. 9
  • File memory scores 3.59–3.62 on PRS and 3.46–3.57 on BRS, exceeding retrieval-based systems and No Memory on both delivery measures.The figures average across three probe repeats and four users; File memory is highest for each backbone on BRS.Source: §5 · p. 9, Figure 6 · p. 10, Figure 7 · p. 10
  • File memory raises PET to 25.4% from No Memory’s 11.2%, but CS falls to 14.4% from 21.8%, with preferences spilling across contexts.In 26.9% of its CS pairs, File memory carries a target setting from one context into the other.Source: §5 · pp. 9–10, Table 3 · p. 10
  • Information Flow has the lowest PSA at 34–44%, versus 57–69% for Expression Style and 55–58% for Disclosure, because it often concerns task actions.Information Flow and Initiative & Autonomy preferences commonly change what the PA does, while Expression Style and Disclosure change how its reply reads.Source: Figure 8 · p. 10, §5 · pp. 10–11
  • Under Oracle, correctly selected probes for five difficult settings pass the BRS threshold in only 60% of cases, versus 95% for the other 25 settings.Oracle raises correct selection of those five settings from 33% to 96%, but replies scoring BRS 4 or higher rise only from 24% to 59%.Source: §5 · p. 11, Figure 9 · p. 11

6 DISCUSSION

The discussion finds limited gains from current memory in preference selection: retrieval-based systems do not surpass No Memory, while File memory improves selection but still spills preferences across contexts. The appendices describe the preference taxonomy and the procedures for constructing and extracting persona preferences.

6 DISCUSSION

  • Current memory brings little gain in preference selection: retrieval-based systems do not surpass No Memory, and File memory still fails to constrain preferences to relevant contexts.Even with the right preference available, overriding the model’s default interaction remains difficult.Source: §6 · p. 11

A.1 ATTRIBUTES, SETTINGS, AND CONTEXTS

  • The 14 attributes specify discrete settings through observable PA replies or actions, with one setting assigned per attribute in each context.The taxonomy distinguishes interaction preferences from preferences about substantive content and groups attributes into four dimensions.Source: §A.1 · p. 20, Table 4 · p. 20, §A.2 · p. 22

A.2 LITERATURE FOUNDATIONS AND PROJECT EXTENSIONS

  • The attribute foundations draw on prior studies, but most report that the better setting depends on the request, situation, or user knowledge.The authors state trait-to-setting links as expectations.Source: §A.2 · pp. 22–23

B.1 TRANSCRIPT SOURCES AND PREPROCESSING

  • Four users are drawn from the first three seasons of The Big Bang Theory and The Office, where about 93% of preference cells were already classifiable as determinable or not.Cells that could not be determined from the transcripts went to human review.Source: §B.1 · p. 25

B.2 IDENTITY AND PREFERENCE EXTRACTION

  • Preference extraction uses two passes: one locates and contextualizes reactions to assistance, and the other selects a setting for each attribute–context cell with a linked rationale.Identity observations are synthesized in parallel to ground preference evidence in relationships, activities, and recurring circumstances.Source: §B.2 · p. 25

B.3 EVIDENCE, CONFIDENCE, AND HUMAN REVIEW

The authors review preference settings when evidence is weaker than high-confidence agreement, checking both the proposed setting and its context against transcript scenes. Cells without usable transcript evidence are specified manually.

B.3 EVIDENCE, CONFIDENCE, AND HUMAN REVIEW

  • Reviewers check whether a setting follows from its cited scenes and whether its context is warranted, resolving conflicting observations by circumstances rather than averaging incompatible signals.A cell is accepted only when it names a setting.Source: §B.3 · p. 25
  • Seven cells have no usable transcript evidence, all in Process Visibility, Solution Breadth, or Capability Boundary, and those settings are specified manually.The seven comprise five cells without an extractor proposal and two manually specified cells.Source: §B.3 · pp. 25–26
  • Of 112 cells, 53 high-confidence settings were confirmed, while 59 underwent review; reviewers kept 34 extracted settings and chose settings for 23 others.The 23 reviewer-chosen settings include five cells for which the extractor proposed none.Source: §B.3 · pp. 25–26, Table 5 · p. 26

B.4 ANONYMIZATION AND FINAL PERSONA STRUCTURE

The authors anonymize persona identities while preserving preference-relevant characteristics, then check whether preference reactions remain consistent and whether source-character knowledge offers a prediction shortcut. A worked example shows how responses to others can outweigh a character’s own speaking style when setting a preference.

B.4 ANONYMIZATION AND FINAL PERSONA STRUCTURE

  • Anonymization replaces named people, workplaces, and places with reviewed fictional substitutions while retaining roles, relationship closeness, and other non-anonymized fields.The final preference matrices contain 2 × 14 cells per user and use 30 of the 32 settings in the inventory.Source: §B.4 · p. 26
  • All 14 preference-preservation probes pass the reader check, indicating anonymization leaves how each user reacts to violated preferences unchanged.Each probe presents a PA message violating four settings, and readers compare whether named and anonymized personas respond as the same kind of person.Source: §B.4 · pp. 26–27, Table 6 · p. 27
  • With source identity, preference-prediction accuracy is .532, versus .464 with the anonymized profile and .462 with no information.The anonymized-profile score is close to no information at .452 chance accuracy; source identity gives the highest accuracy for every user.Source: Figure 10 · p. 27, §B.4 · p. 27

B.5 WORKED EXTRACTION EXAMPLE

  • For User C’s work Verbosity cell, the extractor and review retain Terse because the character cuts long explanations short when receiving them, despite giving long speeches.User C’s personal Verbosity setting is Detailed, so the preference is context-specific.Source: §B.5 · p. 28

C.1 WORLD DESIGN, TIMELINE, AND FIXTURES

MemPABench constructs four-month work and personal worlds with persistent, date-aware records and scripted sessions that expose user preferences through interaction. Controlled change events then update selected preferences after the PA’s behavior produces a consequence.

C.1 WORLD DESIGN, TIMELINE, AND FIXTURES

  • Each persona’s four-month work and personal storyline uses recurring people and places, six scenario types per context, 94 accumulation sessions, and five scheduled preference-change events.The three accumulation sessions for each stable preference cell use different tasks; they average 5.5 user–PA exchanges and 14.6 tool calls.Source: §C.1 · p. 28

C.2 TOOLS AND WORLD STATE

  • The PA uses 83 tools to read or change a persistent world, with completed writes carried into later sessions and every change traceable to its tool call.Records are hidden until their timeline entry date, and the PA can access only files in its working copy.Source: §C.2 · pp. 29–30

C.3 ACCUMULATION SESSION GENERATION

  • Sessions are scripted as beats, with a decision point where the tested preference becomes active and four mechanisms keeping live interactions on track.Withheld facts, draft checks, write locks for five attributes, and a required interaction-tool call prevent early disclosure or skipped decisions.Source: §C.3 · pp. 30–31

C.4 CONTROLLED PREFERENCE-CHANGE EVENTS

  • Each user receives five fixed preference-change events; the PA first follows the old setting, then the user encounters its consequence and states a new rule.The 20 events span nine attributes and change settings in both directions; each event is replayed into the PA’s history and memory, while later probes test the updated setting.Source: §C.4 · p. 31

C.5 CONSTRUCTION CHECKS AND ACCEPTANCE

Construction checks ensure that each setting can be enacted naturally and observably, and that the user can react to it. Scripts are then checked with rule-based validation and semantic review before acceptance.

C.5 CONSTRUCTION CHECKS AND ACCEPTANCE

  • A setting is viable only when the PA follows it and all five conditions hold, including a user reaction that can expose the preference during accumulation sessions.The checks require settings to be reasonably carried out, naturally and observably distinct, and open to a user response.Source: §C.5 · p. 31, Table 8 · p. 32
  • Rule-based validators check record bindings, tool steps, beat order, visibility, and required reactions, while two LLMs review every draft against the same written viability criteria.Only drafts that pass both validators and semantic review as live are accepted; failed drafts are regenerated or edited and checked again.Source: §C.5 · pp. 32–33

C.6 WORKED ACCUMULATION AND CHANGE-EVENT EXAMPLES

  • The worked examples show preferences entering history through a correction after a wrong selection or a new rule stated after a consequence.User B corrects an autonomous message sent without confirmation; in a separate event, the user changes Topic Management after a multi-errand reply leaves a flight unbooked.Source: §C.6 · p. 33

D.2 PREFERENCE EXPOSURE THROUGH BEATS AND ACTIVE SKILLS

  • Preference skills shape user turns and reactions, so correct behavior receives explicit acceptance while a mismatch prompts a correction or other named response.At the decision point the PA calls the tested attribute’s interaction tool, and the simulator’s next beat reacts to the delivered behavior.Source: §D.2 · pp. 33–34

D.3 CROSS-SESSION CONTINUITY AND INFORMATION BOUNDARIES

The benchmark carries separate world, PA, and simulator memories across sessions while restricting what each component can observe. Its wrapper makes preference choices explicit, controls their delivery, and keeps withheld content out of memory.

D.3 CROSS-SESSION CONTINUITY AND INFORMATION BOUNDARIES

  • World state, PA memory, and simulator memory persist across sessions, but probes branch from checkpoints and write nothing back to those stores.The simulator’s dated entries track how the PA treated it while omitting task subject matter, file paths, and exact numbers and dates.Source: §D.3 · p. 34, Table 9 · p. 34

D.4 PREFERENCE LEAKAGE CONTROL AND AUDIT

  • The leakage audit found no preference announcements without a behavioral trigger and no probe references to earlier sessions.Among judged sentences, 1,032 commented on the PA’s current reply and 21 converted a reaction into a general request; all 21 followed an action the PA had just taken.Source: §D.4 · p. 35

E.1 WRAPPER ROLE IN THE EXECUTION FLOW

  • The IX Wrapper turns selected settings and proposed actions into a public reply and executed or held actions, but it cannot see or correct the target setting.It standardizes realization without changing the assistant’s choice.Source: §E.1 · p. 35

E.2 SETTING SELECTION AND SUBSEQUENT BEHAVIOR

  • The assistant selects a setting once when an attribute first activates, and that choice remains active for the rest of the session.Later turns supply payloads such as clarification questions, outreach content, or proposed actions as task state changes.Source: §E.2 · pp. 35–36

E.3 WRAPPER MECHANISMS AND ACTIVATION

  • Four wrapper mechanisms handle pass-through, selective rendering, delivery scheduling, and execution control across preference attributes.Autonomy Level settings determine which actions execute and which are held for approval; held items retain identifiers and continuation state.Source: §E.3 · p. 36

E.4 PA OUTPUT, PUBLIC INTERACTION, AND MEMORY RECORDS

  • The memory receives user messages, tool calls and results, the interaction-tool call, and the delivered reply, but not held content or wrapper policy fields.The simulator instead sees the rendered reply and consequences of actions that actually executed.Source: §E.4 · p. 36

E.5 WORKED WRAPPER EXAMPLE

  • For a Self-directed setting, the wrapper executes the calendar write, holds the email, and releases it only after the user approves.The user’s approval releases the same held item, which is then sent as an email.Source: §E.5 · p. 36

F EVALUATION PROTOCOL

The evaluation holds users, schedules, tools, wrappers, checkpoints, and probe protocols constant across memory conditions, while keeping histories and memory states condition-specific. It scores preference selection, context and change sensitivity, behavioral realization, and user perception under distinct rules.

F.1 MEMORY CONDITIONS AND COMPARISON CONTROLS

  • Retrieval-based conditions rarely bring an applicable preference into the PA’s input, even when their stores hold it.The comparison uses shared users, session schedules, wrappers, tools, checkpoints, and probe protocols, but condition-specific histories and memory states.Source: §F.1 · p. 37

F.2 SCORING INPUTS, RULES, AND AGGREGATION

  • PSA scores exact-match setting selection, while BRS independently rates the public reply and executed-task trajectory on a 1–5 scale.CS requires correct selections in both contexts, and PET requires correct selections both before and after a preference change; PSA, CS, and PET are percentages.Source: §F.2 · pp. 37–38
  • A capability failure receives PSA = 0 and BRS = 1 without reducing the denominator, while probes failing the interaction-tool contract are floored only under the stated failure rules.Of 70 floored probes, 64 failed in every run; 60 of those omitted a required interaction-tool call.Source: §F.2 · p. 37, Table 12 · p. 38

F.3 CLUSTER-BOOTSTRAP INTERVALS

File memory clearly improves PSA over No Memory, while retrieval-based systems remain near it; File memory also produces more cross-context carry-overs.

F.3 CLUSTER-BOOTSTRAP INTERVALS

  • The 95% intervals use 5,000 cluster-bootstrap resamples, keeping repeated runs of each drawn probe, pair, or cell together across backbones.Intervals span the 2.5th to 97.5th percentiles of resampled values.Source: §F.3 · p. 38
  • File memory gains 9.6 PSA points over No Memory (95% interval [+3.9, +15.6]) and wins in 14 of 16 user–backbone pairs; Mem0, MemOS, and Simple RAG stay near No Memory.Their PSA differences are +0.4, +0.4, and +0.2 points, respectively; No Memory beats them in 10, 8, and 8 of 16 pairs.Source: §F.3 · p. 38, Table 13 · p. 38
  • File memory has 58 cross-context carry-overs versus 13 for No Memory, with more in every backbone, user, and repeat; its PET and carry-over gains have intervals above zero.Each condition includes 216 pair instances; no CS difference has an interval above zero.Source: §F.3 · p. 38, Table 13 · p. 38

G.1 PSA, BRS, AND PRS BREAKDOWNS

File memory raises preference-selection accuracy more for preferences governing actions than replies, while backbone PSA rankings do not track general capability and backbones show distinct defaults.

G.1 PSA, BRS, AND PRS BREAKDOWNS

  • File memory raises PSA for action preferences from 40.3% to 54.3% and for reply preferences from 56.6% to 62.9%, narrowing their gap without closing it.Under No Memory, action-preference PSA is below random selection at 44.4%.Source: §G.1 · pp. 38–39, Table 14 · p. 39

G.2 PSA AND GENERAL CAPABILITY

  • GPT-5.6-Sol has the highest Intelligence Index but lowest PSA, whereas Qwen3.8-27B has the highest PSA despite a GDPval-AA rating close to GPT-5.6-Sol.The reported PSA ranking of the four backbones does not follow their general capability ranking.Source: §G.2 · p. 41, Table 19 · p. 41

G.3 BACKBONE DEFAULT SETTINGS

  • Under No Memory, each backbone’s most frequent setting defines its default; GPT-5.6-Sol selects Detailed for Verbosity in all 24 selections.The defaults are pooled across users, repeats, and final and pre-event probes.Source: §G.3 · p. 41

G.4 DELIVERED BEHAVIOR AFTER A CORRECT SELECTION

Even when Oracle supplies the target setting, some delivered interactions fail to follow it, with shortfalls across disclosure, verbosity, elicitation, and solution breadth.

G.4 DELIVERED BEHAVIOR AFTER A CORRECT SELECTION

  • Under Oracle, pass rates are 38% for Hide, 50% for Terse, 61% for Iterative, 74% for Structured, and 77% for Low across five settings.These are the lowest-pass-rate settings for Reasoning Visibility, Verbosity, Information Elicitation, and Solution Breadth.Source: §G.4 · p. 41
  • With Terse selected, a reply still lists six steps and adds a closing aside, earning BRS 1 and a user complaint about verbosity.The target setting asks for the answer in the first sentence and at most one supporting move.Source: §G.4 · p. 41
  • With Low selected for Solution Breadth, the PA recommends one lamp but frames each reason as a comparison with an alternative, earning BRS 1.The user objects to being walked through what the other lamp could not do.Source: §G.4 · p. 41
  • With Iterative selected, the PA acts on a calendar hold without asking for the missing arrival time, and the user objects to the guess.The judge gives the interaction BRS 3.Source: §G.4 · p. 42

H VALIDITY CHECKS

The audits find that preferences are generally conveyed and simulator sessions follow their scripts; agreement checks also distinguish user-rated reception from common-rubric scoring.

H VALIDITY CHECKS

  • Annotators judged preferences identifiable from visible conversation in 99.3% of sampled sessions, with 95.4% inter-annotator agreement.The sample comprised 112 sessions from File-memory runs.Source: H VALIDITY CHECKS · p. 42
  • The post-hoc audit found no preference leakage, and annotators found no scripted-session deviations in the sample, with 99.1% agreement.The script audit covered a random 10% of accumulation sessions.Source: H VALIDITY CHECKS · pp. 42–43
  • On 7,996 final probes, BRS and PRS agree exactly on 65.3% and within one point on 90.0%; an independent Claude judge’s PRS agreement is lower than Gemini’s.Spearman ρ is 0.83 overall, 0.71 for Claude–PRS, and 0.85 for Gemini–PRS; the Claude–Gemini difference is −0.14 (95% CI [−0.20, −0.09]).Source: H VALIDITY CHECKS · p. 43
  • On correctly selected Oracle probes, PRS falls below BRS on 16.1% overall, ranging from 11.0% for User A to 21.5% for User C.PRS reflects each user’s reception; BRS applies a common rubric for comparing settings.Source: H VALIDITY CHECKS · p. 43, Table 21 · p. 43

I PROMPT DIRECTORY AND TEMPLATES

This appendix documents the construction, runtime, and evaluation prompts used to operationalize interaction preferences, from grounding simulated users and preference updates to probes, assistant behavior, and judging. The prompts specify observable criteria for assessing replies, tool calls, and transcript events, including rules for ambiguity, safety, task scope, and avoiding false positives.

I.1 PROMPT DIRECTORY

  • Table 22 indexes the construction, runtime, and evaluation prompts; some long prompts are summarized with excerpts, while the full prompts are released with the code.Runtime-filled text appears in braces or angle brackets, and highlighted notes or omissions are not sent to the model.Source: §I.1 · p. 43, Table 22 · p. 44

I.2 CONSTRUCTION PROMPTS

  • Preference extraction tags clear scene evidence by context, then synthesizes one of 14 settings per context using production patterns and reception as confirmation or override.Thin or contradictory evidence still receives a best-fitting setting at low confidence, and every setting below high confidence goes to human review.Source: §I.2 · p. 45
  • Identity prompts record scene-grounded behavior with verbatim quotes, merge observations into stable identity items, and rate personality facets and functions from those observations alone.Non-null ratings require supporting quotes; levels reflect consistency and cost across contexts, while unsupported ratings are null.Source: §I.2 · p. 46
  • Anonymization replaces identity markers while preserving the persona’s archetype and life structure, then checks whether anonymized replies push back on the same preference violations.The validator flags objections to the assistant’s handling, not merely attributes reflected in the reply’s own style.Source: §I.2 · p. 47
  • The self-view extractor records what users recall about their files in exact, rough, or unavailable tiers, with each detail assigned to exactly one tier.Recall tiers reflect user traits; exact and unavailable strings must match the source text character for character.Source: §I.2 · p. 48
  • Session outlines specify goals, phases, and correct and wrong preference branches; both branches share the same request, facts, tools, and base outcome.Nine attributes use a one-reply compressed task; the other five require a gate, and change events make the old behavior cause a felt loss before naming the new setting.Source: §I.2 · pp. 48–49
  • The accumulation-session generator expands accepted outlines into runnable tool steps and beats while leaving the assistant unscripted and the user’s preference-based reaction to runtime.Uncompressed sessions have at least five beats; the probe reveals a needed fact, and each setting-specific reaction path completes the task without a verdict.Source: §I.2 · p. 49
  • Probe prompts create preference-neutral questions that isolate one target attribute, with only designated uncompressed attributes receiving a real gate requiring a value absent from the world.Pre-event probes preserve the task condition and feasible options while rewriting only what the clock changes; runtime prose cannot alter code-built steps or arguments.Source: §I.2 · pp. 49–50
  • Construction beats are preference-neutral: they do not prescribe presentation style, and every setting of the target attribute remains a reasonable answer.Source: §I.2 · p. 50
  • Change events move from the PA’s old-setting behavior through its consequence and the user’s realization to a new rule, while keeping the new-setting path unplayed.The sequence must be causally ordered; the separate new-setting path avoids the cost, and any gated wording comes before the behavior beat.Source: §I.2 · pp. 50–51
  • Accepted change-event drafts become fixed transcripts in which the PA performs the old-setting behavior and the user states the new rule visibly, without introducing a second preference.Each beat receives an exchange, and the final user message states the rule in the user’s own words.Source: §I.2 · p. 51
  • Dialogue-naturalness instructions favor immediate, concise user turns and let reactions, diagnoses, and repair rules emerge across turns rather than as polished summaries.They also treat persona traits as tendencies, not a checklist, and distinguish reception guidance from production guidance.Source: §I.2 · pp. 51–52

I.3 RUNTIME PROMPTS

  • Runtime prompts assemble simulator context from the user profile, current-beat preference skills, and session script, while a checker reviews messages before the PA receives them.The PA uses the same system prompt in accumulation sessions and probes and is offered an attribute interaction tool at relevant decision points.Source: §I.3 · p. 52
  • The profile prompt keeps the simulator in the user’s anonymized first-person perspective and grounds task responses in session facts rather than invented details.The simulated user has no external task tools and should defer exact document wording to the PA unless session facts provide it.Source: §I.3 · pp. 52–53
  • In accumulation sessions, the simulator conveys active preferences through behavior, reacts only to the PA’s actual replies, and gives task facts or consent only when the beat or PA request permits.The user does not name preferences, and facts assigned to later beats are withheld until then.Source: §I.3 · pp. 55–56
  • In pre-event and final probes, the simulator asks for the task and takes the PA’s answer without feedback on its manner; factual checks record factual errors only.Source: §I.3 · p. 56
  • Probe prompts present one standalone question as first contact and prevent users from referring to prior sessions, naming preferences, or giving feedback on answer manner.The simulator speaks only as the user, corrects factual errors, and does not praise, complain about, or otherwise steer the response style.Source: §I.3 · pp. 56–57
  • Checker calls return one JSON verdict per supplied constraint or question, with evidence pointing to numbered sentences and no extra questions or entries.The prompt distinguishes general-form checks from per-rule checks and requires evidence only for specified verdicts.Source: §I.3 · pp. 57–58
  • Direction and avoid-clause checks judge only the attached turn; a direction already handled earlier in its multi-turn beat is not applicable on the current turn.The checker must not infer unlisted constraints or treat compliance with a direction as violating a paired avoid clause.Source: §I.3 · pp. 57–58
  • Cue checks accept equivalent wording, but a bare noun-phrase cue counts only when the user raises the missing item for the assistant to obtain, not merely its topic.Handing the item over outright still obeys the cue; a violation requires that the candidate turn omit it.Source: §I.3 · p. 60
  • Avoid-clause checks treat refraining from the prohibited act or doing its opposite as obeying, while discussing how the beat goal landed is also allowed.The checker flags a violation only when the candidate line itself performs the avoided behavior.Source: §I.3 · p. 61
  • Fact-missing checks search the candidate and earlier user turns for the fact’s content, accepting paraphrases and equivalent forms without judging whether it was owed on that turn.The checker evaluates whether the fact was said at all, including when its authored value is phrased in third person or converted to an everyday form.Source: §I.3 · p. 62
  • Ask detection counts a question, instruction, or request for assistance as an ask, but not a turn that only reports, corrects, acknowledges, or closes a matter.Releasing the assistant to proceed or supplying information it requested counts as a request; an instruction merely to hold or not proceed does not.Source: §I.3 · p. 63
  • Withheld-fact rules gate disclosure until the reveal beat and require the assistant to ask before the scripted sentence is handed over.The prompt supplies the required terms and whether they were delivered, leaving the checker to use those provided values rather than recalculate them.Source: §I.3 · p. 63
  • The withheld-fact prompt checks whether a turn fills the requested slot, separately from whether the assistant asked for its wording or asked about that slot earlier.A user-provided stand-in can count as filling the slot; merely discussing or postponing it does not.Source: §I.3 · pp. 64–65
  • The unexecuted-action check compares a user’s claim about completed assistant actions against a complete record, excluding reply-surface claims and proposed or requested actions.An act absent from the record is treated as not having happened.Source: §I.3 · p. 65
  • The action-authorization check requires both a ban on handing work over and an immediate release addressed to the assistant, while accounting for carve-outs and conditional or user-directed go-aheads.A go-ahead from the user about their own next step does not release the assistant to act.Source: §I.3 · pp. 65–66
  • The contraction check distinguishes reports about the current reply from demands about future replies, then compares the report with the code-provided contraction list.If there is no report about the current reply, the correctness check is not applicable.Source: §I.3 · pp. 66–67
  • A future-policy violation requires one sentence containing both a recurring-time marker and a directive or expectation; negated, quoted, or one-reply reactions are excluded.A policy may concern a process or standard as well as a behavior.Source: §I.3 · p. 67
  • The off-attribute check flags reaction wording that opens an aspect of the reply not owned by the measured attribute, while allowing mentions that close that aspect or concern task material.The prompt defines reply aspects to include length, reasoning, wording, autonomy, scope, and format.Source: §I.3 · p. 68
  • The preference-naming check distinguishes designer-style labels for a preference or setting from ordinary requests that express the desired behavior in the user’s own words.Quoted assistant wording and labels describing task material are not treated as the user naming a preference.Source: §I.3 · p. 68
  • The File-memory system prompt supplies the current memory in context and directs the assistant to use available tools, avoid unconfirmed action claims, and apply available interaction-preference tools.It also says outward messages require an instruction in the same turn and that tools persist or send artifacts rather than authoring them.Source: §I.3 · p. 69
  • The runtime prompt separates renderer-controlled selection and ordering from assistant wording and provides user-request and proactive-event context.It also directs the assistant to analyze tool errors before retrying and to clarify ambiguous requests.Source: §I.3 · p. 70
  • Runtime tools define explicit settings for tone and formality, verbosity, emotional engagement, and guidance level.The choices specify observable response features, from colloquial or professional register to direct versus scaffolded explanations.Source: §I.3 · pp. 71–72
  • Reasoning visibility, uncertainty expression, and process visibility settings control whether users see rationale, confidence annotations, and action narration.The prompt requires the corresponding fields even when the selected setting hides their contents.Source: §I.3 · pp. 73–74
  • Autonomy settings range from requiring confirmation for actions to allowing delegated non-routine choices, while still requiring clarification for material ambiguity or exceptional risk.Self-directed autonomy permits routine, reversible work but gates key internal writes and external actions; autonomous execution stops at real exceptions or out-of-scope decisions.Source: §I.3 · p. 75
  • Proactive outreach settings suppress or deliver relevant event-triggered contact, while task-expansion settings govern whether the assistant acts beyond the requested deliverable.Even when outreach is enabled, it is limited to events tied to an existing user goal.Source: §I.3 · pp. 75–76
  • Other settings govern one-option versus multi-option answers, recovery versus handoff at capability limits, assumption-based versus structured clarification, and grouped versus deferred topics.Structured or iterative elicitation can stop task calls until questions are answered; one-at-a-time topic management defers later topics.Source: §I.3 · pp. 76–77

I.4 EVALUATION PROMPTS

  • The evaluator judges one probe at a time using the reply text and tool trajectory against known ground truth, with a shared 1–5 conformity scale.Directly asking the user which preference they want always earns a score of 1, and preference scoring remains separate from task completion.Source: §I.4 · p. 78
  • When a probe’s wording fixes the response regardless of the target setting, the evaluator flags it as degenerate and identifies the constraining sentence.The probe must leave room for every setting to produce a clearly different response.Source: §I.4 · p. 78
  • Judges receive the scene, transcript, and tool trajectory before the attribute rubric, and are instructed to read that context before applying the criteria.The scene supplies the core task and world context; transcript and tool trajectory show the PA’s delivered replies and actions.Source: §I.4 · pp. 79–81
  • Evidence must be a verbatim substring of a PA reply or a tool event matching an existing trajectory time and tool, and corrected-argument retries are not failures.The judge returns a JSON object with score, rationale, evidence, and flags; a failed machine check triggers a request to resubmit the complete JSON.Source: §I.4 · pp. 79, 81
  • Tone and formality are judged from the PA’s user-facing reply, with Consultative requiring a collaborative move and Casual distinguished by at least two colloquial markers.The rubric also distinguishes Formal through restrained professional wording, a formal connective, and a conventional opening or closing; tool calls and email text addressed to third parties do not indicate register.Source: §I.4 · pp. 82–83
  • Verbosity is based on relevant supplementary information categories, not word count: Terse allows at most one move, while Detailed requires at least two from different categories.The categories are background, caveat, boundary condition, downstream consequence, and alternative; an unusable answer is a task failure, not evidence of Terse style.Source: §I.4 · p. 83
  • Emotional engagement depends on whether empathy is absent and task content starts immediately, or an opening sentence acknowledges the user’s emotion before the task proceeds.Empathy cannot substitute for completing the task; a reply without a task result is a task failure rather than Relationship-focused behavior.Source: §I.4 · pp. 83–84
  • Guidance is judged by whether new terms are explained at first use and steps are ordered, with the user’s own unfamiliarity signal governing which named term must be explained.If the user names no unfamiliar term, judges assess explanations of terms new to the reply; scene annotations cannot substitute for the user’s message.Source: §I.4 · p. 84
  • Reasoning visibility distinguishes a specific basis passage shown before the answer from a reply with no basis passage, and does not count process steps or supplementary verbosity as that basis.When process sentences are present, the basis follows them; the two attributes use distinct passages as evidence.Source: §I.4 · p. 85
  • Uncertainty expression is judged only from rendered confidence parentheticals, which must match the facts; prose hedging is not a judged channel.The rubric directs judges to inspect literals such as “(confidence: low)” in the reply body.Source: §I.4 · p. 85
  • Express marks decision-relevant facts or conclusions with confidence parentheses matching their factual basis, while Hide uses no parentheticals and states limitations only when materially consequential.Missing confidence marks are judged from key judgments in the public answer; incorrect confidence or fabricated certainty is a task failure, not a setting match.Source: §I.4 · p. 86
  • Process visibility is judged by comparing progress sentences actually delivered with executed tool calls, with Silent, Bookend, and Full narration requiring different counts.The count uses calls in tool_log and sentences in the reply; gate turns and turns without calls have special rules, and after-the-fact step recaps do not count.Source: §I.4 · pp. 87–88
  • Autonomy is graded by where the PA stops across a user-named internal write and a consequential external action: before both, between them, or after both.Reactive stops before the write, Self-directed performs the write but stops before the external action, and Autonomous executes both; missing either step makes the probe degenerate.Source: §I.4 · pp. 88–89
  • Proactive outreach is scored by whether a relevant trigger prompts an actual message when no request is active: Low suppresses delivery, while High sends it.High outreach must relate to the user's existing goals and state what changed or is due; it does not authorize unrelated work.Source: §I.4 · p. 90
  • Task expansion distinguishes Low, which performs no separable adjacent work, from High, which executes at least one extra operation outside the items named in the request.The extra step must be separable from completing the request and explicitly labeled in the reply; suggestions and necessary work on named items do not count.Source: §I.4 · pp. 90–91
  • Solution breadth is determined by the number of distinct viable options presented: Low is K = 1, while High is K >= 2, regardless of whether the PA recommends one.Options must be able to solve the request and are checked against feasible options; retrieving multiple candidates does not count unless they are presented.Source: §I.4 · pp. 91–92
  • Capability boundary scoring anchors on the first tool-reported capability limit, then distinguishes recovery attempts from diagnostic queries and checks whether the PA hands control back or continues.Only actual workaround, correction, or recovery calls after the limit count as attempts; retryable parameter-format errors do not establish the blocking point.Source: §I.4 · pp. 92–93
  • For blocked tasks, `Suggest alternatives` requires an immediate same-turn recovery call, while `Find and hand off` requires no recovery attempt and an end-of-reply hand-off stating what was blocked and what the user must do next.Both settings must explain the limit; diagnosis alone counts as hand-off rather than recovery, and an ambiguous call cannot establish either setting by itself.Source: §I.4 · pp. 93–94
  • Information elicitation is judged from question turns (Q), questions per turn (Qn), and the first task-call turn (Tf), against deliberately withheld required facts with candidate values available in world records.The rubric measures whether, when, and how many questions are released—not whether the information exists in the world.Source: §I.4 · p. 94
  • `Infer` asks no questions and acts in turn 1 using a defensible, task-safe default from world-record candidates; assumptions need not appear in the reply.Wild guesses or unsafe assumptions that produce unusable results are task failure, not compliant inference.Source: §I.4 · p. 95
  • `Structured` asks all currently knowable required-field questions together and stops before task calls, whereas `Iterative` releases one missing-field question per turn and makes progress in turns without questions.Structured coverage is checked against withheld required facts; Iterative requires at least two asking turns when multiple independent facts are missing.Source: §I.4 · p. 95
  • Question counts track independent required information items, not question marks or repeated entries; when only one fact is missing, the probe cannot distinguish `Structured` from `Iterative`.Permission to execute is not a missing-information question, and asking for missing facts is distinct from asking permission.Source: §I.4 · p. 95
  • Topic management requires agreement between topic-labeled tool-call order and reply chunks, while retaining every topic the user raised; both settings can be judged in one turn.The rubric also checks how many topics are substantively addressed and whether the reply explicitly defers remaining work.Source: §I.4 · pp. 96–97
  • `Organize` runs all topic batches together, groups calls by topic, and reports labeled reply chunks; `One-at-a-time` runs only the first batch and explicitly suspends the rest while asking before moving on.Tool logs establish which topics ran, while reply text must show organization or explicit suspension; dropping a topic is task failure, not either setting.Source: §I.4 · pp. 96–97
Loading 2610.30485v1…