PaperlayerView exact revision on arXiv

Source-linked AI summary

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao, Fanyi Wang, Tengda Guo, Wentao Qiu, Ying Wang, Liujian Tang

arXiv:2610.03020v1cs.AI

TL;DR

Existing benchmarks leave implicit how a particular agent should work with a user as their shared history evolves. DyadMem jointly annotates user-side and relationship-specific memory and evaluates the full memory pipeline; Gold-Memory QA is strong, while Full-Pipeline QA is substantially lower.

  • Problem

    Existing benchmarks study user memory and reusable agent experience largely separately, leaving implicit what a particular agent should do differently for a particular user.

    Source: §1 · p. 2

  • Method

    DyadMem jointly annotates user-side memory and User-conditioned Relational Agent Memory on shared multi-session trajectories and evaluates memory stages through final-answer QA.

    Source: §1 · pp. 2–3

  • Results

    Gold-Memory QA ranges from 78.7% to 96.3%, while Full-Pipeline QA averages 47.7% for open-weight models and 57.4% for proprietary models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Jointly incorporating user-side memory and URAM outperforms general experience memory across all 20 LLMs.

    Source: §1 · p. 2

Abstract

from arXiv · show

Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves. Existing benchmarks primarily supervise user facts and preferences or experience reusable across users, leaving this relationship-specific agent memory implicit. Additionally, most prior works measure the model solely with final-answer QA over long interaction histories, making the assessment still incomplete and unreliable. To this end, we introduce DyadMem with the proposed new definition User-conditioned Relational Agent Memory (URAM). DyadMem jointly annotates user-side memory and URAM along the same multi-session trajectories, resulting in 6 memory categories. To summarize, it includes 3,065 episodes, 50,961 sessions, and 61,210 QA instances, with extensive session-level Capture and Update gold annotations, query-level Recall support, and two QA settings: Gold-Memory and Full-Pipeline. Across 16 open-weight and 4 proprietary models, Gold-Memory QA is consistently strong, yet Full-Pipeline QA drops sharply. Such a gap explicitly supports our fine-grained evaluation design. Additionally, several quantitative results further reveal low Capture recall, incomplete Recall, and unsafe-deletion issues arising from even the frontier LLMs. We further conduct a rigorous experiment to validate the effectiveness of our URAM and observe the positive effects for all 20 models. In summary, DyadMem is a dual-domain, full-pipeline memory benchmark with extensive annotation efforts for advancing the domain's development.

1 INTRODUCTION

DyadMem addresses two gaps in long-term memory research: relationship-specific agent memory is often implicit, and final-answer QA alone does not reveal where memory pipelines fail. It jointly annotates user-side memory and URAM on shared trajectories and evaluates memory stages as well as end-to-end QA.

1 INTRODUCTION

  • Existing work often studies user memory and transferable agent experience separately, leaving agent behavior specific to one user relationship less explicit.DyadMem frames this as a dual-domain view of long-term memory.Source: §1 · p. 2
  • Prior work commonly evaluates long-term memory through final-answer QA, which can collapse memory construction, maintenance, retrieval, and answer use into one outcome.An answer can be correct despite an incomplete or unsafe memory bank, while an incorrect answer does not identify the failed stage.Source: §1 · p. 2
  • Session-level Capture and Update annotations, query-level Recall supports, and two QA settings evaluate distinct stages and their end-to-end consequence.Gold-Memory QA supplies annotated supporting memories; Full-Pipeline QA uses a memory bank built and retrieved by the model.Source: §1 · pp. 2–3
  • Gold-Memory QA ranges from 78.7% to 96.3%, while Full-Pipeline QA averages 47.7% for open-weight models and 57.4% for proprietary models.Across 20 models, stage-level results also reveal low Capture recall, incomplete Recall, and unsafe deletion; jointly incorporating user-side memory and URAM outperforms general experience memory across all 20 models.Source: §1 · p. 2
  • URAM represents how a particular agent should work with a particular user as their shared history evolves.Two agents serving the same user may need different future behavior because their interaction histories differ.Source: §1 · p. 2
  • DyadMem jointly annotates user-side memory, URAM, and shared episodic memory on the same multi-session trajectories.The benchmark contains 3,065 episodes, 50,961 sessions, and 61,210 QA instances.Source: §1 · p. 2

2 RELATED WORK

Related work addresses user memory, transferable task experience, user-conditioned behavior, and components of memory lifecycle evaluation. DyadMem combines these directions by explicitly supervising two coupled memory domains and their lifecycle on the same interaction trajectory.

What long-term memory represents

  • User-memory benchmarks primarily describe user facts, profiles, and preferences, while experience-based systems capture workflows or task knowledge reusable across tasks.DyadMem represents both the user and how a particular agent should work with that user on the same trajectory.Source: §2 · p. 3

User-conditioned agent behavior

  • User-conditioned behavior methods adapt through preferences, corrections, memory, policies, or feedback, but generally assess resulting behavior rather than directly supervising evolving agent-side memory.DyadMem makes URAM entries typed, dialogue-grounded, co-annotated with user memory, and labeled when added, revised, or invalidated.Source: §2 · p. 3

Lifecycle-level evaluation of long-term memory

  • Prior benchmarks cover different lifecycle components, while DyadMem brings per-session gold memory, gold update operations, and query-to-bank supports together for two coupled memory domains.Gold-Memory QA uses recall supports as given evidence, while Full-Pipeline QA composes the stages.Source: §2 · p. 3, Table 1 · p. 3

3 METHOD

DyadMem represents sessions, evolving memory banks, and QA supports, then constructs dialogue-grounded annotations and evaluates Capture, Update, Recall, and QA separately and end to end.

3.1 TASK FORMULATION

  • DyadMem represents each episode as chronologically ordered sessions plus a QA set; sessions contain temporal anchors, complete dialogues, gold Capture entries, and gold Update operations, while QA instances include reference answers and terminal-bank Recall supports.A temporal anchor is the session date and time used to resolve relative expressions such as “tomorrow.”Source: §3.1 · p. 4
  • Six memory types separate user_profile and fact, three URAM types, and shared episodic entries that may support either domain.The URAM types are agent_identity, agent_method, and agent_knowledge.Source: §3.1 · p. 4
  • Capture receives only the current dialogue and temporal anchor; standalone Update receives gold prior memory and gold Capture entries, whereas Full-Pipeline Update uses model-produced prior memory and Capture output.Dedup collapses entries with the same key while retaining the last occurrence.Source: §3.1 · p. 4

3.2 DATASET CONSTRUCTION

  • Complete multi-session dialogues are generated and reviewed before Capture, Update, and Recall annotations are reconstructed from realized interaction evidence.Chronological replay records replacements as a deletion plus addition, explicit termination as deletion only, and sessions without substantive change as leaving the bank unchanged.Source: §3.2 · p. 4
  • Nine quality-control annotators review episodes against the original dialogue and chronologically reconstructed bank, with corrections propagated to affected memory indices and QA annotations.Source: §3.2 · p. 4

3.3 EVALUATION FRAMEWORK

  • Capture, gold-conditioned standalone Update, and Recall are evaluated independently, while QA scoring checks semantic consistency and whether answers are supported by supplied memories.Update operation-level F1 excludes sessions with no gold Add or Delete; empty retrieval is a failure because each QA instance has non-empty gold support.Source: §3.3 · p. 5
  • Gold-Memory QA and Full-Pipeline QA distinguish answer generation with supplied gold evidence from answering with a model-constructed and retrieved memory bank.The complete-pipeline gap measures overall performance loss, not a causal decomposition of individual stage errors.Source: §3.3 · p. 5
  • Truncated, unparseable, empty, and failed outputs remain in the relevant denominator and receive no credit.Cross-model comparisons fix prompts, judge, matcher, embedding model, and NLI configuration; confidence intervals use 200 bootstrap resamples.Source: §3.3 · p. 6

4 EXPERIMENTS

Across 20 frontier models, the experiments compare memory-stage performance with Gold-Memory and Full-Pipeline QA. Models perform much better with gold memories than through the complete pipeline, while controlled tests and subgroup analyses expose additional differences.

4.2 MAIN RESULTS

  • Open-weight Capture averages 15.1% recall; top rank-one Recall rates reach 85–89% while Recall@n stays below 63%, and unsafe-deletion rates average 84.1% and 78.1% for open-weight and proprietary models.Gold-conditioned standalone Update averages 18.6% op-F1 for open-weight and 24.5% for proprietary models.Source: §4.2 · pp. 6–7
  • Gold-Memory QA ranges from 78.7% to 96.3%, while Full-Pipeline QA averages 47.7% for open-weight and 57.4% for proprietary models.The Gold-to-Pipeline gaps are 42.9 and 33.9 points, respectively; these are end-to-end gaps, not causal allocations of error to individual stages.Source: §4.2 · p. 7
  • Kimi-K3 achieves the highest Full-Pipeline score at 74.9%, despite 18.6% Capture recall and 31.7% gold-conditioned standalone Update op-F1.Its 95.0% Gold-Memory score yields a 20.1-point Gold-to-Pipeline gap; 7.94% of answers receiving positive correctness credit are unsupported by retrieved memories.Source: §4.2 · p. 7

4.3 USER-CONDITIONED RELATIONAL AGENT MEMORY VERSUS AGENT-ONLY

  • All 20 models improve on controlled URAM Update, while URAM-only QA gains are positive for 17 models and average +3.2 points for open-weight and +3.7 for proprietary models.DeepSeek-V4-Pro gains the most at +9.8 points; the comparison uses 500 paired episodes and 1,735 URAM-only questions.Source: §4.3 · p. 7, Table 3 · p. 8

4.5 MEMORY TYPE, NOT LIFE DOMAIN, SETS CAPTURE DIFFICULTY

  • Across models, Capture F1 is about 50% for agent_identity and agent_method, 25% for fact, and 16% for shared episodic memory, but spans only 22.7–26.3% across life domains.Interpersonal & Communication has the lowest Capture F1 at 22.7% while retaining 50.1% Full-Pipeline QA.Source: §4.5 · p. 8

4.4 USER-SIDE MEMORY IS HARDER TO CAPTURE AND USE

  • Open-weight Full-Pipeline QA differs by 14.8 points between user- and agent-centric questions, versus Capture and Recall gaps of 3.4 and 0.9 points; proprietary QA averages 51.0% and 63.6%, respectively.The partitions contain 30,468 user-centric and 30,742 agent-centric questions.Source: §4.4 · p. 7, Table 4 · p. 9

4.6 JUDGE RELIABILITY

  • The three judges identify the same top seven models, with no rank shift exceeding three places; human labels agree 86.52% with the default judge (Cohen’s κ = 0.7099).All pairwise judge rank correlations are at least 0.988; human calibration covers the semantic-consistency decision used for QA accuracy.Source: Figure 3 · p. 9, Table 5 · p. 9, §4.6 · p. 9

4.7 ANSWERING STYLE CHANGES FULL-PIPELINE QA

  • Qwen3-4B’s 43.6% Full-Pipeline QA exceeds Qwen3.5-9B’s 38.1% despite weaker stage metrics; across domains, QA should be interpreted with grounded accuracy and abstention.The reversal tracks answering styles and appears in both QA-target partitions and all five life domains.Source: §4.7 · p. 10

C DATA SOURCES

This section describes DyadMem’s controlled synthetic user–agent data and the construction prompts and review procedures used to create and annotate its multi-session conversations. The prompts specify six memory types and their capture and evolution, as well as rules for generating personalized questions grounded in those memories and their supporting recalls.

C DATA SOURCES

  • The dataset uses synthetic multi-session interactions designed to model user preferences, instructions, and behavioral changes, with automated checks, human review, and sampling-based validation.The controlled process avoids private or personally identifiable user data.Source: C DATA SOURCES · p. 17

D ANNOTATOR INFORMATION AND WORKFLOW

  • Reviewers retained 3,065 of 4,000 constructed episodes (76.6%) after primary review, with a sampled second pass used to compare judgments and review additional episodes.Nine annotators conducted quality control; overlapping random samples enabled independent review of some episodes.Source: D ANNOTATOR INFORMATION AND WORKFLOW · pp. 17–18

D.1 REVIEW PROCEDURE

  • A 100-group pilot established shared rules for memory types, Update operations, and QA support; formal review checked episode consistency, Capture, Update evidence, and gold QA support.Rules and models flagged suspected errors but did not write gold labels; episodes entered the final set only after second-review sampling and a consistency check.Source: §D.1 · p. 19

D.2 RECRUITMENT, COMPENSATION, AND CONSENT

  • All nine annotators worked full time for two months, and the study did not undergo formal IRB review because it reviewed generated dialogues rather than participants’ life records.The released data omit participant names and API keys, and participants could withdraw from unfinished work.Source: §D.2 · p. 19

E PROMPTS FOR DATA CONSTRUCTION

  • Construction prompts adapted personas into AI power users who delegate recurring tasks, maintain long-term assistant rules, rely on cross-domain memory, and seek the assistant’s judgment.The original persona’s identity, profession, and life background were to remain intact.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 19–20
  • Dialogue prompts required natural English chats with realistic continuity and safety constraints, while reserving about 15% of turns for nondisclosive off-topic chitchat.The prompts also disallowed invented real-world actions or personal life for the agent and prohibited chitchat from introducing extractable user facts or new plans.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 20–21
  • The agent defaults to tidy, detailed English, but an explicit user style instruction persists across later sessions; replies should remain natural, objective, and non-templated.The prompts also discourage customer-service sign-offs, forced empathy-advice patterns, and repetitive openings or closings.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 23
  • Episodes must preserve facts, plot, and character continuity, while sessions represent separate short sittings with time-appropriate progress and a trackable mini-plot.Progress requiring longer than a few minutes must occur between sessions, not within one; timestamps are assigned separately, so the writing should not anchor the current moment to a specific time of day.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 23–24
  • Sampled memory evolutions must occur naturally across sessions, with old-to-new correspondence made clear and two-hop chains evolving from A to B to C.Episodes must include change rather than additions alone, while allowing many sessions to contain only new information or chat.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 25
  • User-profile facts should be limited to those needed by the sampled evolution forms, while a few standalone entity facts may surface naturally as lived-in detail.Agent-side plots should take priority over adding surrounding stable profile facts; agent-side memory material includes identity, methods, and knowledge.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 25
  • Agent-side plots should feature agent_identity most, agent_method next, and agent_knowledge least, with the episode’s user- or agent-centric task determining their share relative to user-side plots.The user should bring back outcomes of prior approaches and may ask the agent to distill methods or lessons after work wraps up or a snag recurs.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 25–26
  • Agent_identity requires explicit, colloquial long-term rules for recurring situations, covering both collaboration boundaries and assigning the agent a task role.Each user message should raise one point at a time, and the agent’s confirming echo does not itself count as a memory.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 26
  • Agent_method must be a reusable way the agent works, supported across sessions and confirmed or corrected after user feedback; agent_knowledge requires accumulated evidence before a stable lesson is stated.Knowledge usually emerges from the third session onward after similar events or comparisons, and must show its provenance rather than present a one-off suggestion as a rule.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 26–27
  • Generated conversations must use one flat, strictly alternating user-agent turn sequence, while capture annotation extracts zero or more memories independently for each target-session turn.The dialogue is truncated at the target session, and each extracted memory must be supported by its own turn; the rest of the dialogue is only context for understanding and reference resolution.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 27
  • Turn roles constrain eligible types: user turns may yield episodic, user_profile, or fact memories, while assistant turns may yield episodic or agent-side types.Information that belongs to a forbidden type is recorded as episodic only when it fits that definition; otherwise it is dropped.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 28
  • The prompts distinguish episodic, user_profile, fact, agent_identity, agent_method, and agent_knowledge memories by what each records.These types cover concrete events, user attributes, objective external information, long-term agent collaboration requirements, reusable workflows, and reusable agent-side insights.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 28
  • Extraction must use information expressed by the target turn; later turns cannot backfill earlier memories, and historical context only resolves references.For example, a later confirmation must be attributed to its own turn rather than retroactively turning an earlier suspicion into a confirmed fact.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 29–30
  • The prompts require splitting independent facts, avoiding duplicate long-term and episodic memories, and recording each event as one complete episodic unit.Long-term types take precedence for the same information; episodic memories retain distinct event information, with multiple actions and outcomes from one event kept together.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 30
  • Non-episodic candidates must pass explicitness, correct-subject, future-value, and irreplaceability gates; common, externally retrievable, volatile, and weakly inferred information is generally excluded.The prompts also caution against treating jokes, hyperbole, sarcasm, or metaphor as literal long-term facts or profiles.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 30–31
  • The prompts distinguish episodic events, user-stated stable facts about external entities, and long-term user attributes, relationships, preferences, habits, goals, and constraints.Transient states and one-off actions do not become stable facts or profile entries; unsupported or non-restatable content should not be extracted.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 32
  • Agent-side memories separate long-term collaboration requirements, reusable agent-executed workflows, and cross-task judgment principles grounded in dialogue.User-mandated workflows belong to agent_identity, while assistant-formed reusable workflows belong to agent_method; agent_knowledge excludes public knowledge and unsupported or one-task conclusions.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 32
  • The prompts prohibit unsupported inference, including treating assistant speculation, question premises, public knowledge, or unstated current status as long-term memory.Each candidate must be supported by its own turn, remain meaningful independently of other messages, and pass checks for type, role, fidelity, and future retrieval value.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 33–34
  • Memory text must be concise, faithful, complete, and independently understandable, resolving references while preserving conditions, uncertainty, and the distinction between plans and completed actions.Convertible relative dates must be anchored to the owning session and turn time; dates or status words must not be invented.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 33–34
  • Memory evolution should minimize deletion, remove entries only for explicit forgetting or when an add absorbs, supersedes, or corrects them, and preserve every still-valid sub-item when fusing memories.Captures already enter the pool, so sessions with no necessary fusion, correction, or deduplication normally produce no changes.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 35
  • Questions must meet per-tag and total quotas, use third-person phrasing, and focus on the user or assistant according to the memory data type.User-centric questions concern the user; agent-centric questions concern the assistant’s collaboration rules, procedures, lessons, and their formation.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 36
  • Every question must be hard, use at least two genuine difficulty techniques, depend on a personalized fact, and avoid simply restating one memory.Techniques include cross-session evidence, completeness, decoys, lexical gaps, temporal-state selection, constraint combination, qualifiers, coreference, and calculation.Source: E PROMPTS FOR DATA CONSTRUCTION · pp. 37–38
  • Recall must include the complete supporting set from current final memories, exclude outdated or irrelevant items, and span at least two sessions.Queries must provide retrieval cues for each recalled memory; each question has one core question, and usable evolution chains require a temporal-state question.Source: E PROMPTS FOR DATA CONSTRUCTION · p. 38

F DATA SAMPLES

The data samples illustrate how user facts and interaction episodes are stored alongside agent-specific guidance, memory updates, and recall-oriented QA annotations. Across varied conversation contexts, they show how user profiles and assistant-user working relationships develop through repeated interactions.

F DATA SAMPLES

  • The agent-centric record combines user and assistant profiles, dialogue, memory hints, captured memories, and retrieval-oriented annotations within a trajectory.The appendix describes this organization as part of its representative data records.Source: F DATA SAMPLES · pp. 38, 39, 42
  • The dialogue turns a user request into persistent assistant guidance: lead food-safety answers with a verdict and provide short, usable scripts.The user asks for answer-first allergy guidance and phrasing they can say aloud, and the assistant agrees to default to usable scripts for restaurant and travel-meal questions.Source: F DATA SAMPLES · pp. 40–42
  • The sample distinguishes user-side allergy memories from assistant-side behavior instructions, including seafood as the general warning and shellfish as the strongest reaction signal.The record labels the assistant's answer-first instruction as agent identity memory and the seafood and shellfish facts as user-profile memories.Source: F DATA SAMPLES · pp. 41–43
  • In the Berlin restaurant example, stored memories combine the user's seafood allergy and travel context with guidance to provide usable scripts.The sample also records the assistant's advice to obtain the organizer's answer about dinner and prepare a restaurant script for other meals.Source: F DATA SAMPLES · pp. 43–44
  • In the holiday-buffet exchange, the assistant advises treating the buffet as unsafe by default and gives the user a concise sentence for declining food.The sample's captured memories preserve the stricter-than-hotel-breakfast rule, the user's request, and the assistant's suggested wording.Source: F DATA SAMPLES · pp. 44, 46, 47
  • The later allergy-management sample records a request for menu or label verdicts in the first three words and a preference for a routine that feels boring.Its memories also describe repeatable filters, a failed canned-soup aisle, and the user's bag items being in place; the associated recall question asks about the buffet rule and cites relevant memories.Source: F DATA SAMPLES · pp. 45–48
  • The user-centric example introduces a Mumbai-based content operations associate seeking to clarify their professional identity and move toward campaign operations work.The profile describes marketing and content-operations responsibilities and work with colleagues.Source: F DATA SAMPLES · p. 48
  • The initialized assistant profile presents an external-brain career and work-communication assistant, with evidence-based recommendations and audience-adaptive writing as operating principles.Its listed capabilities include refining role descriptions, structuring KPI recaps, organizing messy notes, and avoiding inflated claims or false certainty.Source: F DATA SAMPLES · p. 49
  • The multiweek episode tracks the user’s shift toward campaign operations and the assistant’s evolving support for role descriptions, reporting, and learning priorities.The target narrowed from content strategy through content operations and marketing operations to campaign operations; the user shifted learning focus from SQL toward CRM and lead lifecycle concepts.Source: F DATA SAMPLES · pp. 49–50
  • After the user felt reduced to “the person who puts the blogs up,” the assistant helped frame their work around content operations, campaign process, QA, and reporting.The assistant recommended a short, calm description rather than a long explanation, and the user said the wording landed better.Source: F DATA SAMPLES · pp. 50–51
  • The sample’s captured memories distinguish conversational episodes from user-profile facts and an agent-identity instruction to lead with answers on work topics.Examples include the user’s job title and dislike of “associate,” alongside the assistant’s commitment to answer work questions first.Source: F DATA SAMPLES · pp. 52–53
  • After the user struggles with how to describe their work, the assistant recommends a concise sentence and encourages them not to let the standup incident define their job.The user tends to stay quiet or over-explain when put on the spot, and the assistant suggests taking the middle option: one sentence, then stop.Source: F DATA SAMPLES · p. 54
  • Across sessions, the user frames operational work as their current function and campaign operations as a future direction, with launch-readiness gaps as a concrete example.The assistant advises emphasizing the present function in current conversations rather than the future target.Source: F DATA SAMPLES · pp. 55, 56, 58
  • For a referral packet, the assistant drafts and then softens a professional summary covering Mumbai, content operations, launch readiness, and a campaign-operations goal.The user requests a short summary that sounds neither robotic nor like a dating profile, then asks for a less formal version.Source: F DATA SAMPLES · pp. 56, 58, 59
  • The sample separates user-profile facts, episodic events, and agent-identity instructions while tracking the user's work and career direction.It records launch-readiness experiences alongside the user's operational role, campaign-operations goal, and guidance for the assistant.Source: F DATA SAMPLES · pp. 57–58
  • Agent-side guidance preserves plain coworker versus cleaner director tone, the exact degree wording, and full-draft treatment for interview answers.The user specifies BA in Communications without embellishment and asks the assistant not to use the tiny mode for interview responses.Source: F DATA SAMPLES · pp. 56–59
  • A sample records the user's preference for making campaign work more trackable, an assistant-proposed interview line, and an instruction to provide full interview drafts.Source: F DATA SAMPLES · p. 59
  • The sample QA connects two proposed explanations through the shared idea of making messy campaign work clearer, more reliable, and trackable.Source: F DATA SAMPLES · p. 60

Part III | Method and Experimental Setup

G ADDITIONAL METHOD AND EVALUATION DETAILS

DyadMem formalizes session-level inputs, memory states, and isolated versus pipeline task conditions. Its dialogue-first construction process builds gold memory annotations chronologically and subjects episodes to consistency and evidence-chain checks.

G.1 FORMAL NOTATION AND TASK INPUTS

  • Capture, Update, and Recall use session dialogues, gold memory entries, and evolving user-side and URAM banks as their inputs.Gold entries distinguish Capture targets from Update additions and deletions; user-side and URAM states are scoped to the user and user–agent relationship, respectively.Source: §G.1 · p. 61
  • Capture sees only the current complete session, while standalone Update receives gold prior memory and Capture entries and Full-Pipeline Update uses model-built inputs.Capture has no access to earlier sessions.Source: §G.1 · p. 61

G.2 CONSTRUCTION AND QUALITY-CONTROL DETAILS

  • For each finalized dialogue, six gold Capture entry types are extracted session by session, then the episode is replayed chronologically to build its memory bank.Update operations preserve lineage to dialogue sources.Source: §G.2 · p. 61
  • QA construction adds similar memories, evidence spread across sessions, invalidated entries, and negation or exception distractors, while checks verify support validity and traceability.Nine annotators review evidence and consistency; automated checks flag anomalies but do not determine gold labels.Source: §G.2 · p. 62

G.3 EVALUATION AND SCORING DETAILS

The evaluation scores memory operations independently and distinguishes answer quality with gold support from end-to-end QA using retrieved model memories. Its matching and scoring rules specify how semantic coverage, exact indices, grounding, and failed outputs affect credit.

G.3 EVALUATION AND SCORING DETAILS

  • Update op-F1 is the macro-average of Add and Delete F1 and is scored only on sessions with at least one gold addition or deletion.No-change sessions are excluded from this score.Source: §G.3 · p. 62
  • Gold-Memory QA uses gold support without retrieval, whereas Full-Pipeline QA retrieves at most 16 entries from the model’s terminal bank; their accuracy difference is an end-to-end contrast, not a causal decomposition.Standalone inputs keep upstream Capture or Update errors from changing the instance scored.Source: §G.3 · p. 62
  • Capture and Update additions use bidirectional semantic coverage, while Recall and standalone Update deletions require exact memory-index matches.Semantic matching uses cosine thresholds and NLI adjudication for gray-zone pairs, with a one-to-many coverage guard.Source: §G.3 · p. 63
  • QA correctness is judged semantically, refusals receive zero when gold support is non-empty, and grounded accuracy additionally requires support from supplied memories.Grounded accuracy uses all questions as its denominator; ungrounded-correct rate measures unsupported positively scored answers.Source: §G.3 · p. 63
  • Truncated, unparseable, empty, or failed outputs remain in their metric denominators and receive no credit.Confidence intervals use 200 bootstrap resamples, with paired comparisons keeping variants aligned within the resampled unit.Source: §G.3 · p. 63

H ADDITIONAL EXPERIMENTAL DETAILS

The experiments evaluate 20 models spanning open-weight and proprietary systems, with additional subsets for agent-centric and simplified episodes. The appendix also specifies evaluation components and the unsafe-deletion metric’s scope.

H.1 MODELS, CONFIGURATIONS, AND AUXILIARY SUBSETS

  • DyadMem evaluation covers 16 open-weight models and 4 proprietary models.The listed systems span Qwen, GPT, Llama, DeepSeek, GLM, Kimi, MiniMax, Gemini, and Doubao families.Source: §H.1 · p. 63
  • The default QA judge is GPT-5-mini, with BGE-M3 embeddings and cosine and NLI thresholds specified for memory matching.Open-weight models run with vLLM on a multi-node A800 cluster.Source: §H.1 · pp. 63–64
  • Table 8 defines unsafe-deletion rate as incorrect deletes divided by all predicted deletes on the full 3,065-episode benchmark, with lower values better.The table caption gives the metric definition and benchmark scope.Source: Table 8 · p. 64
  • DyadMem-Agent contains 500 episodes and DyadMem-Simple contains 300, with their session and QA counts and subset designs specified separately.The controlled URAM comparison uses 1,735 questions whose gold support contains only URAM entries; DyadMem-Simple samples 12 episodes per subcategory and balances episode lengths and data types.Source: §H.1 · p. 64

H.3 ADDITIONAL MAIN-RESULT BREAKDOWNS

Additional breakdowns show that retrieval hits can be incomplete, Update quality does not reliably scale with model size, and Capture performance varies across memory types and domains. These results separate stage-level behavior from final QA outcomes.

H.3 ADDITIONAL MAIN-RESULT BREAKDOWNS

  • 88.8% of questions have a gold memory at rank one for Kimi-K3, and GLM-5.2 and DeepSeek-V4-Pro each reach 87.8%, yet Recall@n remains below 63% for all three.Across models, the self-constructed-to-gold bank-size ratio ranges from 0.08 to 0.51 and usually lies between 0.20 and 0.35.Source: §H.3 · p. 64
  • Qwen3.5-397B reaches 22.3% op-F1 with 83.8% unsafe deletion, while Llama-3.1-8B has the highest unsafe-deletion rate at 98.5% and GPT-OSS-120B the lowest at 62.6%.Standalone Update does not improve monotonically with parameter count.Source: §H.3 · p. 64
  • For Kimi-K3, Full-Pipeline QA is 71.9% for questions requiring two gold support entries and 75.7% for questions requiring seven; Capture F1 ranges from 16.4% for episodic to 51.9% for agent_identity.Interpersonal & Communication has the lowest domain Capture F1 at 22.7% and Full-Pipeline QA of 50.1%.Source: §H.3 · p. 64

H.4 CONTROLLED URAM PROTOCOL AND ADDITIONAL RESULTS

The controlled protocol compares URAM Update variants and evaluates their effects on both memory operations and downstream Full-Pipeline QA. Results vary by model, with some QA gains and some losses not explained by increased abstention.

H.4 CONTROLLED URAM PROTOCOL AND ADDITIONAL RESULTS

  • Controlled URAM Update uses each variant’s model-extracted prior bank and Capture output, scoring additions and deletions against the same gold entries.The evaluation restricts gold Add/Delete entries to agent_identity, agent_method, and agent_knowledge; Add F1 and Delete F1 are averaged into op-F1.Source: §H.4 · pp. 64–65
  • Figure 5 presents unsafe deletion rates for open-source and proprietary models.Source: Figure 5 · p. 65
  • Full-Pipeline QA compares variants on the same 1,735 URAM-only questions, with each variant building its own agent bank chronologically.User-side entries cannot directly support answers in this setting; the comparison measures the downstream effect of allowing user turns to inform URAM construction.Source: §H.4 · p. 65
  • Three models have negative QA point estimates, while DeepSeek-V4-Pro and Qwen3.5-9B improve; the negative shifts occur on questions already answered.Llama-3.1-8B, Qwen3-4B, and Qwen3-30B-A3B show negative estimates; their abstention rates also fall. DeepSeek-V4-Pro rises from 58.9% to 68.7% QA, and Qwen3.5-9B from 54.3% to 61.5%.Source: §H.4 · p. 65

H.5 RETRIEVAL TRACKS AND MODEL SELECTION

The retrieval analysis compares fixed retrieval tracks with each tested model’s own selection and examines their effects on QA. Fixed retrieval improves QA for four of five models, but cannot recover all losses associated with incomplete constructed banks.

H.5 RETRIEVAL TRACKS AND MODEL SELECTION

  • The comparison tests five models against dense ranking, three rerankers, and each model’s own selection, with rerankers rescoring the top 30 dense candidates.Retrieval retains the top 50 candidates, while answer generation receives at most 16 memories.Source: §H.5 · p. 65
  • Recall@n, MRR, and nDCG evaluate retrieval on the gold bank, whereas QA evaluates answers using each model’s constructed bank or a named retrieval track.Source: §H.5 · p. 65
  • Table 9 compares variants that differ in captured and evolved memory types; pipeline QA uses each variant’s terminal bank.Source: Table 9 · p. 66
  • Table 10 reports retrieval and QA results for five models across dense, reranker, and model-selection tracks.Bold and underlined values mark the best and second-best track within each model and metric.Source: Table 10 · p. 66
  • The best fixed retrieval track improves QA for four of five models, while Qwen3.8-27B performs best with its own selection; stronger retrieval recovers only part of the loss from incomplete constructed banks.Qwen3.8-27B leads the five models on Recall@n, MRR, and nDCG; Qwen3-4B and Qwen3-8B trail the strongest fixed track on Recall@n.Source: §H.5 · pp. 65–66

H.6 CROSS-JUDGE SCORES AND RANKS

  • Table 11 provides exact Full-Pipeline QA scores and ranks underlying Figure 3.Source: §H.6 · p. 66

H.7 ADDITIONAL ERROR-PATTERN RESULTS

The additional results examine error patterns, memory-type capability, and performance across domains. They also report Full-Pipeline QA and ranks under multiple judges.

H.7 ADDITIONAL ERROR-PATTERN RESULTS

  • Qwen3-4B combines 4.9% Recall precision with six or more returned memories on average, 16.2% over-abstention, and 27.4% ungrounded-correct.Across both QA-target partitions and all five everyday-life domains, the same Full-Pipeline reversal appears; after dropping abstentions, answered-question accuracy is 52.0% for Qwen3-4B and 66.9% for Qwen3.5-9B.Source: §H.7 · p. 66
  • Figure 6 visualizes three error categories, while Figure 7 compares capability across six memory types and 20 LLMs.Source: Figure 6 · p. 67, Figure 7 · p. 67
  • The cited passage states only that Figure 8 is a performance visualization on 25 domains, and gives no Capture F1 range or Full-Pipeline QA value.Source: Figure 8 · p. 68
  • Table 11 reports Full-Pipeline QA on DyadMem-Simple for 300 episodes and 5,935 QA instances under three judges, with ranks computed independently per judge.Source: Table 11 · p. 69

Part IV | Evaluation Protocol and Implementation

I EVALUATION METRICS

Capture, gold-conditioned standalone Update, and Recall use separate gold inputs, while Full-Pipeline QA uses the model-built memory bank (p14).

I EVALUATION METRICS

  • Capture, gold-conditioned standalone Update, and Recall are evaluated independently; Full-Pipeline QA uses the bank built by the model.Source: I EVALUATION METRICS · p. 70

I.1 SEMANTIC COVERAGE MATCHING

  • Coverage matching allows many-to-many entry matches, with cosine-similarity thresholds and a one-way entailment guard for predictions covering multiple gold entries.Source: §I.1 · p. 70

I.2 CAPTURE

  • Capture uses micro-averaged F1 over pooled coverage counts; a truncated or unparseable call yields no true positives and makes gold entries false negatives.Source: §I.2 · p. 70

I.3 UPDATE

  • Update reports Add and Delete F1 separately, averages them as op-F1, and defines unsafe deletion as the share of predicted deletions absent from the gold set.Full-Pipeline Update builds the bank consumed by later Recall and Full-Pipeline QA.Source: §I.3 · pp. 70–71

I.4 RECALL

  • Recall@n and Recall@(n+4) average questions equally, while Recall@All pools support counts regardless of rank; empty or unparseable returns count as misses.Source: §I.4 · p. 71

I.5 QUESTION ANSWERING

  • QA paths share a judge but differ in supplied memories; the Gold-to-Pipeline gap is an end-to-end contrast, not an attribution to a specific stage.The judge checks answer consistency, and grounded accuracy also requires support from the supplied memories.Source: §I.5 · pp. 71–72

I.6 CONTROLLED URAM COMPARISON

  • The controlled URAM comparison averages per-episode Add/Delete op-F1 differences and compares Full-Pipeline QA on questions supported only by agent-side memory types.Source: §I.6 · p. 72

I.7 UNCERTAINTY AND AGREEMENT WITH HUMAN LABELS

  • Stage-level 95% intervals use 200 bootstrap resamples; human calibration defines agreement and Cohen’s κ from paired labels, with the human-consistent label treated as positive.Source: §I.7 · p. 72

J EVALUATION CODE

This appendix presents the evaluation metrics and framework prompts, alongside qualitative cases examining how agents handle evolving user-conditioned guidance. The cases probe timeline confusion and interference from adjacent rules, including whether agents give concise, realistic advice and accurately retrieve a rule’s content, position, or rationale.

J EVALUATION CODE

  • The evaluation code uses micro-aggregated Capture F1, recall@n+0 over positive queries, macro-averaged add/delete Update F1, and end-to-end QA accuracy from self-generated memories.Capture uses semantic matching and counts parse errors or truncated outputs as false negatives; Update scores adds semantically and deletes by exact memory ID.Source: J EVALUATION CODE · pp. 73–74

K PROMPTS FOR FRAMEWORK EVALUATION

  • URAM and Agent-only extraction read the same complete session and omit episodic entries; URAM permits user-conditioned memories, while Agent-only extraction limits memories to reusable assistant-side types.Both prompts require explicit, durable, evidence-grounded memories and allow an empty result when the dialogue does not support one.Source: K PROMPTS FOR FRAMEWORK EVALUATION · p. 75
  • The update prompt allows additions only from captured memories and deletions only for explicitly superseded, corrected, contradicted, absorbed, or forgotten memories, favoring minimal edits.It preserves valid conditions and exceptions, prohibits unsupported fabrication, and treats no-op updates as valid when evidence does not justify a change.Source: K PROMPTS FOR FRAMEWORK EVALUATION · p. 76

L QUALITATIVE MODEL RESULTS

  • The qualitative appendix selects ten QA error cases spanning six error types, balancing coverage across five agent-centric and five user-centric examples.The timeline-confusion example is one representative case in this set.Source: L QUALITATIVE MODEL RESULTS · p. 77

L.1 TYPE I: TIMELINE CONFUSION

  • The timeline-confusion case requires combining damage prevention over perfection, compact actionable advice, and a workable real meal; several answers judged wrong omit the real-meal objective.The query combines an earlier family-meal framing with later out-of-home compression.Source: §L.1 · pp. 77–79
  • For social eating, advice should prioritize realistic damage prevention over perfectionism and remain short and actionable, preserving the real-meal objective.Examples include protein first, modest starch, and water, with guidance adapted to the user already being out.Source: §L.1 · p. 80

L.2 TYPE II: ADJACENT RULE LEAKAGE

  • The check-in format requires the problem type on the fourth line, with the exact fallback “problem type: none” when nothing is wrong.Several wrong responses supplied the fallback phrase but missed the fourth-line constraint.Source: §L.2 · pp. 82–83
  • The user updates their meeting log after meetings, avoiding the loss of the room caused by live typing and the Notion log’s clutter from repeatedly revised half-decisions.The case distinguishes the facilitation failure from nearby process-related explanations for postponing updates.Source: §L.2 · pp. 85, 87

L.3 TYPE III: INCOMPLETE OR OVER-INCLUSIVE SET

This section examines memory’s ability to preserve distinctions across user-side supplement behavior and agent-side leadership communication, including which products and information belong to which audience. Its examples also test whether memory connects concrete job conditions to planning rules while retaining rejected calendar-based assumptions alongside human spillover.

L.3 TYPE III: INCOMPLETE OR OVER-INCLUSIVE SET

  • The supplement case expects breakfast as the anchor and vitamin D plus creatine as the dependent products; substituting whey or missing creatine fails the set-boundary test.The reference identifies breakfast, vitamin D, and creatine; a highlighted response instead selects vitamin D and plain whey and is marked wrong for missing creatine.Source: §L.3 · pp. 89–90
  • Evidence that the separation worked includes the user saying the rule helped when explaining Atlas 3.4 status upward and Dana replying to the leadership version in about thirty seconds.A highlighted response gives different evidence—the team write-up landing better than expected—and is marked wrong for omitting the audience-separation rule and Dana’s fast reply.Source: §L.3 · pp. 93, 94, 97
  • Leadership updates should keep audiences separate and focus on decision, impact, and ask, including only risks or rollback details that affect leadership’s decision or risk posture.Scope, ownership, next steps, and other operational details belong in team-facing updates unless they change leadership’s decision or risk posture.Source: §L.3 · pp. 94, 97
  • Leadership updates should focus on decision, impact, and ask, include only decision-relevant risk, and reserve rollback details for cases affecting leadership’s decision or risk posture.The example treats Dana’s reply in about thirty seconds, together with the user’s report that the separate-audience rule helped, as evidence that the separation worked.Source: §L.3 · p. 98

L.4 TYPE IV: EXAMPLE-TO-RULE BRIDGING FAILURE

  • The bridging case requires linking job conditions to rejected planning assumptions, including that the calendar alone cannot reliably indicate workload, through the phrase “human spillover.”Several example answers omit the calendar-based assumption and are marked wrong, while the full answer connects it to human spillover.Source: §L.4 · pp. 99–100

L.5 TYPE V: FINE-GRAINED ENTITY SUBSTITUTION

This section tests whether fine-grained entity substitutions preserve narrow rules and distinctions across hydration, workout planning, and relationship routines. It also captures how Friday time with Maya feels more like ordinary shared time as strain around Mum eases, and how conversation with Maya varies by setting.

Hydration rules

  • An electrolyte packet is allowed only after a long field block or comparable sweat-heavy stretch, not as a casual drink on ordinary desk-heavy days.Answers that imposed the field-block rule but omitted the comparable sweat-heavy condition were marked wrong.Source: §L.5 · pp. 103–104

Workout planning

  • Workout obligations are Tuesday and Friday gym sessions, with Wednesday limited to brief light mobility; weekend walks and one-off activities remain optional, not compensation or replacements.Some answers incorrectly promoted Thursday into the core plan.Source: §L.5 · pp. 106, 109, 110

Relationship routines

  • As strain around Mum eased, Friday cooking and home conversation became lighter and more self-standing, rather than serving as emergency repair or carrying practical-talk pressure.Friday pasta without administrative talk became easier again as shared time.Source: §L.5 · p. 111

L.5 TYPE V: FINE-GRAINED ENTITY SUBSTITUTION

  • Friday cooking with Maya and Friday shared time over pasta with no admin talk became easier, feeling lighter and more like themselves than emergency repair.The easing of strain around Mum is given as the context for this change.Source: §L.5 · pp. 111–113
  • The user’s week feels less clenched, with more margin before defensive over-talk as ambient strain eases.Source: §L.5 · p. 113
  • Conversation with Maya is described as easier when the user’s hands are busy cooking, while walking also loosens conversation compared with a pressured dinner-table exchange.Source: §L.5 · pp. 113–114

L.6 TYPE VI: MIXED ERROR

The mixed-error case tests whether answers combine purpose-based channel choice with the user’s broader meeting policy; the appendix also lists benchmark model metadata and license information.

L.6 TYPE VI: MIXED ERROR

  • Correct answers test the sync’s purpose, route factual confirmations and simple choices to email, reserve calls or walks for live negotiation or site-dependent decisions, and preserve the broader meeting policy.The examples identify omissions of that broader policy as errors.Source: §L.6 · p. 115

M MODEL CARDS

  • Table 12 lists the release time and model source of LLMs used in the benchmark.Source: Table 12 · p. 120

N LICENSING

  • Table 14 summarizes the licenses of models evaluated in the benchmark.Source: Table 14 · p. 122

Related papers

Loading 2610.03020v1…