PaperlayerView exact revision on arXiv

Source-linked AI summary

RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

Arman Behnam, Sunglyoung Kim, Liangwei Yang

arXiv:2610.01780v1cs.AI

TL;DR

Long-term companion systems must infer who a person is and decide when their past matters, but generated benchmarks often prescribe traits and memory needs in advance. RealCompanion releases ten real relationships with evidence-linked records and staged reasoning traces; its results show that memory is rarely needed, difficult to locate when needed, and not reliably selected by the evaluated systems.

  • Problem

    Generated conversations and declared personas can test recall against predetermined facts, but do not establish when past messages matter or how a person’s understanding changes over time.

    Source: §1 · p. 1

  • Method

    RealCompanion releases ten consented companion relationships with conversations and four derived files, including profiles, personas, chat ground truth, and questions, with chat labels traced stage by stage to source messages.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Memory is rarely needed and often distant; recency finds a required message for 0.959 of chat probes overall but only 0.022 of distant items, while no evaluated detector distinguishes when memory is needed on real messages.

    Source: §6 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    The evaluated systems do not reliably distinguish real messages that need memory from those that do not, despite strong pooled recency retrieval.

    Source: §6 · p. 8, §7 · p. 9

    The benchmark covers ten relationships with one companion application, and evaluates chat decisions on participants’ actual messages.

    Source: §5 · p. 6

Abstract

from arXiv · show

A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the past bears on the message in front of it. Testing this requires a real person's record, and such records are private, so benchmarks generate the person and the questions and settle in advance what matters. We release \bench, ten real relationships with an AI companion: 27,218 messages over up to 120 days, released as the conversation and four files derived from it, a profile, a persona, a chat ground truth and a question set, each citing the messages it rests on. Every chat label carries the reasoning trace that produced it, checked stage by stage against the conversation. Three findings follow. First, the past is rarely needed and far away. Pooled measures mislead: a recency window finds the required message for 95.9\% of probes and 2.2\% of those that need memory, and at the natural rate 96\% of the gain from supplying recorded evidence comes from messages that need none. Second, no detector we tried can tell when memory is needed on real messages, authored questions over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. Third, three agent systems reconstruct the persona with the same F1 at a 31-fold difference in cost.

1 INTRODUCTION

Understanding a person over a long relationship requires both a stable account of who they are and a changing judgment about which past messages matter now. RealCompanion releases real conversations with derived records that make both questions checkable against what participants said.

1 INTRODUCTION

  • Generated-dialog benchmarks often make memory necessary, declare persona traits in advance, and leave established facts unchanged over time.These design choices measure recall, but do not establish when memory is warranted or how a person’s understanding changes.Source: §1 · p. 1
  • The benchmark addresses when past messages matter, how a person can be understood across messages, how to use that understanding in replies, and how to check the result.A reply may miss distant context, repeat stated facts, or rely on an incorrect inferred trait.Source: §1 · pp. 1–2
  • RealCompanion releases ten consented relationships with an AI companion, including the conversation and four derived files, with claims tied to their source messages.The files support reconstruction, chat, and written-question evaluation tracks.Source: §1 · p. 2
  • The profile records stated claims, the persona records supported readings of how a person thinks and decides, and the question set covers claims, facts, and traits across 3,312 questions.Each chat ground-truth item also carries a trace of what the person referred to, where it is recorded, and whether the supporting messages justify the reply.Source: §1 · p. 2

2 BACKGROUND

Prior work has developed conversational-memory benchmarks, companion evaluations, and datasets based on real preferences or behavior. RealCompanion positions its contribution as real longitudinal conversations with unmarked memory needs and evidence-linked judgments.

2 BACKGROUND

  • Many memory benchmarks ask questions over constructed histories, so the question itself signals what to retrieve rather than testing whether retrieval is warranted.Selection-focused benchmarks report difficulty identifying which prior context a request depends on and tracking preference changes.Source: §2 · p. 3
  • Companion studies examine long-term interaction, but the cited evaluations either use short sessions or rely on participants’ reports after the interaction.The passage describes CompanionBench as using de-identified real conversations while noting that 20 turns cannot reach the depth of longer disclosure.Source: §2 · p. 3
  • RealCompanion distinguishes itself by using real companion conversations without predeclared traits or marked memory needs, and by linking each judgment to its supporting messages.Related efforts include real chat sequences, human preference data, and non-conversational behavioral traces; other benchmarks score agreement with written person descriptions.Source: §2 · pp. 3–4
  • The paper defines an evaluation item as a probe paired with the messages it relies on, and distinguishes chat replies from questions about a participant.Chat items have a tier describing the context their reply may use; question items have a category describing the fact requested.Source: §2 · p. 4

3 THE DATASET

The dataset captures anonymized, naturally occurring relationships with an AI companion and organizes each participant’s history into records of stated facts, inferred persona, and evaluated replies. Its claims and labels are linked back to the conversation.

3 THE DATASET

  • The corpus records ten relationships from first to last message, with anonymized and rewritten messages that preserve what participants said, its context, and their interaction style.The two largest relationships contain 73% of messages, while the other eight represent lighter use and provide less history per person.Source: §3 · p. 4
  • The profile stores supported stated claims with evidence, confidence, and temporal scope, while the persona records supported readings of how participants think, feel, and decide.Persona entries can include named working hypotheses with their evidence and stated strength; claims in both files resolve to the conversation.Source: §3 · p. 4
  • The question track tests facts across sixteen categories, while the chat track pairs verbatim user messages with reference replies grounded in recorded context.Chat labels include reasoning traces whose stages check against the conversation, so a label does not serve as its own warrant.Source: §3 · p. 5

4 REASONING OVER CONVERSATIONS TO CONSTRUCT THE GROUND TRUTH

The chat ground truth pairs observed probes with context, reference replies, and reasoning traces. Its construction uses stratified probes and staged checks so that evidence, labels, and replies can be inspected against the source conversation.

4 REASONING OVER CONVERSATIONS TO CONSTRUCT THE GROUND TRUTH

  • The derivation records each decision for review, and the ground truth is checked against source messages so that a reader can inspect how each label was produced.The staged process guards against invented evidence, unsupported replies, and errors that make memory appear more necessary.Source: §4 · p. 5, §4.2 · pp. 5–6

4.1 PROBES AND ITEMS

  • Each chat item pairs a probe with permitted context, a reference reply, and a trace; context composition determines its tier, which can be recomputed from the listed messages and claims.Of 1,533 scoreable items, 404 point outside the current thread and 167 of those have reference replies that use the cited material.Source: §4.1 · p. 5
  • The proportional stratum estimates past need from 1,169 scoreable items; the enriched stratum tests systems when the past is needed, and 434 session openers test abstention when nothing prior is needed.Source: §4.1 · p. 5

4.2 TRACED REASONING

  • Five stages resolve a probe’s referent, retrieve and verify records, assign a tier and category, write an evidence-grounded reply, then verify the reply and category.Each stage reads only earlier decisions and records its decision and reason together, forming the item’s reasoning trace.Source: §4.2 · p. 5, Figure 2 · p. 6
  • The final tier differs from the pre-verification signal on 685 of 1,533 items, showing that a surface guess about context needs can be wrong on nearly half the items.The derivation also proposed 806 distant referents for unused messages and verified 422; it kept 752 of 1,544 profile-claim matches after checking their evidence.Source: §4.2 · p. 6

5 BENCHMARK SETUP

The benchmark separates reconstructing who a participant is from deciding whether and how past messages matter to a current message. It evaluates these tasks across three tracks with retrieval, context, response, and reconstruction procedures.

5 BENCHMARK SETUP

  • Three tracks test persona reconstruction from full histories and, for chat and questions, memory selection and its use in replies.The chat reasoning trace supplies reference sets, memory-need labels, tiers, and reference replies; Table 3 summarizes the tracks.Source: §5 · p. 6, Table 3 · p. 7
  • The benchmark covers ten relationships with one companion app, spanning 36–120 days, and scores restraint, retrieval, and application on real participant messages.Empathy, future-behavior prediction, and personal values are out of scope.Source: §5 · pp. 6–7
  • Retrieval baselines range from random and recency windows to BM25 and an oracle, with controls that permute positions or supply known non-gold messages.Methods run on one participant’s history at a time; Table 4 reports Hit@5 on distant items.Source: §5 · p. 7, Table 4 · p. 8
  • A single generator compares five contexts, including recent turns, recorded required messages, BM25 selections, and recency selections.Matched BM25 and recency conditions hold budget, heading, and order fixed; Table 5 reports content match by context condition.Source: §5 · p. 7, Table 5 · p. 8
  • Retrieval uses top-k hit rate and MRR, while responses are scored for content match, restraint, and evidence coverage.Memory-need detection is also evaluated using abstention and AUROC; rates and comparisons are reported with participant-level and stratum distinctions.Source: §5 · p. 7
  • Three agent systems reconstruct each participant’s persona in three runs, with fields scored by type and F1 averaged across runs.Agreement between repeated runs measures consistency within each system.Source: §5 · p. 7

6 RESULTS

Memory is rarely required in the natural-rate chat population, and the required messages can be far back, making pooled retrieval and context results misleading. Detectors struggle to identify when memory is needed on real messages, framing shifts memory use, and persona reconstruction scores are similar across systems despite large differences in token use.

6 RESULTS

  • 3.4% of natural-rate probes carry verified memory demand under the recorded reading and 1.3% under the strict reading.These rates use the proportional stratum; the 364 enriched probes are a census for conditional findings, not population rates, and demand varies across relationships.Source: §6 · p. 8
  • The furthest required message is a median 2,157 messages back, and even the nearest required message is a median 450 messages back.The furthest-message upper quartile is 5,114, with a maximum of 12,542.Source: §6 · p. 8
  • Recency finds a required message for 0.959 of all reference-set probes but only 0.022 of distant items, while lexical matching reaches 0.248 on distant items.On distant items, recency falls to zero when on-screen referents are removed; pooled retrieval reflects corpus composition.Source: §6 · p. 8
  • Replacing recent turns with recorded messages raises content match by +0.140 overall, but 96% of the proportional-stratum gain lands on probes needing no memory.With budget, heading, and order matched, ten BM25-selected messages score 0.189 below ten preceding messages.Source: §6 · pp. 8–9
  • All three reconstruction systems reach F1 0.69–0.70 despite about a thirtyfold difference in processed tokens.Table 6 reports persona reconstruction pooled over ten participants; systems agree with their own runs at 0.93–0.95.Source: Table 6 · p. 9, §6 · p. 9
  • A detector reaches AUROC 0.530 against probes whose referent is absent and 0.548 against cold opens on the proportional stratum.On its own target it reaches 0.601, catching 6 of 40 memory-needed probes while raising 40 false alarms.Source: §6 · p. 9
  • Labeling the same ten messages as retrieved memories raises memory use by 10–14 points, including on probes that need no memory.Without context, misfires are zero across cold opens and probes with no verified locus.Source: §6 · p. 9

7 CONCLUSION

RealCompanion evaluates understanding across real companion histories by measuring when the past is needed and how a person's persona is reconstructed. Its findings show memory demand is rare and poorly detected, while persona F1 is matched across systems at sharply different costs.

7 CONCLUSION

  • In RealCompanion, 3.4% of sampled messages reach outside the current thread, a median of 2,157 messages back; no tested detector distinguishes the two situations on real messages.Where past evidence is needed, supplying recorded evidence raises content match by 0.455.Source: §7 · p. 9
  • Three agent systems reconstruct the persona with the same F1 at a 31-fold difference in cost, adding about seventy to eighty fields per hundred recovered.The demand for memory concentrates in the longest relationships.Source: §7 · p. 9

ETHICS STATEMENT

  • Anonymization does not prevent authorship matching: a stylometric classifier reaches 83.0% and a language model 98.0%, against 10% chance.The release includes special-category data with participant consent; the approving ethics committee was not independent of the operator, and no IRB was involved.Source: ETHICS STATEMENT · p. 10

A THE EVALUATION TASK

  • A subject record pairs a time-ordered message stream with stores derived from it, including profile claims and the messages themselves.The formal task is defined over a person's record regardless of how that record was obtained.Source: §A.1 · p. 18, A THE EVALUATION TASK · p. 18

A.2 SCORING AND AGGREGATION

  • The task evaluates restraint, store locus when retrieval is required, and response quality, with each decision defined on the output of the preceding one.Response consistency is judged across all scoreable probes; attribution and locus are scored only where retrieval is required.Source: §A.1 · p. 18, §A.2 · p. 19

A.1 SUBJECT RECORDS AND PROBES

  • The demand rate is an instance property; when it is small, retrieving on every probe creates restraint errors on a 1 − π fraction of inputs.Authored probes that plant a needed fact make restraint vacuous, and response scores alone do not register unnecessary retrieval.Source: §A.1 · p. 18
  • Chat probes are user messages evaluated against their preceding history, while question-track probes are authored questions evaluated over the full stream.Only the proportional chat stratum supports estimates of how often memory is needed, since its turns are sampled without regard to demand.Source: §A.1 · pp. 18–19

A.2 SCORING AND AGGREGATION

  • Pooled and subject-macro averages answer different questions, and reported rates depend on the probe stratum and scored population.Participant-resampled paired bootstrap intervals reflect the subject-level unit, though intervals are wide with ten subjects.Source: §A.2 · p. 20

A.3 MEASUREMENT UNDER A LOW DEMAND RATE

At a low memory-demand rate, pooled retrieval gains largely reflect benefits on probes that need no memory, while the tested detectors do not reliably identify when memory is needed. The appendix specifies validity conditions for interpreting these estimates and situates the benchmark against prior work that substitutes authored personas and questions for real interactions.

A.3 MEASUREMENT UNDER A LOW DEMAND RATE

  • When false positives are negligible, the labeled memory-bearing group is mostly genuine, but missed referents can make the measured demand rate a lower bound on true demand.The census rules out misses from sampling because stage A ran over every user message, but not misses by stage A itself.Source: §A.3 · p. 20
  • The tried detectors were near chance on natural-rate probes: lexical AUC was 0.546, while the model reached 0.530 [0.488, 0.572].The sampler disagreed with verified tiers on 44.7% of rows; these tests do not establish that no feature map can detect demand.Source: §A.3 · pp. 20–21
  • A pooled response-score contrast ∆ does not identify the memory effect πγ1; identification requires per-probe demand labels and a population with 0 < π < 1.The paper notes that authored corpora with ρ⋆ ≡ 1 cannot identify restraint, regardless of additional data.Source: §A.3 · p. 21
  • 96% of the pooled +0.108 retrieval benefit comes from probes needing no memory, with γ0 = +0.107 and only +0.004 from the memory-bearing term at π = 0.034.The decomposition is reported for the proportional stratum; the first term is πγ1 and the second is (1 −π)γ0.Source: Table 8 · p. 21, §A.3 · p. 21

A.4 VALIDITY CONDITIONS FOR AN INSTANCE

  • Validity conditions require checkable evidence links, fixed reference stores, per-probe window judgments, and a stratum sampled independently of memory demand.Verification may demote a demand label but must reset all labels derived from refuted evidence; the appendix reports that this was repaired in the derivation pipeline.Source: §A.4 · pp. 21–23

B RELATED WORKS

  • The framework distinguishes authored persona descriptions, authored questions, and model judgments as substitutions for the person, the moment needing memory, and the target.Prior work on personas often supplies traits in advance, while conversational-memory benchmarks commonly pose a question at the end of a constructed history.Source: B RELATED WORKS · p. 23, §B.1 · pp. 23–24

B.1 THREE SUBSTITUTIONS

  • Prior work also studies preference updating and persona drift, but whether simulated-user failures hold at the same severity in natural interaction remains unresolved.ANCHOR reports 44.4% trajectory accuracy and near-chance recall of user state; its evaluation asks whether the companion stays itself.Source: §B.1 · p. 24

B.2 WHAT THE SUBSTITUTIONS COST

The substitutions used in prior personalization benchmarks obscure whether memory is needed, where its evidence lies, and how far back it reaches. The paper contrasts prior efforts with a real-conversation release whose evidence, labels, and derived files can be inspected, while documenting constraints on interpretation and release.

B.2 WHAT THE SUBSTITUTIONS COST

  • A question announces that retrieval is needed, preventing authored-probe benchmarks from measuring restraint; ordinary messages needing no memory cannot be authored as scored probes.Resolving a referent’s locus is also usually studied on questions, whereas this work labels ordinary messages.Source: §B.2 · pp. 24–25
  • 450 messages is the median reach to the nearest cited message outside the current thread, versus 2 messages within it; the respective maxima are 12,542 and 15.Outside-thread probes reach a median of 2,157 messages to their furthest cited message.Source: §B.2 · p. 25

B.3 ATTEMPTS TO REMOVE THEM

  • Prior efforts remove different substitutions: AlpsBench uses real assistant exchanges but synthesized queries, CompanionBench uses real companion chats but short sessions, and PRISM collects human preferences from single-session conversations.Behavior-trace benchmarks use real behavior but do not evaluate conversational memory.Source: §B.3 · p. 25

B.4 ADJACENT WORK

  • Memory architectures differ in whether they store retrieved messages or distilled statements, which determines whether exact identifier matching or judged recall can be evaluated.The paper evaluates three systems as deployed products so their write policies are part of what is measured.Source: §B.4 · p. 26
  • 44.7% of 1,533 scoreable items changed final tier from the pre-verification signal, making the stage-by-stage trace contestable in a way a post-hoc rationale is not.A reader can reject a trace at a named stage and identify other items decided by that stage.Source: §B.4 · p. 26

B.5 WHAT RELEASING THE ALTERNATIVE REQUIRES

  • The release provides interaction records but no participant account, and ten users of one product are not a population sample.Because a particular conversation cannot be regenerated if a release is wrong, consent constrains release before scale.Source: §B.5 · p. 26

B.6 THE COMPARISON TABLE

  • The comparison table includes benchmarks scoring responses conditioned on a particular person’s record and distinguishes absent tasks from tasks lacking a property.The table’s criteria and benchmark cells were checked against each benchmark’s own description.Source: §B.6 · p. 27, Table 10 · p. 27

C THE CORPUS

  • The release contains 27,218 messages across ten subjects, with one underived source file and four derived files whose citations resolve to source messages.The profile’s claims cite establishing messages, while 207 of 390 persona content slots are empty.Source: §C.1 · p. 27, §C.2 · p. 28

C.6 PROFILE AND PERSONA CONSTRUCTION

This appendix documents how profile and persona layers are constructed and how chat labels are derived, checked, and mechanically validated. It describes filtering and uneven persona completeness, and clarifies that mechanical checks establish internal consistency—not label correctness—while noting that two gates caught shipped defects and now run on every build.

C.6 PROFILE AND PERSONA CONSTRUCTION

  • 37.0% of 7,176 candidate profile claims were rejected for cause, and persona fields were sparsest when they required an explicit demographic category.The shipped profile has 3,580 claims, 3,579 with evidence; 147 of 160 demographic slots are empty, compared with 12 of 100 remaining descriptive fields.Source: §C.6 · pp. 31–32

C.7 REFERENCE RESOLUTION AND THE VERBATIM INVARIANT

  • Every released reference resolves to a source message, and all 1,533 scoreable chat probes match their recorded source text character for character.Resolution establishes that cited turns exist and probes are quoted exactly; it does not establish that a cited turn supports its claim.Source: §C.7 · pp. 32–33

D.1 THE FIVE PHASES

  • Five derivation phases alternate proposing, checking, rule-based tier assignment, reply grounding, and a final ship-or-withdraw decision.Phase B removes unverified references, phase C assigns tiers from the surviving context shape, and phase D grounds replies only in turns within the recent window, verified references, and probe.Source: §D.1 · pp. 33–34

D.2 PROBE SELECTION AND THE THREE STRATA

  • 44.6% of sampled rows with a comparable bucket were routed to a different verified tier, showing that the routing heuristic poorly predicted memory demand.The proportional stratum estimates traffic without conditioning on memory demand; the enriched stratum is memory-bearing by construction and is not used for rates.Source: Table 17 · p. 34, §D.2 · p. 34

D.3 TIER, CATEGORY AND LOCUS

  • 1,073 rows are basic, 223 intermediate, 181 hard, and 56 edge; verified locus maps thread to basic, profile to intermediate, and episode or companion to hard.Tier is recomputable from the context lists; locus is proposed in phase A and verified in phase B.Source: §D.3 · p. 35

D.4 WHETHER THE REFERENT WAS ON SCREEN

  • 3.4% of proportional-stratum probes were memory-bearing by the recorded reading, falling to 1.3% [0.8, 2.1] when clear on-screen cases were removed.The on-screen field distinguishes clear, borderline, and absent referents across 404 memory-bearing rows; its judgments were made by one person and do not cover probes never marked memory-bearing.Source: Table 21 · p. 36, §D.4 · pp. 36–37

D.5 WITHDRAWAL, DEMOTION AND REPAIR

  • 67 chat rows were withdrawn, 17 flagged rows were retained with defect markers, and all 26 release gates pass on the shipped corpus.The release preserves failures and repairs for inspection; mechanical gates check consistency but do not replace the independent correctness audit.Source: §D.5 · pp. 37–38, §D.7 · pp. 38–39

D.6 THE INDEPENDENT AUDIT

  • 7 of 1,533 reference replies disagreed with independent re-verification (0.46%), while stage A rationales failed to justify the recorded locus on 395 of 1,533 rows (25.8%).Hand reading found no defects among 119 unflagged rows, bounding the miss rate at roughly 2.5%; the audit also regenerated 19 golds, removed unsupported parts from 129, and withdrew 67 rows.Source: §D.6 · p. 38, Table 23 · p. 39

D.7 GATES AND MECHANICAL CHECKS

  • All five check families pass across 2,034 chat rows, with the category and signal families accounting for every row.The rows comprise 1,600 sampled rows and 434 abstention rows.Source: §D.7 · pp. 39–40
  • The structural family passes on the question track after repairs rebuilt multi_hop and knowledge_update and set a span threshold for long_range.Source: §D.7 · p. 40
  • Passing gates establishes internal consistency, not label correctness; gates T and U caught shipped defects and now run on every build.Source: §D.7 · p. 40

D.8 THE QUESTION TRACK

The question track contains authored probes over the same ten records, but its audit found substantial content defects and its probes do not support the paper’s rate claims. The appendix reports repairs, category agreement and revisions, and operational definitions for the evaluation protocol.

D.8 THE QUESTION TRACK

  • The 3,312 authored question items, 3,235 scoreable, support comparison with spoken queries over the same conversations, not the paper’s rate claims.The track was built separately and audited by a model that did not build it, using a pre-written instruction.Source: §D.8 · p. 40
  • 51.3% [49.6, 53.0] of delivered question items had a content defect, including unsupported answers, ill-formed questions, incorrect answerability flags, or non-evidential citations.The identified mechanism was late source re-narration that changed facts while earlier-authored items retained the previous versions.Source: §D.8 · p. 40
  • 12 points came from repairing wording against cited turns, and replacing items that survived both earlier repair stages bought ten more.Reclassifying categories had no effect on content defects; Table 26 reports the staged repairs on the same items.Source: Table 26 · p. 40, §D.8 · p. 40
  • Among items left untreated, 31.7% [29.7, 33.7] had a content defect, compared with 10.2% [7.4, 14.0] of replaced items and none of 50 re-anchored items.Wording-repaired items had a 30.3% defect rate; the overall residual was dominated by untreated items.Source: §D.8 · p. 40
  • 0.880 raw agreement and 0.866 Cohen’s κ measured classifier–auditor category agreement; category disagreement was excluded from the content-defect rate.Table 27 gives agreement by category and identifies the pairs readers confused.Source: Table 27 · p. 41, §D.8 · p. 41
  • Repairs made long_range carry a span threshold, while multi_hop items cross sessions and knowledge_update items cite more than one message.The delivered long_range label encoded a recipe rather than distance: 38 of 39 items cited a single turn, and all 22 released items met the revised span threshold.Source: §D.8 · p. 41
  • 0.480 [0.421, 0.536] Questiontrack BM25 recall@5 followed wording repair, up from 0.356; RECENCY, RANDOM, and ORACLE each moved by at most 0.005.The wording repair restored vocabulary from cited turns to the questions, so the delivered BM25 figure measured a delivery defect.Source: §D.8 · p. 41

E EVALUATION PROTOCOL

  • The evaluation-protocol appendix defines the context ablation, retrieval controls and baselines, response judge, quantity calculations, and models used for each role.Its operational definitions are taken from the code that produced the numbers.Source: E EVALUATION PROTOCOL · p. 41

E.1 THE CONTEXT ABLATION

The appendix tests how retrieval, context, and evaluation choices affect memory-dependent replies. It finds that required evidence can lie far beyond practical context windows, while retrieval controls distinguish position-based from content-based matching.

E.1 THE CONTEXT ABLATION

  • The five context conditions use one fixed generator, and the C2-versus-C1 comparison changes supplied context but also changes its heading, confounding the comparison with a label effect.A matched pair uses the same neutral heading to isolate that component.Source: §E.1 · pp. 41–42, Table 28 · p. 42

E.3 RETRIEVAL BASELINES

  • Retrieval baselines return 20 preceding messages; only BM25 reads probe content, while recency uses the immediately preceding turns and the oracle uses the item’s gold set.BM25 retrieves 40 turns, filters to positions before the probe, then keeps the first 20; values are macro-averaged over participants.Source: §E.3 · p. 43

E.4 THE RESPONSE JUDGE

  • On used-memory and restraint, the judge agrees with human adjudication on 90.6% of replies (κ = 0.804), while exact agreement on content match is 77.4% (κ = 0.545).Human and judge comparisons preserve the same condition ordering despite the judge’s harsher content-match ratings.Source: §E.4 · p. 43

E.5 HOW EACH QUANTITY IS COMPUTED

  • Because ablation systems return no retrieved set, restraint and appropriate application are inferred from whether replies volunteer past details; evidence coverage is a weak measure alone.A reply may repeat an evidence turn without using it, so coverage is reported alongside content match.Source: §E.5 · p. 43

F.1 ESTIMATING THE DEMAND RATE

  • Demand-rate estimates range from one to four percent, with the random-sample stratum providing the paper’s demand rate and the other procedures checking it.A census, a residual sweep, and independent referent adjudication use different denominators and failure modes; the strict adjudication removes 154 of 404 memory-bearing probes.Source: §F.1 · pp. 44–45, Table 32 · p. 45

E.2 RETRIEVAL CONTROLS

  • Retrieval controls replace each item’s gold set with targets that destroy position, content, or gold-set membership while holding the other properties fixed.The offset permutation preserves distances but assigns another probe’s gold offsets; the broken oracle must score zero.Source: §E.2 · p. 42, Table 29 · p. 42, §F.2 · p. 45

F.2 RETRIEVAL IN FULL

  • Recency scores 0.979 with actual gold turns and 0.982 after offset permutation, while lexical matching falls from 0.282 to 0.216, showing recency reads position and lexical matching reads content.Randomizing positions takes recency to 0.104; the broken oracle scores zero for every method.Source: §F.2 · p. 45

F.3 CONTEXT-WINDOW COVERAGE

  • A 128k-token window covers 45.3% of memory-bearing probes and 200k covers 56.7%; full coverage arrives only near a million tokens.On the turn axis, the last thousand turns cover 30.9%, and ten thousand are needed to reach 95%.Source: §F.3 · p. 46, Table 34 · p. 46, Figure 4 · p. 46

F.4 THE ABLATION IN FULL

The full ablation shows that condition rankings change on memory-bearing probes and that prompt labels shift memory use. Detection remains poor on real conversational probes, despite stronger results on authored questions.

F.4 THE ABLATION IN FULL

  • The ten-turn window ties the three-turn window overall but falls below it on memory-bearing rows, while exact context’s advantage survives across every population.Removing probes whose referent was already on screen raises the exact-context gain from 0.437 to 0.541 on the hard tier and from 0.549 to 0.650 on hard and dependent rows.Source: §F.4 · pp. 46–47, Table 35 · p. 47, Table 36 · p. 47

F.5 SELECTIVITY

  • Under matched neutral labels, retrieval misapplies context on 50.8% [42.0, 59.6] of probes needing none, versus 59.2% [50.2, 67.5] for a recency window.Announcing context as retrieved memories raises memory use by 10–14 points on probes needing none, memory-bearing probes, and cold opens.Source: §F.5 · pp. 47–48, Table 37 · p. 48
  • On real conversational probes, the detector is at chance, flags only 6 of 40 probes needing memory, and raises 40 false alarms; authored questions over the same histories yield BM25 0.696 [0.629, 0.758].The model detector reaches 0.530 [0.488, 0.572] on the proportional stratum; on questions, its threshold calls 40% of answerable items unanswerable.Source: §F.5 · p. 48

F.6 PER-SUBJECT VALUES

Per-subject results expose how unequal row counts can distort macro-averages, while deployed-system results and persona reconstruction provide separate measures of performance and agreement. Reconstruction scores depend on the released ground truth and its scoring rules.

F.6 PER-SUBJECT VALUES

  • Lexical matching’s subject macro is 0.424 while its pooled value is 0.248, a macro result no populated subject exhibits because subjects with few rows receive full weight.Under the strict reading, lexical matching is 0.397 macro versus 0.300 pooled, while recency is zero on every subject.Source: §F.6 · p. 48, Table 38 · p. 48
  • The content-match contrast C2 minus C1 is positive for all ten subjects, ranging from +0.071 to +0.286; on memory-bearing rows it is 0.328 macro and 0.233 pooled.The pooled memory-bearing contrast is the γ1 of record for Proposition 2.Source: §F.6 · p. 49, Table 40 · p. 49

F.7 DEPLOYED MEMORY SYSTEMS

  • The Codex port answers 44.5% of answerable questions correctly and 58.1% of chat probes; its recall at five is 0.331 on questions and 0.137 on chat.The other systems cover only subsets or different releases, so their rows should not be compared directly with current-release scores.Source: §F.7 · pp. 49–50, Table 41 · p. 49

G.1 TASK AND SCOPE

  • Reconstruction scores compare each system’s persona and profile fields with release-generated, hand-reviewed ground truth; the two longest histories contain 83% of transcript tokens but only 18% of persona fields and 39% of profile fields.The task scores all ten subjects in both layers.Source: §G.1 · p. 50

G.2 SYSTEMS AND SETTINGS

  • Claude, Codex, and Antigravity use different agent settings and history handling, with only Codex holding the whole history or U10 half in every call.Claude contexts near the window were auto-compacted, and Antigravity read the longest histories in pieces.Source: §G.2 · pp. 50–51

G.3 SCORING

  • Scoring counts a wrong value as both a false positive and a false negative, excludes fields both GT and a run leave empty, and computes F1 over non-empty cells.Big Five is a fixed set; quirks and entities are scored for recall only, while most other categories can incur false positives.Source: §G.3 · pp. 51–52

G.4 OVERALL AGREEMENT

  • All three systems reach persona F1 0.69–0.70, with recall 0.86–0.89 and precision 0.56–0.59; runs agree more with each other than with ground truth.Between-run agreement is 0.93–0.95 against 0.69–0.70 with GT; profile F1 is Claude 0.80, Codex 0.77, and Antigravity 0.78.Source: §G.4 · p. 52, Table 42 · p. 52

G.5 RESULTS BY CATEGORY •

Across persona and profile categories, the systems often reproduce ground-truth entries while adding entries GT does not contain. Their relative strengths vary by category.

G.5 RESULTS BY CATEGORY •

  • Each system matches nearly every GT personality dimension and prose section, but adds about ten to eleven persona dimensions, four to eight profile dimensions, and about two prose sections.Figure 5 distributes GT-filled fields by how many runs match them.Source: §G.5 · p. 52, Figure 5 · p. 53
  • All systems match 62–70% of persona Big Five traits and 64–74% of profile traits, and every miss is exactly one level off..Source: §G.5 · p. 52
  • Claude recovers the most entities and quirks, Codex the most identity fields and profile dimensions, and Antigravity the most fact categories.Antigravity is weakest on identity fields; all systems name 23–44% of GT entities.Source: §G.5 · p. 52

G.6 RESULTS BY SUBJECT AND HISTORY LENGTH

Persona and profile scores move differently with history length, largely because the ground truths change with history length rather than because systems read differently. At matched persona F1, Codex uses about 31× fewer tokens per run than Claude.

G.6 RESULTS BY SUBJECT AND HISTORY LENGTH

  • Persona F1 falls with history length (ρ −0.44 to −0.55, not significant), while profile F1 rises for Claude (ρ +0.83, p = 0.005) and Codex (ρ +0.65, p = 0.049).Antigravity’s profile trend is not significant (ρ +0.18, p = 0.632); both trends mainly reflect how ground truth was built.Source: Table 43 · p. 54, §G.6 · p. 54
  • U09 and U10 have the weakest persona scores for every system, but U09 ranks among Claude’s and Codex’s best profile subjects.After removing categories confounded by ground-truth construction, U09 and U10 still score 0.78–0.84 versus 0.88–0.90 for U01–U08, but this residual is suggestive only.Source: §G.6 · p. 54
  • Persona trends disappear or reverse when ground-truth cutoffs and category choices are adjusted, while profile precision rises as history-dependent fixed lists fill.The chain grounds more dimensions than the persona ground truth retains on long histories; profile ground truths also grow from 5–26 facts on U01–U05 to 227–797 on U07–U10.Source: §G.6 · p. 54

G.7 COST

  • At the same persona F1, Codex processes about 31× fewer tokens per run than Claude, and Antigravity about 15× fewer.Call counts are not comparable across systems; token totals include retries, cached input, and reasoning tokens.Source: §G.7 · p. 55

H DATASHEET

H.1 MOTIVATION

  • The benchmark was assembled to measure when and what to remember on conversations not written for that purpose.No existing corpus pairs real longitudinal conversation with a per-item record of whether memory was required.Source: §H.1 · p. 55

H.2 COMPOSITION

The release combines longitudinal companion conversations with derived records and traceable labels, enabling evaluation and analysis grounded in what people actually said. Its findings include a substantial contribution from messages that do not require memory and transcript-based observations unavailable from self-reports.

H.2 COMPOSITION

  • The composition includes 27,218 turns from ten subjects, item-level labels about retrieval and evidence, and resolved links from derived claims to source turns.The release reports 18,983 citations between derived layers and the conversation, with none failing to resolve.Source: §H.2 · pp. 55–56

H.3 COLLECTION; H.4 PREPROCESSING AND LABELING

  • Ten people used one deployed companion for 36–120 days; preprocessing rewrote wording, shifted dates, and removed shadow replies and game turns.The persona layer is not a validated psychological instrument and was neither administered to nor reviewed by the subjects.Source: §H.3 · p. 56, §H.4 · p. 56

H.5 USES; H.6 MAINTENANCE; I DISCUSSION

I.1 MODELING PEOPLE IN CONVERSATION

  • At the natural rate, probes needing nothing earlier contribute twenty-four times as much of the measured gain as probes that need memory.The passage attributes the difference to real people’s present-focused conversation and notes that authored questions are written to need something.Source: §I.1 · p. 57
  • Per-probe restraint labels and five-phase derivations support supervision of retrieval decisions and intermediate reasoning, beyond scoring final answers.The release has 1,129 negative and 404 positive restraint examples; its coverage curve finds the last 1,000 turns cover 31% of memory-bearing probes, while 95% coverage takes 10,000 turns.Source: §I.1 · p. 57

I.2 AFFECT, MENTAL STATE AND PREDICTION

  • The corpus tracks the same people across months, linking disclosures and profile claims to turns and the periods those claims held.This supplies longitudinal records against which claims about a person’s state at a given time can be checked.Source: §I.2 · p. 58

I.3 THE PSYCHOLOGY OF HUMAN–AI RELATIONSHIPS

  • Only 3.4% of participant messages depend on earlier messages; the companion initiates exchanges on 217 of 430 participant-days.The transcript also shows that 33 of 12,795 participant messages reproduce a tappable reply, while demographic slots are empty 91.9% of the time.Source: §I.3 · p. 58

I.4 CONSUMER RESEARCH AND MARKETING

  • The corpus offers real-person records for validating synthetic personas, but its terms permit research use only and forbid profiling or targeting.Source: §H.5 · p. 57, §I.4 · p. 59

I.5 LIMITATIONS

The reported findings have boundaries arising from the small, product-specific corpus, anonymization, and judgment procedures; some analyses also rest on only five subjects.

I.5 LIMITATIONS

  • Ten users of one chosen application are not a sample of companion users or people, and generalization to other products is unknown.Source: §I.5 · p. 59
  • Anonymization rewrote every turn, costing 30 chat rows; its effect on retrieval and reply scores was not measured.Shifted timestamps preserve intervals but remove calendar position and day-of-week information.Source: §I.5 · p. 59
  • One reader made the on-screen judgments, so they have no agreement rate; graded scores carry the weaker reported agreement, κ = 0.545.The question track’s content defect rate after repair is 3.8% [3.2, 4.6].Source: §I.5 · p. 59
  • Under the strict reading, memory-demand-conditioned quantities are macro-averages over five subjects, not ten.Source: §I.5 · p. 59

J NOTATION AND TERMINOLOGY

Loading 2610.01780v1…