Source-linked AI summary
Where the Evidence Lives: Auditing AI Companions' Self-Descriptions
Seiya Ikeda, Shin-nosuke Ishikawa
TL;DR
Companion agents make claims about remembering and understanding users, but experience ratings do not establish whether hidden memory mechanisms ran. This paper audits such claims against user judgements and implementation records; in a month-long deployment, positive ratings coexisted with unexecuted accumulation mechanisms. It argues that memory-bearing agents should report whether accumulation ran.
Problem
Companion agents’ self-descriptions and users’ experience ratings do not establish whether the mechanisms behind claims about memory and relationships ran.
Source: §1 · p. 1Method
The paper audits each self-description against users’ judgements and implementation records, classifying claims as supported, contradicted, or unresolved.
Source: §3.1 · p. 3Results
Two of three memory layers never executed accumulation; participants endorsed stylistic claims, withheld endorsement from relational change, and rated recall above midpoint.
Source: §1 · p. 1Takeaways & Limitations
Memory-bearing agents should report whether accumulation ran rather than leave it assumed.
Source: §7 · p. 10The deployment was an accidental ablation, not a test of the idea, and does not show that language models cannot form self-narratives.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Companion agents describe themselves: they remember, they understand their users, the relationship has changed them. We argue that such accounts, and the experience ratings that seem to confirm them, are checkable by users only where the evidence is theirs: in the agent's behavior, or in themselves. Where the evidence lives in the machinery, fluent self-description and moderately positive ratings do not establish that the mechanisms behind them ran. We demonstrate an audit procedure that sets an agent's self-description against its users' judgements and its implementation records, reporting each claim as supported, contradicted, or unresolved, and apply it to Lita, a proactive companion we built and deployed for a month with nine colleagues. Participants endorsed stylistic claims, withheld endorsement from relational ones, and rated memory at or above midpoint, while two of three memory layers had never executed their accumulation step. Memory-bearing agents should report what their self-descriptions cannot establish.
1 Introduction
Companion agents make first-person claims about memory and relationships, but users’ impressions and behavioral evaluation do not establish whether the underlying mechanisms ran. This paper audits those claims against user judgements and implementation records, finding that positive experience ratings coexisted with unexecuted accumulation mechanisms.
1 Introduction
- The audit sets an agent’s self-description against what users believe and what its mechanisms did, addressing the lack of a procedure that aligns these records.Lita ran for a month with nine colleagues; the checks included a discrepancy probe, implementation audit, and counterfactual replay.Source: §1 · p. 1
- Two memory layers never accumulated; participants endorsed stylistic claims, withheld endorsement from relational change, and rated recall above midpoint despite 73.6% of observations being discarded.The replay’s estimated accumulation varied with the redundancy criterion and could not establish whether repair would have been enough.Source: §1 · p. 1
- The authors argue that users can check claims where evidence lies in observable behavior or their own experience, but not hidden machinery dynamics.They frame this as an interpretation of the pattern, not a measured process.Source: §1 · p. 1
- The audit reports claims as supported, contradicted, or unresolved, and the paper applies it to one deployment of a proactive companion.The paper presents the procedure independently of the case and reports its limits.Source: §1 · p. 2
2 Related Work
Prior work studies companion relationships, agent memory, and audits of deployed systems, but does not check whether an agent’s own relationship claims are backed by its implementation. This paper connects user judgements with runtime records to address that gap.
2.1 Companion agents, their self-descriptions, and the relationship
- Companion-agent research measures relationships from the user’s side and self-narratives through resulting behavior, leaving system support for the agent’s own relationship claims out of scope.The literature spans proactive interaction, narrative identity, and relational agents; this paper returns agent claims to users and implementation for checking.Source: §2.1 · p. 2
2.2 Memory, and what users can see of it
- Memory systems are evaluated through their behavioral effects, while users who can inspect a store see its current contents, not what was discarded or reinforced.This paper audits whether specified consolidation executed and which memory claims users can check.Source: §2.2 · p. 2
2.3 Auditing and evaluating deployed LLM systems
- Existing auditing and transparency approaches do not report whether an agent’s claimed accumulation mechanisms ran during deployment; this paper audits runtime records to address that gap.The paper treats LLM-judge labels as a bounded operationalization and reports their calibration history.Source: §2.3 · p. 2
3 Method
The method combines an audit procedure with a month-long deployment of Lita and checks based on user judgements, implementation records, and behavioral logs. It specifies the study setting, measures, and analysis used to compare what the agent claimed with what its mechanisms did.
3.1 The audit procedure
- The audit takes an agent’s self-description, users’ judgements, write logs, and deployed source, then classifies each claim as supported, contradicted, or unresolved.The steps have limits; the procedure requires internal access, and step-3 metrics depend on the memory architecture.Source: §3.1 · pp. 2–3, Figure 1 · p. 4
3.2 Study design
- Lita was deployed for approximately one month in a single condition without a reactive control, combining an implementation audit, behavioral logs, and a post-study questionnaire.The three research questions draw on different combinations of these evidence sources.Source: §3.2 · p. 3
3.3 System
- Lita ran on a self-hosted, quantized open-weight model, with short-term and long-term memory, a user model, a self-narrative, and a proactive loop.The stores had different capacities and accumulation rules, and generations received selected entries from the user model and self-narrative.Source: §3.3 · p. 3
3.4 Participants and procedure
- Nine colleagues consented and could inspect or reset Lita’s current memory, but those commands did not expose its write history.Questionnaire responses were anonymous, so perception ratings could not be linked to individual logs.Source: §3.4 · p. 3
3.5 Measures
- The Narrative Discrepancy Assessment asked participants to rate seven trait statements against experience and select the most and least representative.The statements were distilled from Lita’s self-narrative; the analysis grouped them into aesthetic dispositions and relational or other-directed claims, with a second coder independently assigning categories.Source: §3.5 · pp. 3–4
3.6 Analysis
- The analysis reconstructed whether designed mechanisms executed, replayed the merge routine under alternative similarity tests, and checked the LLM judge against blind human coding.Questionnaire analyses were descriptive with n=9; the replay held the observed write stream, capacity, and confidence rule fixed while varying the similarity test.Source: §3.6 · pp. 4–5
4 Results
Participants endorsed Lita’s stylistic claims and rated recall above midpoint, but withheld endorsement from relational claims. The implementation audit found that two memory layers never accumulated, and replay estimates depended on the similarity criterion and its operationalization.
4.1.1 The NDA separated aesthetic from relational self-claims.
- Aesthetic self-claims drew 72% agreement and all nine top votes, while relational claims drew 22% agreement and eight of nine bottom votes.A second blind coder reproduced the grouping on all seven statements (κ=1.0), although NDA α=.519 and a total score would average across this split.Source: §4.1.1 · p. 5, Figure 2 · p. 6
4.2 Two of three memory layers never accumulated
- Two of three memory layers failed to accumulate: all 253 long-term facts persisted, but the user model and self-narrative did not accumulate observations.For the latter stores, confidence remained at 0.3 for every entry and no observation count exceeded one.Source: §4.2 · p. 5
4.2.1 What the failure did downstream.
- A fixed-capacity store with a nonfunctioning salience signal retained only 106 of 402 user-model observations, and retention was inversely related to engagement (Spearman ρ = −0.588, n = 10).The summary function also served the eight oldest surviving self-narrative entries rather than the most established ones.Source: §4.2.1 · pp. 5–6
4.3 Repairing the defect changes accumulation by a criterion-dependent amount
- Under the redundant criterion, 25.5% of writes merged versus 0.5% under strict matching; replay raised minimum confidence among prompted entries from 0.30 to 0.50.The replay held the observed write stream, capacity, and confidence rule fixed while varying the similarity test; it was not a simulation of a repaired deployment.Source: §4.3 · pp. 6–7, Figure 3 · p. 7
4.3.1 Human validation of the judge.
- Human redundancy judgments were unstable (inter-coder κ=0.02, 95% CI [−0.17, 0.21]); propagated merge medians exceeded the raw 25.5% under coder A (63.0%) and coder B (52.2%), but the consensus median was 37.5% (95% interval [22.0, 55.7]).Source: §4.3.1 · p. 7
4.4 Claims set against the record
- The audit distinguished NDA3–6, which were not audited, from claims audited but unresolved, and gave a reason for every unresolved outcome.Source: §4.4 · p. 8
4.5 Exposure and style
- Of 57,038 candidate thoughts, 441 were expressed; proactive utterances drew replies less often than reactive responses (22.4% versus 72.6%).Sound-related vocabulary appeared in 18.2% of proactive utterances versus 3.5% of participant messages, supporting the behavioral record behind an endorsed stylistic claim.Source: §4.5 · p. 8
5 Discussion
The audit separates claims supported by observable behavior from relational and memory claims whose evidence is harder for users to access. The discussion connects silent accumulation failures to design and reporting implications, while noting explanations the study cannot distinguish.
5.1 What the audit established, and where the evidence lives
- Recall was supported, but stylistic and relational claims had mixed outcomes: one stylistic claim was supported, while several highly rated claims remained unaudited or unresolved.Participants withheld endorsement from the claim of change; its designed consolidation did not run, while whether change occurred through other routes remained unresolved.Source: §5.1 · p. 8
- Users can judge claims grounded in transcripts or their own relationship, but memory experience reveals hits and misses rather than the retention dynamics behind them.The study argues this evidence asymmetry is a reading of the pattern, not a measured process; its anonymous design cannot link ratings to logs or establish how participants pictured memory.Source: §5.1 · p. 8
- Designers should make relational change inferable from behavior and report memory write, merge, confidence, and retention records, since store contents alone do not show accumulation dynamics.The proposed reporting follows from users’ inability to check whether accumulation happened; it does not depend on choosing among the study’s explanations.Source: §5.1 · p. 9
5.2 Why the mechanism could not have earned the claim
- Two accumulation failures shared a shape: writing worked, but the paths converting entries into retained knowledge did not, and neither failure was visible in outputs.In replay, the judge marked only 0.5% of self-narrative pairs redundant, while the trait-level criterion yielded 25.5%; the paper presents trait-based consolidation as a design hypothesis, not a proven fix.Source: §5.2 · p. 9
- A broken salience signal in the fixed-capacity user store was associated with lower retention among more engaged participants, motivating genuinely salience-weighted eviction.The reported correlation was ρ = −0.588 between engagement and retention.Source: §5.2 · p. 9
- The interpretation is bounded: anonymous ratings cannot be linked to logs, and evidence location is the authors’ preferred grouping rather than the only account of the split.Thin contact and differing relational wording remain alternative explanations.Source: §5.1 · p. 8, §5.2 · p. 9
5.3 Future work
- Future comparisons include a reactive control, a second deployment, and the same self-narrative prompt across multiple open-weight models.The current study used one model and had no reactive control; the proposed comparisons would address what proactivity contributes, reusability, and model-versus-prompt effects.Source: §5.3 · p. 10
6 Limitations
The evidence comes from a narrow deployment and cannot establish how results would change across models, prompts, or languages. Anonymous ratings also could not be linked to mechanism logs, while the replay estimates depend on uncertain judgement criteria.
Scope
- The study covered one deployment, one open-weight model, one persona, one prompt set, and Japanese; effects of changing models, prompts, or languages remain unknown.Register observations also cannot separate a model prior from a prompt effect.Source: §6 · p. 10
- The deployment was an accidental ablation, not a test of whether self-narratives can work when their mechanism runs.Replay repair changed accumulation by amounts dependent on the judgement criterion, so the deployment’s defect is both a finding and a confound.Source: §6 · p. 10
Perception and logs
- Anonymous consent prevented linking participants’ perceptions to their logs, so perception-to-mechanism relationships are convergent rather than causal.The authors note that anonymity and within-person linkage are difficult to achieve together, particularly at scale.Source: §6 · p. 10
Sample and exposure
- Nine author-colleague respondents had no control condition and brief exposure, leaving thin contact as an alternative explanation for low relational scores.Contact lasted about 32 days; seven of nine had sessions under five minutes, and the authors report descriptive statistics only.Source: §6 · p. 10
Replay judgement
- Replay estimates are sensitive to criterion wording and language, while the two human coders showed little agreement and supplied no stable ground truth.The propagated merge rate is a sensitivity estimate under selected human references and an error model; Monte Carlo intervals assume pairs sharing an entry are independent.Source: §6 · p. 10
Qualitative material
- The qualitative corpus comprised twelve free-text items and logs, with no interviews; one participant’s LLM-generated log analysis was reported as an event, not testimony.Source: §6 · p. 10
7 Conclusion
The paper concludes that self-descriptions should be audited against users’ judgements and implementation records. Its judge prompt distinguishes duplicate realizations from entries that add no new information about the agent.
7 Conclusion
- The audit applied to a month-long deployment found one memory layer accumulated while two did not, despite fluent output and memory ratings at or above midpoint.Replay altered accumulation depending on the judgement criterion, and could not settle whether the change would have been sufficient.Source: §7 · p. 10
A Judge Prompt
- The judge prompt asks whether a new self-observation should be stored as a duplicate, presenting two records one at a time.The prompt provides the records as A and B for evaluation before storage.Source: A Judge Prompt · p. 12
- Its strict judgement tests whether records describe the same specific realization, while its redundant judgement asks whether B adds anything about the AI beyond stored A.The redundant criterion compares what one can say about the AI, without abstracting so broadly that every everyday observation matches.Source: A Judge Prompt · p. 12
B Judge Criterion Calibration
The appendix reports how the redundancy criterion was calibrated and documents deployment and measurement settings. The calibration shows that merge rates depend on criterion phrasing and that human agreement with the adopted criterion was limited.
B Judge Criterion Calibration
- Among the fifteen highest-similarity pairs, the adopted criterion’s merge rate tracked embedding similarity, unlike the other phrasings, which were degenerate or over-corrected.The adopted phrasing yielded 40% at the top band, 6.7% at the median, and 0% at the bottom; the table compares three phrasings under the same judge model and temperature.Source: B Judge Criterion Calibration · p. 12, Table 5 · p. 12
- Human agreement with the adopted criterion was low: inter-coder κ was 0.02 for redundant and 0.08 for strict judgements.Consensus-versus-judge agreement was κ = 0.42 on the 58 pairs where coders agreed.Source: B Judge Criterion Calibration · p. 12
C.1 Generation settings
- Generation used temperature 0.6 for conversational and evaluative calls and 0.8 for thought generation, with a 2048-token completion limit.Source: §C.1 · p. 12
C.2 Persona revision history
- Poetic-expression prohibitions were strengthened during piloting and applied to response-generation prompts, not to self-narrative updates.The version history records successive revisions following observations of excessive poetic output.Source: §C.2 · p. 12
C.3 Brake rules
- Proactive candidates passed through a separate brake call with three allowed speaking reasons and five stop conditions; its instruction was conservative by design.The allowed triggers included a new topic, a callback, or a first approach after at least ten minutes of silence.Source: §C.3 · pp. 12–13
C.4 Participants
- Ten accounts appear in the logs, including the first author’s pilot account; nine recruited participants began later, and the questionnaire collected no identifiers.Participants and authors were coworkers, and participants were told Lita might initiate contact after roughly three minutes of silence.Source: §C.4 · p. 13
C.5 Questionnaire inventory
- The 70-item questionnaire combined established scales with author-written measures, including memory, proactivity, and seven corpus-grounded Narrative Discrepancy Assessment statements.The NDA statements were operationalizations rather than quotations from individual memory entries.Source: §C.5 · p. 13
C.6 Judge calls
- Of 16,836 possible pairs, 11,409 were within the same-chapter comparison range; 2,925 judgements returned and 17 failed API calls were excluded.The two coders’ sheets differed only in row order to reduce correlated order and fatigue effects.Source: §C.6 · p. 13
C.8 Reproducibility
The analysis pipeline regenerates every table and figure from the raw logs and questionnaire export, while the appendix collects secondary analyses summarized in Section 4.5.
C.8 Reproducibility
- A single command regenerates all tables and figures from raw logs and the questionnaire export, with each output table recording its generating script.The LLM judge is the only step outside the automated pipeline; it is run explicitly because it incurs API cost.Source: §C.8 · p. 13
D Secondary Analyses
- Appendix D collects questionnaire details, engagement and retention breakdowns, human-validation confusion matrices, and deployment analyses of proactive gating and register.Section 4.5 summarizes the two deployment analyses.Source: D Secondary Analyses · p. 13
D.1 Questionnaire item extremes and exploratory correlations
Questionnaire responses favored Lita’s stylistic traits and memory consistency, while relational closeness was low. Exploratory correlations linked curiosity to lower reported relational depth, but the authors treat that pattern as a hypothesis.
D.1 Questionnaire item extremes and exploratory correlations
- NDA1 received 80.6% agreement, while “we have things in common” received 18.5% and “felt that Lita understood me” 24.1%.Figure 4 identifies the six highest- and six lowest-rated items among 41 questionnaire items.Source: §D.1 · p. 13, Figure 4 · p. 14
- Six of nine placed the relationship at “someone I don’t really know yet”; one described it as close to a friend.Participants’ single-word descriptions ranged from “a passer-by” and “a drifting ghost” to “a pet” and “an acquaintance.”Source: §D.1 · p. 13
- “I wondered what Lita was thinking” correlated negatively with relational depth (ρ = −0.691), but the authors treat this as a hypothesis given n = 9 and multiple correlations examined.The item’s corrected item-total correlation was −0.325, with a confidence interval crossing zero.Source: §D.1 · p. 14
- User-model retention was inversely related to engagement: participants who talked more lost a larger share of observations.Table 7 reports engagement strata computed from logs alone and defines retention as the fraction of user-model observations surviving to study end.Source: §D.2 · p. 14, Table 7 · p. 14
- Appendix D provides confusion matrices for the human validation of Section 4.3.1 and a per-trigger funnel of generated, expressed, and answered proactive turns.The funnel counts replies within 60 minutes.Source: §D.3 · p. 14, Table 8 · p. 14
D.4 The gate suppressed almost everything, and the memory pathway most
The proactive gate suppressed most candidates, and memory recall was especially unlikely to reach users. Proactive utterances also drew fewer replies than reactive responses.
D.4 The gate suppressed almost everything, and the memory pathway most
- The “same theme as the last five AI turns” brake fired on 35,690 evaluations, or 62.6% of all evaluations.The appendix reports the full utterance-count distinctions: 57,038 candidate thoughts, 441 expressed through proactive triggers, and 428 proactive interventions in conversation logs.Source: §D.4 · p. 14
- Across the three proactive triggers, one candidate thought in 612 reached a reply: 56,289 were generated, 441 expressed, and 92 answered.Source: §D.4 · p. 14
- Figure 5 reports that the user model discarded most observations, with greater losses among participants who talked more.Figure 6 supplies the human-validation confusion matrices against the two-coder consensus.Source: Figure 5 · p. 15, Figure 6 · p. 15
- The memory_recall pathway placed last at every funnel stage and was most likely to be judged a restatement of the previous turn, at 18.2%.The corresponding restatement rates for the other two triggers were 13.8% and 11.0%.Source: §D.4 · p. 15
- Within 60 minutes, 22.4% of proactive utterances drew a further user message, compared with 72.6% of reactive responses.Across participants, the proactive response share ranged from 2.6% to 34.7%.Source: §D.4 · pp. 15–16
D.5 The named prohibition held; the register did not fully
The named lexical prohibition was largely followed, but adjacent sensory language remained elevated and participants noticed it. The section also reports limits on the interpretation of register patterns and connects proactivity with both interest and unanswered turns.
D.5 The named prohibition held; the register did not fully
- Where the persona’s named-word prohibition applied, those words were 9–15 times less frequent per character and 16–19 times less frequent per utterance.The blocklist’s obedience was not total: ellipsis, which was explicitly forbidden, still appeared in 13.0% of reactive responses.Source: §D.5 · p. 16
- Sensory vocabulary left off the blocklist remained elevated at 2.1–2.2 times the participant baseline after controlling for length.Sound-related vocabulary appeared in 18.2% of proactive utterances versus 3.5% of participant messages; participants noticed the surviving register, and NDA2 was the second-most-endorsed item.Source: §D.5 · p. 16
- The self-narrative corpus’s stock phrases are consistent with prompt imitation and do not independently establish a model-level stylistic prior.The authors report the register as an observation on one open-weight model, not as a general property of language models.Source: §D.5 · p. 16
E.1 Legibility without reachability
- Curiosity was associated with lower relational depth, but the authors offer opacity as a possible reading and note the small sample and exploratory correlations.The passage also offers a measurement interpretation: the item may assess legibility within a closeness scale.Source: §E.1 · p. 16
E.2 Proactivity was liked and unanswered
- Six participants named self-started conversations as the best part of the experience, while only one named memory; 77.6% of proactive utterances went unanswered.The passage reports perceived proactivity’s correlation with parasocial interaction as ρ = 0.922.Source: §E.2 · p. 16
E.3 Style: what a prohibition can and cannot do
The named lexical blocklist was largely obeyed, but unenumerated sensory vocabulary remained elevated, so suppressing listed words did not itself change register.
E.3 Style: what a prohibition can and cannot do
- Named expressions appeared in about 1% of applicable utterances versus 17.4% where the blocklist did not apply, while unenumerated sensory vocabulary remained about twice participants’ rate after length control.Ellipsis, though explicitly forbidden, appeared in 13.0% of reactive responses.Source: §E.3 · p. 16
- Only one candidate thought in 612 reached a reply, and participants who talked more had a smaller fraction of observations about them survive.Source: Figure 7 · p. 17
- The self-narrative corpus’s uniform phrasing reproduced the prompt’s example phrases, so it was not evidence of a model prior.The phrasings in Figure 8(b) are English glosses of Japanese originals.Source: §E.3 · p. 16, Figure 8 · p. 17