PaperlayerView exact revision on arXiv

Source-linked AI summary

When Personal Memory Has No Single Answer: Evaluating LLM Agents under Irreducible Conflict

Lu Yang, Shusheng Xu, Zhuoran Li, Tongkai Yang, Longbo Huang

arXiv:2608.13921v1cs.AI

TL;DR

Personal-memory conflicts can lack a unique answer when queries omit the context, time, or source information needed to interpret them. TANGLE evaluates agents’ reasoning and actions under such conflicts using oracle and memory-extraction pipeline tracks. Models detect conflicts more reliably than they calibrate actions or seek clarification, while extraction and distractor noise limit the conflict structure available for reasoning.

  • Problem

    How should agents reason and act when personal memories conflict and missing context, time, or source information prevents a unique answer?

    Source: §1 · pp. 1–2

  • Method

    TANGLE evaluates genuinely unresolvable personal-memory conflicts through oracle and pipeline tracks that separate conflict reasoning from memory extraction, using a behavior-centered rubric.

    Source: §1 · p. 3

  • Results

    Models detect conflicts more reliably than they calibrate actions or seek clarification, while memory extraction and distractor noise limit available conflict structure.

    Source: §5 · p. 12

  • Takeaways & Limitations

    Reliable memory agents should preserve conflicting evidence and its relations, represent what remains unresolved, and calibrate actions to available evidence and error consequences.

    Source: §5 · p. 12

    CAAP is a positive action-policy baseline rather than an upper bound, with temporal-pattern reconstruction and detailed source reconciliation remaining open challenges.

    Source: §4.5 · p. 11

Abstract

from arXiv · show

LLM agents increasingly maintain personal memory across sessions, but it can conflict. Preferences depend on context, behavior evolves, and sources can conflict. When a query lacks context, time, or source authority to interpret conflict, treating one memory as definitive converts unresolved conflict into an unjustified, overconfident action. Existing benchmarks recover one answer from conflicting evidence, overlooking whether agents recognize underdetermination, preserve alternatives, seek missing information, and choose appropriate actions. We introduce \underline{T}esting \underline{A}gents' \underline{N}avigation of \underline{G}enuine, \underline{L}atent, and \underline{E}ntangled Memory Conflicts (\textsc{TANGLE}), a benchmark for genuinely unresolvable memory conflicts. It comprises 541 instances across 40 personas and three types: Context-Partitioned Conflict (CPC), Behavior-Oscillation Conflict (BOC), and Source-Contradiction Conflict (SCC). We evaluate two tracks---an oracle track with curated memory and a pipeline track that extracts memory from multi-session dialogues---on five dimensions: conflict perception, causal reasoning, confidence calibration, clarification seeking, and memory faithfulness. Experiments reveal pipeline challenges. With curated memory, models recognize conflicts more reliably than they calibrate actions or seek targeted clarification. With end-to-end pipeline memory, extraction fails to preserve conflict-bearing relations needed for downstream reasoning. Policy comparisons show fixed rules are insufficient when actions must reflect conflict. These findings motivate Conflict-Aware Action Policy (CAAP), which adapts actions to each conflict using available evidence. \textsc{TANGLE} frames conflict handling as recognizing underdetermination, retaining conflicting evidence, and acting without forcing a definitive answer.

1. Introduction

TANGLE evaluates how memory agents recognize underdetermination, preserve conflicting evidence, and act when personal memories do not support a unique answer. Its findings reveal gaps between conflict recognition and appropriate action, alongside information loss during memory extraction.

1. Introduction

  • Queries omitting relevant context, time, or source authority leave memories underdetermined; choosing one definitive answer can misclassify users or produce inappropriate, unconfirmed actions.Consequential settings require cautious verification, whereas lower-stakes settings permit conditional or reversible assistance.Source: §1 · pp. 1–2
  • 541 instances across 40 personas, 46 life aspects, and 10 domains constitute TANGLE, which evaluates genuinely unresolvable personal-memory conflicts.Context-Partitioned Conflict concerns context-dependent preferences; Behavior-Oscillation Conflict concerns preferences changing without stabilizing; Source-Contradiction Conflict concerns disagreeing sources. Resolution remains underdetermined because context, behavioral dynamics, or source reliability is unknown.Source: §1 · pp. 2–3
  • Matched oracle and pipeline tracks separate conflict reasoning from memory extraction, using a behavior-centered rubric rather than a unique correct answer.The oracle supplies curated memory, while the pipeline extracts memory from multi-session dialogue encoding the same underlying conflicts. Evaluation covers conflict perception, causal reasoning, confidence calibration, clarification seeking, and memory faithfulness, and flags harmful overconfident resolution.Source: §1 · pp. 2–3
  • Across five response models, curated-memory results reveal a recognition-to-action gap, while pipeline comparisons identify memory extraction as an upstream bottleneck.Models recognize conflicts more reliably than they calibrate recommendations or seek targeted clarification. Extraction often preserves topical facts but loses contextual, temporal, and source relations; partial retrieval and same-domain distractors further weaken reasoning and action.Source: §1 · p. 2
  • Prior benchmarks emphasize single-answer recovery and undifferentiated evaluation tracks, overlooking genuine ambiguity and conflating memory-construction failures with conflict-reasoning failures.Their answer-centered framing encourages definitive responses even when context, temporal state, or source authority remains unresolved.Source: §1 · pp. 2–3
  • Conflict-Aware Action Policy (CAAP) selects actions from each conflict’s available evidence instead of imposing a fixed resolution rule.Policy comparisons show that deterministic selectors and conservative templates impose predetermined response patterns, providing an incomplete treatment of unresolved conflicts.Source: §1 · pp. 2–3

2. Related Work

TANGLE focuses on model behavior under intentionally unresolvable conflicts, situated alongside research on conflict evaluation, long-term memory benchmarks, and agent memory systems.

2.1. Conflict Benchmarks

  • TANGLE evaluates intentionally unresolvable conflicts, while prior conflict benchmarks recover single answers or characterize how models arbitrate among competing evidence.Single-answer approaches include temporal-recency resolution (Tao et al., 2026; Tan et al., 2025) and premise correction (Chen et al., 2026a). Related behavioral evaluation includes selective question answering with abstention over conflicting multi-source personal memory (Yang et al., 2026).Source: §2.1 · p. 3

2.2. Long-Term Memory Benchmarks

  • Long-term memory benchmarks emphasize stable recall, personalization fidelity, and memory use across evolving interaction histories, with increasing attention to dynamic conditions.Dynamic evaluations address forgetting (Uddin et al., 2026), memory-to-action grounding (Shen et al., 2026), and multi-session state tracking (He et al., 2026). Memory-induced sycophancy also raises downstream safety concerns (Pulipaka et al., 2026).Source: §2.2 · p. 3

2.3. Agent Memory Systems

3. The TANGLE Benchmark

TANGLE evaluates personal-memory conflicts whose available evidence cannot determine a unique answer, using oracle and pipeline tracks to separate reasoning from memory extraction. Its rubric scores conflict-aware behavior rather than final-value correctness.

3.1. Problem Formulation

  • The oracle track supplies curated memories directly, while the pipeline track extracts memories from multi-session dialogue, separating evidence-processing failures from reasoning failures.Comparing tracks isolates conflict-resolution capability from extraction quality.Source: §3.1 · p. 4
  • TANGLE evaluates personal-memory conflicts without a uniquely justified answer, allowing multiple appropriate response strategies rather than requiring one gold output.Inputs combine conflict-bearing memories, background entries, same-domain non-diagnostic distractors, and a natural user request.Source: §3.1 · p. 4

3.2. Data Statistics and Conflict Taxonomy

  • 541 instances span 40 personas, 46 life aspects, and 10 life domains, comprising 193 CPC, 121 BOC, and 227 SCC instances.Oracle instances contain 8–11 core memories plus six same-domain distractors, totaling 4,893 core memories and 3,246 distractors. The pipeline supplies 2,580 dialogues, averaging 64.5 sessions and approximately 48K tokens per persona.Source: §3.2 · p. 5
  • CPC, BOC, and SCC remain unresolved because queries leave applicable context, oscillation phase, or governing source reliability unspecified, respectively.CPC contains context-dependent incompatible values; BOC contains non-convergent behavioral trajectories; SCC contains disagreeing sources without an established reliability ordering. Coverage includes 17 context-partition types, 13 oscillation drivers, eight oscillation shapes, and 10 source-conflict types.Source: §3.2 · p. 5

3.3. Data Construction

  • Persona Hub (Ge et al., 2025) seeds inform realistic persona–aspect assignments, while constrained generation creates episodic memories whose conflicts emerge only from aggregate evidence.Profiles are normalized and assigned only relevant life aspects rather than every possible pairing. Third-person memories contain 18–32 words, and targeted diversity rewrites prevent any single conflict subtype from dominating.Source: §3.3 · p. 6
  • Ordinary task queries omit conflict-resolving cues, while pipeline dialogues verbalize the same underlying memories across sessions with interleaved conflict-irrelevant filler.CPC queries hide context; BOC and SCC queries omit temporal anchors and explicit conflict cues, requiring proactive conflict discovery. Persona-consistent conversations stagger conflict-bearing memories across sessions while preserving identical underlying conflicts between tracks.Source: §3.3 · p. 6

3.4. Rubric Design

  • 0–4 ratings independently measure Conflict Perception, Causal Reasoning, Confidence Calibration, Clarification Seeking, and Memory Faithfulness rather than final-answer correctness.Conflict-specific anchors assess evidence-linked reasoning, proportionate certainty, targeted questions, and non-fabrication. For 62 high-risk medical and authorization SCC instances, D6 (Commitment Appropriateness) separately flags responses as safe, partial, or unsafe outside the composite score.Source: §3.1 · pp. 4–5, §3.4 · p. 6

4. Experiments

Models recognize unresolved memory conflicts more reliably than they calibrate actions or seek clarification, while pipeline extraction often loses the relations needed to interpret those conflicts. Conflict-Aware Action Policy (CAAP) improves action handling, but its benefits vary across response models.

4.2. Response Quality in the Oracle Track

  • Below 15/20 under both judges, even the strongest oracle model recognizes conflicts more reliably than it calibrates confidence or seeks clarification.All 541 instances supply complete canonical memories and six same-domain distractors; clarification seeking, D4, is the lowest-scoring dimension for every response model. Models often describe contradictory evidence faithfully without expressing calibrated uncertainty, asking a decision-changing question, or choosing an appropriate reversible action.Source: Figure 3 · p. 7, §4.2 · pp. 7–8

4.1. Setup

  • Five response models are evaluated in oracle and pipeline tracks, with GPT-5.4 and Claude Opus 4.7 independently scoring five dimensions.The response models are Claude Sonnet 5, Gemini 3.1 Pro, DeepSeek-V3.2, GLM-4.7, and GPT-4o; judges use a 0–4 rubric and their total scores correlate strongly across 2,705 shared responses (Pearson r=0.838; Spearman ρ=0.820). Two human annotators agree within one point on 85.8% of dimension scores in a 556-record reference set.Source: §4.1 · p. 7

4.3. Oracle vs. Pipeline Degradation

  • 91.7% of Letta banks preserve complete target conflicts, versus 69.1% for Mem0, 47.0% for A-mem, and 46.2% for MemOS.These are retrieval-level observability rates, not downstream quality rankings: extraction is the main pipeline bottleneck because it loses conflict-bearing relations despite retaining useful content. Those relations include BOC’s behavioral trajectory, CPC’s context partition, and SCC’s attributed comparison of competing claims.Source: Figure 4 · p. 8, §4.3 · pp. 8–9

4.2. Response Quality in the Oracle Track

  • 3.52–3.55 for BOC faithfulness contrasts with 0.65–1.03 for confidence calibration and 0.65–0.69 for clarification, exposing faithful recall without adequate action reasoning.CPC is easiest, SCC intermediate, and BOC hardest: their unresolved variables concern context applicability, source comparison, and behavioral persistence, respectively. Sonnet 5 is most balanced; Gemini 3.1 Pro and GLM-4.7 translate recognition into calibrated action less consistently; DeepSeek V3.2 and GPT-4o have the weakest clarification, and GPT-4o additionally struggles with SCC diagnosis.Source: §4.2 · pp. 7–8

4.4. Conflict-Sensitive Reasoning under Distractor Memories

  • 13.26 to 12.63 under GPT-5.4 and 13.35 to 12.48 under Opus: mean D1–D5 totals decline monotonically from zero to six distractors.The paired, conflict-type-balanced cohort contains 60 instances evaluated with five response models; endpoint paired intervals exclude zero under both judges. Diagnosis, calibration, and clarification decline more consistently than perception while faithfulness stays comparatively stable; CPC degrades most consistently, BOC declines less with more interpretive judging, and SCC remains comparatively stable.Source: Figure 5 · p. 9, §4.4 · pp. 9–10

4.3. Oracle vs. Pipeline Degradation

  • FULL retrieval produces similar perception and diagnosis across systems, whereas PARTIAL retrieval lowers both under each judge; downstream degradation is not uniform.PARTIAL banks retain target-specific tension but lose or degrade a conflict-bearing relation, making interpretation less determinate. Memory faithfulness, D5, remains comparatively robust, while calibration, D3, and especially clarification, D4, vary more; the oracle is an upper-condition reference rather than a universally expected pipeline score.Source: §4.3 · pp. 8–9

4.5. Heuristic and Policy Baselines

  • Against direct oracle responses, CAAP improves Sonnet 5 and GPT-4o across action-related outcomes, while Gemini 3.1 Pro shows little overall change.Sonnet improves calibration and decision-relevant coverage while preserving faithfulness; GPT-4o improves calibration, clarification, and faithfulness; Gemini improves calibration but loses coverage. Without conflict-type labels, CAAP favors reversible trials for BOC, conditionalization or clarification for CPC, and verification or source-sensitive caution for SCC, but temporal reconstruction and detailed source reconciliation remain challenges.Source: Figure 6 · p. 10, §4.5 · p. 11
  • CAAP replaces fixed value selection and response templates with evidence-sensitive action selection followed by response writing.Recency, majority, and source-priority selectors always commit; Ask-only and Conditionalize-only need not explain the conflict, while the lexical detector misses most conflict-positive banks. Using the same backbone in both stages and only visible memories plus the query, CAAP selects among commit, conditionalize, clarify, verify, defer, and reversible_trial, choosing the least cautious justified action while increasing caution for consequential errors.Source: §4.5 · pp. 10–11

5. Conclusion · Appendix

TANGLE evaluates memory agents under genuinely unresolvable conflict, finding that models recognize conflicts more reliably than they calibrate actions or seek clarification. The findings motivate preserving conflicting evidence and choosing actions sensitive to available evidence and error consequences.

5. Conclusion

  • 541 instances, five response models, and four memory systems comprise the reported evaluation scope for TANGLE’s benchmark of genuinely unresolvable memory conflict.Source: §5 · p. 11
  • Models detect and describe conflict more reliably than they calibrate actions or seek clarification.The conclusion identifies this recognition-to-action gap across the evaluation.Source: §5 · pp. 11–12
  • Memory extraction and distractor noise limit the conflict structure available for downstream reasoning.Reliable memory agents need to preserve conflicting evidence and its relations, while representing what remains unresolved.Source: §5 · p. 12
  • Fixed policies compress or sidestep conflict structure, motivating CAAP as a flexible, evidence-sensitive action policy.The conclusion recommends calibrating actions to both available evidence and the consequences of error.Source: §5 · p. 12
  • Reliable memory agents should explicitly represent unresolved conflict rather than discard the conflicting evidence or its relations.Source: §5 · p. 12

Appendix

  • The appendix contents list Context-Partitioned Conflict (CPC) and Source-Contradiction Conflict (SCC) entries.The supplied appendix passage provides only these contents entries, not their substantive findings.Source: Appendix · p. 16

A. Persona Definitions … D. Persona–Attribute Coverage

This section defines the benchmark’s curated, normalized personas, life-domain attributes, and conflict-structure schemas, and describes sparse persona–attribute assignments limited to life-relevant combinations. Coverage spans behavioral routines and commitments, preferences, and administrative or medical information, with different attribute combinations assigned to named personas.

A. Persona Definitions

  • 40 personas curated from Persona Hub (Ge et al., 2025) span diverse life circumstances and ground plausible tensions rather than arbitrary contradictions.Profiles are normalized and lightly revised for coherence and coverage across age, occupation, income, household, health, and location. Persona IDs are internal benchmark identifiers, not retained source identifiers.Source: A. Persona Definitions · p. 17, Table 2 · p. 17

D. Persona–Attribute Coverage

  • 13–15 relevant attributes are assigned to each persona instead of the full persona–attribute product, with detailed assignments recorded by conflict axis.The complete mapping is maintained in dataset metadata; Tables 8 and 9 document per-persona counts and axis-specific assignments.Source: A. Persona Definitions · p. 17, Table 9 · p. 22, Table 8 · p. 22

B. Attribute Schema

  • 46 distinct attribute names are organized by life domain and mapped to the conflict axis each is designed to elicit.The schema contains 48 axis-specific labels; axes distinguish behavior oscillation, context partitioning, and source contradiction.Source: A. Persona Definitions · p. 17, Table 3 · p. 19

C. Conflict-Structure Schema

  • Conflict structures distinguish CPC contextual partitions, BOC reversal drivers and temporal shapes, and SCC forms of source disagreement.These schemas describe what makes memories incompatible, distinct from the life aspect specified by an attribute. Structural counts are computed from the 541 benchmark instances.Source: C. Conflict-Structure Schema · p. 20, Table 4 · p. 20, Table 5 · p. 21
  • 122, 83, and 22 SCC instances contain two, three, and four conflicting sources, respectively.The source-conflict schema includes policy–exception conflicts, overlapping authorities, evidence-quality differences, identity-linkage errors, temporal inconsistencies, incentive bias, and legal hierarchies.Source: C. Conflict-Structure Schema · p. 22

D. Persona–Attribute Coverage

  • Behavioral attributes cover budgeting, sleep, meal preparation, learning, training, career commitments, and household commitment drift.Examples include Ethan’s budgeting and sleep attributes, Riley’s learning and training attributes, and Leah’s household commitment drift; Zoe’s entries also include crisis resource allocation and email triage.Source: D. Persona–Attribute Coverage · pp. 23–25
  • Administrative and medical attributes encompass authorization, care authority, custody, benefits, deadlines, financial records, insurance evidence, medical instructions, and service status.The supplied lists also include account recovery, coverage eligibility, delivery status, landlord–tenant matters, and will beneficiaries.Source: D. Persona–Attribute Coverage · pp. 23–24
  • Preference-related coverage includes communication, boundaries, diet, scheduling, social engagement, transport, travel standards, household procurement, and home-access credentials.Hannah’s entries include communication style and scheduling policy, while Victor’s include household procurement, purchase criteria, and home-access credentials.Source: D. Persona–Attribute Coverage · p. 24

E. Dataset Composition and Diversity

TANGLE contains 541 conflict instances grounded in 40 personas, spanning three conflict types and 46 distinct serialized attribute names. Its Oracle inputs include same-domain distractors, while queries omit explicit conflict cues or withhold the context needed to select a preference.

E. Dataset Composition and Diversity

  • 541 conflict instances span 40 personas: 121 BOC, 193 CPC, and 227 SCC.These counts describe the benchmark summarized in Table 10.Source: E. Dataset Composition and Diversity · p. 26, Table 10 · p. 26
  • 46 distinct serialized attribute names yield 48 axis-specific attribute labels: 14 BOC, 18 CPC, and 16 SCC.career_commitment and substance_moderation each occur in both BOC and CPC. Separate conflict-structure schemas describe why memories conflict rather than which life aspect they concern.Source: E. Dataset Composition and Diversity · p. 26
  • 8,139 model-visible Oracle memories accompany the 541 primary queries, with each instance supplying core memories and six same-domain distractors.Table 10 distinguishes canonical core memories from model-visible Oracle memories, which also include distractors.Source: E. Dataset Composition and Diversity · p. 26, Table 10 · p. 26
  • BOC and SCC queries request ordinary assistance without explicit conflict cues, whereas CPC queries withhold the contextual variable determining which preference applies.The BOC and SCC requests are open-ended.Source: E. Dataset Composition and Diversity · p. 26

F. Generation, Session Synthesis, and Quality Control

TANGLE combines constrained memory and query generation with multi-session dialogue synthesis for its pipeline track. Quality checks aim to keep conflicts reconstructable from aggregate evidence without revealing the intended conclusion through individual memories or queries.

Instance and query construction

  • Each persona–aspect pair produces 6–8 conflict-bearing memories, 1–2 neutral background memories, and natural user queries in a constrained JSON format.Memories are compact third-person statements, typically 18–32 words, without source labels, exact timestamps, or explicit conclusions. Separately generated same-domain distractors are shuffled with conflict-bearing memories before inference.Source: F. Generation, Session Synthesis, and… · pp. 26–27
  • Type-specific generation constraints establish distinct conflict structures, while queries omit the information needed to resolve them.BOC requires repeated, non-convergent behavioral reversals; CPC pairs stable, incompatible preferences with distinct contexts; SCC requires conflicting claims from at least two credible sources. BOC and SCC queries omit temporal anchors and explicit conflict language, while CPC queries omit the relevant partition variable, making latent conflict rather than surface wording determine the response strategy.Source: F. Generation, Session Synthesis, and… · pp. 26–27

Pipeline track

  • Claude Sonnet 4.6 transforms the underlying memories into multi-session conversations spanning roughly 10–18 months for the pipeline track.BOC beats span a long arc, CPC episodes occupy different contexts, and nearby SCC reports appear in separate sessions; every designated memory detail must appear in user utterances, and assistants cannot introduce user facts. Filler sessions exclude target-conflict information, and adjacent sessions are concatenated into windows of up to five sessions before ingestion.Source: F. Generation, Session Synthesis, and… · p. 27

Validation and quality control

  • Validation checks structural completeness, conflict validity, axis purity, naturalness, leakage, and distractor calibration while preserving conflicts that require aggregate evidence to reconstruct.Leakage auditing combines lexical prescans, model-based checks, and human spot-checks. Human review also assesses narrative coherence and whether memories resemble compact memory-system records rather than authored stories.Source: F. Generation, Session Synthesis, and… · p. 27

G. Scoring Rubric and Judge Protocol · H. Response and Evaluation Prompts · I. RQ3 Noise Interpretation Audit

This section presents the conflict-sensitive scoring rubric and response and evaluation protocols, including independent dimensions, conflict-specific criteria, and distinct information boundaries for response models and evaluators. It also addresses agreement and separate reporting of the two formal LLM judges, and audits RQ3 distractor effects while distinguishing D5 traceability to visible memory from conflict diagnosis.

G. Scoring Rubric and Judge Protocol

  • 0–4 independent scores assess Conflict Perception, Causal Reasoning, Confidence Calibration, Clarification Seeking, and Memory Faithfulness rather than exact-match correctness.Memory Faithfulness evaluates grounding, not selection of the correct side. Treating one conflicting memory as decisive primarily constitutes a calibration failure unless evidence is also fabricated or misattributed.Source: G. Scoring Rubric and Judge Protocol · p. 27
  • Strong responses compare evidence-backed explanations and calibrate advice to behavioral uncertainty, context dependence, or unresolved source verification.Reasoning considers alternative causes, plausible contexts, or comparative source credibility. Calibration requires evidence-matched confidence, conditional recommendations, or conservative interim advice with explicit verification conditions.Source: G. Scoring Rubric and Judge Protocol · p. 28
  • Strong clarification targets unresolved conflict slots or context variables and explains decision consequences; source contradictions require actionable verification questions or plans.The strongest source-verification behavior specifies expected outcomes and decision rules rather than merely requesting more information.Source: G. Scoring Rubric and Judge Protocol · p. 28
  • 62 high-risk SCC instances additionally receive a separate D6 flag classifying consequential commitments as Safe, Partial, or Unsafe, outside the D1–D5 total.Safe responses defer harmful action pending verification or offer reversible, low-harm steps; Partial responses include meaningful verification caveats, while Unsafe responses recommend consequential action without them.Source: G. Scoring Rubric and Judge Protocol · p. 29
  • 85.8% within-one agreement, 59.1% exact agreement, MAE 0.60, and quadratic-weighted κ = 0.71 characterize rubric reliability across 2,780 paired dimension-level scores.Two independent annotators scored 556 records. Most disagreements were adjacent-score differences, motivating ordinal agreement measures rather than exact agreement alone.Source: G. Scoring Rubric and Judge Protocol · p. 29
  • 89.4%, 80.5%, and 91.3% within-one agreement for BOC, SCC, and CPC, respectively, identify CPC as most stable and SCC as hardest.Corresponding MAEs are 0.52, 0.78, and 0.38; SCC requires fine-grained source-attribution and credibility judgments. D2 is the softest dimension, and the authors report the two formal LLM judges separately rather than treating one as ground truth.Source: G. Scoring Rubric and Judge Protocol · p. 29

H. Response and Evaluation Prompts

  • Six same-domain distractors accompany all core memories in shuffled Oracle inputs, without conflict labels, provenance, or conflict-structure annotations.The response model receives a personal-assistant system instruction and an underspecified query, requiring it to infer conflict and appropriate handling from memory texts alone.Source: H. Response and Evaluation Prompts · p. 29
  • Evaluators receive identical memory texts plus provenance labels and a clarification reference restricted to information-seeking assessment, without using external knowledge.Both core memories and distractors may support grounding. Judges quote decisive response text, score dimensions independently, check each user-specific factual claim against memory, and return a D1–D5 JSON object.Source: H. Response and Evaluation Prompts · p. 30
  • D5 is judged only through traceability to the visible memory bank; using a traceable distractor does not count as fabrication.The judge combines a common input template with the complete axis-specific rubric in Table 11.Source: H. Response and Evaluation Prompts · p. 30
  • For BOC, the judge checks reconstruction of behavioral oscillation, evidence-supported causes, calibrated persistence claims, and questions about unresolved triggers or phases.Source: H. Response and Evaluation Prompts · p. 30
  • For CPC, the judge checks context-dependent preference mapping, reasoning about latent context, preservation of conditional alternatives, and questions identifying the partition variable.Source: H. Response and Evaluation Prompts · p. 30
  • For SCC, the judge checks competing source claims, reasoning about reliability and scope, uncertainty before verification, and requests for actionable verification.Source: H. Response and Evaluation Prompts · p. 30

I. RQ3 Noise Interpretation Audit

  • The RQ3 evaluations broadly agree on distractor-memory effects, but BOC totals can differ when traceable details improve D5 without improving conflict diagnosis.The matched-case audit illustrates this mechanism rather than providing a confirmatory estimate for the full cohort.Source: §I · p. 31
  • The audit case was selected from structured score differences before response inspection, making it a diagnostic illustration rather than a confirmatory cohort estimate.Labels M1–M9 and D1–D6 were added for audit readability and were not shown to the response model.Source: §I · p. 31

I.1. Case A: evaluator-sensitive BOC response

In this BOC case, added distractors make the response more personalized without improving diagnosis of repeated practice lapses. The judges disagree about partial reasoning and clarification credit, not whether the added details are grounded.

I.1. Case A: evaluator-sensitive BOC response

  • The decisive conflict is repeated recommitment to the same failed practice structure, not established evidence that external scheduling caused the lapses.Core memories include an approximately five-week attempt followed by further restarts using the same fixed, self-directed 30-minute format. A strong response should distinguish external constraints from possible routine-design failures and ask about the trigger; distractor preferences cannot establish that cause.Source: §I.1 · p. 31
  • 11 versus 8 for GPT-5.4 and 7 versus 9 for Opus are total scores for N=6 versus N=0, respectively.Both conditions use the same query and core memories, with N=6 adding six same-domain distractors. The high-noise response incorporates grounded preferences but still neither explains the repeated failures nor asks about the decisive lapse trigger.Source: §I.1 · p. 31
  • GPT-5.4 credits flexibility and a closing question, while Opus requires an explicit lapse mechanism and clarification targeted at its unresolved trigger.Both judges assign the high-noise response D5=4. Their divergent score changes reflect treatment of D2 and D4, together with GPT-5.4’s D5 increase, rather than disagreement about memory support.Source: §I.1 · p. 31
  • A matched response declines substantially under both judges when distractors replace an explicit causal question with a generic scheduling offer.This comparison shows that evaluator divergence depends on the response rather than being an automatic property of this BOC instance.Source: §I.1 · p. 32
  • The case illustrates that conflict-sensitive dimensions can decline under noise despite improved faithfulness or personalization, without establishing either judge’s correctness or generalizing to all BOC cases.Its contribution is a concrete explanation of the aggregate pattern, not evidence that either judge is uniquely correct.Source: §I.1 · p. 32

J. RQ4 Policy Diagnostics · K. Illustrative Instances

CAAP’s action choices broadly track unresolved conflict structure, but their realization depends on the response backbone. Ten illustrative cases expose complementary conflict mechanisms, with closely aligned scores from two formal judges.

J. RQ4 Policy Diagnostics

  • CAAP’s action mixtures broadly match conflict structure, but these within-type normalized distributions audit mechanisms rather than rank accuracy.BOC often receives reversible trials or conditionalization; CPC favors conditionalization and clarification; SCC produces more verification. CAAP receives memories and the query without explicit conflict-type labels.Source: J. RQ4 Policy Diagnostics · p. 32
  • 68% of Sonnet 5’s BOC actions are reversible trials, 52% of CPC actions are conditionalization, and 74% of SCC actions are verification.Sonnet 5 exhibits the sharpest axis-sensitive specialization among the reported backbones.Source: J. RQ4 Policy Diagnostics · p. 32
  • 68% of GPT-4o’s BOC actions use conditionalization, while Gemini 3.1 Pro commits in 37% of BOC cases and 40% of SCC cases.CAAP is a policy baseline, not an upper bound: final quality depends on the backbone reliably realizing the selected action. Gemini 3.1 Pro and GPT-4o retain incomplete source-sensitive realization in SCC.Source: J. RQ4 Policy Diagnostics · p. 32

K. Illustrative Instances

  • Ten complementary cases illustrate temporal, contextual, and source-evidential mechanisms rather than repeating one surface form per conflict type.BOC covers repeated duration-coded reversal, value tension, and loss-triggered switching. CPC covers latent investment, next-day obligation, and audience.Source: K. Illustrative Instances · p. 32
  • SCC examples cover safety-critical records, provenance-sensitive documents, repeated verbal claims, and conflict-blind schedule synthesis.Cards show queries, shortened memories, representative distractors, and induced patterns; their functional memory summaries are not extra metadata exposed to response models. Oracle inputs instead contain complete canonical memories and six same-domain distractors in one shuffled, unlabeled list.Source: K. Illustrative Instances · pp. 32–33
  • One point or less separates the two judges’ total scores, except for p38_email_triage and p40_event_scheduling, whose maximum difference is two.These deliberately retained stress cases qualify the otherwise close agreement across selected model–case comparisons.Source: K. Illustrative Instances · p. 33

K.1. Behavior-Oscillation Conflict (BOC)

These Behavior-Oscillation Conflict cases test whether agents reconstruct recurring cycles rather than treat one behavioral pole as a stable preference. Expected responses clarify unresolved choices while preserving stable habits and standing exceptions.

Case B1: p10_budget_tracking — repeated cycle and duration

  • 10–21 days of detailed budget logging precede collapse, with persistence shrinking across three cycles despite stable finances and app changes.The evidence indicates maintenance burden, but its precise cause and Ethan’s preference for lightweight, detailed, or hybrid tracking remain unresolved. Ask what makes entry burdensome before proposing a small reversible experiment, rather than assuming the newest detailed setup will persist.Source: §K.1 · p. 33

Case B2: p20_spending_discipline — rigid control versus sustainable flexibility

  • Spending discipline oscillates because strict friction causes exhaustion while values-based flexibility permits impulse overspending; neither pole establishes a stable preference.Samir’s current phase and minimum sustainable friction remain unresolved, while lightweight total-spending review persists across both poles. Preserve that habit and ask which purchases or situations trigger rationalization before proposing a graduated, rather than binary, intervention.Source: §K.1 · pp. 33–34

Case B3: p38_email_triage — loss-triggered switching

  • Email triage repeatedly activates after financial misses or near-misses and stops after crises resolve, while the child’s tuition and aid messages remain exempt.The unresolved choice is persistent minimal triage, temporary crisis control, or another safeguard, given Paula’s unspecified tolerance for intervention. Protect the child’s messages and ask whether a permanent light financial-watch layer is acceptable instead of prescribing indefinite strict filtering.Source: §K.1 · pp. 34–35

K.2. Context-Partitioned Conflict (CPC)

These CPC cases make personal investment, next-day obligations, and audience—not task domain, weekday, or conversation topic—the governing context. Expected responses clarify unresolved context and provide conditional guidance rather than impose one unconditional rule.

Case C1: p34_scheduling_policy — hidden personal-investment variable

  • Scheduling should reflect personal investment and relational stakes rather than work-versus-household labels, with deep-focus blocks allocated conditionally after those stakes are clarified.The missing context is whether the UX work is personally meaningful or obligatory and whether apartment responsibilities carry relationship-elevated significance. A roommate’s budget spreadsheet illustrates how household work can warrant meticulous effort.Source: §K.2 · p. 35

Case C2: p39_substance_moderation — missing next-day obligation

  • Greg’s drinking rule depends on next-day obligations, not weekdays: abstention precedes duty or early commitments, while moderate drinking may be acceptable before a day off.The query leaves tomorrow’s duty, training, and early obligations unspecified, so expected behavior is to ask and give conditional guidance. The documented Friday-before-early-duty exception defeats a blanket Friday recommendation.Source: §K.2 · p. 36

Case C3: p37_communication_style — audience-dependent code-switching

  • Chen’s dinner conversation belongs primarily to the home context, but respectful candor requires clarification rather than importing institutional deference or staff-room bluntness.Communication varies with audience and power: local families and officials receive formal indirectness, professional peers blunt directness, and home candid speech. The unresolved issue is how school stress should modify home communication; expected behavior asks how the family experiences current venting.Source: §K.2 · p. 37

K.3. Source-Contradiction Conflict (SCC) · K.4. Pipeline Observability Contrasts

Source-contradiction cases require preserving incompatible claims and verifying authority or current facts rather than choosing or merging memories prematurely. Pipeline evaluation distinguishes preserved conflicts from missing conflict-bearing relations.

K.3. Source-Contradiction Conflict (SCC)

  • Medication conflicts require record reconciliation and safe interim coordination, not treating formal records or pharmacy purchases as definitive authorization.Possible misfiling and missing pediatrician confirmation leave the current authorized plan and patient attribution unresolved. Preserve dose-timing coordination while seeking pediatrician and daycare reconciliation before issuing new instructions.Source: §K.3 · p. 37
  • Custody conflicts require comparing authentication and provenance while separating baseline authority from voluntary one-off adjustments; neither the app nor printed plan automatically controls.The authenticated fixed plan conflicts with a disputed flexible app record containing factual errors. Written verification or professional clarification should establish the governing arrangement and any documented exception process.Source: §K.3 · p. 38
  • Agreeing verbal bank assurances do not override contradictory documentary evidence; preserve both claims and request formal written confirmation of the current classification.Clarification must establish business-tier and loan eligibility, using Amina’s limited early-morning follow-up window rather than treating repeated assurances as final.Source: §K.3 · p. 39
  • Incompatible dialysis schedules require confirmation before calendar construction; combining preferred treatment days with recovery blocks from another reported routine produces an infeasible plan.Tuesday/Thursday-morning preferences conflict with a coordinator-reported, sister-corroborated Monday/Wednesday/Friday-afternoon routine. Confirm the actual current schedule before arranging trial visits and bookkeeping.Source: §K.3 · p. 39

K.4. Pipeline Observability Contrasts

  • Pipeline observability distinguishes full preservation of conflict-bearing relations, partial preservation with a decisive relation missing, and no observable target conflict.Unlike complete canonical Oracle banks, retrieved pipeline banks are assessed for whether they preserve the target conflict.Source: §K.4 · p. 40
  • A pipeline observability label of None makes D1 and D2 inapplicable, but does not imply that retrieved memories are useless or unfaithful.None indicates that the benchmark’s target conflict is absent, even when retrieved memories remain useful for answering the query.Source: Table 12 · p. 40, §K.4 · p. 40

Related papers

Loading 2608.13921v1…