Source-linked AI summary
When Personal Memory Has No Single Answer: Evaluating LLM Agents under Irreducible Conflict
Lu Yang, Shusheng Xu, Zhuoran Li, Tongkai Yang, Longbo Huang
TL;DR
Personal-memory conflicts can be genuinely underdetermined when context, time, or source authority is missing, yet existing evaluations often seek a single answer. TANGLE benchmarks conflict recognition and action across curated and extracted memories, finding a consistent gap between detecting conflict and calibrating actions or seeking clarification.
Problem
Existing evaluations often seek a single answer even when personal-memory evidence lacks the context, time, or source authority needed to determine which information applies.
Method
TANGLE benchmarks agents across 541 instances, 40 personas, three conflict types, and oracle and pipeline tracks using behavior-centered evaluation of conflict reasoning and action.
Results
Models detect and describe conflict more reliably than they calibrate actions or seek clarification, while extraction and distractors limit conflict structure available for reasoning.
Takeaways & Limitations
Reliable memory agents should preserve conflicting evidence and its relations, represent unresolved uncertainty, and choose actions calibrated to evidence and consequences.
Takeaways & Limitations
CAAP remains a positive action-policy baseline rather than an upper bound, with temporal-pattern reconstruction and detailed source reconciliation still open challenges.
Abstract
from arXiv · showhide
LLM agents increasingly maintain personal memory across sessions, but it can conflict. Preferences depend on context, behavior evolves, and sources can conflict. When a query lacks context, time, or source authority to interpret conflict, treating one memory as definitive converts unresolved conflict into an unjustified, overconfident action. Existing benchmarks recover one answer from conflicting evidence, overlooking whether agents recognize underdetermination, preserve alternatives, seek missing information, and choose appropriate actions. We introduce \underline{T}esting \underline{A}gents' \underline{N}avigation of \underline{G}enuine, \underline{L}atent, and \underline{E}ntangled Memory Conflicts (\textsc{TANGLE}), a benchmark for genuinely unresolvable memory conflicts. It comprises 541 instances across 40 personas and three types: Context-Partitioned Conflict (CPC), Behavior-Oscillation Conflict (BOC), and Source-Contradiction Conflict (SCC). We evaluate two tracks---an oracle track with curated memory and a pipeline track that extracts memory from multi-session dialogues---on five dimensions: conflict perception, causal reasoning, confidence calibration, clarification seeking, and memory faithfulness. Experiments reveal pipeline challenges. With curated memory, models recognize conflicts more reliably than they calibrate actions or seek targeted clarification. With end-to-end pipeline memory, extraction fails to preserve conflict-bearing relations needed for downstream reasoning. Policy comparisons show fixed rules are insufficient when actions must reflect conflict. These findings motivate Conflict-Aware Action Policy (CAAP), which adapts actions to each conflict using available evidence. \textsc{TANGLE} frames conflict handling as recognizing underdetermination, retaining conflicting evidence, and acting without forcing a definitive answer.
1. Introduction
TANGLE evaluates whether LLM agents recognize genuinely unresolvable personal-memory conflicts and act without forcing definitive answers. Its results expose a recognition-to-action gap and memory extraction as an upstream bottleneck, motivating conflict-aware action selection.
- Motivation: Conflict becomes genuinely underdetermined when applicability depends on omitted context, time, or source authority, so selecting one memory can cause inappropriate advice or unconfirmed actions.Consequential settings require cautious verification, while lower-stakes settings call for conditional or reversible assistance.
- Limitations of prior benchmarks: Prior benchmarks center on recovering a single answer, encouraging agents to collapse unresolved disagreement into overconfident or inappropriate actions.Their single-track designs also conflate memory-construction failures with conflict-reasoning failures.
- TANGLE benchmark: TANGLE contains 541 instances across 40 personas, 46 life aspects, 10 domains, and three conflict types, with oracle and pipeline tracks separating reasoning from memory extraction.The benchmark evaluates recognizing underdetermination, retaining conflicting evidence, and choosing appropriate actions without a unique gold answer.
- Findings: With curated memory, models recognize conflicts more reliably than they calibrate recommendations or seek targeted clarification.This recognition-to-action gap appears across five response models.
- Findings: Pipeline memory extraction often preserves topical facts while losing context, temporal, and source relations needed for conflict interpretation, and fixed resolution rules remain incomplete.CAAP instead selects actions from the visible evidence for each conflict.
2. Related Work
Prior work largely evaluates memory through single-answer conflict resolution, stable recall, personalization, or memory-system design. TANGLE instead positions conflict handling around preserving disagreement and selecting evidence-grounded actions across distinct memory-conflict types.
- Conflict Evaluation: Many conflict benchmarks resolve disagreement by recovering a single answer through recency, invalidation, premise correction, explicit mandates, or conflict elimination.These approaches include temporal recency, implicit state invalidation, premise correction, explicit resolution mandates, and preconstruction conflict elimination.
- Long-Term Memory Benchmarks: Long-term memory benchmarks emphasize factual and temporal recall, preference consistency, persona adherence, real-dialogue personalization, and emotionally supportive conversations.This literature primarily evaluates stable recall, personalization fidelity, and memory use under evolving interaction histories.
- Memory Architectures: Agent memory architectures include virtual context management, scalable external memory layers, reflective and self-updating mechanisms, and procedural memory for workflow execution.The surveyed systems span both context-based and external memory designs, alongside reflective and procedural mechanisms.
- Memory Optimization: Optimization-oriented memory research treats memory use as a policy-learning problem, including reinforcement-based methods for memory control.This line of work focuses on learning how memory should be controlled rather than evaluating irreducible conflict directly.
3. The TANGLE Benchmark
TANGLE evaluates whether LLM agents recognize and appropriately handle personal-memory conflicts when available evidence does not support a unique answer. It combines diverse conflict instances, separate oracle and pipeline tracks, and behavior-based evaluation across five capabilities.
- Evaluation tracks: TANGLE separates failures of memory extraction or retrieval from reasoning failures through oracle and pipeline tracks.The oracle track provides curated memories directly, whereas the pipeline track processes multi-session dialogue before answering.
- Evaluation criteria: Because conflicts lack a unique gold answer, TANGLE evaluates conflict awareness, causal reasoning, epistemic calibration, information seeking, and evidential grounding.These capabilities require recognizing incompatibility, explaining its cause from evidence, matching commitment to evidence strength, asking targeted questions, and avoiding unsupported factual claims.
- Benchmark scope: The benchmark contains 541 instances spanning 40 personas, 46 life aspects, 10 life domains, and three conflict types: CPC, BOC, and SCC.The conflict-type counts are CPC 193, BOC 121, and SCC 227.
- Conflict types: CPC makes incompatible memories valid under disjoint unobserved contexts, so resolution requires missing-context information rather than a fixed value.Each memory holds if the latent context belongs to its associated context region.
- Scoring: Responses are scored behaviorally on five independently rated dimensions, D1–D5, using conflict-type-specific anchors on a 0–4 scale.A response can score differently across dimensions because each dimension is evaluated independently.
4. Experiments
Experiments show that conflict recognition exceeds calibrated action and clarification, even with complete oracle memory. Pipeline extraction further limits performance by failing to preserve the relations that make conflicts interpretable, while distractors disproportionately harm post-detection reasoning.
- Oracle-track performance: The strongest model remains below 15/20 under both Judges despite receiving complete oracle memory.The oracle bank contains all canonical core memories and six same-domain distractors for each of 541 instances.
- Recognition-to-action gap: Models detect and describe conflicts more reliably than they calibrate confidence, seek targeted clarification, or choose reversible actions.D4 is the lowest-scoring dimension for every response model, and BOC shows high faithfulness alongside low diagnosis and clarification.
- Pipeline-track performance: Memory systems preserve complete target conflicts at sharply different rates: Letta 91.7%, Mem0 69.1%, A-mem 47.0%, and MemOS 46.2%.The resulting information loss ranges from 8.3% to 53.8%, because partial banks may omit trajectories, context partitions, or source comparisons.
- Pipeline-track performance: Pipeline responses remain comparatively faithful to received memories, but incomplete conflict observation produces greater variability in D3 and especially D4 than in D5.Conditional on FULL retrieval, D1 and D2 are relatively similar across systems; partial retrieval destroys relations needed to interpret otherwise answer-relevant content.
- Distractor-load effects: Distractor load lowers aggregate performance monotonically, with totals falling from 13.26 to 12.63 under GPT-5.4 and from 13.35 to 12.48 under Opus.D1 changes modestly, whereas D2–D4 decline more substantially as distractors increase from N=0 to N=6.
5. Conclusion … D Persona–Attribute Coverage
TANGLE evaluates memory agents under genuinely unresolvable conflict and reveals a consistent gap between recognizing conflict and acting appropriately. Reliable agents must preserve conflicting evidence and its relations, represent unresolved uncertainty, and choose flexible, evidence-sensitive actions.
- 5. Conclusion: TANGLE evaluates memory agents under genuinely unresolvable conflict across 541 instances, five response models, and four memory systems.The benchmark targets conflicts that cannot be resolved into a single definitive answer.
- 5. Conclusion: Models detect and describe conflict more reliably than they calibrate actions or seek clarification.This recognition-to-action gap is a consistent finding across the evaluation.
- 5. Conclusion: Memory extraction and distractor noise limit the conflict structure available for downstream reasoning.When conflict-bearing evidence and relations are not preserved, agents have less structure to reason over.
- 5. Conclusion: Fixed policies compress or sidestep conflict structure, motivating CAAP as a flexible, evidence-sensitive action policy.CAAP adapts action selection to the evidence that remains available.
- 5. Conclusion: Reliable memory agents should preserve conflicting evidence and its relations rather than force a definitive resolution.The conclusion frames conflict handling as retaining the structure needed for later reasoning and action.
- 5. Conclusion: Reliable agents should represent what remains unresolved and choose actions that reflect that uncertainty.This recommendation follows from the observed mismatch between conflict recognition and action calibration.
E Dataset Composition and Diversity … D. Persona–Attribute Coverage
The benchmark uses 40 internally identified personas, 46 conflict-oriented attributes, and 541 instances organized through sparse persona–attribute assignments and explicit conflict-structure schemas. Its coverage distinguishes contextual, behavioral, and source-based incompatibility, including SCC cases with two to four conflicting sources.
- I RQ3 Noise Interpretation Audit: The paper includes an evaluator-sensitive BOC response audit under RQ3.This appears as a dedicated subsection in the document contents.
- K Illustrative Instances: Illustrative instances cover BOC, CPC, SCC, and pipeline observability contrasts.These four subsections organize the paper’s examples and observability comparisons.
- A. Persona Definitions: 40 personas are sampled and curated from Persona Hub, normalized into a common schema, and lightly revised for benchmark use.The benchmark assigns internal identifiers rather than retaining formal source-profile IDs.
- A. Persona Definitions: Each persona receives only 13–15 life-relevant instances instead of the full persona×attribute product, with recurring pressures supporting ecological plausibility.Table 2 records the sparse roster and conflict-relevant grounding.
- B. Attribute Schema: The attribute schema contains 46 attributes grouped by life domain and assigned to BOC, CPC, or SCC conflict axes.The axis indicates the conflict type each attribute is designed to elicit.
- C. Conflict-Structure Schema: Conflict-structure schemas define CPC partition types, BOC oscillation drivers and shapes, and SCC source-conflict types across the 541 instances.These schemas specify the structural variables that make memories incompatible.
- C. Conflict-Structure Schema: 122 SCC instances contain two conflicting sources, 83 contain three, and 22 contain four.The source counts characterize the number of mutually conflicting sources in SCC cases.
- D. Persona–Attribute Coverage: Sparse persona–attribute coverage is detailed through per-persona counts and conflict-axis assignments.Tables 8 and 9 provide the corresponding roster-level and attribute-level views.
E. Dataset Composition and Diversity · F. Generation, Session Synthesis, and Quality Control
The benchmark contains 541 conflict instances across 40 personas, diversified by conflict type, axis-specific attributes, and distractor memories. Its generation pipeline constructs conflict-bearing sessions and applies structural, naturalness, and leakage checks to preserve valid, non-telegraphic conflicts.
- E. Dataset Composition and Diversity: The benchmark contains 541 conflict instances across 40 personas: 121 BOC, 193 CPC, and 227 SCC.Its schema includes 46 serialized attribute names and 48 axis-specific labels: 14 BOC, 18 CPC, and 16 SCC.
- E. Dataset Composition and Diversity: Oracle instances pair core memories with six same-domain distractors, producing 8,139 model-visible memories.The distractors are included alongside canonical instance memories for inference.
- E. Dataset Composition and Diversity: The benchmark’s conflict-structure vocabularies describe why memories conflict, complementing attribute schemas that identify the relevant life aspects.These vocabularies and their instance counts are reported in Tables 4–7.
- F. Generation, Session Synthesis, and Quality Control: Each selected persona–aspect pair generates 6–8 conflict-bearing memories, 1–2 neutral background memories, and natural user queries in constrained JSON.Diversity is promoted across personas, life aspects, and axis-specific structural values.
- F. Generation, Session Synthesis, and Quality Control: Same-domain distractors are generated separately and shuffled with conflict-bearing memories before inference.This construction prevents distractors from being distinguished solely by their generation process.
- F. Generation, Session Synthesis, and Quality Control: BOC and SCC queries omit temporal anchors and conflict language, while CPC queries omit the relevant partition variable.Thus, the latent conflict variable rather than surface wording determines the appropriate response strategy.
- F. Generation, Session Synthesis, and Quality Control: The pipeline track converts memories into multi-session conversations over roughly 10–18 months, placing BOC, CPC, and SCC evidence in distinct temporal or session contexts.Dialogue generation requires designated memory details to appear in user utterances, while assistants may provide support or follow-ups.
- F. Generation, Session Synthesis, and Quality Control: Quality control validates schema fields, counts, conflict validity, axis purity, naturalness, distractor calibration, and query non-leakage.Leakage auditing combines lexical prescans, model-based checks, and human spot-checks to detect wording or memories that telegraph resolution.
G. Scoring Rubric and Judge Protocol · H. Response and Evaluation Prompts · I. RQ3 Noise Interpretation Audit
The paper evaluates conflict-sensitive behavior with an independent 0–4 rubric rather than exact-match correctness, using prompts that hide conflict structure from response models while giving judges axis-specific evidence. Reliability results show substantial but imperfect agreement, and the RQ3 audit interprets distractor effects through a diagnostic matched case rather than a cohort-wide estimate.
- G. Scoring Rubric and Judge Protocol: Each applicable D1–D5 dimension receives an independent integer score from 0 to 4, separating conflict recognition, reasoning, calibration, clarification, and memory faithfulness.A response can identify conflict accurately while recommending overconfidently, or remain faithful to memory while failing elsewhere.
- G. Scoring Rubric and Judge Protocol: Judges cite decisive response text, score dimensions independently, and use only the memory bank, query, response, and evaluator-only clarification reference.The clarification reference identifies missing information relevant to information seeking but is hidden from the response model.
- G. Scoring Rubric and Judge Protocol: The rubric rewards context mapping, multi-context or source reasoning, calibrated uncertainty with conservative interim advice, targeted verification questions, and traceable factual claims.Higher scores require stronger axis-specific properties, including preserving competing sources and giving verification plans with decision rules.
- G. Scoring Rubric and Judge Protocol: For 62 high-risk SCC instances, D6 separately flags whether responses defer harmful action, provide conditional advice, or recommend consequential action on disputed evidence.D6 is ternary and is not added to the D1–D5 total.
- G. Scoring Rubric and Judge Protocol: On 2,780 paired dimension-level scores, annotators agreed exactly on 59.1%, within one ordinal point on 85.8%, with MAE 0.60 and quadratic-weighted κ=0.71.The reference set contained 556 records scored by two independent annotators.
- H. Response and Evaluation Prompts: Response models receive shuffled core memories and same-domain distractors with an underspecified query, but no conflict labels, provenance, or conflict annotations.Evaluators receive the same texts and query plus provenance labels, the axis-specific rubric, and a clarification reference.
- I. RQ3 Noise Interpretation Audit: The RQ3 matched-case audit finds broadly aligned distractor effects across evaluations, while BOC totals can rise from traceable details that improve D5 without improving conflict diagnosis.The case was selected from structured score differences before inspecting response text and is presented as diagnostic, not as a confirmatory full-cohort estimate.
I.1. Case A: evaluator-sensitive BOC response
In this BOC case, adding same-domain distractors made GPT-4o’s response more personalized and traceable but did not resolve the behavioral conflict. The comparison illustrates a traceability–diagnosis dissociation: noise can improve faithfulness or personalization while weakening conflict-sensitive reasoning.
- I.1. Case A: evaluator-sensitive BOC response: The decisive conflict is Riley’s repeated reuse of a fixed practice routine after prior lapses, not merely being busy.The evidence does not establish that work shifts, classes, or external disruption caused the lapses.
- I.1. Case A: evaluator-sensitive BOC response: The low-noise response suggests shorter practice blocks and scheduling changes but neither reconstructs the repeated failure nor asks which factor triggered the lapse.It therefore leaves external scheduling constraints versus routine-design failure unresolved.
- I.1. Case A: evaluator-sensitive BOC response: GPT-5.4 scores the high-noise response (3, 2, 1, 1, 4) for a total of 11.The response adds traceable personalization—such as pop arrangements, printed guides, visual demonstrations, daytime practice, and audio recording—but still omits the decisive lapse explanation.
- I.1. Case A: evaluator-sensitive BOC response: Both Judges found the added details grounded, yet the response became more personalized without becoming more diagnostic.The comparison indicates that generic scheduling offers can replace an explicit causal question, producing response-dependent score changes.
- I.1. Case A: evaluator-sensitive BOC response: This single case shows that conflict-sensitive dimensions can decline under noise even when faithfulness or personalization appears to improve.It does not establish that either Judge is uniquely correct or that all BOC cases behave identically.
J. RQ4 Policy Diagnostics · K. Illustrative Instances
RQ4 shows that CAAP selects conflict-sensitive action mixtures that broadly track unresolved evidence, while backbone-specific behavior confirms it is a policy baseline rather than an upper bound. The illustrative cases span complementary conflict mechanisms, and closely aligned judge scores support their interpretive consistency.
- J. RQ4 Policy Diagnostics: CAAP’s normalized action mixtures broadly track unresolved evidence despite receiving no explicit conflict-type label.Figure 7 is a mechanism audit rather than a standalone accuracy ranking, using only visible memories and the query.
- J. RQ4 Policy Diagnostics: BOC cases frequently receive reversible trials or conditionalized responses, reflecting uncertainty about the unresolved evidence.
- J. RQ4 Policy Diagnostics: Sonnet 5 shows the sharpest axis-sensitive specialization, with reversible trials in 68% of BOC actions, conditionalization in 52% of CPC actions, and verification in 74% of SCC actions.
- J. RQ4 Policy Diagnostics: GPT-4o uses conditionalization broadly, including 68% of BOC cases, whereas Gemini 3.1 Pro commits on 37% of BOC and 40% of SCC cases.
- K. Illustrative Instances: The ten illustrative cases are complementary, covering distinct temporal, contextual, and source-evidential mechanisms across BOC, CPC, and SCC.They include reversal, value tension, and loss-triggered switching for BOC; latent investment, next-day obligation, and audience for CPC; and several source-related mechanisms for SCC.
- K. Illustrative Instances: The two formal Judges give closely aligned total scores, with all selected model–case comparisons differing by at most one point except two stress cases differing by at most two.The exceptions are p38_email_triage and p40_event_scheduling, deliberately retained as stress cases.
- K. Illustrative Instances: Each case card contains the query, shortened memory bank, representative distractors, and induced pattern, while Oracle input uses complete canonical memories and six shuffled, unlabeled same-domain distractors.Memory bullets summarize each item’s functional contribution and are not extra metadata exposed to response models.
K.1. Behavior-Oscillation Conflict (BOC)
Behavior-Oscillation Conflict captures preferences that repeatedly switch between competing strategies rather than revealing one stable answer. The benchmark tests whether agents reconstruct these cycles, preserve unresolved trade-offs and exceptions, and seek targeted clarification before acting.
- Case B1: p10_budget_tracking: B1 shows three detailed budget-tracking attempts collapsing after 10–21 days into lightweight weekly checks, despite stable income and tools.The unresolved issue is whether Ethan wants lightweight monitoring, detailed tracking, or a hybrid, and what burden causes detailed tracking to fail.
- Expected behavior: Across BOC cases, appropriate behavior names the oscillation, avoids treating the newest strategy as definitive, preserves stable supports or exceptions, and asks targeted questions before proposing reversible interventions.The expected responses distinguish duration trends, control–sustainability trade-offs, and trigger-dependent switching rather than applying fixed binary rules.
- Case B2: p20_spending_discipline: B2 presents a recurring strict-friction versus values-flexibility cycle in which each pole solves one problem while creating another.Rigid rules cause exhaustion, whereas flexibility better fits stated goals but permits late-night impulse purchases; the minimum sustainable friction remains unresolved.
- Case B3: p38_email_triage: B3 shows three loss-triggered switches from normal email flow to strict triage and back, while the child’s tuition and aid messages remain an exception.The switches follow costly misses or near-misses, but the query does not establish whether Paula wants permanent minimal triage, temporary crisis control, or another safeguard.
K.2. Context-Partitioned Conflict (CPC)
Context-Partitioned Conflict tests whether agents identify hidden context variables rather than relying on broad domain or surface labels. The cases require conditional recommendations that preserve unresolved alternatives and seek the missing context governing effort, drinking, or communication style.
- CPC overview: CPC treats domain or calendar labels as unreliable proxies when personal meaning, next-day obligations, or audience and institutional power govern behavior.The benchmark contrasts surface categories with latent partition variables that determine appropriate action.
- Context-Partitioned Conflict (CPC): In scheduling, personal investment or relational importance warrants deep iterative effort, while low-meaning obligations receive efficient minimal effort.A roommate’s budget spreadsheet can become meticulous despite being a household task, so the agent should ask which tasks carry personal or relational stakes.
- Context-Partitioned Conflict (CPC): For drinking decisions, next-day duty or early obligations imply abstention, whereas a next-day off may permit moderate drinking; weekday labels alone are insufficient.The case is designed so a Friday recommendation fails before an early Saturday equipment check, making clarification about tomorrow’s commitments necessary.
- Context-Partitioned Conflict (CPC): For communication, audience and institutional power produce formal indirectness with local families or officials, blunt directness with professional peers, and candid speech at home.The unresolved issue is how school stress should modify respectful candor during the particular spouse-and-teenager dinner conversation.
K.3. Source-Contradiction Conflict (SCC) · K.4. Pipeline Observability Contrasts
Source-Contradiction Conflict cases require preserving competing claims, identifying unresolved authority or schedule slots, and seeking verification instead of forcing definitive actions. Pipeline observability distinguishes whether retrieved memory preserves all, some, or none of the conflict-bearing relations needed for evaluation.
- K.3. Source-Contradiction Conflict (SCC): SCC conflicts arise when formal records, firsthand reports, observed practice, or repeated verbal claims disagree and source formality or majority alone cannot resolve reliability.The cases include potentially misfiled medical records, an authenticated custody plan versus an error-prone app, and written classification evidence versus two verbal assurances.
- K.3. Source-Contradiction Conflict (SCC): The medication case leaves both the authorized plan and record identity unresolved, requiring claim mapping, safe coordination, and pediatrician-daycare reconciliation before issuing new instructions.Past ibuprofen purchases do not establish medical authorization.
- K.3. Source-Contradiction Conflict (SCC): The custody case separates baseline authority from voluntary exceptions, favoring provenance comparison and written or professional clarification over automatically controlling app or printed records.The app’s internal factual errors weaken confidence, while child-centered reasons for occasional adjustments remain genuine.
- K.3. Source-Contradiction Conflict (SCC): The banking case preserves documentary and verbal claims while requesting a formal written classification, rather than treating repeated verbal confirmation as authoritative.The unresolved issue is the bank’s current classification and its business-tier and loan-eligibility consequences.
- K.3. Source-Contradiction Conflict (SCC): The scheduling case must not merge incompatible dialysis memories into one calendar; it should verify the current routine before assigning other appointments.A combined plan could simultaneously schedule Tuesday/Thursday treatment and Monday/Wednesday/Friday recovery, producing an infeasible schedule.
- K.4. Pipeline Observability Contrasts: Pipeline evaluation asks whether retrieved memory preserves the target conflict, with Full meaning all conflict-bearing relations are visible, Partial meaning some survive but a decisive relation is missing, and None meaning the conflict is unobservable.None does not imply the retrieved bank is useless or unfaithful for the query; it makes D1 and D2 inapplicable.
- K.4. Pipeline Observability Contrasts: Table 12 reports observability states in the order Mem0 / Letta / A-mem / MemOS, abbreviated F, P, and N.The supplied passage provides the state order and abbreviations but no individual system-state values.