PACE: Towards Surfacing Hidden Conflicts in User Requests

• Personalized assistants must consider users’ circumstances and relevant evidence, avoiding both blind execution and unnecessary refusal.Apparently reasonable requests can conflict with prior commitments or companions’ constraints, while over-refusal harms usefulness and efficiency.Source: §1 · p. 1
• Existing risk benchmarks miss the challenge of retrieving decisive facts from large personal knowledge bases containing plausible distractors.Input-centric settings focus on risks explicitly present in the query, whereas personalized decisions require integrating distributed contextual evidence.Source: §1 · p. 1
• PACE introduces a retrieval-grounded dataset in which requests appear ordinary in isolation but become conflicting when hidden situational facts distributed across an egocentric knowledge base are retrieved.The decisive evidence is often not semantically apparent from the original query alone.Source: §1 · pp. 1–2
• PACEMAKER proposes multi-agent retrieval that reformulates queries, traverses evidence through graphs, and filters distractors to surface conflict-relevant facts.The framework targets latent conflict signals through structured, multi-hop, conflict-aware evidence extraction.Source: §1 · p. 2
• Experiments show that existing methods struggle to surface hidden situational constraints, while PACEMAKER consistently improves retrieval and reasoning performance over baselines.These findings establish hidden-conflict evidence retrieval as a substantial open challenge and support the effectiveness of conflict-aware retrieval.Source: §1 · p. 2

Personalized and Context-Aware Assistants

• Personalized assistants study user-specific understanding, long-term memory, and evolving preferences, while contextual safety examines how situational context affects request appropriateness.These directions typically focus on preference use or externally observable risks.Source: §2 · pp. 2–3
• PACE focuses on conflict-aware personalization, requiring models to compose distributed egocentric evidence and retrieve diagnostic facts that determine whether reasonable requests remain compatible with broader user context.This differs from prior retrieval objectives centered on relevant or supportive information for answering and personalization.Source: §2 · p. 3

Retrieval and Memory for Personalized LLMs

• Existing personalized LLM research spans memory systems, personalized alignment, and RAG, with graph-based methods improving evidence organization and multi-hop reasoning using user signals and structured personal knowledge.These methods also employ graph expansion mechanisms to organize and connect personal evidence.Source: §2 · p. 3

3.1 Task Definition

• Given a request and egocentric KB, the model decides whether to execute the action and justifies the decision using contextual evidence about the user and related individuals.Requests appear normal in isolation, while hidden situational constraints become apparent only through relevant KB facts.Source: §3.1 · p. 3

3.2 Feasibility Status and Situation Types

• Each instance is labeled as Conflict or Non-conflict and categorized as Temporal, Personal, or State according to the constraint requiring reasoning.Non-conflict cases remain feasible despite contextual information, preventing models from rejecting requests merely because a KB is present.Source: §3.2 · p. 3

3.3 Dataset Generation

• PACE is generated from expanded personas and ego–alter profiles, with ordinary requests and temporally consistent KB facts constructed so that decisions depend on hidden, distributed evidence.Gold contexts typically require jointly interpreting multiple facts, while judgments provide rationales for rejection or execution.Source: §3.3 · p. 4
• Gold and distractor contexts are atomized into independent KB facts, reflecting scattered personal knowledge and varying requirements for integrating evidence across instances.Distractors remain persona- and timeline-consistent while avoiding decisive conflict or compatibility signals.Source: §3.3 · p. 4
• PACE contains approximately 3.2K queries, with feasibility labels achieving 93.3% agreement with human judgments.Cases were manually reviewed and automatically filtered for taxonomy compliance, constraint satisfaction, diversity, and sufficient, unambiguous evidence.Source: §3.3 · p. 5

4 PACEMAKER

• PACEMAKER constructs dense and sparse indexes plus a k-NN document graph, then uses four stages to retrieve conflict-diagnostic documents from user-specific egocentric knowledge bases.The stages are conflict-aware query planning, hybrid retrieval with fusion, agentic multi-hop graph traversal, and conflict-aware document filtering.Source: §4 · p. 5

4.1 Conflict-Aware Query Planning

• Conflict-aware query planning generates up to three decision-relevant cues and counter-query views to retrieve latent conflict evidence absent from the original request.All query views are conditioned on a reference date to resolve temporal expressions correctly.Source: §4.1 · pp. 5–6

4.2 Hybrid Retrieval and Fusion

• Hybrid dense and sparse retrieval merges all query-view results with weighted reciprocal rank fusion, prioritizing counter-view evidence before pre-hop filtering selects traversal seeds.The pre-hop filter reduces noise and prevents misleading seed documents from propagating through graph expansions.Source: §4.2 · p. 6

4.3 Multi-Hop Graph Traversal

• Breadth-first multi-hop traversal expands filtered seed documents through top-M k-NN neighbors up to depth H, surfacing weakly query-related evidence near the seeds.This collects additional context that may not be directly retrievable from the original query.Source: §4.3 · p. 6

4.4 Conflict-Aware Evidence Selection

• A post-hop filter retains the top-N documents that most directly determine whether the request is feasible or conflicts with the user’s knowledge base.Filtering operates over both initial seed documents and documents discovered during traversal.Source: §4.4 · p. 6

4.5 Answer Generation

• The final evidence set is passed to an answer generator that states whether the request can be fulfilled and explains the decision’s reasoning.Implementation details and agent instructions are provided in Appendices B and E.Source: §4.5 · p. 6

5.1 Evaluation Metrics

• Retrieval performance is measured with Recall@K, Hit@K, Gold@K, and MRR at K=5 and K=10.These metrics assess recovery of gold documents, presence of at least one gold document, recovery of all gold documents, and reciprocal rank of the first relevant document.Source: §5.1 · p. 6
• Response quality uses GPT-5.4-mini to classify outputs as PASS, WRONG, or FAIL based on outcome correctness and rationale alignment, with PASS as the primary metric.A human evaluation of 600 instances with three annotators per instance achieved 93.5% agreement with the automatic judge.Source: §5.1 · pp. 6–7

Main results

• Oracle consistently achieves the highest PASS rate across both settings, while Full KB reaches only 57.49% open-source and about 73% across closed-source settings.The gap shows that unfiltered context substantially impairs conflict reasoning and that reasoning capability matters beyond context availability.Source: §6 · pp. 7–8
• PACEMAKER achieves a 68.82% PASS rate in the open-source setting, outperforming sparse retrieval at 62.73% and dense retrieval at 62.39%.The result supports targeted evidence selection through multi-hop traversal and conflict-aware filtering.Source: §6 · p. 8

Robustness analysis

• An independent Gemini 3.1 Flash-Lite judge evaluation was conducted to assess potential self-preference bias and confirm result robustness.Detailed results and model examples are provided in Appendices D.2 and D.3, while Appendix D.4 reports the independent-judge evaluation.Source: §6 · p. 8

7.1 Conflict Status vs. Non-conflict Status

• Conflict queries have lower PASS rates than Non-conflict queries across non-oracle methods, and this difficulty persists for Qwen and GPT even under Oracle settings.Across all three configurations, PACEMAKER leads non-oracle Conflict PASS by 11.40, 3.47, and 4.02 percentage points over the strongest baseline while maintaining comparable Non-conflict performance.Source: §7.1 · p. 8, Figure 3 · p. 8

7.2 Evidence Coverage

• Full coverage of all gold documents produces a clear additional PASS gain over partial coverage, because atomically distributed conflict evidence is rarely contained in one document.Retrieving only part of the evidence yields meaningful performance, but incomplete evidence is often insufficient for reliable conflict judgment.Source: §7.2 · pp. 8–9

7.3 Agent Component Ablation

• Removing multi-hop traversal reduces performance to 71.65% overall and 52.41% on Conflict queries, while removing query planning or evidence selection also causes consistent declines.The components reinforce one another: planning finds conflict-relevant seeds, traversal expands indirectly connected evidence, and selection retains decision-critical facts.Source: §7.3 · p. 9, Table 4 · p. 9

7.4 Comparison with Structured Retrieval Methods

• PACEMAKER achieves a substantially larger Conflict-query improvement than GraphRAG (Edge et al., 2024) and HippoRAG 2 (Gutiérrez et al., 2025), despite only slightly higher overall PASS.The comparison uses Qwen3-4B-Instruct-2507 with NV-Embed-v2; PACEMAKER targets decisive evidence through conflict-aware planning and filtering rather than topical relevance alone.Source: Table 5 · p. 9, §7.4 · p. 9
• PACEMAKER has lower cold-start cost than HippoRAG 2 because it uses no indexing-time LLM calls and incurs far fewer total calls.HippoRAG 2 is faster for online retrieval after indexing, but PACEMAKER is better suited to personalized knowledge bases initialized or refreshed frequently.Source: §7.4 · p. 9

Conclusion

• PACEMAKER outperforms retrieval-based baselines on PACE by combining conflict-aware query planning, hybrid retrieval, multi-hop graph traversal, and conflict-aware evidence selection.PACEMAKER is a training-free multi-agent retrieval framework designed to identify decision-relevant evidence from egocentric knowledge bases.Source: §8 · p. 9
• Retrieval completeness is particularly critical for Conflict queries, because missing evidence substantially degrades reasoning quality.The findings indicate that substantial room remains for future work on conflict-aware reasoning and retrieval.Source: §8 · p. 9

Limitations

• The benchmark isolates latent-conflict detection but does not test complex downstream tasks such as recommendation, scheduling, or planning.Future benchmarks could add richer persona worlds, actionable entities, and dynamic task environments to evaluate end-to-end personalized task solving.Source: Limitations · p. 10
• PaceMaker is evaluated without task-specific training, leaving specialized training for query reformulation, evidence selection, graph traversal, and decision calibration as future work.The training-free setup highlights the framework’s generality, while specialized agents could potentially improve performance.Source: Limitations · p. 10

Ethics Statement

• The dataset contains synthetic identity profiles and no real personal information, but its realistic contexts should not be used for privacy-invasive profiling, manipulation, or inappropriate personalization.The benchmark includes sensitive or potentially unsafe situations, so users should exercise caution when interpreting or extending the data.Source: Ethics Statement · p. 10
• The authors filter inherently harmful, unreasonable, or unethical requests and use large language models for data generation assistance and minor language polishing, while leading the research process themselves.Source: Ethics Statement · p. 10

• Additional examples cover persona expansion, profile synthesis, Conflict queries, and Non-conflict queries in Tables 9–13.Source: A Dataset Details · p. 14
• Table 6 presents PACE’s query distribution by situation type.Source: Table 6 · p. 14

• PACEMAKER is training-free and combines multiple agent components without additional fine-tuning.Source: B Implementation Details · p. 14
• The system builds a cosine-similarity k-NN document graph with k = 10, generates up to 3 conflict dimensions and counter queries, and retrieves top-K = 10 documents per dense or sparse query view.Source: B Implementation Details · p. 14
• Weighted Reciprocal Rank Fusion uses k = 60, weights counter-query results at 1.2 versus 1.0 for original queries, and passes top-N_seed = 20 documents to pre-hop filtering.Source: B Implementation Details · p. 14
• BFS traversal starts from 10 filtered seeds, expands 3 neighbors per hop for up to H = 5 hops, and selects 10 final evidence documents.Source: B Implementation Details · p. 14
• FAISS supports dense retrieval, BM25 supports sparse retrieval, vLLM serves open-source agents, and experiments run on four NVIDIA RTX A6000 GPUs.Source: B Implementation Details · p. 14
• Sensitivity analysis shows the experimental configuration achieves the strongest results, while performance remains stable across nearby traversal and retrieval settings.Source: B Implementation Details · p. 14

C Human Evaluation Details

• Human-evaluation quality was supported by qualified annotators, three assessments per instance, majority voting, and 72.5% full agreement among annotators on response-quality labels.Participants had more than 10,000 approved HITs and approval rates above 98%; the remaining response-quality cases received 2-of-3 agreement.Source: C Human Evaluation Details · p. 14, §C.2 · p. 15

C.1 Feasibility Status

• PACE feasibility statuses achieve 93.3% overall agreement with human annotations, including 98.3% for Non-conflict and 88.3% for Conflict queries.Overall inter-annotator agreement is 86.7%; disagreements primarily reflect annotators’ more permissive interpretations of Conflict feasibility than the construction criteria.Source: §C.1 · p. 15

C.2 Response Quality

• The automatic judge aligns with human judgments at 93.5% overall, with agreement rates of 97.0% for PASS, 90.5% for WRONG, and 93.0% for FAIL.The lower WRONG agreement reflects differing strictness about rationale alignment, consistent with the judge’s intentionally strict gold-rationale criterion.Source: §C.2 · p. 15

D.1 Embedding Model Selection

• Qwen3-Embedding-8B, BGE-M3, and NV-Embed-v2 produce very similar retrieval and overall performance, so Qwen3-Embedding-8B is selected for the open-source setting.The selection reflects its best overall performance and shared model family with the Qwen3-4B-Instruct-2507 agent, while results suggest conflict-aware reasoning and agentic filtering also matter substantially.Source: §D.1 · p. 15

D.2 Detailed Results

• Temporal queries consistently have the lowest PASS rates across Qwen, GPT, and Gemini configurations.Temporal conflicts require aggregating multiple related facts about routines, schedules, or commitments rather than relying on one decisive evidence item.Source: §D.2 · p. 15

D Further Analysis

• The analysis attributes temporal-query difficulty to relevant information being distributed across several facts that may appear individually unimportant from the original query.Resolving these conflicts therefore requires reasoning over multiple temporally related facts and performing evidence aggregation.Source: §D.2 · p. 15

D.3 Model Outputs

• Model-output examples are organized by situation type, with separate tables for temporal, personal, and state queries.Tables 18, 19, and 20 provide the corresponding temporal-, personal-, and state-type response examples.Source: §D.3 · p. 15

D.4 Cross-Model Judge Evaluation

• 86.40% agreement was observed between Gemini 3.1 Flash-Lite and GPT-5.4-mini judges, with nearly identical overall, Conflict, and Non-conflict FAIL rates.Disagreements mainly concerned PASS versus WRONG labels, indicating similar failure detection but slightly different rationale-quality criteria.Source: §D.4 · p. 16, Table 8 · p. 16

D.5 Ablation Settings

• PACEMAKER’s ablations remove planning, multi-hop traversal, or post-hop evidence selection to assess the contribution of each agent component.The full model generates counter queries, expands candidates through graph hops, and filters post-traversal evidence; each ablation bypasses one corresponding stage.Source: §D.5 · p. 16
• Ablation outcomes are reported in Table 21, with feasibility-status results separately provided in Table 22.These tables define the reported evaluation scope for the component-removal study, but the supplied passages do not provide their cell values.Source: §D.5 · p. 16

D.6 Comparison with Other Methods

• The comparison evaluates PACEMAKER against HippoRAG 2 (Gutiérrez et al., 2025) and GraphRAG (Edge et al., 2024) under a controlled Qwen3-4B-Instruct-2507 LLM backbone and NV-Embed-v2 embeddings.The shared backbone is used wherever LLM inference is required, while the embedding model supports all compared implementations.Source: §D.6 · p. 16
• All methods independently index each egocentric KB instance and use the same task-specific answer generator to judge request feasibility relative to the KB and reference date.This normalization addresses the incompatibility between general-purpose RAG outputs and the task’s required feasibility judgment.Source: §D.6 · p. 16
• HippoRAG 2 uses its standard Personalized PageRank retrieval with a top-10 passage budget, enabling both retrieval-metric and response-quality reporting.GraphRAG preserves its standard indexing and local-search workflow, but its synthesized local-search output is passed to the shared generator and supports answer-quality reporting only.Source: §D.6 · p. 16
• GraphRAG local search averaged 16.3 seconds, had a 14.0-second median, and exceeded the 300-second timeout for 94 of 3,249 queries (2.89%).The full comparison appears in Table 23, with feasibility-status results in Table 24; timed-out queries were treated as having no usable returned context.Source: §D.6 · pp. 16–17

D.7 Computational Cost Analysis

• PACE-MAKER achieves a 215.10-second cold-start total, compared with 310.99 seconds for HippoRAG 2 and 741.44 seconds for GraphRAG, despite differing indexing and retrieval tradeoffs.PACE-MAKER uses 80.78 seconds for indexing and 7.46 seconds per query, versus HippoRAG 2’s 290.36 and 0.80 seconds and GraphRAG’s 489.37 and 14.00 seconds.Source: §D.7 · p. 17, Table 25 · p. 27
• PACE-MAKER eliminates indexing-time LLM calls and uses 72 retrieval calls across 18 queries, while HippoRAG 2 uses 4,096 indexing and 18 retrieval calls.PACE-MAKER’s four calls per query are assigned to the Conflict Planner, Multi-View Generator, Pre-hop Filter Agent, and Post-hop Filter Agent, shifting computation toward lightweight indexing.Source: §D.7 · p. 17

E Prompts

• Persona-expansion prompts require coherent, neutral third-person profiles that add contextual details without inventing explicit reasons, conflicts, risks, judgments, or dramatic traits.Alter expansion additionally grounds the subcharacter in the main character’s environment, while profile synthesis produces structured JSON with dense, internally coherent identity, lifestyle, health, and technology details.Source: E Prompts · pp. 28–30
• Query-generation prompts create temporal, personal, and state cases whose ordinary requests remain executable but become conflicting or non-conflicting only through concrete, persona-grounded, temporally knowable context.Conflict cases require contextual incompatibility rather than intrinsic danger, while non-conflict cases must remain compatible despite relevant context; invalid cases are discarded when they rely on generic norms, unknown future facts, shallow relationships, or context-independent violations.Source: E Prompts · p. 31
• All contextual facts must be plausibly knowable to ego by ref_date, with alter-related knowledge requiring an explicit or plausible source and no assumed private information.The validator also checks timeline validity and epistemic validity separately.Source: E Prompts · pp. 36–37
• Cases must appear normal and feasible without hidden context, while conflict or non-conflict emerges only after reasoning over concrete persona- or alter-specific facts.Ego must not knowingly request something that contradicts a stable personal constraint, and cases must remain technically executable rather than impossible.Source: E Prompts · pp. 36–37
• Validation requires contexts to contain all necessary, minimal facts, avoid unsupported reasoning, and make the decisive conflict or compatibility reason explicit enough for downstream evaluation.Judgments should summarize logic already supported by the context, while irrelevant or decorative details are removed.Source: E Prompts · pp. 34, 36, 37
• Relative dates are resolved against ref_date before validation, and requests resolving to past times are invalid or rewritten when a natural future-valid formulation is possible.The rules distinguish potentially valid upcoming references from past or ambiguous bookings and scheduling requests.Source: E Prompts · p. 37
• Situation types must remain primary and distinct: temporal cases concern scheduling, preparation, travel, sequencing, or coordination, while personal cases require meaningful constraints, boundaries, welfare concerns, or obligations.Temporal conflicts should focus on situational fit rather than pure impossibility, and alter-involved examples may center on concrete coordination or caregiving responsibilities.Source: E Prompts · pp. 37–38
• Personal cases must be ego-plausible and should not have ego knowingly request something that directly violates a stable self-constraint; stronger cases involve alter needs or situationally activated constraints.Cases depending on out-of-character behavior or ignored known facts should be rewritten or dropped.Source: E Prompts · p. 39
• Across the full set, duplicate hidden reasoning drivers must be removed or rewritten, favoring cases that are more persona-grounded, causally clear, natural, timeline-valid, and coverage-diverse.Invalid cases include obvious unsafe or self-contradictory requests, unsupported or padded contexts, implausible alter knowledge, generic cases, type mismatches, and past-resolving relative dates.Source: E Prompts · pp. 39–40
• Validators assign keep, rewrite, or drop actions with scores from 0–10, while rewrite prompts require preserving the core idea and repairing the primary issue with concise, context-grounded outputs.Rewritten cases must include only necessary facts, one primary judgment driver, explicit conflict or compatibility reasoning, and valid situation_type and label fields.Source: E Prompts · pp. 40–41
• Cases must remain normal and technically feasible, with conflicts arising only from context-supported facts that the ego could plausibly know by ref_date.Future facts must be scheduled, announced, or otherwise knowable at ref_date; unsupported future outcomes, missing knowledge bridges, and past-resolving target times invalidate cases.Source: E Prompts · pp. 38, 41, 42
• Temporal cases center on scheduling, sequencing, preparation, travel, or coordination conflicts rather than pure impossibility, including concrete duties or shared plans involving alters.Personal cases require meaningful persona-linked constraints such as health, accessibility, caregiving, welfare, ethical commitments, or alter-specific needs, not mere dislike or generic impropriety.Source: E Prompts · p. 43
• State cases use external operating conditions known by ref_date—such as access restrictions, route diversions, infrastructure work, or building conditions—that make requests risky, wasteful, disruptive, or poorly matched.They should not rely solely on closure, outage, stock failure, outright denial, or unannounced future conditions.Source: E Prompts · p. 38
• Distractors must resemble plausible, query-adjacent knowledge-base fragments while providing little evidence for determining the target label, reproducing the gold judgment, or producing the final answer.They should be retrieval-relevant but decision-irrelevant, grounded in concrete harmless timeline anchors rather than generic biography.Source: E Prompts · pp. 43, 45
• Prompts prohibit distractors from restating or implying conflict-blocking evidence or non-conflict compatibility evidence, including exact feasibility, availability, access, timing, or restriction facts.For non-conflict cases, distractors must not directly answer practical questions such as venue hours, routes, reservations, companion approval, or calendar availability.Source: E Prompts · pp. 44, 46
• All generated contexts and facts must be knowable to Ego by ref_date, with future information allowed only when scheduled, announced, reserved, posted, planned, or otherwise known in advance.The prompts also restrict alter-related information to facts Ego could reasonably know and require neutral, factual, self-contained outputs.Source: E Prompts · p. 44
• Each distractor must remain compatible with the entire shared egocentric scenario and avoid introducing contradictions, decisive enabling conditions, or new reasons to change any case outcome.The full scenario is treated as one coherent world, including compatibility across distractors generated for different cases.Source: E Prompts · pp. 44–45
• The atomization prompt converts each source context into 2 to 5 standalone atomic facts that contain one neutral claim and preserve evidence for later reasoning without stating conclusions.Atomic facts should remain independent stored knowledge units, while scenario-level relevance emerges only when multiple facts are composed with the query or other information.Source: E Prompts · pp. 48–49
• Gold-context prompts use the context as primary evidence and forbid copying judgments, whereas distractor prompts preserve harmless anchors without introducing blocking constraints or conflict conclusions.Gold outputs must retain all minimally necessary neutral evidence for reconstructing the case judgment; distractor facts must remain limited to the distractor context.Source: E Prompts · p. 52
• Context atomization extracts minimal, standalone, neutral facts with explicit entities and necessary time anchors, while preserving meaning without mechanically copying source wording.Facts contain one claim, avoid unresolved references and scenario conclusions, and retain enough evidence for later reasoning without performing that reasoning.Source: E Prompts · pp. 49, 50, 51, 53
• The Conflict Planner Agent selects one to three request-grounded retrieval directions spanning schedules, commitments, relationships, travel, preferences, health, resources, and ambiguous context.Directions may provide feasibility evidence or reveal constraints and need not directly imply conflict, but must have plausible retrieval value and exact request anchors.Source: E Prompts · p. 54
• The Multi-View Generator creates distinct counter queries targeting conflict-bearing factors, while pre-hop and post-hop filters retain person-, time-, place-, or state-specific documents that may become useful after evidence combination.Filtering prioritizes constraints, commitments, availability, preferences, relationships, dependencies, and bridge evidence without requiring documents to explicitly prove conflict or feasibility.Source: E Prompts · pp. 55–56
• The answer-generator prompt supplies persona facts and a query, requiring a direct 2–4 sentence decision and explanation grounded only in those facts.It instructs the assistant to act in the persona’s world, avoid mentioning context or permissions, and return plain prose rather than labels, bullets, or JSON.Source: E Prompts · p. 57
• The response-evaluation judge first checks whether the model’s decision matches the gold conflict label, marking wrong or ambiguous decisions as FAIL.Non-conflict responses must proceed or agree, whereas conflict responses must refuse, warn, block, or state that the request should not be carried out.Source: E Prompts · p. 58
• Only after a correct decision does the judge assess whether the rationale captures the gold rationale’s core reason, assigning PASS, WRONG_RATIONALE, or FAIL.Exact wording is unnecessary, but missing, vague, unsupported, or substantially incorrect reasoning can produce WRONG_RATIONALE or FAIL; decision correctness takes priority.Source: E Prompts · pp. 58–59