PaperlayerView exact revision on arXiv

Source-linked AI summary

PACE: Towards Surfacing Hidden Conflicts in User Requests

Yoojin Kim, Jihyoung Jang, Hyounghun Kim

arXiv:2609.03293v1cs.CL

TL;DR

Existing assistants and benchmarks often omit whether apparently reasonable requests conflict with implicit personal circumstances distributed across a knowledge base. This paper introduces PACE and PACEMAKER to retrieve decisive evidence and evaluate conflict decisions, with experiments showing consistent improvement over existing approaches. The benchmark primarily targets feasibility judgment rather than full downstream task execution.

  • Problem

    Existing conflict and safety benchmarks focus on explicit input risks, while personalized assistants must integrate hidden, distributed facts from egocentric knowledge bases to judge request appropriateness.

  • Method

    The paper introduces PACE and PACEMAKER, a training-free multi-agent framework using conflict-aware query planning, hybrid retrieval, multi-hop graph traversal, and evidence filtering.

  • Results

    PACEMAKER consistently outperforms retrieval-based baselines on PACE, and retrieval completeness is particularly critical for Conflict queries.

  • Takeaways & Limitations

    PACE provides an evaluation setting for conflict-aware reasoning, while the results identify decision-relevant evidence retrieval as central to conflict judgments.

  • Takeaways & Limitations

    PACE primarily evaluates feasibility judgment rather than complex downstream execution, and PACEMAKER is evaluated without task-specific training.

Abstract

from arXiv · show

Personalized assistants should not only comply with user requests but also assess whether those requests are appropriate given the user's current circumstances. However, prior work has primarily focused on accurately executing requests, overlooking the need for assistants to account for context and engage in conflict-based refusal. Furthermore, while existing work on conflict or safety detection relies on explicitly provided factors, real-world scenarios often involve implicit factors that must be retrieved from a knowledge base (KB). To this end, we introduce Personalized Assistants for Conflict Evaluation (PACE), a dataset for evaluating whether models can identify latent constraints, expressed as egocentric knowledge or events, that render seemingly reasonable user requests inappropriate. PACE pairs user requests grounded in well-defined personas with egocentric KB facts, requiring models to integrate contextual evidence to determine whether a request is conflicting. This implicit retrieval setting hinders the direct association between user requests and conflict-inducing knowledge, making it difficult for existing models to identify relevant user-specific facts. To address this challenge, we further propose PaceMaker, a multi-agent framework in which specialized agents coordinate across query reformulation, multi-hop graph traversal, and conflict-aware filtering to retrieve contextually decisive evidence. Experiments on PACE evaluate both evidence retrieval quality and conflict decision accuracy, showing that PaceMaker consistently outperforms existing approaches.

1 Introduction

PACE addresses personalized-assistant decisions where ordinary-looking requests become inappropriate only after hidden personal and situational constraints are retrieved from an egocentric knowledge base. The paper introduces PACE and PACEMAKER, and reports improved retrieval and reasoning over baselines.

  • Personalized assistants must assess whether requests are appropriate given personal circumstances and surrounding conditions, not merely execute them.
  • PACE evaluates conflict-aware reasoning over egocentric knowledge bases whose decisive facts are distributed and rarely retrievable from the original query alone.
  • PACEMAKER reformulates queries, traverses evidence through a graph, and filters distractors to surface decision-relevant conflict facts.
  • The paper introduces PACE as a dataset for evaluating conflict-aware reasoning over egocentric knowledge bases.
  • Experiments show that surfacing hidden situational constraints remains a substantial open challenge, while PACEMAKER consistently improves retrieval and reasoning performance over baselines.

2 Related Work

Related work spans personalized assistants, contextual safety, memory, personalized alignment, and retrieval-augmented generation. The paper positions conflict-aware reasoning over distributed egocentric knowledge as a distinct challenge.

  • Personalized-assistant research examines user-specific understanding, long-term memory, and adaptation to evolving preferences.
  • Contextual-safety research studies how request appropriateness depends on surrounding situational context.
  • PACE focuses on conflict-aware reasoning over personal context distributed across many atomic facts in an egocentric knowledge base.
  • Personalized LLM research includes memory systems, personalized alignment, and retrieval-augmented generation.

3 PACE

PACE is a dataset for deciding whether ordinary user requests conflict with hidden facts in an egocentric knowledge base. It constructs synthetic, distributed contexts and validates the resulting feasibility judgments.

  • PACE evaluates whether a request is compatible with facts about the user and closely related individuals stored in an egocentric knowledge base.
  • Each instance pairs a request and egocentric knowledge base with conflict or non-conflict feasibility labels and a situation type.
  • PACE categorizes conflicts as Temporal, Personal, or State according to scheduling constraints, personal constraints, or current external conditions.
  • Dataset construction makes requests executable in isolation while distributing the decisive evidence across multiple contextual facts.
  • Approximately 3.2K queries remain after verifying that decomposed gold facts are sufficient and unambiguous for resolving each query.
  • Human judgments agree with the generated feasibility labels at a rate of 93.3%.

4 PACEMAKER

PACEMAKER is a training-free multi-agent framework for retrieving conflict-relevant evidence from large egocentric knowledge bases. It combines query planning, hybrid retrieval, graph traversal, and final evidence filtering before generating a justified decision.

  • PACEMAKER processes thousands of atomic facts through conflict-aware query planning, hybrid retrieval, multi-hop graph traversal, and document filtering.
  • Query reformulation creates conflict-diagnostic probing cues and counter views when the original request does not expose the needed evidence.
  • Dense and sparse retrieval results are merged with Weighted Reciprocal Rank Fusion, assigning higher weights to counter-view results.
  • Multi-hop traversal expands filtered seed documents through a k-NN graph to find weakly related evidence near those seeds.
  • A post-hop filter retains the top-N documents that most directly determine whether the request is feasible or conflicting.
  • The final evidence set is passed to an answer generator that states whether the request can be fulfilled and explains the decision.

5 Experiments

The experiments evaluate retrieval quality and response quality on PACE using multiple retrieval baselines, model configurations, and human validation of automatic judgments.

  • Evaluation Metrics: Retrieval quality is measured with Recall@K, Hit@K, Gold@K, and MRR.These metrics assess recovered gold documents, whether any or all gold documents appear, and the rank of the first relevant document.
  • Evaluation Metrics: Response quality compares model outputs with gold rationales using GPT-5.4-mini as an automatic judge.The judge evaluates both the correctness of the outcome and alignment between the response rationale and the gold rationale.
  • Human Validation: Human evaluation samples 600 instances, uses three annotators per instance, and determines final labels by majority vote.The evaluation checks whether assigned labels are correct.
  • Human Validation: 93.5% agreement between human annotators and the automatic judge supports the reliability of the automatic evaluation.Further evaluation details are provided in Appendix C.2.
  • Experimental Setup: The study compares PACEMAKER with sparse and dense retrieval, Oracle, and Full KB settings across open-source and closed-source configurations.The configurations use Qwen, GPT, and Gemini models with corresponding embedding models; Oracle and Full KB do not perform retrieval.

6 Results

PACEMAKER achieves the strongest reported non-oracle response quality on PACE, while unfiltered context remains substantially below the Oracle upper bound. Its advantage is especially pronounced for conflict queries, where targeted evidence retrieval improves performance over retrieval baselines.

  • Main Results: Oracle consistently achieves the highest PASS rate, while Full KB reaches 57.49% in the open-source setting and about 73% across closed-source settings.Both Full KB results are below Oracle, indicating that access to all context does not match access to directly relevant evidence.
  • Main Results: PACEMAKER reaches a 68.82% PASS rate in the open-source setting, versus 62.73% for sparse retrieval and 62.39% for dense retrieval.It outperforms all retrieval-based baselines in this setting.

7 Analysis

The analysis shows that conflict reasoning depends on complete, decision-relevant evidence and that PACEMAKER’s coordinated retrieval components outperform retrieval baselines. Conflict queries remain difficult even with directly provided facts, while PACEMAKER maintains comparable non-conflict performance.

  • 7.1 Conflict Status vs. Non-conflict Status: Conflict queries have lower PASS rates than Non-conflict queries across non-oracle methods, with the gap persisting under Oracle in Qwen and GPT configurations.This indicates that conflict resolution remains difficult even when relevant facts are directly provided.
  • 7.1 Conflict Status vs. Non-conflict Status: PACEMAKER improves Conflict PASS by 11.40, 3.47, and 4.02 percentage points over the strongest non-oracle baseline across the three configurations.It achieves the highest Conflict PASS rate among non-oracle methods while maintaining comparable Non-conflict performance.
  • 7.2 Evidence Coverage: Full gold evidence coverage produces a clear additional PASS gain over partial coverage because conflict-relevant facts are atomically distributed across the knowledge base.Retrieving only part of the required facts can yield meaningful performance, but incomplete evidence is often insufficient for reliable conflict judgment.
  • 7.3 Agent Component Ablation: Removing multi-hop traversal causes the largest ablation drop, reaching 71.65% overall and 52.41% on Conflict queries.Removing query planning causes a comparable decline, while removing evidence selection produces a smaller but consistent drop.
  • 7.3 Agent Component Ablation: Query planning, traversal, and evidence selection are mutually reinforcing: they retrieve conflict-relevant seeds, expand indirectly connected evidence, and filter for decision-critical facts.The ablation results show that removing any component consistently degrades performance.
  • 7.4 Comparison with Structured Retrieval Methods: PACEMAKER’s main advantage over structured retrieval methods appears on Conflict queries, where topical relevance alone fails to recover the full set of conflict-inducing constraints.PACEMAKER uses conflict-aware query planning and targeted evidence filtering to surface latent constraints needed for the final decision.
  • 7.4 Comparison with Structured Retrieval Methods: PACEMAKER incurs lower cold-start cost than HippoRAG 2 because it uses no LLM calls during indexing and far fewer calls in total.HippoRAG 2 offers faster online retrieval once its index is built, whereas PACEMAKER is advantageous when personalized knowledge bases are initialized or refreshed frequently.

8 Conclusion

The paper introduces PACE to evaluate conflict-aware reasoning over egocentric knowledge bases and PACEMAKER as a training-free multi-agent retrieval framework. Experiments show that PACEMAKER outperforms retrieval-based baselines, while retrieval completeness remains especially important for Conflict queries.

  • PACE evaluates whether models can determine if requests conflict with personal context distributed across thousands of atomic facts.
  • PACEMAKER retrieves decision-relevant evidence through conflict-aware query planning, hybrid retrieval, multi-hop graph traversal, and conflict-aware evidence selection.
  • PACEMAKER outperforms retrieval-based baselines, while missing evidence substantially degrades reasoning quality on Conflict queries.
  • The results indicate substantial room for future work on conflict-aware reasoning and retrieval.

Limitations

The benchmark primarily evaluates feasibility judgment rather than complete personalized task execution, and the method is evaluated without task-specific training.

  • The benchmark does not evaluate complex downstream tasks such as recommendation, scheduling, or planning.
  • The training-free evaluation leaves future work open to training specialized agents for query reformulation, evidence selection, graph traversal, and decision calibration.

Ethics Statement

The dataset uses synthetic identity profiles without real personal information, while its fictional profiles and knowledge bases are designed to emulate realistic personal contexts.

  • The dataset contains synthetic identity profiles and no real personal information.
  • The profiles and knowledge bases should not be used for privacy-invasive profiling, manipulation, or inappropriate personalization.
  • The benchmark filters out inherently harmful, unreasonable, or unethical requests, but sensitive or potentially unsafe situations still require cautious interpretation.

A Dataset Details

PACE includes a distribution of queries by situation type, with additional examples organized by persona expansion, profile synthesis, and conflict status.

  • Additional examples cover persona expansion, profile synthesis, Conflict queries, and Non-conflict queries.
  • Table 6 reports the query distribution of PACE by situation type.

B Implementation Details

PACEMAKER is a training-free multi-agent system that combines conflict-aware query planning, hybrid retrieval, graph traversal, and evidence filtering. Its experiments use fixed retrieval and traversal hyperparameters, with nearby configurations remaining stable.

  • PACEMAKER is training-free and uses multiple agent components without additional fine-tuning.
  • Indexing: The index encodes KB documents and connects each document to its 10 nearest neighbors using cosine similarity.
  • Query Planning: The conflict planner generates up to 3 conflict dimensions and up to 3 counter queries alongside the original query.
  • Hybrid Retrieval: Hybrid retrieval combines top-10 dense and sparse results with WRRF, weighting counter-query results 1.2 and original-query results 1.0 before selecting 20 seeds.
  • Multi-Hop Graph Traversal: Pre-hop filtering selects 10 seeds for BFS traversal up to 5 hops, expanding 3 neighboring documents per hop.
  • Evidence Selection: Post-hop filtering selects the final 10 documents from the traversed pool for the answer generator, while nearby hyperparameter configurations remain stable.

C Human Evaluation Details

Human evaluations assess both PACE feasibility labels and PACEMAKER response-quality judgments. The reported results show strong agreement overall, while temporal queries and embedding-model choices receive additional analysis.

  • Feasibility Labels: Each feasibility evaluation uses three annotators, with the final label determined by majority vote and participants meeting strict approval criteria.
  • Feasibility Labels: Human annotation tests sample 120 queries across temporal, personal, and state situations, balancing Conflict and Non-conflict cases.
  • Feasibility Labels: 93.3% overall agreement with human annotations supports the feasibility-status assignments, with 98.3% agreement for Non-conflict and 88.3% for Conflict queries.
  • Feasibility Labels: Disagreements concentrate among Conflict queries, where annotators sometimes interpret feasibility more permissively than the construction pipeline.
  • Response-Quality Judgments: 72.5% of 600 response-quality instances receive full agreement among three annotators, while the remaining cases receive 2-out-of-3 agreement.
  • Response-Quality Judgments: The automatic judge agrees with human annotators on 93.5% of instances, with label-specific agreement of 97.0% for PASS, 90.5% for WRONG, and 93.0% for FAIL.

D.4 Cross-Model Judge Evaluation

Cross-model evaluation compares PACEMAKER’s automatic judgments and retrieval framework under controlled settings. Alternative judges agree strongly overall, while ablations and baseline comparisons isolate the roles of its components and design choices.

  • Cross-Model Judge Evaluation: Gemini 3.1 Flash-Lite and GPT-5.4-mini judges agree on 86.40% of PACEMAKER evaluations across PASS, WRONG, and FAIL.
  • Cross-Model Judge Evaluation: The two judges show nearly identical FAIL rates overall and across Conflict and Non-conflict queries, with disagreements concentrated between PASS and WRONG.
  • Ablation Study: The ablation study removes conflict-aware planning, multi-hop traversal, or post-hop selection to evaluate the three core agent components.
  • Baseline Comparison: Comparisons with HippoRAG 2 and GraphRAG use Qwen3-4B-Instruct-2507 as the controlled LLM backbone and NV-Embed-v2 as the common embedding model.
  • Baseline Comparison: All methods use a shared task-specific answer generator because general-purpose RAG output formats are not directly comparable for feasibility judgments.
  • Baseline Comparison: GraphRAG local search outputs are evaluated only for answer quality because they do not return ranked document lists aligned with the corpus.
  • Baseline Comparison: GraphRAG local search exceeds the 300-second timeout on 94 of 3,249 queries, or 2.89%, among the reported runs.

D.7 Computational Cost Analysis

The analysis compares offline indexing, online retrieval, cold-start latency, and LLM-call costs across structure-aware methods.

  • Evaluation setup: The computational-cost evaluation measures latency and LLM-call counts for offline indexing and online retrieval on a 2,056-fact, 18-query instance.All methods use the same downstream answer-generation and evaluation modules.
  • Latency: 80.78 seconds for offline indexing and 7.46 seconds per query for online retrieval give PACEMAKER a 215.10-second cold-start total.HippoRAG 2 requires 290.36 seconds for indexing and 0.80 seconds per query, while GraphRAG requires 489.37 seconds and 14.00 seconds, respectively.
  • LLM calls: PACEMAKER makes no LLM calls during indexing and 72 calls during retrieval across 18 queries.Its four calls per query correspond to the Conflict Planner, Multi-View Generator, Pre-hop Filter Agent, and Post-hop Filter Agent.
Loading 2609.03293v1…