PaperlayerView exact revision on arXiv

Source-linked AI summary

MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation

Jungyeon Lee, Kangmin Lee, Taeuk Kim

arXiv:2507.21544v3cs.CL

TL;DR

RAG systems can encounter inconsistent retrieved documents or knowledge that conflicts with a model’s parametric knowledge, while existing benchmarks have limited conflict types and construction strategies. MAGIC uses knowledge graphs to benchmark varied inter-context conflicts; experiments find that models struggle with conflict detection, particularly for multi-hop cases, and often fail to locate contradictions.

  • Problem

    Existing conflict benchmarks often focus on question answering, use entity substitution, and provide limited analysis of multi-hop conflicts and disagreements among input documents.

    Source: §1 · pp. 1–2

  • Method

    MAGIC uses knowledge-graph structure to benchmark inter-context conflicts, including cases with multiple conflicts and multi-hop reasoning.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    LLMs remain imperfect at conflict detection, especially in multi-hop cases, and often fail to pinpoint the exact conflict.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Explicit graph relations support interpretable representation of conflicting entities and relations for structured analysis.

    Source: §1 · p. 2

    MAGIC is built on Wikidata-based KGs, and localization evaluation currently relies on human judgment.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Knowledge conflict often arises in retrieval-augmented generation (RAG) systems, where retrieved documents may be inconsistent with one another or contradict the model's parametric knowledge. Existing benchmarks for investigating the phenomenon have notable limitations, including a narrow focus on the question answering setup, heavy reliance on entity substitution techniques, and a restricted range of conflict types. To address these issues, we propose a knowledge graph (KG)-based framework that generates varied and subtle conflicts between two similar yet distinct contexts, while ensuring interpretability through the explicit relational structure of KGs. Experimental results on our benchmark, MAGIC, provide intriguing insights into the inner workings of LLMs regarding knowledge conflict: both open-source and proprietary models struggle with conflict detection -- especially when multi-hop reasoning is required -- and often fail to pinpoint the exact source of contradictions. Finally, we present in-depth analyses that serve as a foundation for improving LLMs in integrating diverse, sometimes even conflicting, information.

1 Introduction

RAG can expose LLMs to conflicting knowledge, and identifying disagreements across sources—and pinpointing their locations—remains challenging. MAGIC uses knowledge graphs to construct inter-context conflicts and finds that models struggle particularly with multi-hop detection and conflict localization.

1 Introduction

  • Detecting whether knowledge sources disagree and pinpointing where they disagree are key challenges to generating reliable responses from multiple sources.The paper focuses on inter-context conflict: contradictions among multiple input documents.Source: §1 · pp. 1–2
  • Existing benchmarks often center on question answering, rely on entity substitution, lack systematic single-hop versus multi-hop analysis, and underexplore conflicts among input documents.These limitations motivate a benchmark focused on inter-context conflict rather than only conflicts between parametric and external knowledge.Source: §1 · pp. 1–2
  • MAGIC constructs distinct knowledge chunks from KG subgraphs, introduces conflicts by modifying graph nodes and edges, and converts the graphs into text passages.The resulting graph representation supports greater diversity, complexity, and control in conflict generation.Source: §1 · p. 2
  • Models often miss inter-context conflicts, especially those requiring multi-hop reasoning, and fail to localize the exact contradiction.MAGIC includes simultaneous and multi-hop conflict patterns that are rarely represented in existing benchmarks.Source: §1 · p. 2

2 Related Work

Prior conflict benchmarks have often relied on question answering or entity substitution, while inter-context benchmarks have traded realism for coverage or fluency for representativeness. The paper positions KG-based construction as a way to support broader conflict types and more systematic analysis.

2 Related Work

  • Question-answering benchmarks constrain conflicts to answer-related contexts, while entity substitution is often too simple to capture complex and subtle conflicts.The paper adopts question-free, KG-based methods to support more diverse problem types.Source: §2 · p. 2
  • Existing inter-context benchmarks use curated Wikipedia conflicts for realism or LLM-generated conflicts for scalability, with respective limits in coverage or naturalness.The authors aim to combine KG factual grounding with LLM fluency for quality and scalability.Source: §2 · p. 2
  • Prior work has paid limited attention to fine-grained conflict types; MAGIC organizes conflicts by conflict count and reasoning hops.This contrasts with the two-category convention described in the paper.Source: §2 · p. 2
  • Bi et al. (2024) also use Wikidata triplets, but retrieve seed entities for substitution rather than leveraging the full KG structure.The paper uses KGs as a foundation for generating realistic and nuanced conflict statements.Source: §2 · p. 3

3 MAGIC: Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts

MAGIC constructs an inter-context conflict benchmark from knowledge graphs, combining controlled graph perturbations with human review and paired graph/text representations. Its pipeline extracts subgraphs, generates conflicts, and converts the graphs into text contexts.

3 MAGIC: Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts

  • MAGIC combines KG-based conflict generation with human review and complementary graph and text views for a scalable, interpretable benchmark.The pipeline extracts subgraphs, perturbs them, and converts original and modified graphs into text passages.Source: §3 · p. 3

3.1 Subgraph Extraction

  • The extraction uses Wikidata5M, selecting 46 of its 825 relations for semantic clarity and multi-hop reasoning, grouped into seven domains.Wikidata5M contains approximately 20 million triplets; subgraphs are built by DFS from sampled seed triplets under structural constraints.Source: §3.1 · p. 3

3.2 Knowledge Conflict Generation

  • Conflict categories vary along two axes: reasoning depth (single-hop or multi-hop) and the number of contradictions between contexts (1 or N).The paper describes four resulting categories and allocates instances across them; multi-hop conflicts require reasoning over connected triplets.Source: §3.2 · p. 4
  • To generate conflicts, the method prompts an LLM with a seed triplet and its surrounding subgraph, using three validated examples per relation type.For N-conflict cases, the same graph is reused with different perturbations until the desired number of conflicts is reached.Source: §3.2 · p. 4
  • Human experts filter few-shot demonstrations before prompting and remove trivial or incoherent outputs after generation.This two-stage review is incorporated to improve dataset quality.Source: §3.2 · p. 5

3.3 KG-to-Text Conversion

  • KG-to-text conversion uses GPT-4o-mini to produce coherent contexts that preserve graph semantics, followed by automatic verification using Claude 3.7 Sonnet.Manual inspection of a sample confirms consistently high conversion quality.Source: §3.3 · p. 5

4 Experimental Setups

The experiments evaluate prompted open-source and proprietary LLMs on MAGIC and two existing benchmarks. They use repeated inference runs, ID and LOC metrics, and a stepwise prompting strategy.

4 Experimental Setups

  • The evaluation uses five open-source and proprietary LLMs without task-specific training, prompting them to identify potential contradictions.The models are Llama 3.1 70B Instruct, o1, Mixtral-8x7B Instruct, Claude 3.5 Haiku, and GPT-4o-mini.Source: §4 · p. 5
  • MAGIC contains 1,080 curated examples spanning single-hop and multi-hop contradictions with varying numbers of conflicts, alongside ECON and WikiContradict.ECON contains 168 instances and WikiContradict contains 103 samples after deduplication.Source: §4 · pp. 5–6
  • The stepwise prompting strategy is designed to probe LLMs’ inter-context conflict detection capabilities more fully than prior minimal binary-classification prompts.The cited prior approaches are Jiayang et al. (2024) and Hou et al. (2024).Source: §4 · p. 6
  • Models perform three independent runs, scored on identification (ID) and localization (LOC); researchers compute both metrics manually.ID awards credit if a conflict is detected in any of three attempts, while LOC requires all conflicting locations to be identified correctly.Source: §4 · p. 6

5 Experimental Results

MAGIC is harder for LLMs than ECON and WikiContradict, with lower conflict-identification and localization scores. Multi-hop reasoning and precise localization are particular challenges, while performance also varies by model, domain, relation, and context composition.

Overall results

  • ID and LOC scores on MAGIC were lower than on ECON and WikiContradict, decreasing by up to 10% and 17%, respectively.The results indicate difficulty both identifying conflicts and pinpointing their locations.Source: §5 · p. 6

ID scores per LLM

  • GPT-4o-mini achieves the highest ID accuracy and generalizes well to MAGIC, while Mixtral performs worst overall and Haiku lags behind larger models.Llama fails to respond in 47.4% of runs but has high ID scores when it does; o1's moderate performance suggests detection may depend on more than reasoning ability.Source: §5 · p. 6

LOC scores per LLM

  • GPT-4o-mini has the best LOC scores across datasets; o1 often misses ambiguous conflicts, particularly those requiring multi-hop thinking.One proposed explanation for o1's outcome is the exclusion of explicit reasoning instructions such as “think step by step.”Source: §5 · p. 6

Performance by conflict types

  • Multi-hop conflicts yield lower ID and LOC scores than single-hop conflicts, with localization especially difficult when contradictions span multiple locations.The indirect nature of multi-hop contradictions adds complexity.Source: §5 · p. 7
  • More conflicts facilitate identification but make precise localization harder, lowering LOC when all conflicting sentences must be pinpointed.The pattern is also reported for ECON.Source: §5 · p. 7

Domain- & relation-level analysis

  • Performance is relatively strong in Class/Concept, while Organization results range from around 75% for GPT-4o-mini and Llama to below 45% for Mixtral, Haiku, and o1.Conflicts involving captain and mother are relatively easy; work location and father are more challenging.Source: §5 · p. 7
  • In multi-domain cases, ID scores tend to improve while LOC scores decline, possibly because structural variation complicates span localization.The analysis attributes the ID pattern possibly to greater semantic diversity making conflicts more salient.Source: §5 · p. 7

6 Further Analysis and Discussion

Further analyses compare MAGIC with existing datasets and examine how conflict complexity, prompting, parametric knowledge, context order, and linguistic quality relate to evaluation. MAGIC includes substantial multi-hop coverage and remains challenging in comparisons with WikiContradict and other datasets.

Knowledge graph construction

  • Schema-free knowledge graphs constructed with LangChain support structured conflict representation where predefined ontologies or domain structures are unavailable.The stated challenge is the lack of such predefined structures for these datasets.Source: §6 · p. 8

Dataset composition

  • MAGIC has a more balanced conflict-type distribution, including 28% 1-Multi-Hop and 27% N-Multi-Hop, while 1-Single-Hop accounts for 78% in ECON and 76% in WikiContradict.The latter category is described as typically the easiest case.Source: §6 · p. 8

Length-based analysis

  • As context length increases across MAGIC's four quantile-based bins, ID and LOC scores decline, with LOC dropping more sharply.MAGIC also contains more long-context examples than the other datasets.Source: §6 · p. 8

Comparison with existing KC datasets

  • On the challenging multi-hop subset, MAGIC's ID score is up to 9% lower and LOC score up to 16% lower than WikiContradict's implicit-conflict subset.Even on the shared 1-Single-Hop category, models perform worse on MAGIC than on ECON and WikiContradict.Source: Figure 11 · p. 8, §6 · p. 8

Impact of prompts

  • A multi-step prompting approach outperforms the naïve prompt across cases, with improvement up to 39.41%, though substantial challenges remain on MAGIC.The comparison evaluates the multi-step technique against binary prompting commonly used in prior work.Source: §6 · pp. 8–9

Impact of parametric knowledge

  • GPT-4o-mini and Llama 3.1 have slightly higher ID scores on instances where they already possess the relevant parametric knowledge.Knowledge presence is estimated by verification-question performance; a more systematic investigation is left for future work.Source: §6 · p. 9

Impact of context order

  • Reversing the two contexts leaves ID scores largely consistent across models, with Mixtral 8x7B showing a slight increase and others nearly unchanged.The findings suggest performance is robust to input ordering.Source: §6 · p. 9, Table 4 · p. 9

Linguistic quality of MAGIC

  • MAGIC's naturalness score is close to WikiContradict's, 0.31 lower, and its realism score surpasses ECON's.The evaluation used Claude 3.7 Sonnet to rate sampled contexts for naturalness and realism.Source: §6 · p. 9, Table 5 · p. 9

7 Conclusion

MAGIC is a knowledge-graph-based benchmark for inter-context conflict detection, designed for greater diversity and complexity. Its results show that LLMs still struggle with complex conflicts, especially those requiring multi-hop reasoning.

Conclusion

  • MAGIC is a KG-based benchmark for inter-context knowledge conflict detection, designed to offer greater diversity and complexity.The paper reports that LLMs continue to struggle with complex cases, including those requiring multi-hop reasoning.Source: §7 · p. 9

Limitations

MAGIC’s current benchmark has limits in source coverage, relation alignment, and localization evaluation. The authors identify broader knowledge-graph sources and automated localization assessment as future directions.

Limitations

  • MAGIC is built on Wikidata-based knowledge graphs, so its source coverage is limited; additional or domain-specific sources could improve robustness and applicability.The authors give preliminary DBpedia and YAGO examples in the appendix.Source: Limitations · p. 10
  • Semantically equivalent relations across graphs still need alignment to ensure consistency and coverage.Source: Limitations · p. 10
  • Localization evaluation relies on human judgment; automated approaches could enable finer-grained assessment.Source: Limitations · p. 10

Ethics Statement

The appendix documents MAGIC’s data and prompt procedures, reports human checks of generated contexts, and presents human and model performance analyses. The reported results include strong but imperfect human annotation scores and lower model performance on multi-hop conflicts.

Ethics Statement

  • MAGIC is released for scientific research, contains no personally identifiable information, and may still contain biased or potentially offensive content despite filtering and validation.Source: Ethics Statement · p. 10

A Details of Selected Relation Lists per Domain

  • The selected relations were grouped into seven manually determined semantic domains, with selection based on clarity for controlled manipulation and support for meaningful multi-hop chains.The domains are Human, Geography, Organization, Creative Work, Class/Concept, Cause-Effect, and General.Source: A Details of Selected Relation Lists… · p. 12

B Detailed Prompts used in MAGIC

  • The KG-to-text prompts generate contexts from subgraphs and verify triplet coverage with Claude 3.7 Sonnet; only outputs returning “No error” are retained.The conflict-generation prompt uses related subgraphs for indirect contradictions, and N-conflict cases reuse a graph with different perturbations.Source: B Detailed Prompts used in MAGIC · p. 12
  • 95.21% accuracy for conflict-triplet coverage and 82.04% for subgraph-triplet coverage were obtained in human inspection of KG-to-text outputs.The inspection sampled 167 outputs and used two criteria: expressing the intended conflict and conveying all subgraph triplets.Source: B Detailed Prompts used in MAGIC · p. 12, Table 7 · p. 13
  • The appendix contrasts a binary conflict-detection prompt with the paper’s stepwise detection prompt.Source: B Detailed Prompts used in MAGIC · p. 13

C Human Baseline Performance

  • On a small subset of MAGIC, annotators achieved average scores of 92.5% for conflict identification (ID) and 83.3% for conflict localization (LOC), generally identifying conflicts and their locations with reasonable accuracy.The authors caution that these results are based on a small-scale pilot and should not be taken as definitive.Source: Table 8 · p. 14, C Human Baseline Performance · p. 14

D.1 Performance by each Conflict type

  • As the number of conflicts grows, identification scores tend to rise while localization scores fall; multi-hop conflicts generally score lower than single-hop conflicts on both.Source: §D.1 · p. 14

D.2 Domain-Level Analysis

The analyses show that identification and localization respond differently to domain and conflict complexity. The appendix also reports that subgraph context matters for generating plausible, semantically grounded conflicts and documents the human annotation guidelines.

D.2 Domain-Level Analysis

  • As the number of domains per instance increases, identification tends to improve while localization decreases, indicating easier conflict distinction but harder precise localization.Source: §D.2 · p. 14

E Performance by Number of Conflicts

F Impact of Subgraph Scope on Conflict Generation

  • Generation without surrounding subgraphs often produced trivial negations or incoherent, off-topic facts, whereas subgraph-level prompting enabled more realistic, semantically grounded conflicts.Source: F Impact of Subgraph Scope on Conflict… · p. 14

G Annotation Guideline

  • Human intervention included selecting few-shot examples and filtering trivial or inherent generated conflicts; selection guidelines preferred plausible, relation-appropriate, diverse examples, including multi-hop or indirect conflicts.Source: G Annotation Guideline · pp. 14–15
  • Post-generation filtering requires direct, unambiguous single-hop contradictions, multi-hop contradictions supported by a chain of triplets, independent N-conflicts, and natural, coherent contexts.Source: G Annotation Guideline · p. 15

H The Scalability and Consistency of Human Evaluation

The authors compare human annotation with model-based localization evaluation and describe a protocol intended to reduce human review effort. GPT-4o-mini’s agreement with human judgments varies by evaluated model and remains insufficient to replace human judgment reliably.

H The Scalability and Consistency of Human Evaluation

  • The authors state that human evaluation was necessary because current LLMs are not yet reliable for conflict detection and localization.Source: H The Scalability and Consistency of… · p. 15
  • The human-evaluation protocol ignores “No conflict” predictions because every MAGIC instance contains a conflict and checks localization only when identification is correct.Bulk comparison of multiple model outputs against gold labels reduced the need to assess each case individually.Source: H The Scalability and Consistency of… · pp. 15–16
  • 74.67% agreement for o1 and 48.72% for Mixtral-8x7B were achieved by GPT-4o-mini when judging localization against human annotations.The authors conclude that current LLMs do not yet reliably replace human judgment for conflict localization.Source: H The Scalability and Consistency of… · p. 16

I The Generalizability of the Benchmark Framework

The framework’s relation-pool overlap with DBpedia and YAGO supports applying its benchmark-construction pipeline beyond Wikidata5M. MAGIC also includes single- and multi-hop conflict examples, including a case where GPT-4o-mini struggles to detect a contradiction requiring multi-hop reasoning.

I The Generalizability of the Benchmark Framework

  • The framework’s pipeline generated conflict examples from DBpedia and YAGO, supporting its generalization beyond Wikidata5M.Their relation sets substantially overlap with the framework’s pool; examples include child, capital, and participatedIn.Source: I The Generalizability of the… · pp. 16–17, Table 14 · p. 18

J Examples from MAGIC Dataset

J.1 Qualitative Analysis of Model Failures

  • GPT-4o-mini struggles with a multi-hop contradiction requiring chained equivalence and distinction relations across entities.The cited case contrasts a context stating that John is equivalent to Hans with perturbed contexts implying that they are not equivalent.Source: §J.1 · p. 17, Table 17 · p. 21
Loading 2507.21544v3…