PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework

Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Kamal Premaratne, Uthayasanker Thayasivam

arXiv:2609.30484v1cs.AIcs.LG

TL;DR

Surface-level metrics do not establish whether LLM responses are grounded in the provided context, an important gap for QA. The paper evaluates contextual understanding by comparing knowledge graphs with a hybrid semantic-structural metric and diagnoses triplet-level errors; across benchmarks, it reports strong results against lexical and neural baselines.

  • Problem

    Traditional metrics such as BLEU and perplexity measure surface-level performance, leaving contextual understanding in question answering insufficiently assessed.

    Source: §1 · p. 1

  • Method

    The framework builds knowledge graphs from model responses, gold answers, and context, compares them with hybrid S3KG, and uses triplet-level diagnostics to classify errors.

    Source: §4 · p. 2, §4.3 · p. 3

  • Results

    S3KG achieves best-per-dataset F1 of 0.766–0.932 and AUROC up to 0.973, outperforming lexical and neural baselines on KG-rich datasets while remaining competitive on short-text settings.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The framework makes contextual understanding empirically testable by measuring whether generated responses preserve relational knowledge in reference answers and supporting context.

    Source: §8 · p. 9

    The evaluation is restricted to open-source 7B-parameter models, with larger proprietary models such as GPT-4 left for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While large language models (LLMs) have achieved remarkable linguistic capabilities, a profound question lingers at their core: do these models truly comprehend context or simply excel at pattern matching on an unprecedented scale? Contextual understanding in LLMs refers to the ability to correctly extract relevant information from a given context, integrate it into a coherent internal representation, and reason over it to produce factually consistent and contextually grounded responses. However, traditional methods such as BiLingual Evaluation Understudy (BLEU) and perplexity simply measure surface-level performance. This reveals a critical gap in question answering (QA), where responses must be contextually grounded rather than simply being memorized associations. To fill this void, we propose a novel knowledge graph (KG) based evaluation framework for LLM contextual understanding in QA. Central to this is Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure combining structural and semantic signals into a single score. In addition, a diagnostic analysis framework is developed to identify and categorize reasoning errors at the triplet level, enabling fine-grained analysis of model failures. Together, across nine benchmarks, S3KG achieves F1 gains of up to $+7.6$ points over the strongest baseline and AUROC up to $0.973$.

1 Introduction

LLMs can produce fluent responses, but whether they understand context remains an open question. Existing evaluation often measures surface-level performance, leaving contextual comprehension in QA insufficiently assessed.

1 Introduction

  • LLMs’ fluent, plausible outputs leave open whether they understand processed content or merely generate plausible responses.The cited discussion notes that hallucination detection has largely focused on output-level signals such as semantic entropy and token probabilities.Source: §1 · p. 1
  • Perplexity, BLEU, and token-matching metrics measure surface-level performance rather than semantic comprehension, a concern for high-stakes QA.The passage names medicine and healthcare, defense, and legal analysis as settings where contextual grounding matters.Source: §1 · p. 1

2 Related Work

Prior work probes contextual patterns and logical reasoning, but the cited approaches do not provide fine-grained error localization for QA. Knowledge graphs offer a possible route to analyzing relations in long-form answers.

2 Related Work

  • Zhu et al. (2024) find that across four tasks—coreference resolution, discourse relation classification, dialogue state tracking, and query rewriting—LLMs capture general contextual patterns but fail to reveal misunderstood context parts or extend analysis to QA.Yan et al. (2024) probe reasoning by manipulating in-context examples, including swapping “AND”/“OR”; models do not consistently obey formal reasoning rules or exhibit clearly identifiable error patterns.Source: §2 · p. 1
  • Knowledge graphs are presented as a promising way to evaluate relational triplets in long-form answers, where fine-grained interpretation is difficult to assess.The cited work notes that the approach depends on accurate extraction of relational triplets.Source: §2 · p. 2
  • Existing WL-style graph kernels give no credit to semantically equivalent but syntactically different labels, while clustering-based alignment remains lossy.The passage contrasts exact-label WL and WWL kernels with the SBERT-based clustering approach of Haskins and Adams (2025).Source: §2 · p. 2

3 Our Contributions

The paper contributes a KG similarity metric, a model-level score separating factual accuracy from contextual faithfulness, and a triplet-level diagnostic component.

3 Our Contributions

  • S3KG converts responses and reference answers into KG triplets and combines semantic and structural similarity in one interpretable score.This is the paper’s proposed metric for evaluating LLM outputs against reference answers.Source: §3 · p. 2
  • CUS aggregates factual accuracy (GoldSim) and contextual faithfulness (CtxSim) for cross-model comparison across benchmarks.Source: §3 · p. 2
  • TAU identifies and classifies reasoning failures at the triplet level to support fine-grained behavioral analysis of model outputs.Source: §3 · p. 2

4 Methodology

The methodology constructs comparable knowledge graphs from model responses, gold answers, and supporting context, then scores their similarity and diagnoses low-scoring pairs. Its components specify answer collection, graph construction, S3KG comparison, CUS aggregation, and triplet-level error analysis.

4.1 LLM Answer Collection

  • For answer collection, models receive a question and supporting context; deterministic temperature-zero outputs establish a baseline before higher-temperature runs examine generation diversity.Source: §4.1 · p. 2

4.2 KG Construction

  • KGs are extracted from gold answers, model responses, and context using a shared prompt, then labels are normalized to make the graphs structurally comparable.The normalization includes lowercasing, lemmatization, and whitespace normalization.Source: §4.2 · pp. 2–3

4.3 S3KG: Semantic Structural Similarity for KGs

  • S3KG combines WL structural similarity with SBERT semantic similarity; its WL kernel aggregates neighborhood labels for up to five hops.The semantic score uses mean-pooled triplet embeddings, while the structural score compares refined-label histograms.Source: §4.3 · pp. 3–4
  • S3KG matches triplets by semantic similarity and independently soft-aligns entity and relation labels, allowing equivalent labels to match despite different wording.Alignment uses a cosine-similarity threshold of 0.65; the unidirectional matching treats one triplet set as the reference and emphasizes recall without penalizing surplus triplets.Source: §4.3 · p. 3

4.4 Contextual Understanding Score (CUS)

  • CUS applies a harmonic mean to response-to-gold similarity and response-to-context similarity, representing factual accuracy and contextual faithfulness.The harmonic mean penalizes profiles with one strong and one weak dimension.Source: §4.4 · p. 4

4.5 Triplet Analysis Unit (TAU)

  • TAU analyzes the five percent of lowest-scoring QA pairs, aligning gold and generated triplets before categorizing residual errors.Error classes include relation mismatch, entity mismatch, extra triplets, and missing triplets.Source: §4.5 · p. 4

5 Experiments

The experiments evaluate contextual QA on long-form health datasets and test S3KG across varied text-pair benchmarks. They compare a hybrid graph-and-language similarity method with standard baselines.

5.1 Datasets

  • PubMedQA and MesaQA provide long-form biomedical and consumer-health answers, with MesaQA requiring multi-span evidence integration.PubMedQA contains 273,518 QA pairs with answers typically exceeding 100 words; MesaQA has approximately 6,100 pairs with answers averaging 70 words.Source: §5.1 · p. 4

5.2 LLM Answer Collection

  • Four similarly scaled 7B instruction-tuned models answer questions with supporting context; higher temperatures test how generation diversity affects contextual use.Responses are generated at temperatures 0, 0.3, 0.7, and 1.0.Source: §5.2 · p. 4

5.3 Benchmarking Dataset Collection

  • Ten balanced datasets span human-annotated short texts, KG-perturbed paragraphs, and NLP-perturbed paragraphs, forming binary semantic-equivalence tasks evaluated by maximum F1 and AUROC.The protocol uses 400 positive and 400 negative pairs per dataset.Source: §5.3 · p. 5

5.4 Methods Evaluated

  • S3KG interpolates KG structural embeddings and sentence-transformer representations, with the best variant selected by maximum F1 across an α sweep and compared against seven baselines.The two signals contribute relational and ontological structure and contextual semantic similarity, respectively.Source: §5.4 · p. 5

6 Results

S3KG is evaluated across nine benchmarks, including short-text paraphrase and perturbed-paragraph tasks. Its performance varies with the dataset’s text and graph characteristics, and it also leads on an independently constructed entity-swap control.

6 Results

  • On paragraph-level KG-perturbed tasks, S3KG is top-ranked on 4 of 5 benchmarks, while sparse short-text graphs leave it 6–7 F1 points behind sentence-T5-base on MRPC and STS12.The reported paragraph-level scores include 0.932 on SK-Codex 400, 0.834 on SK-Combined, 0.892 on SK-GloBI, and 0.812 on SK-Oregano.Source: §6 · p. 5
  • On PAWS-Wiki, S3KG scores F1 = 0.766 while ROUGE-1 has AUC = 0.490; in the worked pair example, S3KG alone places the relation-reversal case below threshold.For the negative pair, S3KG scores 0.48 against a threshold of 0.5; all seven baselines assign near-identical high scores to the positive and negative pairs.Source: §6 · p. 6, Table 1 · p. 6
  • On SK-FindKG, sentence-T5-base leads at F1 = 0.848 while S3KG scores 0.767, with noisy triplets limiting performance despite the best α being 0.0.The passage attributes the cap to the dependence of S3KG’s gains on faithful graph recovery from text.Source: §6 · p. 6
  • On the independently constructed Wikipedia Entity-Swap control, S3KG achieves F1 = 0.872 and AUC = 0.890, exceeding MiniLM and sentence-T5-base; ROUGE-1 is excluded as a token-overlap artifact.The passage states that the result rules out circularity as an explanation for gains on KG-perturbed data.Source: §6 · p. 6
  • The best α depends on the dataset: structural perturbations favor lower values, while complementary KG and dense signals favor higher values.Examples include α = 0.0 for SK-FindKG and α = 0.5 for SK-Codex 400 and SK-Combined.Source: §6 · p. 6
  • S3KG leads on 6 of 9 datasets, with gains up to +7.6 F1 over the strongest baseline on KG-rich paragraph datasets.The highest meaningful score is also reported on the Wikipedia Entity-Swap anti-circularity control.Source: Table 2 · p. 7

F1 Score

S3KG’s benchmark results vary with graph richness, while its QA evaluation separates reference-answer alignment from context faithfulness. Across the two health QA datasets, model scores differ by domain and model, and low-similarity cases expose distinct failure patterns.

F1 Score

  • KG-rich paragraph datasets gain up to +7.6 F1 over the strongest baseline, while sparse or noisy KGs yield reduced margins; the KG component is comparatively lightweight at inference.The strongest baselines overall require full finetuned transformer inference.Source: Table 3 · p. 7, F1 Score · p. 7

6.1 LLM Comparison on QA Datasets

  • On MesaQA, Mistral-7B has the best CUS at 0.678, while Falcon-7B scores 0.606; Gemma-7B leads GoldSim at 0.689 and Mistral-7B leads CtxSim at 0.736.The reported CUS range for Gemma-7B, Llama-2-7B, and Mistral-7B is 0.675–0.678.Source: §6.1 · p. 7, Table 4 · p. 8
  • On PubMedQA, CUS ranges from 0.480 for Falcon-7B to 0.592 for Mistral-7B, which also leads CtxSim at 0.733; Gemma-7B leads GoldSim at 0.524.Gemma-7B has the lowest PubMedQA CtxSim, at 0.640.Source: §6.1 · pp. 7–8, Table 4 · p. 8
  • CtxSim exceeds GoldSim in all eight model–dataset combinations, and CUS is about 10 points lower on PubMedQA than MesaQA across models.The passage links the cross-dataset gap to biomedical vocabulary and reasoning challenges across model architectures.Source: §6.1 · p. 8
  • In the 5% lowest-similarity QA cases, Mistral is most reliable and Gemma weakest; generated KGs tend to align more with context than reference KGs, with biomedical terminology and entity variability prominent in PubMedQA failures.The identified failure patterns include drifting to a different interpretation, omitting reference facts, and adding unsupported content.Source: §6.1 · p. 8

7 Discussion

The discussion frames the framework as a way to evaluate relational and structural fidelity beyond syntactic metrics. It emphasizes both S3KG’s graph-comparison design and the distinct insights provided by its two-dimensional scores and triplet diagnostics.

7 Discussion

  • S3KG’s type-separated, one-to-one label alignment addresses limitations of clustering-based approaches while retaining WL-kernel multi-hop neighborhood similarity.The framework evaluates relational and structural fidelity rather than syntactic-level performance alone.Source: §7 · p. 8
  • GoldSim and CtxSim distinguish factual accuracy from contextual faithfulness, while TAU identifies entity substitutions, relation errors, and broader inconsistencies.The discussion says these model-specific trade-offs and pinpointed failure types are not captured by aggregate metrics alone.Source: §7 · p. 8

8 Conclusion

The framework makes contextual understanding in QA empirically testable by comparing knowledge graphs from model responses, reference answers, and supporting context. It reports benchmark performance and triplet-level diagnostics of reasoning failures.

8 Conclusion

  • The framework compares knowledge graphs from generated responses, reference answers, and context to test whether responses preserve relational knowledge.It uses S3KG for similarity and the Triplet Analysis Unit to diagnose reasoning errors.Source: §8 · pp. 8–9
  • Best-per-dataset F1 ranges from 0.766–0.932, and AUROC reaches 0.973, with S3KG outperforming lexical and neural baselines on KG-rich datasets.It remains competitive on short-text settings.Source: §8 · p. 8
  • Evaluations on PubMedQA and MesaQA show model-specific strengths and weaknesses, and the pipeline is presented as reproducible and extendable to other datasets and tasks.The stated aim is to support trustworthy AI development.Source: §8 · p. 8

Limitations

The appendix reports temperature robustness with a Falcon-7B/MesaQA exception, details KG extraction and TAU annotation and evaluation, and identifies scope and implementation limits.

Limitations

  • S3KG is sensitive to noisy or incomplete KG extraction; SBERT inference may be costly at scale without GPUs, and evaluation is limited to open-source 7B models.Handling directional semantic equivalence, such as daughter_of versus mother_of, would require attention-based mechanisms.Source: Limitations · p. 9

A Effect of Temperature on Contextual Understanding

B KG Construction Prompt

  • The shared extraction prompt performs entity detection, coreference resolution, relation extraction, and KG refinement across the three input texts.It standardizes labels while keeping distinct facts separate.Source: B KG Construction Prompt · p. 11
  • The prompt takes the reference answer, model response, and supporting context, and returns three knowledge-graph keys containing subject-relation-object triples.Few-shot examples support the expected output format and label consistency.Source: B KG Construction Prompt · p. 11

C.1 Dataset and Annotation

  • Three medical students independently annotated semantically aligned gold and LLM-generated triplet pairs from a manually annotated MesaQA subset, resolving disagreements through adjudication.The LLM-generated graphs were extracted from model outputs.Source: §C.1 · p. 11

C.2 Annotation Quality

  • Annotation agreement was 0.9433; pairwise F1 scores were 0.9708, 0.9825, and 0.9882, indicating consistency among annotators.Pairwise Jaccard scores were 0.9433, 0.9657, and 0.9766.Source: §C.2 · p. 11

C.3 Evaluation Protocol

  • TAU alignment is evaluated with precision, recall, and F1 using both micro-averaging across triplets and macro-averaging across QA instances.Table 7 compares the method with KEA on manually annotated MesaQA and PubMed subsets.Source: §C.3 · p. 11, Table 7 · p. 12

C.4 Results

Sentence-level semantic alignment improves recall and F1 over KEA, while benchmark results show that S3KG performance and the best structural-semantic balance vary across dataset types. On short-text benchmarks it leads on PAWS-Wiki but trails sentence-T5-base on MRPC and STS12; it leads on four of five KG-perturbed datasets.

C.4 Results

  • +34.8% recall over KEA accompanies higher Micro and Macro F1, despite lower precision from semantic matching that captures relation equivalences such as “treats” and “used for.”The method also supports residual error analysis by categorizing mismatches as relation wrong or entity wrong.Source: §C.4 · p. 12

D Per-Dataset Performance Tables

  • S3KG leads on PAWS-Wiki (F1 0.766, AUROC 0.795) but trails sentence-T5-base on MRPC and STS12.The short-text comparison reflects the different performance of structural signals and dense semantic representations on these datasets.Source: Table 8 · p. 12
  • S3KG attains the highest F1 and AUROC across retained Wikipedia Entity-Swap baselines; ROUGE-1 is excluded because shared surrounding tokens make its score a token-overlap artifact.Entity-swapped pairs differ only in swapped entity tokens while retaining nearly identical surrounding surface forms.Source: Table 9 · p. 12, D Per-Dataset Performance Tables · p. 12

E Hyper-parameter Selection

  • The selected α is dataset-dependent: SK-FindKG, Wiki Swap, and STS12 favour 0.0–0.1, while SK-Codex 400, SK-Combined, and PAWS-Wiki peak at 0.4–0.6.The sweep selects α by maximum F1 per dataset.Source: E Hyper-parameter Selection · pp. 12–13, Table 11 · p. 14, Table 12 · p. 14
  • S3KG leads on four of five KG-perturbed datasets, while sentence-T5-base outperforms it on SK-FindKG, where α = 0.0 is optimal.The caption identifies SK-FindKG as the only dataset where pure dense embeddings dominate.Source: Table 10 · p. 13
  • AUROC varies less across α than F1, indicating that α primarily reshapes score distributions near the decision boundary.α blends the structural WL kernel signal at 0.0 and the SBERT semantic signal at 1.0.Source: E Hyper-parameter Selection · pp. 12–13
Loading 2609.30484v1…