Source-linked AI summary
Evaluation of Contextual Understanding in Large Language Models
Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Uthayasanker Thayasivam, Kamal Premaratne
TL;DR
LLM evaluations often do not reveal whether answers genuinely integrate context, especially when QA requires grounding beyond memorized associations. The paper introduces a KG-based pipeline combining S3KG, CUS, and TAU, and reports strong benchmark performance with interpretable contextual-faithfulness analysis. Its main limitation is that KG extraction quality remains the primary bottleneck.
Problem
Existing evaluation metrics miss relational depth and factual faithfulness, leaving uncertain whether LLMs genuinely understand context in grounded question answering.
Method
The framework constructs knowledge graphs from responses, gold answers, and context, compares them using hybrid structural-semantic S3KG, aggregates CUS, and diagnoses triplet-level errors with TAU.
Results
S3KG achieves best-per-dataset F1 of 0.766–0.932 and AUROC up to 0.973, consistently outperforming lexical and neural baselines on KG-rich datasets.
Takeaways & Limitations
The framework provides a reproducible and interpretable approach for measuring LLM contextual understanding in QA and supports broader trustworthy, verifiable AI evaluation.
Takeaways & Limitations
KG extraction quality remains the primary bottleneck, motivating future directional alignment methods to distinguish semantically inverse relations.
Abstract
from arXiv · showhide
Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.
1. Introduction
Although LLMs perform strongly across NLP tasks, their apparent contextual understanding remains uncertain. This uncertainty is especially consequential in medical and legal QA, where answers must rely on provided context rather than memorized priors.
- LLMs achieve strong performance across tasks including open-domain question answering and complex reasoning, yet their genuine contextual understanding remains unresolved.
- A central concern is whether LLMs understand processed context or exploit statistical correlations to produce plausible outputs without comprehension.
- Contextual grounding is especially critical in high-stakes medical and legal QA, where responses must use provided context rather than memorized priors.
2. Related Work
Existing metrics and graph-comparison methods inadequately capture relational structure together with semantic faithfulness. The paper addresses this gap with S3KG, CUS, and TAU for continuous evaluation and fine-grained diagnosis.
- Perplexity, BLEU, and token-level accuracy measure surface fluency or overlap but miss relational depth and factual faithfulness.
- No existing metric jointly captures relational structure and semantic faithfulness in a continuous interpretable score.
- Existing graph metrics either treat labels as opaque symbols or use lossy clustering, leaving semantically varied but fine-grained relational distinctions inadequately preserved.
- Contributions: S3KG converts LLM responses and reference answers into knowledge-graph triplets and produces a continuous hybrid structural-semantic evaluation score.
- Contributions: CUS aggregates factual accuracy and contextual faithfulness for cross-model benchmark comparison.
- Contributions: TAU categorizes reasoning errors at the triplet level for fine-grained behavioral analysis.
3. Methodology
The methodology constructs knowledge graphs from model responses, gold answers, and supporting context, then compares them with S3KG and diagnoses low-similarity cases through triplet analysis. Experiments use three instruction-tuned 7B models, Falcon-7B as a baseline, and temperature variation across two QA datasets.
- Pipeline: For each QA pair, graphs from the LLM response, gold answer, and supporting context are compared via S3KG to produce GoldSim, CtxSim, and CUS.
- Pipeline: The pipeline extracts structured representations from three sources and uses S3KG to compare them at both triplet and overall graph levels.
- S3KG: Triplet-level matching serializes facts into natural-language strings and uses SBERT cosine similarity to anchor structural comparison to semantically relevant triplets.
- S3KG: S3KG independently aligns entity and relation labels when SBERT similarity exceeds 0.65, preventing cross-type collisions between semantically related labels.
- S3KG: Structural similarity uses a normalized Weisfeiler–Lehman kernel with 5 iterations to capture multi-hop neighborhood patterns over aligned graphs.
- S3KG: Semantic similarity is the clipped-to-[0, 1] cosine between mean SBERT embeddings of graph triples.
- Scoring and diagnosis: Structural and semantic scores are blended with coefficient α = 0.5, while TAU analyzes the 5% lowest-similarity QA pairs using a 0.76 alignment threshold.
- Evaluation: Experiments evaluate PubMedQA and MesaQA with three instruction-tuned 7B models, Falcon-7B as baseline, and temperatures {0.0, 0.3, 0.7, 1.0}.
4. Experiments and Results
The experiments evaluate S3KG as both an isolated graph-similarity measure and an end-to-end contextual-understanding pipeline. Across semantic-equivalence and QA settings, S3KG generally performs strongly, while extraction quality and domain complexity remain important constraints.
- Evaluation setup: S3KG is evaluated through KG construction and a similarity module that produces the Comparative LLM Understanding Score (CUS).The pipeline compares graphs derived from LLM responses, gold answers, and supporting context; S3KG is also benchmarked independently before full QA evaluation.
- Evaluation setup: Nine balanced semantic-equivalence datasets are evaluated with maximum F1 after threshold sweeping, spanning short text, KG-perturbed paragraphs, and Wikipedia Entity-Swap control data.Baselines include ROUGE, BLEU, BERTScore, MiniLM, and Sentence-T5-base.
- Similarity results: S3KG achieves top-1 or top-2 F1 on 7 of 9 datasets, with gains up to +7.6 F1 points on KG-perturbed paragraphs.On PAWS-Wiki, ROUGE-1 reaches near-random performance while S3KG records F1 = 0.766; SK-FindKG is an exception where Sentence-T5-base leads with F1 = 0.848.
- TAU evaluation: The TAU evaluation improves over KEA on Micro F1 and Macro F1, reaching 0.895 versus 0.836 and 0.782 versus 0.622, respectively.The improvement is attributed to a +34.8% recall gain despite a modest precision decrease, supporting alignment of semantically equivalent relations under lexical variation.
- LLM evaluation: Mistral-7B leads CUS on MesaQA at 0.678 and PubMedQA at 0.592, with the highest contextual-similarity values on both datasets.The dataset-level CUS is the mean across N = 400 samples, and CtxSim exceeds GoldSim across all eight model–dataset combinations.
- LLM evaluation: The QA results show a consistent approximately 10 percentage-point CUS gap between MesaQA and PubMedQA, while the lowest-similarity cases favor Mistral-7B and expose biomedical terminology as a failure mode.Gemma-7B is weakest on the 5% lowest-similarity pairs, particularly on MesaQA, and PubMedQA hard cases show lower aligned-triplet recovery across models.
5. Discussion
The discussion presents S3KG as a structure-aware and semantically grounded alternative to surface-level evaluation, while identifying KG extraction quality as the primary bottleneck and proposing richer alignment methods for future work.
- Discussion: S3KG combines type-separated one-to-one label alignment with WL-kernel multi-hop sensitivity to preserve relational distinctions under semantic variation.The GoldSim/CtxSim decomposition also exposes model-specific trade-offs that aggregate metrics do not show.
- Limitations and future work: KG extraction quality remains the primary bottleneck, and future work proposes GPT-4 and attention-based directional alignment for semantically inverse relations.The discussion also reports a consistent PubMedQA–MesaQA domain gap and states that Mistral-7B handles domain-specific health knowledge more reliably than general health QA.
6. Conclusion
The paper presents a KG-based evaluation pipeline combining S3KG and TAU to measure LLM contextual understanding reproducibly and interpretably. S3KG performs strongly across datasets while remaining competitive on short-text settings.
- S3KG and TAU form a reproducible, interpretable KG-based pipeline for measuring contextual understanding in QA.
- S3KG achieves best-per-dataset F1 values of 0.766–0.932 and AUROC up to 0.973.
- The metric consistently outperforms lexical and neural baselines on KG-rich datasets while remaining competitive on short-text settings.
- The S3KG score blends normalized WL-kernel structure with SBERT mean-pool semantics through mixing coefficient α.
- The appendix evaluates the complete α range from 0.0 to 1.0 and reports each dataset’s best-performing setting by F1.
B.1 Short-Text Datasets
Short-text results show that the best α retains a meaningful structural component, with the optimal blend varying across MRPC, PAWS-Wiki, and STS12.
- MRPC peaks at α = 0.3, PAWS-Wiki at α = 0.5, and STS12 at α = 0.1.
- All three short-text datasets favor retaining a KG structural component over a pure dense representation.
- A pure dense-embedding representation at α = 1.0 is suboptimal for MRPC, PAWS-Wiki, and STS12.
B.2 KG-Perturbed Paragraph Datasets
KG-perturbed paragraph datasets generally favor balanced structural–semantic blends, while SK-FindKG uniquely favors pure structural similarity.
- Four of five KG-perturbed datasets favor balanced blends: α = 0.5 for SK-Codex 400 and SK-Combined, α = 0.6 for SK-GloBI, and α = 0.4 for SK-Oregano.
- SK-FindKG peaks with pure KG structure at α = 0.0, achieving F1 = 0.767.
- SK-FindKG is the exception because its structural signal is particularly discriminative and is diluted by dense representations.
C. Temperature Sensitivity Results
The temperature analysis tests whether contextual faithfulness remains stable as generation becomes more stochastic. The supplied passages describe the operating range and benchmark coverage but do not report complete temperature-specific outcomes.
- Temperatures from 0.0 to 1.0 span deterministic inference through high-entropy generation and test robustness of GoldSim and CtxSim.
- Higher temperature broadens the sampling distribution, increasing response variety but also the risk of factual drift from context.
- Tables 7 and 8 report mean GoldSim and CtxSim across temperatures, while most models remain stable within ±0.01–0.02.
- The benchmark appendix also covers short-text, KG-perturbed paragraph, and Wikipedia Entity-Swap results with a summary heatmap.
D.3 Wikipedia Entity-Swap Results
On the Wikipedia Entity-Swap anti-circularity control, S3KG achieved the strongest reported performance, supporting gains beyond artifacts from KG construction. The PAWS-Wiki example further shows that S3KG distinguishes semantic opposition when surface overlap is misleading.
- Entity-Swap control: S3KG achieved the highest F1 (0.872) and AUC (0.890) on Wikipedia Entity-Swap.This control was designed without KG involvement.
- Entity-Swap control: ROUGE-1 was excluded because entity-swapped pairs share surrounding tokens, making unigram overlap trivially near-perfect.The surface-form artifact would make ROUGE-1 uninformative for this control.
- Variant selection: The best-performing S3KG variant per dataset was selected by maximum F1 over α ∈ {0.0, 0.1, . . . , 1.0}.This selection rule is summarized for the reported dataset comparisons.
- Variant selection: Figure 3 compares S3KG and baseline methods using F1 and AUROC across benchmark datasets, with gold borders marking each dataset’s best method.S3KG represents the best-performing α variant selected per dataset.
- PAWS-Wiki illustration: In PAWS-Wiki, S3KG correctly identified reversed subject–object roles as Not Similar despite identical surface tokens.ROUGE-1 incorrectly predicted Similar for the same negative pair.