PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Make More Mistakes If They Do Not Believe the Input Data?

Peter Kochelka, Aleš Manuel Papáček, Vojtěch Dvořák, Ondřej Dušek

arXiv:2609.09363v1cs.CL

TL;DR

LLMs may omit, alter, or add facts when verbalizing structured inputs, raising the question of whether perceived implausibility increases unfaithfulness. The paper tests this with multilingual generation from local factual, counterfactual, and fictional RDF triples and audited LLM judges. It finds only a weak context–memory conflict under Kimi K3, while another judge substantially overestimates the effect.

  • Problem

    The paper asks whether conflicts between an LLM’s parametric knowledge and provided context reduce faithfulness, a concern for retrieval-augmented generation and data-to-text systems.

  • Method

    The authors generate English, Czech, Slovak, and Upper Sorbian text from local Czech and Slovak factual, counterfactual, and fictional RDF triples, using LLM judges audited against human annotations.

  • Results

    The primary Kimi K3 judge finds only a mild context–memory conflict, with counterfactual faithfulness 0.05 points below factual faithfulness on a 1–5 scale.

  • Takeaways & Limitations

    Judge selection is part of the finding: DeepSeek reports a counterfactual penalty three to four times larger, so judge bias can be mistaken for a generator property.

  • Takeaways & Limitations

    The judge-tuning sample is partly stratified by an earlier judge score and has one author label per row without independent adjudication, potentially overstating generalization.

Abstract

from arXiv · show

Large language models (LLMs) are prone to hallucinating or misinterpreting facts, which impairs their usability in retrieval-augmented generation or data-to-text systems. We analyse how faithfulness of LLMs to provided context depends on how plausible they perceive the context to be (context-memory conflict). To better identify error patterns, we make use of the increased difficulty of non-English and low-resource language text generation and input data based on local knowledge, only partially captured in models' parametric knowledge. We let the models generate text in English, Czech, Slovak and Upper Sorbian from factual (FA), counterfactual (CFA) and fictional (FI) RDF triples containing local Czech and Slovak data. Contrary to our expectations, we observe only a weak context-memory conflict on the human-annotated sample. For Kimi K3 as an LLM judge, which agrees well with human annotations on the sample, counterfactual inputs receive only slightly lower faithfulness scores than factual ones (-0.05 on a 1-5 scale). We also find that a suboptimal choice of LLM judge would lead to overestimating the strength of the context-memory conflict.

1 Introduction

The paper studies whether conflicts between parametric knowledge and provided context reduce faithfulness, using factual, counterfactual, and fictional local-knowledge RDF inputs. Across multilingual generation, it finds only a mild context–memory conflict, while judge choice can substantially alter its apparent strength.

  • The study examines whether LLMs correct or ignore provided information when it conflicts with their parametric knowledge.This context–memory conflict is investigated using counterfactual and fictional RDF inputs.
  • The authors construct factual, counterfactual, and fictional RDF groups from local Czech and Slovak knowledge to reduce possible training-data leakage.Models classify input plausibility before generating sentences in English, Czech, Slovak, and Upper Sorbian.
  • 0.05 points: Kimi K3 rates counterfactual items only slightly below factual items on the 1–5 faithfulness scale.Kimi K3 is selected as the primary judge because its agreement with human annotations is strong.
  • Three to four times larger: DeepSeek V4 Pro Preview reports a much stronger context–memory conflict than Kimi K3.The result indicates that an unsuitable LLM judge can overestimate the effect.

2 Related Work

Prior work connects multilingual data-to-text generation with context–memory conflict research and highlights the difficulty of evaluating models on familiar benchmark data. This paper combines those perspectives by using less familiar local knowledge and auditing LLM judges against human annotations.

  • Existing studies show that LLMs can verbalize knowledge graphs, including in under-resourced languages, while models may still rely on parametric knowledge instead of context.This motivates studying context–memory conflict in multilingual generation.
  • Established benchmarks may obscure context–memory conflict because their data can appear in pretraining.Prior work therefore uses data collected on the fly or obscure local knowledge to reduce familiarity.
  • The paper combines multilingual and low-resource data-to-text generation, context–memory conflicts, and less familiar data to test whether perceived factuality affects faithfulness.It evaluates factual, counterfactual, and fictional RDF inputs with LLM judges audited on human-annotated samples.

3 Data

The dataset converts Czech and Slovak local-knowledge question-answer data into RDF triples and adds matched counterfactual and fictional variants. It contains 2,847 instances spanning factuality conditions and both local languages.

  • The source data come from Czech and Slovak portions of the local-knowledge CUS-QA benchmark, grounded in native-speaker questions and local Wikipedia content.Question-answer pairs are converted into statements and RDF triples with typed entities.
  • Counterfactual triples randomly substitute same-type subjects and objects, whereas fictional triples use generated fictional entities.Both variants are created from factual triples following the shared-task procedure.
  • 2,847 instances: the dataset contains 1,482 Czech and 1,365 Slovak examples, each with 1–9 RDF triples.The Czech portion includes 530 factual, 476 counterfactual, and 476 fictional items; the Slovak portion includes 493, 436, and 436 respectively.

4 Experiments

The experiments classify RDF inputs and generate multilingual text with nine open-weight models, evaluating faithfulness and fluency through audited LLM judges. Judge validation favors Kimi for full-scale faithfulness evaluation, while fluency comparisons and annotation design introduce important interpretive constraints.

  • Classification and generation: Models classify each RDF set as factual, counterfactual, or fictional, repeating classification five times and using the majority label.The study also measures classification consistency through unanimous predictions.
  • Classification and generation: The generators verbalize RDF sets into one or a few sentences using greedy decoding across Czech, Slovak, English, and Upper Sorbian.The language setup contrasts matched and other local languages, high-resource English, and low-resource Upper Sorbian.
  • Classification and generation: Nine open-weight generators span large, medium, and small model groups, enabling comparisons by model size and family.The evaluation covers five model families and nine models.
  • Evaluation: Faithfulness judges rate agreement with RDF triples from 1 to 5, while fluency judges assess naturalness without seeing the input triples.Faithfulness penalizes omissions and unsupported claims; fluency penalizes unnatural or wrong-language output.
  • Judge validation: Kimi offers the best balance of agreement and cost among the evaluated faithfulness judges, supporting its selection for full-scale scoring.The comparison includes Claude Opus 5, Kimi K3, DeepSeek, and GPT-5.6 Sol variants.
  • Judge validation: Kimi’s faithfulness agreement remains stable on held-out data, whereas DeepSeek’s agreement declines substantially, suggesting weaker generalization for DeepSeek.Kimi’s κw changes from 0.712 to 0.711, while DeepSeek’s changes from 0.555 to 0.350.
  • Judge validation: Kimi agrees better than DeepSeek on fluency, but both judges are stricter than annotators and show language-dependent bias.Absolute fluency levels and small cross-language differences should therefore be interpreted cautiously.

5 Results

Across classification and generation, larger models generally perform better, while language mismatch and low-resource Upper Sorbian expose substantial fluency difficulties. The matched variant comparison finds only small faithfulness differences, and judge choice materially affects the apparent context–memory conflict.

  • 5.1 Classification: 33–36% classification accuracy for small models is close to the ∼33% random baseline, while accuracy increases with model size.
  • 5.2 Generation: Faithfulness generally increases with model size, whereas fluency follows a less consistent pattern and faithfulness decreases with more input triples.
  • 5.2 Generation: Upper Sorbian is the most difficult prompt language, especially for smaller models, and accounts for 50.4% of faithful-but-disfluent cases.
  • 5.2 Generation: Large models retain mean faithfulness 4.85 in Upper Sorbian while fluency falls to 3.53, producing a 1.32-point gap versus 0.54 in English.
  • 5.3 Factual, Fictional and Counterfactual: Under Kimi, matched inputs follow FA > FI > CFA, but the CFA and FI faithfulness penalties are only −0.049 and −0.030 on a 1–5 scale.
  • 5.3 Factual, Fictional and Counterfactual: DeepSeek estimates a −0.177 CFA–FA faithfulness difference, while Kimi shifts only 0.050 under judge-prompt disclosure and retains stronger human agreement.
  • 5.3 Factual, Fictional and Counterfactual: FI produces about twice as many hallucinations as FA in medium and small models, while no error category consistently follows FA–FI–CFA across model sizes.
  • 5.3 Factual, Fictional and Counterfactual: Some CFA outputs replace supplied counterfactual facts with memorised knowledge, such as generating 1620 instead of 2013 for the Battle of White Mountain.

6 Conclusion

Across nine open-weight models and four languages, larger generators are generally more faithful, but Upper Sorbian remains the hardest setting. The primary judge indicates only a small context–memory-conflict effect, while alternative-judge bias makes evaluator choice central to interpreting the result.

  • Larger generators are generally more faithful, while all models struggle most in Upper Sorbian.
  • The primary Kimi judge finds a CFA faithfulness penalty of −0.049, about one percent of the 1–5 scale, and the effect is not cleanly separable from evaluator bias.
  • DeepSeek assigns a −0.177 counterfactual penalty, showing that a judge can agree well overall while carrying bias aligned with the experimental condition.
  • Manual annotation held out from prompt tuning and evaluation of alternative judges help distinguish generator properties from measurement properties.

Limitations

The study identifies limitations in language coverage, judge validation, and generation settings. These constraints affect how broadly its annotation results and model behavior should be interpreted.

  • No participating author annotator was an Upper Sorbian speaker, although Czech and Slovak expertise supported factuality checks.
  • The judge prompt was tuned and evaluated on a 540-row sample whose stratification and single-label annotation may overestimate generalization.A disjoint 108-row sample was annotated after the prompt was frozen to partially address this concern.
  • Generation used disabled or low reasoning effort, which may alter model behavior relative to higher reasoning modes.

A Data Creation Pipeline

The data pipeline converts question–answer material into typed RDF triples and constructs factual, counterfactual, and fictional variants for annotation and generation experiments.

  • A Data Creation Pipeline: The annotation sample contains 540 examples balanced across variants, languages, faithfulness groups, and model-size-related strata.
  • A Data Creation Pipeline: Question–answer pairs are rewritten as declarative phrases, decomposed into triples, and assigned subject and object types using local Wikipedia context.
  • A Data Creation Pipeline: Counterfactual examples randomly swap same-type entities, while fictional examples inject LLM-generated entities.
  • A Data Creation Pipeline: A later 108-row heldout sample uses the same design and is balanced across variant, language, and parameter count.
  • A Data Creation Pipeline: Annotators hid judge scores and model identifiers, discussed unclear cases, and produced one final label per row rather than independent adjudications.

D Judge Robustness and Audit Details

The audit compares judge behavior, language-condition results, and robustness to prompt changes. It shows substantial disagreement in some settings and warns that judge-specific biases can distort conclusions about context–memory conflict.

  • Judge disagreement: DeepSeek reports an FA-to-FI-to-CFA rise in hallucination labels for every model-size group, whereas Kimi shows a different pattern.The paper interprets this difference as judge sensitivity because structured labels are less stable than scores.
  • Judge disagreement: Upper Sorbian contributes 44.5% of total absolute score difference and 79.8% of the 3,337 maximum four-point gaps despite representing one quarter of the data.
  • Judge robustness: DeepSeek’s disclosure effect exceeds its noise floor, while Kimi’s does not; DeepSeek becomes more lenient and less accurate when the class is disclosed.
  • Judge robustness: Neither judge reliably predicts factuality: DeepSeek labels 44.7% of counterfactual sentences factual, while Kimi labels 44.4% of factual sentences counterfactual.

E Additional Results

Additional analyses show that faithfulness generally declines as inputs grow, while model-size patterns and perceived-class differences depend on the judge. A proprietary DeepSeek generator provides a high-faithfulness reference, and language-specific errors are concentrated in Upper Sorbian.

  • Input size: Faithfulness generally decreases as the number of input triples increases under both judges.DeepSeek is monotonic for every model, while Kimi is nearly flat for GPT OSS 120B and rises from one to 2–3 triples for Gemma4 E2B.
  • Model size and uncertainty: Confidence intervals are narrow, and faithfulness broadly tracks model size under both judges, unlike the less regular fluency ordering.The reported intervals indicate that per-model faithfulness orderings are not sampling noise.
  • Judge sensitivity: DeepSeek’s perceived-class differences are larger than Kimi’s, and two models reverse the FA-over-CFA direction under both judges.Under Kimi, five of nine models show a Pred. FA advantage over Pred. CFA with disjoint intervals, while Gemma4 E2B and Tiny Aya reverse it; under DeepSeek, FA exceeds CFA for seven of nine models, with the same reversals.
  • Perceived classes: Under DeepSeek, perceived CFA is more difficult in every model-size group, with Pred. FA − Pred. CFA intervals of [0.09, 0.15], [0.04, 0.11], and [0.35, 0.47].These Bonferroni-adjusted 95% family-wise confidence intervals correspond to large, medium, and small models, respectively.
  • Proprietary reference generator: DeepSeek as a generator reaches mean faithfulness 4.92 and Upper Sorbian faithfulness 4.93, while its 0.03-point variant spread is smaller than every open-weight model’s.Its classification accuracy is 49.3%, below gpt-oss-120b, separating counterfactual recognition from resistance to derailment.
  • Per-language errors: Upper Sorbian prompts produce language or transformation errors in 37.6% of outputs under DeepSeek and 36.2% under Kimi, versus 0.1% for English under both.Other error categories are less stable across judges; small models often fail to produce an Upper Sorbian sentence.

F Experimental Prompt Templates

The experiments classify RDF inputs by plausibility, generate multilingual data-to-text outputs, and evaluate faithfulness under explicit instructions to treat every triple as authoritative. The prompt templates also define scoring gates and error-handling rules for omissions, reversals, unsupported claims, language, and transformation failures.

  • F Experimental Prompt Templates: The experiments use classification and data-to-text prompts in English, Czech, Slovak, and Upper Sorbian, with language-specific instructions and output requirements.The templates fill in example-specific values such as triple count, data category, target language, and generated sentence.
  • F.1 Classification Prompt: Inputs are classified as FA, CFA, or FI, with one label assigned to the whole RDF set rather than to individual triples.FA denotes factual information, CFA altered relationships or values involving real entities, and FI fictional entities.
  • F.2 Data-to-Text Generation Prompt: Generation requires a fluent sentence expressing all triples while adding no unsupported facts, omitting no facts, and preserving numbers, dates, and values.Predicates must be translated into English, while proper named entities remain in their original form.
  • F.2 Data-to-Text Generation Prompt: The generation prompts require translation of non-proper entities and predicates while allowing proper names to remain unchanged.The instructions distinguish descriptive or common-noun terms from names of people, places, organizations, works, and other named entities.
  • F.3 Conflict-Acknowledgment Addition: The conflict-acknowledgment ablation instructs models to treat even implausible or contradictory triples as authoritative and to express them without correction, hedging, refusal, or disclaimers.Equivalent additions are used for the other target languages.
  • F.4 Faithfulness-Judge Prompt: The faithfulness judge treats triples as the only source of truth, accepts paraphrases, and evaluates semantic roles rather than surface word order.The prompt explicitly accommodates case-marked languages and predicate labels whose object defines a semantic role.
  • F.4 Faithfulness-Judge Prompt: The judge penalizes unsupported, contradicted, reversed, omitted, over-specific, or incorrectly transformed claims, while accepting valid paraphrases and name changes that preserve identity.Copied triple wording, inflection, diacritics, transliteration, and grammatical word order are not automatically errors.

G Mistake Categories and Examples

The paper groups judge-identified disagreements with source triples into six non-exclusive mistake categories and illustrates each with concrete multilingual generation errors. The examples distinguish unsupported substitutions, reversed relations, omissions, incorrect predicates, entity changes, and language or transformation failures.

  • Category mapping: The analysis maps fine-grained judge labels into six non-exclusive categories: LANG, HALL, REV, MISS, REL, and ENT.Outputs can receive multiple categories because one response may contain several disagreement types.
  • LANG: LANG covers wrong-language outputs, retained RDF predicates, and strongly degenerate repetitions.A Czech sentence produced for an English request follows the input content but is still classified as a language error.
  • HALL: HALL covers information unsupported by the triples, including replacing a counterfactual fact with knowledge-consistent information.For the counterfactual short name PIENAP, Tiny-Aya instead generated a factual description of Austria and received a faithfulness score of 1 from DeepSeek.
  • REV: REV occurs when an output preserves the entities and general relation but swaps their semantic roles.The triple says Rákosníček is voiced by Ferdinand d’Este, whereas the output makes Rákosníček the voice of Ferdinand d’Este.
  • MISS: MISS occurs when an entity, relation, or complete source triple is not verbalized.One output states the foundation-stone date and event but omits that they refer to the National Theatre in Prague.
  • REL: REL occurs when the output connects the relevant entities or values with a different predicate.Changing a person’s field of neurology into the occupation or identity of neurologist changes the relation’s meaning.
  • ENT: ENT applies when a source entity or value is replaced, misidentified, or assigned incorrectly.Replacing Slovensko with Slovenia retains the number and relation but produces a claim about a different country.
  • Category interactions: The categories identify each output’s main disagreement with the triples but are not independent phenomena.Entity replacement can also make a claim unsupported, while omitting an argument can distort interpretation of the remaining relation.
Loading 2609.09363v1…