When More Metadata Misleads: Probing Field-Level Grounding in LLM-Generated Museum Descriptions

1 Introduction

• Museum catalogue fields differ in epistemic status, and images can support visible descriptions but cannot ground provenance, date, or historical associations.Uneven field coverage also reflects acquisition histories and cataloguing conventions across regions.Source: §1 · p. 1
• Prior evaluations often aggregate quality or report regional gaps observationally, leaving metadata quantity, field identity, image access, and domain priors difficult to disentangle.Cultural-heritage work has less often examined how metadata configurations change what models include, omit, or infer.Source: §1 · pp. 1–2
• The study combines natural-metadata analysis of 10,250 artefacts with controlled probes that vary field count, field type, and image access within objects.It evaluates reference similarity, field coverage, and claim-level factuality.Source: §1 · p. 2
• More fields improve coverage of provided metadata but can increase unsupported inference over withheld fields, while contextual and content fields produce distinct failure patterns.Place-level variation also reflects field–region interactions and documentation practices, motivating field- and claim-level evaluation.Source: §1 · p. 2

Data-to-text generation

• Data-to-text research spans generation from meaning representations, RDF triples, structured records, and selected table cells, with fidelity and content realization as central concerns.This paper extends that line by analyzing faithfulness at the metadata-field level.Source: §2 · p. 2

Generation from cultural heritage data

• Cultural-heritage LLM research includes exhibition question answering, collection chatbots, image-archive captions, keyword generation, and visitor-facing text from collection databases.The paper generates descriptions from structured records combining images and metadata.Source: §2 · p. 3

Metadata completeness

• Metadata completeness can vary systematically because records reflect subjective, culturally situated documentation practices and aggregation judgments.The study examines how completeness, image availability, and region interact with generation.Source: §2 · p. 3

3 Experimental Setup

• The two-phase framework measures generation under naturally available metadata, then probes metadata composition while holding object identity fixed.Phase 1 reflects realistic variation; Phase 2 is designed to isolate effects that natural records confound.Source: §3 · p. 3, §3.2 · p. 3

3.1 Task and Dataset

• The task generates descriptions from TimeTravel records covering 10,250 artefacts, ten regions, nine metadata fields, images, and reference descriptions.Objects average 6.8 populated fields, with documentation varying across regions.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 Two-Phase Experimental Design

• Phase 2 uses paired conditions over the same objects, with a three-field baseline and matched-count context and content arms that add distinct fields.Object identity, prompt template, and image access remain fixed across matched pairs; remaining conditions bound field range or withhold images.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 Evaluation

• The evaluation is organized around three dimensions.The supplied passage does not specify those dimensions.Source: §3.3 · p. 4

4.1 Phase 1: Surface metrics invert documentation completeness

• In Phase 1, descriptions lengthen as field count rises while BERTScore falls, and surface-metric rankings almost perfectly invert documentation completeness.The observed field-count and region co-variation means their individual contributions cannot be disentangled from Phase 1 results.Source: §4.1 · p. 5

4.2 Metadata composition creates grounding trade-offs

• Primary-field coverage stays between 0.90 and 0.93 as field count changes, while bonus wrong rates exceed correct rates at every observed count and rise alongside bonus completion.The Figure 2 caption identifies primary and bonus-field rates by observed metadata count; the reported wrong bonus rates are 0.18–0.24, versus correct rates rising from 0.05 to 0.19.Source: Figure 2 · p. 6, §4.2 · p. 6
• S_CTX conditions have higher primary coverage and lower hallucination than matched-count S_CONT conditions, while S_CONT5 has the highest primary hallucination rate.Paired tests show S_CTX exceeds S_CONT in primary coverage by 5.6–7.4 pp and falls below it in primary hallucination by 6.2–8.5 pp (all p < 0.001).Source: Table 3 · p. 6, §4.2 · p. 6
• Subjects and Technique have the highest primary hallucination rates, whereas context fields such as Find spot and Place show more omission than unsupported elaboration.Inscription is the only field for which bonus correct exceeds bonus wrong (16.7% vs. 10.6%).Source: §4.2 · p. 6, Table 4 · p. 7

4.3 Grounding reliability varies by region

• Regional averages differ only narrowly in BERTScore, but claim-level results distinguish regional error profiles, including high hallucination in Central America and high silence in China.Regional comparisons are conditional on populated fields; the near-zero Technique hallucination rate in the Roman Empire reflects that Technique is absent from its records.Source: §4.3 · p. 7

4.4 Image helps sparse grounding but increases extrapolation

• Removing images under sparse metadata lowers primary coverage and raises primary hallucination, but full-metadata image ablation yields no significant difference in those measures.Under sparse metadata, coverage falls from 93.3% to 88.5% and hallucination rises from 5.8% to 9.1%; image-inclusive conditions also have more bonus attempts and more bonus wrong mentions in absolute terms.Source: §4.4 · pp. 7–8
• LLAMA-VISION shows the largest model-level R-to-T primary-coverage difference under full metadata, falling from 0.882 to 0.796 (∆= 0.086).The aggregate R-to-T difference is 0.6 pp.Source: §4.4 · p. 8

5 Discussion

• More metadata can improve coverage of provided fields while increasing unsupported inference over withheld fields, so stronger visible-field grounding can coincide with weaker respect for the metadata boundary.The discussion also notes that longer outputs create more opportunities for unsupported elaboration and can receive lower reference similarity.Source: §5 · p. 8
• Context fields tend toward omission when withheld, while content fields invite more elaboration and hallucination; Materials occupies an intermediate position.The described mechanisms include transferring values across neighbouring fields, narrowing broad labels with domain priors, and turning visual observations into field-specific claims.Source: §5 · p. 8
• A plausible or visually supported statement may still be unsupported by the specific catalogue field being evaluated, making field-to-claim verification central to assessment.Examples include treating a Find spot as a Place of origin or adding image-visible details absent from the catalogue subject vocabulary.Source: §5 · pp. 8–9
• Images can reduce hallucination in sparse conditions yet amplify content-field elaboration beyond the catalogue record when metadata is richer.The evaluation treats catalogue metadata as the reference schema, not as an exhaustive description of the object.Source: §5 · p. 9
• Reference similarity, field-level NLI, and claim-level verification capture different aspects of quality; evaluation should track which field supports a claim and whether it stays within that field’s scope.NLI-based coverage can indicate semantic consistency without correctly reproducing the catalogue value.Source: §5 · p. 9

• Increasing field count raises coverage of provided fields but also increases unsupported claims about withheld fields.At a fixed field count, contextual fields are often omitted, while content fields are incorporated more readily but carry higher hallucination risk.Source: §6 · p. 9
• Reference similarity and entailment-based coverage can rank matched conditions differently and do not distinguish field-specific support, contradiction, and omission.Faithfulness assessment requires attributing each generated claim to the metadata field it addresses and checking it against that field’s value.Source: §6 · p. 9
• Images can support visual grounding while also encouraging visual inference beyond the catalogue record.Content fields such as Technique and Subjects also encourage domain-specific elaboration and schema-boundary errors.Source: §6 · p. 9
• Public-facing systems should expose each claim’s provenance so catalogue-supported content can be reviewed separately from content generated from parametric knowledge.Fluent descriptions can present plausible but unsupported interpretation as authoritative object information.Source: §6 · p. 9

• The contextual/content distinction may vary with object type, record format, and catalogue practice.The authors suggest examining categories dynamically based on schema role and object-specific usage.Source: Limitations · pp. 9–10
• Materials is usually grounded reliably when present, but can still prompt unsupported object-type or quality inferences.The example shows that a field category’s typical behavior does not ensure every associated claim is supported.Source: Limitations · p. 10
• Claim-level factuality was assessed with an automatic nugget-verification pipeline and two crowdworkers rather than expert curator annotation.Automated verification may miss subtle curatorial distinctions, so the authors call for expert review, especially for visually plausible claims absent from metadata.Source: Limitations · p. 10
• The evaluation treats catalogue metadata as authoritative, although records may omit details visible in images or known from external sources.Expert references are selective rather than exhaustive, and the authors call for studying how people balance descriptiveness, visual plausibility, and catalogue fidelity.Source: Limitations · p. 10

Ethical Considerations

• In public-facing settings, plausible but unsupported interpretations may be presented as authoritative information about objects.The concern is representational: automated descriptions should not overstate, distort, or homogenise cultural knowledge.Source: Ethical Considerations · p. 10
• Uneven documentation across regions, periods, or communities could allow generated descriptions to reinforce existing documentation imbalances.The authors do not recommend standalone curatorial use and call for domain-expert review and systems that distinguish catalogue-supported, image-inferred, and uncertain claims.Source: Ethical Considerations · p. 10

A.1 Metadata Availability and Place-Relative Sampling

• Field availability varies by region: Subjects is below 10% in the British Isles and Central America, while Inscription ranges from 0% in Central America and China to 100% in Egypt.The passage attributes these disparities to institutional documentation practices and says they motivate place-relative sampling.Source: §A.1 · p. 12
• Phase 2 samples 50 objects per region from the one or two highest-count tiers at the upper end of that region’s completeness distribution.A global completeness threshold was avoided because it would disproportionately exclude lower-field-count regions; the approach represents all ten regions.Source: §A.1 · p. 12

A.2 Collapsed Conditions

• A target condition is collapsed when one or more of its target fields are unpopulated in an object’s catalogue record.Collapse is reported as most frequent in S_CONT6, where Inscription is named as an example.Source: §A.2 · p. 12

A.3 Models and Implementation Details

• Phase 1 generated descriptions for 10,250 objects with four LLMs, while Phase 2 used 17,000 attempted generations across three models and controlled conditions.LLaMA-3.2-Vision was restricted to the four base Phase 2 image-ablation conditions; fewer than five failed or empty Phase 2 responses were excluded.Source: §A.3 · p. 13, Table 7 · p. 14
• NLI field coverage uses DeBERTa-large-MNLI to classify field-value hypotheses against descriptions, counting entailment as coverage, contradiction as active hallucination, and neutral as silence.The hypotheses use templates instantiated with the ground-truth field value.Source: §A.3 · p. 13, Table 8 · p. 15
• Claim-level factuality uses Gemini 2.5 Flash to extract atomic claims and verify each against its relevant catalogue value.Because of API budget constraints, nugget verification covered 6,120 descriptions stratified by region, model, and condition.Source: §A.3 · p. 13

A.4 Claim-Level Factuality Pipeline Validation

• Span validity was 98% across 6,120 evaluated descriptions, and NLI predictions agreed with nugget verdicts at Cohen’s κ > 0.70 over about 30,000 primary field–generation pairs.Most disagreements occurred when NLI returned neutral but Stage 2 found span-level support.Source: §A.4 · p. 14

A.3 Models and Implementation Details

• Generation prompts request continuous prose grounded in supplied metadata and allow an appended image in image-enabled conditions.The prompt asks for cultural, historical, or artistic context where metadata supports it.Source: §A.4 · p. 14

B.1 Phase 1 Results

• LLaMA-Maverick has the highest mean scores across reported metrics, while Qwen2.5-VL-32B performs worst overall; differences from other models are significant.All model comparisons have p < 0.05 under paired Wilcoxon signed-rank tests.Source: §B.1 · p. 15, Table 10 · p. 16
• Place significantly affects every model–metric combination, with the largest effect for BERTScore, explaining up to 43% of score variance.Effects are smaller for METEOR (η2 ≈ 0.19–0.21), ROUGE-L (η2 ≈ 0.10–0.15), and BLEU (η2 ≈ 0.07–0.11).Source: §B.1 · p. 15
• Regional rankings agree across models most strongly for BERTScore (W = 0.990) and least for BLEU (W = 0.310, p = 0.125).METEOR agreement is W = 0.877; ROUGE-L agreement is W = 0.691.Source: §B.1 · p. 15
• Higher field counts are associated with lower BERTScore in Greece (ρ = −0.350), India (ρ = −0.411), and Iran (ρ = −0.337), but not in Roman Empire (ρ = 0.040).In Greece, India, and Iran, output length rises by 14.9–21.0 words per additional field; Roman Empire is the only region where length does not rise.Source: §B.1 · pp. 15–16

B.2 Phase 2 Results

• GPT-4o achieves the highest BERTScore (0.884) and METEOR (0.251), while Qwen2.5-32B achieves the highest BLEU-1 score (0.398).BERTScore is compressed across models, with a range of 0.009.Source: §B.2 · p. 16
• Field-count–BERTScore curves peak at different counts by region, while word-count correlations are uniformly negative and stronger overall than field-count correlations.Overall correlations are r = −0.543 for word count and r = +0.201 for field count; China and Central America have the strongest length–BERTScore associations.Source: §B.2 · pp. 16–17

B.2.1 Per-field Analysis

• Primary-field coverage exceeds 0.90 for most field–region combinations, with the lowest coverage concentrated in Technique and Subjects for Central America and China.Figure 4 displays primary outcomes alongside their corresponding bonus outcomes by region and field.Source: §B.2.1 · p. 17
• Bonus correct rates are uniformly low; Inscription is the only withheld field whose bonus correct rate exceeds bonus wrong, while Find spot recovery is near zero.Inscription bonus correct reaches 0.30 for Roman Empire, 0.28 for India, and 0.25 for British Isles.Source: §B.2.1 · p. 17

C.1 Qualitative Error Patterns

• Place and Find spot errors include substituting a Find spot value for Place, narrowing broad labels with other metadata, adding unsupported qualifiers, and redundant geographic documentation.For example, a record with Place: Central America may be described as Meso-America, a value present only in Find spot.Source: §C.1 · pp. 17, 19
• Technique errors include replacing broad provided terms with specialized subtypes and, when withheld, generating stylistic claims instead of recovering manufacturing processes.Withheld values such as painted, carved, or gilded may prompt claims about naturalism, craftsmanship, or calligraphic style.Source: §C.1 · p. 19
• Models expand sparse controlled Subjects labels with image-consistent details, while Materials mentions are comparatively well grounded and errors tend to involve unsupported inference or neighbouring-field confusion.A label such as bird may be expanded with foliage or vine motifs; material-triggered inferences include treating pottery as a bowl.Source: §C.1 · pp. 19–20
• The dominant qualitative error is schema-boundary overreach across neighbouring fields, visual priors, and curatorial conventions, making plausible prose violate the intended field–value–claim relation.Table 18 summarizes recurring mechanisms rather than exhaustive categories.Source: §C.1 · pp. 17, 20, Table 18 · p. 23

C.2 Generation Samples

• Figure 5 compares descriptions for three objects under sparse, expanded sparse, and all-fields-with-image conditions alongside museum references; Figure 6 shows GPT-4o outputs under the all-fields-with-image condition across civilisations.Descriptions in Figure 5 are truncated to 100 words.Source: §C.2 · p. 20, Figure 5 · p. 21, Figure 6 · p. 22