Not Every Mention Is a Claim: Rethinking Object Hallucination Detection in Vision-Language Models

1 Introduction

• A mention’s usage—existential or non-assertive—depends on the text, while presence depends on the image; detectors should flag only existential mentions of absent objects.Non-assertive examples include objects mentioned as depictions, intended purposes, negations, or hedged possibilities.Source: §1 · p. 1
• Existing benchmarks annotate object presence but not usage, so they cannot distinguish non-assertive absent mentions from genuine hallucinations, and detectors trained on them inherit that conflation.Caption-based metrics also penalize non-assertive language; a mitigation method could improve scores by reducing hedging or negation without changing hallucination frequency.Source: §1 · pp. 1–2
• NAME-Bench contains 25,497 object mentions from five VLMs, labeled for usage and presence, with non-assertive mentions divided into four subtypes.The subtypes are depictive, purpose, negation, and hedged.Source: §1 · pp. 1–2
• On the four newer VLMs, reaching 5–10% non-assertive flags takes five to ten times more data than on LLaVA-1.5 and costs real-hallucination recall.At the natural mention rate, 16–46% of non-assertive mentions remain flagged; both the target rates and the recall trade-off are reported for the newer models.Source: §1 · pp. 2–3
• On LLaVA-1.5, non-assertive examples equal to 10% of training hallucinations reduce flagged mentions from 18–52% to 4–6%, with little change in caught hallucinations.The other four models require five to ten times more data to reach a comparable rate, and then detectors catch fewer real hallucinations.Source: §1 · pp. 2–3

Hallucination Benchmarks and Metrics

• Caption benchmarks such as CHAIR match named objects to image annotations [Rohrbach et al., 2018], while THRONE judges implied existence [Kaul et al., 2024] but counts over half of hedged and depictive mentions as hallucinations.Other work uses polling questions or evaluates attributes, relations, events, open-ended responses, and perceptual versus language-prior errors.Source: §2 · p. 3

Hallucination Detection

• Hallucination detectors use attention, token statistics, hidden representations, visual grounding, and embedding similarity; their training and evaluation commonly rely on presence-based labels [Zhang et al., 2025; Fieback et al., 2024; Zohrabi et al., 2026; Gunjal et al., 2024].Other approaches predict risk before generation or use cross-modal and visual attention.Source: §2 · p. 3

Factuality and Non-assertion in Text

• Factuality research studies certainty, possibility, and counterfactuality [Saurí and Pustejovsky, 2009, 2012; De Marneffe et al., 2012; Rudinger et al., 2018], with related work on hedging, negation, and intensional verbs.Claim-level evaluation verifies atomic assertions rather than all mentions [Min et al., 2023]; this paper applies such distinctions to object mentions in VLM captions.Source: §2 · p. 3

3 Benchmark

• NAME-Bench tests whether detectors distinguish asserted presence from mere mention using usage- and presence-labelled object mentions from five VLMs.The benchmark uses a common evaluation protocol.Source: §3 · p. 3

3.1 Task Definition

• A mention is concrete when asserted and present, hallucinated when asserted and absent, and non-assertive when not asserted, regardless of presence.Non-assertive mentions are divided into purpose, depictive, hedged, and negation subtypes.Source: §3.1 · pp. 3–4

3.2 Captioning and Mention Extraction

• Five VLMs caption COCO images, and candidate mentions are extracted using a curated list of synonyms and plurals for the 80 COCO categories.The data are on-policy: each model's mentions come from its own captions; COCO instance annotations establish presence.Source: §3.2 · p. 4

3.3 Labelling Usage

• GPT-4o judges usage from caption text alone, while COCO instance annotations determine presence; about 45% of candidates initially labeled non-assertive fit none of four subtypes and are excluded from NAME-Bench.Manual labels matched GPT-4o’s usage labels for 90.9% of mentions; agreement on category among mentions with agreed usage labels was 92.1%.Source: §3.3 · p. 4

3.4 Statistics

• Non-assertive mentions make up 1.3% of LLaVA-1.5's object mentions and 4.2–5.0% for the four newer models; hallucination rates are 28.2% and 5.7–8.9%, respectively.NAME-Bench contains 25,497 mentions and oversamples non-assertive mentions for subtype-level analysis.Source: §3.4 · p. 5

3.5 Evaluation Protocol

• Detectors are calibrated to flag 10% of concrete mentions, then evaluated on held-out data using non-assertive false-positive rate and hallucination true-positive rate.The splits are 53.3% training, 13.3% calibration, and 33.3% test; results average three seeds, with key comparisons also repeated on a position-matched subset.Source: §3.5 · p. 5

4.1 Results on Existing Metrics and Detectors

• CHAIR and THRONE each count about half of non-assertive mentions as hallucinations, while neither counts concrete mentions; neither handles hedges well.CHAIR flags 30–68% of negations versus THRONE’s 0–14%; THRONE flags depictive mentions more often than CHAIR (58–66% vs. 36–52%).Source: §4.1 · p. 6
• On newer models, non-assertive mentions account for 28–37% of CHAIR’s counted hallucinations, versus about 2% on LLaVA-1.5.CHAIR recovers only 64–92% of the paper’s hallucinations under its reference-based presence convention.Source: §4.1 · p. 6
• Every evaluated detector flags substantial non-assertive mentions, with error rates varying by subtype and detector family.On newer models, trained probes flag 56–88% of hedged and 41–77% of depictive mentions; SVAR flags 97–100% of negations, while NLL and entropy stay below 35% in every category.Source: §4.1 · p. 6
• The comparison includes trained-probe false-positive rates and hallucination true-positive rates across budgets that add non-assertive examples as training negatives.Table 5 also reports natural-share and maximum-available budgets, but its caption alone does not establish outcome values.Source: Table 5 · p. 7
• Mention position alone separates concrete from hallucinated mentions, but position-matching leaves trained probes’ non-assertive false-positive rates largely unchanged.Position-matched non-assertive false-positive rates remain 48–79% on newer models, while concrete mentions at the same positions are flagged at 11–21%.Source: §4.1 · pp. 6–7

5.1 Training on Non-Assertive Examples

• On LLaVA-1.5, about 100 non-assertive training examples bring all five trained probes’ false-positive rates to 4–6%; newer models need five to ten times more data to reach 5–10%.At maximum mixing, hallucination true-positive rates fall from 64% to 51% on Qwen2.5-VL-7B and from 71% to 61% on LLaVA-OneVision-2.Source: §5.1 · pp. 7–8
• Excluding hedged or depictive examples from training raises false-positive rates two- to six-fold, unlike excluding negation or purpose examples.For newer models, unseen hedges rise from 18–24% to 44–58%, and unseen depictive mentions from 8–15% to 26–39%.Source: §5.1 · p. 8

5.2 Text Filter

• The text filter catches 62–69% of non-assertive mentions and reduces training-free detectors’ false-positive rates from 14–57% to 2–19%, but removes 2–4% of newer-model hallucinations.The filter removes 0.2% of hallucinations on LLaVA-1.5.Source: §5.2 · p. 8

5.3 Why Detectors Fail

• On newer models, hidden-state probes distinguish absent from present objects among non-assertive hedges, while LLaVA-1.5 shows little separation.Hedge AUC is 0.70–0.78 on the four newer models versus 0.52 on LLaVA-1.5; retraining probes while treating hedges as claims does not eliminate the remaining errors.Source: §5.3 · p. 8

6 Conclusion

• On four newer models, non-assertive mentions make up 28–37% of CHAIR’s hallucinations; correcting detectors takes five to ten times more data than on LLaVA-1.5 and costs recall.The analysis suggests these mentions share textual and internal signals with hallucinations.Source: §6 · pp. 8–9
• The paper concludes that hallucination evaluation should distinguish what a caption asserts from what it merely mentions.This follows from the observed conflation of non-assertive mentions with hallucinations by existing metrics and detectors.Source: §6 · pp. 8–9

Limitations

• The benchmark covers only 80 COCO object categories in COCO-image captions from five open models, using one prompt and at most 128 new tokens.Extending coverage to attributes, relations, other image domains, and closed models is identified as a next step.Source: Limitations · p. 9
• Usage labels come from GPT-4o and agree with two annotators on 90.9% of 250 mentions (κ = 0.79); presence labels use COCO annotations, which miss some objects.About 45% of non-assertive mentions are excluded because they are compounds or fit no single subtype, and hedges are labelled non-assertive by convention.Source: Limitations · p. 9

A Category Vocabulary

• Candidate mentions are matched case-insensitively as whole words against COCO category names and listed surface forms, with regular plurals added automatically.Irregular plurals are listed explicitly; orange is matched but excluded from NAME-Bench.Source: A Category Vocabulary · p. 12

B Detectors

• All detectors score one mention using signals from a teacher-forced pass over the generated prompt and caption; thresholds flag 10% of calibration concrete mentions, and held-out results report FPR, TPR, and AUC.Trained detectors use logistic regression or the papers’ MLPs; training-free detectors use published definitions and default settings, with calibration data used only for thresholds.Source: B Detectors · p. 12, Table 8 · p. 13

C Labelling Prompts

• GPT-4o judges whether each candidate is referred to and asserted, then assigns non-assertive subtype; GPT-4o-mini checks whether a mention belongs to a compound expression.Deterministic rules use COCO instance annotations to split existential mentions into concrete or hallucinated, discard ambiguous or non-referring mentions, and apply a negation-cue override except for depictive mentions.Source: C Labelling Prompts · pp. 12–13

D Teacher-Forced Control

• Table 10 repeats the mixing experiment with every detector reading LLaVA-1.5 captions instead of its own model’s captions.The passage identifies the control’s scope but does not provide its results.Source: D Teacher-Forced Control · p. 16

E Text Filter

• The text filter is a class-balanced logistic regression trained to distinguish non-assertive mentions from concrete and hallucinated mentions, using natural-share non-assertive data and a fixed 0.5 threshold.After filtering, detector thresholds are recalibrated to maintain 10% concrete FPR; the evaluation tracks the share of held-out hallucinations incorrectly removed.Source: E Text Filter · p. 16

Voting Rule

• THRONE asks a language model whether each mentioned object class is implied to be in the image, then compares its verdict with COCO annotations.The reported setup uses three Flan-T5 sizes, three question formats, and nine yes/no votes per mention.Source: F THRONE: Voting Rule and Judge · p. 16
• With Flan-T5’s unanimous rule, 61% of newer-model non-assertive mentions are disqualified, while 46% of the mentions it scores are counted as hallucinated.The rule also disqualifies 22% of newer-model real hallucinations, so the paper reports majority voting to give every mention a verdict.Source: F THRONE: Voting Rule and Judge · p. 16

A Stronger Judge

• Replacing Flan-T5 with Llama-3.1-8B-Instruct reduces but does not remove errors: majority voting counts 41% of newer-model non-assertive mentions as hallucinated.It counts 50% of hedged and 47% of depictive mentions as hallucinated; negation remains near zero for both judges.Source: F THRONE: Voting Rule and Judge · p. 16

G Position-Matched Evaluation

• Table 13 compares concrete-versus-hallucination AUC before and after matching token position.The passage does not state the comparison’s outcome.Source: G Position-Matched Evaluation · p. 16

H Mixing by Category

• Table 14 breaks the mixing results down by non-assertive category for the two strongest probes.The passage does not give category-level results.Source: H Mixing by Category · p. 16

I Treating Hedges as Claims

• Table 15 repeats the §5.3 comparison across all five models with every hedge re-labelled as a claim.The passage identifies the changed convention and model coverage, but not the outcomes.Source: I Treating Hedges as Claims · p. 16

• Negation is concentrated on a few concepts, mostly person and car.Source: J Concept Distribution · pp. 16–17
• Depictive and hedged mentions span 41–71 concepts.Source: J Concept Distribution · p. 17
• The false-positive problem extends beyond NAME-Bench’s four subtypes to mentions that do not refer to the target object.Source: J Concept Distribution · p. 17

• 25–37% of mentions initially labelled non-assertive are compound matches, while another 6–13% are valid mentions outside the four subtypes.The latter are mostly generic statements.Source: K Excluded Non-Assertive Mentions · p. 17
• On the four newer models, trained probes flag 36–75% of Other mentions versus 48–80% of retained subtypes; SVAR flags 61–70% versus 46–57%.Source: K Excluded Non-Assertive Mentions · p. 17
• The failure does not depend on labelling hedges as non-assertive: the probes also test held-out depictive, purpose, and negation mentions under that alternative convention.Source: Table 15 · p. 19