HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning

• HalDec evaluates VLM image-text alignment by identifying caption errors, and can also help filter unaligned image-caption pairs from VLM training data.Source: §1 · p. 2
• Evaluating detector generality requires variation in captioners, image domains, and hallucination types because each can introduce distinct language styles and error patterns.Source: §1 · p. 2
• Existing HalDec datasets are constrained by annotation costs, limited model coverage, and insufficient scale; rule-based or artificial errors may not represent diverse VLM-generated hallucinations.Source: §1 · pp. 2–3
• HalDec-Bench combines diverse VLM-generated captions with human sentence-level, hallucination-type, and segment-level annotations to assess detectors across varied styles and error types.Source: §1 · p. 3
• HalDec-Bench reveals model-performance differences not clearly observable in existing image-reasoning and image-text alignment benchmarks.Source: §1 · p. 3
• Detectors tend to judge response-initial sentences as correct regardless of their actual correctness, while strong VLM filters can substantially reduce dataset noise across strong and weak captioners.Source: §1 · p. 3

• Existing HalDec datasets often lack diverse captioners or sufficient samples, while SUGARCREPE and ARO use simple sentences and language-model or rule-based perturbations.Source: §2 · p. 3
• HalDec-Bench exceeds prior alignment datasets in scale, uses naturally occurring VLM hallucinations and broader vocabulary, and records each pair’s source model for model-specific analysis.Source: §2 · p. 3, Table 1 · p. 4
• CHAIR evaluates object-level alignment using an object detector, but its coverage and accuracy constrain its ability to capture varied hallucination types and caption styles.Source: §2 · p. 4
• Prior work has fine-tuned VLMs and used tools such as open-vocabulary detectors, OCR, and uncertainty estimates, yet VLMs’ fundamental caption-hallucination detection ability remains unclear without a benchmark.Source: §2 · p. 4

3 Datasets

• Sentence-level labels facilitate finding unaligned descriptions, while span-level labels and hallucination categories support deeper analysis; span annotation is more subjective.Source: §3 · pp. 4–5

3.1 Collecting Image-Caption Pairs

• The benchmark uses ten image-to-text models and also includes images from five text-to-image models to test detector robustness across diverse image-sentence pairs.Source: §3.1 · p. 5

3.2 Annotation

• Annotators judge whether image details are correctly described; the protocol labels indeterminate cases unknown, while reviewed annotations include sentence-level, segment-level, and eight-category labels.Source: §3.2 · pp. 5–6, Figure 3 · p. 6

3.3 Overview of the Dataset

• Captioner errors often concern subtle visual details or object relationships, making them difficult to detect and requiring fine-grained image-text understanding.Source: §3.3 · p. 6
• Approximately 104.3K sentences are annotated, including 87.5K correct and 16.8K incorrect examples; GPT-5 hallucinates in 6.5% of sentences versus 26.6% for ShareGPT.Source: §3.3 · pp. 6–7
• Attribute errors lead for many models, and the dataset includes enough examples to support evaluation by hallucination type.Source: §3.3 · p. 7
• HalDec-Bench exceeds prior datasets in scale with over 104K sentences, longer multi-sentence captions, and a broader vocabulary spanning diverse and long-tail concepts.Source: §3.3 · p. 7

4 Experiments

• The evaluation scores each sentence–image pair independently and computes AUROC within each captioner; random prediction scores 50.Source: §4 · p. 7

4.1 Overview of Results

• HalDec-Bench reveals performance variation across models; Gemini-3-Pro performs best on average, while Llama-4 is the strongest open-source model.TripletCLIP achieves an AUROC of only around 50, and BLIP-2 shows the same trend.Source: Table 2 · p. 8, §4.1 · pp. 8–9

4.2 Detailed Analysis

• Higher MMMU performance does not necessarily mean higher HalDec-Bench performance, and stronger captioners tend to produce harder-to-detect errors.Qwen-3-VL-235B and Qwen-2.5-VL-32B outperform Llama-4 on MMMU, while Llama-4 outperforms them on HalDec-Bench.Source: §4.2 · pp. 10–11, Table 3 · p. 11
• Detectors assign higher correctness scores to earlier sentences regardless of correctness; Llama-4 also assigns higher likelihood to hallucinated sentences.The likelihood analysis found this tendency both with a randomly sampled image and with the corresponding image, while Llama-4 still performs strongly on some caption sources.Source: §4.2 · p. 11
• Filtering to the top 20% of Llama-4 alignment scores substantially reduces noise across caption sources, while ensembling improves performance in most cases.For GPT-4o captions, filtering reduces the noise rate to 5.0%; ensembling can drop on Llama-4 captions and brings limited gains on Gemini-3.Source: §4.2 · pp. 12, 14, Figure 8 · p. 12, Table 6 · p. 14
• For text-to-image pairs, Stable Diffusion errors are easier to detect than errors from many proprietary generators, and GPT-5 outperforms other models in all reported cases.Source: §4.2 · p. 13, Table 4 · p. 13
• Segment localization remains difficult: Llama-4 (400B), the best model, localizes only 24.2% of hallucinated segments on average.GPT-4o mini also outperforms Qwen-2.5 (32B) on localization, unlike their sentence-level comparison.Source: §4.2 · p. 13, Table 5 · p. 13
• Detectors can struggle on their own outputs, while Direction and Number hallucinations remain difficult to detect.The self-evaluation analysis reports lower AUROC on self-generated captions; Direction errors require fine-grained visual understanding, and Number errors persist even in advanced models.Source: Figure 9 · p. 12, §4.2 · pp. 13–14, Figure 10 · p. 14

5 Conclusion

• HalDec-Bench supports systematic evaluation across caption styles and hallucination types, and strong performance on existing benchmarks does not necessarily transfer to it.Source: §5 · p. 15
• Detector judgments show positional bias, favoring sentences near the beginning of a response.Source: §5 · p. 15
• Using strong VLMs as filters can reduce dataset noise in image-caption pairs.Source: §5 · p. 15

A Limitation

• Sentence-by-sentence scoring targets lightweight detection but ignores context across consecutive sentences, which might improve performance.The authors also report that larger-parameter VLMs perform better.Source: A Limitation · p. 20

B The Use of Large Language Models (LLMs)

• LLMs were used only to polish grammar, clarity, and readability; the authors state they did not influence research design, analysis, interpretation, or results.The authors claim scientific content, data, and conclusions as their own.Source: B The Use of Large Language Models… · p. 20

C Attribution to an icon

• The authors credit Freepik - Flaticon for the chatbot icons used in Fig. 1.Source: C Attribution to an icon · p. 20

D.1 Image-Caption Collection

• CC12M and COCO 2017 validation images were clustered into 50 domains, with 40 images sampled per cluster for 2,000 images per dataset.The collection also employs five text-to-image models.Source: §D.1 · p. 20
• For Captioner models, one of five instructions was randomly selected; instructions requested detailed image descriptions, with variants limiting content to confidently determined or clearly visible and verifiable elements, minimizing aesthetic descriptions, or avoiding emotional language.Source: §D.1 · p. 21

D.2 Voting and Quality Control

• Annotators completed a pilot, were screened for quality, and received ongoing batch checks, with re-annotation required when work fell below standards.Trusted annotators were assigned 1,000–2,000 images each.Source: §D.2 · p. 22
• Weighted multi-round voting determined labels, with stricter review of suspected hallucinations and author adjudication of ambiguous cases.A trusted annotator’s incorrect label or two annotators’ incorrect labels triggered manual review.Source: §D.2 · p. 22

D.3 Annotation Quality

• 97% of 539 sampled sentence annotations were judged correct in manual inspection across 100 images from different models.The authors interpret this as evidence of annotation-pipeline reliability.Source: §D.3 · pp. 22–23

D.4 Hallucination Type and Location Annotation

• The eight hallucination types span object, attribute, number, text, relation, location, direction, and illusion errors.They range from fine-grained object or attribute mistakes to broader contextual errors.Source: §D.4 · p. 23, §D.5 · p. 24

D.5 Basic Statistics by Model

• Most models have high positive-sentence rates, so many generated sentences are needed to obtain sufficient hallucination examples; multiple sentences per image enable finer analysis.Source: §D.5 · p. 23

D.6 Additional Analysis

• 104k sentences and 17.5k unique word types give HalDec-Bench larger textual coverage than prior datasets.The comparison is described in Table E.Source: §D.6 · p. 24
• Captioners tend to make more errors on Text and Illustration images than on real-image domains such as Buildings and Scenary.The authors suggest training-data bias as a possible explanation.Source: §D.6 · p. 24
• Incorrect sentences occur most frequently in positions two to four; rates decline beyond position six, where sentences often serve as conclusions rather than detailed descriptions.The authors note that the first sentence is also less likely to contain hallucinations, possibly because it summarizes the image.Source: §D.6 · pp. 24–25
• The dataset covers varied hallucination types, and Fig. C provides example annotations with hallucinations highlighted in red.Source: §D.6 · p. 25, Figure C · p. 25

E.1 Details of Evaluation

• 104k sentences from nine models and 15 types are reported for HalDec-Bench, which offers human-annotated Captioner hallucinations for detailed model-wise analysis.The table also reports 17.8 words per sentence, 17.5k vocabulary, and 5.5k unique items.Source: §E.1 · p. 26
• Alignment prompts ask detectors to score sentence correctness from 0–100, while unparseable outputs receive an alignment score of 50.The reported model results include models with failure rates below 5%.Source: §E.1 · pp. 26–27
• Models produce fewer errors in first-position sentences, while errors are common in illustration and text domains and in attribute and text categories.The domain and category summaries describe tendencies across models.Source: §E.1 · pp. 27–28, Figure D · p. 27
• For localization, a predicted span counts as correct at IoU ≥ 0.3, and precision is the proportion of predicted spans judged correct.The prompt asks detectors to mark incorrect caption segments with brackets.Source: §E.1 · p. 28

F Additional Experiments

• Filtering Qwen-2-generated captions using Qwen-3-VL alignment scores improves CLIP image-text retrieval Recall@5 on COCO and Flickr8k.The experiment generates 5M captions and selects approximately 1M using Qwen-3-VL scores.Source: F Additional Experiments · p. 29
• Filtering approximately 1M of 5M Qwen-2-generated captions improves the fine-tuned model’s performance in most cases.The comparison uses randomly selected captions versus captions filtered with Qwen-3-VL scores.Source: F Additional Experiments · p. 29

Does incorporating the preceding sentences lead to improved perfor-

• Adding preceding sentences does not clearly improve alignment; it often degrades Llama-4 instruction following and slightly lowers GPT5-mini performance.GPT5-mini has no parsing errors in this context condition.Source: Does incorporating the preceding… · p. 29
• Chain-of-thought generally improves Llama-4, but gains are marginal or negative for some Captioners and mixed for GPT4.1-mini.GPT4.1-mini's changes depend strongly on the evaluation target.Source: Does incorporating the preceding… · p. 30
• Self-ensembling consistently improves Llama-4 performance across Captioners, with gains tending to saturate as ensemble paths increase.The table presents changes relative to the better model used for ensembling.Source: Table I · p. 30, Does incorporating the preceding… · p. 30
• Advanced VLMs can surpass UniHD without its external open-vocabulary detector and OCR engine.The comparison is reported in Table J.Source: Does incorporating the preceding… · pp. 30–31
• Mean-IoU results are consistent with Table 5, while localizing hallucinated caption segments remains difficult even for performant models.The table reports mean IoU for hallucination localization.Source: Does incorporating the preceding… · p. 31
• VLM outputs can err on object locations, relationships, and small visual details, and Llama-4 scores for GPT-4o captions cluster near 0 and 100.The examples include input images, sentences, and inferred correctness scores.Source: Does incorporating the preceding… · p. 31

G Additional Examples of Annotations

• Additional annotation examples cover captions from ShareGPT, LLaVA, Qwen-2, GPT4o, CogVLM, LLaMA-4, Stable Diffusion, and GPT-Gen.They are presented across Figures J–Q.Source: G Additional Examples of Annotations · p. 31