Source-linked AI summary
SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Models
Amanuel Gizachew Abebe, Yasmin Moslem
TL;DR
LVLM hallucination detection must jointly localize hallucinated spans and estimate confidence reliably. SpanCalib-VLM combines a multimodal discriminative tagger with a generative VLM through Union-Calibrated Fusion, achieving Pearson 0.413 and IoU 0.391 on SHROOM-Visions. The authors report strong clean-response accuracy and release the model weights and code.
Problem
SHROOM-Visions requires character-level hallucinated-span localization together with well-calibrated probability scores.
Method
SpanCalib-VLM fuses a multimodal XLM-RoBERTa-Large–SigLIP tagger with a fine-tuned generative VLM by rescoring proposed spans using calibrated probabilities.
Results
Pearson 0.413 and IoU 0.391 were achieved on SHROOM-Visions, with 91.3% clean accuracy.
Takeaways & Limitations
Principled fusion of calibrated discrimination and deliberative span detection is reported as effective for LVLM hallucination detection.
Takeaways & Limitations
The system can miss hallucinations beyond its 512-token truncation boundary and has hallucinated-span IoU of 0.196 despite Pearson 0.413 calibration.
Abstract
from arXiv · showhide
Detecting hallucinations in Large Vision-Language Models (LVLMs) requires both accurate span localization and well-calibrated confidence scores. Fine-tuned generative VLMs excel at identifying hallucinated text spans but suffer from overconfidence and high inference latency. Discriminative sequence taggers offer deterministic speed and superior calibration but exhibit conservative span recall. We present SpanCalib-VLM, a hybrid dual-system for the SHROOM-Visions Shared Task that combines a multimodal sequence tagger, consisting of XLM-RoBERTa-Large fused with a SigLIP vision encoder via cross-attention, with our fine-tuned generative VLM (Qwen3.5-4B-SHROOM-SFT). Through a Union-Calibrated Fusion strategy, candidate spans from the generative model are re-scored with calibrated probabilities from the sequence tagger. On the SHROOM-Visions English evaluation split, our ensemble achieves a Pearson calibration correlation of 0.41 and an overall IoU of 0.39, with a clean-response IoU of 0.91} and overall detection accuracy of 70.7%. We make our model weights and code publicly available.
1 Introduction
LVLM hallucination detection requires character-level span localization together with calibrated probability scores. SpanCalib-VLM combines generative span proposals with discriminative calibration to address complementary weaknesses.
- SHROOM-Visions requires character-level hallucinated-span boundaries and well-calibrated probability scores evaluated via Pearson correlation.
- Generative VLM fine-tuning achieves high span IoU but suffers from overconfidence and latency up to 23 s per sample with chain-of-thought.
- Discriminative sequence taggers provide single-forward-pass calibrated outputs but produce conservative span boundaries.
- SpanCalib-VLM combines XLM-RoBERTa-Large and SigLIP-2 through cross-attention, uses generative spans as proposals, and recalibrates them with tagger probabilities.
- The system reports Pearson 0.413, IoU 0.391, and 91.3% clean detection accuracy on SHROOM-Visions.
2 Related Work
Prior hallucination benchmarks commonly evaluate sentence- or object-level errors, whereas SHROOM requires joint token-level localization and uncertainty estimation. SpanCalib-VLM’s calibration design targets fine-grained, token-dependent uncertainty beyond global logit adjustment.
- POPE, CHAIR, and MME evaluate hallucination at the sentence or object level, while SHROOM requires token-level span identification paired with probability calibration.
- The SHROOM task frames hallucination detection as a joint localization-and-uncertainty problem.
- Temperature scaling and Platt scaling adjust global logit temperature but do not capture fine-grained, token-dependent span uncertainty.
3 Methodology
SpanCalib-VLM uses a multimodal sequence tagger and a fine-tuned generative VLM, then fuses their outputs by rescoring proposed spans with calibrated character-level probabilities.
- 3.1 Task Formalization: The task represents hallucinations with character offsets, categories, and annotator-consensus probabilities, requiring matching spans with calibrated continuous scores.
- 3.2 System 1: Multimodal Sequence Tagger: The sequence tagger encodes concatenated prompt-response text and fuses projected SigLIP2 patch embeddings into language representations through cross-attention.
- 3.2 System 1: Multimodal Sequence Tagger: Three token-level heads predict binary hallucination probability, continuous calibration score, and one of five error categories.
- 3.2 System 1: Multimodal Sequence Tagger: The composite loss combines binary span detection, mean-squared probability calibration, and category classification, with the MSE term supervised by annotator agreement.
- 3.3 System 2: Generative VLM: The generative Qwen3.5-4B model produces structured JSON spans that are converted into a binary character mask for fusion.
- 3.4 Union-Calibrated Fusion: Union-Calibrated Fusion extracts generative candidate spans, aligns tagger token probabilities to characters, and computes probability-guided ensemble scores.
- 3.4 Union-Calibrated Fusion: The fusion weights are w1 = 0.55 and w2 = 0.45, selected by validation grid search to emphasize the tagger’s calibration while retaining generative spatial proposals.
4 Experimental Setup
The experiments fine-tune and evaluate SpanCalib-VLM across multilingual data, with primary baselines assessed on English and cross-lingual performance assessed across four languages. Validation-based checkpoint selection addresses overfitting during training.
- 4.1 Data: The models are fine-tuned on 15,102 multilingual samples covering English, French, Italian, and Chinese.
- 4.1 Data: Main baselines are evaluated on the English validation split, while cross-lingual performance is evaluated across all four languages on validation and test sets.
- 4.2 Training: Training uses five epochs on NVIDIA A40 GPUs with AdamW, 10% linear warmup, batch size 16, and maximum sequence length 512.
- Training Dynamics: Pearson correlation peaks at epoch 2 with 0.3688, after which validation loss rises despite decreasing training loss; best-checkpoint selection preserves the optimal state.
5 Results
On the English evaluation split, SpanCalib-VLM achieves strong calibration, span overlap, clean-response detection, and inference speed, while its fusion remains robust across thresholds and languages.
- Main Results: 0.413 Pearson correlation and 0.391 overall IoU are the highest reported for SpanCalib-VLM on the English evaluation split.The split contains N = 379 samples.
- Main Results: 0.913 clean-response IoU and 70.7% detection accuracy accompany the English evaluation results.
- Main Results: SpanCalib-VLM combines the generative model’s higher hallucinated-sample IoU with the tagger’s superior Pearson calibration.The reported comparison is 0.182 versus 0.155 for hallucinated-sample IoU and 0.369 versus 0.285 for Pearson calibration.
- Multilingual Evaluation: 0.437 Overall IoU and 0.958 Clean IoU are achieved on Chinese, with +0.169 IoU over base Qwen-3.5-4B.French and Italian Hallucinated IoU gains are +0.182 and +0.180 over base models.
- Threshold Robustness: IoU remains 0.391 across τ ∈[0.25, 0.45], reflecting a bimodal fusion-score distribution with few samples near the decision boundary.
- Inference Speed: 5.25 samples/s is 4.5× faster than standard Qwen inference and 120× faster than chain-of-thought mode.The comparison is reported for SpanCalib-VLM sequence tagging and generative decoding.
- Ablation Study: Removing the MSE calibration loss causes a −0.095 Pearson drop, while Union-Calibrated Fusion beats weighted averaging by +0.032 Pearson and intersection by +0.061.The vision tower adds +0.003 IoU while maintaining identical ensemble calibration.
6 Conclusion
SpanCalib-VLM combines a multimodal discriminative tagger with a generative VLM through Union-Calibrated Fusion. The ensemble achieves strong SHROOM-Visions performance, pairing calibrated probabilities with generative span proposals.
- SpanCalib-VLM combines a multimodal discriminative tagger with a generative VLM via Union-Calibrated Fusion.
- 0.413 Pearson, 0.391 IoU, and 91.3% clean accuracy are achieved on SHROOM-Visions.
7 Limitations
The authors identify false positives, input-length constraints, localization weaknesses, English-centered tuning, and irrecoverable misses when both subsystems fail.
- SpanCalib-VLM sometimes flags elaborate descriptions, idioms, and rare but accurate visual details as hallucinations.
- A 512-token input limit makes hallucinations beyond the truncation boundary undetectable.
- Pearson 0.413 calibration contrasts with hallucinated-span IoU 0.196, indicating difficulty precisely localizing character-level boundaries.
- Hyperparameter selection and ensemble fusion thresholds were optimized primarily on the English validation split despite joint training on EN, FR, IT, and ZH.
- Fusion cannot recover hallucinations missed by both the sequence tagger and the generative VLM.
B Training configuration
The supplied passages identify the training-configuration and cross-lingual dataset tables, including multilingual splits and their evaluation organization.
- Table 5 summarizes the training configuration for the multimodal system.
- Table 6 gives the training configuration for the Qwen3.5-4B generative VLM.
- Table 7 compares base zero-shot models with SpanCalib-VLM across English, French, Italian, and Chinese evaluation sets.
- Table 8 details SHROOM-Visions sample counts for 90% training, 10% validation, and unlabeled test sets across languages.
D Training History
The supplied passage identifies Table 9 as the training history for SpanCalib-VLM’s multimodal model.
- Table 9 presents the training history for SpanCalib-VLM (multimodal).
E Ablation Study Results
The supplied passage identifies Table 10 as an ablation study, without reporting its component-level findings.
- Table 10 reports an ablation study.