Source-linked AI summary
Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models
Raúl Vázquez, Aman Sinha, Chuyuan Li, Claudio Savelli, Eduardo Calò, Emilio Raimond, Stella Frank, Hengyu Luo, Flavio Giobergia, Vincent Segonne, Lorenzo Vaiani, Jörg Tiedemann, Timothee Mickus
TL;DR
LVLM hallucination detection must account for errors unsupported by specific visual inputs while avoiding benchmarks tied to a small, changing set of generators. SHROOM-Visions uses the multilingual SHEEP dataset to evaluate character-level span detection and five-class classification across four languages. The strongest systems reach average scores of 0.58 in character-level correlation, 0.46 in label-conditioned correlation, and 0.51 in span-level IoU, while rankings vary across resampled tests and benchmark partitions.
Problem
Existing hallucination-detection benchmarks depend on small, fixed model sets, making model-specific overfitting and obsolescence concerns for evaluation.
Method
SHROOM-Visions evaluates character-level hallucination probabilities and five-class span labels on SHEEP data spanning four languages and both LVLM-generated and human-written examples.
Results
0.58 character-level correlation, 0.46 label-conditioned correlation, and 0.51 span-level IoU were achieved by the strongest systems on average across four languages.
Takeaways & Limitations
Reliable fine-grained hallucination detection remains difficult, and evaluation should combine model-independent data with multilingual annotations and explicit uncertainty reporting.
Takeaways & Limitations
The benchmark has restricted coverage of languages, domains, and LVLM outputs, and its static design carries leakage and overfitting risks.
Abstract
from arXiv · showhide
In 2026, we held the fourth iteration of the SHROOM Shared Task series: SHROOM-Visions (\textbf{S}hared-task on \textbf{H}allucinations and \textbf{R}elated \textbf{O}bservable \textbf{O}vergeneration \textbf{M}istakes in \textbf{Vision} language model\textbf{s}), which is hosted at the UncertaiNLP Workshop co-located with EMNLP 2026. Following the success of the 2024 and 2025 tasks, this time we aim to tackle hallucinations through a model-agnostic detection task focused on large vision-language models. Building on the recently introduced SHEEP dataset, designed for long-term evaluation across model generations, the task invites participants to detect and classify fine-grained hallucination spans in image-conditioned text generation (VQA, image captioning, etc.). The evaluation uses a five-class taxonomy of hallucinations spanning four languages: Chinese, English, French, and Italian. The shared task generated strong interest in the NLP community worldwide, with 27 teams contributing 600+ system submissions. The best systems achieve average scores of 0.58 in character-level correlation, 0.46 in label-conditioned correlation, and 0.51 in intersection-over-union (IoU) across four languages, outperforming the baselines by 30-40 points.
1 “Begin at the beginning”: Introduction
SHROOM-Visions targets reliable, model-agnostic detection of fine-grained hallucinations in multilingual LVLM outputs. It builds on human-authored, labeled errors to reduce dependence on rapidly obsolete model-derived benchmarks.
- LVLM hallucinations require outputs to remain faithful to both linguistic knowledge and the specific visual content of an accompanying image.Multimodal failures include object fabrication, entity misdescription, on-image text misreading, and miscounting visible items.
- Existing hallucination-detection benchmarks rely on outputs from small, fixed model sets, risking overfitting to system-specific idiosyncrasies and declining diagnostic value.
- SHEEP provides 20,000 multilingual samples with human-written hallucinations, LVLM outputs, span-level annotations, and a five-way taxonomy across Chinese, English, French, and Italian.The dataset combines outputs from five LVLMs with 1,600 human-written items.
- Human-written samples achieve higher inter-annotator agreement, controlled hallucination-type distributions, and detector rankings that correlate more strongly with LVLM output rankings than cross-model comparisons.
- SHROOM-Visions evaluates detection on both LVLM-generated and human-written data, testing whether performance generalizes beyond specific generators.
2 Down the rabbit hole: Related works
Prior LVLM hallucination work developed increasingly fine-grained taxonomies and localization resources, but benchmarks often remain narrow in language and model coverage. SHROOM-Visions adopts SHEEP to make multimodal evaluation more durable and model-independent.
- LVLM hallucination taxonomies evolved from object–attribute–relation categories toward event fabrications, miscounting, text misreading, and identity incongruity.
- M-HalDetect and HalLoc provide fine-grained and token-level hallucination localization, respectively.
- Many resources use English outputs from a small number of LVLMs, inheriting source-model idiosyncrasies and losing diagnostic value as models are superseded.
- SHROOM-Visions uses SHEEP to extend SHROOM shared tasks into multimodal detection while treating human-written hallucinations as a first-class evaluation source.
3 “Oh dear! Oh dear! I shall be late!”: Task timeline and organization
The task asks participants to locate and classify hallucinated character spans in image-conditioned responses across four languages. Its data combine random LVLM outputs, model-assisted selections, and human-authored multilingual examples, with hidden multilingual evaluation.
- 3.1 Task definition: Participants identify response character spans corresponding to hallucinations and assign each detected span a hallucination category.
- 3.1 Task definition: The five categories are Invention, Mischaracterization, OCR Problem, Miscounting, and Other.Participants also estimate, for every response character, its probability of belonging to a hallucinated span.
- 3.2 Data: The SHEEP data combine random LVLM-output sampling, LLM-judge-assisted label-balanced preselection, and translated, post-edited human-authored examples.
- 3.2 Data: Human-written items appear exclusively in the test split, while MAP and Random examples contribute to both training and test splits.
- Task timeline and organization: The training phase provided approximately 15.2K annotated samples, and evaluation used a hidden test set of 4.8K samples, with 1.2K per language.
4 “What is the largest number that you know?”: Evaluation metrics
Systems are evaluated against multi-annotator character-level gold labels using confidence correlation, label-conditioned correlation, and span IoU. Separate language and sampling-subset rankings test both detection quality and generalization to human-written data.
- Systems output a hallucination probability for every response character, compared with multi-annotator gold probabilities separately for Chinese, English, French, and Italian.
- Unlabeled Correlation (Corr): Unlabeled Corr measures Spearman correlation between gold and predicted character-level hallucination probabilities, using exact-match scoring when either vector is constant.
- Labeled Correlation (Corrlbl): Labeled Corrlbl computes character-level correlation separately for each hallucination label present in gold or predicted annotations.
- Intersection-over-Union (IoU): IoU binarizes gold and predicted probabilities into covered character sets to evaluate span localization independently of confidence and category.
- The metrics are complementary: IoU assesses localization, Corr assesses calibrated confidence profiles, and Corrlbl assesses hallucination-type categorization.
- Ranking: Rankings are reported independently by language and metric across Random, MAP, and Human-written subsets to test generalization from LVLM-generated to model-independent data.
5 The caucus race: Participating systems
SHROOM-Visions attracted broad participation and diverse detection approaches, spanning LoRA-fine-tuned vision-language models, token- and span-level classifiers, ensembles, and reasoning-based methods.
- Participation: 27 teams submitted systems, producing 623 submissions across the four target languages.
- Recurrent participation: 105 unique teams have participated across the four SHROOM shared-task editions.
- System approaches: Participating systems ranged from LoRA fine-tuning of vision-language models to token- and span-level classifiers using multimodal or text-only encoders.
- Selected systems: vroom-vroom fine-tuned vision-language models to generate inline hallucination-span annotations with category and confidence labels.
- Selected systems: TÜRKSAT combined multimodal-LLM judge ensembles with LoRA-fine-tuned vision-language token classifiers and explored curriculum learning and ensemble strategies.
- Selected systems: Dynamos, Champ, and Smurfcat explored encoder-based detection, probing, thresholding, calibration, adaptation, and augmentation strategies.
6 Through the looking glass: Results and analysis
Results show modest, language-robust performance alongside substantial uncertainty in fine-grained rankings. Evaluation strategy affects absolute scores and can substantially shift individual system positions, while strong systems tend to perform well across localization and correlation metrics.
- Overall performance: Most mean scores fall below 0.4, and no system exceeds 0.6 in any language or metric.Chinese is a partial exception, with average Corr performance slightly above 0.4; English is most challenging.
- Leaderboard stability: Top-five systems exhibit rank intervals of up to 10 positions in Chinese, 14 in Italian and French, and 15 in English.The English leader TÜRKSAT has mean rank 2.9 but a 95% interval spanning positions 1 to 10.
- Leaderboard stability: Chinese has the narrowest top-team rank distributions, whereas English combines the widest spreads with the lowest scores.The authors caution that small leaderboard differences should not be treated as definitive evidence of superiority.
- Strategy-level analysis: Human-written and silver-label examples yield highly similar rankings, with Spearman correlations between .861 and .973.Randomly sampled examples generally produce lower correlations and greater changes in system ordering.
- Strategy-level analysis: Evaluation strategy influences absolute performance estimates more consistently than coarse system ordering, especially for randomly sampled Chinese Corrlbl data.Changes in absolute scores and system ordering need not coincide across languages and metrics.
- Metric relationships: Top-performing systems cluster in the upper-right of IoU–correlation plots, indicating strong hallucination localization and character-level ranking.The plots show strong linear trends between IoU and both correlation metrics, while the metrics capture complementary aspects of performance.
- Cross-lingual performance: TÜRKSAT leads English with ρ=0.55 and IoU=0.48, while vroom-vroom leads French, Italian, and Chinese.vroom-vroom reports FR ρ=0.58 and IoU=0.52, IT ρ=0.56 and IoU=0.48, and ZH ρ=0.61 and IoU=0.53.
7 Out of the rabbit hole: Conclusion
SHROOM-Visions advances multilingual, span-level hallucination detection for LVLMs while showing that both detection accuracy and evaluation reliability remain central challenges.
- 27 teams submitted more than 620 systems for span-level, multilingual, multi-class hallucination detection across English, Chinese, Italian, and French.
- 0.58 character-level correlation, 0.46 label-conditioned correlation, and 0.51 span-level IoU were achieved by the strongest systems on average.
- Leaderboard positions are uncertain because leading systems can occupy substantially different ranks across resampled test sets.
- Absolute scores and relative positions depend on benchmark construction, with random sampling producing greater volatility than human-written and silver examples.
- Progress should therefore be assessed through both detection accuracy and the reliability with which evaluations distinguish systems.
Building a house of cards: Limitations
The task inherits dataset-coverage and benchmark-design limitations, while empty-instance evaluation leaves abstention quality insufficiently measured.
- The benchmark covers restricted languages, domains, and LVLM outputs, which may not represent hallucinations in broader real-world settings.
- The static benchmark creates potential leakage and overfitting risks, although test annotations were kept private.
- 15.5–25.5% of test instances have empty annotations, whereas systems produce empty predictions for 30.7–46.0%.
- Span-overlap and probability-correlation metrics do not explicitly capture systems’ abstention behavior on instances without marked hallucinations.
- For empty instances, exact match is used when no span is available, without directly assessing whether abstention was appropriate.
B Off with their heads! Test phase leaderboard
The test-phase leaderboard combines bootstrapped score and rank uncertainty reporting, emphasizing that small ranking differences often lack robust support.
- The final leaderboard ranks teams by Corrlbl performance and reports best-system scores after bootstrapping 25,000 samples per metric.
- Figure 8 summarizes the best systems’ performance separately for English, French, Italian, and Chinese.
- 95% confidence intervals accompany each score, mean rank, and rank interval in the leaderboard.
- P(> next) values near 0.35–0.47 among adjacent top teams indicate only modest probabilities of consistently outperforming the next-ranked system.
- Observed ordering differences should not be treated as definitive superiority because pairwise inversion probabilities often indicate non-robust rankings.
C Painting the white roses red: Strategy-specific performance scores
The strategy-specific tables expose how annotation strategy changes absolute performance estimates and can also alter individual system ordering.
- Tables 8–19 report complete scores across global and strategy-specific rankings for human-written, randomly sampled, and silver-label partitions.
- The tables separate results by language and evaluation metric while ordering teams by their global-ranking score.
- Annotation strategy can shift most systems’ scores without substantially changing the leaderboard.
- Selective strategy effects can produce rank inversions for individual systems.
- The tables, rank correlations, and rank-flow plots provide evidence for strategy-dependent evaluation effects.
D Curiouser and curiouser! Additional metrics
Additional analyses characterize span extraction quality, leaderboard rankings, partition-specific strategy scores, abstention, and hallucination-class performance across languages. Rank-flow and bootstrap views further examine how rankings vary across data partitions and languages.
- Span-level metrics: Precision, recall, and F1 provide complementary span-level measures of extraction quality and the trade-off between false positives and false negatives.Precision measures the proportion of predicted spans matching valid annotations, while recall measures recovery of annotated spans.
- Leaderboard analysis: Leaderboard tables report English, Chinese, French, and Italian Corrlbl rankings with mean rank, 95% rank intervals, and bootstrap probabilities of adjacent-team separation.A probability of at least 0.95 denotes robust separation, while a value of at most 0.20 indicates a possible inversion.
- Partition comparisons: Rank-flow figures track each system’s rank from the General partition to Human, Silver, and Random subsets for English, Chinese, French, and Italian Corrlbl results.Each node represents a system’s rank within one partition.
- Additional analyses: Bootstrap rank distributions summarize 25,000 samples for Chinese, Italian, and French Corrlbl submissions, while other tables cover abstention, hallucination classes, and the submission interface.The bootstrap distributions include means and 95% confidence intervals.
- Strategy-specific scores: Strategy-specific tables compare Global, Human-written, Random, and Silver-label performance for Corr, Corrlbl, and IoU across the four language leaderboards.Teams are ordered by Global scores, and boldface marks the highest score in each reported column.