Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models

• LVLM hallucinations require outputs to remain faithful to both linguistic knowledge and the specific visual content of an accompanying image.Multimodal failures include object fabrication, entity misdescription, on-image text misreading, and miscounting visible items.Source: §1 · p. 1
• Existing hallucination-detection benchmarks rely on outputs from small, fixed model sets, risking overfitting to system-specific idiosyncrasies and declining diagnostic value.Source: §1 · p. 1
• SHEEP provides 20,000 multilingual samples with human-written hallucinations, LVLM outputs, span-level annotations, and a five-way taxonomy across Chinese, English, French, and Italian.The dataset combines outputs from five LVLMs with 1,600 human-written items.Source: §1 · p. 1
• Human-written samples achieve higher inter-annotator agreement, controlled hallucination-type distributions, and detector rankings that correlate more strongly with LVLM output rankings than cross-model comparisons.Source: §1 · p. 2
• SHROOM-Visions evaluates detection on both LVLM-generated and human-written data, testing whether performance generalizes beyond specific generators.Source: §1 · p. 2

• LVLM hallucination taxonomies evolved from object–attribute–relation categories toward event fabrications, miscounting, text misreading, and identity incongruity.Source: §2 · p. 2
• M-HalDetect and HalLoc provide fine-grained and token-level hallucination localization, respectively.Source: §2 · p. 2
• Many resources use English outputs from a small number of LVLMs, inheriting source-model idiosyncrasies and losing diagnostic value as models are superseded.Source: §2 · p. 2
• SHROOM-Visions uses SHEEP to extend SHROOM shared tasks into multimodal detection while treating human-written hallucinations as a first-class evaluation source.Source: §2 · p. 2

3.1 Task definition

• Participants identify response character spans corresponding to hallucinations and assign each detected span a hallucination category.Source: §3.1 · pp. 2–3
• The five categories are Invention, Mischaracterization, OCR Problem, Miscounting, and Other.Participants also estimate, for every response character, its probability of belonging to a hallucinated span.Source: §3.1 · pp. 2–3

3.2 Data

• The SHEEP data combine random LVLM-output sampling, LLM-judge-assisted label-balanced preselection, and translated, post-edited human-authored examples.Source: §3.2 · p. 3
• Human-written items appear exclusively in the test split, while MAP and Random examples contribute to both training and test splits.Source: §3.2 · p. 3

Task timeline and organization

• The training phase provided approximately 15.2K annotated samples, and evaluation used a hidden test set of 4.8K samples, with 1.2K per language.Source: §3.3 · p. 4

• Systems output a hallucination probability for every response character, compared with multi-annotator gold probabilities separately for Chinese, English, French, and Italian.Source: §4 · p. 4

Unlabeled Correlation (Corr)

• Unlabeled Corr measures Spearman correlation between gold and predicted character-level hallucination probabilities, using exact-match scoring when either vector is constant.Source: §4 · p. 4

Labeled Correlation (Corrlbl)

• Labeled Corrlbl computes character-level correlation separately for each hallucination label present in gold or predicted annotations.Source: §4 · p. 4

Intersection-over-Union (IoU)

• IoU binarizes gold and predicted probabilities into covered character sets to evaluate span localization independently of confidence and category.Source: §4 · p. 4

• The metrics are complementary: IoU assesses localization, Corr assesses calibrated confidence profiles, and Corrlbl assesses hallucination-type categorization.Source: §4 · p. 4

Ranking

• Rankings are reported independently by language and metric across Random, MAP, and Human-written subsets to test generalization from LVLM-generated to model-independent data.Source: §4 · p. 4

Participation

• 27 teams submitted systems, producing 623 submissions across the four target languages.Source: §5 · p. 4

Recurrent participation

• 105 unique teams have participated across the four SHROOM shared-task editions.Source: §5.1 · p. 5

System approaches

• Participating systems ranged from LoRA fine-tuning of vision-language models to token- and span-level classifiers using multimodal or text-only encoders.Source: §5.2 · p. 5

Selected systems

• vroom-vroom fine-tuned vision-language models to generate inline hallucination-span annotations with category and confidence labels.Source: §5.2 · p. 5
• TÜRKSAT combined multimodal-LLM judge ensembles with LoRA-fine-tuned vision-language token classifiers and explored curriculum learning and ensemble strategies.Source: §5.2 · p. 5
• Dynamos, Champ, and Smurfcat explored encoder-based detection, probing, thresholding, calibration, adaptation, and augmentation strategies.Source: §5.2 · pp. 5–6

Overall performance

• Most mean scores fall below 0.4, and no system exceeds 0.6 in any language or metric.Chinese is a partial exception, with average Corr performance slightly above 0.4; English is most challenging.Source: §6 · p. 6

Leaderboard stability

• Top-five systems exhibit rank intervals of up to 10 positions in Chinese, 14 in Italian and French, and 15 in English.The English leader TÜRKSAT has mean rank 2.9 but a 95% interval spanning positions 1 to 10.Source: §6.1 · p. 6
• Chinese has the narrowest top-team rank distributions, whereas English combines the widest spreads with the lowest scores.The authors caution that small leaderboard differences should not be treated as definitive evidence of superiority.Source: §6.1 · p. 6

Strategy-level analysis

• Human-written and silver-label examples yield highly similar rankings, with Spearman correlations between .861 and .973.Randomly sampled examples generally produce lower correlations and greater changes in system ordering.Source: §6.2 · p. 7
• Evaluation strategy influences absolute performance estimates more consistently than coarse system ordering, especially for randomly sampled Chinese Corrlbl data.Changes in absolute scores and system ordering need not coincide across languages and metrics.Source: §6.2 · p. 7

Metric relationships

• Top-performing systems cluster in the upper-right of IoU–correlation plots, indicating strong hallucination localization and character-level ranking.The plots show strong linear trends between IoU and both correlation metrics, while the metrics capture complementary aspects of performance.Source: §6.3 · pp. 7–8

Cross-lingual performance

• TÜRKSAT leads English with ρ=0.55 and IoU=0.48, while vroom-vroom leads French, Italian, and Chinese.vroom-vroom reports FR ρ=0.58 and IoU=0.52, IT ρ=0.56 and IoU=0.48, and ZH ρ=0.61 and IoU=0.53.Source: §6.3 · p. 8

• 27 teams submitted more than 620 systems for span-level, multilingual, multi-class hallucination detection across English, Chinese, Italian, and French.Source: §7 · p. 8
• 0.58 character-level correlation, 0.46 label-conditioned correlation, and 0.51 span-level IoU were achieved by the strongest systems on average.Source: §7 · p. 8
• Leaderboard positions are uncertain because leading systems can occupy substantially different ranks across resampled test sets.Source: §7 · p. 8
• Absolute scores and relative positions depend on benchmark construction, with random sampling producing greater volatility than human-written and silver examples.Source: §7 · p. 8
• Progress should therefore be assessed through both detection accuracy and the reliability with which evaluations distinguish systems.Source: §7 · p. 8

• The benchmark covers restricted languages, domains, and LVLM outputs, which may not represent hallucinations in broader real-world settings.Source: Building a house of cards: Limitations · p. 9
• The static benchmark creates potential leakage and overfitting risks, although test annotations were kept private.Source: Building a house of cards: Limitations · p. 9
• 15.5–25.5% of test instances have empty annotations, whereas systems produce empty predictions for 30.7–46.0%.Source: Building a house of cards: Limitations · p. 9
• Span-overlap and probability-correlation metrics do not explicitly capture systems’ abstention behavior on instances without marked hallucinations.Source: Building a house of cards: Limitations · p. 9
• For empty instances, exact match is used when no span is available, without directly assessing whether abstention was appropriate.Source: Building a house of cards: Limitations · p. 9

• The final leaderboard ranks teams by Corrlbl performance and reports best-system scores after bootstrapping 25,000 samples per metric.Source: B Off with their heads! Test phase… · p. 12
• Figure 8 summarizes the best systems’ performance separately for English, French, Italian, and Chinese.Source: Figure 8 · p. 12
• 95% confidence intervals accompany each score, mean rank, and rank interval in the leaderboard.Source: B Off with their heads! Test phase… · p. 12
• P(> next) values near 0.35–0.47 among adjacent top teams indicate only modest probabilities of consistently outperforming the next-ranked system.Source: B Off with their heads! Test phase… · p. 13
• Observed ordering differences should not be treated as definitive superiority because pairwise inversion probabilities often indicate non-robust rankings.Source: B Off with their heads! Test phase… · p. 13

• Tables 8–19 report complete scores across global and strategy-specific rankings for human-written, randomly sampled, and silver-label partitions.Source: C Painting the white roses red:… · p. 13
• The tables separate results by language and evaluation metric while ordering teams by their global-ranking score.Source: C Painting the white roses red:… · p. 13
• Annotation strategy can shift most systems’ scores without substantially changing the leaderboard.Source: C Painting the white roses red:… · p. 13
• Selective strategy effects can produce rank inversions for individual systems.Source: C Painting the white roses red:… · p. 13
• The tables, rank correlations, and rank-flow plots provide evidence for strategy-dependent evaluation effects.Source: C Painting the white roses red:… · p. 13

Span-level metrics

• Precision, recall, and F1 provide complementary span-level measures of extraction quality and the trade-off between false positives and false negatives.Precision measures the proportion of predicted spans matching valid annotations, while recall measures recovery of annotated spans.Source: D Curiouser and curiouser! Additional… · p. 13

Leaderboard analysis

• Leaderboard tables report English, Chinese, French, and Italian Corrlbl rankings with mean rank, 95% rank intervals, and bootstrap probabilities of adjacent-team separation.A probability of at least 0.95 denotes robust separation, while a value of at most 0.20 indicates a possible inversion.Source: Table 4 · p. 14, Table 5 · p. 14, Table 6 · p. 15

Partition comparisons

• Rank-flow figures track each system’s rank from the General partition to Human, Silver, and Random subsets for English, Chinese, French, and Italian Corrlbl results.Each node represents a system’s rank within one partition.Source: Figure 9 · p. 16, Figure 10 · p. 16, Figure 12 · p. 16

Additional analyses

• Bootstrap rank distributions summarize 25,000 samples for Chinese, Italian, and French Corrlbl submissions, while other tables cover abstention, hallucination classes, and the submission interface.The bootstrap distributions include means and 95% confidence intervals.Source: Figure 13 · p. 17, Figure 14 · p. 18, Table 20 · p. 22

Strategy-specific scores

• Strategy-specific tables compare Global, Human-written, Random, and Silver-label performance for Corr, Corrlbl, and IoU across the four language leaderboards.Teams are ordered by Global scores, and boldface marks the highest score in each reported column.Source: Table 8 · p. 19, Table 9 · p. 19, Table 10 · p. 20