Source-linked AI summary
Two-Token Features and Small-Large Ensembles for VLM Hallucination Detection
Eli Schwartz
TL;DR
Character-level VLM hallucination detection requires reliable span detection and categorization across languages. The paper combines a small hidden-state classifier with OCR, synthetic hallucination data, and a large zero-shot judge, reaching mean Cor 0.487 / Cor-lbl 0.387 on the hidden test set. Its scope is bounded by validation noise and differences between the submitted and described systems.
Problem
SHROOM-Visions 2026 requires identifying and categorizing hallucinated character spans in VLM outputs across four languages, motivating systems that combine per-token reliability with cross-language generalization.
Method
The system fine-tunes a small VLM with a two-token hidden-state classifier, OCR grounding, and large-model-generated synthetic data, then ensembles it with a large zero-shot VLM judge.
Results
0.487 mean Cor / 0.387 Cor-lbl is achieved on the hidden test set, with the entry placing 6th/28 in EN, 6th/21 in FR, 8th/21 in IT, and 7th/22 in ZH.
Takeaways & Limitations
Ensembling the small detector with the large judge lifts mean Cor by +0.132 over the judge alone and +0.033 over the small detector alone.
Takeaways & Limitations
Single-language comparisons are descriptive because each configuration was trained once and synthetic-data swings can reach 0.023 Cor; the submitted system also differs from the described system.
Abstract
from arXiv · showhide
We present our system for the SHROOM-Visions 2026 shared task on character-level VLM hallucination detection. A small ($4$B-parameter) VLM is fine-tuned as a per-token classifier reading a two-token feature from its own hidden states, and is ensembled with a $\sim$400B zero-shot VLM judge at prediction time. Both components see off-the-shelf OCR of any visible in-image text. We use synthetic hallucination data generated by the large model as a source of ensemble diversity, and use validation to select feature layer, training data and OCR grounding. Our official entry reaches mean Cor $0.487$ / Cor-lbl $0.387$ on the hidden test set, placing $6$th/$28$ (EN), $6$th/$21$ (FR), $8$th/$21$ (IT) and $7$th/$22$ (ZH) on the task's primary Cor-lbl metric.
1 Introduction
The task requires character-level detection and categorization of hallucinated spans in VLM outputs. The system combines a small fine-tuned detector with a much larger zero-shot judge to balance per-token reliability and cross-language generalization.
- Task: SHROOM-Visions 2026 evaluates whether each character span in a VLM output is hallucinated relative to an image and prompt.Systems also classify spans as invention, mischaracterization, OCR, miscounting, or other, using Cor and Cor-lbl per-language Spearman correlations.
- Motivation: A large instruction-tuned VLM can identify hallucinated spans zero shot across languages, but its character-level predictions are noisy.
- Motivation: A purpose-built fine-tuned detector is more reliable per token, but training at roughly 400B parameters on 15k examples is not compute-efficient.
- Approach: The proposed system fine-tunes a VLM two orders of magnitude smaller than the zero-shot judge and ensembles both models at prediction time.
- Approach: Validation experiments select a two-token hidden-state feature, synthetic training data, and OCR grounding as the system’s main ingredients.
2 Task and Models
The task scores character-level hallucination probabilities and categories across four languages, while the system pairs a small fine-tuned VLM with a large zero-shot judge and OCR input.
- Task: Each example contains an image, prompt, and generated text, with character-level hallucination probabilities and six-way categories scored against annotator gold.The categories are five hallucination types plus none.
- Models: The small model is a 4B-parameter Qwen3.5-VL checkpoint fine-tuned with LoRA adapters while its visual encoder and multimodal connector remain frozen.
- Models: The large model is Qwen3.5-VL-397B, used zero shot with a JSON-schema prompt requesting hallucinated span strings and categories.
- Classifier: The classifier head concatenates hidden-state features for the current and previous tokens before prediction.
- OCR: Both models receive off-the-shelf OCR text extracted from visible image content as additional prompt input.
3 Small-Model Detector
The small detector uses current and previous token hidden states in a shallow classifier, with validation favoring a mid-network layer and OCR improving performance on relevant examples.
- Two-token feature: Each token feature concatenates current and previous hidden states from the same language-tower layer before a two-layer MLP predicts six classes.The six classes are five hallucination categories plus none.
- Two-token feature: The detector reviews teacher-forced text generated by another VLM, so its hidden-state features come from the reviewing model rather than the generator.
- Layer choice: +0.039 mean Cor separates the embedding output from a broad plateau centered in the middle of the tower, leading to the choice L=16.
- Layer choice: A follow-up run tying both token slots to L=16 adds +0.017, while replacing the previous-token slot costs at most 0.008 Cor and permuting it costs 0.049.
- Training and inference: The model is trained jointly across four languages with six-way cross-entropy, then projects token distributions to character probabilities and labels.
4 Synthetic Data for Ensemble Diversity
Synthetic hallucination data are generated by the large model to make the small detector’s errors more complementary to the large judge. The resulting diversity is measurable as increased token-level disagreement.
- Motivation: The shared-task data alone do not ensure that the small detector produces signal complementary to the large judge.The paper therefore uses synthetic data to shift the small model’s error distribution away from the judge’s.
- Synthetic-data pipeline: The large model generates clean descriptions, targeted hallucinated rewrites, and changed-span categories from image-caption inputs.Character diffs identify changed spans, with approximate matching used when diff counts disagree with declared changes.
- Diversity rationale: Using the same large model for generation and judging is intended to preserve diversity because the two roles use different prompts and inputs.The generator produces perturbed captions, whereas the judge annotates fixed VLM outputs against images.
- Measured diversity: +1.2 percentage points raises token-level hallucination-flag disagreement from 28.9% to 30.1% when synthetic data are mixed with real data.Argmax-over-six-classes and mean L1 distribution distance remain essentially unchanged, indicating the change concerns which tokens are flagged.
5 Results
Validation favored a small detector with synthetic data and OCR, while hidden-test performance benefited from combining it with the large judge. The official entry reached mean Cor 0.487 and Cor-lbl 0.387, with language-specific rankings reported by Cor-lbl.
- Ensemble: The ensemble combines small- and large-model per-character probabilities using a fixed α=0.8 convex combination.The predicted category comes from the model contributing the majority of probability mass at each position.
- Validation ablations: Synthetic data and OCR grounding each add approximately +0.010 mean Cor-lbl, while the best small detector reaches 0.467 mean Cor and 0.398 Cor-lbl on validation.These ablations use the small detector at layer L=16.
- Validation ablations: 28.9% to 30.1%: synthetic training raises token-level disagreement between the small model and large judge by 1.2 percentage points.On validation, the ensemble remains essentially flat because the OCR-grounded large judge reaches the empty-labels floor.
- Hidden-test results: 0.355 mean Cor: the large judge scores well above its validation floor on the hidden test set, making the two constituents useful ensemble partners.The validation–test gap is attributed most likely to differences in annotation quality.
- Hidden-test results: +0.031 mean Cor over the small model and +0.130 over the judge: fusion wins on three of four languages.The large judge is stronger on EN and weaker on the other three languages by 0.12 to 0.14.
- Submission results: 0.487 mean Cor / 0.387 Cor-lbl: the official entry places 6th, 6th, 8th and 7th among 28, 21, 21 and 22 teams.Cor rank equals or beats Cor-lbl rank in every language, indicating stronger span detection than categorisation.
6 Related Work
The task extends character-level hallucination evaluation to visual failure modes and contrasts with common whole-response or object-level VLM benchmarks. It also builds on prior work probing internal states for factuality.
- Task context: SHROOM-Visions 2026 adds OCR errors and miscounting to text-only hallucination scoring while retaining per-character spans.The task uses the same per-character metrics as the previous SHROOM edition.
- Task context: Character-granularity evaluation differs from the whole-response or object-level judgements common in VLM-hallucination benchmarking.The passage frames this granularity as unusual within that benchmarking landscape.
- Prior methods: The paper relates its hidden-state probing to established factuality methods such as SAPLMA.The cited prior work classifies truthfulness from activations.
7 Conclusion
The paper concludes that a small VLM with a two-token mid-network feature can provide competitive per-character hallucination detection. Layer selection, OCR grounding and synthetic data improve validation performance, while ensembling with a large judge improves hidden-test performance.
- Conclusion: A small fine-tuned VLM with a two-token classifier head over a mid-network hidden state provides competitive per-character hallucination detection.The system targets the SHROOM-Visions 2026 shared task.
- Conclusion: +0.039 val Cor: selecting a mid-network layer is the largest reported lever, while OCR grounding and synthetic training each add a further +0.010 mean Cor.These gains are reported for the validation experiments.
- Conclusion: +0.132 mean Cor over the judge alone and +0.033 over the small detector alone: the ensemble places the system in the top third of every language.This result is reported for the hidden-test evaluation.
Limitations
The study’s limitations include compute-intensive synthetic-data generation, single-seed evaluation, restricted model and OCR coverage, and differences between the described system and the official entry.
- Evaluation: All configurations were trained once with seed 42, while per-language synthetic-training swings reached 0.023 Cor.Single-language comparisons are consequently described as descriptive.
- Scope: The study uses one model family on both ensemble sides and one OCR extractor, limiting evidence about cross-family or cross-extractor robustness.The small and large models are Qwen3.5-VL 4B and 397B, and the OCR extractor is PaddleOCR.
- Reproducibility: The submitted system differs from the paper’s described runnable system in layer placement, training-data usage, and how the official-entry result is selected.The entry used cross-layer placement, folded the validation holdout into training, and reported per-language Cor-lbl bests across configurations.
- Validation transfer: The diversity account predicts test-time ensemble gains but not validation behavior because the large judge reaches the empty-labels floor on validation.On the hidden test set, the judge is 0.07 Cor above that floor and the ensemble gains.
- Compute: Synthetic-data generation required approximately 363 H100-GPU-hours, about 90× small-model training and roughly 85% of official-entry compute.The authors therefore make no claim that the full pipeline is compute-efficient relative to approaches that skip synthetic generation.
C Data Statistics
The synthetic corpus is generated from CC6M captions for each language, with substantial filtering and fuzzy alignment; validation is a deterministic 10% hash-based holdout.
- Splits: Validation is a deterministic hash-based 10% holdout of the released training data.The split uses IDs below a fixed SHA-256-derived hash cutoff.
- Synthetic data: Approximately 68% of synthetic rows are usable after generation across the four languages.The synthetic pipeline runs on 24k CC6M input captions per language.
- Synthetic data: The fuzzy-alignment path fires on approximately 41% of rows across all four languages.
- Experimental setup: The layer-placement comparison fixes the previous-token slot and sweeps the current-token layer, then reruns tied and cross-layer placements at L=16.Architecture, data, schedule, and seed are held fixed in the placement rerun.
E Extended Limitations
The paper reports several boundaries on interpreting its validation gains, test improvements, and official submission results. These include confounded comparisons, composite configurations, distribution mismatch, reproducibility discrepancies, and substantial compute requirements.
- Placement effects: +0.017 mean Cor is a head-only same-layer advantage and an upper bound on the full system’s placement contribution.With LoRA and more data, the advantage shrinks to 0.000 mean Cor at the top rung.
- Ensemble interpretation: +0.031 mean Cor is not purely a diversity-driven test gain because synthetic data also improves the small model’s Cor-lbl from 0.345 to 0.356.The large judge is at the empty-labels floor on validation but 0.07 above it on test, creating a val–test asymmetry.
- Evaluation confounds: The test OCR delta combines OCR grounding with folding the validation holdout into training, so it does not isolate OCR’s effect.The clean OCR comparisons are the validation A/B experiments.
- Submission scope: The official-entry row is a per-language Cor-lbl composite of different uploaded configurations rather than one runnable system.Its Cor is not the maximum because selection used Cor-lbl; the honest ensemble Cor gain over the best uploaded small model is +0.028 rather than +0.031.
- Reproducibility: The EN final-ensemble figures differ between upload-time values and the final leaderboard, and the authors cannot account for the discrepancy.The reported pairs are 0.460/0.347 at upload time and 0.466/0.357 in Table 4.
- Data limitations: Synthetic data is category-level distribution-mismatched, with mischaracterization under-generated and other over-represented relative to real spans.This mismatch is identified as the most likely explanation for categorisation being the weakest metric.
- Compute limitations: The system makes no end-to-end efficiency claim because test-time inference requires the large judge and synthetic-data generation uses roughly 90× its training compute.The ensemble weight was fixed rather than learned.
F Negative Results
Several attempted diversity and alternative-signal strategies did not provide useful gains. Image multi-scale augmentation and LoRA dropout produced little disagreement, while verdict-position probabilities were worse calibrated than the adopted classifier head.
- Image multi-scale TTA: Image multi-scale averaging changed the EN-val argmax on only 0.5% of tokens and was not a useful diversity source.The small model was rerun at long-side resizes of 384, 448, and 512.
- MC-dropout on LoRA adapters: LoRA MC-dropout produced 0.17% argmax disagreement at dropout 0.1 and only 0.76% at dropout 0.5, far below inter-model diversity.The disagreement increased monotonically with the inference dropout rate.
- Verdict-position probabilities: Verdict-position log-probabilities were worse calibrated to gold on EN validation than the adopted classifier-head architecture.The experiment used per-sentence verdict-token probabilities from a JSON-completion setup.