PaperlayerView exact revision on arXiv

Source-linked AI summary

alkhder at ImageEval 2026 Shared Tasks: Constrained Selection and Rotation-Averaged Logit Scoring for Multimodal Hallucination Detection

Maher Pasa, Maria Hanini, Amro Najjar, Hasan Alkhder

arXiv:2610.22515v1cs.CL

TL;DR

Vision–language models remain prone to hallucination, including culturally grounded cases where learned associations may diverge from visual evidence. A pre-trained 7B vision–language model uses next-token log-probabilities to select the sole supported statement, with Latin-square answer-slot rotation to reduce position bias. CI scores of 0.0510 on English and 0.0960 on MSA ranked the system 5th and 4th, respectively.

  • Problem

    Vision–language models remain prone to hallucination, including culturally grounded cases where learned associations may diverge from visual evidence.

    Source: §1 · p. 1

  • Method

    A pre-trained 7B vision–language model uses next-token log-probabilities to select the sole supported statement, with Latin-square answer-slot rotation to reduce position bias.

    Source: §1 · p. 1, §6 · p. 5

  • Results

    CI scores of 0.0510 on English and 0.0960 on MSA ranked the system 5th and 4th, respectively.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The submission’s contribution is an inference and prompting strategy for the constrained one-grounded-statement task.

    Source: §6 · p. 5

    The constrained formulation relies on exactly one statement being grounded and does not directly apply to open-ended hallucination detection.

    Source: Limitations · p. 5

Abstract

from arXiv · show

We describe our submission to Subtask~1b of ImageEval 2026. The task consists of selecting the single visually valid statement from three candidates paired with an image, and is in both English and Modern Standard Arabic (MSA). Our system uses a pre-trained 7B vision--language model in a zero-shot setting, without task-specific training or fine-tuning. We formulate the task as a constrained selection problem and choose the answer using next-token log-probabilities. To reduce answer-position bias, we apply a Latin-square rotation to the candidate statements. Our system achieves CI scores of 0.0510 on the English test set and 0.0960 on the MSA test set, reducing CI relative to the official 3B baselines reported on devtest by 83.7\% and 80.4\%, respectively.

1 Introduction

The submission addresses culturally grounded visual hallucination by selecting the image-supported statement from three candidates. It uses zero-shot log-probability scoring with rotation averaging and reports test-set results in English and MSA.

1 Introduction

  • Culturally grounded hallucination detection must separate culturally plausible answers from interpretations supported by the image.The paper gives the example of a model associating an Arab country with Islamic art when the image does not support that interpretation.Source: §1 · p. 1
  • Subtask 1b asks systems to identify the single visually grounded statement among three statements paired with an image.The submission covers both English and MSA tracks.Source: §1 · p. 1
  • A pre-trained 7B vision–language model selects the visually supported statement using next-token log-probabilities, with a Latin-square rotation to reduce answer-position bias.The system uses zero-shot inference without task-specific training or fine-tuning.Source: §1 · p. 1
  • CI scores of 0.0510 on English and 0.0960 on MSA ranked the system 5th and 4th, improving on released 3B baselines by 6.1× and 5.1×, respectively.The English and MSA scores are on blind test sets.Source: §1 · p. 1

2 Background

The task evaluates whether systems can identify visual support amid culturally plausible alternatives in English and MSA. The paper places this task in a progression from captioning hallucinations to image-content probes and multilingual evaluation.

2.1 Task Description and data

  • Subtask 1b pairs an image with three statements, only one visually valid; the same images and answers appear in English and translated MSA statements.Each track is processed independently, and the paper uses the dev split for validation and ablation and test for submission.Source: §2.1 · pp. 1–2

2.2 Related Work

  • Hallucination research moved from image captioning to yes/no image-content probes and has increasingly examined evaluation beyond English and Western visual contexts.ImageEval draws items from OASIS and uses CFHR; the paper combines fixed-answer likelihood scoring with permutation ensembling.Source: §2.2 · p. 2

3 System Overview

The system converts image–statement judgments into a single winner among three candidates. It reads label likelihoods, scores candidates pointwise and jointly, averages across answer-slot rotations, and combines normalized scores for deterministic decoding.

3 System Overview

  • For an image and three statements, the system produces one winning statement index and emits true for that statement and false for the others.Source: §3 · p. 2

3.1 Logit-level answer reading

  • The model reads next-token probabilities rather than generating an answer, pooling label surface-form probabilities with logsumexp to retain confidence and avoid parsing generated text.The paper reports this is faster than generating and parsing free-form text, allowing use of a larger model.Source: §3.1 · p. 2

3.2 Scoring

  • Pointwise scoring judges each image–statement pair independently, while multiple-choice scoring compares all three statements under an explicit exactly-one-supported constraint.Multiple-choice scoring uses the pooled log-probability of the corresponding option token.Source: §3.2 · p. 2

3.3 Rotation averaging

  • Three cyclic permutations place each statement in every answer slot, and averaging its slot scores makes fixed position preferences contribute equally across statements.Source: §3.3 · p. 2

3.4 Fusion and decoding

  • The submission combines normalized score distributions with equal weights and uses deterministic decoding without sampling or temperature, requiring six forward passes per item.Source: §3.4 · p. 3

4 Experimental Setup

Experiments use an off-the-shelf 7B model and English development data for validation and weight selection, with test data for official runs. The setup reports implementation conditions, split use, and evaluation metrics.

4.1 Model and implementation

  • Qwen2.5-VL-7B-Instruct runs without fine-tuning in bf16 on an NVIDIA L4, with a capped vision-encoder pixel budget and throughput of about 4.17 seconds per item.The reported throughput is 4.17 s/item for English and 4.18 s/item for MSA.Source: §4.1 · p. 3

4.2 Weight selection

  • WPW = 0.25 reached 0.9520 combined accuracy versus 0.9500 at WPW = 1.0, a one-item difference judged indistinguishable from noise on 500 English development items.The submission retained the 1:1 configuration; the 95% binomial interval spans ±1.9 points.Source: §4.2 · p. 3

4.3 Split usage

  • The 500-item labelled dev split supported validation, ablation, and weight sweeps, while the 1,000-item unlabelled test split was used for official runs.The train split was unused because the system involves no training, and devtest was unused because the development phase had closed.Source: §4.3 · p. 3

4.4 Evaluation Metrics

  • Contrastive Instability measures the share of items not fully correct among items with at least one correct label; secondary metrics include combined accuracy, CFHR, and Q+/Q− accuracy.Source: §4.4 · p. 3

5 Results

The system achieves low CI on both official test tracks, while ablations, cross-language comparisons, transfer analysis, error analysis, and examples characterize its behavior and limitations.

5.1 System Performance

  • CI was 0.0510 on English and 0.0960 on MSA; the system finished fifth and fourth, respectively, and reduced CI versus released 3B devtest baselines by factors of 6.1 and 5.1.It labelled all three statements correctly on 949 of 1,000 English items and 904 MSA items; the baseline comparison is mainly indicative because the baselines use devtest. Under the one-true decoding constraint, CI = 1 − acc, CFHR = 0, and Q−acc = (1 + acc)/2.Source: §5.1 · pp. 3–4, Table 1 · p. 4

5.2 Ablation Study

  • On English dev, pointwise scoring reached 0.9440 versus 0.9500 for multiple-choice scoring, while equal-weight fusion matched multiple-choice and WPW = 0.25 reached 0.9520.Differences were small, amounting to one to three items; pointwise scoring was chosen despite the small multiple-choice advantage. Without running the pipeline on the 3B baseline, the effects of the constraint, logit scoring, and model scale cannot be separated.Source: §5.2 · p. 4

5.3 English–MSA Gap

  • The English–MSA combined-accuracy gap was the largest among teams entering both tracks, and English-validated prompts, answer-token groups, and score weights were transferred to MSA without Arabic-side tuning.Prediction distributions were close to uniform on both tracks, suggesting little residual slot preference after rotation; the authors attribute the gap mainly to procedure.Source: §5.3 · p. 4, Table 3 · p. 4

5.4 Dev-to-Test Transfer

  • English dev accuracy was 0.9500 and blind-test accuracy was 0.9490, while the authors caution that nearby English leaderboard ranks are not clearly separated by aggregate scores alone.The English interval is approximately ±1.4 points; the 60-item gap to the top MSA system is described as unambiguous.Source: §5.4 · p. 4

5.5 Error Analysis

  • Across nine cultural categories, accuracy ranged from 0.84 to 0.97, but category-correctness tests were not significant; paired analysis found an English–MSA accuracy gap of 0.0450.The paired bootstrap 95% CI was [0.0310, 0.0600], and McNemar’s exact test rejected symmetry (p < 0.0001). Of 44 jointly missed items, 34 selected the same wrong statement and 10 different ones.Source: §5.5 · pp. 4–5, Table 4 · p. 5

5.6 Qualitative Examples

  • Manual examples include a shared scene-stereotype failure and an MSA-only failure associated with translation noise in the gold statement.In the cuisine example, pointwise scores were close but multiple-choice separated the grounded option; another example showed multiple-choice selecting the grounded decorative reading over a plausible seating interpretation.Source: §5.6 · p. 5

6 Conclusion

The submission presents a zero-shot inference and prompting strategy for Subtask 1b in English and MSA. It reports test-set CI scores of 0.0510 and 0.0960, respectively.

6 Conclusion

  • The system uses a pre-trained 7B vision–language model with no task-specific training or fine-tuning, framing the task as constrained selection with rotation-averaged scoring.It uses next-token log-probabilities and rotation averaging to reduce position bias.Source: §6 · p. 5
  • The system achieved CI scores of 0.0510 on English and 0.0960 on MSA, ranking 5th and 4th, respectively.Source: §6 · p. 5
  • Future work should validate prompts and answer-token groups directly on Arabic and compare baseline sizes to separate model-scale effects from the constrained formulation.Source: §6 · p. 5

Limitations

The reported results are limited to one model, prompt family, and run per track, with validation confined to English development data. Baseline and cross-team comparisons also have stated scope constraints, and the constrained formulation relies on a one-grounded-statement dataset guarantee.

Limitations

  • Results use one underlying model, prompt family, and run per track, and prompt-paraphrase variance was not measured.Source: Limitations · p. 5
  • All validation used 500 English development items with no Arabic-side validation, which the authors say likely contributed to the English–MSA gap.Source: Limitations · p. 5
  • Baseline comparisons are indicative because baseline scores are from devtest while the system's scores are from test; the constrained formulation relies on exactly one grounded statement and does not directly apply to open-ended detection.Source: Limitations · p. 5

A Hyperparameters and Configuration

The configuration selected precision and resolution automatically based on the detected GPU.

A Hyperparameters and Configuration

B Qualitative Examples

Table 6 is identified as the submitted system configuration.

B Qualitative Examples

  • Table 6 is identified as the submitted system configuration.Source: Table 6 · p. 7

C Prompts

The prompts first test each statement against the image, then require selecting the single supported statement; the MSA track keeps the English instruction wrapper and uses the released Arabic statements.

C Prompts

  • The statement-level template asks the model to judge only what the image shows, answering True when clearly supported and False when absent or contradicted.It requires exactly one word: True or False.Source: C Prompts · p. 7
  • The selection template presents three statements, says exactly one is image-supported, and requests its index as a single digit.The other two statements are described as plausible-sounding but unsupported.Source: C Prompts · p. 7
  • For MSA, the English instruction wrapper, answer-token groups, and rotation scheme remain identical; only the statement slots use the released Arabic statements.The Arabic statements are passed as released, without Arabic-language instruction wording or translation.Source: C Prompts · p. 7

D Metric Derivation

The derivation defines item counts and the one-true labeling setup, then relates the CI denominator, CFHR, and Q− accuracy to whether the grounded statement is selected.

D Metric Derivation

  • Under one-true decoding, each item has one true label and two false labels; n counts items and c counts items where the grounded statement is selected.If the grounded statement is missed, exactly one label remains correct, so every item has at least one correct label and CI uses denominator n.Source: D Metric Derivation · p. 7
  • CI is presented as 1 − acc, with acc corresponding to the share of items where the grounded statement is selected.The derivation states that the CI denominator is n.Source: D Metric Derivation · p. 7
  • CFHR is zero because selecting the grounded statement leaves both hallucinated statements labelled false.The text says the system therefore never affirms a hallucination after selecting the grounded statement.Source: D Metric Derivation · p. 7
  • Q− accuracy weights solved items as two correct hallucination labels and missed items as one; with c = 949 and n = 1000, it is 0.9745.The example is reported as matching the official scorer.Source: D Metric Derivation · p. 7

Related papers

Loading 2610.22515v1…