PaperlayerView exact revision on arXiv

Source-linked AI summary

HALDETECT at ImageEval 2026 Shared Tasks: Answer-First Contrastive Grounding with QLoRA

Syed Mohaiminul Hoque, Md Sakhawat Hossain

arXiv:2609.11236v1cs.CVcs.AI

TL;DR

HALDETECT tackles fluent visual hallucinations by selecting the single grounded statement from three culturally plausible alternatives in ImageEval 2026 Task 1b. It uses contrastive, answer-first prediction with observable-attribute checks and QLoRA adaptation, achieving CI 0.035 on the 1,000-item test set and third place among eight teams.

  • Problem

    Multimodal models can describe images fluently but incorrectly, especially when hallucinated statements remain culturally plausible and difficult to detect.

  • Method

    HALDETECT treats each item as one contrastive decision, answers before explaining, checks colour/texture, shape/form, and context, and applies QLoRA while freezing the vision encoder.

  • Results

    CI 0.035 on the 1,000-item test set placed HALDETECT third of eight teams.

  • Takeaways & Limitations

    Answer order can matter more than model scale, and observable-attribute reasoning plus adaptation improves the contrastive grounding setup.

Abstract

from arXiv · show

Large multimodal models tend to hallucinate visual detail fluently, which limits their deployment for fine-grained interpretation. We present HALDETECT, our system for the English hallucination-detection track (Task 1b) of ImageEval 2026, in which a system must identify, from an image and three culturally plausible statements, the single visually grounded one. We frame the item as one contrastive decision, emit the answer before its explanation, and structure reasoning around colour/texture, shape/form, and context. Our best submitted adapter fine-tunes Qwen2.5-VL-7B-Instruct with 4-bit QLoRA while freezing the vision encoder and reaches Contrastive Instability (CI) 0.035 on the 1,000-item test set; we placed third of eight teams. Development experiments show that answer order can matter more than model scale and that adaptation beats prompting alone. Retrospective paired analysis of the released gold labels confirms the QLoRA gain over the best prompt but not the small gap between the devtest-selected and best-test adapters, and reseeding all four training sizes shows that the apparent data-scaling curve does not survive a seed change. The 35 residual errors are culturally plausible function, material, and recognition distinctions; naive adapter voting does not help.

1 Introduction

HALDETECT addresses fluent but visually incorrect multimodal descriptions by treating each item as a contrastive choice among one grounded statement and two culturally plausible hallucinations. Its design exposes the exactly-one constraint, answers before explaining, checks observable evidence, and adapts language layers with QLoRA while freezing vision.

  • 1 Introduction: The approach targets deployment trust because convincing but non-factual visual descriptions are difficult to detect when their errors remain culturally plausible.ImageEval 2026 evaluates culturally grounded multimodal systems on Arabic visual content, while this work participates in the English Task 1b track.
  • 1 Introduction: HALDETECT frames Task 1b as one contrastive grounding decision among three statements, rather than three independent verification problems.The task contains one grounded statement and two culturally plausible counterfactual hallucinations, making cultural coherence insufficient as visual evidence.
  • 1 Introduction: The system compares all statements jointly, emits the label before its rationale, evaluates colour/texture, shape/form, and context, and adapts only language layers.The vision encoder remains frozen while QLoRA adapts the language model.

2 Task, Data, and Metrics

Task 1b evaluates whether a system can identify the single visually grounded statement among three culturally grounded candidates, using labels and metrics that explicitly capture their joint constraint. The dataset covers Arab-world scenes across stable categories but has unequal country coverage in test.

  • 2 Task, Data, and Metrics: Task 1b presents one image with three English statements, exactly one grounded, and requires binary labels with exactly one True.This contrasts with benchmarks that judge claims independently, because the candidates form one decision about the same image.
  • 2 Task, Data, and Metrics: CI is the official lower-is-better metric for partially correct items, while combined accuracy requires all three labels to match gold.Acc(Q+) measures accepting the grounded statement, and Acc(Q−) measures rejecting both hallucinations.
  • 2 Task, Data, and Metrics: The decoder’s exactly-one-True output makes CI equal item error, CFHR structurally zero, and combined accuracy equal Acc(Q+).Consequently, CFHR is 0.000 throughout for this decoding strategy.
  • 2 Task, Data, and Metrics: The dataset contains 3,000 labelled train, 500 dev, 500 devtest, and 1,000 test items spanning nine categories and 31 subcategories.Category mixtures are stable across splits, but test covers 13 rather than 18 Arab countries, with 28–98 items per country.

3 System

HALDETECT combines joint answer-first prediction, observable-attribute checking, and parameter-efficient QLoRA adaptation for Task 1b. Answer order improves performance enough to outweigh a larger model in development, while the observable-attribute checklist improves over sequential elimination and the final adapters freeze the vision tower.

  • 3 System: Figure 1 summarizes the system’s three design choices and recipe.The system description centers on joint prediction, answer-first ordering, and adaptation.
  • 3.1 Joint and answer-first prediction: Answer-first prompting lowers devtest CI from 0.092 to 0.050 at 7B and from 0.142 to 0.082 at 3B, so answer-first 3B beats reason-first 7B.The first generated line is the answer, with justification produced afterward.
  • 3.1 Joint and answer-first prediction: Answer-first reordering repairs 29 errors while introducing eight at 7B, with exact McNemar p = 7.5×10^-4, supporting a decision effect rather than formatting disruption.At 7B, only two changed predictions move between different wrong statements; the gap persists on parseable outputs.
  • 3.2 Evidence-guided reasoning: The observable-attribute checklist reaches CI 0.042 versus 0.050 for freeform reasoning and 0.056 for sequential elimination.Against sequential elimination, the checklist difference is −0.014 [−0.026, −0.004], repairing seven items and breaking none.
  • 3 System: The submitted models use 4-bit QLoRA on Qwen2.5-VL-7B-Instruct with the vision tower frozen and rank-8 adapters in language-model attention and MLP layers.Approximately 20M of 8.3B parameters are trained, or 0.24%, fitting on two 16GB T4 GPUs.

4 Experiments and Results

HALDETECT’s experiments compare joint contrastive prompting, answer-first reasoning, attribute checklists, and QLoRA adaptation, then test whether apparent training-size gains survive reseeding. The best submitted adapter reached test CI 0.035, while fresh seeds flattened the apparent data-scaling effect.

  • Development progression: QLoRA surpassed the best prompting-only ceiling of CI 0.042, with 2,348 items reaching devtest CI 0.028 and 2,600 items reaching 0.034.The controlled progression also reports that joint comparison cut CI by 44.7% relative.
  • Prompt ablations: Answer-first ordering was the only large, reliable prompt improvement, reducing paired CI by 0.042, whereas checklist gains were smaller and training-size comparisons crossed zero.The answer-first design requires the model to emit the decision before its rationale.
  • Official test results: CI 0.035 was achieved by the post-hoc best QLoRA submission on the 1,000-item test set, while the devtest-selected adapter scored CI 0.039.The leaderboard-selected 2,600-item adapter was a post-hoc best of ten; strict devtest selection would have chosen 2,348 items.
  • Training setup: The experiment uses a frozen vision tower with rank-8 language-layer LoRA and nested prefixes from one fixed permutation for the reseeded training-size comparison.The submitted recipe trains one clean epoch on two T4 GPUs.
  • Seed replication: Fresh-seed means were nearly flat across 2,000, 2,348, 2,600, and 3,000 items—.0397, .0403, .0407, and .0407—so the single-seed scaling curve did not replicate.Per-seed effects disagreed in sign, and each seed preferred a different training size.

5 Post-Task Test Error Analysis

Post-task paired analysis confirms that QLoRA improves over prompting, but does not support the small advantage of the post-hoc best adapter over the devtest-selected one. Remaining errors cluster around culturally plausible functional, material, and recognition distinctions, and adapter voting does not improve results.

  • Paired test comparison: QLoRA-2,600 reduced CI from .062 to .035 against the best prompt, while its comparison with QLoRA-2,348 was not significant.The paired analysis found 35 fixes and eight breaks against prompting, but only 11 fixes and seven breaks against the devtest-selected adapter.
  • Country analysis: Country-level differences do not establish a country effect because the highest observed rate’s interval overlaps Iraq’s and Yemen’s.Bahrain had the highest rate at 9/35, or 9.2%.
  • Error taxonomy: The 35 residual errors divide into function or intent, visual or material attributes, and object or scene recognition, with similar counts across independent annotations.The first annotation counted 16, nine, and ten errors; the second counted 14, nine, and 12.
  • Ensembling and position: No three- or four-adapter majority vote beats the single best adapter, although an oracle over the adapters would improve performance.Per-position error rates and the confusion matrix also show no position effect.

6 Conclusion

HALDETECT frames hallucination detection as contrastive grounding: compare candidates jointly, commit before explaining, inspect visible attributes, and adapt economically. It reaches test CI 0.035 and third place, while its residual errors and reseeding results identify limits of current evidence.

  • Conclusion: HALDETECT combines joint candidate comparison, answer-first commitment, observable-attribute checks, and economical language-layer adaptation.The system freezes the vision encoder while adapting the language layers.
  • Conclusion: The system achieved test CI 0.035 and third place, but residual errors remained culturally plausible distinctions involving purposes, materials, and fine visual recognition.Paired tests support adaptation over prompting, but not the small gap between the two leading adapters.
  • Conclusion: Reseeding all four training sizes removed the apparent single-seed data-scaling curve, and naive adapter voting did not improve results.The conclusion points toward rationale faithfulness and learned adapter combination as future work.

Limitations

HALDETECT’s reported configuration is compute-bounded rather than a swept optimum, and several uncertainty sources remain unresolved. The study cannot establish whether alternative seeds, adaptation settings, image budgets, or training durations would improve CI.

  • CI 0.035 is a single draw from a spread of about ±.003–.005, while bootstraps quantify finite-test rather than seed uncertainty.Every submitted QLoRA result uses one seed, and reseeding finds that the apparent training-size effect does not replicate.
  • The vision-frozen design lacks a controlled comparison with vision-tower adaptation, leaving the source of development errors partly inferential.Freezing vision was both a compute constraint and a modelling choice, but no vision parameters were adapted.
  • The fixed recipe does not establish an optimum because rank, precision, image budget, and training duration were not swept.Training was capped at one epoch because of the dual-T4 compute budget, so further epochs could improve or overfit.
  • Rationale faithfulness was not evaluated, limiting conclusions about whether generated explanations accurately reflect the model’s visual decision process.

Ethics Statement

The paper reports benchmark-only research using public data and no human-participant study, while documenting reproducibility and limitations of seed-based evaluation. Its replication analysis finds that apparent training-size gains do not survive reseeding.

  • Ethics Statement: The study used only the public Task 1b release, collected no new data, and ran no human-participant study.
  • Ethics Statement: The system is intended for benchmark evaluation only, and country-level rates or scene-purpose inferences should not be treated as evidence about communities or countries.
  • Seed Replication: Fresh-seed means stayed flat at .0397–.0407 across training sizes, while seed 42 alone improved from .049 to .035, indicating seed variance rather than a size effect.
  • Seed Replication: Across twelve replication runs, 92.0% of items were correct under every run, 1.8% under none, and 6.2% flipped with the seed.
  • Seed Replication: Eighteen items appeared in every fresh-size error core, while the submitted adapter had 35 errors and 17 additional errors that could flip with the seed.
  • Reproducibility: Replication CIs were archived for sizes through 2,600, whereas the 3,000-item fresh CIs were recorded from runs rather than recomputed from committed prediction archives.

F Error Taxonomy Definitions

The paper classifies residual failures by the distinction the model must make, emphasizing culturally plausible near-neighbour statements rather than formatting errors. These failures include function, material, and recognition contrasts.

  • F Error Taxonomy Definitions: The taxonomy separates failures into function/intent/event, visual/material, and recognition categories using a fixed tie-break order.
  • F Error Taxonomy Definitions: Figure 3 illustrates one failure per category from the 18 items that every seed replication and the submitted adapter answered incorrectly.
  • F Error Taxonomy Definitions: The 35 failures include distinctions such as decorative versus flavour-related flowers, cotton versus silk, and Arabic versus Hebrew script.

G Retrospective Paired and Ensemble Results

Retrospective analyses show that adapter-size gains are unstable, while the best model’s residual errors cluster in a few culturally grounded categories without systematic position confusion.

  • Selected pairwise comparisons use exact two-sided McNemar tests with Holm correction across 45 system pairs.Fixes count cases where B corrects an error by A; breaks count the reverse.
  • Adapter-size differences are small and uncertain: devtest size steps cross zero, while four adapters agree on 97.3–98.1% of predictions.The 2,348-versus-2,600 error-set Jaccard is 0.609, with 28 shared failures.
  • The 35 missed items include one illustrative failure from each category, spanning material, recognition, and image-unobservable intent distinctions.
  • 35 residual errors are concentrated in Arts & Entertainment, while Food & Cooking has the highest category rate at 6.5%.
  • Gold-position error rates overlap across positions, and the prediction-position table shows no systematic confusion.

H Cross-Family and Negative Results

Cross-family prompting results are mixed: answer-first helps InternVL2, but the checklist is largely ignored, so its lack of benefit does not test the intended attribute reasoning.

  • Answer-first improves InternVL2 CI from 0.298 to 0.232 at 2B and from 0.098 to 0.084 at 8B.
  • The InternVL2 checklist changes only 9/500 decisions, yielding four fixes, four regressions, and one wrong-to-wrong switch.
  • Under InternVL2-8B, 90.2% of checklist outputs omit any rationale and only 9.8% contain an attribute probe.None of the 49 compliant items changes prediction; all nine changes occur among outputs that ignore the instruction.
  • The cross-family null reflects instruction neglect rather than evidence against the usefulness of colour, shape, and context axes, but provides no positive cross-family evidence.
Loading 2609.11236v1…