PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation

Samir Abdaljalil, Hunzalah Hassan Bhatti, Ahlam Bashiti, Farina Amir, Md Arid Hasan, Basel Mousi, Nadir Durrani, Fahim Dalvi, Zien Sheikh Ali, Erchin Serpedin, Hasan Kurban, Mustafa Jarrar, Shammur Absar Chowdhury, Firoj Alam

arXiv:2608.30475v1cs.CLcs.AI

TL;DR

Existing multimodal benchmarks provide limited evidence about culturally specific reasoning and robustness across languages, while standard image metrics may miss cultural distortions. ImageEval 2026 addresses these gaps through Arabic-English spoken VQA, hallucination detection, and culturally grounded text-to-image evaluation, revealing challenges in MSA speech and structural cues in cultural image assessment.

  • Problem

    Existing benchmarks offer limited evidence on culturally specific reasoning across languages, and standard image metrics may miss culturally important distortions in generated images.

  • Method

    The shared task evaluates culturally grounded Arabic multimodal understanding and generation through AynVQA and CRAI-Bench, using spoken VQA, hallucination detection, and cultural accuracy assessment.

  • Results

    The results show that MSA spoken VQA is harder than English, hallucination-detection performance depends on task formulation, and cultural image evaluation is influenced by structural cues beyond visual content.

  • Takeaways & Limitations

    Culturally grounded Arabic multimodal evaluation remains challenging, requiring attention to Arabic speech conditions, hallucination task design, and direct visual assessment of cultural accuracy.

  • Takeaways & Limitations

    The benchmark is limited by constrained hallucination scenarios and a Qatari-only image set of 40 reference and 200 generated images from one text-to-image model.

Abstract

from arXiv · show

We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic multimodal evaluation. It includes two tasks: (i) AynVQA, covering spoken visual question answering and image-grounded hallucination detection in English and Modern Standard Arabic (MSA), and (ii) CRAI-Bench, evaluating the cultural accuracy of text-to-image generation. A total of 14 teams participated in the test phase, with 12 teams submitting system description papers. Participating systems used a range of approaches, including zero-shot prompting, fine-tuning of vision-language models, speech-recognition pipelines, ensembling, and score calibration. We describe the task setup, datasets, evaluation procedure, and participating systems, and summarize the main results across the different tracks. All datasets and evaluation scripts from the shared task are released to the research community. The shared task highlights the challenges of culturally grounded multimodal evaluation, particularly for Arabic speech and image-text reasoning.

1 Introduction

Existing multimodal benchmarks provide limited evidence of culturally specific reasoning, especially across languages and regional varieties. ImageEval 2026 addresses these gaps through unified Arabic-English evaluation of culturally grounded understanding and generation.

  • Motivation: General multimodal benchmarks largely emphasize scene understanding, leaving culturally specific and cross-variety reasoning underexamined.Cross-lingual benchmarks extend language coverage but often preserve the same visual domain.
  • Motivation: Cultural benchmarks show that multilingual coverage does not necessarily yield culturally grounded understanding.Models continue to struggle with artifacts, food, clothing, practices, landmarks, and regional identities.
  • Motivation: Arabic evaluation is especially important because systems must handle MSA, regional dialects, spoken questions, and culturally specific content.Evaluation of culturally grounded VQA and hallucination across Arabic varieties remains limited.
  • Motivation: Text-to-image systems can produce plausible images that omit or distort culturally important details, while standard metrics may miss these errors.Arabic and Gulf contexts risk being reduced to generic regional representations.
  • Contribution: ImageEval 2026 introduces AYNVQA for spoken VQA and hallucination detection and CRAI-BENCH for cultural accuracy in Arabic text-to-image generation.Together, the tasks evaluate cultural grounding across multimodal understanding and generation.
  • Findings: The shared task finds that MSA spoken VQA is harder than English, hallucination detection benefits from task-aware prediction, and cultural evaluation can exploit caption-specificity cues.The best MSA accuracy was 8.7 percentage points below the best English result.

2 Tasks and Datasets

The shared task evaluates culturally grounded Arabic multimodal systems through spoken understanding, hallucination detection, and cultural faithfulness of generated images. It defines task-specific datasets, metrics, baselines, and public evaluation procedures.

  • Tasks: The benchmark covers spoken questions grounded in images, culturally plausible hallucination detection, and cultural faithfulness in generated images.These are organized into the AynVQA and CRAI-BENCH tasks.
  • Task 1: AynVQA: AynVQA offers English and MSA subtasks for selecting answers to spoken questions and classifying one grounded statement against two unsupported alternatives.Task 1a provides three spoken answer options; Task 1b requires true/false judgments.
  • Task 1: AynVQA: Task 1 derives training and development data from OASIS and uses M2CQA for testing, with voice-cloned training speech and human-recorded test speech.Examples include country, cultural category, and subcategory labels.
  • Task 1: AynVQA: Task 1a is scored with accuracy, balanced accuracy, and macro-F1, while Task 1b uses contrastive instability alongside combined and statement-level measures.Contrastive instability evaluates consistency across triplets containing one grounded and two unsupported statements.
  • Task 2: CRAI-BENCH: CRAI-BENCH gives systems a Qatari reference image, caption, and generated image to predict five dimensions of the human-validated Cultural Representation Accuracy Index.The benchmark includes weighted CRAI dimensions and evaluates agreement with human cultural judgments.
  • Task 2: CRAI-BENCH: CRAI-BENCH contains 40 reference images across people and attire, architecture, and objects, each paired with five captions of decreasing cultural specificity.Images were culturally verified by native Qatari consultants.
  • Task 2: CRAI-BENCH: CRAI-BENCH uses reference-image-disjoint 60/20/20 train, development, and test partitions stratified by cultural category.This prevents captions and generated images from the same reference image crossing splits.
  • Task 2: CRAI-BENCH: The primary CRAI-BENCH metric is Spearman correlation with human CRAI_composite rankings, with MAE as a secondary metric and tiebreaker.Higher correlation and lower MAE are preferred.

3 Results and Discussion

Official results reveal three cross-task patterns: spoken VQA is harder in MSA than English, contrastive task formulation improves hallucination detection, and CRAI-BENCH scores can reflect caption structure beyond visual evidence.

  • Task 1a (Spoken VQA): 0.875 best MSA accuracy trails 0.962 in English, with Arabic ASR errors contributing substantially to the gap.The English track was closely matched, with the top two systems differing by 0.001 accuracy.
  • Evaluation overview: The official evaluation covers accuracy-based spoken VQA, contrastive instability for hallucination detection, and Spearman correlation for CRAI-BENCH.Tables 4, 5, and 6 report the blind test-set results for these tracks.
  • Task 1b (Hallucination Detection): 0.029 English CI and 0.036 MSA CI were achieved after reducing baseline instability through joint three-way statement selection.The corresponding baseline reductions were from 0.267 to 0.029 in English and from 0.428 to 0.036 in MSA.
  • Task 2 (CRAI-BENCH): All submitted CRAI-BENCH systems outperformed GPT-4o, with the top four reaching Spearman correlations between 0.781 and 0.826.The highest-ranked system used a caption-version prior with CLIP-based tie-breaking, while DATALabKU achieved the lowest MAE.
  • Task 2 (CRAI-BENCH): Caption specificity strongly associated with human CRAI scores, allowing systems using limited visual evidence to perform well.The results motivate reducing structural cues and emphasizing direct visual assessment in future benchmark versions.

4 System Descriptions

Participating systems combined zero-shot prompting, fine-tuning, ASR pipelines, constrained formulations, ensembling, verification, and score calibration across spoken VQA, hallucination detection, and cultural image evaluation.

  • Approaches: Teams explored zero-shot prompting, task-specific fine-tuning, ASR-based pipelines, ensembling, and score calibration across the three tasks.Table 7 summarizes the models, adaptation strategies, and methods used by submitted systems.
  • Hallucination detection: Dynamos combined constrained single-statement selection, low-rank fine-tuning, multiple adapters, and 7B/32B score averaging.Its final system achieved CI scores of 0.033 for English and 0.056 for MSA.
  • Hallucination detection: Nile Nexus used confidence-based paraphrase voting with constrained selection, achieving English accuracy 0.942 and CI 0.058.The system was built on Qwen2.5-VL-7B-Instruct with 4-bit quantization.
  • Hallucination detection: Team Falcons used three-way image-conditioned classification with QLoRA fine-tuning and dual-model verification, achieving CI 0.029.A prediction changed only when both verifier models agreed, placing the system first on the English leaderboard.
  • Spoken VQA: Spoken VQA systems included joint multimodal processing, answer-label comparisons, LoRA adaptation, and modular ASR-to-reasoning pipelines.Ahmed Ayman achieved accuracies of 0.962 in English and 0.834 in MSA; Digilians achieved 65.6% on the MSA blind test set.
  • CRAI-BENCH: md_faisal used forced-choice hallucination detection and a caption-version prior with CLIP tie-breaking, reaching Spearman 0.8258 and ranking first on Task 2.The approach was zero-shot for both Task 1b and CRAI-BENCH.
  • Cross-task systems: Ahmed Younis reformulated MSA hallucination detection as single-grounded-statement selection, achieving combined accuracy 0.964 and ranking first.For CRAI-BENCH, the team blended a frequency-based prior with two vision-based judges to reach Spearman 0.7806.

5 Related Work

Related work establishes gaps in culturally grounded Arabic multimodal evaluation across spoken QA, hallucination detection, and text-to-image cultural accuracy.

  • Culturally grounded multimodal benchmarks: Existing multimodal benchmarks largely emphasize general scene understanding, providing limited evidence about culturally specific reasoning across languages and regional varieties.Culture-centered benchmarks broaden coverage to artifacts, food, clothing, practices, landmarks, and regional identities, but VLMs still struggle with cultural understanding.
  • Spoken visual question answering: Arabic multimodal QA has focused mainly on image-text settings, while evaluation across Arabic varieties and spoken questions remains limited.OASIS extends speech coverage with 3.7M spoken questions across 18 Arab countries, including MSA and dialectal Arabic.
  • Hallucination benchmarks and metrics: Common hallucination metrics often fail to distinguish visual-recognition errors from acceptance of culturally plausible but visually unsupported alternatives.ImageEval focuses on faithfulness hallucination, and Arabic and MENA hallucination evaluation remains limited.
  • Cultural text-to-image evaluation: Text-to-image cultural benchmarks increasingly evaluate cultural accuracy alongside visual quality and prompt alignment across countries and settings.Prior work includes CUBE, CULTDIFF, CulturalFrames, RusCode, CULTIVate, and MOSAIG.

6 Conclusion

ImageEval 2026 combines AYNVQA and CRAI-BENCH to evaluate culturally grounded Arabic multimodal understanding and generation across English and MSA. Its findings identify persistent challenges in Arabic speech, hallucination detection, and culturally faithful image evaluation.

  • Conclusion: ImageEval 2026 combines spoken VQA and image-grounded hallucination detection in English and MSA with cultural accuracy evaluation for generated images.The shared task had 14 test-phase teams, including 12 teams submitting system description papers.
  • Conclusion: Spoken VQA is more challenging in MSA than English, with Arabic ASR contributing substantially to the performance gap.The conclusion identifies robust Arabic speech processing as a direction for further work.
  • Conclusion: Hallucination detection benefits from task-aware single-choice formulations that directly identify the visually grounded statement.The conclusion motivates visually grounded hallucination detection methods aligned with the task structure.
  • Conclusion: CRAI-BENCH shows that caption-specificity cues can strongly influence cultural evaluation, even when systems use limited visual evidence.The conclusion motivates cultural evaluation methods relying more directly on image content.

Limitations

The shared task has important scope and design limitations spanning speech conditions, hallucination structure, and cultural coverage. These constraints motivate broader sources, settings, and evaluation cues in future editions.

  • Speech conditions: Task 1a mixes synthesized training and development audio with human-recorded test audio, particularly challenging MSA performance.The speech-condition shift may affect how development results transfer to the test setting.
  • Hallucination detection: Task 1b uses exactly one grounded statement and two unsupported alternatives, limiting generalization to less structured hallucination scenarios.The constrained statement format can simplify the detection problem relative to open-ended evaluation.
  • Evaluation cues: Task 2 caption versions correlate strongly with human scores, enabling systems to perform well with limited visual evidence.This structural cue can weaken the extent to which scores reflect image-based cultural evaluation.

Societal/Broader Impact

ImageEval 2026 releases shared resources to support development and comparison of Arabic multimodal systems while exposing gaps in speech recognition and generated-image cultural accuracy. Its scores should be treated as one component of broader human-centered evaluation across diverse Arab contexts.

  • Research access: The released data, evaluation scripts, and baselines can lower barriers to developing and comparing Arabic multimodal systems.The benchmark is intended to help identify weaknesses that may otherwise remain hidden in predominantly general evaluations.
  • Systemic gaps: Results highlight notable gaps in Arabic speech recognition and the cultural accuracy of generated images.These gaps may affect the reliability and inclusiveness of systems deployed for Arabic-speaking users.
  • Evaluation scope: Benchmark scores are not a complete measure of cultural competence because Arab societies are linguistically, geographically, and culturally diverse.The paper recommends expanding coverage across countries, dialects, communities, and cultural perspectives.
  • Evaluation scope: The benchmark should be used as one component of broader human-centered evaluation.This recommendation follows the paper’s emphasis on cultural and regional diversity.

A.1 Development-Phase Results

Development results were evaluated on task-specific splits and show stronger English performance for spoken VQA, persistent MSA difficulty, and improvements over released baselines. Both Task 2 systems substantially outperform the GPT-4o baseline.

  • Evaluation setup: Development submissions were evaluated on the dev-test split for Task 1 and the development split for Task 2.Task 1a used synthetic TTS audio for development evaluation.
  • Task 1a: Spoken VQA: English systems generally outperform MSA systems on Task 1a, with a wider performance spread in MSA.This pattern indicates greater variability among MSA submissions in spoken VQA.
  • Task 1b: Hallucination Detection: MSA remains more challenging overall in Task 1b despite strong performance among leading systems in both languages.Task 1b ranks systems by contrastive instability, where lower values are better.
  • Task 2: Cultural Image Accuracy: Both Task 2 systems substantially outperform the GPT-4o baseline on the development split.Task 2 results are ranked by Spearman correlation of the composite score.
  • Overall pattern: Across tasks, development results show consistent improvements over the released baselines.The overall pattern combines stronger English spoken VQA performance with gains relative to the official baselines.
Loading 2608.30475v1…