PaperlayerView exact revision on arXiv

Source-linked AI summary

DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture

Arijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka, Nemil Shah, Abhilekh Borah, Vanshika Shah, Nishant Mishra, Sriparna Saha

arXiv:2509.19274v1cs.CLcs.MM

TL;DR

Existing benchmarks lack the joint cultural specificity and multimodal grounding needed to assess culturally competent AI across India's diversity. DRISHTIKON evaluates VLMs with a multilingual, multimodal Indian-culture benchmark, and finds performance gaps, particularly for low-resource languages and regional contexts.

  • Problem

    Existing benchmarks often lack culturally specific multimodal grounding and joint coverage of India's languages and regions, limiting evaluation of culturally competent AI.

    Source: §1 · p. 1

  • Method

    DRISHTIKON is a multilingual, multimodal Indian-culture benchmark with 64,288 curated instances, evaluated across diverse VLMs and prompting settings.

    Source: §1 · p. 2

  • Results

    VLMs show gaps on culturally grounded tasks, especially in low-resource languages and region-specific contexts, while GPT-4o mini performs strongly and compact models and Maya deliver competitive results.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings underscore the need for culturally inclusive AI development and identify persistent challenges in culturally grounded model reasoning.

    Source: §1 · p. 2

    Curated image-text pairs may carry annotator bias and may not fully represent India's regional nuances or real-world multimodal complexity.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We introduce DRISHTIKON, a first-of-its-kind multimodal and multilingual benchmark centered exclusively on Indian culture, designed to evaluate the cultural understanding of generative AI systems. Unlike existing benchmarks with a generic or global scope, DRISHTIKON offers deep, fine-grained coverage across India's diverse regions, spanning 15 languages, covering all states and union territories, and incorporating over 64,000 aligned text-image pairs. The dataset captures rich cultural themes including festivals, attire, cuisines, art forms, and historical heritage amongst many more. We evaluate a wide range of vision-language models (VLMs), including open-source small and large models, proprietary systems, reasoning-specialized VLMs, and Indic-focused models, across zero-shot and chain-of-thought settings. Our results expose key limitations in current models' ability to reason over culturally grounded, multimodal inputs, particularly for low-resource languages and less-documented traditions. DRISHTIKON fills a vital gap in inclusive AI research, offering a robust testbed to advance culturally aware, multimodally competent language technologies.

1 Introduction

Existing benchmarks do not jointly provide culturally specific multimodal coverage across Indian languages and regions. DRISHTIKON evaluates cultural reasoning with a broad, aligned question-image benchmark and varied VLMs.

1 Introduction

  • Existing benchmarks often lack joint coverage of Indian languages, visual modalities, nuanced cultural understanding, and all states and union territories.Source: §1 · pp. 1–2
  • The benchmark spans 15 languages and all 28 states and 8 union territories, with diverse visual-textual cultural content.It is presented as the first multimodal, multilingual benchmark tailored to evaluate cultural reasoning in India.Source: §1 · p. 2
  • Its 64,288 curated instances have fine-grained annotations and multiple-choice questions for assessing VLM perception, inference, and cultural alignment.Source: §1 · p. 2
  • The evaluation covers compact, large, proprietary, reasoning-specialized, and Indic-aligned VLMs under zero-shot and chain-of-thought settings.Source: §1 · p. 2
  • DRISHTIKON exposes VLM gaps on culturally grounded tasks, especially in low-resource languages and region-specific contexts.The findings underscore a need for culturally inclusive AI development.Source: §1 · p. 2

2 Related Works

Prior work addresses cultural benchmarking and Indian linguistic or regional diversity in separate ways. DRISHTIKON is positioned as integrating multilingual, multimodal, culturally grounded QA across India's regions.

2.1 Multimodal and Multilingual Cultural Benchmarks

  • Multilingual and multicultural benchmarks such as M5, CVQA, GIMMICK, and CulturalVQA examine broad cultural or geographic variation, but report limits in low-resource or non-Western cultural understanding.The cited work includes evaluations across 41 languages, 26 languages and 28 countries, 728 cultural facets across 144 countries, and 11 countries.Source: §2.1 · p. 2
  • Other benchmarks and datasets address linguistic or regional diversity, but the authors say none match DRISHTIKON's fine-grained, culturally rich coverage of India.Source: §2.1 · p. 2

2.2 Regionally and Culturally Rich Indian Corpora

  • Indian corpora address sociolinguistic bias, dialects, Indic NLP, and factual QA; DRISHTIKON instead combines multilingual, multimodal cultural QA and visual reasoning across all 36 states and union territories.The authors characterize it as the first large-scale benchmark to holistically evaluate cultural competence in generative models using text and visuals.Source: §2.2 · p. 3

3 DRISHTIKON Dataset Construction

DRISHTIKON's construction moves from curated cultural knowledge to annotated and reasoning-augmented question-image pairs, then translates them to support multilingual evaluation.

3.1 Knowledge Curation and MCQ Generation

  • The dataset draws on authoritative repositories, tourism portals, academic collections, and curated crowdsourced platforms for content spanning cultural practices and heritage.Source: §3.1 · p. 3
  • The authors created 2,126 English MCQs covering all states and union territories, using four choices to balance cultural granularity, annotation effort, and model load.The format sets chance-level guessing at 25% and supports consistent accuracy evaluation.Source: §3.1 · pp. 3–4
  • Questions were validated for accuracy, clarity, and cultural sensitivity, paired with relevant images, and augmented with semantically rich formats to reduce superficial pattern matching.The authors chose MCQs for comparability and reproducibility across languages and VLMs.Source: §3.1 · p. 4

3.2 Cultural Categorization and Attribute Tagging

  • A dynamic cultural taxonomy tags question-image pairs, with trained annotators applying standardized guidelines and resolving ambiguous cases through consensus and expert adjudication.Source: §3.2 · p. 4

3.3 Reasoning-based Question Augmentation

  • A balanced subset of 720 base MCQs was augmented into common-sense cultural, multi-hop, and analogy questions, yielding 2,160 additional MCQs.Sampling capped representation at 20 questions per region, and all augmented questions were manually reviewed.Source: §3.3 · p. 5

3.4 Multilingual Translation and Dataset Scale-up

  • The base and augmented questions were translated into 14 Indian languages, with stratified human verification for meaning, fluency, and cultural relevance.The resulting dataset contains 64,288 question-image-language triples spanning 36 regions and 16 cultural themes.Source: §3.4 · p. 5
  • The authors present DRISHTIKON as the first large-scale multilingual, multimodal benchmark explicitly designed to evaluate cultural competence and generalization in generative AI.Source: §3.4 · p. 5

4 Experimental Setup

The evaluation compares a broad range of VLMs under a unified protocol where possible, using accuracy as its primary metric.

4 Experimental Setup

  • The protocol standardizes image resolution at 224 × 224 or higher, uses prompts aligned with model instruction tuning, and sets token limits by architecture.Multi-turn reasoning is allowed when supported by the model.Source: §4 · p. 6
  • The evaluated models span open-source small and large systems, proprietary models, reasoning-specialized VLMs, and Indic-aligned models.Accuracy is the primary metric, measuring the proportion of correctly answered multiple-choice questions.Source: §4 · p. 6

5 Results

Across the evaluated models, performance varies by model, language, question type, and prompting strategy. The results show persistent regional and linguistic disparities, with reasoning-focused questions remaining difficult even when chain-of-thought prompting helps.

5.1 Analysis of Radar Graphs

  • Performance is higher in states with stronger media presence or distinct cultural signatures, but lower in smaller or less-represented regions, and even top models fluctuate across states.The radar analysis identifies Kerala, Gujarat, and West Bengal among stronger-performing states and Lakshadweep, Mizoram, and Dadra and Nagar Haveli among lower-performing regions.Source: §5.1 · pp. 6–7

5.2 RQ1: To what extent does model scale correlate with performance in multilingual multimodal tasks?

  • GPT-4o mini consistently delivers top-tier performance across all languages and question types, while Maya shows competitive accuracy and SmolVLM-256M-Instruct and InternVL3-1B often outperform heavier LLMs in overall accuracy.Kimi-VL-A3B-Thinking and Chitrarth-1 show limited generalization, with accuracies significantly trailing in both zero-shot and CoT settings.Source: §5.2 · p. 7

5.4 RQ3: What types of questions pose difficulties to current vision-language models?

  • General and common-sense cultural questions receive the highest accuracy, while multi-hop reasoning drops sharply and analogy questions vary most across models.CoT moderately lifts multi-hop scores, but gains are not uniformly robust across languages.Source: §5.4 · p. 7, Figure 5 · p. 8

5.3 RQ2: How do vision-language models vary in performance across Indian languages with unequal resource support?

  • Sindhi, Konkani, and Kannada accuracy drops by over 40% in some cases compared with English; Assamese and Odia also show uneven performance despite wide speaker bases.Hindi, Bengali, and Marathi follow English among the better-performing languages.Source: §5.3 · p. 7, Figure 6 · p. 8

5.5 RQ4: How does model typology influence performance across task categories and languages?

  • GPT-4o mini remains strongest, while compact SLMs and Maya are competitive; reasoning-specialized and some Indic-focused models generalize less reliably.Compact SLMs particularly excel on general and commonsense tasks, while Maya outperforms several larger general-purpose LLMs in culturally grounded understanding.Source: §5.5 · p. 9

5.6 RQ5: How does model performance differ between zero-shot and Chain-of-Thought (CoT) prompting across various question types, and which models benefit most from reasoning scaffolds?

  • CoT gains reach 10–15% on multi-hop and analogy questions, but vary by question type, model family, and language.Large proprietary models benefit consistently; smaller instruction-tuned models can be competitive, while reasoning-specialized and Indic-focused models show limited or inconsistent improvements.Source: §5.6 · p. 9

6 Conclusion

DRISHTIKON evaluates vision-language models in the Indian cultural context and reveals strong results from proprietary and compact models alongside persistent language gaps. The conclusion emphasizes inclusive, culturally aware, and efficient models as a continuing need.

6 Conclusion

  • GPT-4o mini performs strongly, compact instruction-tuned models and Maya are competitive, and low-resource Indian languages continue to trail.The authors present DRISHTIKON as a benchmark for future multilingual research.Source: §6 · p. 9

Limitations

The benchmark cannot exhaustively represent India's regional, dialectal, and socio-cultural variation, and curated image-text pairs may not capture real-world multimodal complexity. Models also continue to struggle with abstract analogies and multi-hop reasoning, including under CoT prompting.

Limitations

  • The dataset cannot exhaustively represent India's regional nuances, dialects, and socio-cultural practices; curated image-text pairs may also carry annotator bias and miss real-world complexity.These limits constrain how fully the benchmark represents cultural variation and real-world multimodal scenarios.Source: Limitations · pp. 9–10
  • Even with CoT prompting, many VLMs struggle with abstract analogies and multi-hop reasoning, indicating limits in compositional understanding.Source: Limitations · p. 10

Ethics Statement

DRISHTIKON draws on public resources and licensed materials, with multilingual annotators, validation, and efforts to mitigate bias and harmful stereotypes. Its dataset is intended for research and non-commercial use, with image URLs provided instead of hosted images.

Ethics Statement

  • Dataset sourcing used publicly available resources and licensed materials, though some regional or dialectal variations may remain underrepresented.Source: Ethics Statement · p. 10
  • Language-proficient, culturally familiar annotators were trained and fairly compensated; validation aimed to support consistency, cultural sensitivity, and respectful representation.Source: Ethics Statement · p. 10
  • DRISHTIKON is intended for academic research promoting fairness and inclusion, not for biased, discriminatory, or culturally insensitive use.Source: Ethics Statement · p. 10
  • The dataset is restricted to research and non-commercial use; it supplies URLs rather than hosting images, and users must follow original source licenses.Source: Ethics Statement · p. 10

A.1 Data Sources and Selection Criteria

DRISHTIKON’s source corpus combines encyclopedic, region-specific, culturally focused, visual, and contemporary material, retaining entries only when multiple sources substantiate them.

A.1 Data Sources and Selection Criteria

  • Wikipedia provided referenced summaries on festivals, attire, cuisines, monuments, and personalities, with citations cross-checked for factual accuracy.Source: §A.1 · p. 14
  • Ritiriwaz and Holidify contributed detail on customs and ethnic wear, alongside local attractions, festivals, and seasonal events.Source: §A.1 · pp. 14–15
  • Google Arts & Culture supplied curated visual and narrative material on art, dance, textiles, and heritage, while Times of India added contemporary cultural coverage.Source: §A.1 · p. 15
  • Sources were selected for reliability, geographic and domain diversity, cultural granularity, and multimodal availability; facts were cross-referenced and retained when substantiated by multiple sources.Source: §A.1 · p. 15

A.2 Annotation Methodology and Validation Protocol

The authors describe a multistage validation process for the benchmark’s multimodal questions, with a visual MCQ example provided.

A.2 Annotation Methodology and Validation Protocol

  • The validation process targeted quality and cultural fidelity across 2,126 multimodal multiple-choice questions.Source: §A.2 · p. 15
  • Validation was organized in multiple stages for the multimodal questions.Source: §A.2 · p. 15
  • Figure 7 provides an example of a question used in the validation discussion.Source: §A.2 · p. 15

A.2.1 Annotation Workflow

This section documents DRISHTIKON’s annotation, quality-control, tagging, sampling, and translation-verification processes, along with dataset-distribution analyses and culturally grounded model errors. It also describes prompts designed to elicit visual, cultural, logical, and regional reasoning across several question types.

A.2.1 Annotation Workflow

  • Trained annotators authored culturally grounded, clear, neutral questions, while visual content was cross-checked against two textual sources and screened for answer-revealing overlays or location tags.Annotators were recruited for regional-language fluency and trained on guidelines and quality expectations.Source: §A.2.1 · p. 15, §A.2.4 · p. 16, §A.6 · p. 19

A.2.3 Agreement and Quality Control

  • Cohen’s κ was 0.82 on a random 20% MCQ sample; peer review and expert adjudication resolved factual, linguistic, and regional-overlap disagreements.Disagreements were resolved through discussion or expert input.Source: §A.2.3 · p. 15, §A.2.2 · p. 15

A.3.1 Attribute Tagging Methodology

  • Each MCQ received one cultural attribute; ambiguous cases were adjudicated by representative context, such as tagging Ganga Aarti as a daily ritual rather than a festival.The taxonomy spans categories including festivals, attire, art forms, language, cuisine, geography, and historical heritage.Source: §A.3.1 · pp. 16–17

A.4 Sampling and Reasoning-based Augmentation

  • Reasoning-based augmentation used 720 MCQs (20 per region across 36 Indian states and union territories); stratified sampling retained rare cultural dimensions, and Pearson’s chi-squared tests found no significant attribute-variety loss (p > 0.1).For regions with more than 20 MCQs, the selected questions’ attribute distribution remained approximately proportional to the full regional set.Source: §A.4 · p. 17, §A.4.1 · p. 17, §A.4.2 · p. 17

A.5 Translation Quality and Human Verification Protocol

  • Gemini Pro translations underwent bilingual review and a separate quality-control stage; illustrative Hindi examples preserved the Ganga Aarti question’s meaning and the cultural relevance of Litti Chokha.The workflow aimed to maintain semantic, syntactic, and cultural integrity while scaling multilingual data curation.Source: §A.5 · p. 18

A.7.1 Word Cloud Analyses

  • The vocabulary visualizations show culturally specific terms alongside broader corpus themes, including regional names and concepts such as Jani Shikar, Yakshagana, and Mysuru Dasara.The full-corpus word cloud also highlights both general-knowledge and region-specific focus.Source: Figure 11 · p. 19, §A.7.1 · p. 20

A.7.2 State-wise Question Distribution.

  • Across 36 states and union territories, the latter three question types had 300 questions per region, while General Questions varied with data availability and population-specific cultural variance.This heterogeneity motivated uniform 20-question sampling for reasoning-based augmentation.Source: Figure 12 · p. 20, §A.7.2 · p. 20

A.9 Error Analysis

  • Manual analysis of selected GPT-4o-mini failures found fine-grained semantic confusion, particularly when distractor options were semantically close to the correct answer.The examples comprised English questions, associated images, and incorrect model predictions.Source: §A.9 · p. 20
  • The error analysis identifies reliance on lexical cues over context and gaps in visual grounding when questions require deeper regional or cultural knowledge.Its examples are accompanied by interpretive commentary on improving multimodal and multilingual reasoning.Source: §A.9 · p. 21

A.9.1 Error Case 1: Historical Leader Identification

  • The model predicted Sidho and Kanho Murmu instead of Budhu Bhagat, apparently matching horseback tribal-warrior imagery to the better-known Santhal rather than Kol Rebellion.The commentary attributes the error to surface-level visual patterns and leader popularity rather than historical specificity or regional cues.Source: §4 · p. 21

A.9.2 Error Case 2: Misclassification of Cultural Dance Form

  • The model identified Odisha instead of Jharkhand for Paika Akhara, apparently confusing it with the visually comparable, martial-themed Paika dance of Odisha.The commentary suggests visual and thematic overlap, along with bias toward more widely documented traditions, contributed to the error.Source: §4 · p. 22

A.9.3 Error Case 3: Misidentification of Tribal Art Form

  • The model identified Gujarat rather than Jharkhand as Paitkar painting’s origin, with the commentary pointing to confusion with better-known folk arts and underexposure to eastern Indian tribal art.Paitkar is described as one of India’s earliest scroll-painting traditions.Source: §4 · p. 22

A.9.4 Error Case 4: Misclassification of Cultural Landmark Location

  • Despite English and Hindi signage, the model placed the National Gallery of Modern Art in Mumbai instead of Delhi, failing to distinguish the Delhi branch’s architecture and setting.The commentary notes that NGMA branches in Mumbai and Bengaluru may have contributed to the confusion.Source: §4 · p. 22

A.10 COT prompt

  • The culturally grounded chain-of-thought prompt uses visual insight, cultural memory, logical integration, and regional contextualization, while requiring only the final answer.It is inspired by classical Indian epistemology and is intended to encourage nuanced reasoning with concise output.Source: §A.10 · pp. 22–23

A.11 Frequently Asked Questions (FAQs)

  • The FAQ frames the study as evaluating cultural reasoning through multimodal prompts, with images anchoring artifacts or symbols and prompts requiring contextual, symbolic, or multi-hop reasoning.It also states that models receive the same image-question pairings and prompts for fair comparison.Source: §A.11 · p. 23

A.12 Prompt Designs for Different Question Types

  • The prompt designs target cultural commonsense, multi-hop inference, and analogy, using image references and contextual clues to probe distinct reasoning skills.The multi-hop prompt requires at least two connected facts, while the analogy prompt asks models to relate cultural equivalents or symbols.Source: §A.12 · pp. 23, 25

Related papers

Loading 2509.19274v1…