Source-linked AI summary
MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir, Mueeze Al Mushabbir, Mohammed Saidul Islam, Mir Rayat Imtiaz Hossain, Md Tahmid Rahman Laskar, Sabbir Ahmed
TL;DR
Vision-language models still struggle to interpret memes whose humor depends on cultural knowledge and pragmatic inference. The paper introduces MEMECULT-1K and evaluates thirteen VLMs with and without minimal cultural context, finding consistent context-related gains while identifying persistent, model-specific failure modes. These findings motivate explicit cultural knowledge integration, within a benchmark limited to South Asian languages and culturally subjective evaluation.
Problem
Meme understanding requires implicit cultural knowledge and pragmatic inference beyond recognizing visual content or literal text.
Method
The paper constructs MEMECULT-1K, a multilingual benchmark of South Asian memes with cultural context notes, three human explanations, and a Bengali dialect set, then evaluates thirteen VLMs in two input settings.
Results
Minimal cultural context consistently improves explanation quality across thirteen VLMs, languages, automatic metrics, LLM judging, and human evaluation, while failure modes differ between model families.
Takeaways & Limitations
Culturally grounded meme understanding remains difficult, and the benchmark supports studying explicit cultural knowledge integration in multimodal reasoning.
Takeaways & Limitations
The benchmark covers Bengali, Hindi, and English South Asian memes, and its findings may not generalize to other regions or linguistic traditions.
Abstract
from arXiv · showhide
Meme understanding goes beyond recognizing visual content or literal text; it requires implicit cultural knowledge and pragmatic inference that most vision-language models still lack. We introduce MemeCULT-1K, a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, where each meme is paired with a cultural context note and three human-written explanations, along with a supplementary set of 54 Bengali regional dialect memes. We evaluate thirteen popular Vision Language Models (VLMs) under two settings: meme-only and context-aware. Providing minimal cultural context yields consistent gains across all models and languages: mean SBERT similarity improves from 44.6 to 56.4 (+11.8), BLEURT from 37.3 to 42.3 (+5.0), and LLM-as-a-Judge scores from 2.57 to 3.43 out of 5 (+0.86). Fine-grained error analysis reveals that closed-source models fail mainly on entity and reference misidentification, while open-source models are bottlenecked by broader cultural knowledge gaps, with linguistic and phonological failures proving the most context-resistant across both. These results highlight the difficulty of culturally grounded meme understanding and motivate future work on explicit cultural knowledge integration. Our dataset and code are publicly available at TawsifDipto17/MemeCULT-1K.
1 Introduction
Meme understanding requires culturally grounded multimodal reasoning because visual and textual recognition alone may miss implicit references and pragmatic humor. MEMECULT-1K addresses this gap with a multilingual, context-aware benchmark and fine-grained failure analysis.
- Motivation: Cultural knowledge, social context, and pragmatic inference often determine whether a model understands meme humor.A model can parse captions and visual content yet miss the underlying cultural reference or social situation.
- Research gap: Existing benchmarks emphasize classification, while explanation-centric resources remain predominantly English- and Western-centric.South Asian memes add Bengali, Hindi, English, code-mixed text, and region-specific references involving cricket and cinema.
- Benchmark: MEMECULT-1K contains over 1,000 South Asian memes with cultural context notes and three human-written English explanations per meme.The benchmark covers Bengali, English, and Hindi and includes a supplementary set of 54 Bengali regional dialect memes.
- Evaluation: Thirteen closed- and open-source VLMs are evaluated in meme-only and context-aware settings.The context-aware setting adds a cultural context note to the meme input.
- Contributions: The benchmark combines context-aware evaluation with fine-grained analysis of persistent culturally grounded VLM failures.Its contributions include a multilingual meme-explanation dataset, contextual evaluation, and failure analysis.
2 Related Work
Prior meme benchmarks mainly assess classification, whereas explanation-centric resources better target implicit interpretation but remain limited in linguistic and cultural coverage.
- Classification benchmarks: Existing benchmarks primarily classify sentiment, humor, hatefulness, misogyny, or abusive content rather than recovering a meme’s implicit premise.Classification labels do not directly assess whether a system can explain what makes a meme meaningful and funny.
- Explanation benchmarks: Explanation-centric datasets pair memes with captions or explanatory evidence but remain English- and Western-centric.MemeCap and MEMEX exemplify explanation-focused evaluation resources.
3 Dataset
MEMECULT-1K is a balanced multilingual South Asian meme dataset with contextual annotations, independent explanations, and difficulty information. Its analysis shows that meme understanding depends on varied contextual knowledge beyond image and text content.
- Dataset composition: 1,000 memes are balanced across Bengali (335), English (331), and Hindi (334), with 54 Bengali regional dialect memes as a supplementary set.The memes include frequent code-mixing and culturally specific references.
- Annotation: Each meme has one short cultural context note and three independent English explanations.Thirteen trained annotators produced 4,000 text entries, which were validated through majority agreement and curator review.
- Data creation: 1,850 collected memes were filtered through deduplication, image-quality review, OCR checks, cultural-relevance checks, and content exclusions before yielding the final dataset.The pipeline removed explicit, hateful, political, duplicate, low-quality, and illegible memes.
4 Experimental Setting
The experiments compare diverse closed- and open-source VLMs on Bengali, English, and Hindi memes under meme-only and context-aware conditions. Models generate one explanation per meme using a fixed deterministic prompt, with scores averaged over three references.
- Models: The evaluation spans closed- and open-source VLM families, including GPT, Gemini Flash, multiple open-source parameter scales, and Gemini 2.5 Pro.Gemini 2.5 Pro is included as a reference point for multilingual performance.
- Evaluation protocol: Each model is evaluated on Bengali, English, and Hindi subsets in meme-only and context-aware settings.The context-aware condition supplies a brief cultural context note in addition to the image.
- Generation and scoring: A fixed prompt requests one explanatory sentence, with temperature 0 and a 100-token output cap.Scores are averaged across three human references per meme.
5 Results and Analysis
Across automatic, judge-based, and error analyses, cultural context improves VLM meme explanations but does not eliminate culturally grounded failures. Performance and recovery vary by language, model family, and error type.
- Automatic Metric Evaluation: Cultural context consistently improves performance across all languages and automatic metrics, although scores remain far from saturation.
- Language-wise Trends: English scores highest across models and metrics, while Bengali scores lowest overall but gains most from added context.
- Judge-Based Evaluation: LLM-as-a-Judge scores correlate strongly with human ratings (Spearman ρ = 0.92), while judge-based context gaps exceed those shown by automatic metrics.
- Error Analysis: Gemini mainly exhibits Entity Misidentification, whereas Gemma is dominated by Cultural Knowledge Gap errors, especially in Bengali and Hindi.
- Error Analysis: Linguistic Failure remains among the least recoverable categories, indicating that short context notes do not resolve phonological, cross-script, or dialect-specific wordplay.
- Regional Bengali Dialects: Regional Bengali dialect memes are substantially more challenging, although context improves performance across models.
6 Conclusion and Future Work
MemeCULT-1K benchmarks culturally grounded meme understanding across South Asian languages and shows that minimal context improves explanation quality while important failures persist. The results motivate culturally grounded and cross-script multimodal reasoning.
- MemeCULT-1K is a multilingual benchmark for culturally grounded meme understanding in South Asian languages.
- Across thirteen VLMs, minimal cultural context consistently improves explanation quality across automatic metrics, LLM judging, and human evaluation, but performance remains far from ceiling.
- Closed-source models often misidentify cultural references, open-source models lack regional background knowledge, and linguistic or phonological failures resist context.
Limitations
The benchmark is geographically, linguistically, and methodologically bounded, and its concise context notes may not resolve deeply entrenched cultural or phonological wordplay.
- The dataset covers Bengali, Hindi, and English South Asian memes, so findings may not generalize to other regions or linguistic traditions.
- Per-language scale is modest, making results within a single language-metric cell or error category indicative rather than definitive.
- Automatic semantic-similarity metrics do not fully capture humor, irony, or pragmatic nuance, motivating complementary judge and human evaluation.
- Concise cultural context notes may not fully resolve deeply entrenched cultural or phonological wordplay.
Ethics Statement
The study used publicly available memes screened for harmful content and acknowledged subjectivity in culturally situated humor interpretation.
- All memes were publicly available and screened to remove NSFW or offensive content.
- Humor interpretation remains subjective and culturally situated, and explanations may reflect annotator perspectives.
- All participating annotators were paid well above their country-specific minimum wages.
A.1 Annotation Guidelines
MEMECULT-1K uses culturally informed, independent annotation procedures to produce standardized explanations, context notes, and structured dataset records. Annotator agreement was strong despite the open-ended nature of the task.
- Annotation procedure: Each meme received three independent English explanations from annotators familiar with at least one represented cultural region.Annotators were fluent in English and followed standardized guidelines for culturally sensitive explanations.
- Annotation procedure: Annotators were instructed to write short, objective, grammatically correct explanations without offensive quotations, personal commentary, or political interpretation.The protocol emphasized neutral and culturally balanced phrasing.
- Context notes: Each meme also received a short cultural context note identifying background markers such as cricket, film tropes, or local idioms influencing humor.Context notes were designed to summarize cultural or situational background in two to three sentences.
- Dataset format: Dataset records store the meme filename, context note, three independent explanations, and anonymized annotator IDs in CSV format.The format supports one meme per row and preserves the provenance of each annotation.
- Annotation quality: 0.78 mean pairwise SBERT cosine similarity indicates strong semantic agreement among the three explanations per meme.Agreement was measured across all samples because the explanations were open-ended natural-language texts rather than categorical labels.
A.2 Example Annotations and Contexts
The paper presents representative dataset examples, standardized VLM prompting and evaluation procedures, cross-judge robustness checks, and error-category definitions. These materials clarify how context-aware explanations and model failures are assessed.
- Experimental setup: Models were prompted to produce one self-contained sentence explaining the underlying meaning or irony without describing visual details or using generic meme terminology.The instruction prohibited words such as “meme,” “image,” “picture,” “joke,” and “pun.”
- Experimental setup: Context-aware evaluation prepended the cultural note, used deterministic decoding at temperature = 0, and compared outputs against three reference explanations.Scores were averaged over references and computed separately for each language split.
- Example annotations: Table 7 presents Bengali, Hindi, and English memes alongside their cultural context notes and human explanations.The examples illustrate the multilingual annotation structure used in the benchmark.
- Cross-judge robustness: Human ratings closely matched Gemini judging on Gemini systems, differing by at most 0.06 points in the context-aware setting.For example, Flash received 4.18 from humans versus 4.23 from the Gemini judge.
- Cross-judge robustness: Context gains were positive for every model under all three evaluators, while model rankings showed strong agreement across judges.Spearman ρ values were 0.98 for human–Qwen3-32B, 0.92 for human–Gemini, and 0.93 between judges.
- Error analysis: The error taxonomy distinguishes cultural knowledge gaps, entity misidentification, linguistic failure, literal interpretation, incomplete humor, hallucination, and visual grounding failure.Two annotators assigned one most-salient category to each incorrect or incomplete output, resolving disagreements through discussion.