PaperlayerView exact revision on arXiv

Source-linked AI summary

BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset

Md. Sadman Sakib, Zisan Mahmud, Md. Fahim Arefin, Md Fahim

arXiv:2609.08029v1cs.CLcs.CV

TL;DR

Culturally grounded, metaphor-rich reasoning remains insufficiently studied in multimodal benchmarks. BanglaMemeX provides a 3,000-meme Bangla dataset with multidimensional labels and metaphor explanations, and evaluates modern VLMs. The results show that stronger label prediction does not guarantee faithful metaphor-grounded explanations.

  • Problem

    Many multimodal benchmarks focus on literal tasks, leaving culturally grounded reasoning about metaphor-rich content less studied.

    Source: §1 · p. 1

  • Method

    The paper introduces 3,000 Bangla memes with multidimensional labels and human-written textual and visual metaphor explanations, then evaluates modern VLMs using prompting and parameter-efficient fine-tuning.

    Source: §1 · pp. 1–2

  • Results

    Across prompting, LLM Council reasoning, and LoRA adaptation, strong label prediction does not guarantee faithful metaphor-grounded explanations.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Explanation quality remains fragile for visual metaphors, highlighting the need for supervision and evaluation.

    Source: §6 · p. 9

    BanglaMemeX covers Bangla memes from public online social media and may not capture the full range of dialectal, regional, or community-specific humor and metaphor.

    Source: §7 · p. 9

Abstract

from arXiv · show

Vision Language Models have achieved strong performance on multimodal benchmarks, yet their ability to reason about culturally grounded and metaphor-rich content remains insufficiently studied. Internet memes present a challenging setting where meaning emerges from implicit interactions between image, overlaid text, sarcasm, and shared socio-cultural knowledge rather than literal visual recognition. This challenge is amplified in low-resource languages such as Bangla, where code-mixing, stylized scripts, and culturally specific symbolism introduce substantial distribution shift. In this work, we introduce BanglaMemeX, a culturally grounded multimodal benchmark comprising 3,000 Bangla memes annotated with multi-dimensional labels (humor, sarcasm, offensiveness, motivational intent, and overall sentiment) and human-written explanations that explicitly describe textual and visual metaphors. We systematically evaluate modern VLMs on both classification and explanation generation, revealing that current models struggle to interpret implicit cultural cues despite reasonable surface-level accuracy. Our results highlight the need for culturally-aware multimodal systems capable of grounded reasoning under linguistic and cultural distribution shift.

1 Introduction

Existing multimodal benchmarks and Bangla meme datasets largely emphasize literal understanding or categorical prediction, leaving culturally grounded metaphor reasoning and explanations less directly evaluated. BanglaMemeX addresses this gap with a 3,000-meme dataset and systematic VLM evaluation.

1 Introduction

  • Meaning in memes often depends on metaphor, sarcasm, irony, and shared cultural context, so categorical labels alone provide only a partial view of interpretation.Explanations can help analyze model decisions in socially sensitive domains.Source: §1 · p. 1
  • Most existing Bangla meme datasets emphasize categorical prediction and do not explicitly evaluate whether models interpret metaphor and reasoning correctly.BanMiMe is limited to misogyny-specific cases, while other cited datasets focus on sentiment, hate, abuse, or contextual classification.Source: §1 · p. 1
  • BanglaMemeX contains 3,000 Bangla memes labeled for humor, sarcasm, offensiveness, motivational intent, and overall sentiment, with human-written explanations of textual and visual metaphors.The explanations support evaluation beyond classification accuracy and analysis of model reasoning.Source: §1 · p. 1
  • The paper evaluates modern VLMs with zero-shot prompting, chain-of-thought reasoning, and parameter-efficient fine-tuning, assessing explanations with BERTScore and LLM-as-a-judge evaluation.It also uses structured LLM Council prompting and explores LoRA adaptation for robustness under cultural and linguistic variation.Source: §1 · p. 2
  • The work contributes a culturally grounded dataset with explanation-level annotations and a systematic evaluation of VLM interpretation under cultural and linguistic shift.The evaluation examines whether models interpret metaphoric Bangla memes and how explanation supervision influences reasoning quality.Source: §1 · p. 2

2 Related Work

Prior vision-language evaluation has largely emphasized literal meaning and high-resource languages, while Bangla meme benchmarks have mostly assessed category prediction. BanglaMemeX combines multidimensional labels with textual and visual metaphor annotations and human-written explanations.

Vision Language Models

  • Vision-language models perform effectively on captioning, visual question answering, and instruction-following benchmarks, but culturally nuanced, metaphor-rich content remains less explored.Most evaluation settings emphasize literal semantics and are primarily developed using high-resource languages.Source: §2 · p. 2

Multimodal Meme Understanding in Bangla

  • Bangla meme datasets cover sentiment, hate, abuse, misogyny, humor, sarcasm, and code-mixed text, but most evaluate category prediction rather than explanation alignment and reasoning consistency.Some datasets include structured annotations such as metaphor localization, while BanglaMemeX combines multidimensional labels with textual and visual metaphor annotations and human-written explanations.Source: §2 · p. 2
  • BanglaMemeX extends prior Bangla meme research by supporting evaluation of interpretation accuracy and text-image-metaphor reasoning.Its task focus is culturally situated meme interpretation, unlike the cited implication-level benchmarks’ broader vision-language formulations.Source: §2 · pp. 2–3

3 Dataset

BanglaMemeX is a 3,000-meme dataset with multidimensional communicative labels, metaphor annotations, and explanations. Its construction combines filtering and manual annotation with human and semantic checks of annotation quality.

3 Dataset

  • BanglaMemeX contains 3,000 Bangla multimodal memes annotated for humor, sarcasm, offensiveness, motivational intent, overall sentiment, and explanation-level supervision.Figure 1 depicts the dataset construction workflow.Source: §3 · p. 3

3.1 Data Collection and Annotation

  • The collection began with 105,835 publicly accessible memes from Facebook, Instagram, Reddit, and X, then removed duplicates, near-duplicates, and images with third-party watermarks.After filtering, the final dataset contains 3,000 memes; all textual content was manually transcribed because OCR often struggles with stylized Bangla, mixed scripts, and low-resolution overlays.Source: §3.1 · p. 3
  • Three native Bangla-speaking annotators assigned communicative labels, described textual and visual metaphors, and wrote explanations linking them to cultural meaning or intent.For visual metaphors, annotators selected image regions using Segment Anything Model segmentations and described each region’s symbolic role.Source: §3.1 · p. 3

3.2 Data Validation

  • The dataset uses stratified multi-label sampling for train/test division and measures inter-annotator agreement with Fleiss’ κ using a one-vs-rest formulation for multi-class labels.Human evaluation rated correctness, clarity, and cultural appropriateness; annotations averaging below 3.5 were revised and re-annotated.Source: §3.2 · p. 4
  • BERTScore F1 agreement was averaged across accepted annotations, while complementary overlap metrics were ROUGE-1 F1 85.9, ROUGE-2 F1 50.8, and ROUGE-L F1 84.7.Table 4 reports human evaluation and semantic similarity for textual metaphor descriptions, visual metaphor explanations, and final multimodal explanations.Source: §3.2 · p. 4, Table 4 · p. 5
  • Figure 2 illustrates one instance’s categorical labels, textual and visual metaphor descriptions, and corresponding human-written explanation.Dataset statistics, label distributions, and explanation lengths are reported in Table 2, with lengths measured in characters.Source: §3.2 · pp. 4–5, Table 2 · p. 4, Figure 2 · p. 5

4 Experiment Setup

The setup evaluates BanglaMemeX interpretation through prompt-based inference, a role-based LLM council, and LoRA fine-tuning. It combines multi-label prediction with metaphor-focused explanation generation and evaluates classification and explanation quality.

4 Experiment Setup

  • Three settings evaluate culturally grounded meme interpretation: prompt-based inference, a role-based LLM Council, and LoRA fine-tuning.Figure 3 illustrates the overall experimental pipeline.Source: §4 · p. 5, Figure 3 · p. 6

4.1 Problem Formulation

  • Each meme is used to predict labels for five communicative categories and generate an explanation of interactions between visual elements, textual cues, and implied metaphors.The formulation combines multi-label classification and explanation generation.Source: §4.1 · p. 5

4.2 Prompt-Based Inference

  • Prompt-based inference compares zero-shot predictions with chain-of-thought prompting that elicits visual, textual, and metaphor reasoning before the final output.Both strategies use structured prompts without parameter updates.Source: §4.2 · p. 5

4.3 Role-Based LLM Council

  • The LLM Council uses two explainers, three critics, and one synthesizer to refine interpretations through sequential synthesis rather than majority voting.Critics assess grounding, cultural alignment, and interpretive gaps before the synthesizer reconciles disagreements.Source: §4.3 · p. 5

4.4 Parameter-Efficient Fine-Tuning

  • LoRA fine-tuning adds trainable low-rank matrices to selected projection layers while keeping pretrained weights fixed, and training jointly optimizes classification and explanation losses.The low-rank matrices have rank r much smaller than the dimensions of the weight matrix.Source: §4.4 · p. 6

4.5 Evaluation Metrics

  • The evaluation combines multi-label metrics across five categories with BERTScore, LLM-as-a-judge assessment, and human ratings of explanation quality.Explanation assessment considers textual grounding, cultural alignment, and metaphor interpretation.Source: §4.5 · p. 6

4.6 Implementation Details

  • Implementation uses five training epochs with rank r = 16, scaling factor α = 32, and dropout 0.05; images are processed directly without external OCR.Experiments use predefined train/test splits and fixed random seeds.Source: §4.6 · p. 6

5 Results Analysis

Across settings, label accuracy and metaphor-faithful explanations diverge: closed-source models lead on some zero-shot labels, while visual metaphors remain harder than textual ones and adaptation can trade grounding for accuracy.

Closed-source vs. open-source models

  • Claude-Opus-4.5 leads zero-shot sarcasm (Sarc=68.50), offensiveness (Offe=66.10), and overall sentiment (Over=55.80); GPT-5.2 leads humor (Hum=81.10), and Gemini-3 Flash leads motivational intent (Moti=95.07).Motivational intent is the easiest label, while sarcasm remains difficult because it requires interpreting implicit contrast, irony, and culturally specific framing.Source: §5 · p. 6, Table 5 · p. 7
  • BERTScore remains relatively high across models, but sharp LAVE variation shows that plausible, fluent explanations do not always capture the intended metaphor.Human evaluation supports this observation.Source: §5 · p. 7, Table 5 · p. 7

Do reasoning prompts help?

  • CoT has modest, model-dependent aggregate effects: GPT-5.2 Avg rises from 60.59 to 61.01, while Gemini-3 Flash falls from 60.55 to 58.81.Sarcasm can improve, but several models lose performance on overall sentiment under CoT.Source: §5 · p. 7

Effect of LLM Council and LoRA fine-tuning

  • The LLM Council delivers the strongest explanation quality, but its broader, more narratively appealing interpretations come with lower label-prediction accuracy.Those interpretations can diverge from the classification rubric’s strict decision boundaries.Source: §5 · p. 7, Table 5 · p. 7

Why visual metaphors remain hard

  • The LLM Council achieves the strongest overall metaphor performance (Textual LAVE 77.87; Image LAVE 43.14), yet visual metaphor understanding remains weaker across settings.The Council reduces the textual–visual gap to some extent through iterative explanation, critique, and synthesis.Source: Table 6 · p. 8, §5 · p. 8, Figure 4 · p. 8

Effect of LLM Council and LoRA fine-tuning

  • Qwen3-VL-8B’s classification Avg rises from 54.56 to 73.02 after LoRA, while Textual LAVE falls from 32.65 to 18.47 and Image LAVE from 12.11 to 4.67.The results suggest label-predictive shortcuts may improve label prediction without strengthening metaphor grounding or faithful explanations.Source: §5 · pp. 7–8, Table 6 · p. 8, Figure 4 · p. 8

6 Conclusion

BanglaMemeX is a culturally grounded benchmark for evaluating Bangla meme interpretation and explanation faithfulness. Across evaluated prompting and adaptation approaches, classification strength does not guarantee faithful metaphor-grounded explanations.

6 Conclusion

  • BanglaMemeX jointly evaluates multi-label interpretation and explanation faithfulness for metaphor-rich Bangla memes, with explanation-level annotations.The study systematically evaluates modern vision–language models under cultural and linguistic shift, including whether they interpret metaphoric memes and how explanation supervision influences reasoning quality.Source: §6 · pp. 8–9
  • Across prompting, LLM Council reasoning, and LoRA adaptation, strong label prediction does not guarantee faithful reasoning; visual-metaphor explanation quality remains fragile.Closed-source models provide strong zero-shot baselines, while LoRA boosts open-source classification performance.Source: §6 · pp. 8–9

7 Limitations

The study’s evidence is bounded by its Bangla-only, 3,000-sample benchmark and selected model set.

7 Limitations

  • BanglaMemeX covers 3,000 Bangla memes from public online platforms, so it may miss dialectal, regional, and community-specific humor and metaphor.The study evaluates selected models on one dataset; its findings should not be taken as evidence of generalization to all multimodal or cross-cultural settings.Source: §7 · p. 9

8 Ethical Considerations and Potential Risks

The study treats public-meme content and annotation judgments as sensitive, and describes limits on how the benchmark should be used. AI tools supported specific auxiliary tasks, while human annotators made the final selections and wrote the annotations.

8 Ethical Considerations and Potential Risks

  • Public memes may contain socially sensitive, offensive, or polarizing expressions; annotations involve subjective judgments, and the benchmark is intended for research rather than individual or community profiling.The authors recommend research and evaluation use, not sole reliance for automated moderation or other high-stakes decisions without human oversight.Source: §8 · p. 9

8.1 Use of AI Tools

  • SAM generated region-level segment proposals as aids, but human annotators made all final visual-metaphor selections and interpretations and wrote all annotations.AI models did not automatically assign labels or generate explanations during dataset construction; BERTScore was used post hoc for quality analysis.Source: §8.1 · p. 9

A Data Collection Process

This section describes the collection and filtering of BanglaMemeX memes and their annotation by culturally informed annotators using structured guidance and a SAM-assisted workflow. It covers annotation criteria and reconciliation of categorical labels, as well as annotator safeguards and limitations in dataset coverage.

A Data Collection Process

  • Collection covered public Bangla meme communities on Facebook, X, Instagram and Reddit; deduplication and native-speaker screening removed repeats, non-Bangla or literal memes, and other unsuitable images.Despite multiple sources, coverage reflects the demographics, posting habits and moderation norms of sampled communities.Source: A Data Collection Process · pp. 11–12

B Dataset Annotation Guidelines

  • Annotators assigned one label per meme in five categories—Humor, Sarcasm, Offensive, Motivation and Overall Sentiment—based on dominant intent, using culturally grounded guidelines.All annotators were native Bangla speakers familiar with local cultural practices, public discourse and social-media conventions.Source: B Dataset Annotation Guidelines · p. 12

B.1 Annotator Recruitment and Qualification

  • Three native Bangla-speaking University of Dhaka linguistics undergraduates annotated the memes, calibrated on a shared 50-meme subset, and discussed disagreements to align interpretation.The authors note that future work could use a broader, more demographically diverse annotator pool.Source: §B.1 · pp. 12–13

B.2 Annotation Timeline and Session Protocol

  • Annotation of 3,000 memes was completed over three months using focused 45–50-minute labeling blocks followed by mandatory 10–15-minute breaks, with a soft cap of approximately 40–50 memes per annotator per session.Annotators could flag personally distressing memes and temporarily skip them; on days with a high concentration of offensive or hateful content, an additional 20–30-minute break was provided.Source: §B.2 · pp. 13–14

B.3 SAM-Assisted Visual Metaphor Annotation Pipeline

  • The SAM-assisted pipeline segmented meme regions, let annotators select visual metaphors, transcribe text and assign labels, then record an explanation linking both metaphors to cultural intent.The interface was designed to standardize metaphor-region granularity and provide spatial grounding for downstream localization evaluation.Source: §B.3 · pp. 14–15

Humor

  • Humor labels distinguish strong laughter from mild amusement and from serious, critical or informational memes not intended to amuse.Definitions point to exaggeration, absurd contrast and punchlines for strong humor, versus playfulness or irony for milder humor.Source: Humor · pp. 16–17

Sarcasm

  • Sarcasm labels distinguish overt ridicule or exaggerated mockery, mild irony, and direct communication without meaningful ironic inversion.Source: Sarcasm · p. 17

Offensiveness

  • Offensiveness assesses whether a meme contains insulting, degrading, harmful or discriminatory content, with the hateful-offensive category covering explicit hate or identity-based targeting.Source: Offensiveness · p. 18
  • Offensiveness spans hateful hostility, strong degrading attacks, mild disrespect, and non-hostile content, with examples illustrating the four intensity levels.The examples range from contempt toward an individual and irreverent dark humor to mild irritation and a proverb without an insulting target.Source: Offensiveness · pp. 18–19, Figure 8 · p. 19

Motivational Intent

  • Motivational intent distinguishes constructive encouragement, aspiration, hope, or resilience from content that is critical, sarcastic, observational, or discouraging.Examples classify a drug-addiction euphemism as not motivational and prayer portrayed as protection from hardship as motivational.Source: Motivational Intent · pp. 19–20

Overall Sentiment

  • Overall sentiment is classified on five levels: very positive, positive, neutral, negative, and very negative, according to the meme’s dominant emotional tone.The definitions range from strongly uplifting or hostile content to emotionally balanced observation.Source: Overall Sentiment · p. 20

B.6 Inter-Annotator Agreement

  • Agreement was measured with Fleiss’ Kappa across three annotators for each of the five categories; majority voting set labels, with unanimous disagreement resolved through discussion and consensus.The passage points to Table 7 for the category-specific agreement scores, but does not provide those values here.Source: §B.6 · p. 20

B.7 Explanation Annotation and Evaluation

This appendix specifies how human-written metaphor explanations are produced and checked, details prompts used for model evaluation, and illustrates recurring model failures on Bangla memes.

B.7 Explanation Annotation and Evaluation

  • Explanations are short Bangla sentences that identify implicit metaphors and relevant cultural context while avoiding unsupported literal image description.Source: §B.7 · pp. 20–21

Explanation Quality Evaluation

  • Three annotators score each explanation from 1–5; those averaging 3.5 or above are accepted, while lower-scoring items are collaboratively revised into evaluation references.The retained explanations serve as ground truth for evaluating faithfulness, metaphor understanding, and cultural alignment.Source: Explanation Quality Evaluation · p. 21

C Prompting

  • Prompts ask models to integrate visual and textual cues, identify metaphors, assess five dimensions, and generate Bangla explanations; an LLM Council adds critique and synthesis.Source: C Prompting · pp. 22–23

D Model Failure Examples

  • Examples show models missing underlying metaphors, hallucinating elements, and offering plausible but incorrect explanations despite occasionally correct labels.Source: D Model Failure Examples · p. 23

E Extended Classification Results and Error Analysis

The extended results show that fine-tuning improves open-source models’ classification F1, while stronger classification does not guarantee better explanations. Unimodal systems trail multimodal ones on average, and confusion analyses reveal persistent difficulty with rare or intense labels.

E Extended Classification Results and Error Analysis

  • F1 scores provide a more balanced view than accuracy when label distributions are uneven across meme categories.The section reports F1 across zero-shot, chain-of-thought, LLM Council, and LoRA settings.Source: E Extended Classification Results and… · p. 23
  • Claude-Opus-4.5 has the highest average F1 among prompting baselines, while Qwen3-VL-8B and Gemma-3-12B have the strongest average F1 among fine-tuned models; classification strength does not ensure better explanations.The section recommends evaluating both classification and explanation quality.Source: E Extended Classification Results and… · p. 23, Table 8 · p. 24

Unimodal Ablations

  • Average performance in every text-only and image-only setting falls below the corresponding multimodal systems, though some gaps are small.The ablations include prompting strategies and lightweight supervised classifiers.Source: Unimodal Ablations · pp. 23–24, Table 10 · p. 25, Table 11 · p. 25
  • 76.75 F1 for humor, 53.84 for sarcasm, 62.62 for offensiveness, 94.59 for motivation, and 62.10 for overall sentiment yield 69.98 mean classification F1.Sarcasm is the only dimension with an F1 in the 50s.Source: Unimodal Ablations · p. 24
  • The LLM Council preserves more intensity distinctions but often overpredicts strong labels, while all systems struggle with motivational and extreme sentiment classes.Fine-tuned models do better on common labels but often collapse minority and extreme categories into them.Source: Unimodal Ablations · pp. 24–25

F Reliability Check of Automatic OCR

Automatic OCR performs weakly on Bangla meme text, so the dataset uses manual transcription. The appendix also specifies its human explanation rubrics, evaluator independence, and score aggregation procedure.

F Reliability Check of Automatic OCR

  • Automatic OCR remains weak across models; lower lexical-overlap scores accompany moderate BERTScore, motivating manual transcription to preserve Bangla meme text accurately.Manual extraction accommodates stylized typography, contextual placement, informal spelling, mixed scripts, and cultural conventions.Source: F Reliability Check of Automatic OCR · p. 25, Table 16 · p. 27

G Human Evaluation Rubrics

  • Three independent evaluators score explanations using three 5-point Likert rubrics, with raw averages normalized to the [0, 1] range.The normalized percentage scores are reported in Tables 5 and 8.Source: G Human Evaluation Rubrics · pp. 25–26

G.1 Evaluator Recruitment, Demographics, and Independence from Dataset Annotators

  • The evaluators are separate from the dataset annotators, are blind to model identity and prompting strategy, and do not include paper authors.The evaluators are native Bangla speakers recruited independently of the annotation team.Source: §G.1 · pp. 26, 28

G.2 HumEval: Human Evaluation on Overall Explanation

  • HumEval scores whether an explanation captures the meme’s overall meaning, including its text, image, metaphor, tone, and cultural context.Source: §G.2 · p. 28

G.3 HumTexMetEval: Human Evaluation on Textual Metaphor Explanation

  • HumTexMetEval assesses whether an explanation identifies and explains the metaphor conveyed through the meme text, including its figurative meaning and cultural-linguistic context.Source: §G.3 · p. 28

G.4 HumImgMetEval: Human Evaluation on Image Metaphor Explanation

  • HumImgMetEval assesses whether an explanation identifies and explains image-based metaphor, visual symbols, and relevant cultural or contextual cues.Source: §G.4 · p. 28

G.5 Human Evaluation Score Calculation

  • The scoring procedure averages evaluator scores per sample, averages those values across evaluated samples, then min-max normalizes the raw score for a 5-point scale.K is the number of evaluators and N is the number of evaluated samples.Source: §G.5 · p. 28

H Qualitative Error Analysis of the LLM Council

The LLM Council can recognize a meme’s broad metaphor yet misjudge its pragmatic force, with errors sometimes amplified by the critic and retained by the synthesizer. The examples emphasize that culturally grounded interpretation also requires judging humor’s target and intensity.

H Qualitative Error Analysis of the LLM Council

  • The Council often recognizes a broad metaphor but misjudges pragmatic force; the critic can intensify a plausible reading, and the synthesizer may retain that shift.The error analysis traces whether mistakes emerge or persist across the explainer, critic, and final synthesizer.Source: H Qualitative Error Analysis of the… · p. 28, Figure 15 · p. 35, Figure 16 · p. 36
  • In culturally grounded Bangla memes, correct metaphor recognition alone is insufficient because models must also infer the culturally appropriate intensity and target of humor.Local religious references, social norms, and everyday habits can make distinctions among humor, sarcasm, offensiveness, and sentiment subtle.Source: H Qualitative Error Analysis of the… · p. 28
  • In the park-romance example, the explainer reads the Jannah reference and Shaitan’s reaction as irony but misses the slight offensiveness and negative moral-judgment tone.Its classifications instead mark the meme not offensive and neutral.Source: H Qualitative Error Analysis of the… · p. 36
Loading 2609.08029v1…