Source-linked AI summary
MemeBridge: A Dataset for Benchmarking and Mitigating the Bidirectional Cultural Gap in Meme Interpretation
Hangxiao Zhu, Suliu Qin, Zhuoyan Li, Ming Jiang, Yu Zhang, Meng Xia
TL;DR
MemeBridge targets the limited cultural context in existing meme benchmarks and studies cross-cultural interpretation through a curated, bidirectional dataset of U.S.-originated memes. Its experiments reveal substantial interpretation gaps and show that culturally grounded evaluation and fine-tuning improve the basis for cross-cultural meme understanding.
Problem
Existing meme benchmarks do not explicitly capture the cultural context required for accurate cross-cultural meme interpretation, while LLMs face cultural and multimodal understanding limitations.
Method
MemeBridge combines native U.S. meme contributions, explanations, anticipated cross-cultural misunderstandings, sentiment and emotion annotations, and LLM evaluation and fine-tuning.
Results
58.8% accuracy was achieved for Chinese participants’ meme explanations, with lower sentiment and emotion labeling accuracies, and U.S.-anticipated misunderstandings often failed to match Chinese participants’ misconceptions.
Takeaways & Limitations
The bidirectional framing supports more nuanced benchmarking of cross-cultural meme comprehension, while MemeBridge provides culturally grounded supervision for evaluating and training LLMs.
Takeaways & Limitations
The dataset is relatively small because crowdsourcing rich meme annotations is costly and logistically complex, potentially causing overfitting during fine-tuning.
Abstract
from arXiv · showhide
Communicating across cultures is inherently challenging, especially through culturally dense and ambiguous formats like memes. While people expect large language models (LLMs) to hold promise for bridging such gaps, existing benchmark datasets often fail to capture the cultural context necessary for accurate interpretation. To address this, we introduce MemeBridge, a curated dataset centered on U.S.-originated memes, designed to capture two complementary perspectives: (1) how Chinese participants interpret these memes, and (2) how U.S. participants anticipate how people from other cultures might misunderstand them. Here, context refers to implicit cultural knowledge, including background beliefs, norms, and shared assumptions that shape meme comprehension. The dataset was constructed via a multi-stage crowdsourcing pipeline with rigorous validation, including human agreement checks and GPT-based classification verification. Each meme is annotated with sentiment, emotion, cultural significance, and knowledge type, providing rich supervision for downstream tasks. Notably, we observe that the anticipated misunderstandings from U.S. participants are often inaccurate, highlighting the asymmetries in cultural understanding and the challenges of adopting perspectives beyond one's own. This bidirectional framing, which focuses on both expression and perception, enables more nuanced benchmarking of cross-cultural comprehension. Our probing of multiple LLMs reveals that while models developed in different cultural contexts exhibit partial cross-cultural understanding, they often struggle with sophisticated interpretations. By contrast, fine-tuning with MemeBridge improves model performance, underscoring the value of culturally grounded resources for training and evaluating LLMs in globally diverse settings.
1 Introduction
MemeBridge addresses culturally grounded meme interpretation by framing misunderstanding bidirectionally and evaluating multimodal LLMs on U.S.-based memes. The study finds substantial human and model challenges, while dataset-based evaluation and fine-tuning support cross-cultural interpretation research.
- Memes rely on shared cultural references, societal trends, linguistic variation, and generational knowledge, making them vulnerable to cross-cultural misinterpretation.
- Global-platform exposure can produce unintended misunderstandings or misaligned social interactions when users lack familiarity with another culture’s memes.
- Existing meme datasets support general understanding tasks but do not explicitly address cross-cultural interpretation through bidirectional perspectives and culturally grounded annotations.
- MemeBridge contains native U.S.-contributed memes with explanations, potential cross-cultural misunderstandings, sentiment, and emotion annotations, and supports LLM evaluation and fine-tuning.
- 58.8% accuracy was achieved for Chinese participants’ meme explanations, compared with 45% for sentiment labeling and 48.9% for emotion labeling.
- The cultural gap is bidirectional, while evaluated LLMs showed comparable performance across development origins and responded substantially to explicit cultural-perspective instructions.
2 Related Work
Related work establishes progress in meme analysis, multimodal cultural adaptation, and culturally influenced inference, while motivating datasets designed for diverse cross-cultural meme understanding.
- Existing studies report cultural-symbol biases, regional performance differences, and difficulty reaching human-level cultural understanding in LLMs.
- Research on multimodal cultural adaptation examines how visual and textual information is interpreted across linguistic and cultural contexts.
- The related work motivates resources that reflect cultural diversity for evaluating cross-cultural meme understanding.
- Studies of culturally influenced inference investigate how cultural norms shape reasoning in natural-language inference and figurative-language understanding.
- Prior meme datasets support meme tracking, figurative-language identification, contextual explanation, captioning, and related understanding tasks.
3 MemeBridge Dataset Construction
MemeBridge was built through a staged crowdsourcing and validation pipeline that progressively refined U.S.-originated memes, annotations, and cross-cultural interpretations. The resulting dataset combines culturally grounded labels with topic and knowledge-type classifications supported by human agreement checks.
- Pipeline Overview: A three-stage pipeline collected, validated, and cross-culturally tested memes using U.S. and Chinese participants.The process included collection and cleaning, annotation validation, and Chinese participant interpretation tests.
- Stage 1: Data Collection: U.S. contributors supplied 1,000 memes with explanations, anticipated misunderstandings, sentiment labels, and emotion labels.Sentiment labels were positive, negative, or neutral; emotions included sarcastic, humorous, offensive, and motivational.
- Stage 1: Data Cleaning: Three researchers labeled 200 sampled data points for explanation and misunderstanding quality, enabling a BERT classifier to filter low-quality interpretations.Majority voting determined the labels used for classifier training.
- Stage 2: Data Validation: 79.2% agreement for explanation clarity and 67.1% for misunderstanding level accompanied filtering that left 621 memes in the final dataset.Agreement was also measured for cultural significance, sentiment, and emotion; memes with aggregated cultural significance below 3 were removed.
- Annotation Dimensions: GPT-4 classified memes by topic and required knowledge type, covering five topical categories and distinguishing cultural from general knowledge.The topic categories were Political Satire, Entertainment, Films and Cartoons, Pop Culture, and Social Events.
- Annotation Verification: Human-GPT-4o agreement reached Cohen’s Kappa values of 0.89 for topic classification and 0.85 for knowledge-type classification.The comparison used a random sample of 200 memes with author-provided labels.
4 Evaluating LLM’s Cross-Cultural Meme Understanding
The study evaluates four multimodal LLMs on U.S.-based meme interpretation, cultural-perspective prompting, and aggregate performance. Models show task-specific strengths, role-playing generally improves performance, and fine-tuning especially benefits sentiment and emotion classification.
- 4.1 Assessment on LLMs: The evaluation compares Qwen, GLM, LLaMA, and GPT with Chinese participants across multiple-choice, sentiment, and emotion tasks.The models are tested for human-like interpretation, sentiment and emotion detection, and adaptation to cultural perspectives.
- 4.1 Assessment on LLMs: GPT consistently outperforms Chinese participants in interpreting U.S. memes, while Qwen, GLM, and LLaMA underperform them on sentiment classification.Qwen and GLM outperform Chinese participants on emotion detection, whereas LLaMA performs worse than both other models and humans.
- 4.1 Assessment on LLMs: LLMs and Chinese participants more frequently select LLM-generated distractors than human-assumed distractors when answering incorrectly.The paper interprets this descriptive alignment cautiously because source bias, option position, verbosity, and implicit model cues may confound the pattern.
- 4.2 Detection of LLMs’ Potential Bias: Role-playing prompts improve performance across models relative to the default condition, with US-RP generally outperforming CN-RP and DEF.The study compares default, U.S. role-playing, and Chinese role-playing conditions using similarity, classification, and aggregate performance measures.
- 4.3 Fine-tuning: Fine-tuning produces substantial improvements in sentiment and emotion classification, particularly for models that initially perform poorly.Models with stronger prior task performance may show smaller gains, degradation, or diminishing returns due to overfitting.
5 Discussions
The discussion characterizes meme interpretation as a bidirectional cultural gap: Chinese participants struggle with U.S. memes, while U.S. participants often misjudge Chinese misunderstandings. LLMs can model some of these perspectives, but their adaptation remains uneven and culturally biased.
- 5 Discussions: Chinese participants exhibit relatively low accuracy across multiple-choice, sentiment, and emotion tasks, confirming one direction of the cultural gap.They are frequently outperformed by LLMs when interpreting U.S. memes.
- 5 Discussions: Chinese participants more often select LLM-generated distractors than human-assumed misunderstandings, indicating that U.S. assumptions do not always match their interpretations.The same pattern appears in LLM testing, suggesting that models can partly approximate Chinese participants’ thought processes.
- 5 Discussions: LLMs can help U.S. participants anticipate how shared content may be misinterpreted across cultures.This extends their potential use beyond helping Chinese participants understand U.S. memes.
- 5 Discussions: Explicit role-playing can significantly improve LLM performance on some metrics, but gains vary across tasks and models.The variation indicates that cultural adaptation remains limited by training data and pre-existing model understanding.
- 5 Discussions: LLMs perform better when prompted as native Chinese speakers than by default, but worse than when prompted as native U.S. speakers.The paper relates this asymmetry to English-heavy training data and alignment effects that may suppress cultural biases.
6 Limitations
The study’s limitations concern dataset scale, unvalidated cultural-direction reversal, restricted cultural coverage, and the subjective reliability of imagined misunderstandings.
- Dataset scale: The dataset is relatively small because collecting raw memes with rich annotations was costly and logistically complex.Data collection lasted approximately 40 days and cost around $2000; limited size may contribute to overfitting during fine-tuning.
- Unvalidated reversal: The study did not test the reversed direction in which Chinese participants provide memes and U.S. participants interpret them.Whether the same bidirectional difficulties appear in that setting remains open.
- Cultural scope: The dataset covers only Chinese and U.S. cultural contexts, excluding other linguistic and cultural backgrounds.Future work is proposed across broader cultural settings to examine whether similar bidirectional gaps occur elsewhere.
- Misunderstanding annotations: Imagined misunderstandings from U.S. participants often failed to match Chinese participants’ actual responses.Because misunderstanding is subjective and lacks objectively verifiable ground truth, one-shot prompts may produce unreliable elicitation.
A Participant Demographics
The appendix describes participant demographics and recruitment across three crowdsourcing stages for meme collection, validation, and cross-cultural testing.
- Recruitment: Stage 1 and Stage 2 participants were recruited through Prolific with U.S.-based screening and demographic diversity controls.Table 9 summarizes sex, ethnicity, and employment status for these stages.
- Recruitment: Stage 3 participants were recruited through university mailing lists and social media groups targeting Chinese international communities.All participants were paid $10 per hour.
- Stage 1: Stage 1 participants uploaded at least 10 U.S. memes and supplied cultural context, potential misunderstandings, sentiment, and emotion labels.The stage focused on initial data collection.
- Stage 2: Stage 2 participants validated meme interpretations and rated explanation and misunderstanding quality.This stage provided validation of the collected annotations.
- Stage 3: Stage 3 participants answered multiple-choice interpretation questions for U.S. memes and labeled sentiment and emotions.The stage measured cross-cultural meme interpretation.
E BERT Classifier Performance on Varying Training Sizes
The BERT-based classifier was evaluated on a 50-example held-out test set while varying the number of training examples.
- Results: Classifier performance improved consistently as the amount of training data increased.The reported trend supports evaluating the labeled set at multiple training sizes.
- Results: Performance reached near-perfect levels with 150 training samples, suggesting the labeled set was sufficient for quality filtering.This conclusion concerns the classifier’s filtering role rather than the broader meme-interpretation tasks.
F Additional Details on the Rewriting Process of Original Data
The appendix specifies GPT-4-assisted rewriting and classification procedures for standardizing original meme data while preserving cultural meaning and assigning structured labels.
- Rewriting process: The rewriting prompt standardizes explanations and misunderstandings without changing their meaning.It preserves original keywords, slang, and cultural references, adding only minimal context when needed.
- Rewriting process: Standardized outputs contain one explanation sentence and one potential-misunderstanding sentence with a prescribed opening.The misunderstanding field begins with “People might” or “Some viewers might.”
- Classification process: The classification prompt asks an expert meme analyst to assign knowledge-dependency and topic-category labels from specified options.Knowledge labels distinguish cultural-knowledge-dependent from general-knowledge-based memes, while topic labels include political satire, entertainment, films and cartoons, pop culture, and social events.
- Classification process: The topic classification output uses one label selected from political satire, entertainment, films and cartoons, pop culture, or social events.The prompt requires the result in a specified Topic output format.
H Example Data Instances
The dataset examples pair original explanations with GPT-4 interpretations and human-annotated misunderstandings. Structured prompts elicit explanations, possible cross-cultural misinterpretations, sentiment, and emotions from specified perspectives.
- Figure 7 presents three memes with an original explanation, a GPT-4 interpretation, and a human-annotated potential misunderstanding.
- The Chinese-view prompt assumes limited U.S. cultural knowledge and requests a plausible misinterpretation based on the image and Chinese cultural intuition.
- The prompts ask models to analyze American memes from neutral academic, American, or Chinese perspectives.
- For neutral academic or American perspectives, models describe plausible misunderstandings by non-U.S. audiences caused by cultural differences.
- Outputs also include sentiment labels and emotion categories such as sarcastic, humorous, motivational, and offensive.
K Performance Score Calculation
The performance score aligns classification accuracies with explanation-similarity metrics and normalizes them using expected baseline performance. Emotion accuracy accounts for the dataset’s multi-label structure and observed label counts.
- The performance score combines explanation similarity with multiple-choice, sentiment, and emotion classification performance.Similarity uses original and GPT-4-generated explanations, while classification includes AccMCQ, AccSent, and AccEmo.
- Expected accuracy is approximately 0.33 for three-option multiple-choice and sentiment classification tasks.
- Emotion accuracy treats a prediction as correct when the ground-truth labels form a subset of the predicted labels.
- The emotion-label calculation uses possible label sets, possible correct label sets, and the number of true labels for each meme.There are four possible emotion labels, and models may select between one and four labels.
L Emotion Label Count Distribution
The evaluation considers whether subset-based emotion scoring could reward predicting too many labels. Hidden criteria and observed response distributions indicate that label inflation did not occur in practice.
- Subset-based emotion scoring could allow degenerate solutions in which models predict all available emotion labels.
- The grading rule was hidden from models and human participants, reducing the likelihood of deliberate label inflation.
- Models and humans almost always selected one or two labels, occasionally three, and never all four.
- The expected number of selected labels ranged from 1.27 for humans to 1.63 for LLaMA, below the degenerate maximum of 4.
- Figure 8 indicates that fine-tuning may improve meme interpretation, provided overfitting does not occur.The authors recommend a sufficiently large dataset and their curation pipeline to mitigate overfitting risk.