PaperlayerView exact revision on arXiv

Source-linked AI summary

Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning

Haihan Li, Haihao Li, Zhenfei Xu, Jize Qian

arXiv:2608.26866v1cs.CV

TL;DR

LVLMs have strong single-image capabilities, but it remains unclear whether they perform sequence-aware reasoning over Chinese multi-panel memes. CMPM introduces a 1,214-sample benchmark with order-sensitive tasks and human-rated explanations, finding sharp degradation under shuffling and higher Task 2 preferences for Gemini 3.1 Pro and GPT-5.5 than open models.

  • Problem

    Existing evidence does not establish whether LVLMs can reason over constrained panel order in structured Chinese memes rather than relying on local visual or textual cues.

  • Method

    CMPM combines 1,214 annotated Chinese multi-panel memes with Task 1 structure and order evaluation, context ablation, and Task 2 human ratings of Chinese explanations.

  • Results

    Shuffling reveals order blindness; Gemini 3.1 Pro reaches 75.2% on the primary condition, while Task 2 favors Gemini 3.1 Pro and GPT-5.5 over open models.

  • Takeaways & Limitations

    Canonical-display accuracy alone is insufficient evidence of order understanding, and comment context provides only heterogeneous gains.

  • Takeaways & Limitations

    CMPM focuses on Chinese multi-panel social memes, while results depend on frozen checkpoints, prompts, decoding settings, and future model updates.

Abstract

from arXiv · show

Many multimodal tasks depend on how visual elements are ordered and composed, not only on recognizing them in isolation. Internet memes are a compact case of this problem: their punchline often depends on a constrained reading order and cross-panel visual--textual cues. While large vision-language models (LVLMs) show strong performance on single-image understanding, it remains unclear whether they can perform sequence-aware reasoning over structured meme layouts, especially in Chinese social media. We introduce CMPM, a Chinese Multi-Panel Meme benchmark with 1,214 annotated samples covering five structural types, ordering dependency, panel-order constraints, and optional comment context. We formulate a two-layer evaluation: Task1 probes structure typing and order-sensitive panel sequencing (with a context ablation setting), and Task2 evaluates Chinese meme explanation generation with human ratings on five 1-3 Likert dimensions (visual, panel, humor, context, and faithfulness). We benchmark five representative LVLMs under a unified protocol. Results indicate that canonical-display accuracy is not by itself evidence of order understanding: the primary shuffled condition produces a sharp accuracy drop, revealing a persistent gap in order-sensitive multimodal reasoning. Task2 preferences place Gemini 3.1 Pro and GPT-5.5 above the open models, while comment context yields only a small and mixed Core4 gain. Code and data will be released upon acceptance.

1 Introduction

Multimodal meme understanding depends on panel order and cross-panel relations, yet existing LVLM evaluations often emphasize local cues. CMPM addresses this gap with a Chinese multi-panel benchmark and two-layer evaluation protocol.

  • Reordering panels can destroy a punchline, invert a contrast, or break a conversational turn.
  • Existing meme benchmarks often test classification or single-image explanation without isolating setup-to-punchline dependencies or valid reading order.
  • CMPM evaluates structure and order recovery in Task 1, then Chinese meme explanations using five human-rated 1–3 dimensions in Task 2.The dimensions are visual correctness, panel coherence, humor comprehension, context handling, and faithfulness.
  • 1,214 samples form CMPM, with annotations for structure, ordering dependency, order groups, and optional context.
  • Shuffled display causes a sharp accuracy drop, while comment context produces only a small average gain with helpful and harmful instance-level effects.

2 Related Work

Prior meme and multimodal resources cover contextual interpretation, text-only Chinese memes, comics, documents, or general LVLM capabilities, but rarely isolate Chinese multi-panel reading-order recovery. CMPM positions itself as a focused sequence-aware benchmark with human explanation evaluation.

  • Earlier meme corpora emphasize hate speech, offense, sentiment, or contextual understanding rather than Chinese multi-panel sequence structure.
  • CHIME evaluates phrase-based Chinese Internet memes textually, while MemeReaCon uses post text and comments without making panel order the primary variable.
  • Comic benchmarks study closure, reading order, reordering, or image sequences, but their settings differ from meme-specific controlled shuffling.
  • General LVLM suites and document benchmarks stress perception and grounding but rarely isolate multi-panel meme order as a controlled factor.
  • CMPM contributes sequence-aware understanding of culturally grounded visual memes and complements automatic metrics with human-rated Chinese explanations.

3 Problem Definition

CMPM models each meme as an ordered panel sequence with optional context and evaluates structure typing, constrained order recovery, and explanation quality under controlled presentation conditions.

  • A meme is represented as P = (p1, . . . , pn), with optional context C, type and ordering-dependency labels, and order groups defining precedence constraints.A canonical order is one valid linear extension of those constraints.
  • Task 1 predicts one of five structure types from canonical-order panels with comments hidden, reporting accuracy and macro-/weighted-F1.
  • Order-sensitive sequencing displays strong-order, non-parallel samples under a fixed shuffle and accepts any valid group-aware linear extension.
  • CAS crosses correct versus shuffled displays with no-context versus comment-present settings to measure whether comments help or hurt order recovery.
  • Task 2 generates Chinese explanations scored blindly on visual correctness, panel coherence, humor comprehension, context handling, and faithfulness using 1–3 Likert ratings.

4 The CMPM Dataset

CMPM contains 1,214 Chinese multi-panel memes collected from major social platforms and annotated for structure, ordering, order groups, and context relations. Annotation agreement is high for structure and ordering, but lower for contextual relations.

  • 4.1 Collection and Inclusion: 1,214 memes from Weibo, Bilibili, Xiaohongshu, Tieba, and Douyin contain cropped panels and optional textual context.Single-image cases, including complex collages, are excluded from the main evaluation scope.
  • 4.2 Annotation Schema: CMPM defines five structure types: narrative, progressive, conversational, comparison, and parallel.
  • 4.2 Annotation Schema: Ordering dependency is strong when shuffling makes a meme incomprehensible or removes its punchline, and weak otherwise.
  • 4.2 Annotation Schema: Order groups encode block precedence while permitting permutations within a group.
  • 4.3 Corpus Statistics: 322 samples provide comment or danmaku context for CAS, while 991 have any non-empty textual field and are not equivalent to comment availability.
  • 4.4 Annotation Agreement: Fleiss κ reaches 0.898 for structure type and 0.875 for ordering dependency, whereas context–meme relation agreement is 0.499.

5 Evaluation Protocol

The study evaluates five LVLMs under shared, model-agnostic manifests spanning structure typing, shuffled order recovery, and comment-context ablation, without fine-tuning on CMPM.

  • Models: Five LVLMs are evaluated: InternVL3.5-8B, Qwen3.5-9B, GLM-4.1V-9B-Thinking, GPT-5.5, and Gemini 3.1 Pro.The benchmark includes three open-source and two closed-source systems.
  • Task 1: Type prediction uses 1,214 canonical-order memes with comments hidden.Each request asks the model to predict the meme’s structural type.
  • Task 1: Order prediction uses 868 strong, non-parallel samples with valid order groups under shuffled display and no comments as the primary condition.Hard cases with at least seven panels are retained separately.
  • Context ablation: CAS evaluates a 2 × 2 display-by-context grid on 252 comment-available order-pool memes.The conditions vary correct versus shuffled display and absent versus available comments.
  • Metrics: Ordering uses group-aware logical-order accuracy, while typing reports accuracy, macro-F1, weighted-F1, and parse rate.Task 2 explanations receive blind human ratings on five 1–3 dimensions.

6 Experiments

Experiments show strong differences between closed- and open-source models, persistent shuffled-order difficulty, semantic structure errors, and limited, inconsistent benefits from comments.

  • 6.1 Task 1: Primary Order Recovery: Gemini 3.1 Pro reaches 75.2% and GPT-5.5 57.8% on primary shuffled order recovery, exceeding Qwen3.5-9B at 27.4%.GLM-4.1V-Thinking reaches 15.6% and InternVL3.5-8B 6.1%; the metric is group-aware logical-order accuracy.
  • 6.1 Task 1: Primary Order Recovery: Gemini’s shuffled accuracy remains below correct-display accuracy: 73.0% / 71.3% versus 90.5% / 94.8%.The gap persists across the reported CAS cells.
  • 6.2 Task 1: Structure Typing: GPT-5.5 reaches 76.3% accuracy / 74.9 macro-F1 in structure typing, while Gemini 3.1 Pro reaches 69.7% / 67.9.Open-source models remain in the mid-50% accuracy range.
  • 6.2 Task 1: Structure Typing: Parse rates above 99.7% indicate semantic structure recognition, rather than output validity, is the typing bottleneck.Errors are concentrated near narrative–conversational and parallel–comparison boundaries.
  • 6.3 Context Ablation (CAS): Under shuffled CAS displays, GPT-5.5 reaches approximately 51% and Gemini 3.1 Pro 71–73%, while comments do not close the display gap.InternVL improves from 4.8% to 6.8%, Qwen remains near 23%, and Gemini declines slightly with comments.
  • 6.1 Task 1: Primary Order Recovery: Long strips sharply reduce primary order accuracy: GPT-5.5 falls from 59.1% to 16.0%, and Gemini 3.1 Pro from 76.7% to 24.0%.Open models remain near floor on hard items.
  • Task 2: Pairwise Task 2 rankings place Gemini 3.1 Pro and GPT-5.5 above open models, recovering the broad tiers in scalar ratings.The comparison is based on five-dimension human evaluation and separate pairwise outcomes.

7 Analysis and Discussion

Analysis identifies order blindness as the dominant failure mode, with errors at discourse and structural boundaries; comments are not a universal remedy, and the benchmark has defined scope limits.

  • 7.1 Order blindness: Open models often produce plausible left-to-right sequences after permutation while ignoring causal, contrastive, or dialogue constraints.Closed-source models narrow but do not eliminate the shuffled-order gap.
  • 7.2 Structure typing: Structure errors cluster at narrative–conversational and parallel–comparison boundaries, with progressive cases often absorbed into those classes.The pattern reflects difficulty encoding distinctions among layouts.
  • 7.3 Context: Comments help when restating a punchline but can mislead when ironic, referential, or unrelated to panel order.Visual sequence reasoning therefore remains necessary.
  • 7.1 Order blindness: Contrast swaps, reply-turn misordering, and punchline-first choices are recurring order-recovery failures.These cases require discourse relations beyond recognizing individual panels.
  • 7.5 Limitations: CMPM’s Chinese multi-panel social-meme focus may not generalize to other cultures or single-image templates.Results are also sensitive to prompts, decoding settings, and future model updates.

8 Conclusion

CMPM evaluates order-sensitive reasoning in Chinese multi-panel memes and shows that shuffled displays expose order blindness, while structure typing and comment benefits remain uneven.

  • 8 Conclusion: CMPM contains 1,214 samples for evaluating order-sensitive reasoning in Chinese multi-panel memes.Its evaluation combines structural typing, order recovery, and explanation assessment.
  • 8 Conclusion: Gemini 3.1 Pro reaches 75.2% on the primary shuffled condition, while open models remain lower.The conclusion characterizes this pattern as evidence of order blindness under shuffling.
  • 8 Conclusion: Structure typing is more reliable than boundary-sensitive ordering, and comment gains are heterogeneous.Task 2 preferences favor Gemini 3.1 Pro and GPT-5.5, with moderate agreement cautioning against a single score.
Loading 2608.26866v1…