PaperlayerView exact revision on arXiv

Source-linked AI summary

MMDS-Bench: Benchmarking Multimodal Large Language Models on Dynamic Stance in Social Media Interactions

Yuzhe Ding, Kang He, Li Zheng, Shengwu Zheng, Teng Shi, Fei Li, Chong Teng, Donghong Ji

arXiv:2608.30903v1cs.CL

TL;DR

Existing dynamic stance research mainly studies text-only or target-oriented relations, although social media interactions increasingly combine visual and textual cues. The paper introduces MMDS-Bench, a multimodal diagnostic benchmark with structured reasoning evaluation and challenge-factor annotations, and finds that current MLLMs often understand parent and reply messages separately but struggle to infer their stance relation.

  • Problem

    Existing dynamic stance research and benchmarks mainly focus on textual interactions or external targets, while multimodal social media requires inferring reply-to-parent relations from combined cues.

  • Method

    MMDS-Bench contains 3,482 multimodal parent–reply instances, an 800-instance diagnostic subset, seven stance labels, five challenge factors, and evaluations of 12 MLLMs with reference-grounded reasoning judges.

  • Results

    Current MLLMs often understand parents and replies separately but struggle with stance-relation inference, and performance declines as more challenge factors co-occur.

  • Takeaways & Limitations

    Multimodal dynamic stance understanding remains challenging beyond separate message comprehension, making relational inference a central diagnostic target.

  • Takeaways & Limitations

    MMDS-Bench covers image-based parent–reply interactions on X, excluding videos, audio, longer threads, other platforms, and broader linguistic settings.

Abstract

from arXiv · show

Dynamic stance classification models how a reply responds to its direct parent message, rather than how a post relates to a fixed topic. Existing work has mainly studied this problem in text-only settings, while social media interactions increasingly rely on images, screenshots, memes, reaction images, and cross-modal references. We introduce MMDS-Bench, a diagnostic benchmark for multimodal dynamic stance classification in social media parent-reply interactions. MMDS-Bench contains 3,482 multimodal instances annotated with a seven-label dynamic stance taxonomy, together with an 800-instance diagnostic subset that requires structured reasoning over parent understanding, reply understanding, and stance-relation inference. We further annotate each instance with five challenge factors covering multimodal fusion, parent framing, non-literal expression, interaction reasoning, and label-boundary ambiguity. We evaluate 12 closed-source and open-source multimodal large language models and propose a reference-grounded LLM-judge protocol for assessing reasoning quality. Results show that current MLLMs still struggle with multimodal dynamic stance understanding, especially in cases that require relational inference beyond separate parent and reply comprehension.

1 Introduction

Dynamic stance classification focuses on how a reply responds to its direct parent, but existing research and benchmarks have largely emphasized text-only or target-oriented settings. MMDS-Bench addresses this multimodal interactional gap with diagnostic data, structured reasoning evaluation, challenge annotations, and MLLM assessment.

  • Dynamic stance labels how a reply responds to its direct parent rather than relating a post to a fixed global topic.
  • Existing dynamic stance research has mainly focused on textual interactions, despite social media conversations increasingly combining text, images, screenshots, memes, and reaction images.
  • Multimodal dynamic stance requires recovering parent framing, interpreting reply intent, and inferring how textual and visual cues jointly shape their conversational relation.
  • MMDS-Bench contains 3,482 multimodal parent–reply instances, a seven-label taxonomy, and an 800-instance diagnostic subset requiring parent, reply, and stance-relation reasoning.
  • The benchmark annotates five challenge factors and evaluates 12 closed-source and open-source MLLMs using final-label metrics and reference-grounded reasoning assessment.

2 Related Work

Related work has expanded stance detection from target-oriented text to conversational and multimodal settings, but most benchmarks still define stance relative to an external target. MMDS-Bench instead centers the local relation between a multimodal reply and its direct parent.

  • Target-oriented stance detection models a single post’s relation to an external target, making it less suitable for local conversational interactions.
  • Conversational stance resources introduced deeper and multimodal settings, but many retain target-centered or claim-centered formulations.
  • Multimodal stance research studies textual and visual evidence, including cross-modal alignment, explicit reasoning, retrieval, agents, memes, and screenshots.
  • Multimodal dynamic stance differs by modeling the local relation between two multimodal messages rather than stance toward an external target, topic, or claim.

3 MMDS-Bench

MMDS-Bench is an image-based parent–reply benchmark with seven stance labels, full and diagnostic tasks, challenge-factor annotations, and quality-controlled multimodal instances. Its dataset design supports both final-label evaluation and analysis of message understanding versus stance-relation inference.

  • 3.1 Task Definition: MMDS-Bench determines how a reply responds to its direct parent using textual, visual, and cross-modal cues under a seven-label taxonomy.
  • 3.1 Task Definition: Task 1 evaluates final-label prediction, whereas Task 2 additionally requires parent_understanding, reply_understanding, and stance_reasoning outputs.
  • 3.2 Dataset Construction: The dataset retains 3,482 public X parent–reply pairs in which both messages contain at least one image, after removing missing, duplicated, corrupted, unavailable, or insufficient instances.
  • 3.2 Dataset Construction: Annotators assign stance labels and five binary challenge factors, while an 800-instance diagnostic subset is sampled approximately uniformly across active-factor counts.
  • 3.2 Dataset Construction: Seven-way stance annotation reached average Cohen’s κ of 0.74, while challenge-factor annotation averaged κ of 0.67 across five factors.
  • 3.2 Dataset Construction: Task 1 is dominated by Interaction Reasoning, Multimodal Fusion, and Non-Literal Reply, whereas Task 2 more strongly emphasizes Parent Framing and Label-Boundary Ambiguity.

4 Experimental Setup

The experiments compare 12 proprietary and open-source MLLMs under identical multimodal inputs across classification and structured reasoning tasks. Evaluation combines classification metrics with judge-based scores for parent understanding, reply understanding, and stance reasoning.

  • 4.1 Evaluated Models: The study evaluates 12 MLLMs spanning closed-source, open-source strong, and open-source efficient systems under the same multimodal setting.
  • 4.2 Evaluation: Task 1 reports Accuracy, Macro-F1, and per-class F1, using Macro-F1 as the primary metric because of label imbalance.
  • 4.2 Evaluation: Task 2 adds Parent Understanding Score, Reply Understanding Score, Stance Reasoning Score, Process Score, and Bottleneck Score to classification metrics.
  • 4.2 Evaluation: Task 2 reasoning is assessed with a reference-grounded LLM-judge protocol using three judges whose scores are averaged.
  • 4.2 Evaluation: Prompts provide parent text and images plus reply text and images, instructing models to judge stance toward the direct parent rather than a global topic.

5 Results and Analysis

MMDS-Bench results show strong but uneven MLLM performance, with reasoning-oriented models often ahead and stance-relation inference remaining the central weakness.

  • 5.1 Main Results: 79.93% Accuracy and 42.22% Macro-F1 make Gemini 2.5 Pro the strongest Task 1 model, while Claude Sonnet 4.6 leads Task 2 Macro-F1 at 51.99%.Gemini 2.5 Pro also achieves the highest Task 2 Accuracy at 72.25%.
  • 5.1 Main Results: Reasoning-oriented Qwen3-VL variants outperform their instruction-oriented counterparts on both tasks, especially on Task 2.The reported trend holds for both the 235B and 8B Qwen3-VL variants.
  • 5.1 Main Results: PUS and RUS consistently exceed SRS, indicating that models understand the parent and reply separately better than they infer their stance relation.Gemini 2.5 Pro scores 4.92 PUS, 4.89 RUS, and 4.33 SRS; Ministral3-8B-2512 scores 4.42, 4.12, and 2.80, respectively.
  • 5.2 Reasoning Quality: Correct Task 2 predictions have strong reasoning scores, whereas incorrect predictions retain relatively high message-level understanding but sharply lower SRS.Correct cases score 4.87 PUS, 4.83 RUS, and 4.81 SRS; incorrect cases score 4.51 PUS, 4.24 RUS, and 2.20 SRS.

EG ACP ECP EG

Errors increase with accumulated challenge factors and concentrate around interactional relation mapping, label ambiguity, and systematic label confusion; judge agreement is strongest for stance-relation scores.

  • Reasoning Quality: Incorrect predictions can still receive high reasoning scores, with 362 cases reaching 4.96 PUS, 4.96 RUS, and 4.49 SRS.The analysis links these cases to possible label-boundary ambiguity, label-mapping inconsistency, or mismatch between reasoning and final labels.
  • Challenge Count: Task 1 Accuracy falls from 76.35% with no challenge factors to 26.44% with all five, while Task 2 Accuracy falls from 77.31% to 27.30%.Task 2 SRS also decreases from 4.29 to 2.89 as challenge factors accumulate.
  • EG ACP ECP EG: Task 2 error gaps are largest for Interaction Reasoning at 14.10% and Label-Boundary Ambiguity at 13.65%.Task 1 has its largest gap for Interaction Reasoning at 8.28%, followed by Parent Framing and Non-Literal Reply.
  • Label Confusion Analysis: Disagree is recognized most reliably at 79.3%, but models overpredict it for gold Agree, Elaborates, and NA cases.Disagree receives 39.6%, 37.5%, and 58.7% of predictions in those respective gold-label groups.
  • Judge Consistency: Judge agreement is highest for SRS, averaging 0.92 Spearman correlation, compared with 0.68 for RUS and 0.62 for PUS.SRS is more explicitly anchored to the gold stance label and reference explanation.

6 Conclusion

The paper introduces MMDS-Bench as a multimodal dynamic stance benchmark and shows that current MLLMs often comprehend parent and reply messages separately without inferring their relation.

  • 6 Conclusion: MMDS-Bench contains 3,482 multimodal instances with seven stance labels and an 800-instance diagnostic subset.The subset evaluates parent understanding, reply understanding, and stance-relation reasoning.
  • 6 Conclusion: Experiments across 12 MLLMs show persistent difficulty when stance depends on relation inference, implicit expression, and multimodal evidence.The benchmark supports evaluation of both final stance prediction and the reasoning behind multimodal social interaction understanding.
  • 6 Conclusion: Models often understand the parent and reply separately but fail to infer how the reply responds to the parent.This diagnostic distinction separates message-level comprehension from dynamic stance inference.

Limitations

MMDS-Bench is scoped to image-based parent–reply interactions on X and is intended primarily for diagnostic evaluation rather than task-specific training.

  • Limitations: The benchmark excludes videos, audio, longer conversation threads, other platforms, and broader linguistic settings.Its coverage is limited to image-based parent–reply interactions on X.
  • Limitations: Because it lacks a large task-specific training set, comparison with fully trained task-specific models is limited.
  • Limitations: Challenge factors often co-occur and should be interpreted diagnostically rather than as independent causal variables.
  • Limitations: Task 2 relies on reference-grounded LLM judges, so automatic evaluation may still reflect judge-specific biases despite multiple judges and human validation.

Ethical Considerations

MMDS-Bench is built from public X parent–reply interactions and uses multimodal annotations focused on the reply’s stance toward its direct parent. Its collection, filtering, annotation, and diagnostic procedures address privacy, data quality, and reasoning-level evaluation.

  • Ethical considerations: Four senior NLP Ph.D. students annotated the data for about three months, working approximately 50 hours each and receiving $7 per hour.The dataset contains no personally identifiable information, and collection used generic keywords rather than user-specific information.
  • Data collection: The dataset uses public X parent–reply interactions collected with topic keywords spanning politics, public health, climate, immigration, gender, economy, and social events.The collection preserves text, attached images, and parent–reply structure.
  • Data collection: The benchmark focuses on direct parent–reply pairs, storing parent and reply text, images, and metadata for filtering and deduplication.This design operationalizes dynamic stance as a local relation to the immediate parent rather than a global topic.
  • Data quality: Filtering removes missing structures, unavailable media, duplicates, videos, corrupted images, and insufficiently informative cases, retaining pairs with at least one image on each side.Text may be present or absent, while images remain part of the multimodal input because they can express stance.
  • Annotation: Annotators assign a seven-label local stance relation and independently mark five non-mutually-exclusive challenge factors.Instances can activate multiple factors, and active-factor counts support difficulty analysis and diagnostic subset construction.
  • Diagnostic evaluation: The 800-instance diagnostic subset samples approximately uniformly across challenge-count groups and requires parent understanding, reply understanding, stance reasoning, and a final label.Reference explanations ground LLM judges, while multi-stage training, discussion, consistency review, and reference review support annotation quality.

A.8 Dataset Statistics

The appendix reports complete statistics for both benchmark evaluation tasks across their evaluation dimensions.

  • Dataset statistics: Table 9 reports detailed statistics for Benchmark Task 1 and Diagnostic Subset Task 2 across different dimensions.
  • Dataset statistics: The reported statistics cover both the full benchmark and the diagnostic subset.
  • Dataset statistics: Task 1 and Task 2 are presented together for comparison across evaluation dimensions.

B Task and Judge Prompts

The prompts enforce local parent–reply stance classification and separate direct prediction from structured reasoning and reference-grounded judging. Task 2 decomposes reasoning into parent understanding, reply understanding, and stance-relation inference.

  • B Task and Judge Prompts: All models receive the same multimodal parent–reply input format, with parent and reply text and images when available.
  • B Task and Judge Prompts: The prompts target the reply’s stance toward its direct parent rather than toward a broader topic.This matches MMDS-Bench’s local conversational definition of dynamic stance.
  • B.1 Task 1 Prompt: Task 1 requires direct prediction of one final label from the seven-label taxonomy in a JSON object.The prompt instructs models to use overall intended meaning, use visual information when necessary, and avoid unsupported visual inferences.
  • B.2 Task 2 Prompt: Task 2 requires concise summaries of the parent and reply followed by an explanation of how the reply responds to the parent.The structured output supports diagnosis of parent-understanding, reply-understanding, and stance-relation errors.
  • B.2 Task 2 Prompt: Table 9 reports detailed statistics across evaluation dimensions for the benchmark and diagnostic subset.
  • B.3 LLM-as-a-judge Prompt: The LLM judge receives the multimodal input, gold label, reference explanations, and model output, then assigns independent PUS, RUS, and SRS scores.The prompt also specifies grounding requirements and penalties for unsupported claims and missing stance-relevant cues.

C.1 Per-class F1 Results

Per-class results show strong performance on explicit agreement and disagreement but substantially weaker and less stable performance on subtle, minority, or ambiguous relations. Text-enriched inputs generally help, yet stance-relation reasoning remains below message-level understanding.

  • Label-level performance: 85.28 for Agree and 87.23 for Disagree are Gemini 2.5 Pro’s highest Task 1 F1 scores among the explicitly stance-bearing labels.Other strong models also perform relatively well on these majority labels.
  • Label-level performance: 42.93 and 42.70 are Claude Sonnet 4.6’s and GPT-5.1’s Task 1 F1 scores for Elaborates.Stronger models exceed 50 F1 on Elaborates in Task 2, but still struggle to distinguish elaboration from implicit agreement or disagreement.
  • Label-level performance: Neutral and NA often receive near-zero F1, while Unrelated typically remains below 30 F1 for most models.Models tend to prefer stance-bearing labels even when the correct relation is weak, absent, or ambiguous.
  • Label-level performance: Query F1 is unstable across models and tasks, with some high scores and some Task 2 scores near zero or exactly zero.Because query cases are few, high Query F1 should be interpreted alongside broader macro-level results.
  • Modality composition: Image-only interactions are generally the most difficult, whereas text-enriched parent–reply settings usually produce higher Macro-F1 or accuracy.Adding text to the parent clarifies the parent frame, while reply text often makes communicative intent easier to recover.
  • Modality composition: PUS and RUS remain relatively high across modality groups, but SRS is consistently lower, especially for image-only inputs.Even when both messages contain text and images, multimodal redundancy does not eliminate the stance-reasoning gap.
  • Challenge factors: 40.57% and 38.10% are Task 1 accuracies for Parent Framing and Label-Boundary Ambiguity, compared with 54.77% for Multimodal Fusion and 53.25% for Non-Literal Reply.Parent Framing alone remains difficult at 32.02% accuracy, identifying semantic framing as a key bottleneck.
  • Judge agreement: 0.88 and 0.92 are average pairwise Spearman correlations for human and LLM judges on SRS, respectively.Agreement is lower for PUS and RUS, where partial-credit judgments vary more with wording, granularity, and emphasis.
Loading 2608.30903v1…