PaperlayerView exact revision on arXiv

Source-linked AI summary

ViMU: Benchmarking Video Metaphorical Understanding

Qi Li, Xinchao Wang

arXiv:2605.14607v1cs.CVcs.CY

TL;DR

Video understanding models largely capture surface content, leaving metaphorical, ironic, and socially grounded subtext insufficiently evaluated. ViMU introduces a hint-free benchmark for this capability, and finds that current frontier models achieve below 50% overall performance despite strong surface-level abilities.

  • Problem

    Existing video understanding models and benchmarks largely emphasize observable objects, actions, and temporal relations, leaving socially grounded subtext interpretation insufficiently evaluated.

  • Method

    ViMU constructs a hint-free multi-task benchmark of 588 videos and 2,352 open-ended and multiple-choice questions spanning diverse rhetorical mechanisms and social value signals.

  • Results

    Below 50% overall performance shows that current frontier models struggle substantially with video metaphorical understanding despite strong performance on surface-level tasks.

  • Takeaways & Limitations

    Strong general video understanding does not ensure robust interpretation of rhetorical, social, and culturally grounded meanings.

  • Takeaways & Limitations

    Subjective interpretation leaves residual ambiguity and annotator bias, while benchmark success may not imply robust real-world understanding of nuanced social or cultural meaning.

Abstract

from arXiv · show

Any new medium, once it emerges, is used for more than the transmission of overt content alone. The information it carries typically operates on two levels: one is the content directly presented, while the other is the subtext beneath it-the implicit ideas and intentions the creator seeks to convey through the medium. Likewise, since video technologies became widely adopted, video has served not only as a powerful tool for recording and communicating visual information, but also as a vehicle for emotions, attitudes, and social meanings that are often difficult to articulate explicitly. Thus, the true meaning of many videos does not reside solely in what is shown on screen; it is often embedded in context, style of expression, and the viewer's social experience. Some forms of such video subtext are humorous, while others carry irony, mockery, or criticism. These implicit meanings can also be interpreted very differently across cultural backgrounds and social groups. However, most existing video understanding models still focus primarily on literal visual comprehension, such as recognizing objects, actions, or temporal relations, and lack a systematic ability to understand the metaphorical, ironic, and social meanings embedded in videos. To bridge this gap, we introduce ViMU, the first benchmark designed to systematically evaluate the subtext understanding capabilities of frontier models in videos. ViMU assesses whether video understanding models can go beyond literal perception to infer implicit meaning while grounding their interpretations in multimodal evidence and answering both open-ended and multiple-choice questions. Importantly, all questions are designed to be hint-free, ensuring that no key evidence is disclosed to models before answering.

1 Introduction

ViMU benchmarks whether video-understanding models can infer culturally situated social subtext beyond observable content, using hint-free multimodal questions. The benchmark contains 588 videos and 2,352 questions across four tasks, and evaluations show that current MLLMs struggle with metaphorical understanding.

  • Motivation: Video understanding models often capture surface-visible content but not the deeper social subtext that conveys broader meanings, values, and collective attitudes.Full video understanding requires combining observable content with its underlying subtext.
  • Motivation: Recovering video subtext requires integrating multimodal evidence, culturally situated references, and the creator’s communicative intent beyond explicit visual content.Recognizing objects, actions, and temporal structure alone is insufficient when observable content and implied meaning diverge substantially.
  • ViMU benchmark: ViMU evaluates whether models can recover video subtext in a hint-free setting without advance disclosure of the relevant socio-cultural cues.The benchmark is constructed through multiple rounds of annotation and filtering by advanced closed-source models and human evaluators.
  • ViMU benchmark: 588 videos and 2,352 questions comprise ViMU’s high-quality dataset across four tasks covering both open-ended and multiple-choice questions.The dataset follows a curation process involving repeated annotation and filtering.
  • Findings: Below 50% average performance across the four tasks was achieved even by the most advanced closed-source models among 16 evaluated MLLMs.The results indicate that video metaphorical understanding remains technically challenging for existing MLLMs.
  • Findings: Many models over-predict generic or safer categories while under-predicting more implicit or socially coded ones.This pattern suggests a shared tendency to favor more accessible interpretations.

2 Related Work

Related work has expanded video understanding beyond explicit visual evidence toward contextual, commonsense, humorous, cultural, and social interpretation. ViMU builds on these efforts by evaluating inference from observable video content to latent subtext beyond objects, actions, and temporal relations.

  • Reasoning beyond explicit visual evidence: Recent video-QA work requires answers to be inferred from indirect or partially unavailable visual cues using context and commonsense reasoning.Examples include I-VQA, VisualCOMET, Video2Commonsense, MECD, and MECD+.
  • Humor understanding, meme interpretation, and social meaning: Related benchmarks study multimodal humor understanding and the contextual and cultural interpretation of Internet audio-visual memes.v-HUB focuses on non-verbal short-video humor, while AVMeme Exam evaluates contextual and cultural understanding.
  • Position of ViMU: ViMU targets latent video subtext, rather than meaning exhausted by visible objects, actions, or temporal relations.It differs from prior work by evaluating movement from observable content to latent subtext, including social signals.

3 ViMU: Video Metaphorical Understanding Benchmark

ViMU is a multi-task benchmark for video metaphorical understanding that evaluates implicit subtext beyond surface-level content. It combines diverse semantic categories, multimodal evidence and target subjects, and complementary tasks for interpretation, semantic structure, and evidence grounding.

  • Benchmark Scope: ViMU contains 2,352 questions from 588 videos spanning more than ten rhetorical mechanisms and social value signals.It is designed to assess understanding of subtextual meaning beyond surface-level video content.
  • Semantic Categories: The benchmark covers rhetorical mechanisms such as irony, exaggeration, contrast, deadpan delivery, parody, and bait-and-switch.These mechanisms describe how humor, critique, or commentary is constructed through expression.
  • Evidence and Targets: ViMU organizes interpretation around multimodal evidence sources and diverse target subjects, including individuals, social groups, institutions, and identity-related groups.Evidence may include video frames, audio, on-screen text, editing patterns, transcripts, and vocal tone.
  • Evaluation Tasks: The evaluation combines open-ended interpretation, multiple-choice identification of rhetorical mechanisms and social value signals, and evidence grounding.Together, these tasks probe overall intended meaning, underlying semantic structure, and the multimodal evidence supporting an interpretation.
  • Construction Principles: Dataset construction prioritizes broad coverage of rhetorical and social signals while accounting for both implicit-meaning sources and reference targets.The specified sources include visual frames, on-screen text, editing patterns, audio content, and vocal tone.

4 Experiments and Analysis

Experiments evaluate 16 zero-shot MLLMs on ViMU and show that metaphorical understanding remains weaker than general video understanding. Analyses identify incomplete evidence retrieval, family-level error patterns, limited taxonomy-geometry preservation, and systematic option biases as key model weaknesses.

  • Settings: 16 MLLMs are evaluated zero-shot using official implementations or APIs and uniform video sampling, covering both open-source and proprietary models.The benchmark evaluation uses each model’s official implementation or available API.
  • Overall Performance Analysis: Current models perform substantially worse on metaphorical understanding than on general video understanding, exposing the central gap targeted by ViMU.GPT-5.2 leads open-ended interpretation and evidence grounding, with both results around 70%.
  • Analysis on Evidence Grounding (EG): Evidence-grounding failures are driven mainly by incomplete retrieval of required cues rather than indiscriminate over-selection.Miss-Only and Mixed errors form a substantial share of non-exact predictions, while Extra-Only errors are generally less dominant.
  • Analysis on Rhetoric Mechanisms (RM) and Social Value Signals (SV): RM and SV error profiles cluster by model family, with PC1 and PC2 explaining 32.9% and 18.5% of variance, respectively.OpenAI, Qwen, and Mistral models tend to remain close to their family peers, indicating shared taxonomy-label organization patterns.
  • Analysis on Rhetoric Mechanisms (RM) and Social Value Signals (SV): Models preserve only limited taxonomy geometry, while guidance mainly reweights local decisions without consistently improving global structural fidelity.For many models, geometry distance remains similar or becomes slightly larger after guidance.
  • Analysis on Rhetoric Mechanisms (RM) and Social Value Signals (SV): Models show systematic option biases, over-predicting safer or broader categories such as rhetoric A and social-signal B while under-predicting implicit or ideological category E.With-guidance results remain qualitatively similar to without-guidance results, indicating only small local adjustments to option allocation.

5 Conclusion

ViMU benchmarks video models’ ability to understand rhetorical, social, and culturally grounded subtext beyond literal perception. Despite strong surface-level performance, current frontier models achieve below 50% overall performance and exhibit systematic interpretive gaps.

  • Benchmark contribution: ViMU evaluates video models beyond literal perception by targeting rhetorical, social, and culturally grounded meanings.The benchmark focuses on understanding video subtext rather than only surface content.
  • Model performance: Below 50% overall performance shows that current frontier models struggle substantially with implicit meaning despite strong surface-level task performance.The results indicate a gap between literal or surface comprehension and subtext interpretation.
  • Behavioral analysis: Fine-grained analyses reveal systematic gaps and distinct behavioral patterns in current frontier models’ subtext understanding.These analyses characterize how models fail beyond their aggregate performance.

A Details of the Dataset Curation Process … D Social Value Signals Grouping

ViMU constructs a hint-free video-subtext benchmark through multimodal evidence extraction, LLM-based annotation and question refinement, human review, and structured evaluation. Its open-ended judging emphasizes semantic understanding, while macro-level groupings organize rhetorical and social-value signals.

  • A Details of the Dataset Curation Process: ViMU’s curation pipeline combines multimodal evidence extraction, LLM semantic annotation, hint-free question generation, iterative refinement, task construction, and human expert review.The final dataset contains 588 videos and is validated by 5 human experts.
  • A Details of the Dataset Curation Process: Each video is represented by uniformly sampled frames and an audio transcript, grounding downstream reasoning in observable multimodal evidence rather than external metadata.The evidence representation is Ei = {Fi, ti}.
  • A.1 Prompt Design Details: Prompts separate literal content from intended meaning and require difficult, hint-free questions that test implicit understanding without revealing the target semantic field.Questions must avoid leaked terms, surface-answerability, and presuppositions about hidden meaning.
  • A.1 Prompt Design Details: Validation checks hint-freeness, difficulty, alignment, surface-answerability, and implicit inference, then uses feedback to revise questions while preserving neutral, non-leading wording.The curation process allows at most K = 3 refinement rounds.
  • B LLM-as-a-Judge for Open-Ended Questions: Open-ended answers are judged by semantic content rather than lexical overlap using core intent, implicit signal, social meaning, hallucination, and literal-only dimensions.The scoring rule subtracts hallucination and literal-only penalties, with a maximum possible score of 9.
  • B LLM-as-a-Judge for Open-Ended Questions: The judge produces structured scores, verdicts, and justifications while prioritizing subtext sensitivity and penalizing unsupported claims or surface-level descriptions.A factually correct but purely literal answer receives lower evaluation than one capturing the intended meaning.
  • C Rhetoric Mechanism Grouping: Fine-grained rhetoric mechanism labels are mapped to five macro categories to support structured evaluation and analysis.The macro categories provide a common abstraction for rhetoric mechanisms.
  • D Social Value Signals Grouping: Fine-grained social value signals are likewise mapped to macro-level categories, enabling consistent subtext evaluation across models.This grouping is defined in Table 3.

E Details of the Taxonomy of Rhetoric Mechanisms · F Details of the Taxonomy of Social Value Signals

The taxonomy defines rhetoric mechanisms by how videos construct implicit meaning through literal content, attitude reversal, discrepancy, ridicule, stereotypes, exaggeration, juxtaposition, indirectness, absurdity, role inversion, coded signals, misdirection, delivery, imitation, or uncaptured hybrids. It separately categorizes social value signals ranging from none or negative affect to contempt, exclusion, prejudice, norm violation, anti-mainstream values, and other coded or adversarial stances.

  • E Details of the Taxonomy of Rhetoric Mechanisms: Rhetoric mechanisms range from Literal Only content to Sarcasm, Irony, and Mockery, distinguishing surface meaning from reversed attitudes, appearance–reality discrepancies, and ridicule.Sarcasm typically signals ridicule, dismissal, or criticism, while Irony does not necessarily require direct mocking intent.
  • E Details of the Taxonomy of Rhetoric Mechanisms: Stereotype Invocation, Exaggeration, and Contrast construct meaning through shared caricatures, deliberate overstatement, or juxtaposition of incompatible elements.These mechanisms may support humor, critique, reinforcement, inversion, or meaning beyond literal plausibility.
  • E Details of the Taxonomy of Rhetoric Mechanisms: Innuendo, Absurdism, and Role Reversal rely respectively on indirect socially loaded implication, deliberate irrationality or surreal mismatch, and inverted expected roles or hierarchies.Their effects depend on implication, nonsensical premises, or behavior conventionally associated with another party.
  • E Details of the Taxonomy of Rhetoric Mechanisms: Dog Whistle or Code, Bait and Switch, and Deadpan use audience-specific coded signals, abrupt replacement of expectations, or restrained delivery to produce meaning.Dog whistles target audiences with relevant cultural, political, or subcultural knowledge, while Deadpan makes flat presentation central to the effect.
  • E Details of the Taxonomy of Rhetoric Mechanisms: Parody imitates another form for humor, critique, or commentary, while Other covers mechanisms not captured by the listed categories or cleanly reducible to one type.The taxonomy therefore includes both recognizable imitation and residual or hybrid rhetorical cases.
  • F Details of the Taxonomy of Social Value Signals: Social value signals begin with None and Negative Affect, distinguishing no salient stance from broadly negative emotions such as frustration, discomfort, annoyance, or aversion.Negative Affect does not necessarily specify a stronger social stance.
  • F Details of the Taxonomy of Social Value Signals: Contempt, Exclusion, Discrimination or Prejudice, Norm Violation, and Anti-Mainstream Value capture scorn, denied belonging, group-based bias, transgression, or opposition to mainstream expectations.These categories cover interpersonal superiority, boundary-making, unequal judgment, improper conduct, and rejection of widely accepted norms or tastes.
  • F Details of the Taxonomy of Social Value Signals: Additional social signals include Fatalism or Cynicism, Sexual Implication, Political or Identity Signal, Aggression or Hostility, Humiliation, and Other.They represent resignation or distrust, sexualized meaning, ideological or identity alignment, antagonism, loss of dignity, or uncaptured and combined signals.

G Guided and Unguided Prompts for Structured Subtext Understanding Tasks

ViMU evaluates rhetoric-mechanism and social-value-signal identification under unguided and guided prompts. Guided prompts add definitions for five macro-categories, while both settings require conservative, JSON-formatted option selection grounded in the video.

  • Prompt settings: The two structured tasks compare prompts without guidance against prompts that additionally provide taxonomy definitions for five macro-categories.Unguided prompts include the task question, transcript, options, and output rules; guided prompts add taxonomy guidance.
  • Output constraints: Both prompt formats instruct models to select only clearly supported option letters, avoid guessing, and return an empty list when no option is justified.The implementation specifies valid JSON containing a selected_options array.
  • Rhetoric mechanism taxonomy: Rhetoric guidance organizes meanings into literal, opposition, attitude, amplification, and implicit social-framing categories.Examples include sarcasm, irony, mockery, exaggeration, parody, innuendo, stereotypes, and coded expressions.
  • Social value signal taxonomy: Social-value guidance distinguishes neutral content, emotional attitude, social evaluation, norm and value framing, and identity or ideological signaling.The categories cover signals such as negative affect, cynicism, contempt, humiliation, hostility, norm violation, political identity, and sexual implication.

H The With-Guidance Counterpart of Affinity Bias

The with-guidance results in Figure 12 exhibit a pattern similar to the results shown in Figure 9.

  • With-Guidance Results: Figure 12’s with-guidance results show a pattern similar to Figure 9’s results.The passage identifies Figure 12 as the with-guidance counterpart of the results related to Figure 9.

I Details of Baselines and the Evaluation Process

ViMU evaluates open-ended responses with rubric-based LLM judging and multiple-choice responses with strict set-based scoring. Scores are normalized to percentages, averaged across samples within tasks, and then averaged across tasks for overall performance.

  • Score aggregation: All task outputs are normalized into percentage scores, task-level scores are averaged across samples, and overall performance is the mean across tasks.These steps provide consistent comparison across evaluation settings.
  • Open-ended evaluation: Open-ended predictions are judged against reference answers for intended meaning, implicit signals, social interpretation, hallucination, and literalness, producing a continuous score in [0, 1].The continuous score is scaled to a percentage.
  • Multiple-choice evaluation: Multiple-choice tasks use set-based scoring over predicted options P and ground-truth options G across evidence, rhetoric, and social-value identification.The scoring rule is strict yet interpretable.
  • Multiple-choice evaluation: Any incorrect predicted option makes the score 0; otherwise, the score is the proportion of correctly selected options.This penalizes hallucinated selections while allowing partial credit for correctly identifying subsets, with full credit only for selecting all and only correct options.

J Limitations

ViMU’s metaphorical and socially grounded interpretations remain subjective, with residual ambiguity and annotator bias despite structured annotation and validation. Models may also exploit superficial patterns or dataset-specific regularities because the benchmark is designed for evaluation rather than training.

  • Subjectivity and annotation: Interpretations of metaphorical and socially grounded meaning are inherently subjective, leaving residual ambiguity and annotator bias despite structured annotation and validation.These limitations arise from the interpretive nature of the benchmark’s target meanings.
  • Evaluation scope: ViMU is designed for evaluation rather than training, limiting its role as a direct resource for model learning.The passage explicitly distinguishes the benchmark’s evaluation purpose from training use.
  • Shortcut exploitation: Models may exploit superficial patterns or dataset-specific regularities instead of genuinely understanding video subtext.Such shortcut behavior can affect how benchmark performance should be interpreted.

K Societal Impact

ViMU advances evaluation of multimodal models by exposing limitations in interpreting socially grounded video subtext, while raising risks of bias, harmful misuse, and unsafe deployment. It is intended only as an evaluation benchmark, with future work needed on fairness, cultural coverage, and robustness.

  • Positive impact: ViMU exposes systematic model limitations in interpreting rhetoric, social signals, and culturally situated subtext relevant to safe deployment.These limitations matter for content moderation, assistive technologies, and human–AI interaction.
  • Potential risks: Socially sensitive and potentially offensive content may cause evaluated models to reproduce or amplify harmful stereotypes, biases, or misinterpretations.Improved implicit-meaning interpretation could also be misused for profiling, surveillance, or manipulating user intent, particularly around political or identity-related signals.
  • Responsible use: ViMU is intended solely as an evaluation benchmark rather than a training resource, and users should report model behaviors transparently and avoid deployment based only on benchmark performance.Future work should investigate fairness, cultural coverage, and robustness so advances benefit diverse user groups without reinforcing existing harms.
Loading 2605.14607v1…