MetaphorVU: Towards Metaphorical Video Understanding

Introduction

• Metaphorical videos convey complex ideas through high-order cross-domain mapping from visual elements to underlying concepts (Forceville et al., 2009; Fahlenbrach, 2016; Pan & Tay, 2020; Zhang, 2021).The introduction notes that existing MLLM video work largely focuses on literal perception, leaving metaphorical understanding without systematic study.Source: §1 · pp. 1–2
• MetaphorVU-Bench is the first comprehensive benchmark, with an eight-type taxonomy, 860 real-world videos, and rigorously cross-validated human annotations.Videos were selected through multi-stage filtration from billions of real-world candidates.Source: §1 · p. 2
• Even Gemini-3-Pro and GPT-5 score around 64 on average, nearly 20 points below human-level performance.Evaluation covers 11 representative closed- and open-source MLLMs.Source: §1 · p. 2
• Over 80% of MLLM failures arise from defective cross-domain mapping rather than recognition errors, identifying visual-to-concept linking as a key improvement target.The analysis spans MLLMs of varying capabilities.Source: §1 · p. 2
• MetaphorBoost queries a metaphorical knowledge graph at inference time to scaffold cross-domain mapping and improve metaphor interpretations across multiple MLLMs.The graph connects metaphorical concepts extracted from metaphorical texts and uses video content-recognition results to retrieve references.Source: §1 · p. 2

2.1. Video Metaphor Taxonomy

• The taxonomy defines eight video-metaphor types, including body and atmosphere language, cultural and naturalistic symbols, causal and analogical montage, and surreal and performative narrative.The authors ground the taxonomy in multimodal metaphor theory and extensions in video studies; most videos may contain multiple types, though Figure 2 illustrates only the dominant type.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2. Benchmark Construction

• MetaphorVU-Bench draws on Kuaishou short videos and filters candidates from billions of videos to 860 with definite metaphorical logic, balancing samples across types as much as possible.Filtering retains videos with over 150 comments (70K), uses GPT-5 to narrow candidates to 16K, Gemini-3-Pro to 4K, then human review to select 860; the source provides diverse real-world topics and durations compatible with common MLLMs.Source: §2.2 · pp. 3–4
• Annotations specify which visual elements convey which implicit meanings, while cross-validation, iterative review, and removal of speech and subtitles help ensure reliable, visually grounded interpretations.Each candidate video is checked by three annotators, and each interpretation is produced by one interpreter and reviewed by two others until all accept it; annotators also provide a brief title and refer to the introduction and audience comments.Source: §2.2 · pp. 4–5
• Table 2 reports that current MLLMs have limited benchmark capability, reasoning-enhanced methods fail to improve effectively, and MetaphorBoost is more effective; human interpretations from 100 samples serve as an upper bound.The passage provides no scores, so these comparisons are reported qualitatively.Source: Table 2 · p. 5

2.3. Evaluation Task and Metric

• The task has systems interpret a video and title, and an LLM judge scores free-form outputs against golden interpretations for grounding metaphorical visual elements and revealing implicit meanings.The judge assigns 0–10 scores rescaled to 0–100; its agreement with human judges has Pearson correlation 0.85.Source: §2.3 · p. 5

3.1. Evaluation Settings

• The evaluation covers close- and open-source MLLMs of varied scales and reasoning-enhanced methods, using official APIs or deployed weights, temperature 0.7, and frame-based inputs encoded as base64.Reasoning methods use their official post-training weights or paper-specified inference strategies; video frames are used because not all models support direct video input.Source: §3.1 · pp. 5–6

3.2. Overall Results

• Gemini-3-Pro leads all baselines at an average score of 63.8, but Qwen3-VL-8B-Thinking scores 52.0 versus the human score of 83.4, leaving MLLMs below human level.The strongest overall baseline still falls short of human performance.Source: §3.2 · p. 6
• LTR and ViTCoT degrade Qwen3-VL-8B-Thinking, metaphor-focused prompts and few-shot examples bring limited gains, and VideoRFT and Vision-R1 yield only marginal gains over Qwen2.5-VL-Instruct.The prior post-training methods incur additional data and training overhead.Source: §3.2 · p. 6

3.3. Detailed Analysis

• Visual misrecognition, missing mapping, superficial mapping, and improper mapping are identified as deficiencies; the latter three account for most errors, making visual-to-concept mapping a key improvement target.Incorrect recognition accounts for only a small proportion of deficiencies.Source: §3.3 · p. 6, Table 3 · p. 6
• Both close- and open-source MLLMs perform worse on the latter four metaphor types, whose richer visual metaphors require more cross-domain mapping than the relatively simpler former four types.This pattern supports mapping augmentation as an important improvement direction.Source: §3.3 · p. 6, Figure 5 · p. 6

4.1. Metaphorical Knowledge Graph

• The metaphorical knowledge graph supports interconnected, multi-hop metaphor reasoning and contains 54,687 nodes and 200,268 edges.It is constructed specifically to provide metaphorical knowledge beyond general commonsense.Source: §4.1 · p. 7

4.2. Inference-time MetaphorVU Boosting

• MetaphorBoost identifies video elements as keywords, retrieves up to 10 target concepts within 2 graph hops that link to the most keywords, then uses them to guide interpretation.The MLLM uses retrieved concepts as references to reveal implicit meanings and generate a metaphor interpretation.Source: §4.2 · p. 7, §4.3 · p. 7

4.3. Effectiveness of MetaphorBoost

• Average score rose from 33.8 to 37.9 with MetaphorBoost on Qwen2.5-VL-7B-Instruct, 52.0 to 55.9 on Qwen3-VL-8B-Thinking, and 63.8 to 66.1 on Gemini-3-Pro, reaching state-of-the-art.The gains surpassed previous post-training methods on Qwen2.5-VL-7B-Instruct and previous inference-time scaling methods on Qwen3-VL-8B-Thinking.Source: §4.3 · p. 7

4.4. Ablation of MetaphorBoost

• Ablations show that external knowledge, graph structure, and metaphor-oriented knowledge each improve mapping augmentation over model-generated, plain-text, and commonsense alternatives.Replacing the metaphorical graph with ConceptNet6 also reduces performance, supporting the need for cognition beyond basic commonsense knowledge.Source: Table 4 · p. 8, §4.4 · p. 8

4.5. Detailed Analysis for MetaphorBoost

• Figure 7 shows fewer missing, superficial, and improper mappings with MetaphorBoost, while a case study illustrates how it mitigates all three deficiencies.The analysis attributes improvement to stronger links between visual elements and underlying external concepts.Source: Figure 7 · p. 8, §4.5 · p. 8

Metaphor Understanding

• Textual metaphor research detects metaphors and identifies source and target domains, while image work builds datasets and explores multimodal fusion (Prystawski et al., 2023; Yang et al., 2025b; Saakyan et al., 2025; Chakrabarty et al., 2022; Qian et al., 2025).Video is temporal and conveys richer information, but existing video-metaphor datasets focus on advertisements and may not reflect complex real-life scenarios (Kalarani et al., 2024; Jia et al., 2025; Long et al., 2025).Source: §5 · p. 8

Deep-semantic Video Understanding

• Deep-semantic video studies examine outcome prediction, novel problem solving, event-logic inference, and spatial or motion reasoning, but the field remains early-stage (Deng et al., 2025; Hu et al., 2025; Chen et al., 2025; Swetha et al., 2025; Yang et al., 2025a).This work introduces metaphorical video understanding as a new challenging task in that direction.Source: §5 · p. 8

Comparison with MMR-V

• Unlike MMR-V, where metaphor is one scenario among broad reasoning abilities, this work focuses on metaphorical video understanding with a taxonomy and benchmark spanning diverse metaphor types and topics.The works can complement each other in evaluating high-order cognitive capabilities.Source: §5 · p. 9

Multimodal Sarcasm

• Multimodal sarcasm relies on apparent contradictions and mainly conveys critical or negative thoughts, whereas metaphorical video understanding requires cross-domain mapping and encompasses more diverse implicit meanings.Metaphorical video understanding links visual elements to underlying concepts and covers forms prevalent in everyday life.Source: §5 · p. 9

• The authors design the first systematic video metaphor taxonomy and construct MetaphorVU-Bench for comprehensive evaluation.These contributions address a gap in prior research on metaphorical video understanding.Source: §6 · p. 9
• Experiments find that current MLLMs struggle with accurate metaphorical video understanding, primarily because of defective cross-domain mapping.Source: §6 · p. 9
• MetaphorBoost uses a metaphorical knowledge graph for mapping augmentation and consistently improves MLLM performance.The paper presents this approach as a promising direction for MLLM advancement and future research.Source: §6 · p. 9

Impact Statement

• The impact statement says the work aims to advance machine learning and that no societal consequences require specific highlighting.Source: Impact Statement · p. 9

A. Theoretical Basis for Video Metaphor Taxonomy

• The first systematic video-metaphor taxonomy distinguishes frame-arrangement and symbolic-sign systems as foundations for classifying metaphors.It draws on multimodal metaphor theory (Forceville et al., 2009; Forceville & Urios-Aparisi, 2009) and video-field extensions. Frame-based forms include body language and atmosphere language; symbols are cultural or naturalistic, with meanings dependent on cultural knowledge or shared human experience, respectively.Source: A. Theoretical Basis for Video… · p. 14
• The taxonomy also classifies montage-based metaphors as causal or analogical, and narrative-based metaphors as surreal or performative.Causal montage implies causal relations across temporally juxtaposed shots, while analogical montage prompts cross-domain mapping; surreal narratives use impossible or dreamlike virtual performances, whereas performative narratives rely on human actors and theatrical conventions.Source: A. Theoretical Basis for Video… · p. 14

B.1. Prompt for LLM Filtration

• GPT-5 first screens videos at scale by analyzing introductions, automatic speech recognition results, and audience comments for metaphorical logic.The paper presents the detailed prompt in Figure 9.Source: §B.1 · p. 14

B.2. Prompt for MLLM Filtration

• Gemini-3-Pro then checks whether the initial text-based analysis agrees with the original videos, addressing its lack of direct visual information.The paper presents the detailed prompt in Figure 10.Source: §B.2 · p. 14

B.3. Prompt for Human Filtration

• Human reviewers retained 860 videos with definite metaphorical logic and annotated metaphor types while balancing sample counts as much as possible.Final filtration used the original video, its introduction, and audience comments.Source: §B.3 · p. 14

B.4. Manual Annotation Guideline

• Annotators interpret metaphors using the video introduction and audience comments, specifying which visual elements convey which implicit meanings.The paper provides a fixed-format guideline in Figure 12.Source: §B.4 · p. 15

C.1. Prompt for Evaluation

• The evaluation prompt asks MLLMs to recognize visual content, map it to external concepts, infer implicit meanings, and identify which content conveys each meaning.The prompt’s details are shown in Figure 13.Source: §C.1 · p. 15

C.2. Prompt for LLM Judge

• Because interpretations are free-form, the evaluation uses DeepSeek-V3.2 as an LLM judge, following prior free-form QA metrics (Li et al., 2025b;e; Yu et al., 2025; Long et al., 2025).The passage notes that rule-based metrics may not align with human scoring habits (Mayfield et al., 2024; Li et al., 2025d); the judge prompt is in Figure 14.Source: §C.2 · p. 15

C.3. Consistency Experiments for LLM Judge

• Human and LLM judge scores on 100 sampled instances had a Pearson correlation of 0.85 (p = 3e-20, p < 0.001).The authors interpret this as a strong, statistically significant positive correlation that validates the LLM judge.Source: §C.3 · p. 15

D.1. Details of Metaphorical Textual Datasets

• The textual metaphor datasets contain extensive metaphorical concept pairs, with their details reported in Table 5.Source: §D.1 · p. 15
• To support the knowledge graph’s universality, GPT-5 translates portions of the originally Chinese data into English.Source: §D.1 · p. 15

D.2. Prompt for Extracting Metaphorical Concept Pairs

• Following prior reports of LLM information-extraction ability (Tang et al., 2023; Li et al., 2024; Tang et al., 2024; Li et al., 2025a), DeepSeek-V3.2 extracts metaphorical concept pairs as graph nodes.Source: §D.2 · p. 15

E.1. Prompt for Identifying Visual Elements

• At inference, MetaphorBoost prompts the given MLLM to identify video visual elements and output them as a keyword list for sourcing mapping nodes.Source: §E.1 · p. 16

E.2. Prompt for Generating Video Metaphor Interpretation

• MetaphorBoost queries the knowledge graph using the identification results and retrieves concepts for augmentation.Source: §E.2 · p. 16
• Using the retrieved concepts as augmentation, the given MLLM unveils implicit meanings and generates a video metaphor interpretation.Source: §E.2 · p. 16

F. Details of Reasoning-based Baselines

• VideoRFT (Wang et al., 2025b) uses supervised fine-tuning with chain-of-thought annotations and semantic-consistency reinforcement learning, but focuses on foundational video cognition.Its focus on object recognition and event understanding limits its capability for metaphorical video understanding.Source: F. Details of Reasoning-based Baselines · p. 16
• Vision-R1 (Huang et al., 2025) trains on a 200K multimodal chain-of-thought dataset and uses Progressive Thinking Suppression Training, but targets logical and mathematical reasoning.Like VideoRFT, it does not target the cross-domain mapping required for metaphor interpretation.Source: F. Details of Reasoning-based Baselines · p. 16
• ReAd-R (Long et al., 2025) targets implicit meanings in advertisement videos, but its domain-specific training limits generalization to broader metaphorical video understanding.It is more relevant to the task than VideoRFT and Vision-R1 because advertisements often contain implicit meanings.Source: F. Details of Reasoning-based Baselines · p. 16
• LTR recursively decomposes complex video questions into a language-centric logical tree, but may not capture metaphorical cross-domain mapping.Its bottom-up reasoning improves transparency on video QA benchmarks.Source: F. Details of Reasoning-based Baselines · p. 16
• ViTCoT interleaves visual and textual information so models can re-examine video during reasoning, but focuses on explicit content rather than metaphorical mapping.It improves general video understanding through better visual integration.Source: F. Details of Reasoning-based Baselines · p. 16
• Prompt Engineering (Wei et al., 2022) prompts models to map visual content to implicit meanings through intermediate reasoning steps.The paper uses this chain-of-thought approach as a straightforward baseline for metaphorical video understanding.Source: F. Details of Reasoning-based Baselines · p. 16
• Few-shot Example adapts models through annotated demonstrations of projecting explicit visual content onto abstract concepts.Together with Prompt Engineering, it is presented as a direct way to adapt existing models to metaphorical video understanding.Source: F. Details of Reasoning-based Baselines · p. 16

Query strategy

• MetaphorBoost queries the knowledge graph to a maximum of h=2 hops and retains the Top-z=10 target nodes most associated with query keywords.This strategy is designed to exploit the graph’s multi-hop structured reasoning.Source: G. Experiments about Query Strategy… · p. 17
• Randomly retaining graph results instead of prioritizing nodes connected to the most query keywords decreases average performance.The authors attribute this to the knowledge graph’s ability to provide low-noise augmentation through structured federated querying.Source: G. Experiments about Query Strategy… · p. 17

Hyperparameters

• Variants using h=1 or z=5 have lower average scores than MetaphorBoost’s defaults, despite fluctuations across subsets.The defaults are h=2 and z=10; the authors interpret the results as support for reasonably deep, low-noise graph augmentation for cross-domain mapping.Source: G. Experiments about Query Strategy… · p. 17

H. More Examples of MetaphorVU-Bench

• Examples cover Body Language, Atmosphere Language, Cultural Symbol, Naturalistic Symbol, and Causal Montage.Each figure shows the dominant metaphor type for convenient illustration, although videos may contain multiple types.Source: H. More Examples of MetaphorVU-Bench · p. 17, Figure 18 · p. 27, Figure 19 · p. 28
• Further examples cover Analogical Montage, Surreal Narrative, and Performative Narrative.These figures likewise show only the dominant type in each example, while videos may simultaneously contain multiple metaphor types.Source: H. More Examples of MetaphorVU-Bench · p. 17, Figure 23 · p. 32, Figure 24 · p. 33