PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning

Chenhao Zhang, Yazhe Niu, Hongsheng Li

arXiv:2602.10575v1cs.CVcs.AIcs.CY

TL;DR

MLLMs remain limited in understanding the cultural, emotional, and contextual implications embedded in images, despite strong literal perception. MetaphorStar addresses this gap with end-to-end visual reinforcement learning using TFQ-Data, TFQ-GRPO, and TFQ-Bench, achieving substantial benchmark gains and improved complex visual reasoning. The paper also identifies evaluation bias as a limitation through its analysis of the OSQ Paradox.

  • Problem

    MLLMs often excel at literal perception but struggle with image implication, which requires multi-hop reasoning, cultural context, and Theory of Mind capabilities.

  • Method

    MetaphorStar is an end-to-end visual reinforcement-learning framework comprising TFQ-Data, TFQ-GRPO, and TFQ-Bench.

  • Results

    MetaphorStar significantly improves image implication performance and enhances complex visual reasoning across evaluations.

  • Takeaways & Limitations

    Learning image implication tasks can improve general understanding, especially complex visual reasoning, while general visual understanding is maintained or slightly improved.

  • Takeaways & Limitations

    LLM-as-a-judge evaluation can favor verbose SFT outputs over more concise and accurate end-to-end RL answers, biasing subjective OSQ scores.

Abstract

from arXiv · show

Metaphorical comprehension in images remains a critical challenge for Nowadays AI systems. While Multimodal Large Language Models (MLLMs) excel at basic Visual Question Answering (VQA), they consistently struggle to grasp the nuanced cultural, emotional, and contextual implications embedded in visual content. This difficulty stems from the task's demand for sophisticated multi-hop reasoning, cultural context, and Theory of Mind (ToM) capabilities, which current models lack. To fill this gap, we propose MetaphorStar, the first end-to-end visual reinforcement learning (RL) framework for image implication tasks. Our framework includes three core components: the fine-grained dataset TFQ-Data, the visual RL method TFQ-GRPO, and the well-structured benchmark TFQ-Bench. Our fully open-source MetaphorStar family, trained using TFQ-GRPO on TFQ-Data, significantly improves performance by an average of 82.6% on the image implication benchmarks. Compared with 20+ mainstream MLLMs, MetaphorStar-32B achieves state-of-the-art (SOTA) on Multiple-Choice Question and Open-Style Question, significantly outperforms the top closed-source model Gemini-3.0-pro on True-False Question. Crucially, our experiments reveal that learning image implication tasks improves the general understanding ability, especially the complex visual reasoning ability. We further provide a systematic analysis of model parameter scaling, training data scaling, and the impact of different model architectures and training strategies, demonstrating the broad applicability of our method. We open-sourced all model weights, datasets, and method code at https://metaphorstar.github.io.

1. Introduction

Image implication requires models to move beyond literal object recognition toward culturally and contextually grounded conceptual understanding. MetaphorStar addresses this gap with an end-to-end visual reinforcement-learning framework built around TFQ-Data, TFQ-GRPO, and TFQ-Bench.

  • Motivation: MLLMs often excel at literal perception but struggle to interpret cultural, emotional, and contextual implications in images.Image implication demands multi-hop reasoning and Theory of Mind capabilities beyond conventional VQA.
  • Motivation: Visual metaphors require connecting concrete scenes with abstract meanings, such as interpreting a split road as a consequential choice.The paper frames metaphorical comprehension as a cognitive tool for understanding surroundings rather than merely identifying objects.
  • Related approaches: Existing approaches use explicit metaphor mapping, passive implicit reasoning, or contextual alignment, but face challenges from complex mappings, search spaces, and dynamic cultural references.These limitations motivate a method that can actively reinforce non-literal reasoning pathways.
  • Approach: MetaphorStar proposes the first end-to-end visual reinforcement-learning framework for image implication, comprising TFQ-Data, TFQ-GRPO, and TFQ-Bench.The framework is presented as a unified response to the gap between literal perception and conceptual understanding.
  • Results: 82.6% average performance improvement is reported on image implication benchmarks for the MetaphorStar family trained with TFQ-GRPO on TFQ-Data.MetaphorStar-32B is reported to achieve state-of-the-art results on Multiple-Choice and Open-Style Questions and outperform Gemini-3.0-pro on True-False Questions.

2. Related Work

Image implication research has expanded from specialized humor and sarcasm recognition toward comprehensive evaluation of metaphorical understanding. Existing methods rely on metaphor mapping, prompted reasoning, or external knowledge, each facing limitations in representing complex non-literal relationships.

  • Scope: Image implication includes humor, sarcasm, and broader metaphorical understanding, motivating more comprehensive evaluation frameworks.DeepEval introduced a systematic taxonomy, followed by II-Bench as an English image implication benchmark.
  • Task demands: Image implication understanding requires sophisticated multi-hop reasoning and Theory of Mind capabilities beyond conventional visual-language tasks.These requirements distinguish the field from approaches focused on isolated implication categories.
  • Existing methods: Explicit metaphor mapping links visual features to metaphor ontologies but is limited by many-to-many relationships and static ontologies.Static representations do not adequately capture dynamic cultural references.
  • Existing methods: Contextual alignment depends on retrieved external knowledge, making the strategy computationally intensive and vulnerable to unreliable information.This introduces a practical constraint beyond the core reasoning challenge.

3. Method

The method introduces TFQ as a dense, learnable, and verifiable foundation for image-implication evaluation, then uses TFQ-GRPO to optimize structured visual reasoning with outcome and format rewards.

  • 3.1. True-False Question (TFQ) For Image Implication Understanding: TFQ complements MCQ and OSQ by evaluating multiple image propositions across implication and visual content, spanning foundational to highly open-ended reasoning.It is designed as a lower-bound, fine-grained assessment while MCQ provides medium difficulty and OSQ represents the upper bound.
  • 3.1. True-False Question (TFQ) For Image Implication Understanding: TFQ offers high knowledge density, clearer learnability, and definitive verifiability, making it suitable for reinforcement-learning reward signals.Its statements provide multiple propositions per image and a less noisy signal than OSQ.
  • 3.2.1. Data Generation: The dataset contains 1,434 metaphorical images and 14,099 generated True/False questions, with each image receiving an average of 5-10 question-answer pairs.Questions were generated from image descriptions and ground-truth implications, then manually verified.
  • 3.2.1. Data Generation: TFQ questions combine metaphorical implication with primary visual information and use plausible false distractors across hierarchical difficulty levels.This design targets both conceptual understanding and basic VQA-style perception.
  • 3.2.2. Dataset and Benchmark Splits: TFQ-Data and TFQ-Bench are separated into training and evaluation sets, including Lite subsets for rapid experimentation and standardized testing.TFQ-Bench-Lite is strictly disjoint from TFQ-Data-Full.
  • 3.3. TFQ-GRPO: TFQ-GRPO fine-tunes reasoning with Group Relative Policy Optimization, using structured Image Description → Implication Analysis → Final Answer prompts.Its reward combines binary answer accuracy with a penalty for outputs that violate the specified tag structure.

4. MetaphorStar Family

The MetaphorStar family includes 3B, 7B, and 32B models based on QwenVL-2.5, trained with end-to-end TFQ-GRPO. Entropy analysis examines uncertainty across TFQ, MCQ, and OSQ generation.

  • 4. MetaphorStar Family: MetaphorStar comprises 3B, 7B, and 32B models using the QwenVL-2.5 series as the base model.The family is analyzed across model sizes in subsequent experiments.
  • 4. MetaphorStar Family: Token entropy visualizations compare MetaphorStar-7B responses across TFQ, MCQ, and OSQ, with red indicating uncertainty and blue indicating confidence.The figure uses token-level entropy to inspect generation behavior across task formats.
  • 4. MetaphorStar Family: Direct end-to-end TFQ-GRPO training is adopted because SFT warmup was found to constrain intrinsic reasoning and produce weaker generalization.All models use the direct end-to-end RL strategy with TFQ-Data-Lite.
  • 4. MetaphorStar Family: High-entropy tokens concentrate around logical connectors, quantifiers, and pronouns, whereas low-entropy tokens usually reproduce image facts or deterministic phrases.The pattern suggests uncertainty is concentrated in inferential and relational language rather than factual reproduction.

5. Experiment

MetaphorStar is evaluated across TFQ, MCQ, and OSQ image-implication benchmarks, where it achieves strong performance and generalizes to broader visual reasoning and understanding tasks.

  • Image implication benchmarks: MetaphorStar achieves SOTA performance on TFQ-Bench-Lite, with MetaphorStar-3B scoring 62% versus Gemini-3.0-pro at 58%.
  • Image implication benchmarks: MetaphorStar-7B improves TFQ accuracy by 150% relative to QwenVL-2.5-7B, while MetaphorStar-3B gains 210% over QwenVL-2.5-3B.
  • Image implication benchmarks: MetaphorStar-32B is the top-performing open-source model on MCQ, while MetaphorStar-7B improves 60% over its base model.
  • Image implication benchmarks: On OSQ, MetaphorStar-32B achieves the best score at 3.94, exceeding Gemini-3.0-pro at 3.82 and Claude-4.0-Sonnet at 3.46.
  • Generalization: Training improves visual reasoning by 3.2 points for MetaphorStar-7B and 2.9 points for MetaphorStar-32B on average, including a 16.2-point MMMU gain for 32B.
  • Generalization: General visual understanding remains stable or slightly improves across 14 benchmarks, including average gains of 0.3 points for MetaphorStar-7B.

6. Ablation Study

The ablation study finds that TFQ-GRPO benefits from increased model and data scale, generalizes across architectures, and outperforms SFT-based alternatives on core implication tasks. It also identifies an entropy-related failure mode in SFT warmup.

  • Model Parameter Scaling: TFQ-GRPO enables monotonic OSQ gains as model size increases, unlike base models with inconsistent scaling.MetaphorStar scores rise from 3.06 to 3.22 to 3.94, while the 7B base model scores 2.34 versus 2.44 for the 3B base model.
  • Training Data Scaling: Training data quantity improves performance across TFQ, MCQ, and OSQ, with the full dataset producing the strongest results.MetaphorStar-7B-Full uses 1.4k images and achieves 84% on TFQ.
  • Different Model Architecture: TFQ-GRPO substantially improves LLaVA-1.5-7B across TFQ, MCQ, and OSQ, demonstrating applicability beyond QwenVL architectures.LLaVA performance changes from 0% to 6% on TFQ, 16% to 34% on MCQ, and 2.06 to 2.78 on OSQ.
  • Different Training Strategy: End-to-end TFQ-GRPO performs best on TFQ and MCQ, whereas SFT-based strategies reduce MCQ performance from 46% to 28%.SFT-based models receive the highest OSQ judge score, but the paper attributes this to verbosity and evaluation bias.
  • Different Training Strategy: SFT creates a low-entropy policy that persists after RL, while end-to-end RL retains higher entropy for broader reasoning-policy search.Entropy is reported as 0.30 for SFT, 0.29 after SFT plus RL, 1.33 for the base model, and 1.23 for end-to-end RL.

7. Discussion

The discussion argues that image implication training transfers to complex visual reasoning because it combines non-literal multi-hop inference with fine-grained truth verification and perceptual grounding. It also explains why end-to-end RL is better suited than SFT warmup for this task.

  • Why SFT Warmup Lose?: SFT warmup is characterized as actively detrimental to image implication performance rather than merely unnecessary.The paper terms this failure mode the “SFT Curse.”
  • Why SFT Warmup Lose?: SFT favors high-probability imitation, whereas end-to-end RL can reinforce creative, low-probability reasoning paths that lead to correct answers.The discussion frames image implication as creative generalization requiring connections between semantically distant concepts.
  • Why SFT Warmup Lose?: SFT trains stylistic plausibility more readily than the discriminative reasoning needed to identify and reject incorrect options.The paper contrasts a format-oriented “talker” with a reasoning-oriented “thinker.”
  • Why SFT Warmup Lose?: LLM-as-a-judge evaluation can favor verbose SFT outputs over more concise and accurate end-to-end RL answers on OSQ.Contradictory viewpoints may be misread as deeper thought, creating the reported OSQ Paradox.
  • Why Image Implication Tasks Can Help with Visual Reasoning?: Image implication training improves downstream visual reasoning and general VQA, according to the paper’s generalization experiment.The authors attribute this effect to the task’s reasoning demands and training methodology.
  • Why Image Implication Tasks Can Help with Visual Reasoning?: Non-literal implication reasoning connects literal perception, abstract conceptualization, and final inference across disparate concepts.The paper links these abilities to formal logic, mathematical reasoning, and other complex visual reasoning benchmarks.
  • Why Image Implication Tasks Can Help with Visual Reasoning?: TFQ’s multiple fine-grained propositions train models to verify truth values rather than merely generate answers.The paper describes this as transforming the model into a “propositional verifier.”
  • Why Image Implication Tasks Can Help with Visual Reasoning?: TFQ-Data combines factual visual statements with implication statements, preserving perceptual accuracy while developing abstract inference.This design is intended to prevent ungrounded abstraction and support general VQA.

8. Conclusion

MetaphorStar addresses MLLM difficulty with non-literal image implication through TFQ resources and end-to-end visual RL. The paper reports stronger complex visual reasoning, identifies the SFT Curse, and releases its models, data, and code.

  • Conclusion: MetaphorStar combines TFQ-Data, TFQ-Bench, and TFQ-GRPO in an end-to-end visual RL framework for image implication.The released MetaphorStar models are trained with this framework.
  • Conclusion: Image implication training significantly enhances complex visual reasoning performance.The conclusion presents this as a central experimental insight.
  • Conclusion: The SFT Curse is attributed to an entropy bottleneck that harms generalization, whereas end-to-end RL is presented as more suitable for these tasks.The conclusion extends this claim to visual reasoning tasks.
  • Conclusion: The authors open-source the models, datasets, and code to support research beyond literal perception.The stated goal is deeper conceptual understanding in MLLMs.

A. Dataset Statistics

The dataset statistics section describes the source image collection and the construction of TFQ-Data-Lite and TFQ-Bench-Lite from representative metaphorical images.

  • Dataset Statistics: The source collection contains 1,434 high-quality metaphorical images spanning six domains and multiple visual formats.Domains include Life, Art, Society, Psychology, Environment, and Others; formats include illustrations, memes, posters, comics, logos, and paintings.
  • Dataset Statistics: TFQ-Data-Lite and TFQ-Bench-Lite are manually constructed from 50–100 high-quality, diverse, representative images.The passage describes these as manually constructed subsets.

B. Experiment Setup

The experiments compare MetaphorStar with diverse MLLMs across image-implication, reasoning, and multimodal-understanding benchmarks. Evaluation uses specified sampling settings, repeated GPT-4o scoring for OSQ, and human-model scoring consistency of 96.5% on OSQ.

  • Evaluation Settings: 96.5% human-model scoring consistency is reported for OSQ, while GPT-4o evaluation uses temperature 0 and averages more than three runs.TFQ and MCQ use temperature 0.5; OSQ uses temperature 0.7, with top p fixed at 0.9.
  • Main Experiment: Models spanning 7B to 300B parameters are selected to compare diverse open-source and closed-source MLLMs.The selection covers general and reasoning models across varying complexity and capability.
  • Main Experiment: The high-level English benchmark contains 50 manually constructed images and evaluates Multiple-Choice Question and Open-Style Question performance.The images include varied types such as illustrations and comics.
  • Generalization Experiment: Reasoning evaluation covers MathVision, MathVerse, WeMath, LogicVista, VisuLogic, VisualPuzzles, V*, ZeroBench, and MMMU.These benchmarks are selected to assess multimodal reasoning ability.
  • Generalization Experiment: Understanding evaluation covers SEEDBench, SEED-2-Plus, MMBench, MMStar, OCRBench, AI2D, ScienceQA, POPE, MMT-Bench, RealWorld QA, BLINK, HallusionBench, and MMVet Hard.The benchmarks assess perception and comprehension across text-and-image formats.
Loading 2602.10575v1…