Source-linked AI summary
Beyond Pixels: Visual Metaphor Transfer via Schema-Driven Agentic Reasoning
Yu Xu, Yuxin Zhang, Juan Cao, Lin Gao, Chunyu Wang, Oliver Deussen, Tong-Yee Lee, Fan Tang
TL;DR
Existing generative models often preserve visual appearance without capturing the abstract logic that makes metaphors meaningful. The paper introduces Visual Metaphor Transfer with Schema Grammar and a closed-loop multi-agent pipeline, reporting improved metaphor consistency, analogy appropriateness, and conceptual integration over baselines. It also identifies a trade-off: deeply migrated metaphors can require substantial cultural and multi-step inference from viewers.
Problem
Existing generative models remain largely focused on pixel-level alignment and surface appearance, failing to capture the abstract logic required for genuine metaphorical generation.
Method
The method formalizes Visual Metaphor Transfer with Schema Grammar and uses perception, transfer, generation, and diagnostic agents in a closed-loop pipeline.
Results
The design improves metaphor consistency, analogy appropriateness, and conceptual integration compared with baselines.
Takeaways & Limitations
The framework supports automated visual metaphor transfer by extracting reference logic, finding a new carrier, generating the target image, and backtracing failures.
Takeaways & Limitations
Some migrated metaphors impose excessive cognitive demands, especially when they depend on cultural knowledge or long chains of inference.
Abstract
from arXiv · showhide
A visual metaphor constitutes a high-order form of human creativity, employing cross-domain semantic fusion to transform abstract concepts into impactful visual rhetoric. Despite the remarkable progress of generative AI, existing models remain largely confined to pixel-level instruction alignment and surface-level appearance preservation, failing to capture the underlying abstract logic necessary for genuine metaphorical generation. To bridge this gap, we introduce the task of Visual Metaphor Transfer (VMT), which challenges models to autonomously decouple the "creative essence" from a reference image and re-materialize that abstract logic onto a user-specified target subject. We propose a cognitive-inspired, multi-agent framework that operationalizes Conceptual Blending Theory (CBT) through a novel Schema Grammar ("G"). This structured representation decouples relational invariants from specific visual entities, providing a rigorous foundation for cross-domain logic re-instantiation. Our pipeline executes VMT through a collaborative system of specialized agents: a perception agent that distills the reference into a schema, a transfer agent that maintains generic space invariance to discover apt carriers, a generation agent for high-fidelity synthesis and a hierarchical diagnostic agent that mimics a professional critic, performing closed-loop backtracking to identify and rectify errors across abstract logic, component selection, and prompt encoding. Extensive experiments and human evaluations demonstrate that our method significantly outperforms SOTA baselines in metaphor consistency, analogy appropriateness, and visual creativity, paving the way for automated high-impact creative applications in advertising and media. Source code will be made publicly available.
1. Introduction
The paper introduces Visual Metaphor Transfer to move beyond appearance-focused image editing by preserving abstract metaphorical logic across target subjects. It proposes a schema-driven multi-agent framework that operationalizes Conceptual Blending Theory and reports stronger metaphor consistency, analogy appropriateness, and visual creativity than baselines.
- Existing text-to-image and image-to-image models primarily align instructions at the pixel level or preserve surface appearance, rather than modeling abstract metaphorical logic.The cited limitations include preserving style, texture, or subjects while failing to capture deeper relational structure.
- Visual Metaphor Transfer deconstructs a reference image’s creative essence and re-materializes its abstract logic onto a user-specified target subject.Unlike subject customization or style transfer, VMT focuses on transferring metaphorical structure rather than visual appearance.
- VMT requires structured extraction of domain-independent relational invariants and autonomous discovery of a new carrier that preserves the original cognitive tension.These challenges motivate a shift from passive pixel synthesis toward active, agentic visual reasoning.
- The proposed framework operationalizes Conceptual Blending Theory through Schema Grammar and four specialized agents for perception, transfer, generation, and hierarchical diagnosis.The diagnostic agent backtraces failures across abstract logic, component selection, and prompt encoding.
- The method outperforms existing baselines in metaphor consistency, analogy appropriateness, and visual creativity.The contribution summary identifies these three evaluation dimensions as the principal experimental outcomes.
2. Related Work
Related work distinguishes metaphor interpretation from synthesis and emphasizes that appearance preservation is insufficient for symbolic meaning. Multimodal and multi-agent approaches provide relevant foundations, while the proposed framework adds iterative targeted refinement for more faithful metaphor transfer.
- Visual metaphors communicate abstract concepts through symbolic imagery rather than literal visual similarity.This distinguishes metaphor understanding and generation from subject customization and style transfer.
- Vision-language models struggle more with metaphor understanding than with literal images, motivating methods that model abstract symbolic relationships.Prior work includes prompting techniques that link metaphor text to visual concepts.
- Multimodal large language models support reasoning across vision and language, and related image-generation systems decompose complex objectives into specialized subtasks.These developments establish a foundation for multi-agent image-generation frameworks.
- The framework evaluates outputs from previous agents and performs targeted refinement, enabling iterative improvement toward more faithful metaphor transfer aligned with intended symbolic meaning.This connects multi-agent decomposition with closed-loop correction.
3. Computational Modeling of Visual Metaphors
The paper models visual metaphors as structured blends of mental spaces and formalizes their transferable logic with Schema Grammar. This representation separates relational structure from concrete entities, enabling target-specific carrier discovery and schema instantiation.
- Conceptual Blending Theory: Conceptual Blending Theory describes metaphors as dynamic integrations of four mental spaces, including input spaces and a generic space of domain-independent relational invariants.The generic space captures relations shared across the input domains.
- Schema Grammar: Schema Grammar maps the blending spaces into a structured 7-tuple representation for computational metaphor reasoning.The representation is introduced as G = {S, C, AS, Aes, G, V, I}.
- Framework architecture: The framework architecture maps a reference metaphor and structured graph Gref to a target context and graph Gtgt through specialized agents and feedback.The agents are perception, transfer, generation, and diagnostic components.
- Schema components: The schema represents entity instantiation through a subject, carrier, inherent attributes, and additional attributes used to characterize cross-domain embedding.The subject is the primary entity, while the carrier supplies the visual or metaphorical context.
- Schema components: Decoupling relational logic from specific entities allows the metaphor’s creative essence to be manipulated as a structured representation.The relational components include generic structure, violations, and emergent meaning.
- Transfer formalization: Successful transfer preserves the reference’s abstract relational logic while discovering a contextually appropriate target carrier and violations.This reframes VMT as structured search and instantiation in Schema Grammar space rather than pixel-level reconstruction.
4. Method
The framework decomposes visual metaphor transfer into four agentic stages, using Schema Grammar to preserve abstract relational logic while adapting carriers and targets. A diagnostic agent then backtracks through prompt, transfer, and extraction errors to refine the final image.
- The pipeline comprises Perception, Transfer, Generation, and Diagnostic Agents for schema extraction, cross-domain synthesis, visual realization, and iterative refinement.
- The perception stage uses VLM-guided reasoning to identify entities, abstract the Generic Space, derive Violation Points, and infer Emergent Meaning.
- Schema Grammar isolates domain-independent relational invariants so the same abstract logic can be re-instantiated with different subjects and carriers.
- The transfer stage profiles the target, maps a cross-domain carrier sharing the Generic Space relationship, synthesizes violations, and aligns the emergent meaning.
- The generation stage translates the target schema into a high-fidelity prompt using structural anchoring, semantic juxtaposition, and affective encoding before image synthesis.
- The diagnostic agent evaluates subject salience, violation realization, relational coherence, and meaning alignment, then performs hierarchical backtracking until quality is satisfactory or the iteration limit is reached.
5. Experiments
Experiments evaluate the framework on a heterogeneous collection of visual metaphors against multimodal image-generation baselines using VLM-based assessment. Qualitative comparisons indicate stronger abstraction and metaphorical alignment than competing models.
- The study evaluates the framework across qualitative, quantitative, human, ablation, and generalizability analyses.
- Baselines include BAGEL-thinking, Midjourney-imagine, GPT-Image-1.5, and Gemini-banana-pro.
- The dataset contains 126 visual metaphors spanning product ads, memes, film posters, comics, and other creative works.
- Three frontier VLMs assess generated images across multiple dimensions because metaphor transfer requires high-level conceptual and analogical evaluation beyond low-level visual similarity.
- The implementation uses Gemini-3-pro for VLM and LLM roles, Banana-pro for image generation, and sets the refinement threshold τ to 5.
- Qualitative comparisons report that the method decodes abstract metaphoric logic and re-materializes it in novel targets, whereas baselines often perform surface substitutions or lose alignment.
5.3. Quantitative comparisons
The method outperforms baselines in quantitative and human evaluations, particularly in metaphorically appropriate carrier selection, while preserving visual quality and user preference.
- 65 participants evaluated outputs across five Likert dimensions, including metaphor recognizability, ingenuity, violation appropriateness, visual integration, and visual quality.Each participant assessed 100 images generated by the proposed method and four baselines.
- 4.57 MI and 4.45 VA were achieved, indicating stronger creative and purposeful metaphorical designs than Banana-pro and GPT-Image.The method also achieved 4.64 VI and 4.77 VQ, showing that conceptual reasoning did not compromise aesthetic fidelity.
- Over 60% “Ours Better” ratings were obtained against every baseline in pairwise GSB evaluation.The method was preferred over GPT-Image in 63.54% and Banana-pro in 61.85% of cases, with fewer than 10% inferior judgments.
- The method surpassed Midjourney and BAGEL with 71.54% and 76.15% “Ours Better” ratings, respectively.These pairwise preferences were reported as evidence of stronger metaphorical resonance and conceptual effectiveness.
5.5. Ablation study
Ablations show that CBT, preparatory reasoning phases, and diagnostic refinement each contribute to metaphor comprehension, carrier selection, and structurally faithful synthesis.
- Removing CBT produces plausible but generic outputs that lack structural creativity and complex carrier migration, with a significant AA decrease.The ablation discards source-to-recipient composition, uses a disjointed cactus analogy, and weakens action-consequence causality.
- Removing Phase 4 causes perception, transfer, and generation failures, with quantitative decreases across various scores.The full model instead selects a battery, finds a precise carrier, and enforces structural constraints.
- The full model correctly identifies the battery metaphor, selects rope for textural analogy, and enforces dual-panel structures.These examples illustrate the distinct contribution of the framework’s reasoning and diagnostic components.
5.6. Generalizability analysis
The framework remains metaphorically consistent across different reasoning and rendering backbones, while allowing language models to vary the narrative interpretation.
- Two reasoning backbones, Gemini and GPT-4, were paired with three T2I generators: Nano-Banana-Pro, GPT-Image, and FLUX.This evaluation tested the framework’s robustness and model-agnostic nature.
- Across T2I models, the LEGO scenario preserved the logical contrast between disorganized effort and total inaction.The framework rendered these concepts as a chaotic LEGO pile and an empty baseplate, respectively.
- Gemini and GPT-driven variants generated different forest-protection narratives while maintaining visual coherence.Gemini produced active retaliation scenes, whereas GPT produced cultural satire; all T2I models faithfully rendered the resulting prompts.
- The framework separates creative flexibility in reasoning from high-fidelity visual execution in generation.The reported backbone combinations preserve metaphorical structure while permitting divergent narrative interpretations.
5.7. Badcase
Some migrated metaphors impose high cognitive demands, trading semantic depth for immediate interpretability.
- Culturally specific and multi-step metaphors can increase viewer decoding effort and hinder instantaneous communication.Examples include the Achilles’ Heel band-aid allusion and the starved Siren headphone metaphor.
- Without the required mythological context, the Achilles’ Heel metaphor may be reduced to a literal historical injury and lose persuasive power.
5.8. Applications
The framework supports visual-metaphor applications in advertising and meme generation, handling both reference-guided and text-guided creative workflows. It transfers underlying logical mechanisms and satirical intent while preserving metaphorical structure and visual coherence.
- Commercial product advertisements: The framework supports commercial advertisements by mapping product attributes onto novel creative carriers.It accepts both text-based descriptions and image-based references as promotional-design inputs.
- Meme generation: The framework demonstrates potential for automated internet-meme generation, where communicative impact and humor rely on visual metaphors and cognitive dissonance.
- Commercial product advertisements: It flexibly handles both visual-to-visual and text-to-visual creative workflows.
- Meme generation: Figure 10 presents meme generation without a reference image.
- Meme generation: Generic Space extraction transfers meme templates’ logical mechanisms and satirical intent to new target entities while preserving metaphorical structure.The resulting Category Violations enhance visual wit and compositional coherence.
6. Conclusions
The paper introduces Visual Metaphor Transfer as a task for extracting metaphor logic from a reference and re-instantiating it on a new subject. Its Schema Grammar and closed-loop multi-agent pipeline improve metaphor consistency, analogy appropriateness, and conceptual integration compared with baselines.
- Visual Metaphor Transfer extracts underlying metaphor logic from a reference image and re-instantiates it on a user-specified subject.
- Schema Grammar formalizes metaphor structure, while four agents perform schema extraction, carrier transfer, prompt generation, and hierarchical failure backtracing.
- The design improves metaphor consistency, analogy appropriateness, and conceptual integration compared with baselines.