PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM: A Category-Theoretic Framework for Measuring and Refining Multimodal Analogies

Mirella Zeisler, Ojas Shirekar, Mircea Licǎ, Chirag Raman

arXiv:2610.01383v1cs.AI

TL;DR

Existing multimodal analogy approaches lack a unified, interpretable way to represent and assess relational structure across tasks and modalities. PRISM uses category-theoretic relational mappings, a pullback score, and iterative feedback-driven refinement. On 200 metaphors, refinement significantly improved metaphor consistency and analogy appropriateness according to VLM judges and human raters.

  • Problem

    Existing approaches lack a unified, human-interpretable representation of analogical structure across modalities and task types, while strong benchmark performance does not necessarily establish relational abstraction.

    Source: §1 · pp. 1–2

  • Method

    PRISM represents analogies as category-theoretic relational mappings, derives a pullback score from VLM-extracted domain graphs, and uses that score to guide iterative test-time image generation.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Across 200 metaphors, refinement yielded statistically significant improvements in metaphor consistency and analogy appropriateness according to VLM judges and human raters.

    Source: §7 · p. 9

  • Takeaways & Limitations

    In visual metaphor generation, the results support the pullback score as meaningful analogical signal and its use in a feedback-driven refinement loop.

    Source: §7 · p. 9

    Qualitative inspection found refinement can produce cluttered compositions, while literal text can earn high pullback scores without clearer visual correspondence.

    Source: §6 · p. 9

Abstract

from arXiv · show

Analogical reasoning involves identifying and preserving relational structures across domains. However, existing approaches to AI-driven multimodal analogy generation lack an interpretable measure of whether this structure is understood and maintained in the generated output. We address this gap with Pullback Refinement via Interpretable Structural Mapping (PRISM), a modality-agnostic framework for measuring and improving relational alignment in multimodal analogies, evaluated on visual metaphor generation. PRISM represents analogies as explicit relational mappings grounded in category theory and uses VLMs to instantiate these structures across modalities. Its first component, the pullback score, quantifies relational alignment from the resulting graph representation. On the AnaloBench benchmark, selecting the correct analogy purely by pullback score achieves 82.5% accuracy, demonstrating that the score captures meaningful relational information. PRISM's second component is an iterative refinement loop that uses the pullback score as an in-context feedback signal to iteratively revise the generated image towards greater relational depth. VLM-as-a-judge and human evaluations show that PRISM consistently improves metaphor consistency and analogy appropriateness over zero- shot generation, with human participants preferring the refined output in 57.65% of pairwise comparisons. However, a qualitative analysis reveals that refinement can favour visually crowded compositions rather than genuinely deeper relational correspondences.

1 INTRODUCTION

AI analogy systems can perform well without demonstrating relational abstraction, while existing structured approaches often remain task-specific. PRISM proposes an interpretable, modality-agnostic representation and pullback score, then uses score feedback to refine multimodal analogies.

1 INTRODUCTION

  • Analogical reasoning transfers relational structures across domains and supports learning, creativity, and generalisation.The paper frames this capability as important for AI systems used in open-ended, multi-domain settings.Source: §1 · p. 1
  • Existing AI systems can exploit training regularities rather than infer relational structure, and VLMs struggle to generalise relational rules across visual transformations.Even correct outputs or explanations may not correspond to interpretable analogical mappings.Source: §1 · p. 1
  • Structured prompting and schema-based approaches address specific tasks but do not generalise across task formulations.The stated gap is a unified, human-interpretable representation that works across modalities and task types while using foundation models.Source: §1 · pp. 1–2
  • PRISM represents analogies as explicit relational mappings grounded in category-theoretic constraints and uses VLMs to populate them across multimodal inputs.The paper evaluates the framework on visual metaphor generation.Source: §1 · p. 2
  • PRISM derives a pullback score to measure preserved relational structure and uses it as feedback in an iterative refinement loop.The framework aims to encourage deeper correspondences rather than superficial similarities.Source: §1 · p. 2

2 RELATED WORK

Prior work spans symbolic, neural, neuro-symbolic, and graph-based approaches to analogy, alongside methods for visual metaphor generation. PRISM extends these directions with an explicit graph scaffold and grounds refinement in a category-theoretic alignment measure.

2 RELATED WORK

  • Analogy research has used symbolic, neural, and neuro-symbolic systems, with newer work adding symbolic scaffolds to language and vision-language models.Graph-based approaches make entities, relations, and higher-order correspondences explicit.Source: §2 · p. 2
  • PRISM extends graph-based work with a formal graph-theoretic scaffold populated and used by VLMs, combining soft relational similarity with pullback operations.This formulation constructs and quantifies cross-domain correspondences.Source: §2 · p. 3
  • Visual metaphor methods have used prompt detail, explicit structures, iterative feedback, or image references, while PRISM uses an explicit category-theoretic measure of relational alignment for refinement.The cited prior approaches include heuristic or learned rewards and VLM-generated critiques.Source: §2 · p. 3

3 BACKGROUND

The background formalises analogies as structure-preserving correspondences between coloured domain graphs. The pullback restricts possible pairings to edges sharing a colour, and an analogy is a consistent subgraph of those matches.

3 BACKGROUND

  • An analogy maps entities and relations between coloured domain graphs, with edge colours shared when relations are similar enough to count as analogous.A pullback restricts the product of the two graphs to colour-matched edges; individual analogies are subgraphs, and the preferred mapping preserves the most relations.Source: §3 · p. 3
  • In the solar-system and Bohr-model example, shared node roles and edge colours mark analogous entities and relations across the domains.The figure illustrates the coloured-graph representation used to express correspondences.Source: Figure 2 · p. 3
  • The graph pair separates source and target entities and relations, assigns nodes semantic roles, and assigns edges colours from a shared set.Source and target edges remain structurally independent before the pullback operation.Source: §3 · p. 3
  • The pullback pairs source and target edges with the same colour, and an analogy is a subset whose endpoints define one consistent node correspondence.The paper states that this construction is developed into a computable score later.Source: §3 · p. 3

4 METHODOLOGY

PRISM extracts coloured domain graphs from metaphors, scores structural correspondence, and uses that score with graph feedback to revise generated images. The pullback score shows relational signal on AnaloBench, while the refinement loop supplies iterative test-time feedback.

4.1 THE PULLBACK FRAMEWORK FOR STRUCTURAL ANALOGY EXTRACTION AND EVALUATION

  • PRISM extracts source and target domains, entities, and relations into coloured graphs before computing the pullback score.The VLM pipeline incrementally conditions each stage on structured output from the previous stage.Source: §4.1 · p. 4, Figure 3 · p. 4
  • The score retains greedily matched edge pairs above a similarity threshold when they preserve one consistent node mapping, then adds a bonus for long cycle-free directed paths.Its similarity term measures relational correspondence, while the chain bonus rewards coherent relational chains.Source: §4.1 · p. 5
  • 82.5% accuracy comes from selecting AnaloBench analogies purely by pullback score, versus 88.5% for zero-shot GPT-4.1 Mini.Pullback selection also reaches 0.895 mean reciprocal rank and 91.5% top-2 accuracy; the correct answer never scores zero, supporting use of the score as a refinement signal despite lower standalone accuracy.Source: §4.1 · p. 5
  • Removing the chain bonus lowers AnaloBench accuracy to 76%, while removing semantic-role abstraction lowers it to 24%, near the 25% four-choice chance rate.The two approaches also have different error distributions: McNemar p = 0.0227.Source: §4.1 · p. 5

4.2 FEEDBACK-DRIVEN REFINEMENT OF ANALOGIES

  • At each refinement round, PRISM generates an image, extracts its graphs, computes structural feedback, and revises the prompt using the score and graph signals.Feedback includes structural quality, relational coverage, image-text alignment, matched pairs, and extracted graphs; the process ends at k rounds or score convergence.Source: §4.2 · p. 6

5 EXPERIMENTAL SETUP

The evaluation compares PRISM with zero-shot generators and iterative prompt optimization, using annotated metaphor datasets and both model-based and human ratings.

Baselines

  • Baselines include zero-shot outputs from four image generators and GEPA optimized with either the pullback score or CLIPScore.The four zero-shot generators are Flux.2 Klein 4B, Stable Image Core v1.1, GPT Image 1.5, and Ideogram 4.Source: §5 · p. 6

Dataset

  • The 250-example dataset combines 125 human-created visual metaphors with 125 textual metaphors from HAIVMet.The human-created subset includes advertisements, cartoons and artworks, awareness campaigns, and memes; metaphorical intent and source and target domains were annotated.Source: §5 · pp. 6–7

Metrics

  • Two VLM judges score metaphor consistency, analogy appropriateness, and conceptual integration; 80 participants compare zero-shot and refined outputs.Each human-reviewed metaphor is independently assessed by two participants, who rate five pairs and select which image better captures relational structure.Source: §5 · p. 7

Implementation

  • GPT Image 1.5 is the primary generator; tuning selects Claude Sonnet 4.6, k = 10, ε = 0.05, and θ = 0.40.The authors also rerun experiments with GPT-6 Astra to test applicability with different models.Source: §5 · p. 7

6 RESULTS

PRISM improves metaphor ratings and human preference over zero-shot generation, while comparisons and qualitative inspection reveal dependencies and failure modes.

Human evaluation

  • 57.65% of forced-choice comparisons favored PRISM-refined images; refined images also received higher scalar ratings more often than zero-shot outputs.Ties were mostly favorable, although a substantial proportion of metaphors favored zero-shot generation.Source: §6 · pp. 7–8, Figure 6 · p. 8

Qualitative comparison

  • Qualitative examples show clearer domains and relations, including smokestacks linking industrial emissions to global ice loss.Figure 5 presents examples involving global ice melting, thought, and the mind as a desert.Source: §6 · p. 7, Figure 5 · p. 7

Alternative model

  • With GPT-6 Astra, all three evaluation dimensions improve, but by smaller margins than with Claude Sonnet 4.6.The authors suggest higher Astra zero-shot scores may leave less room for improvement.Source: §6 · p. 8

Overall comparison

  • 7.87 overall: PRISM outperforms zero-shot baselines, GEPA variants, and PRISM ablations.Table 2 reports the overall average across all axes and both judges.Source: Table 2 · p. 8, §6 · p. 8

Ratings

  • PRISM improves metaphor consistency and analogy appropriateness over zero-shot generation, with larger gains than for conceptual integration.The comparison covers all axes in both human and VLM evaluations; human raters did not assess conceptual integration.Source: §6 · p. 7, Table 1 · p. 8

Evaluation caveat

  • Human and VLM ratings agree only moderately or weakly, and human agreement is low for absolute scores; relative improvement directions are more reliable.Human and VLM correlations are weaker for refined images; raters agree on the direction of improvement for 71.5% of metaphors in analogy appropriateness and 74.7% in metaphor consistency.Source: §6 · p. 8

Baselines and ablations

  • 6.76 overall (GEPA + CLIPScore) exceeds 6.62 (GEPA + pullback) but trails 6.83 (GPT Image zero-shot); PRISM ablations score 6.47 and 4.93.The 6.47 score follows single-stage extraction; removing domain guidance from refinement feedback yields 4.93, the largest degradation.Source: §6 · pp. 8–9

Failure modes

  • Refinement can clutter images or use written labels as shortcuts, allowing high pullback scores without clearer visual correspondence.The reported failure modes include redundant monetary cues and labels such as “social distancing” or “limit.”Source: §6 · p. 9, Figure 7 · p. 9

7 CONCLUSION

PRISM combines a category-theoretic pullback score with feedback-driven image refinement; evaluations report improved metaphor consistency and analogy appropriateness, while noting sample and reproducibility constraints.

7 CONCLUSION

  • Across 200 metaphors, refinement significantly improved metaphor consistency and analogy appropriateness in VLM-judge and human evaluations.The direction of improvement was replicated with GPT-6 Astra, but gains were smaller, indicating dependence on the models used.Source: §7 · p. 9

A.1 EXTRACTION PIPELINE PROMPTS

  • PRISM represents visual metaphors as source and target domain graphs, then uses the pullback score as feedback for iterative, test-time image generation.The extraction prompts identify domains, entities, and relations independently; the refinement prompt asks for both concepts and missing relationships to be made visible.Source: §7 · p. 9, §A.1 · pp. 17–19, §A.2 · p. 20

AI USE STATEMENT

  • The authors used generative AI for framework development, methodology, implementation, result interpretation, figure modification, literature review, and editing, and reviewed the AI-assisted work.The authors state that AI was not used to propose or refine hypotheses or to generate synthetic datasets.Source: AI USE STATEMENT · pp. 9–10

ETHICS STATEMENT

  • The human-evaluation sample was recruited through a survey exchange platform and is largely students and researchers, so it should not represent the general audience.The authors note that metaphor interpretation is subjective and culturally situated.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Reproduction is constrained because pipeline stages are probabilistic, image generation has no seed control, graph extraction varies across repeated calls, and many experiments use proprietary API models.Those models might be deprecated in favor of newer versions.Source: REPRODUCIBILITY STATEMENT · p. 10

A.2 ITERATIVE REFINEMENT PROMPT

  • PRISM’s refinement prompt asks for both metaphor domains to be recognizable in one scene and for missing relationships to be depicted, while retaining prompt elements that worked.The revised prompt is requested as a single paragraph under a word limit.Source: §A.2 · p. 20

B CAN LLMS EXTRACT AND POPULATE DOMAIN GRAPHS?

The authors tested whether an LLM judge could distinguish valid from altered SCAR domain graphs, then compared graph-extraction models with and without reasoning enabled.

B CAN LLMS EXTRACT AND POPULATE DOMAIN GRAPHS?

  • Claude Opus 4.6 judged altered and valid domain graphs with 88% accuracy and an F1 score of 0.867 on the manually annotated evaluation.Of 50 invalid graphs, it classified only one as valid; the authors report conservative errors and deem it suitable for this evaluation context.Source: B CAN LLMS EXTRACT AND POPULATE DOMAIN… · p. 20, Table 3 · p. 21
  • The judge evaluation used a random sample of 100 SCAR domains, with half altered by changing or removing edges to create invalid graphs.The authors used human annotations to evaluate whether the LLM judge distinguished valid from invalid graphs.Source: B CAN LLMS EXTRACT AND POPULATE DOMAIN… · p. 20
  • Gemma 4 with reasoning and Claude Haiku 4.5 with extended thinking achieved accuracies above 80% on graph extraction, with quality differing across models.The authors report that various models could extract valid graphs, especially with reasoning modules enabled.Source: B CAN LLMS EXTRACT AND POPULATE DOMAIN… · p. 20, Table 3 · p. 21

C CAN THE PULLBACK SCORE PROVIDE USEFUL SIGNALS IN ANALOGICAL REASONING?

The AnaloBench analysis examines pullback-score margins and failure cases, alongside the score’s greedy matching procedure and qualitative zero-shot examples.

C CAN THE PULLBACK SCORE PROVIDE USEFUL SIGNALS IN ANALOGICAL REASONING?

  • Correct pullback-score predictions generally had larger confidence margins than incorrect predictions, suggesting greater certainty when the score selects the correct analogy.For correct predictions, the margin is between the top two scores; for incorrect predictions, it is between the top wrong score and the true answer’s score.Source: C CAN THE PULLBACK SCORE PROVIDE… · p. 21, Figure 8 · p. 21
  • Qualitative failures suggest the pullback score may favor denser, more connected graphs, including incorrect candidates with more unmatched relations than the correct candidate.The authors identify this as a potential limitation and a direction for future refinement.Source: C CAN THE PULLBACK SCORE PROVIDE… · p. 22

D ALGORITHM DESCRIPTION

  • GREEDYPULLBACK sorts candidate edge pairs by similarity and accepts them in a single pass, subject to a node-consistency check.The algorithm description identifies this as a greedy matching heuristic.Source: D ALGORITHM DESCRIPTION · p. 22

F ZERO-SHOT GENERATION EXAMPLES

  • The appendix provides selected zero-shot examples, with each table row showing a source metaphor, its visual elaboration, and the generated image.Source: Table 4 · p. 25

G EXTENDED ZERO-SHOT RESULTS

Extended zero-shot results show that current generators rarely depict both intended metaphor domains, even when their images contain measurable relational structure. Retrieval accuracy also varies with how concretely the domains can be visualised.

Domain fidelity

  • Fewer than 10% of images recognisably depict both input domains in every condition; GPT-Image zero-shot reaches 9.4%, while either domain is recognisable in 59.2%.The appendix reports extended results for Flux.2 Klein, GPT-Image 1.5, and Stable Image Core.Source: G EXTENDED ZERO-SHOT RESULTS · p. 26, Table 5 · p. 26

CLIP retrieval

  • 84% image-to-text Top-1 accuracy for GPT-Image zero-shot exceeds Stable-Core chain-of-thought's 59%; GPT-Image zero-shot ranks highest in both retrieval directions.Zero-shot prompting outperforms chain-of-thought prompting for all three generators.Source: G EXTENDED ZERO-SHOT RESULTS · p. 26, Table 6 · p. 26

Relational structure

  • Pullback scores cluster between 1 and 4 and differ nonsignificantly from human scores except for Stable-Core zero-shot (pbonf = 0.040), but they measure extracted-domain alignment, not intended-analogy fidelity.Human-created metaphors have non-zero scores in 94.7% of cases, compared with 77.0–93.8% across generated conditions.Source: G EXTENDED ZERO-SHOT RESULTS · pp. 26–27, Figure 11 · p. 27

Metaphor properties

  • Concreteness uses the lower source or target rating (1–5) and correlates positively with retrieval accuracy in every condition (r = 0.186–0.276), while nearness has no significant corrected relationship.The reported correlations are small but consistent.Source: Table 7 · p. 27, G EXTENDED ZERO-SHOT RESULTS · p. 28

H MODEL SELECTION & HYPERPARAMETER TUNING

The tuning experiments select Claude Sonnet 4.6, k = 10, ε = 0.05, and θ = 0.40 as operating settings. These choices prioritise pullback performance while considering evaluation noise and computational cost.

Refinement model selection

Stopping criteria

  • The k = 15, ε = 0.05 setting scored 4.1079 versus 4.1000 for k = 10, ε = 0.05, but required 50% more iterations; k = 10 was selected for efficiency.Gains in pullback and VLM evaluation metrics were negligible across configurations with k ≥10.Source: H MODEL SELECTION & HYPERPARAMETER… · pp. 28–29
  • At k = 10, ε = 0.05 yields mean pullback 4.1000 versus 3.9788 for ε = 0.10; ε = 0.10 has slightly higher Metaphor Consistency and Analogy Appropriateness scores.The judge-score differences came from only 25 holdout metaphors and a preliminary three-judge panel; ε is the convergence threshold on pullback score.Source: H MODEL SELECTION & HYPERPARAMETER… · p. 29

Pullback matching threshold

  • θ was calibrated on 125 human-authored metaphors, not generated images; at θ = 0.55, zero scores reach 26.4%, and θ = 0.40 was selected as a conservative operating point.Increasing θ makes matching more restrictive, lowering mean pullback scores and increasing zero scores.Source: H MODEL SELECTION & HYPERPARAMETER… · p. 29, Table 9 · p. 29

I FULL QUALITATIVE COMPARISONS

The figure caption describes a comparison between four zero-shot generation pipelines and two PRISM versions; it supplies the systems and models but no outcome.

Zero-shot pipelines

  • The comparison includes four zero-shot pipelines: GPT-Image 1.5, Flux.2 Klein 4B, Stable Image Core, and Ideogram 4.Source: Figure 12 · p. 30

Comparison scope

  • The figure compares those four zero-shot pipelines against two versions of PRISM.Source: Figure 12 · p. 30

PRISM models

  • The two PRISM versions use Claude Sonnet 4.6 and GPT-6 Astra as extraction and refinement models.Source: Figure 12 · p. 30
Loading 2610.01383v1…