Relational Visual Similarity

1. Introduction

• Relational similarity supports reasoning and creativity, including analogy and abstract thought, beyond attribute-based recognition and classification.The introduction gives the Earth–peach correspondence as an example of recognizing structure without shared visible attributes.Source: §1 · p. 2
• Current visual similarity frameworks focus on attributes and struggle to recognize shared relational logic, such as match burning and banana ripening.The match and banana share a gradual transformation over time, not a particular appearance.Source: §1 · p. 2
• Anonymous captions describe an image’s relational logic rather than its visible objects, providing a link between images with similar underlying structure.The example caption is “transformation of {subject} over time”.Source: §1 · p. 2
• The authors filter LAION-2B to 114k images, train an anonymous captioning model, then train relsim to bring together images with similar captioned abstractions.The filtering step targets low-quality, mislabeled, or relationally uninformative images.Source: §1 · p. 2
• The paper contributes a relational dataset and tuned metric, and demonstrates applications in image retrieval and generation.It also analyzes how relational and attribute similarity relate and evaluates limitations of existing similarity models.Source: §1 · p. 3

2. Related Works

• Relational similarity compares relationships between objects, complementing attribute similarity based on matching properties such as size, shape, and color.The section connects relational similarity to Gentner’s Structure-Mapping theory.Source: §2 · p. 3
• Modern metrics such as LPIPS [1], DINO [20], and CLIP [2] support deep-feature or semantic comparisons, but assume similarity is based solely on attributes.Earlier approaches include pixel-level metrics and handcrafted features; deep-feature metrics better align with human perceptual judgment.Source: §2 · p. 3
• The authors use VLMs because relational similarity requires understanding image content beyond what is directly depicted.The section contrasts explicit visual content with meanings such as parental care.Source: §2 · p. 3

3.2. Modeling Relational Visual Similarity

• Using InfoNCE, a visual extractor is trained against a frozen text encoder; image-pair scores track cosine similarity between their anonymous-caption embeddings.The training objective encourages visual features to align with relational concepts represented in the captions.Source: §3 · p. 3, §3.2 · pp. 4–5

3.1. Creating a Relational Dataset

• 93% agreement with human judgments was achieved by the image filter, which selected 114k relationally interesting images from LAION-2B.The filter was fine-tuned using 1.3k positive and 11k negative human-labeled examples.Source: §3.1 · p. 4
• Across 532 manually curated groups of 2–10 images, a frozen VLM drafts one object-anonymized shared caption, which humans verify before pairing with each image.The trained captioning model is then applied to all filtered images to produce the image-caption dataset.Source: §3.1 · p. 4

3.2. Modeling Relational Visual Similarity

• Vision-only encoders remain insufficient for relational similarity even after tuning, so the paper uses a VLM as its visual extractor.The authors say relational reasoning goes beyond visual recognition and cite the VLM’s higher-level semantic knowledge as a reason for its use.Source: §3.2 · p. 5

4.1. Settings

• The 114k-image dataset was split 100k/14k; evaluation used 1,000 queries against 28k-image databases and GPT-4o plus human judgments.The model uses a LoRA-tuned Qwen2.5-VL-7B-Instruct visual encoder.Source: §4.1 · p. 5
• Baselines included LPIPS, DINO, DreamSim, and CLIP-I, plus caption retrieval using CLIP-T and Qwen-T.The caption-based baselines used the original Qwen model to produce anonymous captions from individual images.Source: §4.1 · p. 5

4.2. Evaluations

• 6.77 was relsim's score, above CLIP-I (5.91), DINO (5.14), and LPIPS (4.56) on relational retrieval.The caption describes Figure 6 as also comparing tuned models; its reported score for relsim is the highest.Source: §4.2 · pp. 5–6, Figure 6 · p. 6
• Single-image caption baselines scored 5.33 and 4.86, as their captions often leaked semantic or attribute information and retrieval overfocused on semantics.The authors contrast these prompted captions with captions generated from groups of similar images to identify shared relational structure.Source: §4.2 · p. 6
• Tuned CLIP and DINO improved with anonymous-caption training but remained below the VLM-based model.The ablation uses the same training data and loss for the pure vision encoders; the authors attribute the remaining gap likely to VLMs' language-based world knowledge.Source: §4.2 · p. 6
• Combining relational and attribute similarity reveals matches with the same logic but different appearances, as well as matches sharing both.For the “A dog holding a camera” query, examples include other animals performing human-like activities; the authors suggest the two kinds of similarity are most powerful together.Source: Figure 7 · p. 7, §4.2 · p. 7
• Users preferred relsim across all baseline comparisons, with preference rates ranging from 42.5-60.7%.Figure 8 reports an A/B user study; gray bars indicate ties.Source: Figure 8 · p. 7, §4.2 · p. 7

Relational image retrieval

• Relational retrieval supports searches by interactions and functions when attribute-based matching fails, including for creative inspiration.One example is finding images that decorate food with human eyes.Source: §5 · p. 7, Figure 9 · p. 7

Analogical image generation

• Analogical generation transfers relational structures and conceptual ideas rather than merely changing surface attributes such as shape or texture.The benchmark used 200 manually collected image pairs with human-written instructions, arranged as input, instruction, and example-output triplets.Source: §5 · pp. 7–8, Figure 11 · p. 8
• 0.88 relsim was the highest score for logically similar outputs; CLIP was lowest at 0.66 and LPIPS highest at 0.60.The benchmark evaluates semantic, perceptual, and relational structure preservation between input and edited images.Source: §5 · p. 8, Table 2 · p. 8
• Open-source models tended to preserve visual similarity (CLIP: 0.8x) but trailed closed-source models on relational structure (relsim: 0.7x vs. 0.8x).The authors identify the results as evidence of a performance gap and a need for more challenging analogical-generation datasets for open-source training.Source: §5 · p. 8, Figure 10 · p. 8

• Relsim captures image logic and abstraction not effectively measured by existing attribute-based similarity metrics, with applications in visual exploration, image retrieval, and analogical image generation.Source: §6 · p. 8
• The anonymous captioning model is trained on 532 manually curated image groups that may be imperfect, biased, and unscalable, and it can produce biased or hallucinated captions.Source: §6 · p. 8
• An image may support multiple relational structures and valid mappings, and specifying the intended structure with text prompts remains an open question.Source: §6 · p. 9

• Hyperparameter details are referred to the project’s GitHub repository and Hugging Face Datasets page.Source: §7 · p. 11
• The interesting-image filter uses Qwen2.5-VL-7B-Instruct with LoRA, trained on 1.3k positive and 11k negative images, and keeps about 0.7% of images.Source: §7 · p. 11
• The single-image caption prompt asks for a concise, reusable caption capturing underlying logic, layout, structure, or creative concept, with placeholders for variations.The section includes a predicted-caption example in Figure 13.Source: §7 · p. 11, Figure 13 · p. 11
• The image-group caption prompt asks for a reusable description of shared logic with placeholders wherever image variations occur.Source: §7 · p. 11
• The retrieval judgment prompt scores whether two images share underlying logic and directs evaluators not to rely on visual or semantic similarity.Source: §7 · p. 12

• Additional image retrieval results are provided in Figures 14–15.Source: §8 · p. 12

• Annotators were proficient in English and familiar with computer vision, with PhD students or holders among them.Source: §9 · p. 12
• Three annotators selected interesting images from a random sample of 15k LAION-2B images, with 92% agreement among them.Source: §9 · p. 12
• Nine annotators proposed about 400 image groups after viewing 10 examples; three annotators verified each group, retaining only unanimously accepted, non-duplicate patterns, about 85% of proposed groups.Source: §9 · p. 12

Data Attributions

• All images used in the paper come from publicly available LAION-2B and are used for research purposes.The authors do not own the images and acknowledge the dataset creators and/or original copyright holders.Source: Data Attributions · p. 12
• Figure 14 presents additional image-retrieval results.Source: Figure 14 · p. 13
• Figure 15 presents additional image-retrieval results.Source: Figure 15 · p. 14