GeneCIS: A Benchmark for General Conditional Image Similarity

• The paper’s framing foregrounds the need to decide what things a machine treats as similar or equal.This framing is attributed to Karl Popper, 1963.Source: §1 · p. 1

• GeneCIS studies whether models can adapt to different similarity notions, including object focus, attributes, and changes specified by conditions.Different conditions can make different images the most similar to one reference.Source: §1 · p. 1, Figure 1 · p. 1
• Most supervised and self-supervised representation-learning methods assume one similarity notion, while generic conditional models are difficult to build because exhaustive annotations are impossible.Prior conditional approaches also commonly restrict domains or the set of conditions.Source: §1 · p. 1
• GeneCIS evaluates conditional retrieval over diverse similarity notions using an open-set, zero-shot benchmark built from four evaluation datasets.The benchmark is intended to test adaptation without fine-tuning.Source: §1 · p. 2
• The proposed training approach automatically mines conditional-similarity information from large-scale image-caption datasets rather than requiring exhaustive annotations.The authors report substantial gains over zero-shot baselines and validation on related image-retrieval benchmarks.Source: §1 · p. 2

• Deep metric learning, contrastive learning, and vision-language contrastive training encode fixed or unconditional notions of similarity.These approaches respectively emphasize category similarity, augmentation invariance, or image-caption alignment.Source: §2 · p. 2
• Conditional similarity networks use multiple subspaces but are trained with predefined conditions or evaluated in constrained domains.GeneCIS targets an open set of conditions and natural-image zero-shot evaluation instead.Source: §2 · p. 2
• Compositional learning and composed image retrieval compose visual information and conditions, but commonly use finite primitives, edit instructions, or specialized domains.GeneCIS also evaluates on natural-image CIR datasets and uses the Combiner architecture for additional comparisons.Source: §2 · pp. 2–3
• GeneCIS differs by training on open-world image-caption data and evaluating flexible conditional similarity zero-shot across natural-image conditions.This design avoids a closed set of similarity conditions or visual primitives.Source: §2 · p. 3, Figure 2 · p. 3

3. Conditional Similarity

• Conditional similarity is defined as f(I_T, I_R, c), scoring a target image against a reference given an explicit text condition for retrieval.Standard image similarity omits the explicit condition and therefore implicitly assumes one.Source: §3 · p. 3

3.1. Challenges in training and evaluation

• Possible similarity conditions are effectively infinite, so GeneCIS evaluates a broad subset of practical conditions rather than attempting exhaustive coverage.The paper treats zero-shot gains across the benchmark as evidence of more flexible adaptation.Source: §3.1 · p. 3
• Human annotation cannot feasibly cover the diversity of conditions, including cases requiring sensitivity to objects under some conditions and colors under others.The proposed solution leverages existing large-scale image-text datasets to learn an open set of conditions.Source: §3.1 · p. 3

4. The GeneCIS Benchmark

• GeneCIS combines object-versus-attribute focus with positive focus or specified changes, yielding four conditional retrieval tasks.The tasks are Focus on an Attribute, Change an Attribute, Focus on an Object, and Change an Object.Source: §4 · pp. 3–4
• Each task presents a reference image, text condition, and gallery with exactly one positive target to identify.The benchmark uses galleries of 10–15 images and spans over 400 attributes and 100 objects.Source: §4 · p. 4
• Distractors separately match the reference scene, condition object, object category, or attribute, forcing models to use the condition together with image content.This design targets models that rely only on the reference image or only on the condition.Source: §4 · p. 4
• The benchmark repurposes VAW for attribute tasks and COCO Panoptic Segmentation for object tasks.VAW supplies object crops and positive or negative attribute annotations, while COCO provides dense category annotations.Source: §4 · p. 4
• GeneCIS v0 remains noisy because underlying Visual Genome, VAW, and COCO inconsistencies require ongoing manual verification.The authors are collecting human annotations and verification for the benchmark templates.Source: §4 · p. 5

5.1. Preliminaries

• The conditional similarity model composes reference-image and condition-text features, then scores a target image by their dot product.Separate encoders produce image and text features, while a Combiner transforms the reference and condition into a joint representation.Source: §5.1 · p. 5
• Training optimizes a contrastive loss that raises the score of each paired target relative to other targets in the batch.The loss uses temperature τ over similarities between the combined reference-condition feature and batch target features.Source: §5.1 · p. 5

5.2. Scalable training for conditional similarity

• The method mines conditional-similarity triplets from large-scale image-caption data instead of exhaustively annotating the expanding space of conditions.A text-to-scene-graph parser extracts Subject → Predicate → Object relationships from captions.Source: §5.2 · p. 5
• It filters extracted entities by visual concreteness, then samples same-subject relationships with different objects to form reference, target, and condition triplets.The target condition is the concatenated Predicate and Object from the sampled target relationship.Source: §5.2 · pp. 5–6
• Although the mined triplets are biased toward Change an Object, their abstract relationships require using the condition to modify diverse information from the reference.This training requirement is presented as central to the broader conditional similarity problem.Source: §5.2 · p. 6

6.1. Baselines and Specific Solutions for GeneCIS

• GeneCIS evaluates Image Only, Text Only, Image + Text, Combiner (CIRR), and task-specific solutions under zero-shot conditions.Specific solutions use CLIP for attribute-based tasks and Detic for object-based tasks, while the CIRR Combiner is fine-tuned on manually collected CIRR data.Source: §6.1 · p. 6, §6 · p. 6

6.2. Implementation Details

• The strongest model is trained on 1.6M visually concrete triplets mined from CC3M with a ResNet50×4 CLIP initialization.Training uses contrastive loss with temperature τ = 0.01, batch size 256, and 28K gradient steps.Source: §6.2 · p. 6

6.3. Analysis on GeneCIS

• Our automatically mined-data model substantially outperforms all baselines on average across GeneCIS and at Recall@1 on every task, including the manually trained Combiner (CIRR).The benchmark reports Recall@K for K = {1, 2, 3} and Average R@1 across tasks.Source: §6.3 · pp. 6–7
• Most task-specific solutions outperform the general model on their own tasks, but they remain independently challenging and fail on some conditions.CLIP struggles with object attributes, while Detic struggles with ‘stuff’ categories in COCO Panoptic.Source: §6.3 · pp. 6–7
• The Image Only baseline slightly exceeds the proposed method at higher recalls on Focus Attribute, where standard image embeddings suit some conditions such as color.Combiner (CIRR) performs better on object-conditioned tasks because its multi-object image distribution aligns more closely with them.Source: §6.3 · p. 7

6.4. Comparisons to Prior Work

• Zero-shot evaluation on MIT-States surpasses state-of-the-art supervised methods, although those methods use smaller models than the pretrained CLIP backbone.MIT-States closely reflects GeneCIS’s ‘Change Attribute’ task.Source: §6.4 · p. 7, Table 3 · p. 7
• On CIRR, the model substantially outperforms comparable zero-shot baselines and exceeds some supervised methods, while narrowing the gap to specialist Combiner models.The improved supervised Combiner implementation remains a strong upper bound.Source: §6.4 · pp. 7–8, Table 4 · p. 7
• CIRR’s curated-gallery evaluation is omitted because Text Only performs comparably to state-of-the-art models there, exceeding 60% Recall@1.The reported comparisons instead use CIRR’s official test-server setting.Source: §6.4 · p. 7

Ablations

• Filtering non-visually concrete relationships and fine-tuning the entire backbone both strongly affect performance, while SBU Captions also works despite CC3M performing slightly better.These findings support the robustness of mining across different image-caption datasets.Source: §7 · p. 8

Comparing pretrained backbones

• A 10% Top-1 gain on ImageNet yields only a 1% improvement on GeneCIS, indicating weak correlation between ImageNet accuracy and conditional-similarity performance.The authors report that scaling the underlying CLIP model gives a smaller boost than their method.Source: §7 · p. 8, Figure 6 · p. 8

Scaling the number of triplets

• Increasing the number of mined triplets improves GeneCIS performance, both with and without concreteness filtering.The experiments reduce the 1.6M-triplet training set by successive factors of four.Source: §7 · p. 8, Figure 5 · p. 8
• The method’s substantial boost over Image + Text exceeds the gain from scaling the underlying CLIP model.The authors note that open-source caption datasets contain up to five billion images, suggesting room to scale training data further.Source: §7 · p. 8

8. Conclusion

• GeneCIS is evaluated zero-shot across an open set of similarity conditions, with training and evaluation datasets designed to cover diverse conditional retrieval use cases.The benchmark is built as sparse but broad coverage rather than exhaustive coverage of the conditional-similarity space.Source: §8 · p. 8

A.1. Task construction

• Attribute tasks use VAW annotations to construct same-object retrievals that either preserve or change attributes within a specified attribute type.Distractors separately test whether a model relies only on the condition or only on reference-image content.Source: §A.1 · p. 12
• Object tasks use COCO scenes, selecting targets and distractors according to shared object categories and conditioning objects so that both scene similarity and the condition matter.The target is designed to be conditionally most similar, rather than simply having the highest object-category overlap.Source: §A.1 · pp. 12–13

A.2. Implementation Details

• The attribute taxonomy contains 45 types and 663 constituent attributes, with tasks roughly balanced across types after manually filtering unsuitable categories.The taxonomy is refined from VAW for visually grounded retrieval templates.Source: §A.2 · p. 13
• Object crops are dilated before square padding to preserve context and object aspect ratio, while object-based datasets come from COCO Panoptic images and categories.The reported dilation factor was selected using the discrepancy between Image + Text and Image Only Recall@1.Source: §A.2 · p. 13

A.3. Dataset noise

• Annotation noise and subjective taxonomy boundaries can make GeneCIS templates ill-posed, especially for ambiguous object labels and attributes such as “wide” and “narrow.”The paper notes that similar ambiguity exists in widely used computer-vision datasets, including ImageNet [72].Source: §A.3 · p. 13
• Ten random seeds and ten cross-validation splits yield approximately 0.2% standard deviation in Recall@1 per task, while Recall@{1, 2, 3} accommodates potentially valid distractors.The authors present GeneCIS v0 while continuing manual template filtering and verification.Source: §A.3 · p. 13

A.4. Discussion on symmetry

• GeneCIS similarity is generally directional: Focus tasks are symmetric given their conditions, whereas Change tasks are not.A changed target can be similar to a reference under the condition without the reverse relation holding.Source: §A.4 · pp. 13–14

B. Specific Solutions

• Specific solutions use CLIP for attribute tasks and Detic for object tasks, while remaining tied to each task’s construction and underlying VAW or COCO taxonomy.These solutions are designed for zero-shot evaluation but are not general conditional-similarity models.Source: B. Specific Solutions · p. 14

C. Results with ViT-B/16 on GeneCIS

• With a ViT-B/16 backbone, the model outperforms all CLIP-only baselines on every GeneCIS task and at every recall.This removes the higher-recall Focus Attribute exception reported for the ResNet50×4 comparison.Source: C. Results with ViT-B/16 on GeneCIS · p. 14
• The Focus Attribute task is anomalous because its condition gives an attribute type rather than the specific attribute, which can confuse existing vision models.In this setting, Image + Text does not outperform the other baselines.Source: C. Results with ViT-B/16 on GeneCIS · p. 14

D. Combiner Architecture

• The Combiner maps reference-image and condition-text features into one vector using parallel functions for image, text, and joint representations.Its output combines image-only, text-only, and interaction terms.Source: D. Combiner Architecture · p. 14
• The weighting coefficient λ is computed dynamically from the reference-image and condition features.This allows the composition to adjust the relative contribution of the separate feature pathways.Source: D. Combiner Architecture · p. 14

E. Further Implementation Details

• Models trained on CC3M use 28K gradient steps, jointly tune the CLIP backbone and Combiner head, and select checkpoints by CIRR validation Recall@1 before zero-shot evaluation.Training uses Adam, cosine learning-rate decay, normalized features, and 16 A100 GPUs for approximately 12 hours.Source: E. Further Implementation Details · p. 14
• Table 6 reports that the ViT-B/16 version performs best and surpasses baselines at all recalls on every GeneCIS task.The table evaluates the proposed method alongside CLIP-only baselines.Source: Table 6 · p. 15

F. Extended Related Work

• Unlike text-based image editing, GeneCIS retrieves existing images rather than synthesizing new ones, and unlike related compositional methods, it targets open-set zero-shot conditional similarity.The related-work discussion contrasts GeneCIS with fixed augmentation concepts and constrained category or domain settings.Source: F. Extended Related Work · p. 15

G.1. GeneCIS examples

• Appendix examples display reference images, condition text, and curated galleries in which the positive target is the most similar image given the condition.Additional figures show model predictions, failures, and automatically mined CC3M training triplets.Source: §G.1 · p. 15, §G.2 · p. 15, §G.3 · p. 15