PaperlayerView exact revision on arXiv

Source-linked AI summary

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Netanel Y. Tamir, Michael Green, Issar Tzachor, Or Shimshi, Dvir Samuel, Or Patashnik, Rami Ben-Ari

arXiv:2610.22602v1cs.CV

TL;DR

Existing benchmarks offer limited coverage of attribute–object binding, leaving performance across modality pairings and more complex settings unclear. The paper introduces SABER and uses text-only binding supervision to improve retrieval, reporting gains across SABER and related benchmarks.

  • Problem

    Existing attribute-binding benchmarks are limited in modality coverage, vocabulary, and bindings per sample, leaving model robustness across pairings and complex settings unclear.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    The paper introduces SABER, a controlled benchmark that varies attribute–object assignments, and applies targeted text-only finetuning for binding.

    Source: §1 · pp. 2–3, §8 · p. 9

  • Results

    Text-only finetuning achieves the best overall SABER performance across all four modality pairings, outperforms existing models on all 9 tested attribute-binding benchmarks, and improves negation and instance retrieval.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings suggest compositional precision is a distinct dimension of embedding quality that warrants explicit evaluation and optimization.

    Source: §8 · p. 10

    Attribute-binding performance should not be interpreted as evidence of broader fairness, safety, or deployment suitability.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. One key compositional challenge is attribute--object binding: encoding which attributes belong to which objects. Existing benchmarks provide a limited view of this capability, typically covering a subset of modality pairings, simple two-object swaps, or small attribute sets. Consequently, it remains unclear how robustly current models preserve attribute--object bindings across different modality pairings and more complex settings. We therefore introduce SABER, a controlled benchmark for evaluating attribute--object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects. Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders. These findings motivate us to ask how binding can be improved efficiently. While multimodal supervision may be effective, collecting suitable images is difficult and costly. Text, on the other hand, is well suited to binding supervision: it is cheap to generate, supports a broader range of attributes and enables precisely controlled hard negatives. We thus finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. Despite the absence of image supervision, the gains transfer to cross-modal and image--image retrieval while preserving general multimodal embedding performance. Our model achieves SOTA results on SABER while also improving on negation and instance-retrieval benchmarks, demonstrating broader benefits of our binding-focused supervision.

1 INTRODUCTION

Reliable multimodal retrieval requires embeddings to represent not only which concepts occur but how attributes bind to objects. The paper introduces SABER to test these bindings and reports that text-only finetuning improves binding across modalities and other retrieval benchmarks.

1 INTRODUCTION

  • Existing benchmarks often restrict modality pairings, attribute vocabularies, or evaluations to simple two-object swaps, leaving the generality of binding failures unclear.The unresolved scope includes modality, scene complexity, and attribute type.Source: §1 · p. 2
  • SABER is a controlled benchmark that holds objects and attributes fixed while varying their assignments to isolate attribute–object binding.It supports unimodal and cross-modal retrieval across images and text, diverse vocabularies, and scenes with up to ten objects.Source: §1 · pp. 2–3
  • SABER evaluation finds substantial binding failures in both CLIP-style dual encoders and newer MLLM-based embedders across all four modality pairings and scene complexities.These results motivate testing whether targeted binding supervision can reduce the failures.Source: §1 · p. 3
  • Synthetic text offers explicit, inexpensive binding supervision with controllable hard negatives, whereas constructing controlled images is difficult and costly.The authors finetune a pretrained MLLM-based embedder exclusively on synthetic attribute-binding text.Source: §1 · p. 3
  • Text-only finetuning achieves the best overall SABER performance, outperforming evaluated baselines across all four modality pairings despite no image supervision.The gains also extend to all nine tested attribute-binding benchmarks, negation, and instance retrieval while largely preserving general multimodal embedding performance.Source: §1 · p. 3

2 RELATED WORK

Prior work develops multimodal embedding models and probes compositional failures, while existing attribute-binding benchmarks vary in scope and design. The paper positions text-only controlled supervision and broader benchmark coverage as responses to these gaps.

2.1 MULTIMODAL EMBEDDING MODELS

  • Dual encoders such as CLIP and ALIGN established contrastive image–caption embedding, while newer MLLM-based embedders support diverse multimodal retrieval settings.Examples include VLM2Vec, LamRA-Ret, and Qwen3-VL-Embedding; the paper evaluates both dual encoders and MLLM-based embedders for binding.Source: §2.1 · p. 3

2.2 COMPOSITIONALITY IN VISION–LANGUAGE REPRESENTATIONS

  • Compositionality studies report bag-of-words behavior and use hard negatives, while mitigation work targets training signal, data quality, or limitations of global embeddings.The paper instead uses synthetic text-only supervision to manipulate bindings precisely at scale without controlled image generation or changing the single-vector architecture and inference procedure.Source: §2.2 · pp. 3–4

2.3 ATTRIBUTE-BINDING BENCHMARKS

  • Existing attribute-binding benchmarks differ in modality coverage and control, with some limited to image–text retrieval and others using restricted vocabularies or rendered scenes.SABER is positioned against benchmarks including natural-image, synthetic-image, color-binding, and CLEVR-based evaluations.Source: §2.3 · p. 4, Table 1 · p. 4

3 SABER BENCHMARK

SABER addresses limits in prior benchmark coverage with controlled binding questions that support both multiple-choice and large-gallery evaluation. Its construction varies attribute assignments while maintaining the objects and attributes under comparison.

3 SABER BENCHMARK

  • Prior benchmarks limit modality coverage, vocabulary, bindings per sample, or large-gallery evaluation with multiple valid positives, leaving only a partial view of binding ability.SABER is introduced to address these limitations.Source: §3 · pp. 4–5
  • SABER covers all four image–text modality pairings, uses 232 everyday objects and 50 visual attributes, and scales to scenes with up to ten objects.Its four-way multiple-choice candidates can also be pooled for large-gallery retrieval.Source: §3 · p. 5
  • Each SABER question has one candidate preserving the query bindings and three hard negatives that alter assignments, generally keeping objects and attributes constant.This isolates binding from other challenges such as object recognition, except for one negative in the two-object setting.Source: §3 · p. 5
  • SABER contains 200 multiple-choice questions per scene size from two to ten objects, totaling 1,800 queries and 7,200 candidates.Half of the questions are organized into nested chains, where each successive query adds one binding while preserving earlier ones.Source: §3 · p. 5, Figure 3 · p. 5
  • The benchmark evaluates four-way multiple-choice ranking and retrieval against all 7,200 candidates, counting any candidate containing every query binding as positive.Nested-chain queries can have multiple positives, reflecting retrieval of queried bindings within larger scenes.Source: §3 · pp. 5–6

4 TEXT-ONLY FINETUNING FOR ATTRIBUTE BINDING

The paper evaluates existing embedders, then uses synthetic text-only triplets to train an MLLM-based embedder for attribute binding. The approach is motivated by precise, scalable text supervision and a proposed route for transfer to image representations.

4 TEXT-ONLY FINETUNING FOR ATTRIBUTE BINDING

  • Existing CLIP-style dual encoders and recent MLLM-based embedders show clear binding failures on SABER, motivating targeted finetuning.The evaluation includes compositional and attribute-binding variants as well as newer MLLM-based models.Source: §4 · p. 6
  • Synthetic text-only supervision makes bindings explicit and enables controlled hard negatives without incidental variation from generated images.The training set comprises 200,000 triplets with queries specifying two or three attribute–object pairs and negatives that permute attributes or objects.Source: §4 · p. 6
  • The training objective encourages a binding-preserving positive to be closer to a query than a binding-conflicting negative across all text and image modality pairings.The defined query, positive, and negative may each be text or image, and similarity is cosine similarity in the shared embedding space.Source: §4 · p. 6
  • Text-only optimization can affect image representations because MLLMs share parameters across modalities; the authors finetune Qwen3-VL-Embedding models and apply LiNeS post hoc.The appendix reports additional ablations, backbone experiments, and a comparison showing stronger cross-modal retrieval from textual than image supervision.Source: §4 · p. 6

5 EXPERIMENTS

Text-only binding finetuning improves SABER performance across modality pairings and retrieval settings, transfers to existing binding benchmarks, and largely preserves general multimodal performance.

5.1 CAN TEXT-ONLY FINETUNING IMPROVE ATTRIBUTE BINDING ACROSS MODALITIES?

  • 84.2% mean SABER-MCQ accuracy versus 52.5% for Qwen3-VL-Embedding-8B; the finetuned 8B model leads all four pairings in both protocols.Its text–text accuracy is 99.3%.Source: §5.1 · p. 7, Table 2 · p. 7
  • 19.2 points higher mean mAP than Qwen3-VL-Embedding-8B, the finetuned 8B model performs best in independent retrieval.In the 7,200-candidate gallery, CLIP-based models retrieve unrelated scenes in many queries, especially for cross-modal pairings.Source: §5.1 · p. 7
  • 54.1% mean mAP versus 38.3% for Qwen3-VL-Embedding-8B in nested retrieval, where containment remains a distinct challenge.Candidates containing all query bindings plus extras outrank exact matches in 33–39% of comparisons, on average across modalities.Source: §5.1 · p. 7

5.2 DO THE GAINS GENERALIZE BEYOND SABER?

  • Across 12 settings on nine existing benchmarks, the 8B model beats every baseline by 15.4 points on average, while the 2B model gains 8.2 points over its pretrained counterpart.The gains extend beyond synthetic scenes and templated text to varied images, text sources, and retrieval settings.Source: §5.2 · pp. 7–8

5.3 HOW MUCH GENERAL MULTIMODAL PERFORMANCE IS LOST AFTER FINETUNING?

  • Aggregate MMEB-v3 performance changes by only −0.5 points for 2B and +0.5 points for 8B, despite no MMEB-v3 finetuning data.The evaluation spans 182 image, video, visual-document, text, and agent tasks.Source: §5.3 · p. 8, Table 4 · p. 8

6 FROM ATTRIBUTE BINDING TO NEGATION

Binding-focused finetuning improves negation understanding, especially for hybrid statements that require assigning negation to the correct object. Nearly all gains persist after removing negation-containing training examples.

6 FROM ATTRIBUTE BINDING TO NEGATION

  • Finetuning improves NegBench accuracy by 9.3 points on COCO and 10.5 points on VOC 2007, driven mainly by hybrid cases.Hybrid choices can share the same words while assigning negation to different objects, resembling attribute-binding judgments.Source: §6 · p. 9, Table 5 · p. 9
  • The negation-filtered variant retains nearly all gains, suggesting transfer from binding supervision rather than direct exposure to negation terms.The removed terms appeared in 7.1% of finetuning examples.Source: §6 · pp. 8–9
  • The Qwen3-VL-Reranker-8B remains stronger on pure-negation cases, but is comparable overall to the finetuned model.It serves as an additional reference because it is a stronger pairwise model.Source: §6 · p. 9

7 BROADER BENEFITS: INSTANCE-LEVEL RETRIEVAL

Binding-focused finetuning also improves instance retrieval, where attributes can distinguish individual objects within a category. Gains appear across all evaluated settings and fine-grained categories.

7 BROADER BENEFITS: INSTANCE-LEVEL RETRIEVAL

  • The finetuned model improves over its pretrained counterpart by 2.1–9.9 mAP across eight image–image settings and one text–image setting.The evaluation covers ILIAS, INSTRE, PODS, Paris6K, Oxford5K, and Sculptures6K.Source: §7 · p. 9, Table 6 · p. 9
  • Within ILIAS fine-grained categories, the model improves over its pretrained counterpart in all 38 categories, with mean mAP increasing by 8.5 points.Instances may be distinguished by attributes such as shirt color.Source: §7 · p. 9

8 DISCUSSION

The paper presents SABER and text-only binding supervision as tools for evaluating and improving attribute–object composition. Its appendix details how the benchmark and training data are generated, varied, and validated.

8 DISCUSSION

  • The authors report that SABER is synthetic, and generated images may contain unintended variation in appearance, layout, object size, or orientation.They also state that results should not be interpreted as evidence of broader fairness, safety, or deployment suitability, and may inherit biases from pretrained models, generative models, and datasets.Source: §8 · p. 10, ETHICS STATEMENT · p. 10
  • SABER varies attribute–object assignments while holding the concepts fixed, testing image and text retrieval across broad vocabularies and scenes of up to ten objects.Its scenes use 232 objects and 50 visual attributes grouped into six semantic categories.Source: §A.1 · p. 15, Table 7 · p. 15

A.2 EXAMPLE STRUCTURE

  • SABER candidates include binding-preserving, attribute-swap, object-swap, and joint-swap assignments, with distinct assignments enforced for scenes containing at least three objects.For two-object cases, a fourth candidate instead preserves one binding and replaces an element in the other.Source: §A.2 · p. 15

A.3 DUAL TEXTUAL REALIZATION

  • Each example has a basic text form and a fully instantiated generation form, reusing each attribute realization across candidates to limit unrelated appearance variation.Images are generated as separated objects on a neutral background, horizontally flipped with probability 0.5, and manually screened for correct bindings.Source: §A.3 · p. 16, §A.4 · p. 16

A.5 BENCHMARK SUBSETS

  • SABER includes 200 multiple-choice examples at each object count from two to ten, split between independent samples and nested chains that retain earlier bindings.Nested chains add one binding at a time, making earlier binding-preserving candidates additional positives in large-gallery retrieval.Source: §A.5 · p. 16

B.1 GENERATION SETUP

  • Text-only finetuning examples pair randomized binding queries with contextualized positives and hard negatives that permute attributes or objects while preserving the same words.The 100,000-example dataset uses seven construction types, including structures where bindings must be recovered from sentence structure rather than local word co-occurrence.Source: §B.1 · p. 16, §B.2 · p. 17, Table 8 · p. 17

B.2 CONSTRUCTION TYPES

  • Training-data diversity controls vary contexts, writing styles, length, binding distance, distractors, and vocabulary, while validation checks that each example has its intended binding structure.The construction types include pair swaps, triple-bind variants, attribute bundles, predicative forms, and two deferred-bind forms.Source: §B.2 · p. 17

B.5 FINETUNING AND EVALUATION

The finetuning setup holds out both deferred-bind constructions to test generalization to unseen binding structures. The finetuned models outperform baselines on the held-out test and both deferred-bind evaluations.

B.5 FINETUNING AND EVALUATION

  • Both deferred-bind construction types are held out entirely, while the remaining 80,000 examples are randomly split 80/10/10 into training, validation, and test sets.Each example yields two triplets, producing 40,000 held-out deferred-bind triplets.Source: §B.5 · p. 19
  • Finetuning uses LoRA adapters in decoder layers while keeping the vision tower frozen, with a one-epoch triplet-loss training procedure.The 8B model has 3.8M trainable parameters; training uses a single B200 GPU and a 5×10^-5 learning rate.Source: §B.5 · pp. 19–20
  • Triplet accuracy measures how often the positive is closer to the query than the negative, and the held-out deferred-bind tests evaluate generalization to structures unseen during training.Chance performance is 50%.Source: §B.5 · p. 20
  • 93.6% triplet accuracy for Ours-8B versus 60.4% for pretrained Qwen3-VL-Embedding-8B on the held-out test set; gains also appear on both unseen deferred-bind types.Ours-8B rises from 53.8% to 72.9% on deferred bind anaphoric and from 50.8% to 56.6% on deferred bind respective.Source: §B.5 · p. 20, Table 10 · p. 20

C.1 TRAINING DATA

The appendices examine training-data overlap and format, the LiNeS trade-off, and text-side attention patterns. Binding gains generalize beyond lexical overlap, while LiNeS and direct attention supervision expose trade-offs or limited additional benefit.

C.1 TRAINING DATA

  • Even without SABER vocabulary overlap, finetuning improves accuracy across all four SABER pairings; lexical overlap adds only modest gains, while MMEB-v3 scores remain nearly unchanged.NoOverlap improves text–text from 53.7 to 99.3, text–image from 34.9 to 65.7, image–text from 36.3 to 71.8, and image–image from 84.9 to 93.1.Source: §C.1 · p. 20, Table 11 · p. 21
  • Mixed-length training raises mean SABER accuracy by 1.4 points, but the authors caution that closer alignment with SABER’s templated format may partly explain the gains.MMEB-v3 remains at 55.2.Source: §C.1 · pp. 20–21

C.2 EFFECT OF LINES

  • Without LiNeS, mean SABER accuracy is 9.4 points higher, but MMEB-v3 drops from 55.2 to 47.8; LiNeS recovers broader embedding performance at the cost of some binding gains.The comparison is between the finetuned model before and after applying LiNeS.Source: §C.2 · p. 21, Table 12 · p. 21

C.3 ATTENTION-BASED BINDING ANALYSIS AND DIRECT SUPERVISION

  • In layers 18–26, finetuning retains 1.3–1.4× the base binding margin, and the no-LiNeS variant retains 1.9–2.4× across queries, positive captions, and SABER.The base model’s positive margin in early layers decays toward zero in the final third; Figure 4 reports the layer-wise pattern.Source: §C.3 · pp. 21–22, Figure 4 · p. 22
  • Adding direct attention supervision provides little additional benefit: with LiNeS it raises mean SABER accuracy by 0.4 points, while without LiNeS it lowers SABER accuracy from 93.5 to 93.2.MMEB-v3 changes from 55.2 to 55.0 with LiNeS and from 47.8 to 47.2 without LiNeS.Source: §C.3 · p. 22, Table 13 · p. 23

D GENERALIZATION ACROSS MULTIMODAL EMBEDDERS

The authors apply the binding-focused finetuning recipe to UEmbed and WeMM-Embedding at two model sizes. It improves SABER performance across the tested pairings and largely preserves general-purpose retrieval performance.

D GENERALIZATION ACROSS MULTIMODAL EMBEDDERS

  • The evaluation uses the same data, learning rate, and LiNeS as the main experiments, with rank-4 LoRA applied to full-attention layers and one-epoch checkpoints evaluated.The models use their official prompting and pooling schemes; UEmbed uses its dense embedding.Source: D GENERALIZATION ACROSS MULTIMODAL… · p. 23
  • Finetuning improves SABER across both model families, both sizes, and all four modality pairings, with gains transferring to cross-modal and image–image retrieval.WeMM-Embedding-9B improves from 30.6 to 58.7 in text–image and from 32.6 to 64.4 in image–text retrieval; Table 14 summarizes the comparisons.Source: D GENERALIZATION ACROSS MULTIMODAL… · p. 23, Table 14 · p. 23
  • MMEB-v3 decreases by 1.0 point for UEmbed-9B and by 0.2 and 0.6 points for WeMM-Embedding-2B and 9B, while UEmbed-2B drops by 2.6 points.The authors characterize the recipe as substantially improving binding while largely retaining broader retrieval capabilities.Source: D GENERALIZATION ACROSS MULTIMODAL… · p. 24

E COMPARING TEXT AND IMAGE FINETUNING FOR ATTRIBUTE BINDING

The section compares text-only and image-only finetuning on SABER, asking how their binding gains transfer across modality pairings. Text supervision transfers better to cross-modal retrieval in this controlled setting, while the comparison’s scope is limited to one synthetic benchmark.

E COMPARING TEXT AND IMAGE FINETUNING FOR ATTRIBUTE BINDING

  • With the full training split, text–text finetuning leads image–image finetuning by 13.1 points on text–image retrieval and 7.5 points on image–text retrieval.Text supervision performs best on text–text retrieval, while image supervision is stronger on the matched image–image setting.Source: E COMPARING TEXT AND IMAGE FINETUNING… · p. 24, Table 15 · p. 24
  • MMEB-v3 overall scores vary only from 54.8 to 55.1 across supervision modalities and training-set sizes.The authors suggest this stability may reflect the relatively small amount of finetuning data used.Source: E COMPARING TEXT AND IMAGE FINETUNING… · p. 24
  • The transfer comparison is limited to one relatively small benchmark using a single family of synthetic images and templated texts.The authors caution against treating this result as evidence that text supervision is generally superior to image supervision.Source: E COMPARING TEXT AND IMAGE FINETUNING… · p. 24

F.1 DATASETS AND EVALUATION PROTOCOLS

The section evaluates instance retrieval, binding beyond attribute swaps, and instruction-based alternatives to finetuning. It reports broad retrieval gains and finds that binding-specific instructions do not replace finetuning’s gains.

F.1 DATASETS AND EVALUATION PROTOCOLS

  • Across 1,232 ILIAS image queries, the finetuned model performs better on 923, the pretrained model on 108, and they tie on 201.Evaluations use controlled conditions across benchmarks, with full images unless a benchmark specifies a ground-truth query crop; auxiliary distractor sets are excluded.Source: §F.1 · p. 25

F.2 WHY MIGHT BINDING-FOCUSED FINETUNING IMPROVE INSTANCE RETRIEVAL?

  • The finetuned model improves ILIAS fine-grained retrieval across every top-level category, with gains from +6.1 to +11.2 mAP points.Toy gains +11.2 and Fashion +10.0; Table 16 reports mAP averaged over fine-grained classes within each category.Source: Table 16 · p. 25, §F.2 · p. 25

G ATTRIBUTE BINDING BEYOND ATTRIBUTE SWAPS

  • On CLEVR-bind, the finetuned 8B model improves over its pretrained counterpart and ranks highest on both CLEVR-bind and the corresponding two-object SABER setting.Across ten models, the two settings are positively correlated (Spearman ρ = 0.89; Kendall τ = 0.78).Source: Figure 7 · p. 28, G ATTRIBUTE BINDING BEYOND ATTRIBUTE… · p. 28

H CAN BINDING-SPECIFIC INSTRUCTIONS REPLACE FINETUNING?

  • Binding-specific instructions trade higher SABER accuracy for lower MMEB-v3 image performance, but do not replace finetuning’s gains.With the default instruction, the finetuned model exceeds the pretrained model’s best SABER accuracy across 40 binding-specific instructions.Source: H CAN BINDING-SPECIFIC INSTRUCTIONS… · p. 28, Figure 8 · p. 29

Related papers

Loading 2610.22602v1…