PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings

Shubhang Bhatnagar, Dheeraj Baiju, Narendra Ahuja

arXiv:2606.15134v1cs.CVcs.AIcs.LG

TL;DR

Class-label metric learning gives retrieval encoders a uniform pairwise signal, despite images differing in only a few attributes. SAGA uses a frozen MLLM to provide attribute-aware training gradients, improving Recall@1 by 3–6% over state-of-the-art baselines across four zero-shot retrieval benchmarks.

  • Problem

    Class-label metric learning applies one uniform pull-or-push signal across attributes, limiting supervision for zero-shot retrieval distinguished by attribute combinations.

  • Method

    SAGA uses a frozen MLLM with GRPO and attention distillation to provide attribute-aware gradients for a visual encoder, alongside standard metric learning.

  • Results

    Recall@1 improves by 3–6% over state-of-the-art baselines on four zero-shot retrieval benchmarks using the same vision backbone.

  • Takeaways & Limitations

    The MLLM supplies attribute-aware supervision during training, while deployment retains only the vision encoder and pooler at standard metric-learning cost.

  • Takeaways & Limitations

    Training is slower than standard deep metric learning and requires a supervisor that follows structured prompts and resolves enough pairs correctly.

Abstract

from arXiv · show

Vision encoders for retrieval are typically trained with class-label supervision: each training pair reduces to a scalar that uniformly pushes the embedding apart or pulls it together, as if every visual attribute either differed or matched. A multimodal large language model (MLLM), shown the same pair, can articulate those attributes and use them to predict whether the images share a class. We propose \textbf{SAGA}, a framework that turns this language-grounded, attribute-aware perception into a training signal for the encoder itself. Specifically, we use Group Relative Policy Optimization (GRPO) to reward the MLLM for correct predictions on the vision encoder's tokens. Since correct predictions require those tokens to expose the specific attributes that differ or match between the pair, the gradient pushes the encoder to encode them, replacing the uniform pair-level scalar with attribute-resolved supervision. An auxiliary attention-distillation loss anchors the encoder's embedding to tokens the MLLM attended to, and a standard metric-learning loss shapes the embedding geometry for nearest-neighbour retrieval. The MLLM is frozen throughout and discarded at inference, matching the deployment cost of a metric-learning baseline. SAGA improves Recall@1 by 3 to 6 points over state-of-the-art baselines on CUB-200-2011, Cars-196, FGVC-Aircraft, and iNaturalist Aves on zero-shot image retrieval.

1 Introduction

SAGA uses a frozen multimodal LLM to provide attribute-aware training supervision for visual retrieval encoders, replacing class-label scalars with gradients tied to attributes used in pairwise judgments. It combines reinforcement learning, attention distillation, and metric-learning components while retaining only the encoder and pooler at deployment.

  • Motivation: Class-label metric learning applies one binary signal uniformly across visual attributes, even when image pairs share some attributes.The framework motivates resolving pair differences into attributes such as shape, pattern, texture, and structural proportions.
  • Method: SAGA uses a frozen MLLM as a training-time supervisor whose correct same-class or different-class judgments reinforce attributes relevant to the decision.For example, identifying Bird 2’s orange wing bars supports a different-species verdict, while shared blue plumage and gray legs receive no such reinforcement.
  • Method: Attention distillation teaches the pooler to weight image regions attended to by the MLLM while aggregating patch tokens into the retrieval vector.The pooler forms the embedding used for nearest-neighbor search at inference.
  • Deployment: The MLLM is frozen and discarded after training, leaving only the vision encoder and pooler for retrieval at standard metric-learning deployment cost.Training requires only pairwise class labels already used by the metric-learning objective and no attribute annotations.
  • Results: 3–6%: SAGA improves Recall@1 over state-of-the-art baselines on CUB-200-2011, Cars-196, FGVC-Aircraft, and iNaturalist Aves.The evaluation uses zero-shot image retrieval benchmarks and the same vision backbone.

2 Related Work

Related work spans deep metric learning, multimodal large language models, and language-guided visual representation learning. These lines of research respectively establish semantic embedding geometries, fine-grained language-grounded visual reasoning, and textual supervision for visual representations.

  • Deep metric learning: Deep metric learning trains vision encoders whose embedding spaces represent semantic geometry and generalize to disjoint test classes for zero-shot retrieval and face verification.The passage identifies DML as the dominant framework and names zero-shot image retrieval and face verification as downstream tasks.
  • Multimodal large language models: MLLMs such as LLaVA, Qwen-VL, and InternVL articulate fine-grained visual attributes through language, localize them in images, and reason over them.GRPO is described as a standard recipe for aligning these models with non-differentiable rewards, including grounded visual reasoning.
  • Language-guided visual representation learning: Language-guided representation learning uses textual descriptions to supervise visual representations, including contrastive image-text pretraining and LLM-generated class descriptions for zero-shot recognition.The passage cites CLIP and SigLIP for image-text alignment and subsequent adaptations for zero-shot recognition.
  • Language-guided visual representation learning: CAP-FGVC extends language-guided representation learning toward fine-grained retrieval.The supplied passage introduces CAP-FGVC as an extension of LLM-generated class-description methods, but its final phrase is truncated.

3 Method

SAGA uses a frozen MLLM during training to provide attribute-aware gradients to a trainable vision encoder, while attention distillation and deep metric learning shape retrieval embeddings. The MLLM is discarded at inference, leaving standard DML deployment cost.

  • Vision encoder and retrieval pooler: SAGA factorizes the image-to-embedding map into a Qwen3-VL vision encoder producing patch tokens and a trainable retrieval pooler producing compact embeddings.The pooler can use mean, max, or attention pooling, with both encoder and pooler parameters trainable.
  • Inference: The deployed system contains only the trained vision encoder and pooler, so inference cost matches a standard deep metric-learning pipeline.The MLLM serves solely as a training-time supervisor and is discarded at inference.
  • Frozen MLLM-guided supervision: GRPO rewards a frozen MLLM only for correct same/different-class verdicts, using generated attribute descriptions as implicit chain-of-thought without extra annotations.The MLLM receives paired patch tokens and a structured prompt requesting JSON attributes, key differences, and a class-sharing prediction.
  • Why GRPO provides attribute-aware gradients: GRPO back-propagates through the frozen MLLM into encoder tokens, selectively strengthening visual dimensions used to predict rollout tokens and thereby producing attribute-aware gradients.The MLLM parameters remain frozen, but gradients flow through it into the vision encoder during training.
  • Overall objective: The total objective combines LDML, LGRPO, and LKL: metric learning organizes embedding geometry, GRPO determines what the encoder encodes, and KL distillation teaches where the pooler attends.LDML back-propagates through both encoder and pooler, while LGRPO targets the encoder and LKL targets the pooler.
  • Attention alignment: Attention distillation aligns the pooler’s spatial weights with Qwen3-VL layer 26 attention on attribute-generation tokens from correct rollouts, teaching the pooler where to look.Teacher maps and encoder tokens are detached, so the alignment gradients update only the pooler.

4 Experiments

SAGA is evaluated for zero-shot image retrieval on four fine-grained, attribute-distinguished benchmarks under class-disjoint evaluation and outperforms standard DML baselines. Ablations show that GRPO supplies most of the attribute-aware improvement, while KL alignment provides complementary supervision and gains persist across DML losses.

  • Datasets: SAGA is compared with state-of-the-art DML baselines on zero-shot retrieval benchmarks CUB-200-2011, Cars-196, FGVC-Aircraft, and iNat-Aves.The datasets cover fine-grained bird, car, aircraft, and iNaturalist Aves categories distinguished by visual attributes.
  • Evaluation Settings: Evaluation uses disjoint train and test class halves, with models tested on unseen classes at 224×224 resolution.CUB-200-2011 and Cars-196 use canonical splits; FGVC-Aircraft and iNat-Aves use the same first-half/second-half convention.
  • Main Results: SAGA outperforms PotentialField by 6.3% on CUB-200-2011, 3.3% on Cars-196, 6.1% on FGVC-Aircraft, and 4.5% on iNat-Aves in Recall@1.The gains are largest on the most attribute-driven benchmarks, where subtle visual attributes distinguish classes.
  • Loss component analysis: Adding GRPO improves R@1 by 5.4% on CUB-200-2011 and 4.9% on FGVC-Aircraft, whereas KL alone improves it by 0.5% and 0.7%.These results identify GRPO as the primary source of attribute-aware supervision and KL as complementary pooler supervision.
  • DML loss-agnostic ablation: SAGA gains over bare DML are 8.3%, 7.8%, and 6.3% for MS, PA, and PF on CUB-200-2011, with 9.8%, 9.6%, and 6.1% on FGVC-Aircraft.The gains remain substantial when the underlying DML loss changes.
  • Prompt sensitivity: Removing the per-dataset attribute vocabulary reduces the GRPO lift over PF from +5.4/+4.9 to +2.5/+2.2 R@1 on CUB-200-2011/FGVC-Aircraft.The generic variant omits KL because named-attribute attention spans are unavailable.

5 Limitations

SAGA’s training is slower than standard DML because each pair requires multiple frozen-MLLM rollouts and differentiable replay. Its inference cost remains identical to vanilla DML, but training presumes a capable structured-comparison supervisor.

  • 5 Limitations: Training is slower than standard DML because each contributing pair requires G frozen-MLLM rollouts and differentiable replay through the language backbone.The added computation is incurred only during training.
  • 5 Limitations: Inference uses only the vision encoder and pooler, making its cost identical to a vanilla DML pipeline.The frozen MLLM is not involved at inference.
  • 5 Limitations: The framework presumes a supervisor that can follow the structured comparison prompt and correctly resolve a non-trivial fraction of pairs.This capability is required for GRPO training.

6 Conclusion

SAGA uses a frozen MLLM as a training-time supervisor for a retrieval vision encoder, replacing uniform pair-level scalar supervision with token-level gradients concentrated on discriminating attributes.

  • 6 Conclusion: SAGA turns a frozen MLLM into a training-time supervisor for the vision encoder in a retrieval system.The MLLM guides training but is not described as part of inference.
  • 6 Conclusion: GRPO over the MLLM’s verdict cancels gradients on tokens with agreeing rollouts and concentrates them on discriminating tokens.This routes learning toward directions that resolve the attributes used by the supervisor’s judgment.

Supplementary Material for Beyond Scalar Distances: Semantic Attribute Gradients from

The supplementary material provides additional information beyond the main paper, covering benchmark statistics and licenses, implementation and optimization details, and further ablations and retrieval analyses.

  • Overview: The supplement contains information omitted from the main paper and is organized into five sections.The supplied passage explicitly states that the material is divided into five sections.
  • Section A: Section A gives full statistics and licenses for the four image-retrieval benchmarks.
  • Section B: Section B reports implementation and optimization details, including the training algorithm, optimizer, loss weights, batching, and hardware.
  • Section C: Section C reports additional ablations over embedding dimension and vision backbone that were omitted from Section 4.3 for space.

A Dataset Details

The study evaluates four publicly available fine-grained retrieval benchmarks under a standard zero-shot protocol that separates training and evaluation classes. The datasets span birds, cars, and aircraft, with detailed annotations or hierarchical labels available for several benchmarks.

  • Evaluation protocol: All four benchmarks use disjoint training and evaluation classes, with models tested on classes never seen during training.This follows the standard zero-shot retrieval protocol of Song et al. [2016].
  • CUB-200-2011: CUB-200-2011 contains 11,788 images from 200 bird species, annotated with 312 binary attributes, 15 keypoints, and one bounding box.The first 100 species are used for training and the remaining 100 for evaluation.
  • Cars-196: Cars-196 contains 16,185 images from 196 make-model-year car classes, split into 98 training and 98 evaluation classes.Classes differ through cues including body style, grille and headlight design, side profile, badge placement, and apparent era.
  • FGVC-Aircraft: FGVC-Aircraft provides approximately 10,000 images across 100 variants, with the first 50 variants for training and the remaining 50 for evaluation.Retrieval is performed at the variant level using the standard 100-variant release.
  • iNaturalist 2021 Aves: iNaturalist 2021 Aves contains 1,486 species at 50 images per species, totaling approximately 74,300 images, split into 743 training and 743 evaluation species.The benchmark uses the Aves supercategory from the train_mini split and sorts species directories lexicographically.
  • Image preprocessing: Images are resized to 224 × 224 for the Qwen3-VL-8B vision encoder without bounding-box cropping, preserving full-image background context.The resolution matches the base model’s pre-training resolution.

B Additional Implementation Details

SAGA trains the vision encoder and attention pooler through separate embedding, GRPO rollout, and policy-update phases, while discarding the MLLM supervision machinery at inference. Its implementation uses a single-query attention pooler, normalized losses, and specified optimization and sampling procedures.

  • Training algorithm: Training alternates embedding/DML updates, no-gradient GRPO rollouts with dynamic sampling, and gradient-based policy updates over buffered contributing pairs.Rollouts are retained only when reward variance σr > 0; policy and KL losses are then backpropagated before gradient clipping and an optimizer step.
  • Inference: At inference, only fθ and cϕ produce a single ℓ2-normalized embedding per image for nearest-neighbour retrieval.GRPO rollouts, KL alignment, and the frozen MLLM supervisor are discarded.
  • Attention pooler: The attention pooler is a single-query cross-attention head whose softmax patch weights β are directly supervised by LKL, with no additional importance-scoring head.The pooled vector is projected to dimension De and ℓ2-normalized before retrieval or distance learning.
  • Optimisation: The vision encoder and pooler use AdamW learning rates 2×10^-5 and 1×10^-4, cosine annealing over 3 epochs, 5% warm-up, and global-norm clipping at 1.0.The frozen language backbone pψ receives no gradient updates.
  • Batching and GRPO sampling: G = 8 completions are sampled per image pair with temperature 0.7, top-p 0.95, and at most 1024 tokens; contributing pairs accumulate until K = 8 before one optimizer step.Each micro-batch contains 64 class-balanced images, and only pairs with σr > 0 contribute.

C Additional Ablations · C.1 Lower-Dimensional Embeddings · C.2 Vision Backbone Transfer

Extended ablations show that SAGA preserves its retrieval advantage under lower-dimensional embeddings and after transferring to the InternVL3.5-8B vision-language backbone. These studies evaluate compressed embeddings on CUB-200-2011 and FGVC-Aircraft, and backbone transfer on CUB-200-2011.

  • C Additional Ablations: The ablations use the Sec. 4.1 setting, including Qwen3-VL-8B, the learned attention pooler, GRPO group size G = 8, and DAPO target K = 8.Evaluation uses zero-shot retrieval with Recall@K and NMI.
  • C.1 Lower-Dimensional Embeddings: Lower-dimensional experiments target d ∈{512, 128}, motivated by storage- and compute-constrained retrieval applications.The main-paper embedding dimension is d = 4096; d = 512 matches a standard DML choice, while d = 128 is more aggressive compression.
  • C.1 Lower-Dimensional Embeddings: SAGA and PotentialField are retrained at d ∈{512, 128} and evaluated zero-shot on CUB-200-2011 and FGVC-Aircraft.The remaining procedure follows the standard Sec. 4.1 setting.
  • C.1 Lower-Dimensional Embeddings: SAGA retains its R@1 margin over PotentialField at both compressed dimensions on CUB-200-2011 and FGVC-Aircraft.Table 4 reports R@1 and R@4 (%) at d = 512 and d = 128, with method ordering preserved relative to the d = 4096 main-paper setting.
  • C.2 Vision Backbone Transfer: The backbone-transfer study swaps Qwen3-VL-8B for InternVL3.5-8B while keeping the attention pooler, GRPO, and KL alignment unchanged.This tests whether SAGA’s gain transfers beyond one MLLM family.
  • C.2 Vision Backbone Transfer: On CUB-200-2011, SAGA and PotentialField are trained with InternVL3.5-8B as the encoder, while its language backbone supervises the trained runs.The zero-shot InternVL3.5-8B encoder is also evaluated without fine-tuning; the study is limited to CUB-200-2011 because of compute constraints.
  • C.2 Vision Backbone Transfer: Table 5 reports R@1, R@2, R@4, and R@8 (%) on CUB-200-2011 for the zero-shot encoder, PotentialField, and SAGA under the swapped backbone.For trained runs, the supervisor LM pψ is the InternVL3.5-8B language backbone.

D Qualitative Retrieval Gallery

SAGA’s qualitative retrieval gallery evaluates top-5 nearest neighbors for held-out test images across four zero-shot benchmarks. Clean queries yield visually consistent same-class neighbors, while residual errors occur between visually adjacent classes with similar pose, color, or silhouette.

  • Gallery setup: The gallery reports top-5 nearest-neighbor rankings from SAGA embeddings for held-out test images across four benchmarks.Queries come from the standard zero-shot retrieval split, with classes disjoint from training.
  • Gallery setup: The candidate pool is stratified by SAGA top-5 hit count to avoid trivial wins and degenerate background-dominated queries.Each benchmark includes a clean row drawn from queries with four or five same-class neighbors in SAGA’s top five.
  • Gallery setup: Figure 5 shows two queries per dataset and their five neighbors ordered by descending cosine similarity.Green borders indicate same-class retrievals, red borders indicate cross-class errors, and the query has a neutral border.
  • Qualitative results: Clean rows contain same-class neighbors that are visually consistent with the query, whereas informative-row errors involve visually plausible cross-class matches.These residual errors reflect similarities in pose, color, or silhouette and occur at boundaries between visually adjacent classes rather than coarse categories.

E Prompts and Attributes

SAGA uses a structured pair-comparison prompt that elicits dataset-specific visual attributes, key differences, and a same/different verdict in JSON. A generic prompt ablation removes the expert role and attribute list while preserving verdict parsing.

  • E.1 Prompt Template: The supervisor MLLM describes both images using fixed attribute groups, summarizes key differences, and emits a JSON same/different verdict.The verdict field is parsed into the binary GRPO reward r ∈{0, 1}.
  • E.1 Prompt Template: The template is parameterized by expert role, photographed-item word, verdict question, and dataset-specific attribute list, with substitutions reported in Table 6.The same prompt structure is used across all four datasets.
  • E.1 Prompt Template: Each JSON response contains one entry per attribute for each image, differences, confidence, reasoning, and a yes-or-no verdict key.The response format requires matched attribute keys for the two images.
  • E.2 Per-Dataset Attribute Vocabularies: Attribute groups are dataset-specific and target cues specialists use to discriminate species, car make and model, or aircraft variant.The vocabularies are designed for the relevant taxonomic or identification level.
  • E.2 Per-Dataset Attribute Vocabularies: CUB-200-2011 and iNaturalist Aves share a 28-attribute bird vocabulary formed by collapsing CUB’s 312 binary attributes into 28 groups.The groups include bill, head, eye, body, wing, and tail characteristics.
  • E.2 Per-Dataset Attribute Vocabularies: Cars-196 uses 17 attribute groups covering make-and-model cues such as body style, grille, lights, profile, wheels, badging, proportions, era, and finish.The groups are chosen to discriminate at the make-and-model level.
  • E.2 Per-Dataset Attribute Vocabularies: FGVC-Aircraft uses 17 groups to distinguish variants such as Boeing 737-700 versus 737-800, including engines, wings, stabilizers, windows, gear, size, era, and markings.The vocabulary targets variants rather than only manufacturers or aircraft families.
  • E.3 Generic Comparison Prompt: The generic prompt ablation drops the expert role and attribute list, requests only free-form differences and a verdict, and preserves the verdict key for unchanged reward parsing.Per-dataset substitutions still follow Table 6’s placeholder convention.
Loading 2606.15134v1…