PaperlayerView exact revision on arXiv

Source-linked AI summary

Factorized MLLM Embeddings for Composed Image Retrieval

WonJun Moon, Seongchan Kim, Dahun Kim, Seungryong Kim

arXiv:2611.05193v1cs.CVcs.IR

TL;DR

Composed image retrieval must resolve how a target differs from a reference, but direct single-embedding MLLM approaches do not explicitly resolve the requested edit. FactEmbed uses factorized latents, target imagination, and retrieval-oriented alignment to build global and factor-level representations; it outperforms strong baselines on refined CIR benchmarks.

  • Problem

    Directly mapping a composed query to one MLLM embedding provides no explicit signal to resolve the requested visual edit before forming a discriminative retrieval representation.

    Source: §1 · p. 1

  • Method

    FactEmbed trains factorized latents to imagine target factors, aligns them for retrieval, and distills their decomposition into deeper MLLM attention maps.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    Across refined CIRR and FashionIQ benchmarks, FactEmbed outperforms strong baselines, including a 2.7% improvement in Recall@1 on Refined-CIRR over CSMCIR.

    Source: §1 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    A single forward pass supplies a global [EOS] embedding for recall and mean-pooled latent evidence for reranking a short candidate list.

    Source: §1 · p. 2

    Evaluation covers only CIRR and FashionIQ, so generalizability as a unified multimodal retriever remains to be verified.

    Source: §5 · p. 8

Abstract

from arXiv · show

Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized. This stems from directly mapping composed queries into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. In this work, we introduce Factorized MLLM Embeddings (FactEmbed), which decouples compositional query comprehension from building discriminative embeddings. Rather than directly projecting the composed query into a single embedding without internal reasoning, FactEmbed employs a set of factorized latents that simulate the intended visual modifications via an intermediate target imagination objective. This process encourages the latents to ground into target semantics before forming discriminative representations for retrieval. Subsequently, to propagate these factorized representations throughout the model for retrieval, we optimize a factor-level contrastive loss while distilling the decomposed representations into a deeper attention layer of the MLLM. Finally, for efficient inference, we retain the [EOS] token to establish a global-level embedding on top of these factorized latents, while we exploit the mean-pooled factorized latents to rerank a short candidate list at negligible cost, as both are produced by the same forward pass. Extensive evaluations demonstrate that FactEmbed outperforms prior works across CIR benchmarks.

1. Introduction

CIR lets users search by describing how a reference image should change, but direct single-embedding MLLM approaches do not explicitly resolve the requested edit. FactEmbed uses factorized latents to imagine target factors, align them for retrieval, and combine them into a global embedding with factor-based reranking.

1. Introduction

  • CIR lets users specify how a reference image should differ from the desired result when describing the target itself is difficult.A composed query consists of a reference image and modification text.Source: §1 · p. 1
  • Directly compressing an MLLM query into one embedding provides no explicit signal to resolve the requested edit before forming a retrieval representation [2, 4–6].These approaches use contrastive similarity between a composed query embedding and a target image embedding.Source: §1 · p. 1
  • FactEmbed uses factorized latents to simulate the requested edit and ground target factors through a target-imagination objective.A lightweight decoder trains the latents to reconstruct fine-grained factors not explained by a coarse target-layout prior, without dense annotations or explicit reasoning traces.Source: §1 · p. 1
  • Because reconstruction alone does not ensure retrieval-compatible factors, FactEmbed applies factor-level contrastive learning and distills the decomposition into deeper MLLM attention maps.This addresses the possibility that query and target factors are decomposed faithfully but embedded in incompatible spaces.Source: §1 · pp. 1–2
  • FactEmbed reports state-of-the-art CIR benchmark results, using [EOS] for global retrieval and mean-pooled latents to rerank a short candidate list.Both representations come from one forward pass, and the additional pooled vector adds no further encoding.Source: §1 · p. 2

2. Related Work

Related CIR work includes embedding-based composition, methods for noisy annotations, and zero-shot approaches that avoid annotated triplets. FactEmbed instead places query decomposition inside a local MLLM and uses target reconstruction to ground its latent factors.

2. Related Work

  • FactEmbed embeds target imagination within the retrieval model, unlike zero-shot approaches that generate a proxy image or use a world model.The approach requires neither a generative model nor an additional inference pass.Source: §2 · p. 2
  • Earlier CIR methods addressed representation conflict through composition inside CLIP or BLIP embeddings, while later lines studied noisy annotations and zero-shot retrieval [1, 10–14, 20–31].The noisy-annotation line includes loss dynamics, similarity curves, MLLM-constructed anchor sets, and unlearning; zero-shot methods aim to avoid costly annotated triplets.Source: §2 · p. 2

3. FactEmbed: Factorized MLLM Embeddings

FactEmbed uses factorized latents to imagine target semantics before forming retrieval embeddings, separating query comprehension from discriminative embedding learning. It trains these factors for retrieval, transfers their spatial decomposition into the MLLM, and uses a global embedding with latent-based reranking at inference.

3. FactEmbed: Factorized MLLM Embeddings

  • FactEmbed inserts learnable factorized latents into the multimodal input, reconstructs target patch features through target imagination, and trains factor- and instance-level retrieval representations.A lightweight decoder reconstructs target features from an intermediate MLLM layer; factor-level InfoNCE uses optimal transport to match query and target latents.Source: §3 · p. 3, Figure 1 · p. 3

3.1. Factorized Latents with Target Imagination

  • Target imagination makes query latents infer target residual details without seeing the target, while anchoring target latents to comparable factor slots.The shared coarse layout prior accounts for broad structure, so the imagination loss focuses on residual details not covered by that prior.Source: §3.1 · pp. 3–4

3.2. From Imagination to Retrieval

  • Factor-level InfoNCE with Sinkhorn optimal transport aligns query and target factors through flexible factor-to-factor correspondences.The [EOS] token attends to the factors and forms the global embedding trained with instance-level contrastive loss.Source: §3.2 · p. 4
  • Spatial distillation transfers decoder alpha masks into deeper-layer latent-to-patch attention, encouraging query latents to retain localized semantics at inference.Distillation is applied on the target side, where target patch attention and decoder masks share a spatial grid.Source: §3.2 · p. 4

3.3. Training Objective and Inference

  • Training warms up latents and the decoder with reconstruction before jointly optimizing the MLLM backbone, latents, and decoder with retrieval, imagination, and distillation losses.A post-training augmentation stage adds MLLM-generated triplets to address false negatives and mislabeled triplets associated with sparse CIR supervision.Source: §3.3 · pp. 4–5
  • At inference, FactEmbed searches with the single-vector [EOS] embedding, then reranks the top 100 candidates using a mean-pooled latent vector at negligible additional cost.The mean-pooled vector is a compact proxy for the OT-aligned factor set and is not explicitly optimized during training.Source: §3.3 · p. 5

5. Conclusion

FactEmbed uses factorized latents to imagine visual edits before forming retrieval embeddings, and the authors report state-of-the-art performance. Its evaluation is limited to CIRR and FashionIQ, with fixed latent counts and mean-pooled reranking identified as limitations.

5. Conclusion

  • FactEmbed uses factorized latents to imagine residual target factors beyond a coarse layout prior, then bridges imagination and retrieval with factor-level InfoNCE and deeper-layer attention distillation.At inference, [EOS] retrieves globally and mean-pooled latents rerank a short candidate list.Source: §5 · p. 8
  • The authors report state-of-the-art performance and suggest factor-level supervision on top of instance-level matching improves MLLM-based CIR.Source: §5 · p. 8
  • Evaluation covers only CIRR and FashionIQ, so generalizability as a unified multimodal retriever requires further verification.Source: §5 · p. 8
  • A fixed K ignores input complexity, while mean-pooled reranking leaves some fine-grained information unused compared with full-factor retrieval.The authors suggest query-adaptive latents and Matryoshka representation learning as future directions.Source: §5 · p. 8

A. Additional Visualizations for Imagination

The imagination visualizations use a shared PCA mapping of ground-truth DINOv2 feature maps and imagined representations; despite a coarse layout prior, they show reconstructed local geometry and position.

A. Additional Visualizations for Imagination

B. Diversity of Factorized Latents

The latent-diversity analysis finds that retrieval supervision alone can make factors collapse toward a shared instance representation, while imagination and distillation encourage distinct roles. Low within-instance similarity without auxiliary objectives does not by itself translate into retrieval gains.

B. Diversity of Factorized Latents

  • Factor-level InfoNCE without imagination increases within-instance latent similarity, consistent with collapse toward a shared instance-level representation.The retrieval objective requires collective discrimination across instances but does not incentivize distinct latent roles.Source: B. Diversity of Factorized Latents · p. 9
  • With imagination and distillation, within-instance similarity after per-latent mean removal is 0.548, versus 0.599 and 0.841 for the compared variants.The passage identifies this as the lowest within-instance similarity, with a clear margin.Source: B. Diversity of Factorized Latents · p. 9
  • Latents without auxiliary objectives average 73.21 on Refined-CIRR and 79.60 on CIRR, below the no-latent baseline at 74.65 and 79.80, respectively.The passage attributes their low similarity to orthogonal initialization and lack of direct supervision.Source: B. Diversity of Factorized Latents · p. 9
  • Under the same setting without post-training, the paper's latents score 76.05 and 82.35, indicating they are distinct and informative for retrieval.Source: B. Diversity of Factorized Latents · p. 9

C. Inference Cost

The inference design uses [EOS] for initial retrieval and a mean-pooled factor vector to rerank a fixed top-100 shortlist. This adds little latency, while full-latent matching offers slightly higher R@1 at increasing storage and scoring costs with gallery size.

C. Inference Cost

  • Reranking the top-100 shortlist adds one vector per image and 0.17ms per query at batch size 1, or 0.003ms amortized, inducing less than 0.05% total-latency overhead.The reranking score sums cosine similarities from [EOS] and the mean factorized vector.Source: C. Inference Cost · p. 9
  • The default reranking configuration has storage comparable to or slightly smaller than recent CIR methods maintaining multi-vector galleries.Source: C. Inference Cost · p. 9
  • Full OT matching offers slightly higher R@1, but its storage and scoring cost grow linearly with gallery size, unlike fixed top-100 reranking.The full latent variant stores K vectors per image.Source: C. Inference Cost · p. 9

F.4. Limitation of Contrastive Captioning

Contrastive captioning reduces ambiguity between an original target and a hard negative, but may leave semantically overlapping sibling negatives as false negatives when augmentation scales. Qualitative examples show FactEmbed better handles fine-grained visual constraints and compositional relationships than the contrastive-learning baseline.

F.4. Limitation of Contrastive Captioning

  • As more hard negatives are mined per query, sibling negatives may overlap semantically, so a rewrite conditioned only on the original target can describe multiple candidates and create false negatives.The proposed augmentation strategy should condition instructions across both original and augmented sets; despite this limitation, the augmented triplets consistently improve CIR performance in the experiments and will be released.Source: §F.4 · p. 12

G. Qualitative Results

  • In FashionIQ examples, FactEmbed ranks animal-print details higher, follows requested strap and sleeve constraints, and suppresses a negated Superman logo more reliably than the baseline.The examples span dress and shirt queries.Source: G. Qualitative Results · p. 12, Figure 7 · p. 13, Figure 8 · p. 13
  • On CIRR examples, FactEmbed grounds a smartphone query in the requested screen map and more faithfully captures relative-counting and relational conditions than the baseline.The baseline is distracted by generic smartphone images and exhibits weak compositional understanding on the relational example.Source: G. Qualitative Results · p. 12, Figure 10 · p. 14

Related papers

Loading 2611.05193v1…