PUMA: Post-Hoc Sparsification of Universal Multimodal Embeddings for Efficient Retrieval

Matteo Attimonelli; Alessandro De Bellis; Franco Maria Nardini; Claudio Pomo; Cosimo Rulli; Rossano Venturini; Tommaso Di Noia

Source-linked AI summary

PUMA: Post-Hoc Sparsification of Universal Multimodal Embeddings for Efficient Retrieval

Matteo Attimonelli, Alessandro De Bellis, Franco Maria Nardini, Claudio Pomo, Cosimo Rulli, Rossano Venturini, Tommaso Di Noia

arXiv:2608.25780v1Aug 26, 2026cs.IR

TL;DR

Dense universal multimodal embeddings support broad retrieval but impose substantial storage and scoring costs, and post-hoc sparsification for this setting remains underexplored. PUMA converts frozen embeddings into sparse codes by preserving dense dot-product geometry before retrieval-aware fine-tuning, achieving competitive or better retrieval on most evaluated datasets while reducing storage and search costs. The paper also identifies insufficient pre-TopK support and retrieval-misaligned active support as failure modes.

Problem
Post-hoc sparsification for retrieval serving is underexplored: the paper asks whether frozen universal multimodal embedders can become sparse retrievers without retraining while preserving quality across diverse tasks.
Method
PUMA uses a TopK sparse autoencoder to convert cached dense embeddings from frozen multimodal backbones into sparse retrieval codes, first preserving dense dot-product geometry and then fine-tuning sparse retrieval behavior.
Results
Across five M-BEIR tasks, PUMA is competitive with dense retrieval, improving significantly on CIRR and Fashion200K, near parity on FashionIQ and MSCOCO, and trailing on VisualNews; it also reduces storage by 8–16× and speeds exact search by up to 25×.
Takeaways & Limitations
PUMA offers a practical post-hoc path to efficient multimodal retrieval while retaining the generality of universal embedders at a fraction of storage and search cost.
Takeaways & Limitations
The failure analysis identifies support-limited and retrieval-misaligned active features but does not solve these bottlenecks; frozen-backbone biases may also propagate into sparse codes.

Abstract

from arXiv · show

Universal multimodal embedders enable retrieval across text, image, and combined queries, but their dense representations incur high memory and inference costs. Post-hoc sparsification could reduce these costs but remains underexplored for multimodal retrieval. We introduce PUMA, a sparse autoencoder recipe that maps universal multimodal embeddings to compact sparse codes without retraining the backbone: a pretraining stage preserves dense dot-product geometry, after which the sparse encoder is fine-tuned for retrieval. We evaluate on five benchmarks covering text-to-image and composed image retrieval. On Qwen3-VL-Embedding-2B, PUMA is statistically indistinguishable from or improves over dense retrieval on four of five datasets. We further identify two failure modes of post-hoc sparsification: insufficient pre-TopK support and retrieval-misaligned active support. PUMA reduces vector storage by 8-16x (FP32) and is up to 25x faster than exact dense scoring on larger candidate pools, enabling efficient multimodal retrieval.

1 Introduction

PUMA addresses the storage and scoring costs of dense universal multimodal embeddings by converting frozen-backbone representations into retrieval-aware sparse codes. Across five retrieval datasets, it largely matches or exceeds dense retrieval while reducing storage and search costs, though performance varies by task.

Motivation: Universal multimodal embedders unify text, image, and mixed-modality retrieval in one shared space, but dense high-dimensional vectors are costly to store and score.Exact scoring over n candidates with d-dimensional embeddings costs O(nd) per query.
Approach: PUMA is trained first to preserve dense dot-product geometry, then fine-tuned using sparse dot products for retrieval.Each resulting vector has exactly k active features and can be scored as a sparse dot product.
Evaluation: PUMA is evaluated on five M-BEIR datasets spanning text-to-image and composed image retrieval, with candidate pools from roughly 5K to 542K images.Composed image retrieval combines a reference image and textual modification into one query representation.
Results: On Qwen3-VL-Embedding-2B, PUMA improves over dense retrieval on CIRR and Fashion200K, is statistically indistinguishable on FashionIQ and MSCOCO, and trails dense retrieval on VisualNews.On CIRR, PUMA reaches 0.533 Hit@5 and 0.436 nDCG@10, compared with 0.522 and 0.424 for dense retrieval.
Efficiency: PUMA uses 8–16× less per-vector storage and makes exact sparse search up to 25× faster on larger candidate pools than dense retrieval.The comparisons include dense TopK pruning, matched-memory PCA, a dense autoencoder, and encoder-only sparse training.
Approach: PUMA converts dense embeddings from frozen universal multimodal embedders into compact sparse codes without retraining the backbone.The method formulates post-hoc sparsification as a retrieval problem motivated by dense representation costs.

2 Related Work

Related work establishes universal multimodal retrieval and alternative efficiency strategies, while PUMA applies post-hoc sparse coding to frozen multimodal embedders through a staged pipeline.

Universal multimodal retrieval: Universal multimodal retrieval uses one embedding model across heterogeneous query–candidate pairs, including cross-modal and mixed-modality queries.M-BEIR provides a unified benchmark across tasks and modalities, while MMEB expands evaluation to instruction-conditioned tasks.
Universal multimodal retrieval: Recent multimodal embedding backbones adapt multimodal or vision-language models into general-purpose dense embedding systems.Examples include VLM2Vec, MM-Embed, GME, Qwen3-VL-Embedding, and RZenEmbed.
Sparse retrieval and embedding compression: Dense retrieval efficiency methods include quantization, approximate search, and dimensionality reduction, whereas sparse retrieval changes representation structure to store feature–weight pairs.Sparse representations can reuse lexical-search infrastructure.
PUMA pipeline: PUMA caches embeddings from frozen embedders, trains a TopK sparse autoencoder, fine-tunes the sparse representations for retrieval, and then ranks candidates.The pipeline converts dense multimodal embeddings into sparse retrieval codes.

3 PUMA

PUMA converts cached dense embeddings from a frozen universal multimodal embedder into exactly-k sparse retrieval codes. It first preserves dense geometry and cross-modal structure, then fine-tunes sparse scoring for retrieval.

3.1 Problem Setting: PUMA maps dense embeddings into sparse codes with exactly k active features and ranks candidates using sparse dot products.The sparse encoder operates in an overcomplete dictionary space, where D is much larger than the original embedding dimension.
3.2–3.5 PUMA Pipeline: Training caches embeddings from the frozen backbone, pretrains the sparse autoencoder, and fine-tunes the encoder with retrieval supervision.All subsequent sparse training reads from the cache, making its cost independent of backbone execution.
3.4 Stage ②: Retrieval-Aware Sparse Pretraining: PUMA combines reconstruction, retrieval-aware distillation, cross-modal alignment, feature revival, and progressive k-annealing during sparse pretraining.Progressive annealing starts with a larger support and ends at the target retrieval sparsity.
3.4 Stage ②: Retrieval-Aware Sparse Pretraining: Cross-modal alignment encourages paired modalities to activate overlapping dictionary atoms and aligns their magnitudes on co-active features.For retrieval pairs, this aligns text queries with positive images and composed image–text queries with positive target images.
3.5 Stage ③: Retrieval Fine-Tuning: The final fine-tuning stage optimizes sparse codes with the same sparse dot product used at inference while retaining reconstruction and distillation regularizers.This shifts optimization from reconstruction-oriented toward retrieval-oriented sparse scoring.

4 Experimental Setup

The experiments evaluate PUMA across five M-BEIR datasets covering text-to-image and composed image retrieval, using shared metrics and multiple dense or sparse baselines. Models are trained on cached embeddings with frozen backbones and evaluated across candidate pools from 5K to 542K images.

4.1 Datasets: Five M-BEIR datasets cover two composed image retrieval tasks and three text-to-image retrieval tasks.Candidate pools range from 5,000 to 542,246 images.
4.2 Metrics: The evaluation reports Hit@5, Recall@10, and nDCG@10, with Hit@5 serving as the headline metric for CIRR and FashionIQ.nDCG@10 is used for validation.
4.3 Backbones: Main experiments use Qwen3-VL-Embedding-2B with 2,048-dimensional embeddings and additionally evaluate Qwen3-VL-Embedding-8B.The models directly produce shared embeddings for text, images, and image–text inputs.
4.4 Baselines: PUMA is compared with full dense retrieval, raw dense TopK pruning, matched-memory PCA, trained dense autoencoding, and encoder-only TopK sparsification.These baselines test storage-matched dimensionality reduction, dense autoencoding, and the contribution of PUMA’s autoencoding structure.
4.5 Implementation Details: Sparse models use cached dense embeddings with the backbone frozen, a 16× expansion ratio, and validation-selected checkpoints evaluated once on test.Experiments run on a single NVIDIA H100 NVL GPU.

5 Results

Across five M-BEIR tasks and multiple backbones, PUMA generally preserves or improves dense retrieval quality while substantially reducing storage and exact-search time. Its performance varies by task and backbone, with diagnostics attributing failures to unavailable pre-TopK support or retrieval-misaligned active features.

Main results on Qwen-2B: PUMA matches or exceeds dense retrieval on four of five Qwen-2B tasks by nDCG@10, improving significantly on CIRR and Fashion200K while trailing only on VisualNews.On FashionIQ and MSCOCO, differences are small and not statistically reliable.
Scaling and cross-family transfer: 0.027 nDCG@10 improvements over dense retrieval occur on both VisualNews and Fashion200K with Qwen-8B.PUMA remains close to dense on MSCOCO and FashionIQ but falls below dense on CIRR.
Scaling and cross-family transfer: PUMA transfers unevenly to RZenEmbed, exceeding dense on FashionIQ and Fashion200K but remaining below dense on CIRR, VisualNews, and MSCOCO.The Fashion200K gain is statistically reliable, while FashionIQ trends positive.
Failure diagnostics: Support availability is necessary but insufficient: Qwen-8B CIRR lacks enough pre-TopK activations, whereas RZenEmbed VisualNews has support but retrieval-misaligned active features.The two failure modes are insufficient pre-TopK support and active support that does not preserve dense rankings.
Alternative compression techniques: PUMA outperforms raw TopK, matched-memory PCA, encoder-only sparse training, and trained dense autoencoders, supporting the full reconstruction-plus-retrieval recipe.On CIRR, PUMA reaches 0.4356 nDCG@10 versus 0.3769 for raw TopK, 0.4153 for PCA, and 0.3631 for encoder-only training; trained dense AE is below PUMA in 12 of 15 settings.
Storage and search time: 8–16× lower FP32 per-vector storage and up to 24.5× faster exact search demonstrate PUMA’s efficiency advantage on larger candidate pools.On VisualNews 8B with 542K candidates, search takes 423.8 s dense versus 17.3 s sparse; dense remains faster on the smaller MSCOCO pool.
Ablation: All Stage ② losses contribute, while Stage ③ fine-tuning improves both Qwen-2B tasks but can reverse on support-starved Qwen-8B CIRR.On Qwen-2B, removing Stage ③ decreases nDCG@10 by 0.019 on CIRR and 0.009 on Fashion200K.

6 Conclusion

PUMA converts frozen universal multimodal embeddings into compact sparse retrieval codes and remains competitive with dense retrieval across five M-BEIR tasks. Its remaining challenge is jointly ensuring sufficient pre-TopK support and retrieval-aligned active support.

PUMA converts frozen universal multimodal embeddings into compact sparse retrieval codes.
On Qwen3-VL-Embedding-2B, PUMA significantly improves on CIRR and Fashion200K, reaches near parity on FashionIQ and MSCOCO, and trails dense retrieval on VisualNews.
PUMA’s gains are not explained by dimensionality reduction alone, as several dense and sparse baselines fall short in representative comparisons.
The remaining challenge is jointly ensuring sufficient pre-TopK support and retrieval-aligned active support.

Limitations

The paper’s main experiments focus on the Qwen embedding family, while transfer results suggest sparsification behavior depends on frozen-backbone geometry. Its diagnostics identify bottlenecks but do not solve them, and post-hoc codes inherit biases from the dense backbone.

Main experiments focus on the Qwen embedding family, with RZenEmbed serving as a cross-family check.
Evaluating additional universal multimodal embedders is needed to clarify how broadly PUMA applies.
The failure analysis diagnoses two bottlenecks but does not solve them: insufficient dictionary support and retrieval-misaligned active features.
Because PUMA is post-hoc, domain imbalance or bias in the frozen dense embedding space may propagate to sparse codes.
Deployment-sensitive settings should audit the sparsified retriever alongside the original dense model.

A Bootstrap Confidence Intervals

Table 7 presents representative 95% bootstrap intervals for nDCG@10 on Qwen-2B, organized by whether PUMA wins, ties closely, or loses to dense retrieval.

Table 7 reports representative 95% bootstrap intervals for nDCG@10 on Qwen-2B.
Rows distinguish a PUMA win, near parity, and a dense win.

B Support-Usage Diagnostics

Support-usage diagnostics measure pre-TopK activations, surviving active features, and exact realization of the requested sparsity. The 2B checkpoints have saturated support, whereas the 8B SAE becomes support-limited before TopK selection.

On 2B, encoder-only training and the final SAE are well past support saturation, realizing the requested k on every example.
On 8B, the SAE’s positive pre-activations drop sharply, explaining the support-limited regime.
Support-usage diagnostics use k=128 and report mean positive pre-activations, surviving active features, and the fraction realizing k exactly.

C Additional Ablation Study

The ablations examine how sparsity affects retrieval quality and cross-modal support alignment. PUMA remains effective into the ultra-sparse regime, while paired queries and candidates show greater TopK-support overlap than random pairs.

Effect of the target sparsity k: Quality saturates around k = 128, while k = 64 is already near-saturated on FashionIQ and Fashion200K.CIRR is more sensitive, but its curve also flattens past k = 128.
Cross-modal alignment impact: +0.177, +0.134, and +0.088 are the paired-minus-random TopK-support Jaccard lifts on CIRR, FashionIQ, and Fashion200K, respectively.The largest lifts occur on the two composed-retrieval datasets.
Ultra-sparse regime (k ≤64): 96% of dense nDCG@10 is reached on CIRR at k=48, while Fashion200K exceeds dense retrieval at k=48 with 0.110 vs 0.079.At k=16, PUMA captures 74% of dense score on CIRR and 69% on Fashion200K.
Ultra-sparse regime (k ≤64): ∼82 bytes per vector at k = 16 is ∼100× smaller than FP32 dense storage, while k = 48 uses ∼246 bytes, or ∼33× less.These settings trade retrieval quality against index size on the evaluated datasets.
Effect of the target sparsity k: Figure 2 plots PUMA nDCG@10 against target sparsity k on CIRR, FashionIQ, and Fashion200K, with the dashed line representing the dense baseline.The sweep uses k∈{64, 96, 128, 144, 160}.

D Training Details

PUMA trains sparse autoencoders in two stages using cached query and candidate embeddings. The training recipe combines reconstruction, retrieval, alignment, feature-revival, contrastive, and distillation objectives with fixed cross-dataset loss weights.

Training stages: PUMA trains SAEs in two stages on cached query/candidate embeddings from each M-BEIR task.Stage ② performs sparse pretraining, followed by Stage ③ retrieval fine-tuning.
Training stages: Stage ② combines cosine reconstruction, retrieval-similarity supervision, AuxK feature revival, and a small contrastive blend.Stage ③ emphasizes InfoNCE on sparse codes, while reconstruction and distillation remain small regularizers.
Loss weights: Loss weights are held fixed across datasets and backbones, with each weight assigned a deliberate role rather than selected by grid search.The cited examples include λa for alignment and λx=1/32 for AuxK.
Optimization: AdamW uses weight decay 10−2, learning rates 3×10−4 for Stage ② and 1×10−4 for Stage ③, with linear warmup followed by cosine decay.The warmup schedules are 1,000 and 500 steps, respectively.
Optimization: 40,000 Stage ② steps and 5,000 Stage ③ steps are run with batch size 64 and validation-based checkpoint selection.Selection uses sparse nDCG@10 at the target k on a 10% slice of training pairs.