Source-linked AI summary
CAMIE: Co-Engagement-Aware Multimodal Item Embeddings for Snap Dynamic Product Ads Retrieval
Xiaodong Liu, Siman Wang, Congfei Zhang, Hsiang-wei Chao, Xiao Bai, Wen Zhang, Jingxiao Ma, Zhe Liu, Yunzhi Zhou, Yajun Wang, Jinchao Li, Yu Zhang
TL;DR
Snap DPA I2I retrieval is fragmented across modality-specific encoders and is trained mainly for content similarity rather than co-engagement behavior. CAMIE uses a shared LLM/MLLM multimodal embedding backbone fine-tuned on journey-mined co-engaged pairs. It improves offline Recall@10 and online CTR/CVR while supporting multiple retrieval modes from one production checkpoint.
Problem
Production content-based I2I retrieval uses separate modality-specific encoders and content-only supervision, leaving embeddings misaligned with co-engagement behavior.
Method
CAMIE represents item images and metadata through a shared LLM/MLLM multimodal backbone and fine-tunes it on journey-mined co-engaged pairs with symmetric in-batch InfoNCE.
Results
+0.390% CTR / +10.832% CVR over the multimodal control and +18.958% CTR / +13.12% CVR over the text control were reported online, alongside superior offline Recall@10.
Takeaways & Limitations
One CAMIE checkpoint supports multimodal, text-only, and image-only retrieval through the existing ANN path and is deployed in Snap DPA.
Abstract
from arXiv · showhide
Item-to-item (I2I) retrieval is a core primitive in large-scale recommendation and advertising systems. In production Snap Dynamic Product Ads (DPA), I2I retrieval faces two challenges: separate visual, textual, and multimodal encoders fragment the retrieval stack, and content-only training does not align embeddings with the co-engagement behavior that drives downstream conversions. We present CAMIE, a co-engagement-aware multimodal item embedding framework for Snap DPA retrieval. CAMIE builds on LLM/MLLM backbones, using their native multimodal interfaces to represent item images and metadata in a shared embedding space. It then fine-tunes the backbone on co-engaged item pairs mined from user journeys with a symmetric in-batch InfoNCE objective. Offline, CAMIE outperforms the strongest commercial multimodal embedding model on Recall@10 and serves text-only retrieval from the same checkpoint with minimal quality loss. Online, CAMIE serves as a drop-in replacement for two deployed content-based I2I encoders, delivering +0.390% CTR / +10.832% CVR over the multimodal control, +18.958% CTR / +13.12% CVR over the text control, and +0.211% CTR / +1.911% CVR on overall DPA traffic. CAMIE is deployed in production.
1 Introduction
CAMIE addresses fragmented modality-specific retrieval stacks and content-only supervision by aligning a shared multimodal embedding with co-engagement behavior. It supports multiple serving modes from one checkpoint and reports offline and production gains in Snap DPA.
- Motivation: Content-based I2I systems maintain separate text, image, and multimodal encoders, each with its own fine-tuning, refresh pipeline, and ANN index.A unified encoder that also handles single-modality inputs is missing.
- Motivation: Content-only training optimizes similarity to item metadata rather than the engagement space reflected in user journeys.Co-engagement events include views, clicks, add-to-cart actions, and purchases involving related items.
- CAMIE: CAMIE renders images and metadata as one multimodal record, fine-tunes a shared backbone on co-engaged item pairs, and serves vectors through the existing ANN path.The same checkpoint supports multimodal, text-only, and image-only I2I retrieval.
- CAMIE: CAMIE uses symmetric in-batch InfoNCE and supports multimodal, text-only, and image-only retrieval from one checkpoint.The framework is intended as a general-purpose embedding model for industrial Snap DPA retrieval.
- Offline evaluation: CAMIE outperforms the strongest commercial multimodal embedding model on Recall@10 and serves text-only retrieval at near-parity with a dedicated text-only fine-tune.A matched-recipe comparison attributes most of the gain to co-engagement supervision, while the MLLM backbone adds multimodal and text-only advantages.
- Online evaluation: +0.390% CTR / +10.832% CVR over the multimodal control and +18.958% CTR / +13.12% CVR over the text control were observed in online A/B tests.Overall DPA traffic improved by +0.211% CTR / +1.911% CVR, and CAMIE was deployed as a drop-in replacement for two content-based encoders.
2 Related Work
Related work includes unified multimodal item representations, graph-based association methods, and behavior-aware language-model retrieval. CAMIE differs by using a native MLLM interface, DPA co-engagement pairs, and one checkpoint across serving modalities.
- Industrial multimodal and unified item embeddings: ItemSage, OmniSearchSage, and MERLIN learn shared representations across industrial recommendation or retrieval surfaces using transformer, compatibility, or graph-based designs.These systems address unified item or query-item representation with differing architectures and supervision.
- CAMIE’s distinction: CAMIE uses an MLLM through its native multimodal interface rather than fusing frozen unimodal features.Its supervision comes from co-engagement pairs mined from DPA journeys rather than content or graph associations.
- CAMIE’s distinction: CAMIE evaluates and deploys one checkpoint across multimodal, text-only, and image-only serving.This distinguishes it from prior systems described as targeting other unified retrieval or recommendation mappings.
- LLM/MLLM-based retrieval: NoteLLM and NoteLLM-2 apply co-engagement contrastive learning to note recommendation, with NoteLLM-2 adding multimodal visual contrastive routing.CAMIE targets industrial multimodal I2I retrieval rather than note recommendation and avoids those architectural additions.
3 Method
CAMIE constructs co-engagement supervision from user journeys, renders items according to serving modality, and encodes them with one shared backbone and projection head. Symmetric contrastive training uses in-batch and cross-device negatives, while the resulting normalized vectors support ANN retrieval across serving modes.
- 3.2 Co-engagement pair construction: CAMIE mines unordered positive item pairs from user journeys and trains on their co-engagement relationships.The pair miner can use the full engagement funnel and apply attribution, field, scope, temporal, and taxonomy filters.
- 3.5 Retrieval serving: Training vectors are normalized and materialized in an ANN index for catalog retrieval, with cached or freshly encoded query representations used online.Active catalog items are encoded once at refresh time under the target modality before being written to the index.
- 3.3 Modality-conditioned item encoder: Each item is rendered as a modality-conditioned record containing available image, title, brand, description, and category fields.Multimodal mode includes image and text; text-only omits the image, while image-only omits text fields.
- 3.3 Modality-conditioned item encoder: The same embedding backbone and projection head encode every serving mode, so modality changes only the rendered item fields.The backbone uses its native visual input pipeline and tokenizer, without a modality-specific projection head.
- 3.4 Symmetric in-batch InfoNCE: Symmetric weighted InfoNCE treats both directions of each unordered pair as positive matches and uses local and cross-device negatives.Symmetry matches the unordered nature of item co-engagement, and the unweighted configuration sets every pair weight to 1.
4 Experiments
The experiments evaluate CAMIE across model comparisons, modality-restricted serving, training-data composition, matched backbones, optimization dynamics, and online production traffic. Results show strong offline retrieval, unified modality support, co-engagement supervision as the dominant gain source, and positive online conversion effects.
- Q1: Model comparison: 137.7% R@10: CAMIE exceeds the pretrained Qwen3-VL checkpoint’s 100.0% and the strongest zero-shot commercial baseline, Gemini Embedding 2 at 129.9%.CAMIE’s advantage is concentrated at K≥10, while Gemini remains ahead at K=1.
- Q2: Modality support: Text-only serving from the multimodal-trained checkpoint comes within 1.1pp R@10 and 0.5pp MRR of a dedicated text-only fine-tune.The comparison is 87.9% versus 89.0% on R@10 and 88.7% versus 89.2% on MRR.
- Q3: Training-data scale vs. conversion-type purity: +7.2% relative R@10: training on all-event pairs beats high-intent-only training, because scale outweighs the purity benefit of filtering.At matched scale, high-intent filtering is slightly better by +1.6% relative R@10, but full all-event training improves R@10 by +8.9% relative at matched filter.
- Q4: Backbone vs. co-engagement supervision: The matched-recipe comparison attributes most of the gain to co-engagement supervision: fine-tuned backbones are within 5.5pp multimodally, versus a 37.7% relative gain from fine-tuning the MLLM.The MLLM retains a statistically significant multimodal advantage of 4.3pp over SigLIP2 and 5.5pp over CLIP.
- Q4: Backbone vs. co-engagement supervision: CAMIE improves monotonically during training while both dual encoders plateau after roughly 40% of the epoch, widening its gap to SigLIP2 from 2.3pp at 20% to 4.3pp at the end.This comparison uses the matched multimodal runs and normalized optimizer-step curves.
- 4.6 Online experiments: +1.911% overall DPA CVR: online A/B tests show positive aggregate conversion impact, while overall CTR rises +0.211% without reaching significance.Per-surface lifts are sharper: CAMIE gains +10.832% CVR over I2I_MM and +18.958% CTR / +13.12% CVR over I2I_TEXT.
5 Conclusion
CAMIE aligns multimodal item representations with DPA co-engagement signals while preserving a shared embedding checkpoint across retrieval modes. It achieves strong offline and online results and has launched in production.
- CAMIE uses LLM/MLLM backbones to place item images and metadata in a shared space, then applies symmetric in-batch InfoNCE to co-engagement signals.
- The resulting embeddings fit the existing ANN serving path and support multimodal, text-only, and image-only retrieval modes.
- CAMIE outperforms pretrained open-source backbones and the strongest commercial multimodal embedding model on offline Recall@10 while retaining near-parity for text-only retrieval.
- +0.390% CTR / +10.832% CVR over the multimodal control, +18.958% CTR / +13.12% CVR over the text control, and +0.211% CTR / +1.911% CVR on overall DPA traffic.
- Matched-recipe comparisons attribute most quality gains to co-engagement supervision rather than backbone family, while the MLLM adds multimodal and text-only advantages.