Source-linked AI summary
CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning
Hao Yu, Zhuokai Zhao, Shen Yan, Lukasz Korycki, Jianyu Wang, Baosheng He, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hanchao Yu
TL;DR
Existing LVLMs generate effectively but often produce inadequate retrieval embeddings, while representation-focused adaptation can weaken language generation. CAFe combines contrastive learning with autoregressive modeling to unify both capabilities, achieving strong retrieval, multimodal understanding, and object-hallucination robustness.
Problem
Existing LVLMs are limited in high-fidelity representation learning for retrieval, while representation-focused fine-tuning can compromise generative capabilities.
Method
CAFe fine-tunes pretrained MLLMs with a contrastive objective and autoregressive language modeling to generate multimodal embeddings while maintaining language generation.
Results
CAFe achieves state-of-the-art multimodal retrieval while also demonstrating strong multimodal understanding and robustness against object hallucinations.
Takeaways & Limitations
CAFe provides a unified framework that combines representation learning and generation in a single multimodal model.
Abstract
from arXiv · showhide
The rapid advancement of large vision-language models (LVLMs) has driven significant progress in multimodal tasks, enabling models to interpret, reason, and generate outputs across both visual and textual domains. While excelling in generative tasks, existing LVLMs often face limitations in tasks requiring high-fidelity representation learning, such as generating image or text embeddings for retrieval. Recent work has proposed finetuning LVLMs for representational learning, but the fine-tuned model often loses its generative capabilities due to the representational learning training paradigm. To address this trade-off, we introduce CAFe, a contrastive-autoregressive fine-tuning framework that enhances LVLMs for both representation and generative tasks. By integrating a contrastive objective with autoregressive language modeling, our approach unifies these traditionally separate tasks, achieving state-of-the-art results in both multimodal retrieval and multimodal generative benchmarks, including object hallucination (OH) mitigation. CAFe establishes a novel framework that synergizes embedding and generative functionalities in a single model, setting a foundation for future multimodal models that excel in both retrieval precision and coherent output generation.
1 Introduction
LVLMs perform well in multimodal generation but remain limited in high-fidelity representation learning for retrieval. CAFe addresses this trade-off by jointly supporting representation learning and language generation, with strong retrieval, understanding, and hallucination-mitigation results.
- Motivation: LVLMs remain suboptimal for retrieval because decoder-only architectures are less suited to generating expressive, high-fidelity representations.These models excel at generative tasks but struggle with unimodal and cross-modal retrieval.
- Approach: CAFe introduces unified contrastive-autoregressive fine-tuning that enables LVLMs to produce retrieval embeddings without compromising language generation.The framework combines contrastive learning for multimodal alignment with autoregressive language modeling.
- Motivation: CAFe bridges encoder-based retrieval and MLLM generation by integrating representation learning with generative capabilities in one framework.Encoder models such as CLIP support aligned embeddings but lack generative reasoning, whereas MLLMs show the opposite imbalance.
- Results: CAFe achieves state-of-the-art multimodal retrieval while retaining strong multimodal understanding and robustness against object hallucinations.These results support the framework’s ability to address both representation and generation tasks.
- Contribution: CAFe is presented as the first unified contrastive-autoregressive fine-tuning framework for LVLMs.The authors position it as a foundation for developing better-rounded multimodal models.
2 Related Work
Related work spans multimodal large language models, contrastive vision-language pretraining, and emerging multimodal representation learning. CAFe builds on these directions while addressing the unexamined effect of representation adaptation on MLLM language generation.
- Multimodal Large Language Models: MLLMs combine an LLM, a modality encoder, and a projector to process multimodal inputs, achieving strong results across image, multi-image, and video settings.LLaVA-OneVision is cited as achieving state-of-the-art results across these vision scenarios.
- Vision-Language Representation Learning: CLIP pioneered dual-encoder contrastive pretraining on image-text pairs, enabling strong visual and textual representations for zero-shot retrieval and classification.Later vision-language models extend this foundation through refined data and training approaches.
- Joint Contrastive and Generative Learning: Contrastive-language-modeling methods such as CoCa and BLIP jointly optimize representation or understanding objectives with generation objectives.These approaches demonstrate transferability across vision-language understanding and generation.
- Multimodal Representation Learning: Recent work adapts MLLMs for multimodal representation learning, but does not investigate how this adaptation affects their language generation capabilities.CAFe addresses this gap by jointly targeting multimodal representation learning and natural language generation.
3 Method
CAFe jointly trains an MLLM to produce multimodal embeddings and language responses by combining contrastive learning with autoregressive language modeling. Prompt-based embedding instructions support flexible image, text, and multimodal encoding within the unified pipeline.
- Contrastive-Autoregressive Joint Training: The framework processes paired image-text data separately for embedding alignment while applying language modeling to the same paired input.Contrastive learning operates on corresponding image and text embeddings, while autoregressive training preserves language-generation behavior.
- Contrastive-Autoregressive Joint Training: CAFe jointly optimizes contrastive and autoregressive language-modeling objectives so one MLLM produces robust multimodal embeddings and free-form language responses.The combined objective uses scaling parameters αlm and αcon.
- Prompt-Based Representation: Specialized embedding instructions prompt the MLLM to encode image, text, or multimodal inputs, with either modality optionally absent.The embedding prompt is designed for generating multimodal embeddings from flexible input combinations.
- Contrastive Learning: The contrastive objective uses InfoNCE to bring positive visual-text pairs closer while separating negative samples.The normalized embeddings are computed for each visual-text pair, with batch size N and temperature τ scaling the logits.
- Autoregressive Language Modeling: The autoregressive objective trains the model to predict each next token from preceding context using the target token and its predicted vocabulary distribution.This objective is applied through language instructions in the joint training pipeline.
4 Experiments
CAFe is evaluated as a model-agnostic framework across retrieval, multimodal understanding, and hallucination-reduction tasks. Experiments show that contrastive-autoregressive fine-tuning improves multimodal retrieval while preserving comparable understanding performance and reducing object hallucinations.
- Training setup: The experiments fine-tune LLaVA-OneVision backbones using combined contrastive and language-modeling losses, while CAFe is described as model-agnostic.Training uses existing instruction data, including 248K image-text samples, rather than introducing new data.
- Zero-shot image-text retrieval: CAFe-7B outperforms all baselines on zero-shot image-text retrieval, while the smaller 0.5B model also performs strongly.The comparison uses MSCOCO and Flickr30K test sets with frozen parameters during embedding extraction.
- Multimodal retrieval: A 10.2 percentage-point gain in MMEB overall score over the previous state of the art shows CAFe’s strongest reported retrieval result.The 7B model improves upon the previous state of the art, while the 0.5B model is eight times smaller than VLM2REC yet achieves comparable performance.
- Multimodal understanding: CAFe remains comparable to LLaVA-OneVision on MMMU and MMStar, indicating that contrastive-autoregressive fine-tuning preserves multimodal understanding performance.The evaluation is conducted in a zero-shot setting on benchmarks requiring multimodal reasoning.
- Object hallucination reduction: CAFe consistently outperforms baselines in POPE accuracy and F1 score, demonstrating reduced object hallucination across random, popular, and adversarial splits.POPE evaluates object-presence questions on 500 MSCOCO validation images, with scores averaged across the three splits.
5 Ablation and Analysis
The analysis examines how contrastive and language-modeling objectives balance representation and generation, and how CAFe affects modality alignment. Larger contrastive weighting improves representation ability but reduces multimodal understanding, while CAFe narrows the modality gap.
- 5.1 Ablation on Training Objectives: Larger contrastive-loss weighting improves representation ability but worsens multimodal understanding, revealing a trade-off between representation and generation.The tested α_lm : α_con ratios were 0, 0.02, 0.1, 0.2, and 1, with the contrastive loss scaled to one-tenth of the language-modeling loss.
- 5.2 Modality Gap: UMAP visualizations of 250 MSCOCO and Flickr image-text pairs compare CAFe with LLaVA-OV in a two-dimensional embedding space.The analysis follows prior modality-gap work and evaluates text and image embeddings produced by both models.
- 5.2 Modality Gap: LLaVA-OV exhibits a significant text-image modality gap, whereas CAFe reduces this separation in its multimodal embedding space.The comparison uses embeddings from the same model for both modalities in the LLaVA-OV case, while CAFe is evaluated with the same visualization procedure.
6 Conclusion
CAFe fine-tunes pretrained multimodal language models to produce multimodal embeddings while retaining language generation through embedding instructions and combined objectives. Experiments report improvements in retrieval, multimodal understanding, modality consistency, and object-hallucination robustness.
- 6 Conclusion: CAFe combines contrastive and autoregressive fine-tuning to support both multimodal embedding generation and language generation in one model.The framework adapts a pretrained MLLM with embedding instructions and jointly uses contrastive and autoregressive language objectives.
- 6 Conclusion: CAFe achieves significant improvements in cross-modal and multimodal retrieval while retaining strong multimodal understanding capability.The experiments evaluate retrieval across image-text and multimodal tasks.
- 6 Conclusion: CAFe removes the modality gap and demonstrates robustness against object hallucinations, with THRONE and POPE results highlighting improved mitigation.The authors attribute retrieval gains partly to leveraging LLM-based multimodal processing and understanding beyond simple concatenation.
A Qualitative Case Study on Hallucination Reduction
The qualitative case study compares captions from LLaVA-OV and CAFe on MSCOCO images to examine object hallucination. CAFe produces fewer hallucinated objects and longer, more detailed captions in the reported examples.
- A Qualitative Case Study on Hallucination Reduction: The case study compares captions generated by LLaVA-OV and CAFe on MSCOCO images, using absent objects as the hallucination criterion.The supplied examples include generated captions from LLaVA-OV for individual images.
- A Qualitative Case Study on Hallucination Reduction: CAFe generates fewer hallucinated objects than LLaVA-OV in the MSCOCO captioning case study.The comparison marks objects absent from the original image in red and reports hallucinations such as “sink” and “bird” for LLaVA-OV.
- A Qualitative Case Study on Hallucination Reduction: CAFe produces longer and more detailed captions than LLaVA-OV in the reported qualitative comparison.