Source-linked AI summary
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, Yu Qiao
TL;DR
SPHINX addresses the challenge of building MLLMs with strong alignment, broad task coverage, and fine-grained high-resolution perception. It jointly mixes model weights, tuning tasks, visual embeddings, and image scales, and reports strong performance across multimodal benchmarks and applications.
Problem
Existing MLLMs face limited cross-modal alignment, narrow visual-task coverage, and low input resolution that hinders fine-grained perception.
Method
SPHINX unfreezes and mixes LLM weights from real-world and synthetic data, jointly tunes diverse tasks, mixes visual embeddings, and combines image scales with high-resolution sub-images.
Results
SPHINX achieves superior multimodal understanding and fine-grained visual perception across a wide range of evaluation benchmarks and applications.
Takeaways & Limitations
The resulting model can serve as a vision generalist for object detection, region-level captioning, human pose estimation, and other visual tasks.
Abstract
from arXiv · showhide
We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, to enable multi-purpose capabilities, we mix a variety of tasks for joint visual instruction tuning, and design task-specific instructions to avoid inter-task conflict. In addition to the basic visual question answering, we include more challenging tasks such as region-level understanding, caption grounding, document layout detection, and human pose estimation, contributing to mutual enhancement over different scenarios. Additionally, we propose to extract comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity, providing language models with more robust image representations. Based on our proposed joint mixing, SPHINX exhibits superior multi-modal understanding capabilities on a wide range of applications. On top of this, we further propose an efficient strategy aiming to better capture fine-grained appearances of high-resolution images. With a mixing of different scales and high-resolution sub-images, SPHINX attains exceptional visual parsing and reasoning performance on existing evaluation benchmarks. We hope our work may cast a light on the exploration of joint mixing in future MLLM research. Code is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.
1 INTRODUCTION
SPHINX combines model-weight, task, visual-embedding, and resolution mixing to build a versatile MLLM for diverse and fine-grained visual reasoning.
- 1 INTRODUCTION: SPHINX mixes model weights, tuning tasks, visual embeddings, and high-resolution sub-images within one versatile MLLM.The approach targets diverse visual understanding and perception tasks.
- 1 INTRODUCTION: Unfreezing the LLM during pre-training improves cross-modal alignment while combining vision-language data with RefinedWeb helps preserve detailed language generation.The strategy addresses the limitations of frozen LLMs and mitigates forgetting.
- 1 INTRODUCTION: Weight mixing combines LLMs tuned on real-world and synthetic vision-language data to integrate domain-specific semantics without directly mixing the datasets.The mixed model is formed by linearly combining the fine-tuned and real-world model weights.
- 1 INTRODUCTION: Joint visual instruction tuning combines VQA, region-level understanding, caption grounding, document layout detection, and human pose estimation with task-specific instructions.The task mixture is designed to support multi-purpose visual capabilities while avoiding inter-task conflict.
- 1 INTRODUCTION: Mixing visual scales and high-resolution sub-images produces longer visual-token sequences that support fine-grained perception while maintaining encoding efficiency.The input is divided into high-resolution sub-images and also represented at low resolution before concatenating token groups.
- 1 INTRODUCTION: SPHINX demonstrates diverse visual reasoning and fine-grained high-resolution perception, with state-of-the-art performance reported on MMBench, MME, and POPE.The examples include object detection, caption grounding, region-level description, and interpreting complicated figures.
2 RELATED WORK
Related MLLM research has explored visual-language alignment, multimodal reasoning, and visual instruction following through progressively broader models and data.
- 2 RELATED WORK: VisualGPT and the BLIP series established visual-input alignment for image captioning and question answering.These approaches illustrate early efforts to connect LLM-style language generation with visual inputs.
- 2 RELATED WORK: Flamingo and Kosmos-1 extended multimodal understanding to image-text interleaved contexts.Their results broadened the application setting beyond isolated image-question pairs.
- 2 RELATED WORK: GPT-4 and Bard demonstrated vision-language understanding and reasoning across diverse multimodal tasks through large-scale pre-training and model sizes.The passage presents them as prominent large-scale multimodal systems.
3 SPHINX
SPHINX jointly mixes model weights, tuning tasks, visual embeddings, and high-resolution sub-image tokens to build a versatile MLLM. Its pipeline combines two-stage training, multi-task instruction tuning, multi-encoder visual representations, and efficient high-resolution processing.
- Overall architecture: SPHINX uses two-stage training: vision-language pre-training followed by visual instruction fine-tuning, with model-weight mixing and task mixing applied respectively.The model contains an LLM, mixed vision encoders, and two linear projection layers.
- Model-weight mixing: Unfreezing the LLM during pre-training combines vision-language data with RefinedWeb language data to improve cross-modal learning while alleviating forgetting of detailed language generation.
- Model-weight mixing: SPHINX mixes LLM weights trained on real-world and synthetic domains, producing aggregated semantics through θmix = β · θreal + (1 −β) · θsyn.The strategy is presented as more scalable to new data domains than jointly fusing domains during pre-training.
- Mixed tuning tasks: Joint visual instruction tuning covers VQA, region-level referring, detection and relation reasoning, chart/document VQA, and human pose estimation using task-specific instructions.The instructions are designed to avoid inter-task conflict while training a vision generalist.
- Mixed visual embeddings: SPHINX ensembles visual embeddings from varied backbones and information granularity, placing image tokens before language instructions to provide visual context.
- High-resolution processing: For high-resolution inputs, SPHINX mixes a downsampled full image with spatial sub-images, concatenating token groups to capture fine-grained semantics efficiently.The method addresses low-resolution limitations without simply upsampling, while LLM attention enables interaction among otherwise separate sub-images.
- Extended applications: SPHINX extends to language-referred segmentation by passing its predicted boxes to SAM, and to image editing by combining SPHINX, SAM, LaMa, and Stable Diffusion.
- Extended applications: After joint mixing, SPHINX supports visual recognition fine-tuning, reaching 70.8% ImageNet-1K accuracy after one epoch without data augmentation.The paper contrasts this convergence with traditional ResNet and ViT training that typically uses about 300 epochs and strong augmentation.
4 EXPERIMENTS
SPHINX is evaluated through two-stage training, quantitative benchmarks, and qualitative demonstrations spanning general and fine-grained visual understanding. Joint text-image pre-training preserves text modeling, while high-resolution variants improve several tasks requiring detailed visual information.
- Training details: SPHINX uses image-caption datasets for multimodal alignment and jointly trains on RefinedWeb to preserve text reasoning during full language-model fine-tuning.The pre-training stage fine-tunes the LLM and visual projections while keeping visual encoders frozen.
- Training details: Without RefinedWeb, text-only loss grows significantly within less than 1/10 of the training schedule, despite similar image-caption loss descent.The experiment was early-stopped once forgetting became obvious.
- Quantitative evaluation: SPHINX is evaluated on 10 MLLM benchmarks, with results reported alongside comparisons to prior multimodal language models.The evaluation combines quantitative metrics with qualitative assessments.
- Quantitative evaluation: SPHINX surpasses previous state-of-the-art performance on 6 out of 10 MLLM benchmarks, while high-resolution sub-images improve visual hallucination performance on POPE.The comparison includes BLIP-2, InstructBLIP, Shikra, Qwen, Fuyu, and LLaVA1.5.
- Quantitative evaluation: SPHINX-1k and SPHINX-2k significantly outperform SPHINX on VQAv2 and text-oriented VQA requiring fine-grained visual information.SPHINX remains below Qwen-VL-7B on text-oriented VQA because of limited text-related pre-training data.
- Quantitative evaluation: SPHINX outperforms most state-of-the-art models on visual grounding, and higher-resolution variants yield consecutive improvements over the base model.The high-resolution variants narrow the gap to UNINEXT, a specialist model using larger input images.
5 CONCLUSION
SPHINX combines model-weight, task, and visual-embedding mixing, then extends this design to high-resolution images through mixed scales and sub-images. It achieves strong multi-modal benchmark performance and supports diverse vision-generalist applications.
- 5 CONCLUSION: SPHINX jointly mixes pre-trained LLM weights, diverse visual instruction-tuning tasks, and visual embeddings from different vision backbones.The model additionally mixes visual scales and sub-images for high-resolution processing.
- 5 CONCLUSION: Mixed visual scales and high-resolution sub-images improve SPHINX's fine-grained visual understanding performance.The approach combines different visual scales with spatially divided sub-images.
- 5 CONCLUSION: SPHINX achieves impressive performance across a wide range of multi-modality evaluation benchmarks.The conclusion characterizes SPHINX as a strong vision generalist.
- 5 CONCLUSION: SPHINX supports object detection, region-level captioning, and human pose estimation as a vision generalist.Examples also include emergent referring pose estimation and multi-level dense captioning.
- 5 CONCLUSION: SPHINX can integrate with SAM for language-referred segmentation and Stable Diffusion for image editing.These integrations extend its functionality beyond the listed vision-generalist tasks.