PaperlayerView exact revision on arXiv

Source-linked AI summary

SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, Yu Qiao

arXiv:2311.07575v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Fine-grained visual perception in MLLMs is constrained by low-resolution inputs. SPHINX jointly mixes model weights, visual instruction-tuning tasks, visual embeddings, and high-resolution image scales, and reports strong benchmark performance, surpassing prior state-of-the-art MLLMs on 6 of 10 benchmarks.

  • Problem

    Low-resolution inputs constrain fine-grained visual perception in MLLMs, while simply upsampling images can harm spatial cues and increase computation.

    Source: §1 · p. 2, §3.2 · p. 7

  • Method

    SPHINX jointly mixes real-world- and synthetic-domain model weights, diverse instruction-tuning tasks, visual embeddings, and high-resolution scales and sub-images.

    Source: §1 · p. 2, §3.1 · pp. 5–6, §5 · p. 17

  • Results

    SPHINX reports strong performance across multimodal benchmarks, surpassing prior state-of-the-art MLLMs on 6 of 10 benchmarks.

    Source: §4.2 · p. 12, §5 · p. 18

  • Takeaways & Limitations

    SPHINX can serve as a vision generalist for object detection, region-level captioning, and human pose estimation.

    Source: §5 · p. 18

Abstract

from arXiv · show

We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, to enable multi-purpose capabilities, we mix a variety of tasks for joint visual instruction tuning, and design task-specific instructions to avoid inter-task conflict. In addition to the basic visual question answering, we include more challenging tasks such as region-level understanding, caption grounding, document layout detection, and human pose estimation, contributing to mutual enhancement over different scenarios. Additionally, we propose to extract comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity, providing language models with more robust image representations. Based on our proposed joint mixing, SPHINX exhibits superior multi-modal understanding capabilities on a wide range of applications. On top of this, we further propose an efficient strategy aiming to better capture fine-grained appearances of high-resolution images. With a mixing of different scales and high-resolution sub-images, SPHINX attains exceptional visual parsing and reasoning performance on existing evaluation benchmarks. We hope our work may cast a light on the exploration of joint mixing in future MLLM research. Code is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.

1 INTRODUCTION

SPHINX combines model-weight, task, visual-embedding, and high-resolution-image mixing to build a versatile multi-modal model. It supports diverse visual tasks and achieves state-of-the-art performance on several evaluation benchmarks.

1 INTRODUCTION

  • SPHINX unfreezes the LLM during pre-training, combining vision-language data for cross-modal alignment with RefinedWeb for language-specific tuning to alleviate forgetting.The paper says this enables the LLM to learn more cross-modal knowledge while retaining detailed language response generation.Source: §1 · p. 2
  • SPHINX linearly combines weights from real-world-pretrained and synthetic-data-finetuned LLMs to integrate domain-specific knowledge without mixing potentially contradictory data directly.The synthetic-data model is fine-tuned from the real-world-pretrained model before their weights are combined.Source: §1 · p. 2
  • Joint visual instruction tuning combines VQA, region-level referring, detection and relation reasoning, chart/document VQA, and human pose estimation for generalist visual instruction following.The paper also includes caption grounding among the challenging tasks motivating its broader multi-task approach.Source: §1 · p. 2
  • SPHINX mixes embeddings from CNN and ViT backbones, supervised and self-supervised pre-training, and global and local features to strengthen visual representations and alignment.The embeddings are mixed both channel-wise and sequence-wise.Source: §1 · p. 2
  • For high-resolution images, SPHINX combines a downsampled image with spatial sub-images as token groups, maintaining encoding efficiency while LLM attention enables information exchange among otherwise isolated sub-images.This approach addresses limited input resolution without simply upsampling images, which the paper says can disrupt positional priors and increase computational cost.Source: §1 · pp. 2, 4
  • SPHINX achieves state-of-the-art performance on MMBench, MME, and POPE, with strong visual reasoning and fine-grained perception of high-resolution images.The introduction also highlights object detection, relation parsing, and interpretation of complicated figures.Source: §1 · p. 4

2 RELATED WORK

Prior work developed instruction-following LLMs and increasingly capable vision-language models; SPHINX builds on LLaMA-2 and LLaMA-Adapter experience with a three-fold mixer for multimodal input.

LLMs and MLLMs

  • SPHINX builds on LLaMA-2 and LLaMA-Adapter instruction-tuning experience, using a three-fold mixer of model weights, tuning tasks, and visual embeddings for multimodal input.The paper frames this mixer as extending the capability of instruction-following LLMs.Source: §2 · pp. 4–5
  • The LLM landscape includes pre-training and task fine-tuning, scaling, instruction tuning, and higher-quality instruction datasets, alongside open-source LLaMA-based models.SPHINX adopts LLaMA-2 (Touvron et al., 2023b) and draws on LLaMA-Adapter instruction-tuning experience (Zhang et al., 2023a; Gao et al., 2023b).Source: §2 · p. 4
  • Prior MLLMs aligned LLMs with vision for captioning and question answering, interleaved image-text understanding, visual instruction following, and additional modalities and tasks.Examples include VisualGPT and BLIP, Flamingo and Kosmos-1, LLaVA and MiniGPT-4, and ImageBind-LLM and PandaGPT (Chen et al., 2022; Li et al., 2023d; Alayrac et al., 2022; Huang et al., 2023; Liu et al., 2023d; Zhu et al., 2023; Han et al., 2023; Su et al., 2023).Source: §2 · pp. 4–5

3 SPHINX

SPHINX combines LLM weight mixing, multi-task visual instruction tuning, and heterogeneous visual embeddings, then extends this design to high-resolution images and downstream applications.

3.1 THE JOINT MIXING OF MODEL WEIGHTS, TUNING TASKS, AND VISUAL EMBEDDINGS

  • SPHINX unfreezes the LLM during alignment pretraining while freezing vision encoders, adds language-only text samples to preserve long-sentence generation, and mixes LLM weights tuned on real-world and synthetic data.The separately tuned weights are combined by weighted mixing to aggregate domain-specific semantics and allow flexible extension to new data domains.Source: §3.1 · pp. 5–6
  • Instruction tuning spans VQA, grounding, detection and relation reasoning, chart/document VQA, and pose estimation, while task-specific prompts reduce conflict and visual embeddings mix architectures, pretraining paradigms, and granularity.The visual features combine patch-level encodings with Q-Former global representations, which are projected and concatenated before being supplied as context to the language model.Source: §3.1 · pp. 6–7

3.2 THE MIXING OF SCALES AND HIGH-RESOLUTION SUB-IMAGES

  • SPHINX encodes a downsampled global view and four 224×224 corner crops as separate token groups, preserving fine detail with 224×224 positional encodings and LLM-based patch interaction.For a 448×448 input, the five image groups are concatenated; this avoids directly upsampling the image and its positional encodings, which would increase computation and risk harming spatial cues.Source: §3.2 · pp. 7–8

3.3.1 INTEGRATION WITH SAM AND STABLE DIFFUSION

  • SPHINX’s predicted boxes prompt SAM for language-referred segmentation; the resulting masks guide LaMa object removal and Stable Diffusion generation for language-driven image editing.This integration combines SPHINX’s semantic reasoning and detection with SAM’s class-agnostic segmentation.Source: §3.3.1 · p. 8

3.3.2 FINE-TUNING SPHINX FOR VISUAL RECOGNITION

  • 70.8% ImageNet-1K classification accuracy is achieved after one epoch without augmentation; the passage contrasts this with ResNet and ViT’s roughly 300 training epochs and strong-augmentation requirements.SPHINX is adapted by converting classification into a single-turn instruction-and-response format and fine-tuning on the resulting text-converted dataset.Source: §3.3.2 · p. 9

4 EXPERIMENTS

Experiments describe SPHINX’s two-stage training, benchmark evaluation, and demonstrations of capabilities across visual tasks. The reported results include broad benchmark performance and qualitative examples of cross-task reasoning and perception.

4.1 TRAINING DETAILS

  • Training uses image-caption data with RefinedWeb during pre-training, then diverse instruction, VQA, OCR, detection, pose, and grounding datasets for multi-task fine-tuning.The two-stage pipeline fine-tunes the LLM and visual projections while keeping visual encoders frozen; fine-tuning datasets are converted to multi-turn conversation format.Source: §4.1 · pp. 9–10

4.2 QUANTITATIVE EVALUATION

  • Jointly training with RefinedWeb preserves text-modeling capability while adapting the model for cross-modal understanding; without it, text-only loss grows significantly.Figure 7 shows image-caption loss descending similarly without RefinedWeb, while the text-only loss rises in less than one-tenth of the training schedule.Source: Figure 7 · p. 10, §4.2 · p. 11
  • SPHINX surpasses previous state-of-the-art MLLMs on 6 of 10 benchmarks, while high-resolution sub-images improve visual hallucination performance on POPE.The comparison includes BLIP-2, InstructBLIP, Shikra, Qwen, Fuyu, and LLaVA1.5.Source: Table 2 · p. 11, §4.2 · p. 12
  • SPHINX-1k and SPHINX-2k significantly outperform SPHINX on VQAv2 and text-oriented VQA, while SPHINX remains below Qwen-VL-7B on text-oriented VQA.The authors attribute the gap to a lack of text-related pre-training data and report comparative results across the VQA benchmarks.Source: §4.2 · p. 12, Table 3 · p. 12
  • SPHINX outperforms most state-of-the-art models, including G-DINO-L and Qwen-VL-7B, across referring-expression comprehension splits; higher-resolution variants improve over SPHINX.SPHINX-1k and SPHINX-2k also narrow the gap to UNINEXT, a specialist model using a larger input image size.Source: §4.2 · p. 12, Table 4 · p. 13

4.3 DEMONSTRATIONS

  • Demonstrations show SPHINX working with SAM for segmentation and handling pose estimation, region-focused reasoning, and diagram layout detection.The examples include accurate masks for small objects and pose keypoints for each of five people.Source: §4.3 · p. 13
  • Object-detection feedback helps SPHINX select relevant objects for coordinate-based reasoning, while additional user hints improve its identification of requested objects.The examples also illustrate contextual reasoning that integrates information from different image elements.Source: §4.3 · pp. 13–14
  • Mixing referring-object and pose-estimation tasks yields referring pose estimation from text descriptions, and SPHINX also localizes unsharp anomalies without related training data.The demonstrations additionally show multi-level dense captioning by detecting objects, then describing each detected region with simple and detailed captions.Source: §4.3 · pp. 14, 16

5 CONCLUSION

SPHINX jointly mixes pretrained LLM weights, diverse visual instruction-tuning tasks, and visual embeddings, with high-resolution scale and sub-image mixing to support fine-grained visual understanding. The conclusion presents it as a strong vision generalist with broad benchmark performance and potential for integration with other visual foundation models.

5 CONCLUSION

  • SPHINX jointly mixes LLM weights trained on real-world and synthetic data, diverse visual tasks, and embeddings from different vision backbones, and adds high-resolution scale and sub-image mixing.The high-resolution strategy is designed to improve fine-grained visual understanding.Source: §5 · pp. 16–17
  • SPHINX performs impressively across multimodal benchmarks and acts as a vision generalist for detection, region-level captioning, and pose estimation, with examples of anomaly detection and dense captioning.The conclusion also illustrates emergent referring pose estimation.Source: Figure 11 · p. 17, Figure 12 · p. 17, §5 · p. 18
  • SPHINX can integrate with SAM (Kirillov et al., 2023) for language-referred segmentation and Stable Diffusion for image editing; future work will broaden its vision-language tasks.Source: §5 · p. 18

Related papers

Loading 2311.07575v1…