Source-linked AI summary
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, Yu Qiao, Peng Gao
TL;DR
SPHINX-X addresses limited task-domain coverage and parameter choices in open-source MLLMs. It modifies SPHINX’s architecture and training, expands data coverage, and finds that larger datasets and parameter scales improve multimodal performance.
Problem
Open-source MLLMs remain constrained by limited task domains and LLM parameter choices, weakening coverage of specialized fields and limiting deployment flexibility.
Method
SPHINX-X combines architectural and training changes with a large multi-domain dataset and supports MLLMs spanning 1B to 8×7B parameters.
Results
Extensive evaluations show that SPHINX-Plus surpasses SPHINX and that scaling parameters from 1.1B to 7×8B consistently boosts multimodal understanding.
Takeaways & Limitations
Training-data scale and LLM parameter size both play critical roles in MLLM performance across models targeting mobile devices through high-end computers.
Takeaways & Limitations
SPHINX-Plus marginally underperforms existing methods on video analysis because it was not fine-tuned with video data.
Abstract
from arXiv · showhide
We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX. To improve the architecture and training efficiency, we modify the SPHINX framework by removing redundant visual encoders, bypassing fully-padded sub-images with skip tokens, and simplifying multi-stage training into a one-stage all-in-one paradigm. To fully unleash the potential of MLLMs, we assemble a comprehensive multi-domain and multimodal dataset covering publicly available resources in language, vision, and vision-language tasks. We further enrich this collection with our curated OCR intensive and Set-of-Mark datasets, extending the diversity and generality. By training over different base LLMs including TinyLlama1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities. Comprehensive benchmarking reveals a strong correlation between the multi-modal performance with the data and parameter scales. Code and models are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory
1. Introduction
SPHINX-X addresses limited task coverage and model-scale choices in open-source MLLMs by combining architectural and training changes with broader data and parameter scaling. Evaluations report improved generalization, with performance increasing as dataset diversity and model parameters expand.
- Results: Demonstrations cover high-resolution detection, bilingual understanding, text spotting, visual programming, and screenshot analysis.The examples illustrate SPHINX-X as a multi-purpose vision generalist across diverse domains.
- Motivation: Open-source MLLMs often specialize in natural images, leaving OCR, tables, charts, and mathematics underrepresented while 7B–13B models remain costly for portable deployment.The paper frames both limited task coverage and constrained parameter choices as central limitations.
- Approach: SPHINX-X extends SPHINX with broader task coverage and parameter scales, while modifying its architecture and training process for efficient large-scale multimodal training.The family spans models from fast mobile deployment to complex reasoning settings.
- Approach: The framework retains CLIP-ConvNeXt and DINOv2, represents fully padded sub-images with learnable skip tokens, and replaces multi-stage training with one-stage all-in-one optimization.These changes reduce redundant visual processing, shorten sequences, and train all collected datasets together.
- Data and model scaling: SPHINX-X combines public vision, language, and vision-language datasets with curated OCR-intensive and Set-of-Mark data in a unified conversational format.The dataset expansion targets broader multimodal task coverage and specialized capabilities.
- Results: SPHINX-Plus surpasses the original SPHINX, while comparisons from 1.1B to 7×8B show that larger datasets and parameter counts consistently improve multimodal understanding.The reported family includes TinyLlama-1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral-8×7B bases.
2. Related Work
Related work extends language models toward multimodal perception through diverse architectures and task-specific systems. Existing efforts include general vision-language models, referring detection, document and chart analysis, interleaved images, and visual mathematics.
- Positioning: The related-work landscape motivates SPHINX-X’s attempt to combine broad multimodal instruction tuning with coverage across diverse tasks and model scales.This connection follows from the breadth of architectures and specialized systems surveyed.
- Multimodal architectures: BLIP, MiniGPT-4, Flamingo, and LLaMA-Adapter represent major approaches for connecting visual features or interleaved image-text inputs with language models.These systems use query transformers, gated cross-attention, or adapter-based designs.
- Task-specific MLLMs: Task-oriented systems target referring object detection, tables, documents, scientific diagrams, multi-image instruction tuning, and visual mathematical problem solving.The related systems trade broad multimodal coverage for specialized capabilities in particular domains.
3. Method
SPHINX-X upgrades SPHINX with a concise architecture, efficient training pipeline, broad multimodal data, and multiple base LLM scales. Its design combines visual simplification, skip tokens, one-stage training, curated datasets, and varied language-model backbones.
- Architecture: SPHINX-X removes two of SPHINX’s four visual encoders, retaining DINOv2 and CLIP-ConvNeXt as complementary visual experts.The retained encoders differ in learning approach and architecture, providing complementary visual knowledge while reducing computational cost.
- Architecture: Fully padded high-resolution sub-images are represented with learnable skip tokens, shortening LLM input sequences while preserving their relative positions.The skip-token strategy avoids encoding sub-images whose pixels are all zero.
- Training: SPHINX-X replaces SPHINX’s manually configured two-stage pipeline with one-stage all-in-one training over uniformly formatted multimodal datasets.All LLM and intermediate projection parameters are trained together, while the two visual encoders remain frozen.
- Data: The training collection combines language, vision, and vision-language instruction data with curated OCR-intensive and Set-of-Mark datasets.The OCR-intensive data targets challenging real-world document understanding, while Set-of-Mark data adds fine-grained image-text correspondence.
- Model Family: SPHINX-X provides four base-LLM choices spanning TinyLlama-1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral-8×7B.These variants support comparisons across parameter scales, bilingual capability, and sparse mixture-of-experts modeling.
- Model Family: SPHINX-Plus uses the 13B LLaMA2 backbone with a more diverse, larger-scale multimodal dataset and one-stage training, while SPHINX-Plus-2K increases resolution to 672 × 672 with 3 × 3 sub-images.InternLM2-7B is used to explore bilingual multimodal reasoning, and TinyLlama-1.1B targets lower-resource settings.
4. Experiment
SPHINX-X is evaluated across broad multimodal, VQA, grounding, specialized, and video benchmarks. The results show strong generalization, while video performance is constrained by the absence of video training and temporal modeling.
- Benchmark coverage: SPHINX-X is assessed on a broad suite of multimodal benchmarks spanning general understanding, VQA, grounding, specialized reasoning, and video analysis.The evaluation includes MLLM benchmarks, academic and text-oriented VQA, referring expression comprehension, MathVerse, SciVerse, and Video-Bench.
- Visual question answering: SPHINX-X excels across general VQA benchmarks, covering visual understanding, relational reasoning, scientific contexts, and symbolic visual reasoning.It also achieves competitive performance on text-related benchmarks despite using only a limited portion of OCR data.
- Specialized benchmarks: SPHINX-MoE achieves the best performance among open-source models on MathVerse and SciVerse, but struggles on multi-discipline MMMU and CMMMU tasks.The authors attribute the multi-discipline limitation to insufficient multi-modal multi-disciplinary training data.
- Video analysis: SPHINX-Plus significantly outperforms video-specialized models on Video-Bench, especially for video-exclusive understanding and prior-knowledge question answering.It uses evenly sampled frames with the middle frame selected as the representative input because the model is image-based and lacks video training.
- Video analysis: Video performance is lower on challenging motion datasets because SPHINX-Plus lacks video fine-tuning and does not model temporal relationships.The reported limitation concerns comparison with existing state-of-the-art methods on these datasets.
- Qualitative capabilities: SPHINX-X demonstrations cover high-resolution detection, text spotting, bilingual conversations, visual programming, GUI analysis, geometric reasoning, and pose estimation.The appendix further demonstrates Set-of-Marks understanding, document layout detection, and character recognition from OCR-intensive training.
5. Conclusion
SPHINX-X is a family of multimodal instruction-tuned models spanning 1B to 8×7B parameters. It combines architectural and training simplifications with a large multi-domain dataset and reports strong benchmark performance and generalization.
- Conclusion: SPHINX-X is a family of multi-purpose multimodal instruction-tuned models with LLM parameter sizes ranging from 1B to 8×7B.The family is built on the original SPHINX framework.
- Conclusion: The framework removes redundant visual encoders, uses skip tokens for fully padded sub-images, and simplifies multi-stage training into one-stage all-in-one training.These changes target the architecture and training process of SPHINX-X.
- Conclusion: SPHINX-X uses a large multi-domain dataset containing public resources and targeted constructed data for multimodal training.The targeted data extends the collection for specialized multimodal capabilities.
- Conclusion: Extensive benchmarks demonstrate strong performance and generalization capacity for SPHINX-X.The conclusion presents this as the overall outcome of the proposed model family and training approach.
Impact Statement
SPHINX-X may improve multimodal services and language inclusivity, while its deployment raises dataset-bias and deceptive-content risks requiring responsible governance.
- Enhanced Multimodal AI Applications: SPHINX-X could support more sophisticated systems that understand and interact with text and visual input.Potential applications include automated translation, image recognition, and assistive technologies for people with disabilities.
- Inclusivity and Language Diversity: Training on diverse, multi-domain, and multimodal data may broaden language support and help bridge communication gaps.
- Ethical and Bias Considerations: Expansive training datasets require careful curation to avoid perpetuating biases and to support fair, ethical responses.
- Misuse Risks: Misuse of MLLMs to generate deceptive content presents a risk that calls for robust governance and ethical-use frameworks.
- Responsible deployment requires considering these potential impacts to maximize benefits and minimize negative consequences.
A.1.1. INFERENCE WITH DIFFERENT NUMBERS OF ACTIVATING EXPERTS
SPHINX-MoE generally performs best when inference activates the same two experts used during training, although MME instead favors four activated experts.
- Two activated experts yield the best results on ScienceQA, TextQA, RefCOCO, and Mathvista.This setting matches the training configuration.
- Four activated experts perform best on MME, while two experts produce the second-lowest performance there.The authors describe this departure from the training setting as inconsistent and interesting.
A.1.2. EXPERTS’ USAGE DISTRIBUTION ON DIFFERENT DOMAINS AND DIFFERENT MODALITIES
Expert usage in SPHINX-MoE shows no obvious specialization by the tested domain or modality, while layer-end distributions are flatter than those in middle layers.
- Different domains: Experts show similar usage distributions across artwork, celebrity, and OCR subtasks.
- Different modalities: No specific experts mainly handle one modality across vision, language, and vision-language inputs.
- Expert usage is flatter in layers at both ends of the model than in its middle layers.
A.1.3. PRUNE SOME OF THE EXPERTS WHEN INFERENCE
Pruning experiments show that SPHINX-MoE’s retained ability depends strongly on which experts are removed, not only on how many remain.
- The pruning study retains n experts and randomly removes 8 − n experts from each layer, averaging three runs per n.The x-axis reports the number of retained experts after pruning.
- Some experts are important: pruning particular experts can cause major ability loss, whereas other runs preserve most ability with the same pruning count.
- Keeping important experts while pruning less important ones preserves most of SPHINX-MoE’s ability.
A.2. Video Analysis on MVBench
On MVBench, SPHINX-Plus performs strongly despite being image-based, but its lack of video fine-tuning limits performance on tasks requiring temporal modeling.
- SPHINX-Plus significantly outperforms existing video-specialized models on MVBench despite being an image-based model.It particularly excels in video-exclusive understanding and prior knowledge-based question answering.
- SPHINX-Plus shows strong visual perception and knowledge extraction capabilities across MVBench’s 20 video-understanding sub-aspects.MVBench enables finer-grained comparison by decomposing video understanding into 20 sub-aspects.
- SPHINX-Plus performs slightly worse than existing state-of-the-art methods on challenging MOT datasets.The paper attributes this gap to temporal relationships because SPHINX-Plus was not fine-tuned on video data.