Source-linked AI summary
M6: A Chinese Multimodal Pretrainer
Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, Jie Zhang, Jianwei Zhang, Xu Zou, Zhikang Li, Xiaodong Deng, Jie Liu, Jinbao Xue, Huiling Zhou, Jianxin Ma, Jin Yu, Yong Li, Wei Lin, Jingren Zhou, Jie Tang, Hongxia Yang
TL;DR
Chinese multimodal pretraining lacks datasets and models at the scale needed for broad single- and cross-modal learning. This work introduces M6-Corpus and the M6 multitask mega-transformer, scales models to 10B and 100B parameters, and reports strong downstream performance alongside controllable text-guided image generation.
Problem
Large-scale Chinese multimodal pretraining is constrained by insufficient Chinese datasets and pretrained models, especially resources containing both text and image-text pairs.
Method
M6 unifies single-modality and multimodality understanding and generation through Transformer-based multitask pretraining on Chinese text and multimodal data.
Results
M6 outperforms strong baselines by 11.8% in VQA, 18.4 in image captioning, and 10.3% in image-text matching.
Takeaways & Limitations
The finetuned M6 can perform controllable text-guided image generation using learned relations between text and discrete image codes.
Takeaways & Limitations
Expert overloading can be severe: tokens exceeding an expert's capacity are dropped from that computation and passed through residual connections.
Abstract
from arXiv · showhide
In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation, and show that the finetuned M6 can create high-quality images with high resolution and abundant details.
1 INTRODUCTION
M6 addresses the shortage of large-scale Chinese multimodal pretraining resources by combining a large Chinese corpus with a unified multimodal multitask model. Scaled versions support diverse downstream applications, outperform strong baselines, and enable text-guided image generation.
- 1 INTRODUCTION: The corpus contains over 1.9TB of images and 292GB of texts covering diverse domains and data types.It includes encyclopedia, question answering, forum discussion, and product description data.
- 1 INTRODUCTION: M6 combines a large Chinese multimodal corpus with unified pretraining for single-modality and multimodality understanding and generation.The model is based on the Transformer and is pretrained with multiple tasks.
- 1 INTRODUCTION: M6 is scaled to 10 billion and 100 billion parameters, with M6-100B described as the largest model pretrained on Chinese data.The work applies M6 to product description generation, visual question answering, community question answering, and Chinese poem generation.
- 1 INTRODUCTION: A two-stage framework uses discrete image codes and pretrained M6 to learn text-code relations for controllable text-to-image generation.The image representation is produced by a trained vector-quantized generative adversarial network.
- 1 INTRODUCTION: M6 outperforms strong baselines by 11.8% in VQA, 18.4 in image captioning, and 10.3% in image-text matching.The contribution summary reports these comparisons across three multimodal tasks.
2 DATASET
The dataset is designed to support large-scale Chinese multimodal pretraining through broad domain coverage, paired images and text, and extensive preprocessing. It addresses the limited scale of existing Chinese corpora while emphasizing data quality.
- 2.1 Existing Datasets: Existing Chinese pretraining datasets remain too small for super-large-scale pretraining and generally lack the image-text pairs needed for multimodal learning.The passage contrasts existing resources with GPT-3's 570GB text training corpus.
- 2.2 Data Requirements: The construction requirements prioritize fluent text, realistic images, adequate image resolution, broad domain coverage, and high-quality multimodal data.These criteria are presented as desirable properties for multimodal pretraining corpora.
- 2.3 Dataset Construction: M6-Corpus combines plain texts and image-text pairs across encyclopedia, webpages, community questions, forums, and product descriptions.Its domain coverage includes science, entertainment, sports, politics, and commonsense of life.
- 2.3 Dataset Construction: The corpus is described as the first large-scale, multimodal, multidomain corpus for Chinese pretraining and is much larger than previous Chinese datasets.The collected data are compatible with both text-only and multimodal pretraining tasks.
- 2.3 Dataset Construction: Sophisticated preprocessing removes HTML markup, duplicate punctuation, short content, and sentences flagged by in-house spam detection.The stated filtering targets include malformed text and content associated with specified undesirable categories.
3 M6 FRAMEWORK
M6 is a unified multimodal pretraining framework that combines visual and linguistic inputs, shared transformer representations, and multiple masking-based tasks. It scales through dense and MoE variants to support large Chinese multimodal models.
- 3.1 Visual and Linguistic Inputs: The pretraining corpus combines plain text and image-text pairs from diverse sources and domains, including encyclopedia, web, community QA, forums, and product descriptions.The dataset statistics tables summarize sources, images, tokens, passages, average lengths, and data sizes.
- 3.1 Visual and Linguistic Inputs: M6 unifies visual and linguistic inputs in a transformer with modality-specific segment embeddings for cross-modal representation learning.Image and word embeddings are combined into one sequence before transformer processing.
- 3.3 Pretraining Methods: Different masking strategies let M6 encode and decode visual inputs, masked language, and complete language within one framework.Bidirectional masking is used for visual and masked linguistic inputs, while causal masking is used for complete linguistic inputs.
- 3.3 Pretraining Methods: M6 supports text-to-text, image-to-text, and multimodality-to-text transfer for single-modal and cross-modal understanding and generation.The pretraining tasks include text denoising, language modeling, image captioning, and related multimodal transfer settings.
- 3.4 Scaling up to 10 and 100 Billion Parameters: M6-100B uses GPU-based MoE expert parallelism, routing each token to a small set of experts to reduce computation relative to dense models.The model combines multiple expert FFNs with sparse gating and implements model parallelism through the Whale framework.
- 3.4 Scaling up to 10 and 100 Billion Parameters: M6-100B achieves similar performance to dense M6-10B within 2–3 times shorter training time, with negative log perplexity of −2.297 versus −2.253.The comparison supports MoE-based scaling as more time-efficient than dense scaling in this setting.
4 APPLICATIONS
M6 is applied across text-to-image generation, visual question answering, image captioning, general question answering, poem generation, and image-text matching. Across these applications, it produces high-quality images, supports broad-domain generation, and outperforms baselines on several evaluated tasks.
- 4.1 Text-to-Image Generation: M6 enables controllable text-to-image generation through a two-stage framework that learns relations between text and discrete image codes.The framework first represents images with discrete codes, then finetunes M6 to model text-code sequences.
- 4.1 Text-to-Image Generation: Generated images have high quality, resemble real objects, and can depict novel concepts such as military-style camouflage high heels and robot clip art.The model also adapts to more general image domains beyond e-commerce products.
- 4.2 Visual Question Answering: +6.2% accuracy over the baseline is achieved by M6-base on FMIQA, while scaling to M6-10B adds a further 5.2% improvement.The evaluation uses human assessment on a manually sampled test set.
- 4.3 Image Captioning: M6-base outperforms the transformer baseline in image-captioning grammar, correctness, and richness, with gains of +18.2% and +14.4% in correctness and richness.M6-10B further improves correctness by about 14.7% and richness by 7.0%.
- 4.4 General Question Answering and Poem Generation: Without finetuning, pretrained M6 answers general questions, generates fluent explanatory text, and produces poems under format constraints.The model can provide related background knowledge, although detailed visual attributes remain difficult for pseudo-parallel image-text pretraining.
- 4.6 Image-Text Matching: M6-base outperforms similarly sized InterBert by 10.3% on image-text matching, partly because patch features avoid limitations from sparse detected objects.The paper notes that product images generally contain relatively few detected objects.
5 RELATED WORK
Related work extends NLP pretraining and Transformer architectures to cross-modal representation learning. Existing multimodal models mainly use single-stream or dual-stream designs with tasks such as masked language modeling, masked object classification, and image-text matching.
- 5 RELATED WORK: Cross-modal representation learning is motivated by the success of NLP pretraining and Transformer architectures.The related work positions multimodal pretraining as an extension of these approaches.
- 5 RELATED WORK: Existing multimodal pretraining architectures are mainly single-stream or dual-stream models trained with masked language modeling, masked object classification, and image-text matching.The passage cites VisualBERT, Unicoder-VL, UNITER, and Oscar as representative examples.
6 CONCLUSIONS
The paper introduces M6-Corpus and M6 for large-scale Chinese multimodal pretraining, scaling models to 10B and 100B parameters. It applies the models broadly and reports high-quality text-guided image generation.
- 6 CONCLUSIONS: M6-Corpus contains over 1.9TB images and 292GB texts across diverse domains, while M6 supports single-modal and cross-modal understanding and generation.M6 is scaled to 10B and 100B parameters, described as the largest Chinese multimodal pretrained models.
- 6 CONCLUSIONS: The finetuned M6 produces high-quality images in the paper’s text-guided image-generation task.The authors plan to continue increasing data and model scale and explore additional downstream applications.