Source-linked AI summary
Towards artificial general intelligence via a multimodal foundation model
Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guoxing Yang, Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang, Hao Sun, Ji-Rong Wen
TL;DR
Most existing AI methods provide only single-cognitive abilities, motivating a foundation model that can address broader cognitive tasks. The paper develops BriVL through self-supervised pre-training on massive weakly correlated image-text data and reports promising downstream performance together with visually revealed imagination. The authors present these capabilities as a step toward AGI while noting risks and unresolved challenges.
Problem
Most existing AI advances focus on matching or exceeding human performance in single cognitive abilities rather than supporting a broad class of cognitive tasks.
Method
BriVL is self-supervisedly pre-trained on 650 million weakly correlated image-text pairs crawled from the Internet and uses multimodal interpretability tools.
Results
Experiments across downstream tasks report promising cross-modal understanding and cross-domain transfer, while interpretability tools visually reveal strong imagination or association abilities.
Takeaways & Limitations
The authors conclude that BriVL exhibits imagination and reasoning abilities and represents a step toward more general foundation models and AGI.
Takeaways & Limitations
Foundation models may learn prejudices and stereotypes from pre-training data, can be misused to generate fake content, and remain incompletely understood.
Abstract
from arXiv · showhide
The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of human. Despite tremendous success in the AI research, most of existing methods have only single-cognitive ability. To overcome this limitation and take a solid step towards artificial general intelligence (AGI), we develop a foundation model pre-trained with huge multimodal data, which can be quickly adapted for various downstream cognitive tasks. To achieve this goal, we propose to pre-train our foundation model by self-supervised learning with weak semantic correlation data crawled from the Internet and show that promising results can be obtained on a wide range of downstream tasks. Particularly, with the developed model-interpretability tools, we demonstrate that strong imagination ability is now possessed by our foundation model. We believe that our work makes a transformative stride towards AGI, from our common practice of "weak or narrow AI" to that of "strong or generalized AI".
Introduction
BriVL is proposed to address the limitation that most AI systems specialize in single cognitive abilities by pre-training a multimodal foundation model on weakly correlated image-text data. The model is designed for broad downstream adaptation and is evaluated for cross-modal understanding, transfer, and imagination.
- Most existing AI advances target individual cognitive abilities, motivating a model that can process multiple modalities and support broader cognition.
- Foundation models are pre-trained on broad data at scale so they can be adapted to varied downstream cognitive tasks.
- BriVL is self-supervisedly pre-trained on 650 million weak-semantic-correlation image-text pairs crawled from the Internet without human annotation.
- Because weakly correlated image-text pairs lack necessary fine-grained region-word matches, BriVL uses image-text matching with separate image and text encoders in a two-tower architecture.
- Unlike comparable models described in the paper, BriVL retains the weak semantic-correlation assumption and crawls Internet data with limited filtering beyond pornographic or sensitive content.
- BriVL is evaluated across downstream tasks and reported to show cross-modal understanding, cross-domain transfer, and strong imagination revealed through interpretability tools.
Results
BriVL exhibits imagination and cross-domain transfer after multimodal pre-training on weakly correlated image-text data. Visualizations and downstream experiments show coherent concept generation, stronger results across several tasks, and improved associations over single-modal baselines.
- Neural Network Visualization: BriVL visualizations turn abstract concepts into concrete objects, suggesting learned common-sense representations from weakly correlated multimodal data.Examples include plants for “nature,” a clock for “time,” and a face with glasses and a flask for “science.”
- Neural Network Visualization: BriVL generates visualizations that capture literal and implicit meanings in complex sentences, including imagery from ancient Chinese poems.The model represents scenes such as bamboo with peach blossoms and mountains, sunset, river, and boat imagery.
- Neural Network Visualization: Shared prompts produce context-sensitive mountain scenes, linking specific objects such as forests, stones, snow, and waterfalls with broader visual settings.The generated imagery changes coherently with the semantic restriction while preserving the mountain context.
- Text-to-Image Generation: BriVL-generated images are more realistic and globally coherent than CLIP examples, while independently generated images maintain visual coherence across a sentence series.The authors attribute these differences to BriVL’s web-crawled, weakly correlated training data and whole-image-text modeling.
- Text-to-Image Generation: BriVL generates plausible images for rare or nonexistent concepts, including “blazing sea,” “glowing forest,” “cyberpunk-styled city,” and “castle in the clouds.”These examples are presented as evidence of generalization and imagination beyond memorizing pre-training examples.
- Downstream Cognitive Tasks: Zero-shot experiments report stronger cross-domain transfer than comparison systems in remote sensing and news classification, with multimodal BriVL variants outperforming corresponding single-modal baselines.On remote sensing, BriVL exceeds ZSSC and CLIP variants; on news classification, BriVL improves over RoBERTa-base and RoBERTa-large.
Discussion
BriVL is presented as a multimodal foundation model that may support more cognitive and general capabilities through weakly correlated image-text pre-training. The discussion also identifies societal risks, adaptation challenges, and substantial open directions for understanding and extending such models.
- Discussion: BriVL is trained on 650 million weakly correlated image-text pairs and is reported to show cross-domain transfer, imagination, and reasoning abilities.The authors attribute these advantages mainly to fusing complex emotions and thoughts contained in weakly correlated data.
- Discussion: The framework is intended to support larger models with more modalities and facilitate applications of multimodal foundation models across AI-related fields.The authors specifically mention extending BriVL to larger capacity and additional modalities.
- Discussion: Foundation models may learn prejudices and stereotypes from pre-training data, while increasingly capable models may also be misused to manipulate or generate fake content.The paper calls for careful handling before training and monitoring or addressing these issues in downstream applications.
- Discussion: Key academic challenges include deeper interpretability, larger multimodal datasets, and more effective downstream adaptation and finetuning techniques.These challenges concern both extending model capabilities and applying foundation models effectively across tasks.
- Discussion: The authors state that understanding what BriVL has learned and what it can do is only beginning, leaving room for further study and new use cases.They suggest multilingual data, translation as a by-product, and video or audio modalities as future directions toward AGI.
Methods
BriVL uses separate image and text encoders to map both modalities into a shared embedding space, trained with cross-modal contrastive learning and large negative queues. Its image pathway combines multiscale patch pooling, self-attention, and an MLP, while text is encoded with a Transformer and similarly projected.
- Cross-modal pre-training: BriVL learns image and text encoders that map both modalities into the same d-dimensional semantic space for image-text retrieval.The training objective identifies corresponding image embeddings for text embeddings and vice versa.
- Contrastive learning: Cross-modal InfoNCE learning matches corresponding image-text embeddings while contrasting them against positive and negative samples from separate queues.BriVL maintains image and text negative queues, with momentum encoders used to stabilize features collected across mini-batches.
- Image encoder: The image encoder uses a CNN backbone, multiscale patch pooling, self-attention, and an MLP to produce image embeddings.Images are split into 37 patches at two scales, pooled into patch features, fused by self-attention, and projected into the joint space.
- Text encoder: The text encoder tokenizes sentences, applies a pre-trained Transformer and self-attention, then projects the representation through a two-layer ReLU MLP.The resulting text embedding is d-dimensional and lies in the joint cross-modal embedding space.
- Architecture: BriVL uses a four-tower architecture with momentum encoders and negative queues for multimodal pre-training.The design extends MoCo for multimodal learning in a more GPU-resource-saving way.
- Text-to-image generation: For text-to-image generation, BriVL optimizes a randomly initialized code collection whose generated image is matched to the input text embedding.A pre-trained VQGAN codebook and CNN generator convert the learned spatial code collection into an image.
Data availability
The study identifies public locations for the datasets used in evaluation and pre-training. The listed resources cover remote sensing, news classification, cross-modal retrieval, visual question answering, and the pre-training corpus.
- Evaluation datasets: The remote sensing datasets are UC Merced Land-Use and AID.Their listed resources support remote sensing scene classification.
- Evaluation datasets: The news classification datasets are THUCNews and Toutiao News.Both resources are listed as datasets used for news classification.
- Evaluation datasets: The Chinese cross-modal retrieval dataset AIC-ICC is available through the listed GitHub repository.The repository is identified as the source for AIC-ICC.
- Evaluation datasets: The Visual7W dataset is listed as the visual question answering resource.Its Stanford webpage is provided as the dataset location.
- Pre-training data: The pre-training dataset is available through the WuDao AI resource site.The listed resource corresponds to the dataset used to pre-train BriVL.
- Usage: Datasets (1)–(4) are sufficient for finetuning BriVL to interpret, verify, and extend the research.This availability statement concerns the four evaluation dataset groups listed above.
Code availability
The pre-trained BriVL model and its inference code are publicly available through a GitHub repository under a specified Creative Commons license.
- Code availability: The pre-trained BriVL and its inference code are available on GitHub under the CC BY-NC-ND 4.0 International Licence.The repository link is provided in the paper’s code-availability statement.
Competing interests
The supplementary material identifies the figure as an illustration of BriVL’s design and existing pre-training architectures. The paper separately states that the authors declare no competing interests.
- Competing interests: The authors declare no competing interests.This statement appears as the paper’s competing-interests disclosure.
- Supplementary figure: Fig. S1 compares single-modal textual and visual, single-tower multimodal, two-tower multimodal, and BriVL architectures.Its panels cover BriVL’s algorithm design, image-encoder architecture, and comparison with existing pre-training architectures.
Architecture Overview
The supplementary material points to schematic illustrations of BriVL’s overall model, image encoder, and existing pretraining architectures.
- Fig. S1a presents a schematic illustration of the proposed BriVL model for large-scale multimodal pretraining.
- Fig. S1b shows the network details of BriVL’s image encoder.
- Fig. S1c illustrates existing architectures for pretraining.
Ablation Study
The ablation study evaluates self-attention and contrastive-learning choices using reduced training data, while retrieval examples illustrate weak-correlation learning.
- The ablation study examines self-attention layers in both encoders and alternative contrastive losses, including MoCo-based cross-modal contrastive learning.
- 22M image-text pairs were used for training and 11K for evaluation because full-dataset ablations were too costly.All ablation experiments used the same nine machines, each with eight NVIDIA A100 GPUs.
- BriVL with SimCLR performed worse than standard MoCo-based BriVL on all 7 evaluation metrics.The comparison used a total batch size of 2,160 for SimCLR versus 1,728 for standard BriVL, which also maintained two negative-sample queues of size 10,368.
- BriVL’s top-five text retrievals for an image of a cup of tea were all philosophical sentences.The example used a standard BriVL model trained on 22M data, with texts drawn from the 11K test set.
Results on English Tasks
BriVL-en is pretrained on English image-caption datasets and evaluated on visual question answering and image captioning, where finetuning outperforms direct training.
- BriVL-en was pretrained on four publicly available image-captioning datasets.The described collection includes MSCOCO and Flickr30K training data, among other datasets.
- The experiments were intended to demonstrate BriVL-en’s potential on visual question answering and image captioning.The reported similar results were presented as a feasible solution beyond specific languages.
- Visual Question Answering: On Visual7W, BriVL-en with pretraining and finetuning outperformed direct training by large margins across all evaluation metrics.The zero-shot pretrained variation performed much worse, mainly because of the domain gap and different data distributions.
- Image Captioning: On Flickr30K image captioning, BriVL-en with pretraining and finetuning outperformed direct training on all five evaluation metrics.Zero-shot experiments were unavailable because image captioning required an additional Transformer decoder.
- Image Captioning: Captioning examples showed the finetuned model identifying a white parked car, a dog catching a ball, and a dog running on a beach.The examples were presented as evidence of commonsense-related interpretations in image descriptions.
Image Sources
The work uses publicly sourced images, primarily from Pexels, and documents the image sources in supplementary materials.
- Images used in the main manuscript and supplementary note, except those owned by the authors, were taken from Pexels.Pexels provides free stock photos and permits free downloading under its license.
- Table S1 lists the images taken from public sources.