PaperlayerView exact revision on arXiv

Source-linked AI summary

ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, Chang Zhou

arXiv:2305.11172v1cs.CVcs.CLcs.SDeess.AS

TL;DR

Existing representation models rarely provide a general solution spanning vision, audio, and language while supporting both within-modality learning and cross-modal alignment. ONE-PEACE addresses this gap with an extensible adapter-and-fusion architecture and two modality-agnostic contrastive tasks, achieving leading results across a broad range of uni-modal and multi-modal tasks. Its main limitation is weaker performance in zero-shot image-text retrieval and vision-language understanding.

  • Problem

    General models spanning vision, audio, and language remain rare, while pretraining must support both within-modality information and cross-modal alignment.

  • Method

    ONE-PEACE combines modality adapters, shared self-attention, modality FFNs, cross-modal contrastive learning, and intra-modal denoising contrastive learning.

  • Results

    Across 3 modalities, 11 tasks, and 16 datasets, ONE-PEACE achieves leading results across a wide range of uni-modal and multi-modal tasks.

  • Takeaways & Limitations

    The flexible architecture and modality-agnostic tasks support aligning and integrating representations across vision, audio, and language without vision or language pretrained initialization.

  • Takeaways & Limitations

    ONE-PEACE falls short of state-of-the-art results in zero-shot image-text retrieval and vision-language understanding tasks.

Abstract

from arXiv · show

In this work, we explore a scalable way for building a general representation model toward unlimited modalities. We release ONE-PEACE, a highly extensible model with 4B parameters that can seamlessly align and integrate representations across vision, audio, and language modalities. The architecture of ONE-PEACE comprises modality adapters, shared self-attention layers, and modality FFNs. This design allows for the easy extension of new modalities by adding adapters and FFNs, while also enabling multi-modal fusion through self-attention layers. To pretrain ONE-PEACE, we develop two modality-agnostic pretraining tasks, cross-modal aligning contrast and intra-modal denoising contrast, which align the semantic space of different modalities and capture fine-grained details within modalities concurrently. With the scaling-friendly architecture and pretraining tasks, ONE-PEACE has the potential to expand to unlimited modalities. Without using any vision or language pretrained model for initialization, ONE-PEACE achieves leading results on a wide range of uni-modal and multi-modal tasks, including image classification (ImageNet), semantic segmentation (ADE20K), audio-text retrieval (AudioCaps, Clotho), audio classification (ESC-50, FSD50K, VGGSound), audio question answering (AVQA), image-text retrieval (MSCOCO, Flickr30K), and visual grounding (RefCOCO/+/g). Code is available at https://github.com/OFA-Sys/ONE-PEACE.

1 Introduction

ONE-PEACE addresses the limited availability of general models spanning vision, audio, and language by combining an extensible architecture with modality-agnostic pretraining tasks. It achieves leading results across diverse uni-modal and multi-modal benchmarks without vision or language pretrained initialization.

  • Prior work largely uses separate uni-modal architectures, while general models spanning vision, audio, and language remain rare.
  • ONE-PEACE uses modality adapters, shared self-attention, and modality-specific FFNs to align and integrate representations across three modalities.Adapters convert raw inputs into feature sequences; shared attention enables cross-modal interaction, while modality FFNs extract within-modality information.
  • The model is designed for extensibility: adding a new modality requires injecting its adapter and FFNs into the architecture.
  • Cross-modal contrastive learning aligns semantic spaces, while intra-modal denoising contrastive learning captures fine-grained masked-unit information.The two tasks are designed to support both cross-modal retrieval and downstream fine-tuning.
  • 89.8% accuracy on ImageNet and 63.0% mIoU on ADE20K demonstrate strong vision performance without vision or language pretrained initialization.
  • ONE-PEACE achieves leading results across image, audio, vision-language, and audio-language tasks, including retrieval, classification, question answering, and visual grounding.

2 Related Work

Related work develops specialized vision-language, audio-language, and joint vision-audio-language models, but ONE-PEACE emphasizes a flexible architecture and universally applicable pretraining tasks without external models.

  • Vision-language pretraining evolved from detector-based region features toward Transformer models that jointly learn vision-language data.
  • Audio-language research includes unified speech-text encoders trained with masked prediction and paired-data alignment, alongside environmental-sound contrastive learning.
  • The architecture contains modality adapters and a modality fusion encoder, and can be disassembled into task-specific branches such as a vision branch.
  • Recent vision-audio-language models jointly learn video, text, and audio, often using cross-modal contrastive learning or external discretization models.
  • ONE-PEACE differs by combining a multi-modal-compatible architecture with universally applicable pretraining tasks that do not require external models.

3 Method

ONE-PEACE combines modality-specific input adapters with shared self-attention and modality-specific FFNs, enabling both modality processing and cross-modal interaction. It is pretrained with cross-modal alignment and intra-modal denoising contrastive tasks.

  • Architecture: Three modality adapters convert vision, audio, and language inputs into feature sequences for the shared model.The adapters use modality-appropriate processing before producing unified features.
  • Architecture: Shared self-attention enables interaction between modalities, while separate V-, A-, and L-FFNs extract modality-specific information.The architecture can also be assembled into uni-modal or multi-modal branches.
  • Pretraining tasks: Cross-modal contrastive learning maximizes similarity for related pairs and minimizes it for unrelated pairs across image-text and audio-text data.Global special-token representations are projected and normalized before the contrastive loss is applied.
  • Pretraining tasks: Intra-modal denoising contrastive learning masks units, reconstructs their features with a lightweight decoder, and contrasts masked features against target features.The method applies to image patches, audio time-steps, text tokens, and paired multimodal data.
  • Pretraining tasks: The denoising objective pulls masked-unit representations toward positive units and away from negative units, giving units more distinctive semantic meanings.The figure illustrates this with green positive connections and red negative connections.

4 Pretraining Details

ONE-PEACE pretraining uses publicly available image-text and audio-text datasets, with a 4B-parameter model and memory- and speed-oriented training techniques.

  • Pretraining Datasets: ONE-PEACE uses LAION-2B for image-text pretraining and publicly available open-source environmental sound datasets for audio-text pretraining.The datasets are divided into image-text and audio-text pairs, with reproducibility supported by public availability.
  • Pretraining Settings: The 4B-parameter model uses a two-layer Transformer decoder to recover masked units from visible units during pretraining.The decoder has 768 hidden size, 2048 intermediate size, and 12 attention heads.
  • Pretraining Settings: ONE-PEACE is randomly initialized except for the A-adapter audio feature extractor, which uses WavLM feature-extractor weights.The authors report that incorporating WavLM’s feature extractor significantly improves performance.
  • Image Classification: ONE-PEACE achieves state-of-the-art ImageNet classification using publicly available data and less token length.The comparison includes systems pretrained on privately collected or supervised datasets.
  • Semantic Segmentation: Semantic-segmentation comparisons on ADE20K distinguish single-scale mIoUss from multi-scale mIoUms results.The table caption defines the two evaluation settings but does not provide their values here.
  • Training Acceleration: Memory-efficient attention, gradient checkpointing, and Fused LayerNorm are introduced to accelerate training and reduce memory use.Gradient checkpointing enables training with a larger batch size.

5 Experiments

Across vision, audio, and vision-language benchmarks, ONE-PEACE demonstrates strong transfer, retrieval, grounding, and multimodal understanding performance, including leading results without external vision or language pretraining. Ablations further support shared self-attention with modality-specific FFNs and intra-modal denoising contrastive learning.

  • Vision Tasks: 89.8 top-1 accuracy on ImageNet demonstrates strong image-classification transfer without external pretrained models.ONE-PEACE is trained from scratch without external models and uses less token length than the cited setup.
  • Vision Tasks: 63.0 mIoU on ADE20K establishes a new state-of-the-art for semantic segmentation.The result is reported after adapting ONE-PEACE with ViT-Adapter and Mask2Former.
  • Vision Tasks: 88.1 top-1 accuracy on Kinetics-400 is achieved with the full encoder frozen, outperforming cited larger or privately pretrained models.Several MLP adapters and an I3D head are added while the whole model remains frozen.
  • Audio(-Language) Tasks: ONE-PEACE achieves large audio-text retrieval gains, improving R@1 by 21.1% and 11.4% on AudioCaps and 23.1% and 5.4% on Clotho.The improvements correspond respectively to text-to-audio and audio-to-text retrieval.
  • Vision-Language Tasks: ONE-PEACE achieves leading results across image-text retrieval, visual grounding, and vision-language understanding tasks.It is best on fine-tuned MSCOCO and Flickr30K retrieval, reaches new SOTA on RefCOCO-family grounding without extra grounding data, and reports 82.6 VQA test-dev and 82.5 test-std scores.
  • Ablation Study: Denoising contrastive learning complements cross-modal contrastive learning, while shared self-attention with separated FFNs gives the fastest convergence among tested architectures.Applying DCL to vision-language data gives the best evaluation metrics, and denoising contrastive loss improves COCO retrieval and image classification over alternative losses.

6 Conclusion, Limitation and Future Work

ONE-PEACE is presented as a scalable general representation model that aligns multiple modalities and supports emergent zero-shot retrieval, while future work targets broader modality coverage and LLM integration. Its main limitation is weaker performance on zero-shot image-text retrieval and vision-language understanding.

  • Conclusion: ONE-PEACE aligns and integrates vision, audio, and language representations across 3 modalities, 11 tasks, and 16 datasets, with leading results across a wide range of tasks.The reported tasks include image classification, semantic segmentation, audio-text retrieval and classification, audio question answering, image-text retrieval, and visual grounding.
  • Limitation: ONE-PEACE falls short of state-of-the-art results in zero-shot image-text retrieval and vision-language understanding tasks.The paper identifies limited image-text pretraining and the absence of language-pretrained initialization or pure text data as possible reasons.
  • Conclusion: ONE-PEACE shows emergent zero-shot retrieval between modalities and modality combinations that lack paired data during pretraining.The examples use images retrieved from ImageNet-1K and MSCOCO.
  • Future Work: Future work will evaluate vision-audio-language tasks, extend pretraining to video and 3D point clouds, and explore interaction with large language models.The stated LLM directions include building a more powerful general representation model and a more general multimodal language model.

A.1 Pretraining Datasets

The pretraining pipeline uses cleaned image-text and audio-text data, with vision-language pretraining followed by audio-language pretraining. The two stages use distinct training schedules, and the environmental sound datasets are publicly available.

  • Pretraining Data: Image-text pretraining data come from LAION-2B and are filtered by image dimensions, aspect ratio, CLIP score, language, emoji content, and text length.The source is web-crawled and may contain noisy pairs before preprocessing.
  • Pretraining Data: Audio-text pretraining uses environmental sound datasets processed from prior work, yielding about 2.4 million audio-text pairs after text cleaning.Some tag-only datasets are rewritten into captions with a pretrained T5 language model.
  • Dataset Statistics: The environmental sound datasets used for pretraining are publicly available.This statement is given in the dataset statistics table caption.
  • Training Stages: ONE-PEACE pretraining is divided into vision-language and audio-language stages.The stages are described as sequential components of the pretraining procedure.
  • Training Settings: Vision-language pretraining runs for 200K steps with batch size 32768, while audio-language pretraining runs for 10 epochs with batch size 3072.The vision-language stage uses 256 × 256 images and maximum text length 70; the audio-language stage uses maximum audio duration 15s.

B.1 Vision Tasks

The vision-task experiments cover image classification, semantic segmentation, object detection, and video action recognition, using task-specific heads, adapters, and fine-tuning procedures. Detailed settings are organized in the vision-task configuration table.

  • Task Coverage: The vision experiments include image classification, semantic segmentation, object detection, and video action recognition.These task implementations are summarized as the main vision-task evaluation settings.
  • Image Classification: Image classification uses pooled image tokens followed by LayerNorm and a linear classifier, with optional intermediate fine-tuning on ImageNet-21k.ImageNet-1k fine-tuning uses exponential moving average, while ImageNet-21k intermediate fine-tuning does not.
  • Configuration: Fine-tuning settings for the vision tasks are listed in Table 15.The table is the referenced configuration summary for these experiments.
  • Semantic Segmentation: Semantic segmentation uses Mask2Former and intermediate fine-tuning on COCO-Stuff before fine-tuning on ADE20K.Both segmentation experiments use cosine learning-rate decay.
  • Object Detection: Object detection on COCO uses ViTDet, large-scale jitter, and 50 training epochs.The learning rate is decayed at epochs 44 and 48.
  • Video Action Recognition: Video action recognition on Kinetics-400 freezes the pretrained model and adds spatial and temporal MLP adapters in each Transformer layer.The backbone learning-rate multiplier is 0.1.

B.2 Audio-(language) Tasks

The audio-task experiments cover audio-text retrieval, audio classification, and audio question answering. They use branch-specific feature extraction, similarity or classification objectives, and task-specific datasets and metrics.

  • Audio-Text Retrieval: Audio-text retrieval is evaluated on AudioCaps and Clotho using merged training data from AudioCaps, Clotho, and MACS.Audio and text features are extracted with A-Branch and L-Branch, and recall@k is the evaluation metric.
  • Audio Classification: Audio classification is evaluated on ESC-50, FSD50K, and VGGSound with dataset-specific zero-shot or supervised procedures.ESC-50 uses audio-label embedding similarity for zero-shot classification; FSD50K reports mean average precision and VGGSound reports test accuracy.
  • Configuration: Fine-tuning settings for audio and audio-language tasks are listed in Table 16.The table is the referenced configuration summary for these experiments.
  • Audio Question Answering: Audio question answering is evaluated on AVQA by combining audio and questions with candidate answers and optimizing a pairwise hinge loss.Visual information is excluded from the videos during this task.

B.3 Vision-language tasks

ONE-PEACE is evaluated across image-text retrieval, visual grounding, visual question answering, and visual reasoning using task-specific multimodal branches and standard benchmark metrics.

  • Overview: The section covers implementation details for four vision-language tasks, with detailed hyperparameters provided in Table 17.The tasks are image-text retrieval, visual grounding, visual question answering, and visual reasoning.
  • Image-Text Retrieval: Image-text retrieval is evaluated on MSCOCO and Flickr30K using cosine similarity between image and text features, with recall@k as the metric.The V-Branch and L-Branch separately extract image and text features.
  • Visual Grounding: Visual grounding predicts normalized bounding-box coordinates from jointly processed image and text features on RefCOCO, RefCOCO+, and RefCOCOg.Multi-head attention pooling aggregates image-patch features, and Acc@0.5 is reported.
  • Visual Question Answering: Visual question answering uses jointly processed images and questions to classify answers among the 3,129 most frequent answers on VQAv2.Training uses VQAv2 training and validation data plus additional Visual Genome question-answer pairs.
  • Visual Reasoning: Visual reasoning on NLVR2 treats each example as two image-text pairs and classifies the concatenated pooled representations.Accuracy is reported on the dev and test-P sets.

C Effects of Pretrained Audio Feature Extractor

The pretrained audio feature extractor strongly affects ONE-PEACE performance despite representing only about 1% of total parameters, and pretrained initialization outperforms random initialization on ESC-50.

  • Ablation Study: 85.3 accuracy is achieved on ESC-50 with a randomly initialized audio feature extractor.The feature extractor contains 4.6M parameters, approximately 1% of the total model.
  • Ablation Study: Pretrained Hubert, Wav2Vec 2.0, and WavLM feature extractors yield better ESC-50 performance than random initialization.The comparison is reported as a zero-shot accuracy ablation in Table 18.
  • Additional Analysis: 56.6% recognition accuracy is obtained when locating named One Piece characters in a complex anime image.The model was fine-tuned on RefCOCOg and had not seen anime pictures in that dataset.

E More Examples of Emergent Zero-shot Retrieval

The paper presents emergent zero-shot retrieval examples spanning audio-to-image, audio+image-to-image, and audio+text-to-image settings using audio from ESC-50 and images from ImageNet-1K and MSCOCO.

  • Retrieval Examples: The examples cover audio-to-image, audio+image-to-image, and audio+text-to-image retrieval.They are presented as demonstrations of ONE-PEACE's emergent zero-shot abilities.
  • Retrieval Examples: The audio examples are selected from ESC-50, while retrieved images come from ImageNet-1K and MSCOCO.The examples are intended to help readers perceive the model's retrieval behavior.
  • Visual Grounding Example: Figure 6 shows ONE-PEACE locating 5 of 9 named Straw Hat Pirates in an image.The visualization concerns visual grounding of different One Piece characters.
  • Retrieval Examples: Figures 7–9 illustrate audio-to-image, audio+image-to-image, and audio+text-to-image retrieval, respectively.Each figure corresponds to one of the multimodal retrieval settings described in the section.
Loading 2305.11172v1…