PaperlayerView exact revision on arXiv

Source-linked AI summary

Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks

Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang, Jifeng Dai

arXiv:2112.01522v1cs.CV

TL;DR

Existing perception systems require task-specific designs for different modalities and tasks, increasing the effort needed to build generic models. Uni-Perceiver unifies inputs, targets, and task formulations in a shared architecture, achieving reasonable zero-shot performance on novel tasks and approaching or matching state-of-the-art results with limited or full downstream tuning.

  • Problem

    Task-specific inputs and prediction heads limit unified multi-task perception and require redesign, downstream data, and separate parameter maintenance for new tasks.

  • Method

    Uni-Perceiver uses lightweight modality-specific tokenizers and a modality-agnostic Transformer to map arbitrary-modality inputs and targets into one representation space, with tasks modeled by representational similarity.

  • Results

    The pre-trained model achieves reasonable zero-shot performance on novel tasks; prompt tuning with 1% of downstream data approaches state-of-the-art, while full-data fine-tuning is on par with or better than state-of-the-art on almost all tasks.

  • Takeaways & Limitations

    A shared perception model can transfer across modalities and tasks, including tasks absent from pre-training, with performance strengthened by prompt tuning or full-data fine-tuning.

  • Takeaways & Limitations

    The method currently applies only when the target set is discrete, and its extension to regression tasks remains questionable.

Abstract

from arXiv · show

Biological intelligence systems of animals perceive the world by integrating information in different modalities and processing simultaneously for various tasks. In contrast, current machine learning research follows a task-specific paradigm, leading to inefficient collaboration between tasks and high marginal costs of developing perception models for new tasks. In this paper, we present a generic perception architecture named Uni-Perceiver, which processes a variety of modalities and tasks with unified modeling and shared parameters. Specifically, Uni-Perceiver encodes different task inputs and targets from arbitrary modalities into a unified representation space with a modality-agnostic Transformer encoder and lightweight modality-specific tokenizers. Different perception tasks are modeled as the same formulation, that is, finding the maximum likelihood target for each input through the similarity of their representations. The model is pre-trained on several uni-modal and multi-modal tasks, and evaluated on a variety of downstream tasks, including novel tasks that did not appear in the pre-training stage. Results show that our pre-trained model without any tuning can achieve reasonable performance even on novel tasks. The performance can be improved to a level close to state-of-the-art methods by conducting prompt tuning on 1% of downstream task data. Full-data fine-tuning further delivers results on par with or better than state-of-the-art results. Code shall be released.

1. Introduction

Uni-Perceiver addresses the limits of task-specific perception models with a shared architecture that unifies modalities, tasks, and representations. Its pre-training supports zero-shot novel-task inference, while prompt tuning and full-data fine-tuning improve performance toward state-of-the-art levels.

  • Motivation: Task-specific inputs and prediction heads limit collaboration across tasks and require redesign, sufficient data, and separate parameter maintenance for new tasks.These costs become more inefficient as the number of tasks and model size grow.
  • Approach: Uni-Perceiver encodes inputs and targets from arbitrary modalities into a unified representation space and models their relationship through representational similarity.This removes task-specific formulations and aligns pre-training with downstream task modeling.
  • Approach: The architecture processes varied modalities and tasks with a single siamese model and shared parameters, using lightweight modality-specific tokenizers and a modality-agnostic Transformer encoder.Different perception tasks are formulated as finding the maximum-likelihood target through similarity between representations.
  • Pre-training: Uni-Perceiver is pre-trained on uni-modal and multi-modal tasks, including image and video classification, language modeling, image-text retrieval, and language modeling with image clues.The pre-training task mix spans both single-modality and cross-modality perception settings.
  • Results: Reasonable zero-shot performance is achieved on novel tasks without tuning, performance approaches state-of-the-art with prompt tuning on 1% of downstream data, and full-data fine-tuning reaches comparable or better results.The reported pattern covers downstream tasks that were absent from pre-training.

2. Related Works

Related work moved toward Transformer-based unified architectures for multiple modalities, but generally retained task-specific heads and training. Uni-Perceiver instead uses a unified representational-similarity formulation across tasks and modalities, with prompt tuning supporting few-shot adaptation.

  • Architecture: Transformers enabled unified architectures by converting inputs from different modalities into token sequences with modality-specific tokenizers.This paradigm was applied to image, video, and visual-linguistic recognition.
  • Architecture: Existing multi-modal architectures still require prediction heads specifically designed and trained for each perception task.This preserves task-specific design despite a shared Transformer-style backbone.
  • Uni-Perceiver: Uni-Perceiver replaces task-specific heads by encoding task inputs and targets in one representation space and modeling their relationship through unified representational similarity.The modification is intended to enable collaboration across modalities and tasks and zero-shot inference on unseen tasks.
  • Pre-training: The model is pre-trained simultaneously on image and video classification, language modeling with and without image clues, and image-text retrieval.Regional features and corresponding pre-training tasks are not used.
  • Prompt Tuning: Prompt tuning updates learnable prompt tokens while keeping other model parameters fixed, providing an alternative to full fine-tuning for few-shot adaptation.The paper focuses on zero-shot and few-shot settings, while sufficient downstream data permits whole-model fine-tuning.

3. Method

Uni-Perceiver unifies arbitrary-modality perception tasks through shared tokenization, a modality-agnostic Transformer, and representation-similarity modeling. Simultaneous pre-training supports zero-shot novel-task inference and adaptation with prompt tuning or fine-tuning.

  • Unified architecture: Uni-Perceiver converts arbitrary-modality inputs and targets into unified token sequences, then encodes them with a modality-agnostic Transformer into a shared representation space.Lightweight modality-specific tokenizers support text, image, and video inputs.
  • Unified task formulation: Each perception task uses the same formulation: select the maximum-likelihood target for an input from the similarity of their representations.The model estimates the joint probability through cosine similarity between encoded inputs and targets, using a learnable temperature parameter.
  • Modality coverage: The architecture supports inputs and targets composed of images, texts, and videos, including arbitrary modality combinations when corresponding tokenizers are available.Classification targets may be class names, descriptions, or images representing class indexes.
  • Transfer to downstream tasks: The unified formulation replaces task-specific input and prediction-head designs, enabling zero-shot inference on novel tasks and adaptation with prompt tuning or fine-tuning.The model learns joint distributions over input-target sequences during pre-training and reuses the formulation downstream.
  • Pre-training tasks: Pre-training jointly combines image and video classification, language modeling with and without image clues, and image-text retrieval tasks.Class names are represented as text sequences, providing weak supervision across image, video, and text representations.

4. Experiments

Uni-Perceiver is evaluated across pre-training and novel downstream tasks using multiple datasets and tuning settings. It achieves reasonable untuned performance, approaches state-of-the-art with 1% prompt tuning, and matches or exceeds state-of-the-art with full-data fine-tuning.

  • 4.1. Datasets: The model is pre-trained on image, video, language, image-text, and multimodal datasets, with novel evaluations on Flickr30k, MSVD, VQA, and GLUE.Pre-training uses ImageNet-21k, Kinetics-700, Moments in Time, Books&Wiki, PAQ, and several image-text datasets.
  • 4.3. Evaluation on Pre-training Tasks: The experiments evaluate image classification, video classification, image-text retrieval, and image captioning against task-specific state-of-the-art methods with similar model sizes.ImageNet-1k, Kinetics-400, COCO Caption, and Flickr30k are used for evaluation.
  • 4.3. Evaluation on Pre-training Tasks: Without tuning, the pre-trained model achieves reasonable performance on the pre-training tasks, although it is slightly worse than state-of-the-art methods.The model processes more tasks, datasets, and modalities than comparably sized task-specific methods.
  • 4.3. Evaluation on Pre-training Tasks: With 1% downstream data, prompt tuning brings performance close to state-of-the-art while tuning only a small parameter subset and keeping the encoder shared.This setting indicates low marginal cost across tasks.
  • 4.3. Evaluation on Pre-training Tasks: With 100% downstream data, fine-tuning achieves performance on par with or better than state-of-the-art methods across the evaluated tasks.Prompt tuning also consistently outperforms fine-tuning with the same data amount in few-shot comparisons.
  • 4.4. Generalization to Novel Tasks: On novel tasks including video captioning, video-text retrieval, VQA, and language understanding, zero-shot performance is reasonable, while 1% prompt tuning approaches state-of-the-art.VQA performance is unsatisfactory on the Yes/No and Number subsets, but full-data fine-tuning reaches parity with or exceeds state-of-the-art results.

5. Conclusion

Uni-Perceiver uses one shared architecture for multiple modalities and tasks, supporting zero-shot novel-task inference and strong adaptation with limited or full downstream data. Its main scope limitation is that targets must currently be discrete.

  • A unified perception architecture processes various modalities and tasks with a single model and shared parameters.
  • Zero-shot inference is possible on novel tasks, while 1% prompt tuning approaches state-of-the-art performance and full-data fine-tuning reaches comparable or better results.
  • The method currently applies only when the target set is discrete, leaving extension to regression tasks unresolved.
  • Large-scale training may consume substantial electricity, increase carbon emissions, and inherit biases from its datasets.

A. Tokenizer

Uni-Perceiver converts text, image, and video inputs into modality-aware token sequences for a shared Transformer encoder. Each modality uses a lightweight tokenizer with modality-specific positional and type embeddings.

  • Unified tokenization: Modality-specific tokenizers convert raw text, image, and video inputs into token sequences for the Transformer encoder.The resulting tokens receive modality type embeddings identifying their source modality.
  • Text tokenizer: Text is split into subwords with BPE, linearly projected, and augmented with 1D positional embeddings and a trainable textual modality embedding.Text inputs share one vocabulary, and the maximum text length is 256.
  • Image tokenizer: Images are resized to 224×224, divided into 16×16 patches, and represented with linear projections, 196 positional embeddings, and a visual modality embedding.The positional embedding length is 14 × 14 = 196.
  • Video tokenizer: Videos tokenize each frame into 16×16 patches and add spatial, temporal, and visual modality embeddings.With N = 8 frames by default, a video produces N × 14 × 14 tokens and uses temporal positions up to length 8.

B. Implementation Details for Auto-regressive Language Modeling

Autoregressive language modeling uses special <SPE> tokens to predict the current word rather than relying on ordinary next-token attention. During inference, an <SPE> is appended to the predicted sequence and its output generates the next prediction.

  • Prediction mechanism: <SPE> tokens are trained to predict the current word, replacing the conventional next-word prediction setup.During training, several <SPE> tokens are appended after the input sequence.
  • Inference: During inference, <SPE> is appended to the end of the predicted tokens, and its corresponding output is used for prediction.This defines how the autoregressive sequence is extended at inference time.
  • Attention mask: Figure 4 depicts the attention mask used for autoregressive language modeling.The figure provides the mask design associated with the autoregressive setup.

C. Prompt Tuning

Prompt tuning updates a small set of parameter groups, including special tokens, normalization parameters, prompts, and task-specific heads. The resulting trainable parameter count remains below 1% of the pre-training model parameters for the reported classification example.

  • Learnable components: Prompt tuning learns <SPE>, layer normalization parameters, prompt tokens, and linear heads.These are the four parameter groups identified for prompt-tuning.
  • Prompt configuration: Each Transformer layer receives 10 randomly initialized learnable input prompts, while tokenizer and layer-normalization weights and biases are tuned.The prompts do not come from the preceding layer's output, and <SPE> is shared between inputs and targets.
  • Task formulation: The novel-task format represents inputs as x and targets as y, with f(x) and f(y) used to calculate their joint probability distribution.The figure presents input and target formats for the novel tasks and omits tokenizer and encoder details.
  • Linear component: A linear layer combines weights w and bias b with a learnable scalar α, initialized respectively at 0, 0, and 1.The passage specifies the initialization of the linear parameters and scalar.
  • Parameter count: 227K parameters are used for non-classification tasks, while ImageNet-1k classification uses 995K with a linear head, still below 1% of pre-training parameters.The linear head applies only to classification tasks.

D. Formulation of Novel Tasks

Uni-Perceiver expresses novel perception tasks through a shared formulation that matches representations across inputs and targets. This formulation covers video captioning, video-text retrieval, visual question answering, and natural-language understanding.

  • Video Caption: Video captioning predicts each word autoregressively from the video and previously predicted words.The video and generated words are concatenated with a <SPE> token, whose output feature is matched against vocabulary words.
  • Video-Text Retrieval: Video-text retrieval represents videos and text sequences with <SPE> tokens and calculates their joint probability.The formulation replaces image sequences with video sequences while retaining the same retrieval structure.
  • Visual Question Answering: Visual question answering is formulated as masked language modeling over an image-question input and candidate answer sequences.Questions and candidate answers use <SPE> tokens to calculate their joint probability.
  • Natural Language Understanding: Natural language understanding treats sentences or sentence pairs as inputs and textual class labels as targets.Examples include sentiment labels in SST-2 and yes/no labels in MRPC.

E. Extra pre-training details

The extra pre-training details specify task weighting, retrieval negatives, augmentation, memory-saving parallelization, and dataset-overlap removal. Table 10 summarizes the pre-training ingredients and hyper-parameters.

  • Sampling Weight & Batch Size & Loss: Cross-entropy trains language-modeling tasks, while other tasks use cross-entropy with 0.1 label smoothing.Video classification receives loss weight 0.05; other loss weights default to 1.0.
  • Retrieval Training: Retrieval tasks use other samples in the same batch as negatives, typically with batch size 127.The method does not use a memory bank or gather features across GPUs for additional negatives.
  • Data Augmentation: Image and video augmentation is applied to reduce overfitting during pre-training.ImageNet-21k uses Rand-Aug, random erasing, mixup, and cutmix, while other datasets use resizing and random cropping.
  • Data Parallel for Vocabulary and Class Labels: Data parallelism is used for large vocabulary and ImageNet-21k class-label comparisons because naive implementations exceed memory limits.Language modeling compares a <SPE> feature with vocabulary-token features, and classification applies a similar method to class labels.
  • Removing Overlap: Videos overlapping the Kinetics-400 validation set are removed from the Kinetics-700 pre-training set.This overlap removal is applied specifically to the K700 training data participating in pre-training.

F. Licences of Datasets

The listed datasets carry varied usage terms, including ImageNet, Flickr, Creative Commons, open-source, and platform-specific licenses. The section records these licensing boundaries for the pre-training data sources.

  • Dataset Licences: ImageNet-21K is subject to the ImageNet terms of use.
  • Dataset Licences: Kinetics-700 and Kinetics-400 are licensed under a Creative Commons Attribution 4.0 International License.
  • Dataset Licences: Replicate Toronto BookCorpus is open source and licensed under GNU GPL, Version 3.
  • Dataset Licences: Most Wikipedia text is co-licensed under CC BY-SA 3.0 and the GNU Free Documentation License, with some text restricted to CC BY-SA-compatible reuse.
  • Dataset Licences: YFCC photos and videos use one of the Creative Commons copyright licenses.
  • Dataset Licences: CC12M, CC3M, and Visual Genome are governed by their respective platform or Creative Commons terms.CC12M follows Conceptual 12M terms, CC3M follows Conceptual Captions terms, and Visual Genome uses a Creative Commons Attribution 4.0 license.
  • Dataset Licences: COCO Caption and SBU Caption images are subject to Flickr terms of use.
  • Dataset Licences: PAQ is licensed under the Attribution-NonCommercial 4.0 International License.
Loading 2112.01522v1…