PaperlayerView exact revision on arXiv

Source-linked AI summary

Prismer: A Vision-Language Model with Multi-Task Experts

Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, Anima Anandkumar

arXiv:2303.02506v3cs.LGcs.AIcs.CV

TL;DR

Vision-language models commonly demand large models and massive training datasets, while tasks can require specialized visual reasoning skills. Prismer connects frozen, pre-trained experts through lightweight trainable components and reports competitive benchmark performance with substantially less training data. Its design focuses on fine-tuned multimodal generation and does not support inference with only a partial set of experts without a notable performance drop.

  • Problem

    Vision-language models often use massive image-text datasets to train monolithic models to learn specialized visual skills, while large pre-trained models also demand substantial compute and data.

    Source: §1 · p. 1

  • Method

    Prismer connects frozen, pre-trained vision, language, and task experts through lightweight trainable components for open-ended vision-language reasoning.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Prismer achieves competitive image captioning, VQA, and image classification performance compared with models trained on up to two orders of magnitude more data.

    Source: §6 · p. 17

  • Takeaways & Limitations

    The results support using a small number of trainable components to connect diverse pre-trained experts for vision-language reasoning tasks.

    Source: §1 · p. 2, §6 · p. 17

    Because Prismer entangles features from its included experts, using only a partial set of experts at inference causes a notable performance drop.

    Source: §6 · p. 18

Abstract

from arXiv · show

Recent vision-language models have shown impressive multi-modal generation capabilities. However, typically they require training huge models on massive datasets. As a more scalable alternative, we introduce Prismer, a data- and parameter-efficient vision-language model that leverages an ensemble of task-specific experts. Prismer only requires training of a small number of components, with the majority of network weights inherited from multiple readily-available, pre-trained experts, and kept frozen during training. By leveraging experts from a wide range of domains, we show Prismer can efficiently pool this expert knowledge and adapt it to various vision-language reasoning tasks. In our experiments, we show that Prismer achieves fine-tuned and few-shot learning performance which is competitive with current state-of-the-arts, whilst requiring up to two orders of magnitude less training data. Code is available at https://github.com/NVlabs/prismer.

1 Introduction

Prismer uses frozen pre-trained experts and lightweight trainable components to address the data and compute demands of vision-language learning. It reports competitive reasoning performance with substantially less training data than many state-of-the-art models.

1 Introduction

  • Vision-language tasks can require fine-grained recognition, detection, counting, and 3D perception, while typical approaches train one large model on massive image-text datasets to learn these skills from scratch.The introduction frames these demands as adding to the challenges of vision-language learning.Source: §1 · p. 1
  • Prismer instead uses distinct experts that can be optimized for specific tasks, letting the model integrate specialized skills and domain knowledge rather than learn everything at once.The paper presents this design as a way to improve training efficiency and scale down multimodal learning.Source: §1 · p. 2
  • Prismer combines frozen, individually pre-trained vision and language backbones and task experts with lightweight trainable components comprising roughly 20% of total network parameters.The task experts provide low-level signals such as depth and high-level signals such as instance and semantic labels.Source: §1 · p. 2
  • Trained on 13M publicly available image/alt-text examples, Prismer is competitive with many state-of-the-art vision-language models trained on one or two orders of magnitude more data [3] [88] [90].Reported tasks include image captioning, image classification, and visual question answering.Source: §1 · pp. 2–3

2 Related Work

Related work spans shared or separate vision-language architectures, multi-task learning, and systems that connect pre-trained experts. Prismer distinguishes itself by using expert predictions as auxiliary inputs in a unified model for multimodal reasoning.

Vision-Language Models

  • Earlier single-stream models used pre-trained object detectors to encode images as visual-word sequences, whereas dual-stream models aligned separate vision and language features, often for classification and retrieval.Prismer uses expert predictions as auxiliary signals while retaining the original images for visual encoding, and emphasizes generation alongside contrastive alignment.Source: §2 · p. 3
  • Many vision-language models combine multiple pre-training objectives, while Prismer uses a single autoregressive objective and targets fine-tuned performance with smaller, diverse pre-trained models.The cited related work describes other systems as data-intensive and computationally demanding, often requiring billions of training examples.Source: §2 · p. 3

Multi-task and Auxiliary Learning

  • Multi-task methods predict several outputs or model task relationships, while Prismer uses vision-task predictions solely as auxiliary input for multimodal reasoning with a single-task objective.Unlike the cited pseudo-label approaches, Prismer does not focus on task-agnostic features learned through multi-task supervision.Source: §2 · pp. 3–4

Unifying Pre-trained Experts

  • Socratic models, ViperGPT, and Visual Programming connect experts through language or modular programs, whereas Prismer aims to bridge experts through a unified architecture.The paper characterizes the cited systems as zero-shot methods within experts’ pre-trained domains, with errors from earlier experts potentially carried forward.Source: §2 · p. 4
  • Prismer’s experts are independently pre-trained models encoding explicit task or domain knowledge, unlike Mixture of Experts submodules trained within one network under a shared objective [72].Source: §2 · p. 4

3 Prismer: Open-ended Reasoning with Multi-Task Knowledge

Prismer turns multi-task expert predictions and an RGB image into free-form text with an encoder-decoder model. Frozen pre-trained experts are connected through resampling and adaptor components, and the model is trained with an autoregressive text-generation objective.

3.1 Model Overview

  • Prismer is an encoder-decoder transformer: its vision encoder processes an RGB image and predictions such as depth or segmentation, and its language decoder uses cross-attention to generate text tokens.Vision and language backbones provide the core network, with multi-task features supplying auxiliary information.Source: §3.1 · pp. 4–5
  • Prismer formulates vision-language tasks as language modelling or prefix language modelling, generating answers or captions autoregressively or ranking fixed completions by log-likelihood.The prefix can include an image, its multi-task tokens, and a question for visual question answering.Source: §3.1 · p. 6

3.2 Pre-trained Experts

  • Prismer uses frozen backbone experts and up to six frozen vision task experts that predict low- and high-level signals, including depth, surface normals, edges, object labels, segmentation, and OCR.The task experts act as black-box predictors, and their labels become model inputs; high-level labels are embedded with CLIP text features and reduced to 64 channels with PCA.Source: §3.2 · p. 6

3.3 Key Architectural Components

  • The Experts Resampler uses cross-attention to compress variable-length task features into a fixed number of tokens, keeping self- and cross-attention memory constant as the number of experts varies.Task-specific convolutional stems first map expert labels to a shared dimensionality; adaptors down-project and up-project features with residual connections.Source: §3.3 · pp. 7–8, Figure 3 · p. 8

3.4 Training Objective

  • Training uses one autoregressive next-token objective, but the generation focus makes Prismer less suitable for discriminative tasks such as image-text retrieval and visual entailment.Expert-label collection is a one-time preprocessing step; the single objective requires one forward pass to compute gradients.Source: §3.4 · pp. 8–9

4 Experiments

Prismer combines pretrained vision and language backbones with task-specific experts and evaluates model variants on vision-language benchmarks. The results show competitive performance with substantially less training data, while larger-data models retain advantages on some comparisons.

4.1 Prismer Model Variants

  • PrismerZ removes task experts and the Experts Resampler, requiring only RGB images at inference; BASE and LARGE share a roughly 50M-parameter resampler over different ViT and RoBERTa backbones.The full Prismer model requires processing expert labels at inference, while PrismerZ is more efficient and has lower predictive performance.Source: §4.1 · p. 9

4.2 Training and Evaluation Details

  • Experiments use 11M unique images or 12.7M image/alt-text pairs and evaluate captioning openly, while VQA and classification rank likelihoods over predefined answers.The evaluation treats vision-language tasks as language modeling; fine-tuned captioning uses the COCO Caption Karpathy split, and VQAv2 training includes Visual Genome samples.Source: §4.2 · p. 10, §4.3 · p. 10

4.3 Results on Vision-Language Benchmarks

  • Across COCO Caption, NoCaps, and VQAv2, Prismer variants outperform similarly sized VLMs and approach larger-data systems, though a gap remains to current state-of-the-art models.PrismerBASE and PrismerLARGE match BLIP and LEMON on image captioning with 10 and 20 times less data, respectively; PrismerLARGE has VQAv2 accuracy comparable to GIT with 60 times less data.Source: Table 3 · p. 11, §4.3 · p. 11
  • Zero-shot Prismer reaches state-of-the-art captioning on COCO Caption and NoCaps, outperforming SimVLM by a large margin with 140 times less training data.Its zero-shot results also exceed the fine-tuned performance of OSCAR and VinVL.Source: §4.3 · pp. 11–12, Figure 4 · p. 12
  • Few-shot ImageNet fine-tuning substantially improves on Prismer’s ViT-B and ViT-L backbones, but trails GIT and Flamingo, which use larger backbones and more pretraining data.The comparison uses lightweight fine-tuning for Prismer and its full language decoder as an image classifier.Source: §4.3 · p. 12, Figure 4 · p. 12

5 Additional Analysis

The analysis tests how expert quantity, quality, and noise affect Prismer, examines expert contributions to domain-specific reasoning, and evaluates architecture and training choices. It reports competitive VSR results and identifies design choices associated with stronger performance and stable training.

5.1 Intriguing Learning Strategy of Prismer

  • The analysis uses BASE on 3M Conceptual Captions and SBU examples at 224 × 224 for VQAv2, while expert utility is evaluated zero-shot on VSR and Text-VQA.The utility experiment uses PrismerLARGE trained on VQAv2; VSR tests spatial relations and Text-VQA tests reading and reasoning about image text.Source: §5.1 · p. 14, §5.2 · pp. 14–15
  • Performance improves as expert quantity and depth-expert quality increase, while gains from adding experts eventually plateau.The quantity and quality trends are shown in Figure 5a and Figure 5b.Source: §5.1 · p. 14, Figure 5 · p. 15
  • Prismer maintains performance with noisy experts, and adding noise can improve over RGB-only training, which the authors describe as implicit regularisation.This robustness allows experts to be included without degrading performance even when they are not informative.Source: §5.1 · p. 14, Figure 5 · p. 15

5.2 Utility of Task Experts

  • Prismer outperforms VisualBERT, LXMERT, and ViLT on VSR without dataset-specific fine-tuning, and surpasses BLIP-2 [OPT 2.7B] and OFAHUGE [89].Table 5 reports zero-shot accuracies on VSR and Text-VQA.Source: §5.2 · p. 15, Table 5 · p. 16
  • Object detection is a key contributor on both VSR and Text-VQA, while depth and OCR detection contribute to spatial and text reasoning, respectively.The authors connect depth to 3D spatial understanding and OCR detection to reading text.Source: §5.2 · p. 15

5.3 Architecture Design and Training Details

  • The standard residual, encoder-decoder adaptor performs best among tested designs, and increasing its bottleneck size improves performance.Adding an adaptor at each transformer layer or a learnable gating mechanism performed worse.Source: §5.3 · pp. 15–16
  • Lightweight resampler layers and latent variables are important for stable training; learnable resampling performs slightly better than random sampling.Using RGB as resampler input or adding a resampler after the vision encoder reduces performance.Source: §5.3 · p. 16
  • Freezing pretrained parameters helps avoid overfitting and catastrophic forgetting while saving GPU memory, and freezing the vision encoder benefits downstream fine-tuning.The resampler and adaptors remain trainable in the downstream setting described.Source: §5.3 · p. 16

6 Conclusions, Limitations and Discussion

Prismer connects diverse pretrained experts using a small set of trainable components and reports competitive vision-language benchmark performance. The discussion identifies boundaries in in-context prompting, expert adaptation and availability at inference, and expert-label representation.

6 Conclusions, Limitations and Discussion

  • Prismer uses a small number of trainable components to connect diverse pretrained experts and achieves competitive captioning, VQA, and image-classification performance against models trained with up to two orders of magnitude more data.The conclusion characterizes Prismer as parameter-efficient.Source: §6 · p. 17
  • Prismer does not support few-shot in-context prompting by design because it uses a small-scale language model.The authors state that zero-shot in-context generalisation is an emergent property only in very large language models [8] [91].Source: §6 · p. 17
  • Using a different segmentation expert with a different set of semantic information leads to limited adaptability and a notable performance drop.The experiment applies the same language model to encode semantic labels.Source: §6 · pp. 17–18
  • Because Prismer entangles features from its pretrained experts, using only a partial set of experts at inference leads to a notable performance drop.A masked auto-encoding objective for arbitrary expert counts degraded fine-tuned performance.Source: §6 · p. 18
  • Prismer represents expert labels as image-like 3D tensors after task-specific post-processing, while other representations remain a possible future direction.The authors suggest text-token sequences for object-detection labels as one alternative.Source: §6 · p. 18

A Detailed Training Strategy and Hyper-parameters

The training strategy specifies image resolutions and evaluation tasks, with detailed hyperparameters provided separately.

A Detailed Training Strategy and Hyper-parameters

  • Models are pretrained at 224 × 224, then fine-tuned at 480 × 480 for captioning and VQA or 384 × 384 for image classification.Automated data augmentation is used for both pretraining and fine-tuning.Source: A Detailed Training Strategy and… · p. 28
  • Table 7 lists hyperparameters and training strategy for reproducibility, using the same settings for BASE and LARGE.Source: Table 7 · p. 28

B Comparison of Training Cost

The comparison reports Prismer’s training cost alongside other vision-language models, with costs defined to exclude original expert pretraining.

B Comparison of Training Cost

  • PrismerLARGE takes 8 days to train on 32 NVIDIA V100 GPUs, compared with SimVLM’s 5 days on 2048 TPUv3, GIT-2’s 1.5 months on 1500 NVIDIA A100s, and Flamingo’s 2 weeks on 1536 TPUv4.The authors characterize Prismer as more efficient than these models.Source: B Comparison of Training Cost · p. 29, Table 8 · p. 29
  • Prismer’s training-cost definition excludes the original pretraining data used to train its expert models.Table 8 compares training and inference costs; inference estimates use 256 image tokens and 30 text tokens.Source: B Comparison of Training Cost · p. 29, Table 8 · p. 29
Loading 2303.02506v3…