PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models are General-Purpose Interfaces

Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, Furu Wei

arXiv:2206.06336v1cs.CL

TL;DR

The paper frames connecting diverse foundation models to a language-model task layer as a design challenge. METALM combines pretrained encoders and a causal language model through semi-causal pretraining, and shows strong finetuning and in-context learning across language-only and vision-language tasks.

  • Problem

    The paper frames connecting diverse foundation models with a language-based task layer as a challenge spanning tasks and modalities.

    Source: §1 · p. 4

  • Method

    METALM docks pretrained non-causal encoders with a causal language-model task layer and jointly pretrains them using a semi-causal objective.

    Source: §1 · p. 4, §2.2 · p. 6

  • Results

    METALM exhibits strong finetuning and in-context learning performance across a wide range of language-only and vision-language tasks.

    Source: §6 · p. 21

  • Takeaways & Limitations

    The framework supports in-context customization of finetuned models and multimodal multi-turn interaction.

    Source: §2.4 · p. 7

Abstract

from arXiv · show

Foundation models have received much attention due to their effectiveness across a broad range of downstream applications. Though there is a big convergence in terms of architecture, most pretrained models are typically still developed for specific tasks or modalities. In this work, we propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer. We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders. We subsume the advantages and capabilities from both causal and non-causal modeling, thereby combining the best of two worlds. Specifically, the proposed method not only inherits the capabilities of in-context learning and open-ended generation from causal language modeling, but also is conducive to finetuning because of the bidirectional encoders. More importantly, our approach seamlessly unlocks the combinations of the above capabilities, e.g., enabling in-context learning or instruction following with finetuned encoders. Experimental results across various language-only and vision-language benchmarks show that our model outperforms or is competitive with specialized models on finetuning, zero-shot generalization, and few-shot learning.

1 Introduction: Design Principles

The paper frames language models as a general-purpose interface for diverse tasks and foundation models. Its semi-causal approach connects bidirectional encoders with a causal language model to combine their capabilities.

1 Introduction: Design Principles

  • A language-model task layer can express predictions as free text and support tasks ranging from classification to question answering and multi-turn dialogue.The interface uses natural-language instructions and explanations, presents understandable outputs, and supports conversational interaction.Source: §1 · pp. 4–5
  • Causal modeling supports zero- and few-shot learning, while bidirectional encoders are presented as advantageous for finetuning and cross-task, cross-language, and cross-modal transfer.The paper motivates combining these approaches to draw on their complementary capabilities.Source: §1 · p. 4
  • Semi-causal pretraining links bidirectional encoders with a causal language model, combining their respective strengths in one interface.The encoders represent diverse inputs while the causal model serves as a universal task layer.Source: §1 · p. 4
  • The framework casts non-causal encoders as a perception layer and the causal language model as a sequential task layer.The paper relates these modules, respectively, to System 1 and System 2.Source: §1 · pp. 4–5

2 METALM: Meta Language Model

METALM connects pretrained encoders for diverse modalities to a causal language model that acts as a general-purpose task layer. Semi-causal pretraining and downstream-task procedures support in-context learning, finetuning, customization, and multimodal conversation.

2 METALM: Meta Language Model

  • METALM docks pretrained encoders for diverse modalities with a language model that unifies tasks as free-text generation.The language model serves as the general-purpose interface and universal task layer.Source: §2 · p. 5

2.1 Input Representation

  • The framework uses contextualized encoder representations and text-token embeddings, summed with positional embeddings, as its input representations.A connector projects encoder outputs before they enter the general-purpose interface.Source: §2.1 · p. 5, §2.2 · p. 6

2.2 Model Architecture

  • The architecture pairs a left-to-right decoder with multiple bidirectional encoders, whose outputs are projected by connectors to match the task layer's dimensions.Input representations combine encoder outputs and text-token embeddings with positional embeddings before entering the interface.Source: §2.1 · p. 5, §2.2 · p. 6

2.3 Proposed Objective: Semi-Causal Language Modeling

  • Semi-causal pretraining autoregressively generates sequence tokens while bidirectionally encoding selected spans, jointly pretraining and connecting the interface and foundation models.The number and positions of non-overlapping encoded spans are typically sampled randomly.Source: §2.3 · pp. 6–7

2.4 Capabilities on Downstream Tasks

  • METALM supports zero- and few-shot prompting, finetuning to generate free-text targets, post-finetuning in-context customization, and multimodal multi-turn interaction.In-context adaptation uses instructions or demonstrations without updating parameters; finetuning conditions generation on bidirectionally encoded inputs.Source: §2.4 · p. 7

3 Experiments on Language-Only Tasks

Language-only experiments evaluate METALM through multitask and single-task finetuning, instruction tuning, and in-context learning using a shared free-text interface. Across these settings, the model shows strong finetuning results while retaining zero- and few-shot capabilities.

3.1 Evaluation Settings

  • Evaluation spans multitask and single-task finetuning, instruction tuning, and in-context learning; diverse downstream tasks share a free-text format.Source: §3.1 · pp. 7–8, Table 1 · p. 8

3.2 Pretraining Setup

  • The semi-causal model is pretrained from scratch on Pile for 300k steps, while the encoder starts pretrained and freezes except for its last two layers.Source: §3.2 · p. 9

3.3.2 Results

  • METALM beats GPT on almost all multitask clusters, with 14.1% and 9.6% gains on natural language inference and reading comprehension; struct-to-text is the exception.Finetuning uses a mixture of datasets capped at 30k examples per dataset.Source: §3.3.1 · p. 9, §3.3.2 · p. 10, Figure 5 · p. 10

3.4.2 Results

  • On MNLI, METALM improves accuracy over GPT by 3.4 absolute points despite updating far fewer parameters, and matches or exceeds three large bidirectional encoder baselines.Source: §3.4.2 · p. 11, Table 3 · p. 11

3.5.2 Results

3.6.2 Results

  • Across zero-, one-, and four-shot settings, METALM's average accuracy is better than GPT's, with consistent gains on Winograd and completion tasks.Source: §3.6.2 · p. 13, Table 5 · p. 13

4 Experiments on Vision-Language Tasks

Vision-language experiments evaluate METALM through zero-shot generation, in-context learning, and finetuning across understanding and generation tasks. Results show strong performance across these settings, including competitive or better finetuning results and improvements over comparison methods on several tasks.

4.2 Pretraining Setup

  • Vision-language pretraining uses bidirectional fused representations of image tokens and a random-length text prefix to autoregressively predict the remaining text, jointly with text-only data.The non-causal encoder is a pretrained vision-language foundation model docked with the language-model interface.Source: §4 · p. 14

4.1 Evaluation Settings

  • The nine-dataset evaluation spans visual question answering, visual reasoning, image captioning, and explanation generation, covering understanding and generation.Tasks are evaluated through zero-shot generalization, in-context learning, and finetuning; inputs and prompts go to the encoder while the language model generates open-ended outputs.Source: §4.1 · p. 14

4.2 Pretraining Setup

  • The vision-language model uses a 12-layer non-causal encoder and 24-layer language model, with a three-layer feed-forward connector.The language model has 353M parameters and the encoder 192M; pretraining uses about 4M images and 10M image-text pairs alongside OpenWebText.Source: §4.2 · p. 14

4.3.2 Results

  • METALM outperforms recent strong methods on zero-shot image captioning across COCO, NoCaps, and Flickr30k, and beats Frozen and VLKD on VQAv2 and OK-VQA.The captioning comparisons use one shared prompt, “Summarize this image:”; the results across five datasets demonstrate zero-shot generalization and open-ended generation.Source: Table 7 · p. 16, §4.3.2 · p. 16, Table 8 · p. 16

4.4.2 Results

  • Adding in-context demonstrations improves VQAv2 and OK-VQA performance over zero-shot results, and more examples bring larger improvements on both datasets.METALM outperforms Frozen despite its relatively small model size and adapts without modifying the underlying vision-language model.Source: Table 10 · p. 17, §4.4.2 · p. 17

4.5.2 Results: Visual Question Answering and Visual Reasoning

  • METALM achieves competitive or better finetuning results on VQAv2, VQA Karpathy, and NLVR2, with the best out-domain VQA Karpathy results.On OK-VQA it improves significantly without additional knowledge bases; competitive performance across datasets accompanies open-ended prediction.Source: Table 11 · p. 18, §4.5.2 · pp. 18–19, Table 12 · p. 19

4.5.3 Results: Visually Grounded Language Generation

  • METALM improves on other models for COCO caption finetuning and outperforms previous strong models on most E-SNLI-VE explanation-generation metrics.It jointly generates entailment labels and explanations, with good performance on both explanation generation and entailment-label understanding.Source: Table 14 · p. 20, §4.5.3 · p. 20, Table 15 · p. 20

5 Related Work

Prior work studies causal and bidirectional language-model objectives, as well as systems that unify tasks and modalities. METALM combines these modeling approaches in an interface for foundation models.

5.1 Language Model Pretraining

  • Causal language models support few-shot and in-context learning, while non-causal models perform better for finetuning; METALM introduces semi-causal modeling to combine these strengths.The paper presents this combination as a route to finetuning and in-context learning through a general-purpose interface to foundation models.Source: §5.1 · pp. 20–21

5.2 General-Purpose Modeling

  • Prior general-purpose systems span multitask learning, unified language understanding and generation, multimodal architectures, and transfer of frozen-language-model in-context learning to vision-language tasks.The cited examples include MT-DNN, UniLM, T5, Perceiver IO, and work by Tsimpoukelli et al.Source: §5.2 · p. 21

6 Conclusion

The conclusion presents METALM as a general-purpose interface built from pretrained non-causal encoders and a causal decoder, jointly pretrained with semi-causal language modeling. It reports strong finetuning and in-context learning performance and outlines extensions for future work.

6 Conclusion

  • METALM pairs a causal decoder universal task layer with multiple pretrained non-causal encoders and uses semi-causal language modeling to pretrain them.The paper reports strong finetuning and in-context learning across language-only and vision-language tasks.Source: §6 · p. 21
  • The authors identify scaling, multilingual use, handling more modalities, extending the task layer to vision tasks, and parameter-efficient finetuning as future directions.The proposed modality extensions include language, vision, audio, and multimodality.Source: §6 · p. 21

A.1 Pretraining

The appendix points to tables containing language-only pretraining hyperparameters and settings for multitask finetuning and instruction tuning.

A.1 Pretraining

A.2 Multitask Finetuning and Instruction Tuning

B Datasets Used for Language-Only Experiments

The language-only experiments use a curated Pile corpus for pretraining and datasets spanning diverse tasks for multitask finetuning and instruction tuning.

B.1 Pretraining

  • Pretraining uses 19 sources from the 800 GB English Pile after excluding GitHub, arXiv, and PubMed Central.The sources are grouped into academic, internet, prose, dialogue, and miscellaneous categories.Source: §B.1 · p. 29

B.2 Multitask Finetuning and Instruction Tuning

  • Finetuning and instruction tuning cover structured-text generation and summarization, using CommonGen, E2ENLG, AESLC, SamSum, and XSum.Each dataset has ten FLAN templates; multitask finetuning uses the first, while instruction tuning uses all ten.Source: §B.2 · p. 30
  • The dataset suite includes natural language inference, sentiment classification, paraphrase detection, and coreference resolution benchmarks.Examples include ANLI, IMDB, QQP, and Winogrande, respectively.Source: §B.2 · p. 30
  • Other finetuning tasks include commonsense reasoning, reading comprehension, miscellaneous language tasks, and closed-book question answering.The listed datasets include HellaSwag, DROP, CoLA, and ARC-easy, respectively.Source: §B.2 · p. 30

B.3 In-Context Learning

In-context learning is evaluated across four categories: cloze and completion, Winograd-style tasks, commonsense reasoning, and SuperGLUE tasks.

  • Cloze and completion evaluation uses StoryCloze and HellaSwag.Source: §B.3 · p. 30
  • Winograd-style evaluation uses Winograd and Winogrande.Source: §B.3 · p. 30
  • Commonsense reasoning evaluation uses ARC-easy, ARC-challenge, and PIQA.Source: §B.3 · p. 30
  • Two SuperGLUE datasets, BoolQ and Copa, provide another in-context learning category.Source: §B.3 · p. 30

C Detailed Results of Multitask Finetuning in Section 3.3

The appendix reports detailed language-only multitask finetuning results across ten task clusters, with separate tables for each cluster.

D Hyperparameters of Vision-Language Experiments

The vision-language appendix reports separate hyperparameter settings for pretraining and finetuning, with finetuning using 384×384 images and greedy decoding.

D.1 Hyperparameters of Vision-Language Pretraining

D.2 Hyperparameters in Vision-Language Finetuning

  • Vision-language finetuning uses 384×384 images and greedy decoding across tasks, with settings and prompts summarized in Table 31.Source: §D.2 · p. 32, Table 31 · p. 32
Loading 2206.06336v1…