PaperlayerView exact revision on arXiv

Source-linked AI summary

CM3: A Causal Masked Multimodal Model of the Internet

Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer

arXiv:2201.07520v1cs.CL

TL;DR

Existing generative models do not jointly capture the full structure of multimodal documents, while causal and masked objectives trade off generation and bidirectional context. CM3 introduces a causally masked multimodal model trained on structured web and Wikipedia documents, and demonstrates broad zero-shot capabilities with strong results on summarization and entity tasks. Its scope is constrained by the limited image diversity and ethical filtering of its training data.

  • Problem

    Prior work did not jointly model formatted hypertext, hyperlinks, and images in their naturally co-occurring document contexts.

  • Method

    CM3 trains causally masked decoder-only models on structured multimodal documents, generating left to right while inferring masked spans with bidirectional context.

  • Results

    CM3 performs a wide range of zero-shot unimodal and cross-modal tasks and improves state-of-the-art zero-shot summarization, entity linking, and entity disambiguation while remaining competitive after fine-tuning.

  • Takeaways & Limitations

    A single CM3 model can recover capabilities associated with image generation, image captioning, and structured-text tasks across different supervision levels.

  • Takeaways & Limitations

    CM3 trains on an order of magnitude fewer unique images than DALL-E and therefore does not generate fictional images well.

Abstract

from arXiv · show

We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked language-image models on large-scale web and Wikipedia articles, where each document contains all of the text, hypertext markup, hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they appear in the original HTML source (before masking). The resulting CM3 models can generate rich structured, multi-modal outputs while conditioning on arbitrary masked document contexts, and thereby implicitly learn a wide range of text, image, and cross modal tasks. They can be prompted to recover, in a zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM. We set the new state-of-the-art in zero-shot summarization, entity linking, and entity disambiguation while maintaining competitive performance in the fine-tuning setting. We can generate images unconditionally, conditioned on text (like DALL-E) and do captioning all in a zero-shot setting with a single model.

1 INTRODUCTION

CM3 extends generative sequence modeling to structured documents containing text, hypertext, links, and images. Its causally masked objective supports broad zero-shot multimodal capabilities and improves several structured-text tasks.

  • Motivation: CM3 models structured multimodal documents by jointly representing formatted hypertext, hyperlinks, and images in their document context.The training corpus includes close to a terabyte of web-based data and image tokens from VQVAE-GAN.
  • Approach: Causally masked modeling generates tokens left to right while relocating a small number of masked long spans to the sequence end.This combines full generative modeling with bidirectional context for infilling.
  • Models: CM3 models are trained at 2.7 billion and 13 billion parameters, named CM3-Medium and CM3-Large.
  • Results: CM3 performs a wide range of zero-shot unimodal and cross-modal tasks, including non-trivial image generation similar to DALL-E.
  • Results: The models improve over state-of-the-art zero-shot summarization, entity linking, and entity disambiguation, while fine-tuning sets new state of the art for entity linking and entity disambiguation.

2 CAUSALLY MASKED OBJECTIVE

The causally masked objective combines causal generation with optional bidirectional context. It masks a few long spans and generates them later, enabling generative infilling while retaining left-to-right modeling.

  • Motivation: Causal language models provide left-only context, whereas masked objectives provide bidirectionality but decode only a subset of input tokens during training.
  • Objective: CM3 selects a small number of non-overlapping spans in a document and masks them for generation at the end of the sequence.The number of masks is sampled from a clipped Poisson distribution, and span boundaries are sampled uniformly.
  • Objective: The objective combines per-token causal generation with optional bidirectionality tailored to prompting.
  • Objective: CM3 excludes loss contributions from randomly placed mask tokens because their locations carry no information about the underlying sequence.
  • Implications: The resulting objective supports causal language modeling while allowing bidirectional context when needed for prompting.

3 CM3

CM3 extends structured-document pretraining to multimodal HTML data by adding images and links to a decoder-only causally masked model. The paper reports healthy scaling while defining data and modeling constraints that shape its scope.

  • Model and data: CM3 extends structured-document pretraining to full document structure, including images and hypertext links, using a decoder-only causally masked objective.
  • Data: HTML documents are simplified by removing low-semantic-value elements and retaining structurally relevant content.
  • Data: The dataset uses a filtered subset of Common Crawl News and all English Wikipedia rather than all Common Crawl because of documented ethical concerns.
  • Data: Images are resized to 256x256 pixels, tokenized with VQVAE-GAN, and represented by 256 image tokens inserted into the HTML source.
  • Training: CM3 is trained at 125M, 800M, 2.7B, and 13B parameters, with downstream evaluation using the 2.7B and 13B models.
  • Scaling laws: The multimodal setting complicates traditional scaling assumptions because text and image tokens have different distributions and images can occur at unrestricted locations.
  • Scaling laws: The trained models show healthy scaling without pathological cases, while a deeper analysis of causally masked scaling laws remains outside the paper’s scope.

4 ZERO/FEW-SHOT PROMPTING

CM3 uses structured multimodal prompting to perform zero-shot image generation, image infilling, captioning, entity disambiguation, entity linking, and summarization. Across these tasks, the model produces coherent outputs and reaches strong or state-of-the-art results, while remaining subject to data- and representation-dependent limitations.

  • 4.1.1 UNCONDITIONAL IMAGE GENERATION: CM3 can generate unconditional images from prompts that either let it self-caption before image tokens or directly request image tokens.The self-captioning prompt yields qualitative improvements, and random samples are shown without cherry-picking.
  • 4.1.1 UNCONDITIONAL IMAGE GENERATION: CM3-Large approaches modern GAN FID performance on 30k unconditional MS-COCO samples without textual conditioning or task-specific optimization.The evaluation follows the MS-COCO FID methodology and compares the generated distribution with the benchmark.
  • 4.1.2 IMAGE IN-FILLING: CM3-Large produces semantically coherent image infills without text and substantially improves them when conditioned on ground-truth text.The causally masked objective permits conditioning contiguous image sections on surrounding context, with text-conditioned qualitative results shown separately.
  • 4.2.2 CONDITIONAL IMAGE GENERATION: CM3 generates recognizable images from text prompts, but can omit specified objects or misrender detailed content such as a sheep’s face.For each prompt, 32 samples are generated and CLIP selects the top four.
  • 4.2.2 CONDITIONAL IMAGE GENERATION: CM3 uses about an order of magnitude fewer unique images than DALL-E and performs poorly on fictional images because its data comes from news and Wikipedia.Larger candidate pools for CLIP selection qualitatively fix some of these issues; quantitatively, conditional CM3-Large approaches DALL-E’s FID performance while using fewer data.
  • 4.2.3 CAPTIONING: CM3 performs zero-shot image captioning through alt or title prompts, producing non-trivial semantically coherent captions.Caption-Beam searches over prompts, whereas Caption-CLIP selects among 128 candidates; discrete image tokens can lose texture and blur embedded text.
  • 4.3.1 ENTITY DISAMBIGUATION / 4.3.2 ENTITY LINKING: CM3 shows strong zero-shot named entity disambiguation and fine-tuned HTLM-Large reaches a new state of the art across benchmarked entity-linking datasets.The entity-linking evaluation includes in-domain and out-of-domain settings, with CM3-Large also achieving a new state of the art.
  • 4.3.3 SUMMARIZATION: CM3-Large achieves new state-of-the-art zero-shot summarization on three datasets despite less pretraining text and prompts tuned for HTLM.Performance degrades on Reddit-TIFU, which the authors attribute to missing comparable summarization content in CM3’s pretraining data.

5 FINE-TUNING

CM3 is evaluated as a transferable representation for fine-tuning across GLUE, using standard, adversarial, and prompt-based settings. Both model sizes are competitive with parameter-matched T5, and HTML prompting improves performance without making multimodal or causally masked pretraining detrimental.

  • Fine-tuning setup: CM3 is evaluated on GLUE using standard fine-tuning, adversarial fine-tuning, and prompt-based fine-tuning settings.The comparison includes T5, RoBERTa, and HTLM-derived models.
  • Results: Both CM3 models are competitive with T5 at the same parameter setting.
  • Results: HTML prompts containing natural-language task utterances non-trivially improve end-finetuning performance.
  • Results: Causally masked language modeling does not harm fine-tunable representations, and jointly modeling image tokens is likewise not detrimental.

6 ETHICAL CONSIDERATIONS

The paper probes CM3 for gender and racial bias and for harmful image generation. CM3 shows less measured bias than competing models in the reported experiments, while its image-generation behavior is difficult to steer toward harmful content.

  • Bias evaluation: CM3 is evaluated for gender and racial bias using GWEAT and GSEAT because it generates both language and visual content.
  • Bias results: Both CM3 models show significantly less bias than competing models in the reported gender and race experiments.The authors associate this result with training on Wikipedia and recent CC-NEWS articles, while noting that CM3-Large still shows some bias.
  • Image safety: The authors qualitatively find it difficult to prompt CM3 to produce harmful or objectionable images.They also report that limited distinctive-feature generation from VQVAE-GAN benefits privacy preservation.

7 RELATED WORK

CM3 extends HTLM with causal masking, image tokens, and larger scale to unify structured hypertext and multimodal generation. Its capabilities overlap with DALL-E for image generation and GENRE for autoregressive entity linking and disambiguation.

  • Relation to HTLM: CM3 extends HTLM by adding the causally masked objective, VQVAE-GAN image tokens, and more than an order-of-magnitude scaling.
  • Relation to DALL-E: CM3’s conditional and unconditional image generation is most similar in approach to DALL-E’s left-to-right modeling of text and VQ-VAE visual tokens.
  • Relation to GENRE: CM3’s autoregressive entity linking and disambiguation connects to GENRE’s use of autoregressive modeling for those tasks.
  • Related multimodal methods: Tokenizing non-discrete modalities for sequence modeling has also been explored in DALL-E, Jukebox, and vq-wav2vec.

8 CONCLUSION

CM3 combines a causally masked objective with broad zero-shot multimodal capabilities and strong fine-tuning transfer. A single model supports image and text generation tasks while achieving strong results on language benchmarks.

  • CM3 uses a causally masked objective that supports full generative modeling with bidirectional context.
  • A single CM3 model can be prompted for image generation, image captioning, and unconditional image generation.
  • CM3 improves over state-of-the-art zero-shot summarization, entity linking, and entity disambiguation.
  • Fine-tuned CM3 reaches state-of-the-art entity linking and entity disambiguation while remaining highly competitive with T5 on GLUE.

A.1 MODEL ARCHITECTURE

CM3-Medium and CM3-Large use dense Transformer architectures corresponding to 2.7B and 13B parameter models. Their FairSeq architecture designations are tabulated in the appendix.

  • CM3-Medium and CM3-Large use the same architectures as dense 2.7B and 13B models described by Artetxe et al.
  • The architecture description covers both CM3 model sizes using the corresponding dense-model designs.
  • The appendix includes a FairSeq architecture designation table for CM3 models.

A.2 UNIFORMITY OF VQVAE-GAN TOKENS

The appendix examines the distribution of VQVAE-GAN image tokens and reports somewhat clear uniformity across a 100k-token subset. Nearby material also lists experiment hyperparameters and related tables.

  • A histogram of image tokens in a 100k-token data subset shows somewhat clear uniformity in token usage.
  • Figure 7 presents the histogram of VQ-VAE-GAN tokens in the CM3 training dataset.
  • The surrounding appendix material references hyperparameters for GLUE and fine-tuning experiments.
Loading 2201.07520v1…